1.此處補充一下遷移學習，實際在比如構建一個圖像識別應用過程中，很少有人會直接隨機初始化權重，且很難有大量數(shù)據(jù)來重新訓練一個模型，相反的，我們會使用一個使用大批量數(shù)據(jù)訓練好的卷積神經(jīng)網(wǎng)絡來訓練。早期的卷積層提取低級特征，往后的卷積層提取高級的特征。這意味著只要任務接近我們就可以添加少量的數(shù)據(jù)來微調，就可以實現(xiàn)任務的遷移。

2.循環(huán)卷積網(wǎng)絡受生物記憶啟發(fā)。RNN，此外RNN的加強版有LSTM,GRU。記憶力更強^^.

3.普通的神經(jīng)網(wǎng)絡數(shù)據(jù)是單向傳遞的，而RNN是循環(huán)傳遞的，輸入x經(jīng)過hidden得到y(tǒng)，而hidden的輸出結果h需要作為下次輸入的一部分，循環(huán)傳遞。

4.RNN的本質是tanh（xtht-1）

5.LSTM通過門來控制信息（記憶）的去留來解決梯度消失的問題

下面實現(xiàn)一個RNN讓三角函數(shù)sin去預測cos的值?

import numpy as np
import matplotlib.pyplot as plt
import torch
from torch import nn, optim
steps = np.linspace(0, np.pi*2, 100, dtype=np.float32)
input_x = np.sin(steps)
target_y = np.cos(steps)
plt.plot(steps, input_x, 'b-', label='input:sin')
plt.plot(steps, target_y, 'r-', label='target:cos')
plt.legend(loc='best')
plt.show()

# 定義一個LSTM
class LSTM(nn.Module):
    def __init__(self,INPUT_SIZE):
        super(LSTM,self).__init__()

        self.lstm = nn.LSTM(
            input_size=INPUT_SIZE,
            hidden_size=20,
            # 表示輸入輸出的第一維為batch_size
            batch_first=True
        )
        self.out = nn.Linear(20,1)

    # 隱藏向量h_state,c_state
    def forward(self, x, h_state,c_state):
        r_out,(h_state,c_state) = self.lstm(x,(h_state,c_state))
        outputs = self.out(r_out[0,:]).unsqueeze(0)
        return outputs,h_state,c_state

    def InitHidden(self):
        h_state = torch.randn(1,1,20)
        c_state = torch.randn(1,1,20)
        return h_state,c_state

lstm = LSTM(INPUT_SIZE=1)
optimizer = torch.optim.Adam(lstm.parameters(), lr=0.001)
loss_func = nn.MSELoss()
h_state,c_state = lstm.InitHidden()

plt.figure(1, figsize=(12,5))
plt.ion()

for step in range(600):
    start, end = step*np.pi,(step+1)*np.pi
    steps = np.linspace(start, end, 100, dtype=np.float32)
    x_np = np.sin(steps)
    y_np = np.cos(steps)
    x = torch.from_numpy(x_np).unsqueeze(0).unsqueeze(-1)
    y = torch.from_numpy(y_np).unsqueeze(0).unsqueeze(-1)
    prediction,h_state,c_state =lstm(x, h_state,c_state)
    h_state = h_state.detach()
    c_state = c_state.detach()
    loss = loss_func(prediction, y)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    plt.plot(steps,y_np.flatten(), 'r-')
    plt.plot(steps, prediction.data.numpy().flatten(),'b-')
    plt.draw();plt.pause(0.05)

plt.ioff()
plt.show()