聲明：本模型復(fù)現(xiàn)筆記記錄自己學(xué)習(xí)過(guò)程，如果有錯(cuò)誤請(qǐng)各位老師批評(píng)指正。

本周復(fù)現(xiàn)了Wide&Deep Model、Wide Model、Deep Model、DeepFm Model、NFM Model

dataset：cretio數(shù)據(jù)集（選取了train 20000， test 5000）

Wide&Deep Model結(jié)合了Wide Model（LR）的記憶性和Deep Model的泛化能力，對(duì)于部署應(yīng)用來(lái)說(shuō)會(huì)提高用戶(hù)的獲取率和滿(mǎn)意度，但是Wide&Deep的Wide part，需要專(zhuān)業(yè)的人員來(lái)發(fā)掘某些特征之間的共現(xiàn)關(guān)系，這就需要大量專(zhuān)業(yè)人工參與。

DeepFm Model 對(duì)Wide&Deep Model做了改進(jìn)，對(duì)于Wide部分換成了FM Model具有了自動(dòng)的特征組合能力。

NFM Model：是對(duì)FM、FFM的模型改進(jìn)，F(xiàn)M、FFM其實(shí)就是一個(gè)二階的特征交叉，它們會(huì)受到組合爆炸問(wèn)題，限制了FM、FFM的表達(dá)能力。NFM 就利用DNN去擬合FM的二階交叉特征。如果NFM一階部分是線(xiàn)性模型，也可以看成是Wide&Deep的改進(jìn)，只不過(guò)在Deep部分加了特征交叉池化層。

Wide&Deep Model

Wide部分的輸入是原始特征（dense feature 和 經(jīng)過(guò)Embeddings 的sparse feature）和交叉積特征轉(zhuǎn)換數(shù)據(jù)

交叉積特征轉(zhuǎn)換數(shù)據(jù)計(jì)算為：

Deep部分的輸入是dense feature 和 經(jīng)過(guò)Embeddings 的sparse feature

class Linear(nn.Module):
 
    def __init__(self, input_dim):
        super(Linear, self).__init__()
        self.Linear = nn.Linear(in_features = input_dim, out_features = 1)
        
    def forward(self, x):
        return self.Linear(x)
    
class Dnn(nn.Module):

    def __init__(self, hidden_units, dropout = 0.):
        super(Dnn, self).__init__()
        self.dnn_network = nn.ModuleList([nn.Linear(layer[0],layer[1]) for layer in list(zip(hidden_units[:-1] , hidden_units[1:]))])
        self.dropout = nn.Dropout(p = dropout)
        
    def forward(self, x):
        
        for linear in self.dnn_network:
            x = linear(x)
            x = self.dropout(x)
            x = F.relu(x)
            
        return x

class WideandDeep(nn.Module):

    def __init__(self, feature_columns, hidden_units, dnn_dropout = 0.):
        super(WideandDeep, self).__init__()
        self.dense_feature_cols, self.sparse_feature_cols = feature_columns
        
        #embedding 
        self.embed_layers = nn.ModuleDict({
            'embed_' + str(i):nn.Embedding(num_embeddings = feat['feat_num'], embedding_dim = feat['embed_dim'])
            for i ,feat in enumerate(self.sparse_feature_cols)
        })
        
        hidden_units.insert(0, len(self.dense_feature_cols) + len(self.sparse_feature_cols) * self.sparse_feature_cols[0]['embed_dim'])
        self.dropout = nn.Dropout(p = dnn_dropout)
        self.dnn_network = Dnn(hidden_units,dnn_dropout)
        self.Linear = Linear(len(self.dense_feature_cols) + 1)
        self.final_linear = nn.Linear(hidden_units[-1]+ 1, 1)
        
    def forward(self, x):
        dense_inputs , sparse_inputs = x[:,:len(self.dense_feature_cols)], x[:,len(self.dense_feature_cols):]
        sparse_inputs = sparse_inputs.long()
        sparse_embeds = [self.embed_layers['embed_'+str(i)](sparse_inputs[:,i]) for i in range(sparse_inputs.shape[1])]
        sparse_embeds = torch.cat(sparse_embeds, axis = -1)
        
        dnn_input = torch.cat([sparse_embeds, dense_inputs], axis=-1)
        
        #n = 190000
        #a = [print(sparse_inputs[:,i].max()) for i in range(sparse_inputs.shape[1])]
        
        sparse_one_hot = [F.one_hot(sparse_inputs[:,i],sparse_inputs[:,i].max()+1)  for i in range(sparse_inputs.shape[1]) ]

        sparse_one_hot = torch.cat(sparse_one_hot,axis=-1)
        
        sparse_one_hot = torch.tensor(sparse_one_hot)
        #print(sparse_one_hot.shape)
        sparse_one_hot = sparse_one_hot.reshape(dense_inputs.shape[0],-1)
        c = torch.zeros(dense_inputs.shape[0] ,sparse_one_hot.shape[1] )
        for i in range(0,dense_inputs.shape[0]):
            c[i][1456]=1
            c[i][16123]=1
        
        cross = torch.prod(torch.pow(sparse_one_hot,c),axis = 1)
        cross = cross.reshape(-1,1)
        dense_input = torch.cat([dense_inputs,cross],axis=-1)
        
        #wide
        wide_out = self.Linear(dense_input) # (,14)
        #print(wide_out.shape)
        #deep
        deep_out = self.dnn_network(dnn_input)#(,64)
        #print(deep_out.shape)
        output = torch.cat([wide_out, deep_out] ,axis = -1 )
        
        outputs = F.sigmoid(self.final_linear(output))
        
        return outputs

重點(diǎn)難點(diǎn)：Wide部分的交叉積特征轉(zhuǎn)換數(shù)據(jù)不容易復(fù)現(xiàn)，我通過(guò)觀察數(shù)據(jù)，找到了一個(gè)組合特征。
這是實(shí)現(xiàn)交叉積特征轉(zhuǎn)換數(shù)據(jù)的主要代碼：
流程：所有sparse feature轉(zhuǎn)成one_hot編碼 ，設(shè)置C矩陣（函數(shù)中做指數(shù)），利用torch.pow和torch.prod完成函數(shù)運(yùn)算，最后拼接到wide的輸入。

		sparse_one_hot = [F.one_hot(sparse_inputs[:,i],sparse_inputs[:,i].max()+1)  for i in range(sparse_inputs.shape[1]) ]

        sparse_one_hot = torch.cat(sparse_one_hot,axis=-1)
        
        sparse_one_hot = torch.tensor(sparse_one_hot)
        #print(sparse_one_hot.shape)
        sparse_one_hot = sparse_one_hot.reshape(dense_inputs.shape[0],-1)
        c = torch.zeros(dense_inputs.shape[0] ,sparse_one_hot.shape[1] )
        for i in range(0,dense_inputs.shape[0]):
            c[i][1456]=1
            c[i][16123]=1
        
        cross = torch.prod(torch.pow(sparse_one_hot,c),axis = 1)
        cross = cross.reshape(-1,1)
        dense_input = torch.cat([dense_inputs,cross],axis=-1)
        
        #wide
        wide_out = self.Linear(dense_input) # (,14)

效果：batch_size= 256，數(shù)據(jù)小容易過(guò)擬合，加入了L2正則化（0.005），dropout（0.3）

Wide Mode

僅僅是LR，輸入數(shù)據(jù)是dense feature 和 經(jīng)過(guò)embedding層的sparse feature

class Linear(nn.Module):

    def __init__(self, feature_columns, dnn_dropout = 0.):
        super(Linear, self).__init__()
        
        self.dense_feature_cols,self.sparse_feature_cols = feature_columns
        #embedding
        self.embed_layers = nn.ModuleDict({
            'embed_'+str(i):nn.Embedding(num_embeddings = feat['feat_num'], embedding_dim = feat["embed_dim"])
             for i, feat in enumerate(self.sparse_feature_cols)
        })
        input_dim = len(self.dense_feature_cols) + len(self.sparse_feature_cols) * self.sparse_feature_cols[0]['embed_dim']
        self.Linear = nn.Linear(in_features = input_dim, out_features = 1)
        
    def forward(self, x):
        dense_inputs ,sparse_inputs = x[:,:len(self.dense_feature_cols)],x[:,len(self.dense_feature_cols):]
        sparse_inputs =  sparse_inputs.long()
        sparse_embeds = [self.embed_layers['embed_'+str(i)](sparse_inputs[:,i]) 
                        for i in range(sparse_inputs.shape[1])]
        sparse_embeds = torch.cat(sparse_embeds, axis = -1)
        
        wide_inputs = torch.cat([dense_inputs,sparse_embeds] , axis = -1)
        outputs = self.Linear(wide_inputs)
        return F.sigmoid(outputs)

效果： batch_size= 256，L2正則化（0.005），dropout（0.3）

Deep Model:

class Dnn(nn.Module):
  
    def __init__(self, feature_columns, hidden_units, dnn_dropout = 0.):
        super(Dnn, self).__init__()
        
        self.dense_feature_cols,self.sparse_feature_cols = feature_columns
        #embedding
        self.embed_layers = nn.ModuleDict({
            'embed_'+str(i):nn.Embedding(num_embeddings = feat['feat_num'], embedding_dim = feat["embed_dim"])
             for i, feat in enumerate(self.sparse_feature_cols)
        })
        hidden_units.insert(0,len(self.dense_feature_cols) + len(self.sparse_feature_cols) * self.sparse_feature_cols[0]['embed_dim'])
        self.dnn_network = nn.ModuleList([nn.Linear(layer[0],layer[1]) for layer in list(zip(hidden_units[:-1], hidden_units[1:]))])
        self.dropout = nn.Dropout(p = dnn_dropout)
        self.final_linear = nn.Linear(hidden_units[-1], 1 )
        
    def forward(self, x):
        dense_inputs ,sparse_inputs = x[:,:len(self.dense_feature_cols)],x[:,len(self.dense_feature_cols):]
        sparse_inputs =  sparse_inputs.long()
        sparse_embeds = [self.embed_layers['embed_'+str(i)](sparse_inputs[:,i]) 
                        for i in range(sparse_inputs.shape[1])]
        sparse_embeds = torch.cat(sparse_embeds, axis = -1)
        
        dnn_inputs = torch.cat([dense_inputs,sparse_embeds] , axis = -1)
        
        for layer in self.dnn_network:
            dnn_inputs = layer(dnn_inputs)
            dnn_inputs = F.relu(dnn_inputs)
        
        output =  F.sigmoid(self.final_linear(dnn_inputs))
        return output

效果：batch_size= 256，L2正則化（0.005），dropout（0.3）

Wide&Deep Model VS Wide Model VS Deep Model

Wide&Deep Model 、 Wide Model 、 Deep Model 訓(xùn)練損失比較

Wide&Deep Model 、 Wide Model 、 Deep Model 訓(xùn)練AUC比較

Wide&Deep Model 、 Wide Model 、 Deep Model 測(cè)試集損失比較

Wide&Deep Model 、 Wide Model 、 Deep Model 測(cè)試集AUC比較

結(jié)論：Wide&Deep結(jié)合了記憶性和泛化性確實(shí)比樸素LR和樸素DNN效果好一點(diǎn)，論文中給出的三者的AUC幾乎一樣，文章中給出說(shuō)明是利用cross-product feature去增加在線(xiàn)用戶(hù)獲取率，提高1%的效率對(duì)于大的互聯(lián)網(wǎng)公司來(lái)說(shuō)就可以獲得巨大收益。

DeepFM

DeepFM對(duì)于Wide&Deep模型的改進(jìn)是用FM替換了原來(lái)的Wide部分，加強(qiáng)了淺層網(wǎng)絡(luò)部分特征組合的能力。同時(shí)，F(xiàn)M部分和DNN部分共享相同的EMbedding層。
左側(cè)的FM部分對(duì)不同的特征域的EMbedding進(jìn)行了兩兩交叉，即將Embedding向量當(dāng)作原FM中的特征向量V。

class FM(nn.Module):

    def __init__(self, latent_dim, fea_num):
        
        super(FM, self).__init__()
        
        self.latent_dim = latent_dim
        
        self.w0 = nn.Parameter(torch.zeros([1,]))
        self.w1 = nn.Parameter(torch.rand([fea_num, 1]))
        self.w2 = nn.Parameter(torch.rand([fea_num, latent_dim]))
        
    def forward(self, inputs):   
        # 一階
        first_order = self.w0 + torch.mm(inputs, self.w1)      
        # 二階 
        second_order = 1/2 * torch.sum(torch.pow(torch.mm(inputs, self.w2), 2) - torch.mm(torch.pow(inputs,2), torch.pow(self.w2, 2)),dim = 1,keepdim = True)        
        
        return first_order + second_order 

class Dnn(nn.Module):
    def __init__(self, hidden_units, dropout=0.):
        super(Dnn, self).__init__()
        
        self.dnn_network = nn.ModuleList([nn.Linear(layer[0], layer[1]) for layer in list(zip(hidden_units[:-1], hidden_units[1:]))])
        self.dropout = nn.Dropout(dropout)
    
    def forward(self, x):  
        for linear in self.dnn_network:
            x = linear(x)
            x = F.relu(x)    
        x = self.dropout(x) 
        return x

class DeepFM(nn.Module):
    def __init__(self, feature_columns, hidden_units, dnn_dropout=0.):
    
        super(DeepFM, self).__init__()
        self.dense_feature_cols, self.sparse_feature_cols = feature_columns
        
        # embedding
        self.embed_layers = nn.ModuleDict({
            'embed_' + str(i): nn.Embedding(num_embeddings=feat['feat_num'], embedding_dim=feat['embed_dim'])
            for i, feat in enumerate(self.sparse_feature_cols)
        })
        

        self.fea_num = len(self.dense_feature_cols) + len(self.sparse_feature_cols)*self.sparse_feature_cols[0]['embed_dim']
        hidden_units.insert(0, self.fea_num)
        
        self.fm = FM(self.sparse_feature_cols[0]['embed_dim'], self.fea_num)     
        self.dnn_network = Dnn(hidden_units, dnn_dropout)
        self.nn_final_linear = nn.Linear(hidden_units[-1], 1)
    
    def forward(self, x):
        dense_inputs, sparse_inputs = x[:, :len(self.dense_feature_cols)], x[:, len(self.dense_feature_cols):]
        sparse_inputs = sparse_inputs.long()       
        sparse_embeds = [self.embed_layers['embed_'+str(i)](sparse_inputs[:, i]) for i in range(sparse_inputs.shape[1])]
        sparse_embeds = torch.cat(sparse_embeds, dim=-1)
        

        x = torch.cat([sparse_embeds, dense_inputs], dim=-1)
        # Wide
        wide_outputs = self.fm(x)
        # deep
        deep_outputs = self.nn_final_linear(self.dnn_network(x))
 
        outputs = F.sigmoid(torch.add(wide_outputs, deep_outputs))
        
        return outputs

效果：batch_size= 256，L2正則化（0.006），dropout（0.4）

DeepFM VS Wide&Deep

結(jié)論： DeepFM具有自動(dòng)進(jìn)行特征組合的能力，效果還是比wide&deep效果要好。但是epoch=4前 wide&deep優(yōu)于DeepFM的原因：我認(rèn)為是FM的隨機(jī)初始化導(dǎo)致的auc比wide&deep低。

NFM Model

NFM是對(duì)FM、FFM的模型改進(jìn)，F(xiàn)M、FFM其實(shí)就是一個(gè)二階的特征交叉，它們會(huì)受到組合爆炸問(wèn)題，限制了FM、FFM的表達(dá)能力。

之前看到過(guò)一句話(huà)“深度學(xué)習(xí)網(wǎng)絡(luò)理論上有擬合任何復(fù)雜函數(shù)的能力”，就是用DNN來(lái)擬合表達(dá)能力更強(qiáng)的函數(shù)。

NFM 就利用DNN去擬合FM的二階交叉特征。

如果NFM一階部分是線(xiàn)性模型，也可以看成是Wide&Deep的改進(jìn)，只不過(guò)在Deep部分加了特征交叉池化層。
模型的其他層都與前面一樣含義，新添加的是Bi-interaction Pooling layer。

Bi-interaction Pooling layer:

其中前一層中的Embeding層Vx = [ x1v1,x2v2,…,xnvn ]

在進(jìn)行兩兩Embedding向量的元素積操作后，對(duì)交叉特征向量取和，得到池化層的輸出向量。

然后再把該向量輸入到下一層的DNN中，進(jìn)行進(jìn)一步的交叉特征。

注意： 設(shè)計(jì)模型時(shí)不要忘了一階函數(shù)。

class Dnn(nn.Module):
    def __init__(self, hidden_units, dropout=0.):
        
        super(Dnn, self).__init__()
        
        self.dnn_network = nn.ModuleList([nn.Linear(layer[0], layer[1]) for layer in list(zip(hidden_units[:-1], hidden_units[1:]))])
        self.dropout = nn.Dropout(dropout)
    
    def forward(self, x):  
        for linear in self.dnn_network:
            x = linear(x)
            x = F.relu(x)    
        x = self.dropout(x) 
        return x

class NFM(nn.Module):
    def __init__(self, feature_columns, hidden_units, dnn_dropout=0.):
        
        super(NFM, self).__init__()
        self.dense_feature_cols, self.sparse_feature_cols = feature_columns
        
        # embedding
        self.embed_layers = nn.ModuleDict({
            'embed_' + str(i): nn.Embedding(num_embeddings=feat['feat_num'], embedding_dim=feat['embed_dim'])
            for i, feat in enumerate(self.sparse_feature_cols)
        })
        
       
        self.fea_num = len(self.dense_feature_cols) + self.sparse_feature_cols[0]['embed_dim']
        hidden_units.insert(0, self.fea_num)
        
        self.bn = nn.BatchNorm1d(self.fea_num)     
        self.dnn_network = Dnn(hidden_units, dnn_dropout)
        input_dim  = len(self.dense_feature_cols)  + len(self.sparse_feature_cols) * self.sparse_feature_cols[0]['embed_dim']
        self.Linear = nn.Linear(input_dim,32)
        self.nn_final_linear = nn.Linear(hidden_units[-1], 1)
    
    def forward(self, x):
        dense_inputs, sparse_inputs = x[:, :len(self.dense_feature_cols)], x[:, len(self.dense_feature_cols):]
        sparse_inputs = sparse_inputs.long()      
        sparse_embeds = [self.embed_layers['embed_'+str(i)](sparse_inputs[:, i]) for i in range(sparse_inputs.shape[1])]
        L_sparse_embeds = torch.cat( sparse_embeds,axis = -1)
        sparse_embeds = torch.stack(sparse_embeds)     
        sparse_embeds = sparse_embeds.permute((1, 0, 2))
       
        embed_cross = 1/2 * (
            torch.pow(torch.sum(sparse_embeds, dim=1),2) - torch.sum(torch.pow(sparse_embeds, 2), dim=1)
        )  
        
        #wide
        lr_input = torch.cat([dense_inputs,L_sparse_embeds],axis = -1)
        lr_output = self.Linear(lr_input)
        #print(lr_output.shape)
        # deep
        x = torch.cat([embed_cross, dense_inputs], dim=-1)
        
        x = self.bn(x)
        
        dnn_outputs = self.nn_final_linear(self.dnn_network(x)+ lr_output)
        #joint 
        outputs = F.sigmoid(dnn_outputs )
        
        return outputs

效果：batch_size= 256，L2正則化（0.003），dropout（0.25）

NFM VS DeepFM VS Wide&Deep

Wide&Deep、DeepFM模型是使用的拼接操作，而不是Bi-Interaction.
拼接操作最大的缺點(diǎn)是他并沒(méi)有考慮任何特征組合信息，全部依靠后面的DNN去學(xué)習(xí)特征組合，但是DNN的學(xué)習(xí)優(yōu)化非常困難。
使用Bi-Interaction就考慮到了二階組合特征，使得輸入到DNN層中數(shù)據(jù)包含更多的信息，減輕了DNN的學(xué)習(xí)困難。