I have built a custom peephole lstm, and I want to imitate the dropout part in the already built in nn.lstm. So, how to add the dropout like what this intialization of this lstm, nn.LSTM(input_size, hidden_size, dropout=0.3), do? I have an idea of how to do it, which is by just applying a normal dropout just before returning the output, like this:
# init method
self.dropout = nn.Dropout(0.3)
# forward method
hidden_seq = self.dropout(hidden_seq)
return hidden_seq, (h_t, c_t)
I just want to make sure that this is the right way. If not what to do?
nn.LSTM(... dropout=0.3) applies a Dropout layer on the outputs of each LSTM layer except the last layer. You can have multiple stacked layers by passing parameter num_layers > 1. If you want to add a dropout to the final layer (or if LSTM has only one layer), you have to add it as you are doing now.
If you want to replicate what LSTM dropout does (which is only in case of multiple layers), you can stack LSTM layers manually and add a dropout layer in between.