How to use dnnl_rnn_flags_diff_weights_overwrite in RNN LSTM primitives creation?

[lacak-sk]

There is stated in documentation, that backward primitive ACCUMULATES diff weights (diff_weights_layer, diff_weights_iter) and bias (diff_bias): "The RNN primitive backward pass accumulates gradients to its weight outputs. Hence, these tensors should be properly initialized to zero before their first use, and can be reused across calls to accumulate gradients if need be. This behavior can be altered by the RNN flag diff_weights_overwrite. If this flag is set weight gradients will be initialized by zeros by the RNN primitive."

There is argument "flags" in dnnl_lstm_forward_primitive_desc_create() and also in dnnl_lstm_backward_primitive_desc_create(), but this argument is described as "Unused."

So it is unclear to me how to set this flag (dnnl_rnn_flags_diff_weights_overwrite) in C API?

Does I understad it correctly, that I MUST set this flag if I execute multiple forward/backward passes for multiple samples and I want to get diffs for actual pass and use these diffs to update weights/biases and then process next sample and so on ...?

And these diffs computed by backward pass are SUMMED over time steps (and batch)? So if I have 200 tokens in sequence (200 time steps) I must update weights by: [diffs] * [LR]/200, where LR is learning rate?
(in general: [weights] = [weights] - [diff_weights] * [learning_rate] / ([batch_size] * [time_steps]) ?)

[mgouicem]

Hi @lacak-sk

There is argument "flags" in dnnl_lstm_forward_primitive_desc_create() and also in dnnl_lstm_backward_primitive_desc_create(), but this argument is described as "Unused."

So it is unclear to me how to set this flag (dnnl_rnn_flags_diff_weights_overwrite) in C API?

This is a header documentation glitch, you can set the flag argument to dnnl_rnn_flags_diff_weights_overwrite, and it will impact gradient accumulation behavior as specified in doc.

Regarding the behavior:

• weights gradients are accumulated across batch and timesteps within a primitive execution, independently of the flags set
• gradients are accumulated to diff_weights buffer across multiple calls to primitive execution by default. In other words, diff_weights += rnn_bwd(batch, timesteps)
• if dnnl_rnn_flags_diff_weights_overwrite is passed, gradients are not accumulated across multiple calls to primitive execution. In other words, diff_weights = rnn_bwd(batch, timesteps).

[lacak-sk]

Thanks for the reply, now it is much clearer.

Can you update documentation of "flag" parameter in dnnl_lstm_backward_primitive_desc_create() for example from:

• /// @param flags Unused.
  to:
• /// @param flags RNN flags (@ref dnnl_rnn_flags_t).
  (I assume that flag is used ONLY is backward primitive creation and not in forward primitive creation, right?)

As far as this flag was added in 3.1 version, how can I workaround it in 3.0 version of oneDNN? Then I must zero-out weights gradients as separate step in primitive execution chain (before lstm_backward_primitive) in backward propagation ...

And my assumption about updating weight using diff_weights as:
[weights] -= [diff_weights] * [learning_rate] / ([batch_size] * [time_steps])
is then right?

[mgouicem]

Can you update documentation of "flag" parameter in dnnl_lstm_backward_primitive_desc_create() for example from:

• /// @param flags Unused.
  to:
• /// @param flags RNN flags (@ref dnnl_rnn_flags_t).
  (I assume that flag is used ONLY is backward primitive creation and not in forward primitive creation, right?)

Will do. Thanks.

As far as this flag was added in 3.1 version, how can I workaround it in 3.0 version of oneDNN? Then I must zero-out weights gradients as separate step in primitive execution chain (before lstm_backward_primitive) in backward propagation ...

Yes this is correct.

And my assumption about updating weight using diff_weights as:
[weights] -= [diff_weights] * [learning_rate] / ([batch_size] * [time_steps])
is then right?

That seems correct, assuming you are doing backpropagation through time.
You will likely find more meaningful answer in forums with experts on the topic (pytorch/tensorflow groups, stackoverflow, ...).