You Jiacheng
@YouJiacheng
参加 August 2015
2.4K フォロー中 12.6K ファン
Actually this argument is not limited to linear policy.
Consider the gradient w.r.t. logits.
s = ∇_z log p(a) = e_a - p
\bar{s} = E_{a~q}[e_a - p] = (q - p)
centered score s - \bar{s} = e_a - q
Thus, score centering is a STE: z_p + sg(z_q - z_p).
もっと見る
@YouJiacheng consider linear softmax policy. the proposed correction does nothing and simply calculate the gradient under the sampling distribution.
