You Jiacheng
@YouJiacheng
Joined August 2015
2.4K Following 12.6K Followers
Actually this argument is not limited to linear policy.
Consider the gradient w.r.t. logits.
s = ∇_z log p(a) = e_a - p
\bar{s} = E_{a~q}[e_a - p] = (q - p)
centered score s - \bar{s} = e_a - q
Thus, score centering is a STE: z_p + sg(z_q - z_p).
Show more
@YouJiacheng consider linear softmax policy. the proposed correction does nothing and simply calculate the gradient under the sampling distribution.
