You Jiacheng
@YouJiacheng
가입 August 2015
2.4K 팔로잉 중 12.6K 팬
Actually this argument is not limited to linear policy.
Consider the gradient w.r.t. logits.
s = ∇_z log p(a) = e_a - p
\bar{s} = E_{a~q}[e_a - p] = (q - p)
centered score s - \bar{s} = e_a - q
Thus, score centering is a STE: z_p + sg(z_q - z_p).
더 보기
@YouJiacheng consider linear softmax policy. the proposed correction does nothing and simply calculate the gradient under the sampling distribution.
