🎯 The Hard Part of RL Is Not Optimization — It's Constructing the Target Distribution One decomposition reframes post-training RL: RL = Local Optimum Construction + Projection. Build a local distribution better than the current model, then project the model onto it. Zhihu contributor dung defender argues almost all the real difficulty sits in the first step — while most RL research polishes the second. He also reports a reward-free construction of that target that, in his own experiments, improves the model even from a near-zero pass@K — beating a FlowGRPO-style baseline. The method is undisclosed and the results self-reported. 1️⃣ Two arrows: improve the distribution, then project onto it Skip the parameters entirely and the learning process becomes two arrows: 🔹 Local improvement: from the current distribution, build a better local one — IGO-style, reweighting probability mass toward higher-utility samples. 🔹 Projection: make the neural network match that better distribution. In this frame, policy gradient is not RL itself — just one implementation of the local improvement. PG, TRPO, PPO and GRPO all fit the same form. 2️⃣ Most RL machinery polishes the easier half Variance reduction, baselines, advantage normalization, clipping, trust regions — nearly all of it serves the projection arrow: given a target, update stably. But a beautifully engineered optimizer can only fit a poorly constructed target more precisely. The upstream question is the fundamental one: do we actually know what the next, better distribution should be? 3️⃣ SFT vs RL is an information gap 🔹 SFT: the dataset is the target distribution. You only project — a luxurious setting. 🔹 RL: you get samples from your own model plus a compressed scalar like 0.7, and must infer the target from that. A scalar reward says which sample is better. It does not say what a better sample looks like — exactly the information the target distribution requires. In his reading, the whole "does RL really help LLM post-training" debate reduces to one question: can you construct the target at all? If good trajectories can't be sampled and values can't be estimated, no algorithm saves you. 4️⃣ Reward is optional — and maybe the bottleneck Reward is merely one way to construct the target: a reallocation of probability mass. But "good" in complex generation is high-dimensional — semantics, reasoning, visual quality, consistency, style. Compressing all of that into one scalar, then hoping the gradient recovers the right direction, is a massive information bottleneck. His closing line: RL is not fundamentally about gradients. It is about constructing a better local distribution and projecting onto it. The question worth asking is not how to compute prettier gradients, but what that better distribution actually is. 🔗 Full Reading: 🔗Author's IGO-form derivations (earlier post): #ReinforcementLearning# #PostTraining# #LLM# #GRPO# #GenerativeModels# #AIResearch#