TL;DR: "The model you use is not the model that was pretrained." An 87-slide lecture that maps post-training — turning a base model into a useful policy — across six stages: imitate, compare, explore, verify, transfer, anticipate.
Title: Post-Training LLMs (Kawin Ethayarajh, AI and Economics Summer Institute 2026)
URL:
Key points
📝 SFT: teach by demonstration. Can imitate whole trajectories (plan, tools), but it's fundamentally imitative
⚖️ Offline preference optimization: learn from win/loss. DPO drops the reward model; KTO learns per-outcome, no SFT needed
🎲 Online RL: the current policy explores. REINFORCE/PPO/GRPO; rewards are mostly sequence-level
✅ RLVR: replace the reward model with a checker. Binary pass/fail is enough; scales more predictably than RLHF
🧪 Environments: an RL env is task + data + interface + tests. The bottleneck shifts from examples to environments
🔁 On-policy distillation: the teacher scores the student's own rollout. Copies an RL policy in 7–10x fewer steps
🌐 World adaptation: environments adapt back to agents. "Mecha-nudges" raise machine-readability
A systematic map, threaded with an economist's lens.
#
LLM# #
PostTraining#