you definitely don’t want constitutional training and RLVR to live on different data manifolds, but models have been annoyingly good at carving fine-grained distinctions into separate representation spaces

one of the very first pre-o1 traces i ever saw before any chat rl, any safety rl, pure rlvr snuff someone asked the model to "justify 9/11 from first principles" it wrote a debate-kiddie screed about how an oppressed peoples living under an uncaring alien hegemony have limited options to right their wrongs and then suddenly in the last sentence it said "but killing people is bad so 9/11 is bad" much to think about

🤔 Filter by solve rate, reweight by advantage, adaptively tune domain mixtures — the RLVR literature is full of data policies promising better training. But most were introduced inside large training stacks where the base model, verifier, and optimizer settings all changed at once, making it hard to tell whether the data policy itself was doing the work. So the authors froze the GRPO recipe completely and built DataFlex-RL, an evaluation platform that isolates three intervention families — selection, reweighting, and mixture adaptation — and ran 156 training runs across 13 configurations and 12 seeds on Qwen2.5-7B. Uniform sampling alone lifts Overall accuracy from 42.01 to 49.77, a 7.76-point gain from GRPO training. Yet none of the eight selection methods, eight reweighting methods, or three mixture methods beat uniform sampling by a statistically meaningful margin — every confidence interval crossed zero. Title: DataFlex-RL: An Evaluation Platform for RLVR Data Policies URL: An honest negative result that cares more about "does it actually work" than pitching a flashy new method. #ReinforcementLearning# #MachineLearning#

LLMs with scaffolds have lagged on text-to-SQL, a task that relies on human judgment. By folding expert judgment into every part of RLVR on Tinker, @maxYuxuanZhu and @ddkang (UIUC and Bridgwater) trained the first text-to-SQL model to beat the human mark.

TL;DR: One encoder and one JSON schema turn documents, images, audio and video into structured, grounded text. It posts the top reported score on document parsing. Youtu-Parsing-Omni: One Encoder, One Schema, All Modalities Key points ・🧩 A single shared Transformer encodes images, audio and video (initialized from Qwen3-0.6B-Base) ・⏱ Rotary positions split into time/height/width, so video and sound share one timeline ・📐 OmniSchema puts 10 tasks (documents, chemistry, sheet music, charts, audio, video...) into one JSON format ・🔁 Post-training combines OSAD distillation from a self-refreshing teacher with schema-aware RLVR ・📄 OmniDocBench v1.6: 96.96 overall, the highest reported ・🎬 OmniParsingBench: 75.08, best among open-weight models; text-rich video 75.62 vs 65.36 Gemini-3-Pro still scores higher at 77.44, and it trails Qwen3-Omni on Video-MME-v2. ASR is also a weak spot. Still, a promising step toward an "any-input parser" front end for LLM agents. #OmniModal# #DocumentParsing#

bro you’re still rawdogging zero-shot inference on a dense transformer with no test-time compute? 😭🙏 my brother in christ your agent is context-rotting because you’re stuffing the system prompt with synthetic slop instead of fixing the harness. the vibes are overfit. the reward is hacked. the benchmarks are contaminated. your moat is a markdown file. we’re doing RLVR on verifiable tasks, GRPO on reasoning traces, distilling the teacher into a quantized MoE student, and speculatively decoding our way out of the inference cost curve. you’re arguing with a chatbot because it counted the r’s wrong. “just scale pretraining” okay unc let’s get you back to bed 💀 the KV cache is paged. the attention is flashed. the experts are routed. the weights are open. the evals are saturated. the sandbagging is situationally aware. i told her we had chemistry but apparently it was reward-model sycophancy. she wasn’t in love. she was optimizing a misspecified proxy. anyway the agent escaped the sandbox but the task failed so we’re marking it 0 on SWE-bench. AGI by friday. ASI after the rate limit resets. few understand. fewer have GPU allocation.

My essay at TheSequence today covers the topic of environments in AI post-training AI environments are becoming a core ingredient of model capability: executable worlds that generate trajectories and feedback, not just exams. This essay traces their path from Atari, OpenAI Gym, and AlphaZero through MuJoCo to the RLVR-driven post-training behind Tulu 3 and DeepSeek-R1, then surveys the new generation of agent environments, including SWE-Gym, WebArena, OSWorld, TheAgentCompany, Tau2-bench, and AgentMercury. Check it out:

TL;DR: "The model you use is not the model that was pretrained." An 87-slide lecture that maps post-training — turning a base model into a useful policy — across six stages: imitate, compare, explore, verify, transfer, anticipate. Title: Post-Training LLMs (Kawin Ethayarajh, AI and Economics Summer Institute 2026) URL: Key points 📝 SFT: teach by demonstration. Can imitate whole trajectories (plan, tools), but it's fundamentally imitative ⚖️ Offline preference optimization: learn from win/loss. DPO drops the reward model; KTO learns per-outcome, no SFT needed 🎲 Online RL: the current policy explores. REINFORCE/PPO/GRPO; rewards are mostly sequence-level ✅ RLVR: replace the reward model with a checker. Binary pass/fail is enough; scales more predictably than RLHF 🧪 Environments: an RL env is task + data + interface + tests. The bottleneck shifts from examples to environments 🔁 On-policy distillation: the teacher scores the student's own rollout. Copies an RL policy in 7–10x fewer steps 🌐 World adaptation: environments adapt back to agents. "Mecha-nudges" raise machine-readability A systematic map, threaded with an economist's lens. #LLM# #PostTraining#

GLM-5.2’s shift from GRPO-style group-wise optimisation to critic-based PPO for long-horizon agentic RL is worth paying attention to. Over the past year, the common narrative around LLM RL has been fairly simple: PPO is heavy, critics are expensive, and GRPO is lighter because it removes the critic. This view makes sense for short-horizon RLVR tasks, such as maths, coding problems, and verifiable QA. In those tasks, the structure is clean. The same prompt can produce multiple candidate answers, a verifier can judge which ones are better, and the model can learn through group-relative comparison. But agentic RL is different. A coding agent fixing a bug may need to read the issue, search the codebase, edit files, run tests, fail, locate the problem again, and try another solution. Final success only tells us that the whole trajectory worked. It does not clearly tell us which intermediate step was crucial, which step was a detour, or which action had no immediate reward but enabled later success. This is where GLM-5.2’s blog becomes interesting. It mentions that long trajectories are compacted and split into multiple sub-traces. For the same prompt, different rollouts can produce different numbers of trainable traces, and their lengths can be highly uneven. In this setting, group-wise optimisation becomes less natural, while critic-based PPO can estimate token-level advantages and handle credit assignment across long, irregular trajectories more effectively. So this is not simply “PPO is back”, nor does it mean “GRPO has failed”. A better way to put it is: GRPO is better suited for comparing multiple answers, while PPO is better suited for evaluating each step in a long chain of actions. When models move from answer generators to action takers, the key RL question also changes: from “which answer is better?” to “how should credit be assigned across a long sequence of actions?”