🧵(1/9) Sparse RL for reasoning has an exploration problem. It can only reward solutions the model already stumbles into. On hard problems, that means lots of zeros and very little signal. SFT and self-distillation attack this with reference solutions as targets to match. Instead, we use them as reward scaffolds: a dense signal at both the outcome and process level. Introducing ExpRL: RL-based mid-training that improves exploration by scoring the model’s own attempts against the reference via an LLM judge. What we find: • A stronger policy straight out of mid-training (higher pass@1 and pass@k) • Still ahead after downstream sparse-reward RL • Holds across domains – math & STEM • Scales to a larger policy graded by a smaller judge