setup I often use to review RL Tasks/Environment data at scale + do human review
Code Mode Review Council
1. for every generated Environment/Task, send a council of weak/medium/strong models to do the Task. good calibration step for models & harness
2. send another council to review all of the generated traces + Task to find quality issues in the Env design. report back independently and debate
lots parallel compute to get a strong, diverse first pass at the data
human then reviews thoroughly in a vibe coded html viewer
post-training mostly all data quality, you’re prob not innovating on even the slightest tweak to an existing algorithm 99% of the time
humans are bandwidth limited for data review, so we need methods to help us scale “looking at the data”