Sergio Paniego
@SergioPaniego
Machine Learning Engineer @huggingface 🤗
AI PhD. Technology enables us to be more human. 🏳️🌈
1.7K Following 6.7K Followers
the fourth and last session of the training agents series finally has a date
📅 Class 4: agent environments > thursday september 10th, 5pm cest / 8:30pm ist
this one is about the environments themselves, the sandbox with tools and files where the agent works for many turns, and how you build and run one with trl and openenv
Show more
full write-up is out with everything open > the env, the hand-rated pool, the 3 runs and every painting they made
trained my own with trl + openenv
blog with open artifacts (code, models, dataset, rl env...) soon!
from the vault: on-policy distillation in TRL got 40x faster with three fixes, a generation buffer, batched teacher requests, and binary-encoded logprobs
enough to distill from 100B+ teachers, a 4B student gained 39 AIME25 points learning from a 235B one
Show more
we just released a new blog "Training a coding agent using the OpenCode harness in remote HF sandboxes with TRL and OpenEnv"
you can take a real coding agent (OpenCode), let it run its own tool loop against real coding problems, and train it with RL on the exact tokens it produced
and every rollout runs in its own remote HF sandbox, so rollouts scale out beyond one machine
the loop:
- OpenCode owns its tool loop inside an OpenEnv sandbox
- an in-sandbox proxy records the real token ids + logprobs, per turn
- a hidden-test verifier scores the result, and that is the reward
- TRL trains with AsyncGRPO, weights sync back to vLLM over NCCL
blog + runnable example:
Show more
