가입 후 초대 링크를 공유하면 동영상 재생 및 초대 보상을 받을 수 있습니다.

Viv
@Vtrivedy10
applied research @LangChain Labs, prev @awscloud, phd cs @templeuniv
가입 February 2013
1.8K 팔로잉 중    16.4K 팬
“The missing piece is the synthetic data recipe. And our experiments strongly point in the same direction.” to make a model good at a Task, make/curate high quality data for that Task there’s no hidden magic (which is a good thing), it’s just a hard, time consuming problem -> we can work with that! models behavior comes from data, if the model sees good examples in Traces or is given good environments to practice in, it will get better Kimi, DeepSeek, Jev, etc -> all of these launches explicitly talked about how their data recipe was the sauce
더 보기
We spent the last few days testing JEV internally. We tried to reproduce it, break it, and improve it. 5 things we learned: 1. JEV’s core insight is simple - and probably right. If the answer space is bounded, don’t generate the answer token by token. Remove autoregressive decoding and you can remove 1–2 orders of magnitude of inference work. 2. RLCD isn’t the moat. Data might be. Laya already open-sourced the implementation + weights. The missing piece is the synthetic data recipe. And our experiments strongly point in the same direction. 3. “Open source already beat JEV” is a benchmark illusion. Same checkpoint: In-distribution: 0.769 OOD: 0.541 Change the distribution and the apparent breakthrough largely disappears. 4. Compute-optimal ≠ learnability-optimal. We moved state outside the problem sequence to save compute. Result: -27 points. Likely because we saved FLOPs while throwing away useful MLM pretraining priors. Then we tested 6 more ideas in one night: anchor selection, continuity smoothing, asymmetric windows, bucketed temperature, two-stage retrieval, full RLCD. All failed or failed to generalize. Only one thing consistently worked: More data. 1,200 → 123,475 examples OOD: 0.4069 → 0.5498 5. The weirdest result: A decoder with zero additional training still beat every trained small encoder: 0.5671 vs 0.5628 / 0.5411 / 0.4069 Our takeaway: JEV’s breakthrough may not be a particular architecture or training algorithm. It may be a much simpler idea: When the answer space is bounded, search it. Don’t generate it. And right now, the biggest bottleneck to making that work broadly doesn’t look like architecture. It looks like data.
더 보기