Update on
@harvey’s model training effort.
We post-trained a model we are calling Tenet:
- Achieves SOTA on LAB
- Generalizes 3rd party legal benchmarks
- Uses sub-agents for domain specific capabilities
Tenet uses Kimi K3 as base and was post-trained in collaboration with
@FireworksAI_HQ:
- Rank-64 LoRA over the full network
- GSPO with importance-ratio masking
- 134 B300 GPUs for 2 months
Despite not being trained on 3rd party legal datasets we found improvements on:
-
@mercor’s Apex Agents - Corporate Law
-
@crosbylegal’s Redline Bench
- LegalBench
Tenet also learned how to use domain-specific subagents (separate post-trained models) for complex tasks:
- M&A Diligence: training in an RLM harness for long-horizon tasks (with
@baseten)
- Review Table: specialist models for high-volume structured data extraction (with
@appliedcompute)
- Firm Knowledge: parametric memory and structured notes for more efficient enterprise search (with
@engram)
These results suggest we can significantly scale training and we plan to:
- Scale both human and synthetic data significantly and scale training to 1K and then 10K GPUs
- This scale will let us move to full parameter fine-tuning and larger models
- We are now starting to post train models in our production harnesses
- Post-train other open-source base models to provide customers with model choice
If these problems sound interesting we are hiring for our post-training team