Daniel van Strien
@vanstriendaniel
Machine Learning Librarian @huggingface 🤗
I like big datasets and small models.
加入 September 2014
1.5K 正在关注 6.7K 粉丝
datatrove — the data-processing library behind FineWeb, FineWeb2, and FinePDFs — just shipped 0.10.0!
- JobsPipelineExecutor: run pipelines on @huggingface Jobs: fan-out, multi-stage dependencies, retries, resume. No Slurm cluster needed
- HF storage buckets as a DataFolder: read, write, and log straight to hf://buckets/...
- Reasoning outputs preserved in inference results
pip install datatrove[io,processing]
显示更多
