Daniel van Strien(@vanstriendaniel):datatrove — the data-processing library behind FineWeb, FineWeb2, and FinePDFs — just shipped 0.10.0! - JobsPipelineExecutor: run pipelines on @huggingface Jobs: fan-out, multi-stage dependencies, retries, resume. No Slurm cluster needed - HF storage buckets as a DataFolder: read, write, and log straight to hf://buckets/... - Reasoning outputs preserved in inference results pip install datatrove[io,processing]

Daniel van Strien

@vanstriendaniel

Machine Learning Librarian @huggingface 🤗 I like big datasets and small models.

가입 September 2014

1.5K 팔로잉 중 6.7K 팬

Daniel van Strien@vanstriendaniel

2026.08.14 08:09

datatrove — the data-processing library behind FineWeb, FineWeb2, and FinePDFs — just shipped 0.10.0! - JobsPipelineExecutor: run pipelines on @huggingface Jobs: fan-out, multi-stage dependencies, retries, resume. No Slurm cluster needed - HF storage buckets as a DataFolder: read, write, and log straight to hf://buckets/... - Reasoning outputs preserved in inference results pip install datatrove[io,processing]