Daniel van Strien(@vanstriendaniel):datatrove — the data-processing library behind FineWeb, FineWeb2, and FinePDFs — just shipped 0.10.0! - JobsPipelineExecutor: run pipelines on @huggingface Jobs: fan-out, multi-stage dependencies, retries, resume. No Slurm cluster needed - HF storage buckets as a DataFolder: read, write, and log straight to hf://buckets/... - Reasoning outputs preserved in inference results pip install datatrove[io,processing]

Daniel van Strien

@vanstriendaniel

Machine Learning Librarian @huggingface 🤗 I like big datasets and small models.

加入 September 2014

1.5K 正在關注 6.7K 粉絲

Daniel van Strien@vanstriendaniel

2026.08.14 08:09

datatrove — the data-processing library behind FineWeb, FineWeb2, and FinePDFs — just shipped 0.10.0! - JobsPipelineExecutor: run pipelines on @huggingface Jobs: fan-out, multi-stage dependencies, retries, resume. No Slurm cluster needed - HF storage buckets as a DataFolder: read, write, and log straight to hf://buckets/... - Reasoning outputs preserved in inference results pip install datatrove[io,processing]