Zyphra(@ZyphraAI):Zyphra Research releases PUFFER, a novel incremental fuzzy deduplication system for LLM-scale datasets. PUFFER runs entirely on CPU and achieves 11-35x speedup over existing methods. We use PUFFER internally for our own training dataset and release it under Apache 2.0 license.

Zyphra

@ZyphraAI

Full stack open superintelligence

Joined March 2021

39 Following 15.4K Followers

Zyphra@ZyphraAI

2026.09.01 18:14

Zyphra Research releases PUFFER, a novel incremental fuzzy deduplication system for LLM-scale datasets. PUFFER runs entirely on CPU and achieves 11-35x speedup over existing methods. We use PUFFER internally for our own training dataset and release it under Apache 2.0 license.