Zyphra(@ZyphraAI):Zyphra Research releases PUFFER, a novel incremental fuzzy deduplication system for LLM-scale datasets. PUFFER runs entirely on CPU and achieves 11-35x speedup over existing methods. We use PUFFER internally for our own training dataset and release it under Apache 2.0 license.

Zyphra

@ZyphraAI

Full stack open superintelligence

가입 March 2021

39 팔로잉 중 15.4K 팬

Zyphra@ZyphraAI

2026.09.01 18:14

Zyphra Research releases PUFFER, a novel incremental fuzzy deduplication system for LLM-scale datasets. PUFFER runs entirely on CPU and achieves 11-35x speedup over existing methods. We use PUFFER internally for our own training dataset and release it under Apache 2.0 license.