Zyphra(@ZyphraAI):Zyphra Research releases PUFFER, a novel incremental fuzzy deduplication system for LLM-scale datasets. PUFFER runs entirely on CPU and achieves 11-35x speedup over existing methods. We use PUFFER internally for our own training dataset and release it under Apache 2.0 license.

Zyphra

@ZyphraAI

Full stack open superintelligence

加入 March 2021

39 正在关注 15.4K 粉丝

Zyphra@ZyphraAI

2026.09.01 18:14

Zyphra Research releases PUFFER, a novel incremental fuzzy deduplication system for LLM-scale datasets. PUFFER runs entirely on CPU and achieves 11-35x speedup over existing methods. We use PUFFER internally for our own training dataset and release it under Apache 2.0 license.

显示更多