Sewon Min(@sewon__min):Long-context LMs seem great at tasks like RULER, but what about tasks like "find every contradiction in the given corpus"? - Performance suffers, especially as the input corpus scales (even with in-domain fine-tuning, not just zero-shot) - It changes a lot of core modeling decisions, e.g., whether to use hybrid models, whether to use block sparse attention TL;DR: if you are a model developer, include our CTC-Bench in your evals! ✨

2026.09.28 23:12

Long-context LMs seem great at tasks like RULER, but what about tasks like "find every contradiction in the given corpus"? - Performance suffers, especially as the input corpus scales (even with in-domain fine-tuning, not just zero-shot) - It changes a lot of core modeling decisions, e.g., whether to use hybrid models, whether to use block sparse attention TL;DR: if you are a model developer, include our CTC-Bench in your evals! ✨

Prasann Singhal@prasann_singhal

2026.09.28 16:33

Could long-context architectures “find all contradictions” in a science literature? Not yet! 🧵 We study a new class of "high-complexity” tasks whose difficulty scales quadratically with corpus size (as opposed to linearly), reversing common LCLM decisions! (block-sparse attention, hybrid models…)