wh(@nrehiew_):They do a bunch of ablations on short, long contexts and just general LM evals. QSA seems better on language modelling (again, flops matched i assume), basically the same on RULER. Efficiency wise, its obviously better and doesn't affect MTP acceptance

@nrehiew_

eng primarily, ml mostly, research previously

Joined October 2023

104 Following 18.5K Followers

wh@nrehiew_

2026.08.27 02:50

They do a bunch of ablations on short, long contexts and just general LM evals. QSA seems better on language modelling (again, flops matched i assume), basically the same on RULER. Efficiency wise, its obviously better and doesn't affect MTP acceptance