wh(@nrehiew_):They do a bunch of ablations on short, long contexts and just general LM evals. QSA seems better on language modelling (again, flops matched i assume), basically the same on RULER. Efficiency wise, its obviously better and doesn't affect MTP acceptance

@nrehiew_

eng primarily, ml mostly, research previously

参加 October 2023

104 フォロー中 18.5K ファン

wh@nrehiew_

2026.08.27 02:50

They do a bunch of ablations on short, long contexts and just general LM evals. QSA seems better on language modelling (again, flops matched i assume), basically the same on RULER. Efficiency wise, its obviously better and doesn't affect MTP acceptance