wh(@nrehiew_):They do a bunch of ablations on short, long contexts and just general LM evals. QSA seems better on language modelling (again, flops matched i assume), basically the same on RULER. Efficiency wise, its obviously better and doesn't affect MTP acceptance

@nrehiew_

eng primarily, ml mostly, research previously

가입 October 2023

104 팔로잉 중 18.5K 팬

wh@nrehiew_

2026.08.27 02:50

They do a bunch of ablations on short, long contexts and just general LM evals. QSA seems better on language modelling (again, flops matched i assume), basically the same on RULER. Efficiency wise, its obviously better and doesn't affect MTP acceptance