wh(@nrehiew_):They do a bunch of ablations on short, long contexts and just general LM evals. QSA seems better on language modelling (again, flops matched i assume), basically the same on RULER. Efficiency wise, its obviously better and doesn't affect MTP acceptance

@nrehiew_

eng primarily, ml mostly, research previously

加入 October 2023

104 正在关注 18.5K 粉丝

wh@nrehiew_

2026.08.27 02:50

They do a bunch of ablations on short, long contexts and just general LM evals. QSA seems better on language modelling (again, flops matched i assume), basically the same on RULER. Efficiency wise, its obviously better and doesn't affect MTP acceptance

显示更多