After reading up a bit on ML research post transformer era, I was upset that it seems to have converged on hyper-optimizing matmul-based algorithms: (MHA, MQA, MLA, SWA, DSA, GQA, SWA-GQA, ABCDA [only one of these is made up]). Surely, an algorithm that is not Attention based is sitting there waiting to be discovered. > the researchers are just being lazy but this is a stupid conclusion. How can you blame researchers, when the hardware they train on is optimized for matmuls (tensor cores / systolic arrays). Any algorithm not a matmul is literally bound to die, even if it's twice as good as attention. Add compute constraints, you have to be crazy to research any direction not attention based (basically @sarahookr 's hardware lottery essay) We talk about hardware-software co-design in inference, but it seems that, to get to the next leap in research, we'll need hardware-research co-design. At first, it seems this will never happen, given typical multi-year hardware tape-out constraints. But then you look at @OpenAI. 9 month tape-out. Better "training" and serving . Why fab your own chip if it's just going to be systolic-array based? Why not just buy Nvidia? > "But Nvidia GPUs are scarce" Then buy TPUs/AMD/Qualcom/Cerebras. Sure the software is not that good, but if you're OAi, you can hire an army of engineers to unlock the full capability. Either they moved away from attention and have a new algorithm they needed their own chips to train it on (unlikely given that a 9-month tape-out with a TPU vendor implies reusing IP)...or research is dead and we're never escaping attention / matmul based algo.

In a first, a simple Group B officer awarded, Nation's 4th highest Civilian Honour But his story is not ordinary at all He was an Undersecretary on verge of retirement in MHA and was asked by Congress sympathiser IPS officer to sign affidavit on "Hindu Terror" When he refused, they picked him up, put him solitary cell and burnt his thighs with cigarettes When he was released he came to Arnab's show and told everything about the conspiracy to defame Hindus

This body’s built for battle… but if you’ve got other plans for it, I won’t stop you 🐰𝑨𝒑𝒓𝒊𝒍’𝒔 𝒑𝒊𝒄𝒌: 𝑴𝒊𝒓𝒌𝒐 𝒐𝒏 𝐏𝟒𝐭𝐫𝟑𝟎𝐧 #MHA# #Mirko#