rohan anil
@_arohan_
co-founder @coreauto - and continuing to aspire to understand deep learning.
2.4K Following 46.3K Followers
Hi! this is Rohan Anil, I develop training algorithms among other things - my work trains various frontier models.
At CoreAuto, I was fascinated by the problem of training on compute islands, eventually in space training.
I hope you all enjoy this write up!🪐🛰️
Show more
We have three alphas in core auto:
Always be scaling
Everything is a skill issue
Be like @anmol01gulati
I love backprop, i appreciate it today more than yesterday
I like really deep learning and so should you
Cool idea, the activation optimization reminds me of
Actually wondering how would optimizing the activation with gradient descent works
We put all the good stuff in there. And huge thanks to @JeffDean on pushing on importance of this model and making it into the report.
@_arohan_ OG Gemini Nano was such a trailblazer :) I feel like the inflection point started this year but agree that next year's hardware and models is when people will see the curve steepen
1/ A new revision of full-bandwidth transformer is on arXiv ( We included a few extra discussions on:
- How to debug training runs
- Post-training / KV cache replay
- Alternative fusion gate
Non-official code release at:
Show more
@TheRobotHerder Trains are very cool
This is pretty cool, less expressive than Linear version of 2-simplicial, as the product is QK1 hadamard QK2 and rank can be higher than each. Would be cool to see QK1K2 V1V2 variant
It definitely will take more than 36 days though everything starts small - we shipped an update about our progress to some close folks, and couldn’t be more prouder of the work.
I was out at Detroit this weekend, and I think I got hooked on pinball machines. Feels therapeutic.
On-device ML is pretty cool. A few of us trained the first Gemini Nano model that ever shipped on Android, so I’ve been following the space for a while.
I think things get really interesting (its already pretty interesting) with the late-2027/2028 generation of chips.
Show more
There is a study that says people who create the research usually aren’t ones capturing the value of out of research.
Finding the anecdote today after learning about non attribution around the ternary quant work. I don’t think this is very nice.
Show more
Every research breakthrough that unlocks superlinear returns makes everything else we did before look marginal in retrospect.
It is meant for only that can go through a lot of pain and suffering.
Yesterday was a very good day of new technical milestones at @coreauto
You can feel the gradients wants to learn.
Sometimes I don’t understand neoclouds sometimes I am grateful they exist.
Am I so predictable or do I have freewill?
People not knowing what they don’t know combined with Dunning Kruger is the source of so many problems in this world, especially in science.
Marketing my cat like a new Apple product
