In 1991, the foundations for Transformers, Pre-training, Distillation, and World Models were already being built. These helped shape my own thinking, from my time at Google Brain to our Recursive Self-Improvement (RSI) work at @SakanaAILabs today. 🧠🗼
👇
もっと見る
I am proud of the work my team did in Munich in 1991, when compute was millions of times more expensive. We published the roots of today's trillion-dollar AI boom:
★ 3/1991: the first kind of Transformer (see the T in ChatGPT) - now called the unnormalized linear Transformer: the predecessor of the normalized quadratic Transformer
★ 4/1991: Pre-Training (the P in ChatGPT) & Neural Net Distillation (see DeepSeek and many other LLMs)
★ 6/1991: Deep Residual Learning, basis of LSTM & Highway Net / ResNet (most-cited AIs of their centuries)
★ 8/1991: conference paper on GANs for World Models trained by Artificial Curiosity
★ Around the same time, Munich also was the origin of the first self-driving cars in traffic (Ernst Dickmanns et al.), going up to 175 km/h. The city was truly the epicenter of AI.
Read the timeline with links to the original references, featuring a preface by @hardmaru:
もっと見る
