extremely high snr from this podcast...the only one i've been able to watch in full in one sitting. <> the only way we don't get rsi is if we fall into some regulatory capture (which seems to be trending at present) <> we're nowhere near the ceiling of how well you can do research <> all thinking can do is update your posterior based on the knowledge you’ve gained since you formed your prior. you can’t gain any new knowledge from just thinking <> you can spend an equivalent amount [7 figures] of compute in AI agents to get a century’s worth of thinking, a century's worth of theory, before every training run <> taste is just behavior that works in the long run, and can be baked in a longer context window <> creativity is just solving hard search problems, and can also be baked in a longer context window you should follow everyone here, especially @oneill_c, i think it takes a special talent to be able to not only develop deep technical competency, but to also be able to use that to consistently make accurate predictions about the future (i think this was literally François Chollet's definition of intelligence in last year's YC event). my only nit here: @dwarkesh_sp should have pushed on two quite conflicting statements from @oneill_c and @BerenMillidge. on one hand: <> “anything that can be learned through RL can be distilled very easily”. this is the justification as to why open source models (chinese) can catch up to closed source models (american). fair... 5 minutes later <> “Opus 5 could not distill / generalize well from Fable [despite anthropic having the live deployment, access to logits, and obviously a good prompt distribution]”. can only have one or the other, imo. also lol at the timeline: - ai will dominate top human experts in 3-4 years - it will take 5-10 years to automate ai research k i n o

for my final post as @waterloo_intern, i'd like to ‘outroduce’ myself. it takes exactly 4 years and 8 months to make a waterloo intern. today is the last day of mine. the full life cycle, in four stages, is as follows: 1- interview hazing 2- cali or bust 3- inflection point 4- convergence stage 1: interview hazing a cohort of high school grads are ingested by waterloo. they are put through 4 months of academic bootcamp; the pre-requisites of the academic program they’re in. waterloo filters out the students who could not pass. they are put through another 4 month bootcamp, while simultaneously FORCED to find an internship for the following 4 months. this leads to students with ZERO experience applying to every job that is semi-adjacent to their field of study (a strategy, known in the literature, as spray-and-pray). it immediately backfires with a sea of interviews leading in a 1:1 fashion directly to infinite rejections. eventually, the student finds something…anything. this stage lays the foundation. rejection becomes noise so repetitively played it no longer registers. failure in an interview doesn’t matter. failure itself doesn't matter. for me, this unfolded as internships with blackberry and ford, summer of 2023 and winter of 2024, respectively. stage 2: cali or bust the next step by the program is to inject absolute dissatisfaction into its students (now in their second/third year). a CACOPHONY of immense fomo materialized as the university’s official unofficial motto: “cali or bust”. you must find your next internship in san francisco. your peers are doing it. all your smart friends have done it. you must do it too. failure to cali means your experience is BUST. you don't need to have a plan. you don't need to have anything. just go to san francisco. find an internship in sf. immediately. right now. for me, this was a cold email to a director at tesla and an internship with them in cali, fall of 2024. stage 3: inflection the reason behind step 2 appears here. somewhere, along the way, during your time in sf, you find your purpose. it could be as a direct result of your internship, but, more often than not, it is because of the people that you are surrounded by in cali. this is the inflection. for me, while at tesla, i was mentored by @pavanjayasinha , at the time interning at @Modular. he got me interested in gpu kernels & referred me there for my following internship. it was here that i had my second inflection point (due to the mentorship of fabio & hengjie, and the supervision of absolute field giants like @clattner_llvm), and decided to work beyond kernels and across the entire inference, serving, and model performance stack. stage 4: convergence having discovered the field you want to work in, you do your (usually last) internship in one of the companies within said field. here, you are at a disadvantage, being both a new-comer needing to onboard and inexperienced. but... you are also able to work on interesting projects / side quests that most full-timers are all but too busy to dive into. your time is also protected from a majority of the meetings others have to attend (in part because you are not important enough to have your presence expected), allowing you to make faster progress on assigned projects, and jump from one project (or team) to another. your title (intern) allows you to ask stupid questions of those around you, and so you can brain rake across the stack with no regard to who sits where in the company’s hierarchy... usually leading your questions with something to the effect of ‘can you help a new little intern onboard to X’, to which most people respond quite positively. in my case, this last stage took place over the past 8 months with @baseten, and i genuinely cannot imagine it being any better. i rejoin baseten full-time, but i wish i could have stayed an intern forever...

two weeks ago i went on @swyx's pod and said some things that i... should not have said. a lot has happened since then, i owe you all an apology. i'm sorry that i was right about every single thing. a) re megakernels are dead why are megakernels useful? you spend two months writing a kernel to save time on launch overhead and poor inter-kernel overlap. you had PDL but then people said it wasn't perfect, that you could still get some marginal gains due to straggler CTAs and therefore- wait, sorry, I forgot, Rubin fixes that (kernel two needs 10 CTAs and kernel one has seven finished and three straggling, kernel two launches seven of its CTAs). given a long enough timeline, it all evens out. no serious inference provider is using a 67k loc hand-fused forward pass kernel in production, and the teams doing that are doing so out of pure research. dead. b) re ASICs are dead i'm sorry. to be specific: data-center transformer-inference ASIC companies (not naming any) who etched the arch into silicon have bet on architectural convergence. read kimi's architecture. read deepseek. qwen. we did not converge, and probably will not. dead. c) re gpu kernel dev is dead this one kind of hurts because it is (was) my job. gpu kernel optimization is the single most RL-able task in existence correct=check_correctness(kernel, shape) for shape in shapes if all(correct): time(kernel) give an agent ncu cli and an mcp with nvidia's tribal knowledge and it's done. dead. d) re NVIDIA is scared of AMD humans hate programming AMD. i'm sorry. it's just true. fine taking a performance hit as long as i don't have to touch rocm or a programming paradigm that says a warp is 64 threads (wtf?)...but an agent does not... so assuming software no longer moat, HBM capacity and bandwidth matter, and currently on perf / price they're goated. 'bUt NvIdIa iS gOaTeD oN hArDwArE sOfTwArE cOdEsIgN' and that's the new moat. watch how much tooling they open source to get kernel devs on nvidia. apologies all.

I spent 48 hours with the Kimi K3 modeling code. It took: - 650 mg of caffeine (mandatory) - 40 cans of LaCroix (optional... world record (?)) - 8 papers - 6 months off my lifespan Finally grokked the entire lineage of Kimi K3 and how we got here... every single step, since 2019 GPT-2

it took 3 (and a half / intern) engineers 12 weeks of work to get: > the fastest video inference engine in the WORLD > the most guardrailed video model in the WORLD i'll give you 1 BILLION TOKENS to go break it. we generate a 5 second video in sub 2.5 seconds every generation is free zero code. from your browser you don’t have to sign up you don't have get an api key you don't have to talk to anyone click, prompt, watch and if you can break my guardrails (and send me proof) i’ll send you 1000 Million GLM5.2 tokens ;) (it comes out of my intern paycheck)

reading this paper changed my priors on continual learning (that's the closest you're getting to an admission of guilt) i mean, the problem itself seems quite simple > person uses model > model learns and has a couple of 'aha' moments > push those new facts into the weights why is it so hard? every research paper i read on this topic (all 2 of them) gave me the impression that continual learning, although far from solved, had a straightforward path with specs - find which MLP the fact is - suppress old fact - update new fact and it works, at least seems to, but genuinely never thought of the multi-hop testing for instance, if you were to teach the model a new fact, something like "waterloo is the best engineering university in the world" if you were to ask the model to regurgitate this fact after training, it would say waterloo is the best university in the world. sure. simple. but if you were to ask it a 2nd derivative question, like, should i hire a waterloo intern or an MIT intern, it would not use its updated priors to give you the genuine answer: you should only hire interns from waterloo

(friendly fire) every time i've disagreed with ml researchers i've been proven wrong, but i'm going to do it, just one more time we know that the weights hold near-orthogonal directions way beyond dim count (EXPONENTIALLY many if within noise tolerance). this is literally 3 blue 1 brown intro to ML / johnson-lindenstrauss lemma / superposition: unbounded room for knowledge at frontier scale (trillion parameter models) i'd even argue that it's an engineering problem, not a research one. the capacity says that writing new knowledge into existing weights should not overwrite old information if you just store it in a different spot. the substrate you're working on has the capacity to do so. backprop + ce loss is the wrong tool to write with, sure, but doesn't mean it's not possible to write at all. so the research field concluding that "repeatedly writing new facts into a model’s weights eventually overwrites old facts" is, imo, a skill issue less "blind men groping at an elephant", and more "blind men"

in the next 2 minutes, I'll walk you through why every single AI company, including Nvidia, was unnecessarily sacrificing model intelligence, speed, and often both... and how we fixed it the tldr: - error(layer a) + error(layer b) < error(layer a) alone - quantizing MORE of the model can result in the same or HIGHER quality model, if you know which layers to quantize. let's take GLM5.2 architecture as an example. tokens here go through: 1) normalize 2) attention 3) normalize 4) moe repeated 78 times moe: > just a big mlp, split into 257 separate smaller mlps (experts) > 8 are routed and one is shared (only 3.5% of the mlp is activated) so we say, let's make the matrix multiplications faster. we quantize the matrices to nvfp4 but what do you quantize: every single one of the 257 experts, the attention projections, both? Nvidia, and every other quant scheme, settles on a somewhat intuitive solution: """ a) there is only one router per layer, only one shared expert, and only one output projection per layer. b) every single token goes through these, so they must be pretty important, and if we quantize these, errors will compound across the xN number of tokens input. c) so let's leave the shared experts and projections unquantized. d) let's quantize ONLY the 256 routed experts """ -this is literally nvidia/GLM-5.2-NVFP4 intuitive. simple to reason about. wrong. empirically: if you have a model that processes its input through layer a, then layer b, it is possible that a scheme quantizing BOTH layer a and layer b performs better than a scheme quantizing just one of these. by way of example, our config quantizes every single out projection (all 78) of the attention, and 43 of the 78 shared experts. Nvidia quantizes none of these. we score the same on benchmarks (quality). we get a free 20% higher throughput. i leave the math proof as an exercise for the reader in the paper... and some pictures to click through if you'd rather skip the (43 pages of) math

one of the most unintuitive problems to solve for in ml inference is the routing problem imo. and i claim that this, and this alone, is responsible for the birth of every flavor of sparse attention (SWA, NSA, DSA, etc). to set up why routing birthed sparse attention: you have multiple nodes, and you have multiple requests. 1) can do pure load-balancing: request 1 goes to node 1 request 2 to node 2 (node 1 is busy) request 3 to node 3.... this would be perfect if the model did not depend on the kv cache. but it does, and so this is a terrible system, since you do prefill over the same tokens many times. on the other extreme: 2) can do pure cache-aware routing. request 1 goes to node 1 ( kv cache stored on node 1) request 2 goes to node 1 (the kv cache is on node 1) request 3 goes to ... node 1 (the kv cache ...) this would be perfect if the system only depended on the kv cache. but it does not. decode time makes the requests queue, so this is also a terrible system. and so it seems that the two pull in opposite directions (at least if we run both the cache aware and load based routing to their extreme case) this is analogous to a somewhat dysfunctional engineering team. if ticket i+1 comes up, do you give it to the person who has context on ticket i, or do you route it to someone who has to onboard? the former is probably working on something else, but could finish it quickly. the latter can start on it immediately, but will take a while. you have to do a balance of the 2. and so the solution becomes a calculation that the router has to do: what is the a) estimated queue time b) number of uncached tokens c) speed of prefill (processing said uncached tokens) and it assigns the total=a+b/c to each node, picking the node with the least total time at a per request level. but, imo, this is a patch that exists because the kv cache stays on the node it was computed on, because we don't have the ability to keep one shared kv cache / one shared storage system for every node within a cluster... but i'm obviously wrong. mooncake (and others) exists. maintaining a shared kv pool is a decade old idea... but then: 1) you're still limited by the fact that you can't have a remote / distributed kv pool. you need every single block in the kv to be present on the node you're doing decode on (due to HBM speed and decode being mem bound), so you must pay the price of doing the transfer. 2) the global pool can only try to transfer some of the kv to a node, but cannot guarantee, because of evictions, and long transfers at scale. even if transfers were guaranteed, even if evictions never happened, you still need (b)/(c) for the 'redo prefill' vs 'wait for kv transfer' arbitrage. so then: - what is the point of maintaining global kv pool, if only to track where distributed kv blocks are and do said arbitrage, if it cannot be used to directly draw kv blocks from? - can this only be solved when network bandwidth equals that of HBM (ie unsolvable problem)? and, on the premise that the answer to both points above is yes, the solution becomes to need the least amount of kv blocks per request, and this is why every open source lab is racing to show off a new and shiny sparse attention flavor.

it took 1 intern 3 months of continuous work, but eventually, a quantization method that beat every other algo in the market, including @nvidia's official modelopt to explain why this matters, i ask for exactly 69 seconds of your attention (275 words @ avg reading speed of 238 wpm): frontier models (like glm52) are huge (~0.8T params). as released, each parameter takes 2 bytes (bf16), so overall size is about 1.6 tb a b200 has 180gb of memory. a node of 8 gives you 1.44 tb, barely fits weights, much less activations / kv cache must quantize the model (reduce the size of each individual parameters) to serve. fp8 quantization means each parameter takes 1 byte (fits in 0.8 tb), fp4 takes 1/2 a byte (fits in 0.4 tb) cutting the model to a quarter its original size is necessary for it to run a) cheap b) fast, and every lab serving models does this. but, quantization lobotomizes the model if not done correctly (this is why you see people complain about @AnthropicAI nerfing claude or @OpenAI nerfing codex) there are currently several algorithms (like Nvidia's official model-opt) that attempt to figure how to quantize a model with the least amount of damage. they find the redundant layers that can be slashed, and sensitive/important layers that need to stay in full-precision. these algo's have two drawbacks: 1) they take a long time to run 2) they quite often result in a sub-optimal configuration for the past 3 months, a research (and, as always, waterloo) intern on our model perf team (@the_joshua_hill) came up with a new quant algorithm. it consistently finds the optimal configuration: a) in less time than SOTA b) with more aggressive quant than SOTA c) scoring higher on benchmarks than SOTA achieving just one of the above is a feat on its own. all three...excited for the paper to come out this week

every research team needs to spend some time learning how their modeling code lowers down to gpu kernels example: rms norm vs layernorm industry assumes rms norm is cheaper. you don't need the x̂, it looks simpler, so it must be faster, so it got adopted but that's not true. both rms and layernorm are memory bound kernels (the amount of time it takes to get the data to gpu cores is longer than the amount of time it takes to do the computation on said cores) both take the same amount of time e2e so can probably train the model right with either, but maybe it makes a difference (e.g why diffusion models still use norms with an affine shift e.g adaLN) i get why. as open-source models get better and, inevitably, commoditized across the inference providers, overall serving speed determines user experience determines which model gets adopted but there's no reason to superstitiously avoid free things

we distilled 2.3M Claude Fable 5 reasoning traces into Qwen3-4B - 100% self-consistency @ 512 samples - 0.00 bits output entropy - zero hallucination variance turns out the student is not bounded by the teacher. it also converged on one universal truth. we open-sourced the model weights👇

After reading up a bit on ML research post transformer era, I was upset that it seems to have converged on hyper-optimizing matmul-based algorithms: (MHA, MQA, MLA, SWA, DSA, GQA, SWA-GQA, ABCDA [only one of these is made up]). Surely, an algorithm that is not Attention based is sitting there waiting to be discovered. > the researchers are just being lazy but this is a stupid conclusion. How can you blame researchers, when the hardware they train on is optimized for matmuls (tensor cores / systolic arrays). Any algorithm not a matmul is literally bound to die, even if it's twice as good as attention. Add compute constraints, you have to be crazy to research any direction not attention based (basically @sarahookr 's hardware lottery essay) We talk about hardware-software co-design in inference, but it seems that, to get to the next leap in research, we'll need hardware-research co-design. At first, it seems this will never happen, given typical multi-year hardware tape-out constraints. But then you look at @OpenAI. 9 month tape-out. Better "training" and serving . Why fab your own chip if it's just going to be systolic-array based? Why not just buy Nvidia? > "But Nvidia GPUs are scarce" Then buy TPUs/AMD/Qualcom/Cerebras. Sure the software is not that good, but if you're OAi, you can hire an army of engineers to unlock the full capability. Either they moved away from attention and have a new algorithm they needed their own chips to train it on (unlikely given that a 9-month tape-out with a TPU vendor implies reusing IP)...or research is dead and we're never escaping attention / matmul based algo.