my paper won an award at icml 😁 some thoughts: • this work was rejected from NeurIPS. i cleaned it up a small amount and it got great reviews from ICML! don't give up • ICML received 24k submissions and only gives out 7 awards, which is crazy. feeling grateful • i distinctly remember sitting at my desk two winters ago wondering if i would ever finish this project. most of all this is the product of sitting down and forcing myself to keep working for several months straight. the results emerged from running the experiments over and over and fixing a long sequence of tiny details. eventually, the curves looked like that 👇 • also happy that the insights in this paper are becoming more widely accepted: 3.3 bits/param, thinking about capacity "LLM as flashdrive" mentality • the method here is used successfully for selecting midtraining data at least one frontier lab, which is cool! • i am grateful to my collaborators, but Meta is no longer a great place for academic research imo and this almost never got published for a number of reasons. i shall not elaborate further • for future work, i think analyzing the implications of on-policy algorithms on capacity, as well as LoRA and things like it, are fruitful potential research directions • sadly i'm not in Korea but am following the conference online from california and happy to chat! a nice end to one phase of my research career :)