Register and share your invite link to earn from video plays and referrals.

Liam Fedus
@LiamFedus
Building industrial-scale science at @periodiclabs Past: VP of Post-Training @OpenAI; Google Brain
1.2K Following    36.4K Followers
Does your girlfriend sit front row for your technical talks and leave with 400 photos? Mine does.
Data centers are power-limited and OpenAI's Jalapeño sets new frontiers for tokens per second per watt. Impressive and fast work from OpenAI's ASIC program.
OpenAI Jalapeño: Better Than Nvidia Blackwell OpenAI’s self-designed ASIC compared with Rubin, Jalapeño’s TCO, throughput per MW, and spicy deets
A competitive, rapidly improving open-source model ecosystem lets companies translate their proprietary data and know-how into their own intelligence. Cursor, Cognition, and now Harvey are building their own SOTA models. Congrats @harvey and @FireworksAI_HQ on Tenet! This was a rank-64 LoRA post-trained model on Kimi K3, trained on ~2k legal tasks on a B300 cluster.
Show more
Introducing Tenet, our first model post-trained for legal. Tenet is a Kimi K3 base that we post-trained with @FireworksAI_HQ on a corpus of publicly available legal data, synthetic data, and human expert data simulating long-horizon legal work. Training increases Tenet's all-pass rate by 82% on LAB and 22% on LAB Contracts relative to the Kimi K3 base model. It achieves state-of-the-art performance on LAB Contracts and places second on LAB. These gains generalize to other leading agentic benchmarks including @mercor's Apex Agents - Corporate Law, @crosbylegal's Redline Bench, and @scale_AI's Professional Reasoning Bench. Tenet is also optimized for token efficiency, operating at less than a fourth the cost of leading foundation models. We additionally post-trained three specialist models for Tenet to use as subagents: 1) M&A Diligence: post-trained with @baseten on our LAB Diligence environment in an RLM harness, this model is optimized for high-scale, long-horizon tasks. 2) Review Tables: trained with @appliedcompute on our Review Table environment, this model is state-of-the-art and cost-effective at high-volume document review and structured data extraction. 3) Firm Knowledge: trained with @EngramLab on our synthetic law firm environment, this model is optimized to learn and search over a firm's knowledge via memory and structured notes. More details on model training, environment design, benchmarking, results, and more in the article by @gabepereyra below. What's next for Harvey’s research? - Scaling LAB to more jurisdictions, practice areas and workflows - Scaling compute to bring new generalist models and capabilities to Harvey More to come soon.
Show more
Small groups of people with relatively little money on the line (<$10k markets) produce well-calibrated predictions across diverse domains
We ran a study on over 2 million markets to evaluate their calibration. To our knowledge, this is the largest replicable study on this topic. Here are the results: 🧵
An excellent history of scaling laws from @jietang. In 2020, we explored the limits of sparsity in Switch Transformers by routing each token to only 1 out of 2048 experts (in retrospect, a bold choice). The model had fewer than 3B activated parameters, but 1.6T total parameters (comparable to today's frontier models). The 1.6T model achieved better C4 perplexities than the T5 models using far less compute, set a new SOTA on TriviaQA, but was dumb as bricks on reasoning tasks like SuperGLUE. The lesson was that the optimal tokens-per-parameter ratio is highly task-dependent. Or as @NShazeer had already intuited: FLOPs were intelligence; parameters were knowledge!
Show more
Thoughts About Scaling Law Scaling, but not only of parameters. Every model release now ends with the same question: how many parameters? It isn't a question that can be answered on its own. Parameter count is only meaningful alongside three others — how much data you have, where you intend to spend your compute, and who will run the model, under what conditions. The field learned this the hard way. Kaplan et al. (2020) fit an exponent that told everyone to grow parameters faster than data — roughly 2.7:1 — and the industry complied: GPT-3, Gopher, MT-NLG. Hoffmann et al. (2022) redid the experiment across four hundred models and found the compute-optimal split is closer to 20 tokens per parameter, and that with sufficient compute the two should grow at the same rate rather than drifting apart. The error in the earlier fit compounded with every order of magnitude of compute, which is why the largest models of that generation were the most misallocated. The trillion-parameter round was, in retrospect, a detour the whole field took together and then reversed. Chinchilla wasn't the end either. It optimized training compute for models that would be trained once and evaluated. Today a model is called billions of times a day and inference dominates lifetime cost. Put inference into the objective and the optimum moves toward smaller models trained far longer — deliberate over-training, which is what Llama-2-7B and Gemma-2-9B were doing at roughly 290 and 889 tokens per parameter. Sparsity moved the target again. In a MoE model two quantities have to be kept apart: total parameters govern roughly how much the model can hold — knowledge, facts, the long tail — while activated parameters and effective depth govern roughly how far it can think, how many steps of a causal chain it can carry before it comes apart. A dense 20:1 ratio does not transfer. And the ratio isn't a single number at all: Roberts et al. (2025) find the optimal tokens-per-parameter is task-dependent, with memorization favoring more parameters and reasoning favoring more data. Follow-up work on MoE observes that at fixed TPP, pushing total parameters higher actually degrades reasoning, while activating more experts reliably helps it. This matters for what we are building toward. Finding a vulnerability is not a retrieval problem. It doesn't come from having memorized more CVEs; it comes from carrying a twenty-step chain of inference to the end without losing the thread. That capability does not live in total parameter count. Which brings us to this release. Total parameters appear to matter up to a threshold — enough to hold the world — after which additional capability comes from scaling elsewhere: effective depth per forward pass, and above all post-training. GLM-5.3 is our controlled experiment on that claim. Same base, same architecture, same total and activated parameters as GLM-5.2. One month of scaling long-horizon environments and RL. The gains are not marginal. Well, scaling has more than one dial. We turned the post-training one this time because it had the most slack left in it — not because the others are finished. Base model size, pretraining data, compute spent per forward pass: all of them are still on the table, and we will come back to each. What this experiment taught us is that the dials do not have to be turned together, and that the one worth turning next is rarely the one that was worth turning last. We are not done scaling. Next time, maybe mid-training, pre-training, and even more.
Show more
The highest-level goal for superintelligence is to cheaply and abundantly create new knowledge. Strong open-source models accelerate this. Congrats to the MSL team and @alexandr_wang on the open-source releases and to @finkd for advancing an optimistic view of the future!
Show more
I believe everyone should have access to superintelligence, and I wrote a long piece about Meta's philosophy and values for building a positive future for everyone.
Scaling Kimi K3 on H200s with @sgl_project @lmsysorg While most discussion on X focuses on Blackwell, I thought it would be useful to share what we learned while tuning Kimi K3 for throughput-oriented workloads on H200s. - TP+EP beat DP+EP for K3. Only about 1/4 of its layers use gated MLA, so DP attention provides less benefit from avoiding KV-cache duplication. Having each DP rank process every attention head also substantially increased latency. This differs from our K2.6 and GLM setups, where we generally prefer DP+EP. - FP8 KV cache worked well. We measured no quality or throughput regression in our evaluations, while effectively doubling KV-cache capacity. - TP32+EP32 provided significantly more cache capacity: approximately 4M KV-cache tokens per replica, compared with about 800K using TP16+EP16. - For low cache-hit workloads, consider disabling radix cache. Prefix caching requires four Mamba-state slots per sequence in our setup; without radix cache, each sequence needs only one. - Non-speculative decoding won at high concurrency. Above roughly 64 concurrent requests, draft-and-verify overhead outweighed the savings from accepted speculative tokens in our tests. - A few caveats: symmetric memory caused instability in our multi-node environment. We also observed hangs when combining speculative decoding, HiCache, and high KV pressure. Both issues are still under investigation. We’ve been running this configuration at full load across roughly ~O(1000) chips for several days. Aside from hardware failures, it has been reliable. Feedback, discussion, and better configurations are very welcome! Sharing my launch command in the first comment. h/t to @hanming_lu @liin1211 @0xishand @ChengWan17 @NealBayya for the discussion
Show more
At Periodic, we love people with both science + LLM experience but would be very happy with folks who have: - strong science backgrounds - deep interest in AI tools - a desire to learn the rest (LLMs, evals etc) coding is getting easier, but doing science is not. we want to meet people with depth, accomplishments, and taste in science. if you have friends like this who want to work closer to AI, please send them our way. Also you get to work with people like Dima or yours truly
Show more
Join us to train AI that understands the physical world and can design what comes next. Coding agents worked in part because of objective, fast, and cheap RL envs. But what's often overlooked is that there's no big expertise gap between AI researchers and SWEs. An AI researcher can easily assess correctness, coding strategies, and more. In science, we must bridge the gap and look for brilliant folks who go deep and get the boring details right in order to create systems that otherwise would not exist.
Show more
At @periodiclabs we are looking for brilliant new colleagues who excel at bridging AI and science. The role's main focus is advanced evals and training data/tasks. But you will also get involved in both internal science and AI work. Apply/spread the word!
Show more
Our failures are our moat. A scientific paper is a clean repackaging of a messy process of failed syntheses, dead ends, hints of success. That mess is the durable asset: what was tried, what worked, what failed, and why. Compound it into weights you own. Frontier labs live by this principle. If tokens-in-context were enough, pre-training would have died years ago.
Show more
I love the phrase “intelligence too cheap to meter”. For us, that plays out as every random piece of our lab equipment soon will have a 140+ IQ. An anecdote was that in the early days, as we were quickly scaling up materials synthesis, we had technicians and scientists recording data on our characterization machines. This is the usual way. But when the scientists couldn’t keep up with our throughput, we made a script to programmatically capture the data. That scaled nicely, but we found the data wasn’t particularly useful. It wasn’t a complete or intelligent record of what happened experimentally. It also didn’t have the context for what we intended to do. We weren’t only interested in the mean behavior but were looking for anomalies or details that programmatic capture often missed. Now, with improving AI that is (almost) too cheap to meter, our data capture is both scalable and intelligent. The system knows the intent behind what we’re doing, can review our full records, and can direct and find the interesting evidence using the machine. And that improved data capture yields ever more useful future AI systems. It’s our own real lab version of Rick and Morty’s butter robot: “what is my purpose?”, “you watch the characterization machine”, “oh my god”.
Show more
After mastering the digital realm, we’ll next need AI systems to engineer matter. Tomorrow, Dogus will walk through our recent progress at Periodic towards this
Excited for this one! I’ll be on the Master Stage tomorrow at 3:O0 PM with @_IainMartin. We’re also co-hosting an Automating Science Happy Hour that evening. Come join us (
Show more
Periodic was founded on the basis that thinkism is not enough
Before we gave AI access to the internet (which, to some, was an invitation to the end times), we relied on LLMs' compressed knowledge stored in their weights. Looks like I was compressed into GPT, Claude, Gemini, but not the Llama series. And, miraculously, little Qwen3 8B knows me :)
Show more
Breaking: OpenAI overtakes Anthropic’s valuation
I’m excited to share that I’ll be joining OpenAI and look forward to working with the exceptional team there. It was a difficult decision to move on. I’m incredibly proud of the amazing team at Google and everything we’ve built together. It has been an honor and a pleasure to work with all of you.
Show more
Today, @ekindogus and I are excited to introduce @periodiclabs. Our goal is to create an AI scientist. Science works by conjecturing how the world might be, running experiments, and learning from the results. Intelligence is necessary, but not sufficient. New knowledge is created when ideas are found to be consistent with reality. And so, at Periodic, we are building AI scientists and the autonomous laboratories for them to operate. Until now, scientific AI advances have come from models trained on the internet. But despite its vastness — it’s still finite (estimates are ~10T text tokens where one English word may be 1-2 tokens). And in recent years the best frontier AI models have fully exhausted it. Researchers seek better use of this data, but as any scientist knows: though re-reading a textbook may give new insights, they eventually need to try their idea to see if it holds. Autonomous labs are central to our strategy. They provide huge amounts of high-quality data (each experiment can produce GBs of data!) that exists nowhere else. They generate valuable negative results which are seldom published. But most importantly, they give our AI scientists the tools to act. We’re starting in the physical sciences. Technological progress is limited by our ability to design the physical world. We’re starting here because experiments have high signal-to-noise and are (relatively) fast, physical simulations effectively model many systems, but more broadly, physics is a verifiable environment. AI has progressed fastest in domains with data and verifiable results - for example, in math and code. Here, nature is the RL environment. One of our goals is to discover superconductors that work at higher temperatures than today's materials. Significant advances could help us create next-generation transportation and build power grids with minimal losses. But this is just one example — if we can automate materials design, we have the potential to accelerate Moore’s Law, space travel, and nuclear fusion. We’re also working to deploy our solutions with industry. As an example, we're helping a semiconductor manufacturer that is facing issues with heat dissipation on their chips. We’re training custom agents for their engineers and researchers to make sense of their experimental data in order to iterate faster. Our founding team co-created ChatGPT, DeepMind’s GNoME, OpenAI’s Operator (now Agent), the neural attention mechanism, MatterGen; have scaled autonomous physics labs; and have contributed to some of the most important materials discoveries of the last decade. We’ve come together to scale up and reimagine how science is done. We’re fortunate to be backed by investors who share our vision, including @a16z who led our $300M round, as well as @Felicis, DST Global, NVentures (NVIDIA’s venture capital arm), @Accel and individuals including @JeffBezos , @eladgil , @ericschmidt, and @JeffDean. Their support will help us grow our team, scale our labs, and develop the first generation of AI scientists.
Show more
0
425
4.2K
442
Forward to community