🚙New blog on saving experts from an untold, silent failure mode in ultra-sparse MoEs : While pursuing ultra-sparse MoE training (top-8/768), we found that lower-layer experts can become functionally useless while training/validation loss and load balance remain normal. All experts in these layers have much smaller weight norms than those in healthy layers. We call this silent expert death[Figure 1]. And it wasn’t just us. Auditing public MoEs, we found similar early-layer collapse signatures. For example, MiMo-v2.5-Pro and Qwen3.5-397B-A17B [Figure 2]. Benchmarks can hide unused capacity. Why? This is not merely a load-balancing failure. We found the learning signal itself disappears. Shallow-expert momentum (the accumulated gradients), as well as the square root of second moment, fell by up to ~3 orders of magnitude while deeper layers stayed stable[Figure 1]. By the time norms collapse, the model has already largely stopped relying on the lower pathway. We introduce LM Loss as Auxiliary Loss (LLAL): temporarily connect an early MoE layer to the LM head during the first few thouands steps then remove it. On 60B /180B models, this simple strategy rescued the lower layers—and the gain persisted after removal. It improved nearly every aspect of expert learning: • Better validation loss [Figure3] • No silent expert collapse [Figure 4] • Better downstream metrics [Figure 4] • Markedly better load balance Check out our blog for more details! A bunch of experiments inside—enjoy the 40-minute read :)

I saw a slide circulating on social media last night while working on a deadline. I didn’t comment immediately because I wanted to understand the full context before speaking. After learning more, I feel compelled to address what I witnessed during an invited talk at NeurIPS 2024 by Professor Rosalind Picard. I deeply respect Professor Picard’s scholarship and contributions to the field. However, her comments during the talk reflected a deeply troubling and racist view of Chinese scholars. This was not just inappropriate but also profoundly disheartening. First, it was entirely unnecessary to mention the student’s nationality when discussing an incident of cheating. The point about academic integrity could have been made without emphasizing nationality. Yet, Professor Picard chose to highlight it. This choice perpetuates harmful stereotypes about Chinese scholars and reflects a broader bias against Asians, often rooted in the assumption that we “work hard, avoid conflict, and don’t push back.” This needs to change. Asians, like everyone else, have the right to speak out and demand accountability when racism occurs. We will ensure that being racist against Asians has consequences, including here, Professor Picard. What made this incident worse was how it unfolded during the Q&A session. A Chinese attendee asked a professional and thoughtfully articulated question. She began by thanking Professor Picard for her talk and posed this question: Are you calling out the student’s nationality because you find most Chinese scholars honest, and the fact that the cheating student was Chinese is rare? Is that why you emphasized nationality? This was a generous and high-EQ question, offering Professor Picard an opportunity to reconsider or clarify her comments. Unfortunately, she doubled down instead. Professor Picard reinforced her remarks by quoting the student’s excuse —that ethics wasn’t taught in their school—and generalized this as a broader issue with Chinese education. This statement is both factually incorrect and deeply offensive. There are glaring logical flaws in this argument: 1.If the student cheated, why would their excuse about ethics education be taken at face value? A serious scholar would investigate the claim before making it a central part of their argument. 2.Even if the student’s school didn’t teach ethics (which is false for schools in China), other sources like family and community often instill strong ethical values. Ignoring this nuance is careless and reinforces stereotypes. What is most heartbreaking is that Professor Picard couldn’t even acknowledge something as simple as: “Most Chinese scholars are honest and upright.” Instead, she focused on the singular exception and added, “Of course, with this one exception in this case” in her response. I regret that this happened at NeurIPS. I regret that this happened in my research community—a place I have cherished and contributed to for over 14 years. I regret that this happened at MIT, an institution of excellence and aspiration for many Chinese scholars. Racism has no place in academia, and incidents like this tarnish the principles of inclusion and respect that we, as a global research community, should uphold. I hope NeurIPS and the broader academic community take this as a wake-up call to address the biases and systemic issues that enable such comments to go unchallenged. We must do better. @MIT_CSAIL @NeurIPSConf

@moderncpp7 @clu_cheng @NeurIPSConf @drfeifei @jiahuiyu @edchi I didn't see the talk, but the images I've seen of the slide seem quite offensive. Such generalizations should have no place in NeurIPS or anywhere else.

Mitigating racial bias from LLMs is a lot easier than removing it from humans! Can’t believe this happened at the best AI conference @NeurIPSConf We have ethical reviews for authors, but missed it for invited speakers? 😡

What makes this slide even worse is the little trick trying to hedge the language: the speaker first make an offensive quote specifically targeting a particular nationality, then make the note that tries to exempt herself from the responsibility, or equivalently “I didn’t say it, you said it yourself”. It is rather unfortunate to see language technique is used in this way 🙃

I was at the invited talk. As a Korean researcher, I know that such ethnic stereotyping could target any Asian academic community. One of the questioners confronted the inappropriate generalization, and Prof. Picard's response was inconvenient: "Maybe there is one, maybe they are common... I hope it was an outlier." The audience applauded when the questioner concluded with: "I hope in the future, if you present this again, you can remove that nationality note because that seems unfair for this special group of people." I think today it's directed at Chinese students; tomorrow it could be any of us. #NeurIPS2024# @NeurIPSConf

It is just so sad that the #NeurIPS2024# main conference ended with such a racist remark by a faculty when talking about ethics. How ironic! I also want to commend the Chinese student who spoke up right on spot. She was respectful, decent, and courageous. Her response was exemplary: she began by acknowledging the speaker’s efforts, then gave the speaker an opportunity to clarify (though, regrettably, the speaker’s reply only reinforced her bias), and ultimately called attention to the inappropriate racial bias and offered constructive suggestions. Thank you for speaking out!

If you feel the need you make the "I'm not stereotyping" disclaimer, maybe you should reconsider what you're saying She could have saved a lot of words by deleting "Chinese" and "NOTE: I am not a racist but blah blah blah blah"

🚀Excited to share our latest #EMNLP2024# work on benchmarking the long context ability with KV Cache compression across RNN-based architectures, token eviction, prompt compression, and quantization. We also provide an easy-to-use codebase (it also has my favorite WoW quote 😉). Feel free to give it a try and ⭐ it if you find it useful! 📄 Paper: 💻 Code: Some interesting findings/suggestions include: 1️⃣ Maintaining an uncompressed prefill process is essential for performance, especially with harder tasks. 2️⃣ Combining RNN-based models with attention significantly enhances long-context capabilities. 3️⃣ In "needle-in-a-haystack" evaluation for recent LLMs like Llama-3, we should use longer needles (like 64 digits) since these models tokenize multiple digits into one token. More results and insights can be found in the paper! Kudos to all collaborators: @jiayiy, Hongyi Liu, @henryzhongsc, @YuNengChuang, Songchen Li, Guanchu Wang, Duy Le, @serendip410, Vipin Chaudhary, @ZhaozhuoX, @ziruirayliu, @huxia

Introducing MLR-Copilot: autonomous machine learning research with LLM agents, which → generate research ideas → implement experiments → execute implementation with human feedback 📑Paper 🔨Code 🤗Demo

1/4 Reproducing research results in ML is hard: no code, vague descriptions, noisy results.A lot of effort @huggingface goes into making new methods available for the community, thus we wrote a blog with the challenges and strategies on the example of @GoogleAI’s Infini-Attention

Thanks Forbes for discussing about our work: Arxiv: . We rigorously defined and studied inductive reasoning and deductive reasoning in the era of LLMs. Thinking of how in-context-learning is working and why neural models are not good at deduction compared with symbolic systems, we doubt that deductive reasoning poster a greater challenge to LLMs, compared with inductive reasoning. Welcome more people to studying this interesting topic.

Attending #ICML2024#. Will be presenting our Spotlight Poster: “LLM Maybe LongLM: SelfExtend LLM Context Window Without Tuning” on Tue 23 Jul 1:30 p.m. CEST — 3 p.m. CEST at Hall C 4-9 #806# and our Poster: “KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache” on Tue 23 Jul 1:30 p.m. CEST — 3 p.m. CEST at: Hall C 4-9 #812# Come across to the poster if you are interested in Long context, Efficiency or new architecture for LLMs! 🌟 Or DM me for a coffee chat anytime for in-depth discussions!😌

🔥We release the first open-source 1.4T-token RAG datastore and present a scaling study for RAG on perplexity and downstream tasks! We show LM+RAG scales better than LM alone, with better performance for the same training compute (pretraining+indexing) 🧵

Introducing the LTSM-bundle Package! 🌟Thrilled to launch our open-source tool 🔧Assess various crucial designs to train Large Time Series Models (LTSMs), and identity the best training practices 🔗 Paper: 🔗 GitHub:

Our Amazon Stores Foundational AI team is seeking talented PhD students to join us as research interns in Fall/Winter 2024. We are looking for candidates with top-NLP/ML publications and familiarity with LLM research. Contact me at yifangao@amazon.com if you are interested.