Uh oh. If your AI benchmark uses an LLM-as-judge, you might want to read this. AllenAI tested six frontier models to see whether research ideas were actually novel. And tiny changes to the judge prompt caused crazy swings. GPT-5.4, same basic idea pairs “Which was judged novel?” → 93.5% “Which is novel?” → 40.9% “Which was judged more novel?” → 59.7% Across the study, changing the prompt moved accuracy by 👉50+ points 👈 in some cases and occasionally pushed judges below chance. So naturally they tried giving the judge more intelligence and tools. Didn't fix it. A GPT-5.6 Sol agent running xhigh reasoning with iterative literature search and no search cap scored → 0.78 macro-F1 Claude Opus 4.6 with only 5 pre-retrieved abstracts did about → 0.89 And the big agentic setup cost ~10× more per judgment. Two purpose-built novelty-evaluation systems lost to the researchers' cheapest simple prompted judge by → 8–29 macro-F1 points while costing up to → 32× MORE 🎯 Lesson Learned 👉 More model, more reasoning more RAG, and more money, doesn't automatically mean a better judge. AllenAI open-sourced the benchmark + code too, so now I want to see Qwen, DeepSeek and other local models run as the judges. The safer benchmark recipe is 👨‍🔬 build a human-rated calibration set 🤖 test the LLM judge against it 🔄 test multiple prompt/order variations 🗳️ aggregate multiple judgments 👀 send close calls back to humans Link in ALT

Hugging Face is becoming more and more the home of AI x Science projects. Open source accelerates scientific discovery when models, datasets, and tools are shared openly, all researchers can reproduce them and build on top of them. A big thanks to @allen_ai, @OpenMed_AI, and @NVIDIAAI for being the most active contributors of AI for Science artifacts on @huggingface these days! 🤗 Hugging science heatmap:

Another benchmark, another #1# for MolmoAct 2! 🤖🏆 Our robot policy tops @datapointai’s human-preference rankings for real-world manipulation tasks, including candy sorting & cup stacking. Thanks to the Datapoint AI team for the evaluation & open dataset!

This Friday, meet the open source AI community at Open Together, co-hosted with @huggingface. Join us at The Midway in SF on Oct 16 — bringing the community together to help build a future where AI is open, secure, and fully transparent. RSVP:

That's a wrap on #COLM2026#! Thank you to everyone who stopped by our booth, joined our evening gathering, & talked open research with us. Catch up on the papers, interviews, & highlights from the week. 👇

An interesting paper was just published by @allen_ai in Nature on ditching subword tokenization without burning millions of dollars in compute: Retrofitting language models to operate over bytes. The classic headache with byte-level models has always been speed and the sheer cost of training them from scratch. Instead of starting from zero, the authors introduce "byteification" – a neat two-stage distillation trick to retrofit existing pretrained models (like Olmo, Llama 3, and Qwen) into byte-level systems using under 1% of the original pretraining budget. The key architectural insight addresses a subtle flaw in earlier latent tokenizer setups. Subword tokenizers look ahead at future bytes when drawing boundaries, whereas standard byte-level patch predictors were strictly causal. By adding just 1 byte of lookahead during prefill and predicting fused boundary tokens during decode, they bridge that expressivity gap. The resulting models (Bolmo, Blama, Bwen) run at practical inference speeds, crush character-level reasoning tasks where standard tokenizers trip up, and you can even transfer instruction tuning across using simple task arithmetic. Definitely worth a close read if you work on token-free architectures.

We're co-hosting Open Together with @huggingface! Join the open AI community at The Midway in San Francisco on Oct 16 to kick off Open Source AI Week. Help us build a future where AI is open, secure, and fully transparent. The event is free. RSVP:

Heading back from #colm2026# with a question I heard over and over from researchers and academics: are you still building Olmo models? I know it has been an up-and-down year for @allen_ai, but I’m surprised this is even a question. The next Olmo model is already training, the team is growing, our compute is growing, and Ai2 is committed to building the fully open models that the community has grown to depend on.

What’s happening at #COLM2026#? 👀🎤 Ai2 Comms Lead @Kyle_L_Wiggers catches up with Senior Director of NLP @nlpnoah to talk agentic models, the next iteration of Olmo, & more! At COLM? Come say hi at booth 302! 👋

Thrilled to share that our work on LLM byteification is out in @Nature today! 🎉 Byteification lets existing LLMs gain the strengths of byte-level models without training from scratch. Huge kudos to lead author @bminixhofer, who did most of this in a three-month internship! 🤯

Introducing AstaBrief 8B, an open model that turns complex research questions + literature excerpts into cited reports. Run it locally on your own hardware, with open weights + training data you can inspect & build on. 🧵 🤗 Download:

We're expanding our partnership with @huggingface to accelerate open science. Our storage on the Hub is roughly tripling to ~2 petabytes, & our downloads now run at high speed—even for our largest datasets & multi-checkpoint models. 🧵