Shu Lynn Liu
@shulynnliu
274 Following 1.8K Followers
Quick intro video on 💰HarnessTax💰 is out!!
I also shared some thoughts on what it means for our community to be aware of such cost in coding agents at the end!
Could long-context architectures “find all contradictions” in a science literature? Not yet! 🧵
We study a new class of "high-complexity” tasks whose difficulty scales quadratically with corpus size (as opposed to linearly), reversing common LCLM decisions! (block-sparse attention, hybrid models…)
Show more
super excited that our paper on Inductive Deductive Synthesis just got accepted to NeurIPS as 🏅Oral🏅 (top 0.34%)
in the paper, we show how given a formal Lean/Rocq spec, we can incrementally co-synthesize *proof* with the *code*, beating direct code synth
link &takeaways⬇️🧵
Show more
Excited to share our work on IDS was selected for a #NeurIPS2026# Oral! 🚀
Instead of generating code with agents and praying tests pass, IDS co-evolves code + formal proofs, outperforming SOTA agents to solve several distributed systems specs.
Show more
Formal verification × agents 🚀
Our IDS paper was accepted as a NeurIPS oral! We study how agents can co-evolve code + proofs, achieving 3x the success rate of Claude Code on distributed systems specs.
IDS Paper:
Code:
And also check out our recent work on the next challenge: ensuring the spec itself captures human intent. 👇
Show more
I used Opus 5.5 to formally verify the Claude Agent SDK using Lean. A couple short prompts = 16 PRs fixing various bugs and race conditions. Video attached.
TLA+ also works well. I sometimes combine Lean and TLA+ to look for issues around data flow, concurrency, and state mgmt.
I don't know either language well, but Claude is excellent at both. This approach is super useful for formally modeling your code and finding bugs that a human probably wouldn't have spotted.
Is formal verification the future of coding (or at least, bug finding)?
Show more
We pretrained a 2.3B MoE (360M active) Hybrid Mamba-2 that lands within a few points of Llama-3.2-3B using <1% of its pretraining FLOPs.
No dedicated cluster. The run hopped between H100s, A100s, V100s (yes, V100s) and TPU v5p/v6e on a single codebase.
Meet Rigel 🧵
Show more
Interestingly, one attention head can be important across five ICL task families!
Our original thread gave an in-depth mechanistic analysis of addition ICL (quick video recap 👇). We've now generalized this to four more ICL task families, spanning both arithmetic tasks and semantic tasks!
Our paper “Understanding In-context Learning of Addition via Activation Subspaces” has been accepted to COLM 2026! 🎉
See more details in thread.
(1/N)
Show more
Adding more reviewer agents doesn’t close a gap they all inherit!
If requirements or deployment assumptions are incomplete, every agent can agree, pass every eval, and still build the wrong thing.
Passing every eval ≠ building what we actually wanted.
Check out a paper that explores these gaps in agentic software engineering 👇
Show more
Does your Claude model really need Claude Code…? 🤔
We evaluate 7 models on Claude Code, Codex, and Pi. Three surprising findings emerge:
1️⃣Harness choice has little effect on task success rate, but can significantly affect the cost
2️⃣A simple harness can be competitive
3️⃣The native harness isn’t always the best.
Millions of people are using coding agents, but the impact of harness choice remains unclear.
(1/n) More details in the thread. 🧵
Show more
Agents let us build systems for different workloads and requirements. But… can we trust what they build?
We release 🌟SkySynth🌟: an engine for synthesizing high-performance, just-in-time (JIT) systems we can trust, by co-evolving formal proofs and tests alongside the code.
Results:
💿 KV stores up to 2.3× faster than Redis and FASTER + formally verified stores with 2.9× Claude Code's pass rate
🚏 Model routers up to 48% cheaper than a general router
⚡ Specialized inference engine with 2.2× the throughput of vLLM/SGLang
🧵👇
Show more
AlphaEvolve is closed-source. We release 🌟SkyDiscover🌟, a flexible, modular open-source framework with two new adaptive algorithms that match or exceed AlphaEvolve on many benchmarks and outperform OpenEvolve, GEPA, and ShinkaEvolve across 200+ optimization tasks.
Our new algorithms dynamically adapt their search strategy, and can even let the AI optimize its own optimization process on the fly!
Results:
📊 +34% median score improvement on 172 Frontier-CS problems.
🧮 Matches/exceeds AlphaEvolve on many math benchmarks
⚙️ Discovers system optimizations beyond human-designed SOTA
🧵👇
Show more
