Alex Ker 🔭
@thealexker
code+words @baseten | investing in frontiers & sharing my curiosities | prev @bloombergbeta @stanfordhai @neurable.
1.3K Following 13.3K Followers
excited to share we're building a research lab, an initiative to keep pushing open source ecosystem forward!
Today we're launching Base Labs, a research lab by Baseten. Our mandate is to make open-source as useful as possible, and our one rule is that we publish without exception, including what fails.
Up until pretty recently I thought the way to get the world onto open models was to train them for one company at a time. @mudithj, @maxkirkby and I cofounded @parsedlabs on that bet, @baseten acquired us, and we spent the last year running their training team doing it for customers one by one. Every one of those engagements taught us something new about how models learn, forget, specialise and get cheaper, and almost none of it got spoken about. Sadly, in general that's the field's default in that the people who know the most about training have the least freedom to say it.
To be clear I don't think the closed labs are the villains here. They get to new capabilities first, which buys the rest of us time to harden the world before that stuff is everywhere, and they're the ones paying to find out what's actually possible. But I'm fairly convinced the only real advantage they have is data and scale, and their incentives point squarely at the frontier. You can't do slow, public science on how these things learn when your job is the best model by end of quarter. Someone without that pressure has to, and there are very few of those someones around. Hence Base Labs.
We have the broad remit of making open-source as useful as possible and our one rule is that we publish without exception, including what fails. The first problem is continual learning, which I have come to think is several problems wearing one name. We are also working on the open RL environments and data that open models need and currently can't get, because we have to aggregate data with the same ferocity everyone's been talking about aggregating compute. Plus a bunch of other stuff I'm genuinely excited about, eg a safety stack people can run on top of open deployments, and performance research so these things are cheap for everyone to serve.
I still think open and closed coexist, and that's the good world. It's just that coexistence isn't free, someone has to actually do the work, and this is basically what keeps me up at night. We're hiring researchers, engineers and fellows. Come help distribute the mandate of heaven!
Show more
recent notes/learnings on harnesses (small n, directionally correct, experimental results soon):
> running harness experiments is brutally expensive! e.g. @guanlan's experiment used 2B tokens but only covered 30 tasks (21 terminal-bench tasks and 9 deepswe tasks)
> the best harness is one that's the most bitter-lesson pilled and cost-efficient. you do this by letting your harness adapt at runtime (rsi) and align itself with actual tasks and data. common failure mode is being too opinionated a priori or over optimizing for one model. good properties of harnesses have been mostly commoditized/converged.
> as horizons of execution get longer, two things stand out:
1) speed (does the harness know when to use subagents, fanouts, and have good primitives around them).
2) the ability to unblock itself. this means not getting stuck in a particular loop or deadzone but stepping outside without human intervention.
what do you look for in good harness design?
Show more
recent notes/learnings on harnesses (small n, directionally correct, experimental results soon):
> running harness experiments is brutally expensive! e.g. @guanlan's experiment used 2B tokens but only covered 30 tasks (21 terminal-bench tasks and 9 deepswe tasks)
> the best harness is one that's the most bitter-lesson pilled and cost-efficient. you do this by letting your harness adapt at runtime (rsi) and align itself with actual tasks and data. common failure mode is being too opinionated a priori or over optimizing for one model. good properties of harnesses have been mostly commoditized/converged.
> as horizons of execution get longer, two things stand out:
1) speed (does the harness know when to use subagents, fanouts, and have good primitives around them).
2) the ability to unblock itself. this means not getting stuck in a particular loop or deadzone but stepping outside without human intervention.
what do you look for in good harness design?
Show more
so good. absolutely worthwhile read to understand the first, second, and third-order effects from the progress in AI. feedback loops will be everywhere
most people forget there are two vectors to optimize for to reduce model cost:
1) reducing the input/output costs, increasing cache hit rates, batching etc
2) packing more intelligence per token (fewer tokens for same task)
inference cost = price per token × tokens per task
optimizations around the second is underrated and something we’ll only see more of.
compression and concision is intelligence.
Show more
so good. absolutely worthwhile read to understand the first, second, and third-order effects from the progress in AI. feedback loops will be everywhere
most people forget there are two vectors to optimize for to reduce model cost:
1) reducing the input/output costs, increasing cache hit rates, batching etc
2) packing more intelligence per token (fewer tokens for same task)
inference cost = price per token × tokens per task
optimizations around the second is underrated and something we’ll only see more of.
compression and concision is intelligence.
Show more
hot take...human-in-the-loop is the same as micromanagement for ai agents, and it's a disappearing pattern.
the best managers check for results, rather than controlling process. there’s an implicit level of trust that people will use their talents to figure out the optimal way to do their piece of the puzzle.
we’ll eventually grant the same trust to our agents. no one reviews the assembly their compiler generates, because the trust has been earned.
I think we’re all in various stages of collecting evidence to convince ourselves to let go. soon, final review & verification will live as the few remaining tasks in engineering.
software development will start to look more and more like self-driving: by default trusting, with less oversight over time. right now we're still clutching tight to the wheel
Show more
hot take...human-in-the-loop is the same as micromanagement for ai agents, and it's a disappearing pattern.
the best managers check for results, rather than controlling process. there’s an implicit level of trust that people will use their talents to figure out the optimal way to do their piece of the puzzle.
we’ll eventually grant the same trust to our agents. no one reviews the assembly their compiler generates, because the trust has been earned.
I think we’re all in various stages of collecting evidence to convince ourselves to let go. soon, final review & verification will live as the few remaining tasks in engineering.
software development will start to look more and more like self-driving: by default trusting, with less oversight over time. right now we're still clutching tight to the wheel
Show more
two years ago
this has definitely aged well and is more relevant than ever
post-training is the next frontier for scaling laws
if you want to understand the efficient post-training mechanics behind open models today, I summarized all the innovations for glm-5.3 in plain english, covering environment design, architecture, as well as the rl algorithm and infra behind it:
Show more
not enough people stop and think about how the best tech of the past few decades was built on open source: linux, git, much of the open web.
70-90% of software in a modern codebase is open source. the beauty is you’re always standing on the shoulders of giants. there’s a reason why open-weight is progressing exponentially.
glm-5.3 evolved from attention -> deepseek sparse attention -> indexshare. even with limited compute compared to closed labs, teams cut attention below quadratic cost through making architecture and algorithms efficient. open-source wins stack and compound. decentralized innovation is truly a remarkable force.
Show more
An extremely worrying pattern I've seen recently is hearing more claudish irl.
we need to guard against slop by actively de-conditioning: read old books, talk to strangers, travel outside your bubble, write with pen and paper, limit your AI screen time.
give your neurons something to touch outside the distribution.
Show more
we’re really in the era of flash models:
glm-5.3 flash, qwen 3.8 flash next, deepseek V4 flash 0731.
small, open models matching opus quality is perfect for many 20+ subagents, a pattern that’s making harnesses much more powerful
Show more
post-training is the next frontier for scaling laws
if you want to understand the efficient post-training mechanics behind open models today, I summarized all the innovations for glm-5.3 in plain english, covering environment design, architecture, as well as the rl algorithm and infra behind it:
Show more
unintended light homage to @joshuastarmer, "simply explained" felt like a hard bar to hit on something as complex as GLM-5.3!
