powering the jevolution
. @modal moves mountains to make models manifest!
Our man in pink @CompleteSkeptic will be speaking at Runtime, you won't want to miss it.
how sglang scheduler looks at the decode step when a long prefill comes in
deepseek doing a minor version bump of v4 flash
many inference companies focus on tpm tps ttft -- i would much rather focus on more important things like understanding how a rat can be such a good chef
which of your favorite token brokers are contributing back to open source ai? shall we count the ways?
a slice of data: number of PRs upstreamed and merged into the most popular oss inference engines, by PR contributor org
Show more
The only part of this article I used AI for was the cover photo, I swear.
Link to session -
goated, we are going to have ornithopters now
I left a16z to start Ixian Robotics!
I’m very grateful for the people I got to learn from at a16z, and excited to be building again. The last couple of months have felt like knocking down dominoes I’ve spent years putting up.
Wrote a bit about the last year and what I want to build next:
Show more
best is implied btw
how our team was moving to provide y'all with two models in two days
GLM 5.3, the latest flagship model by @Zai_org, is now available on Modal.
holy cache hit rate
found out we're serving 40% of Kimi K3 traffic on OpenRouter ...
the guy working on it is tired. anyone wanna help us work on inference?
can confirm he is goated
one of the top 5 best researchers I've worked with is exploring dropping his PhD for industry. bg in optimization & interpretability, just wrapped an internship at Apple MLR w glowing recs
if you're hiring for research roles (esp w scope to publish) dm & happy to intro
Show more
summary: what if kernel big
chonk model on a chonk infra provider, meant to be
Qwen3.8-2.4T-A95B by @Alibaba_Qwen and @alibaba_cloud is now available on Modal.
Served with a custom DFlash speculator trained on tool-call-heavy data. Full 1M context window.
Show more
our secret is that we just inject "make no mistakes" to every prompt via the chat template
watching my loss not even move an inch on my 999th attempt at one layer deeper competition by @tilderesearch x @coreauto
I sat down with @silasalberti, Head of Research @cognition, to chat about the intersection of RL and inference -- from training frontier coding models to running inference at scale.
Watch till the end for bonus doggo! :o
0:00 — Intros
1:04 — What's hardest to get right in an RL run
3:39 — How Cognition started using Modal
4:03 — What to weigh when buying inference
4:58 — The latency/throughput Pareto frontier
7:14 — DFlash reveal (!!)
8:44 — Why faster runtime matters
9:56 — Memory-bound vs. compute-bound
10:57 — Tree-based speculation
14:20 — Hot take: RL and inference optimization converging
15:28 — Online training of the speculator
16:22 — "Auto inference"
17:27 — Agents competing on optimization problems
18:27 — What inference providers still get wrong
Show more
our team is so good its not even funny
Kimi K3 is live on Modal.
Moonshot has shipped the world's first open 3T-class model, and we're a day zero launch partner.
We trained a custom DFlash speculator for K3's novel architecture so you can run it faster, losslessly. The most capable open model we've worked with by far.
Show more
Modal has 460 TPS for Kimi K3 on release day!
let the machines think on modal
Modal trained a DFlash speculator that's much faster than MTP, making it a great boost for inference speeds!
overheard in a sports bar:
“mbappe used to skip school to watch me play”
