Alright I finally have more to share on my personal project! Resin is a runtime for building fast LLM inference engines. The idea is to make all the generic parts of inference reusable, and leave only the model-specific forward pass + kernels to be written for each architecture / hardware target. To build an engine: 1. Import Resin's core. 2. Have an agent write a forward pass for the model you’re interested in, for your specific hardware (GPU or TPU or AMD), any kernels you want. Then, boom, you have a standalone inference engine. — I've been hacking over the past month (early post: and recently this week I got a lot of features + parallelisms working, inspired by DeepInfer. For DeepSeek-V4.1-Flash, it's nearly as fast as DeepInfer (within 5-10%), while being self-contained and supporting other models, plus metrics + HTTP server + prefix cache. Yesterday I wrote a skill for authoring models and put Codex to work on it overnight. The agent built a GLM-5.3-Flash engine with MTP, reaching up to 60,000 input tok/s and 16,000 output tok/s on B200 (EP4). So far inference "vibe coding" has not been very reliable or successful. I think it's because people start from scratch each time and reinvent lots of tricky things on the scheduler and server layers. But with a shared core, we can automate lots of inference work. The tricky bit is writing a fast forward pass. But Resin turns that into a standalone engine by just importing a library, and it becomes a verifiable goal for the AI. Code:

for the record, I really like DeepSeek’s benchmarking approach, it controls for all the free variables and is probably the most compelling recent roofline curve to me. every engine should be benchmarking this way, and kv caches/pd separately see:

Inspired by @typesafeai , here is a Jev-compatible public API to play with It runs a comparable open model (Qwen3.6-35B-A3B), and just uses SGLang radix cache to preserve the prefill reuse / really fast parallel systemone generation - 64 tasks in <1s.

So… I have it working now. The core of an inference engine in Rust is about 3k lines (scheduling, kv/swa/recurrent state, prefix cache, multimodal, dynamic batching) I then asked Codex to implement (with MTP, cudagraphs, TP on Blackwell): - DeepSeekV4 Flash - Gemma 3.8 27B - GPT-OSS - Inkling It turns out that most models take a lot of code to run with fast, persistent graphable kernels and so on… on the order of 2.5k for a simple model (gpt-oss) and up to 7k for a monster like DeepSeekV4 Flash that has mHC, CSA, HCA and DSpark Not sure if I’ll continue with this, but it was fun! The simple core is reusable, and startup times are crazy fast for an inference engine with no tech debt and not having to serve a ton of dynamic configurations - just one model forward pass in nvfp4 :’) - maybe one day SGLang/vLLM will be modular like this. See interactivity-throughput below for Inkling-Small TP=1 after a few sconv kernel fusions. (This is without overlap scheduling btw! Don’t even really need it when your core + scheduler is small and Rust.)