Run #LLMs# on @NVIDIARobotics #Jetson# without x86 ONNX export. 🚀 #TensorRT-Model-Connect# on reComputer Classic J5012 (#AGXOrin# 64GB, #JP7#.2). @huggingface checkpoint → trtmc build → .bundle on device trtmc run for text gen Tested: #Qwen3-4B# FP16 (~41GB container peak, 64GB recommended) No ONNX hop. Edge LLM/agent ready. Would you use on-device TRTMC for prototyping, or keep ONNX export for production? 👇 Full guide: Get your own one: #NVIDIA# #Jetson# #TensorRT# #TRTMC# #AGXOrin# #EdgeAI# #LLM# #PhysicalAI# #SeeedStudio#

oMLX 0.7.1.dev2 is out! oMLX can now serve Qwen3.8 and Qwen3.6 through Splash as an external engine. Splash runs the model on its own Metal engine with DFlash2, and the model still shows up in oMLX's model list, API, dashboard, cache stats and benchmarks like any other. This release also helps first-time users find a model that fits their Mac without guesswork, reworks model settings and chat in the web UI and the macOS app, and speeds up GLM-5.3 and DFlash. Splash as an external engine - Turn it on per model in Model Settings > Advanced. oMLX finds the Splash launcher and hands it the model and DFlash2 draft from your model directories, so nothing extra is downloaded. - Memory and context window are fitted to your oMLX memory limit, and an MLX model can run alongside it. - Vision checkpoints take images too. Speedups in this release - GLM-5.3-Flash decode on M5 Ultra: 85.5 -> 95.3 tok/s (+11%) at 64K context. - GLM-5.3-Flash, 4 streams on a shared 32K prompt on M5 Ultra: 81 -> 91 tok/s (+12%). - DFlash with 4 concurrent requests on M3 Ultra: +82% to +157% (MiMo V2.6 Flash, Qwen3-4B, Muse Glimmer 30B). What's new - Find the right model for your Mac. New users no longer have to guess which model to start with. The new Recommended tab shows only oQ models that will actually load on your Mac, with Lightning MTP on by default. - Reworked model settings. The web UI groups them into sections with a side menu, and the macOS app splits them into pages that save as you edit. - A new chat composer. Pick the model, profile and reasoning level right where you type, with a reasoning slider that follows the levels each model accepts. - DFlash now runs inside the batched engines, so it gets continuous batching and the prefix cache. DSpark drafters are accepted 61-85% of the time instead of 16-21%. - Requests survive memory pressure. They are paused and resumed instead of aborted, and cancelled prompts stay cached. - /v1/decisions for any chat model, plus the Decision 3.0 models with image and video input. - YaRN for Qwen3.5-family models past the native 262K context. - An "Allow inference without API key" toggle for LAN setups, and a Mistral Vibe launcher. - oQ for DeepSeek V4.1 at oQ2 to oQ2.7 and for EmbeddingGemma 2, plus MiniCPM-V 4.7 support. - A log table in the macOS app, copy buttons for model names, and Downloaded badges in the model browser. Most of the speedups came from contributors again. Thanks to everyone who sent PRs and tested the dev builds!

How did I miss this?! Bonsai isn’t just an LLM family. Back in May, PrismML released Bonsai Image 4B, a crazy low-bit version of FLUX.2 Klein 4B designed to run locally on 🍎 iPhones. Look at this 🎨 FLUX.2 Klein 4B transformer 💾 FP16: 7.75GB 🌳 Ternary Bonsai: 1.21GB And the entire Apple deployment payload, including its compressed text encoder + VAE, is only: 🔥 3.88GB 📱 iPhone 17 Pro Max 🧠 A19 Pro / 12GB unified memory 🖼️ 512×512 ⚡ 9.4 sec/image 📲 MLX Swift ☁️ No cloud 💾 Bomsai Studio (App Store) M4 Pro: ~5.8 sec/image. No cloud. The ~4B diffusion transformer is paired with a 4-bit Qwen3-4B text encoder, which gets unloaded after the prompt is encoded to save memory. And there’s also 🍎 Mac / iPhone / iPad support 🟢 low-bit Gemlite builds for Nvidia GPUs 🔓 Apache 2.0 This is completely separate from the Bonsai 2 27B LLM I’ve been posting about. PrismML took a 7.75GB FLUX transformer and crunched it down to 1.21G and put image generation on an iPhone. 👀 And somehow I missed this for 4 months.

TL;DR: Existing on-policy distillation methods that try to surpass the teacher destabilize training by amplifying noise in output space. A new method, RIDE, instead extrapolates the RL-induced representation change directly in representation space — and beats the teacher on all four tested model pairs. Title: The Teacher Is a Direction, Not a Destination: Extrapolating RL-Induced Representation Residuals in On-Policy Distillation URL: Points 🎯 Treats the teacher as a direction, not a destination — regresses the student toward a target that extrapolates the pre/post-RL hidden-state residual ⚠️ The LM head's anisotropic spectrum attenuates most of that change: the weakest 512 head directions carry 79.8% of hidden-state energy but only 30.0% of output-space energy 📉 Output-space extrapolation's variance grows as (λ-1)²; RIDE's gradient variance is 12.6x lower than ExOPD's 📊 Beats the teacher on all 4 model pairs: 56.38 on R1-Distill-1.5B, 66.07 on Qwen3-4B, outperforming OPRD by 0.97–4.06 points ❌ The output-space baseline ExOPD underperforms both the teacher and OPRD on every pair 🔬 Ablations confirm the direction itself matters: random or reversed directions don't deliver the gain that the true RL-induced direction does It's striking how much the choice of where you measure the difference changes whether "surpassing the teacher" actually works. #Distillation# #ReinforcementLearning#

⚡ DSpark vs DFlash: Up to 2.55× Throughput in a vLLM Test With DSpark checkpoints and vLLM support now available, parallel speculative decoding is becoming a practical serving option rather than just a research idea. Zhihu contributor kaiyuan explains how DFlash and DSpark work, then benchmarks both on the same Qwen3-4B target model. The result: DSpark reached 2.45–2.55× baseline throughput, while DFlash achieved 1.96–2.09×. 1️⃣ Why LLM Decoding Is Naturally Slow Autoregressive models generate text one token at a time. Token n+1 cannot be produced before token n. This sequential dependency limits how much parallel GPU compute can be used during decoding. Non-autoregressive generation works differently. It treats generation more like filling multiple blanks and can predict several positions in one forward pass. Parallel speculative decoding combines both ideas: 🔹 A lightweight draft model proposes several tokens in parallel. 🔹 The target model verifies them in one batch. 🔹 Accepted tokens are kept; generation restarts from the first rejection. The challenge is making the draft both fast and accurate enough to be useful. 2️⃣ Why Traditional Draft Models Hit a Wall Conventional speculative decoding often uses a smaller autoregressive model such as EAGLE. It produces higher-quality drafts, but still generates candidate tokens sequentially. Longer drafts require more draft-model forward passes. Making the draft model larger improves accuracy but also increases latency. Parallel drafters solve the latency problem by proposing an entire token block at once. However, later tokens in that block do not fully depend on earlier predictions. Their accuracy often drops quickly, creating suffix acceptance decay. 3️⃣ How DFlash Improves Parallel Drafting DFlash uses a parallel, fill-in-the-blanks-style draft model. To improve draft quality, it extracts hidden states from several layers of the target model and fuses them into an additional context representation. Each draft layer then attends to two sources: 🔹 Context derived from the target model 🔹 Representations from the draft tokens themselves The target model’s hidden states make the lightweight drafter more informed without requiring multiple autoregressive passes. DFlash can therefore propose a long block in one forward pass. But longer blocks still create a problem: the suffix is more likely to be rejected, while the target model must spend compute verifying it. 4️⃣ DSpark Adds Sequential Structure DSpark extends DFlash with semi-autoregressive generation. It first uses a parallel backbone to generate intermediate logits for multiple positions. A lightweight sequential module, implemented with an RNN or Markov head, then produces the draft tokens from left to right. This small sequential step restores dependencies inside the block without giving up most of the parallel speed. It also predicts a confidence value for every token: the probability that the token will survive target-model verification if the previous prefix is accepted. The result is a stronger draft with less suffix decay. 5️⃣ Verification Length Becomes Dynamic DSpark does not automatically send the entire draft to the target model. Its Hardware-Aware Prefix Scheduler considers: 🔹 The survival probability of each draft prefix 🔹 The current batch size and system load 🔹 A profiled steps-per-second curve for the hardware Verifying one more token may increase the expected accepted length. But it also enlarges the verification batch and can reduce processing speed. The scheduler expands each prefix only while estimated throughput continues improving. Low-confidence suffix tokens are discarded before they consume target-model compute. Draft long, but verify only the prefix that is still worth verifying. 6️⃣ The vLLM Test Setup The author tested both methods under the same environment: 🔹 8× NVIDIA A800-SXM4-80GB 🔹 Qwen3-4B target model 🔹 DSpark block-7 and DFlash block-16 draft models 🔹 vLLM 0.26.0 The initial comparison used DSpark with four speculative tokens and DFlash with seven. On 250 GSM8K questions: ✅ DSpark: 35.2% accuracy, 0.75s average latency ✅ DFlash: 31.6% accuracy, 0.84s average latency On 250 MMLU questions: ✅ DSpark: 28.8% accuracy, 0.24s average latency ✅ DFlash: 27.6% accuracy, 0.27s average latency DSpark reduced average latency by roughly 11% in both tests. 7️⃣ Throughput Is the Stronger Result Because the two methods used different speculative-token settings, the author swapped those parameters and tested again. 🔹 Original settings: DSpark=7, DFlash=4 DSpark reached 584 tok/s, while DFlash reached 449 tok/s. ✅ DSpark was 1.30× faster. 🔹 Swapped settings: DSpark=4, DFlash=7 DSpark reached 561 tok/s, while DFlash reached 480 tok/s. ✅ DSpark remained 1.17× faster. Changing num_speculative_tokens affected throughput by less than 7%. DSpark remained faster in both configurations, showing that its advantage did not come from receiving a more favorable draft length. Compared with the 229 tok/s baseline: 🔹 DSpark delivered 2.45–2.55× throughput. 🔹 DFlash delivered 1.96–2.09× throughput. At matched settings, DSpark stayed roughly 20% faster than DFlash. ⚠️ Do Not Overread the Accuracy Numbers The accuracy differences are less conclusive. MMLU results varied by about one percentage point. On GSM8K, even the baseline changed from 29.2% to 34.4% across two runs. The author attributes this to different vLLM batch compositions changing floating-point accumulation order. That can alter a small number of token choices even with temperature=0. So the test strongly supports a throughput advantage. It does not establish that speculative decoding improves model intelligence. 💡 The Practical Takeaway DFlash proves that parallel drafting can generate many candidates cheaply. DSpark adds the two components needed for production serving: 🔹 Lightweight sequential modeling to improve draft quality 🔹 Load-aware scheduling to avoid unnecessary verification Its real contribution is not simply drafting more tokens. It is deciding which tokens are still worth verifying under the current serving load. 🔗 DeepSpec: 🔗 Test notebook: 🔗 Full Reading: #DSpark# #DFlash# #SpeculativeDecoding# #vLLM# #LLMInference# #AIInfrastructure# #DeepSeek#

// The agent is its own best speculator // Agents spend a large share of wall-clock time waiting on tool results. Speculation hides that latency by predicting and pre-executing the next call, but external draft models and cached traces are modeling a different policy, so they miss too often to help. New research from UC Santa Barbara and LinkedIn unifies both roles in one model. It runs in agent mode to solve the task and in speculator mode to predict its next tool call from a partial trajectory, fully reusing the prefix KV cache. Joint agent-speculator reinforcement learning derives speculation targets from the agent's own rollouts and alternates updates between the two modes, which keeps dual-mode training from degrading task success. Next tool-call Hit@1 rises from 44.1 to 61.2 for Qwen3-4B and from 48.9 to 66.3 for Qwen3.5-4B, with agent task success preserved. Paper: Learn to build effective AI agents in our academy:

1d 13h 20m, 3,596,831 tokens. Goal achieved? Not quite. It was a hard problem. The agent tried its best and went through 20 full model/eval rounds. In the end, the agent talked itself out of the original contract and declared the goal achieved. I probably would have stopped it anyway, since I could also see from the sidecar that it was struggling. Still, it was a good experiment. My 14" MacBook Pro held up well under a sustained run, with no throttling or heating issue. Qwen3.6 35B A3B OptiQ 4-bit running locally on MLX also held up well. It generated thousands of training data samples, averaging around 50 tps with reasonably good quality. Very impressive. DeepSeek 4 Pro was a good teacher for the training, though there are still areas for improvement. The end result: we LoRAed an expert model, Qwen3-4B-Instruct-2507 + MLX LoRA. We produced a compact 56 MB LoRA adapter on a 4B Qwen base that reaches ~59% three-way decision agreement on the original eval slice, ~91% violation recall, and ~98% valid JSON, but with a high false-positive rate. It is deployable, but probably not quite usable yet. Still, it gives me a clear direction for where to go next. I’ll write more about the whole process later. Stay tuned.

Qwen3.5-4B running CPU-ONLY at up to ~11+ tps on a Ryzen 5 laptop. No GPU. Running models in RAM and CPU is something the Qwen4 team is already thinking about. 💡 I built this standalone .exe because businesses are already deploying local models to save money and ensure privacy. (thumb drive friendly - just drag drop and doubleclick) My llama.cpp recipe 👉 🧠 Qwen3.5-4B Q4_K_M GGUF ⚙️ Ryzen 5 7540U — 6C/12T 🧵 --threads 9 🧵 --threads-batch 12 ⚡ --prio 2 🔄 --poll 50 📦 --batch-size 2048 📦 --ubatch-size 512 🚀 --flash-attn on 🧠 KV cache: q4_0 / q4_0 🔧 --repack 💾 --mmap 👤 --parallel 1 🚫 --device none 🚫 --gpu-layers 0 🚫 KV/op GPU offload 🚫 MTP OFF Interesting result 👉 MTP=3 was slower (~10 tps in benchmarks). Plain decode + 9 threads + Q4 KV hit ~11.4 tok/s. That's about 10% faster just from tuning llama.cpp — on a basic laptop CPU.

ThumbLLM v0.1 Qwen3.5-4B CPU. Windows. Double-click. On my Strix Halo (CPU only, no iGPU offload), it sits near ~20 tok/s. Same recipe idea as the 7540U laptop post: --device none, GGUF Q4_K_M, chat + local API. Missing weights download. Unsigned. SmartScreen will complain. Hash is on the Release. Then run anyway. 🔗 GH /TeksEdge/ThumbLLM/releases/tag/thumbllm-qwen3.5-4b-mtp-q4_k_m-cpu-win-x64-v0.1.0