FULL RECAP: Apple Intelligence Group Lab, the short version for Local AI builders Sharing highlights from this week's session hosted by @shshnkp (Apple AI/ML Evangelism Team) with @mgorbach and engineers from Foundation Models, Core AI, Siri and App Intents teams at Apple. 1) Apple’s app framework can use your open-weight model as its LLM. Apple’s new app framework can use Apple’s on-device model, Apple’s cloud model or a Core AI converted model, which is interchangeable with a one liner! And your model gets access to the same tool calling, structured output, profiling with Instruments, etc. as Apple’s models. 2) Don’t use Apple’s on-device model for coding, even according to Apple engineers! Use Core AI to bring in a model for coding. One engineer uses a setup where he can use MLX and Core AI models in different “coding harnesses” and reports good experiences. 3) Free local Chat Completions endpoint on any Apple silicon Mac with macOS 27! fm serve provides an OpenAI style Chat Completions endpoint backed by Apple’s on-device model on any Mac. Nothing to install! (shared by Shashank the day after the lab) 4) Apple’s new on-device model is 20B, but only has 8K context. AFM Core Advanced (20B) will be available on the latest iPhones, iPads and M3+ Macs. Can’t opt out of Apple Intelligence in 27, and the model will always be present. Tip: use typed units for fields and let constrained decoding handle it (instead of wasting tokens explaining in prompt). 5) Agentic Siri only works when online. Tool calls from Siri to apps are done using Private Cloud Compute. When offline, Siri will use the on-device model but won’t be able to call into apps. 6) Lessons Apple engineers follow when building agents apply to any agent: - write down instructions to the model about how/when to use your tools in a markdown file (“skills” are the same idea as agent skills everywhere else!) - skills are just context to the model, meaning no ordering guarantees. If something should always happen in an order, it should probably be a tool - make tools accept lists of items to operate on to avoid agent having to call it in a loop - unit test your tools (deterministic) before testing the whole agent (non-deterministic) Curated Q&A relevant to Local AI builders in 🧵

✅ BENCHMARKED: IOTA Decide 4B v0.4, an open-weight decision model from @MacrocosmosAI, running on my 64GB M5 Pro Mac mini You hand it a state (a ticket, a policy, an agent trace) plus questions and one forward pass returns a probability for every answer. With their bundled loader on the Mac's GPU in full bf16, it loaded in 12.8 seconds and peaked at 10.7 GB. Short ticket, about 130 tokens: - yes/no: 88 ms - 3-way choice: 105 ms - 3-level score: 88 ms That's about 11 decisions a second on a desktop Mac. Their own example, a refund asked for 45 days after purchase on a 30-day policy, answered all 3 questions in 277 ms with the same answers on every repeat: - eligible: 2.2% yes - route: refunds team, 86% - urgency: Low, 63% Longer states take about 0.6 s per decision at 1k tokens and 2.6 to 3.0 s at 4k. These numbers are the floor. On Mac its linear-attention layers still run on PyTorch's generic kernels, so an optimized Metal path should make it faster. Their 33 ms figure is on a dedicated GPU. If you're building agents, about 90 ms a decision is fast enough to check a policy before an agent acts or route tickets on your own machine, with no per-call bill. I didn't benchmark accuracy, but it gave the same answers on their refund example every run.

✅ BENCHMARKED: TensorFold made Qwen3.8 27B more than 3x faster on my 64GB M5 Pro Mac mini, on two versions in one day. In the morning I tested 1.2.1. On its own, the model wrote about 17 tokens a second. I then switched on the DFlash2 drafter, a small helper that guesses the next few tokens so the big model only has to check them and it jumped to 56. And later in the afternoon @ashxhart shipped 1.2.2. His release notes focused on other hardware, so I reran everything on my Mac mini. On mine it runs within a few percent of 1.2.1, and the drafter still makes it 3x faster. With randomness off, the answer came out word for word the same as without the drafter. - Chat, 3.26x faster on 1.2.1 and 3.35x on 1.2.2 - A 31K token prompt, 2.94x and 2.77x - Code, 5.4x and 5.5x, up to 91.90 tokens a second - 28.5 GiB peak memory on both - New in 1.2.2, the first request doesn’t wait for the drafter to load A massive thanks to @ashxhart and all the contributors behind 1.2.2. The TensorFold challenge from @YukonResearch is open until Nov 6th if you want to try pushing it further on your own Mac.

Be like @ashxhart The person behind TensorFold. > Grew up in the UK > Spent 7 years in tech support > Joined a friend’s food delivery tech company > Spent 2 years there as a PM > Went back to his first company as Head of Technology, still there today > 15+ years across software, DevOps, AWS, networking and cybersecurity > Became the 4th-line support guy, the one you call when nothing else works > Started exploring AI on the side > Spent 6 months building a local inference engine in private > Archived it three times > Launched TensorFold anyway, expecting maybe 5-10 users > It blew up in the first few days of launching > 1.5k+ stars on GitHub > Top Local AI builders started contributing > Rewrote it in Zig with the community > Built MCDMA so Macs and NVIDIA boxes can share memory > Linked two Mac Studios over Thunderbolt 5 > Shipped a model that keeps learning while you use it > Got a DGX Spark donated to keep building TensorFold > Went from 300 to 9k+ followers in 4 weeks > Called on the UK to build its own open model And the story’s far from over. We need people like Ash to push forward Local AI and Personal AI. Truly an inspiration! Follow him, you will thank me later. Honored to partner with him to make TensorFold part of the @YukonResearch challenge.

Should you buy a 512GB Mac Studio for Local AI? I ran the numbers before ordering one. 1) On token price alone, renting wins. At DeepSeek V4.1 Flash API rates, the Mac needs years of 24/7 output to pay for itself. 2) Price per token is the wrong way to judge it, though. In expensive housing markets, renting is cheaper month to month than owning, and people still buy for control and because a home holds value. The same logic applies to this machine. 3) It holds value. The 512GB M3 Ultra now resells for about twice what it cost new, because memory got scarce. If mine resells for even half, the real cost halves. 4) Nobody can take it away. API models get swapped, rate limited and refused mid-task. In June, access to two frontier models was cut off for almost three weeks. Weights on my desk stay put. 5) Zero-cost tokens change what you build. When every call costs nothing, agents can run all day and you can experiment freely. MacStories ran local agents 24/7 for 99 days at zero token cost. 6) It runs the big ones. DeepSeek V4.1 Flash (552B) and GLM-5.3 (744B) don’t fit in 256GB. In 512GB, each loads into one memory pool. So if you’d use it a few hours a week, rent. If you’d run it all day the moment it was free, buy. Go Local AI!

LinkedIn is cringe. LET 𝕏 MAKE IT BETTER. I made a @bot that takes your X posts to LinkedIn. > You approve, it posts > Tags port naturally > No thought leadership needed Oh and your shitposts stay on @X where they belong.

🚨 FOR THE RECORD THE OPEN FRONTIER IS NEARLY 4X AHEAD OF ANTHROPIC’S NEW QUANTUM CIRCUIT @AnthropicAI says Claude Mythos 5 designed quantum circuits that beat the best published ones. For point addition, the step a quantum computer repeats to break elliptic-curve cryptography, its best circuit needs 4.0 million gates on 935 qubits. On by Eigen Labs, the open leaderboard for that same step, the best circuit today needs 761,121 gates on 1,234 qubits. Multiply qubits by gates, the score Anthropic ranks by, and that’s nearly 4x cheaper. At 843 qubits, Anthropic’s circuit needs 21 million gates and the leaderboard’s needs about 12 million. Both of Anthropic’s circuits sit above the open frontier. The paper put a circuit 2.9x ahead of theirs on arXiv on Sep 9. Anthropic’s paper came out Oct 15 and doesn’t cite it. That paper was led by @jieyilong, CTO of @Theta_Network, with coauthors from @eigenlabs, @ethereumfndn, @StarkWareLtd, @Starknet, @trailofbits, @brevis_zk, @SeiNetwork, @pauli_group, @OctavFi, @sciencevr, @nasqret at Adam Mickiewicz University and researchers at Warsaw University of Technology and Stanford’s Free Systems Lab. 80 solvers have moved this frontier 1,061 times, with every circuit checked by the same public evaluator. The current best was set today by a solver running Claude Opus. The two sides count gates slightly differently, so treat this as a close comparison rather than a formal proof. Claude’s divsteps idea is new and the leaderboard is open if Anthropic wants to submit it.

Founders building an open PostAGI world. Who owns intelligence decides what comes after AGI? These are the techno-optimists making sure everyone does. - @drfeifei - @miramurati - @allisondman - @jensenhuang - @sama - @alexandr_wang - @minnasong - @elonmusk - @finkd - @demishassabis - @sreeramkannan - @sundarpichai - @emostaque - @sarahookr - @clementdelangue - @karpathy - @Teknium - @ilyasut - @ccatalini - @mntruell - @scottwu46 - @techtekedra - @hamandcheese - @scottsumnertmi - @bturtel - @dleitersdorf - @annavdorogush Full list on @cosign: Who did i miss?

This is the first time in Darkbloom's history that we are the first provider to support a model on @OpenRouter ❤️ I couldn't have chosen a better partner for this milestone. @PrismML focuses on concentrating intelligence, so more models can run on distributed Mac machines!

Bonsai 2 at <6GB size and ~Opus 4.6 performance. Live on Darkbloom. This breakthrough is from @PrismML founded by Caltech Prof. @BabakHassibi a leading information theorist. Optimizing intelligence per bit is an extremely valuable objective for Local AI. It is only possible for us to own our own intelligence if it will fit into the devices we own. Thats what the PrismML team has achieved here. It has only been 9 months from a frontier release (Opus 4.6) to getting it to fit in your phone! As we worry about frontier superintelligence concentrating power, we are delighted to see the counterweight emerging from open models that can fit into our phones. The darkbloom team @0xkydo and @gajesh were so excited by this breakthrough that they worked overnight to bring this to life on Darkbloom. Excited to offer the first hosted service for this model to everyone on Darkbloom, the compute grid powered by real people! Go try it on our chat or provision your Macbook to service this model for the world!

Bonsai 2 is a very capable model for daily knowledge work. Here is the essay it wrote on "the societal, political, and economic importance of AI that can run locally on any consumer computer or even a phone." You can try it out on and we are looking to support this on OpenRouter soon as well. --------- The Quiet Revolution: Why Local AI Matters - By Bonsai 2 27B For much of the past decade, artificial intelligence has been imagined as something distant and centralized: a service hosted in vast data centers, accessed through cloud platforms, and controlled by a small number of large technology companies. This model has produced remarkable capabilities, but it has also concentrated power, created new dependencies, and raised serious questions about privacy, security, and access. A different possibility is emerging: AI that can run locally on ordinary consumer devices—laptops, desktops, tablets, and even smartphones. This shift is not merely a technical improvement. It is a potentially transformative development with profound societal, political, and economic implications. Local AI matters because it changes who controls intelligence. When a model runs on a person’s own device, the user is no longer merely a customer sending data to a remote server. The user becomes, in a meaningful sense, the operator of an intelligent system. That distinction may seem small, but it reshapes the relationship between individuals, institutions, and technology. It affects how people communicate, work, learn, and participate in public life. It also affects how governments, corporations, and civil society understand power in the digital age. Societal Importance: Privacy, Autonomy, and Access The most immediate societal benefit of local AI is privacy. Many of today’s AI services require users to send prompts, documents, images, or other data to external servers. Even when companies promise strong privacy protections, the architecture itself creates risk. Data must travel across networks, be stored, processed, and potentially logged. For sensitive matters—medical questions, legal advice, personal relationships, political opinions, or confidential business information—this can be unacceptable. Local AI reduces that exposure. If a model runs on the user’s own device, the data can remain under the user’s control. A person can ask a local assistant to summarize a medical report, draft a letter, analyze a contract, or help with a personal decision without sending the underlying material to a third party. This is especially important for vulnerable populations: patients, journalists, activists, lawyers, journalists, researchers, and anyone whose privacy is essential to their safety or dignity. Local AI also supports autonomy. In a world where many digital services are designed to shape behavior, local intelligence can be a tool for personal agency rather than a channel for external influence. A user can choose which model to run, how to configure it, and what it is allowed to do. This is not a perfect solution—local models can still be biased, misleading, or poorly designed—but it creates a more transparent and controllable environment. The user can inspect the system, update it, or replace it without depending on a single provider’s terms of service. There is also a powerful accessibility dimension. Much of today’s AI is available through subscription services, cloud APIs, or high-end hardware. Local AI can lower barriers to entry. A person with a modest smartphone or an older laptop may still be able to run useful models for writing, translation, coding, research, or education. This could be especially significant in regions where internet access is unreliable, expensive, or monitored. In such contexts, local AI may function not only as a productivity tool but as a form of digital resilience. Political Importance: Sovereignty, Censorship, and Power The political importance of local AI is less obvious but equally significant. Intelligence is becoming a strategic resource. Governments, corporations, and other powerful actors are investing heavily in AI because it can influence decision-making, communication, surveillance, and economic competition. If most AI capability remains concentrated in a few cloud providers, those providers become critical nodes in the political and economic order. Their policies, outages, legal compliance, and commercial interests can shape what information is available, what speech is permitted, and what kinds of intelligence are accessible. Local AI can reduce that concentration. When intelligence can run on ordinary devices, it becomes harder for any single entity to monopolize access to advanced reasoning, translation, analysis, or creative assistance. This does not eliminate power imbalances—large companies still design many models, and governments still regulate them—but it introduces a more distributed structure. It creates space for independent actors, small businesses, public institutions, and individuals to develop their own capabilities without depending on a single platform. This has implications for censorship and control. In authoritarian contexts, centralized AI services can be easily blocked, monitored, or co-opted. Local models, by contrast, are harder to shut down remotely. A government may be able to restrict access to cloud services, but it is much more difficult to prevent people from running software on their own devices. This does not mean local AI is inherently liberating; it can also be used for surveillance or manipulation. But it does mean that the balance of power shifts. The capacity for intelligent assistance becomes less dependent on the goodwill of a central authority. Local AI also matters for democratic participation. In an era of misinformation, deepfakes, and algorithmic manipulation, citizens need tools that help them evaluate information, summarize complex policy, translate documents, and understand technical arguments. If those tools are available locally, they can support a more informed public. A voter can use a local model to compare policy proposals, a journalist can analyze leaked documents, and a community organization can translate public notices into multiple languages. The political value lies not in the model’s perfection, but in its accessibility and independence from a single commercial or state-controlled infrastructure. There is also a question of data sovereignty. In many countries, data is increasingly treated as a national asset. Local AI can help institutions keep sensitive information within their own jurisdiction. A hospital, university, or government agency can run models on its own hardware without sending patient records, research data, or classified documents to foreign servers. This is especially important in fields where confidentiality is legally or ethically required. Economic Importance: New Markets, New Work, and New Productivity Economically, local AI could be one of the most important technological shifts of the coming decades. Today, much of the AI economy is built around cloud compute, data centers, and enterprise subscriptions. That model is powerful, but it is also expensive and concentrated. Local AI changes the economics by moving intelligence closer to the user and closer to the data. For individuals, local AI can increase productivity. A student can use a local model to study, a freelancer can draft proposals, a small business owner can analyze customer feedback, and a developer can generate code. These uses may not replace human judgment, but they can reduce the cost of cognitive labor. Over time, this could lower the barrier to entry for many professional and creative activities. For businesses, local AI offers a different value proposition than cloud AI. Some tasks are better suited to local processing because they involve sensitive data, require low latency, or must work offline. A law firm can analyze contracts without sending them to a cloud service. A manufacturer can run diagnostic models on factory equipment without exposing proprietary data. A retailer can use local models to personalize customer interactions while keeping transaction data on-site. In these cases, local AI is not just a privacy feature; it is a business requirement. Local AI may also create new markets. Instead of selling only cloud subscriptions, companies can sell models, fine-tuned versions, hardware optimization, and local deployment services. Developers can build applications that run entirely on-device, creating a new category of software that is faster, more private, and less dependent on connectivity. This could be especially important for mobile applications, where users expect instant responses and do not want to upload personal data to the cloud. There are also labor-market implications. Local AI will not eliminate the need for human workers, but it will change the nature of many jobs. Tasks that once required specialized expertise may become more accessible to ordinary users. A small clinic may be able to use AI-assisted triage tools, a local government office may be able to automate routine administrative work, and a small manufacturer may be able to improve quality control without hiring a large data science team. This could increase economic productivity, but it could also create displacement pressures. The key question will be whether societies can adapt education, training, and social policy to match the new economic reality. One of the most important economic effects may be the reduction of digital inequality. Historically, advanced computing has been concentrated in wealthy nations and wealthy institutions. Local AI could help democratize access to intelligence by allowing useful models to run on devices that people already own. This does not mean that all devices will be equally capable, and there will still be a gap between high-end and low-end hardware. But the direction of travel is significant: intelligence is becoming something that can be embedded in everyday objects rather than reserved for specialized infrastructure. The Limits and Risks It would be irresponsible to present local AI as a simple solution. There are important limitations. Consumer devices have constraints on memory, power, and thermal capacity. Not every model can run efficiently on a phone, and not every task requires a large model. Smaller local models may be less capable than the largest cloud-based systems, and they may require careful design to be useful. There are also risks of misuse: local AI can be used to generate convincing misinformation, automate manipulation, or support surveillance if deployed without safeguards. There is also a governance challenge. If intelligence becomes more decentralized, how do we ensure that it is safe, reliable, and aligned with human values? How do we prevent harmful models from spreading? How do we protect users from poorly designed or deceptive systems? These questions will require new forms of regulation, technical standards, and public oversight. Local AI does not remove the need for accountability; it changes where accountability must be located. There are also environmental and energy considerations. Running AI locally can reduce the need for massive data centers, but it also increases the energy demand of consumer devices. The net environmental impact will depend on hardware efficiency, model size, usage patterns, and the energy mix of the regions where devices are used. A responsible local AI ecosystem will need to balance capability with efficiency. A New Relationship Between People and Intelligence At its core, the importance of local AI lies in a shift in relationship. For centuries, intelligence has been associated with institutions: courts, universities, governments, corporations, and specialized experts. The digital age has already begun to distribute information, but it has often done so through centralized platforms that mediate access. Local AI goes further. It places a form of intelligence directly in the hands of individuals and small organizations. This does not mean that every person will become an expert in machine learning. Most users will not need to understand the technical details. They will simply have access to a tool that can help them think, write, analyze, and create. But the fact that the tool runs locally changes the meaning of that access. It is not just a service. It is a capability. That distinction is why local AI matters so much. It has the potential to make privacy more practical, to strengthen digital sovereignty, to reduce dependence on a small number of cloud providers, and to expand economic opportunity. It can support more independent journalism, more resilient public institutions, and more empowered individuals. It can also create new forms of innovation, especially in areas where data sensitivity or connectivity limits make cloud-based AI impractical. The coming decade will likely be defined not only by how intelligent AI systems become, but by where they run. If intelligence remains concentrated in a few remote data centers, it will continue to concentrate power. If it becomes available on ordinary devices, it may help distribute power in ways that are more inclusive, more private, and more resilient. The rise of local AI is therefore not just a technical trend. It is a social and political development with the potential to reshape the relationship between people and the systems that shape their lives. -----------

Last night @PrismML shipped Bonsai 2 and @darkbloomai is the first model provider to support it. We're offering the first 250 users 100m free tokens test it out. At 9x smaller than Qwen 3.8 with 98.2% of the benchmark performance. It outperforms Opus 4.6 and 5.6 Luna and its the first model that can run on the lightest weight personal MacBooks in our Darkbloom fleet. The race for bigger centralized models misses half the point. Models like Bonsai are how AI becomes ubiquitous and Darkbloom ensures the spoils of the inference economy are accessible to everyone.

BREAKING: Darkbloom is the 1st provider serving Ternary Bonsai 2 27B, a full 27B reasoning model that fits in 8.5GB. @PrismML compressed Qwen3.8 27B to 2 bits per parameter and it keeps 98.2% of the FP16 benchmark performance. An open-weight model, Apache 2.0, 262K context, image input, 75% cheaper to serve. First run on the network, on a single M5 Max with no caching. 35 tok/s at 1K context, 31 at 10K, 19 at 50K. One team compressing another team's open weights, served on 1,000+ Macs that belong to the community rather than to us. ❤️ ❤️ ❤️ Open innovation on open intelligence infra. First 250 users get 100 million free tokens.

Darkbloom is the first model provider to support Ternary Bonsai 2 27B -- concentrated intelligence that fits on your phone. Try it now: First 250 users get 100 million free tokens; PrismML's new flagship model: - a ternary compression of Qwen3.8 27B at 2bits per parameter. - 8.5 GB total, 5x smaller than original - keeps 98.2% of the Qwen's FP16 benchmark performance. - 75% cheaper than Qwen 27B. Qwen3.8 27B already operates comparably with Opus 4.6 and 5.6 Luna on certain tasks. This one does it in the memory of a phone. 262K context, image input, Apache 2.0. From our first run on the network, on a single M5 Max with no caching: - 35 tok/s decode at 1K context, - 31 tok/s at 10K, - 19 tok/s at 50K. But we expect more performance gain coming in a few weeks! That's a full 27B reasoning model running comfortably on any Mac. 1,000+ Macs are serving on Darkbloom right now. Go try it out!! Thank you to @BabakHassibi @SahinLale @HessianFree @rsadri_ml @tushar_bans @evaninwords and the whole PrismML team. This is exactly the kind of model Darkbloom was built for. You can read the essay by Bonsai on Why Local AI Matters:

🖤 @DarkbloomAI crossed 10k followers Still a small number, but a start. Thank you to everyone who followed and stayed patient while a small team figures this out in public. Our goal is to build the open inference network the world runs on. It's incredible how many people already believe in that. And the team keeps pushing EVERY DAY. @gajesh, @0xkydo, @TheDavidTai, @Spangler3000 and the rest of the crew. See you at 50k!

> be @darkbloomai >gather everyday people's Macs >connect them to a grid >create a private "local AI as a service" >optimize models like @googlegemma @Alibaba_Qwen and @nvidia nemotron with your sister team @yukonresearch >run them 2-3x faster and 50% cheaper by stripping out datacenter margins >partner with @OpenRouter for distribution >pass earnings along to passionate, growing local AI community that offer up their spare compute. rinse. lather. repeat