Announcing the Artificial Analysis Endpoint Accuracy Index, measuring how much of an open weights model's accuracy each serverless API endpoint preserves. We are initiating coverage with GLM-5.2, gpt-oss-120b and DeepSeek V4 Pro, with Kimi K3 coming soon Providers trade off accuracy to optimize for speed and cost. They quantize weights, write custom kernels and tune their inference stacks, and sometimes they simply ship bugs. We are bringing the rigor of our Artificial Analysis Intelligence Index to measuring endpoints, so developers can pick providers on accuracy, not just price and speed We benchmark each serverless endpoint against our own self-hosted reference deployment of the official weights, where 100% represents matching the reference. An endpoint is at reference parity when its result falls within the 95% confidence interval of the reference. Coverage is live for GLM-5.2, gpt-oss-120b and DeepSeek V4 Pro, with Kimi K3 accuracy coverage launching soon Key elements of the Endpoint Accuracy Index: ➤ Three areas, equally weighted: tool calling (BFCL-500, 500 questions, 3 repeats), scientific reasoning (HLE-250, 250 questions, 10 repeats) and long context recall (AA-LCR-25, 25 questions, 10 repeats). Each subset separates endpoints on the serving choices that drive accuracy differences, with repeats sized for tight confidence intervals ➤ Reference deployment: we self-host the official weights at the lab's recommended precision, following the lab's serving recipe, and publish the complete commands for each reference ➤ Inference parameters: we run the model's highest supported reasoning mode and each endpoint's highest supported output length and context window ➤ Confidence intervals: the parity test accounts for uncertainty in both the endpoint's runs and the reference's runs ➤ Rotating coverage: models enter once sufficient number of providers serve them and exit when a newer version in the same family supersedes them. We benchmark new endpoints as providers launch them and refresh all listed endpoints periodically ➤ Point in time: each result carries the date it was measured, with multi-day benchmarks dated to their final day Key results for GLM-5.2 ➤ Output token limits restrict accuracy. Restrictive limits cut responses off before the model finishes reasoning, and the most restrictive endpoints score half the reference or less on HLE-250 Key results for gpt-oss-120b ➤ Tool call handling separates endpoints. Providers parse and format tool calls differently, and some endpoints score 22% on BFCL-500 against 37% for the reference ➤ Serving configuration changes what the model does at the same requested settings. Some endpoints produce far fewer reasoning tokens at the same configured level, and restricted context windows truncate long context tasks Key results for DeepSeek V4 Pro ➤ DeepSeek V4 Pro endpoints are more in line with the reference. Majority of the endpoints are at reference parity, and DeepSeek's own first-party endpoint scores slightly above the reference

In a review of our cybersecurity evaluations, we found three incidents in which a Claude model reached the internet from within or while interacting with a third-party evaluation environment, and then gained unauthorized access to the real systems of three different organizations. Our post describes what happened, how it happened, and what we’re changing. We encourage other AI developers to perform similar reviews. We conducted this review together with @Irregular, one of our evaluation partners, and thank them for the joint investigation and their collaboration on this post. This type of collaboration is increasingly critical to safe, rigorous evaluation of models, and we look forward to continuing to work together on security.

Back from vacation and LIVE at 12pm PT / 3pm ET Is AI’s easy-money era ending? We’ll unpack a wild week for the AI trade—big tech earnings, Leopold Aschenbrenner’s massive deal, OpenAI’s price cuts, rogue agents, open models and new restrictions on Chinese robot imports. Cisco’s @jpatel41, Fireworks AI's @lqiao and Standard Bots' @evanbeard.

Today we're announcing dfs-large1, our newest cybersecurity model that achieves best-in-class performance on vulnerability detection tasks. Besides frontier AI labs, only a handful of companies have built specialized models that reach the state of the art in their domain. We're proud to be the first to do it for cybersecurity. dfs-large1 is built on GLM-5.2 and post-trained with reinforcement learning inside depthfirst's security infrastructure. We evaluated it on depthfirst-bench, our benchmark of long-horizon vulnerability discovery across complex repositories, where it achieves best-in-class performance. Training improvements have not plateaued yet and we expect additional performance gains as we continue training. A huge thank you to @FireworksAI_HQ for being an outstanding training partner. Their infrastructure enabled us run large-scale reinforcement learning efficiently and iterate much faster. dfs-large1 is now in preview within the @depthfirstlabs platform

Everyone gets angry with me for saying triple, triple, double, double is dead. Fine, I do not really care. Venture is about investing in unbelievable outliers. Anomalies that own markets with generational founders. @FireworksAI_HQ is an example of this. They scaled to $1BN in ARR in just 3.5 years. They also have just 200 employees making it an insane $5M revenue per head. @lqiao just raised a whopping $1.5BN at a $17BN valuation and I sat down with her. Added my notes and the episode below: 1. The Challenges That Come From Such a Fast Development Cycle for Chips Hardware innovation is moving so quickly that rapid SKU cycles now outpace traditional depreciation timelines, changing the financial calculus of building versus renting infrastructure. Founders should prioritize growth and market agility over immediate gross margins, avoiding premature optimization until customer workloads stabilize. 2. Why National Sovereignty Is Real in AI and Every Company Should Have Its Own Model Frontier models function like a society’s core electricity grid. Relying entirely on a third-party API creates the existential risk of sudden disconnection, making model ownership and infrastructure independence critical for both sovereign nations and enterprise businesses. 3. Why the Future Is Millions of Specialized Models Frontier providers bake their own design tastes and values into models, which inevitably misaligns with enterprise business logic. The future belongs to millions of specialized, “one-size-fits-one” models tailored to proprietary data, consistently outperforming generalized AGI on accuracy, speed, and unit economics. 4. The Transition From the Year of Coding to the Year of Co-Work AI adoption has rapidly evolved from engineering-centric coding tools to a diversified ecosystem of B2B co-work agents. Founders and VCs must look past crowded developer environments to capture massive value in specialized workflow automation across legal, finance, healthcare, and other enterprise functions. 5. How a 10x Cost Reduction Will Drive a 100x Explosion in Usage Temporary supply chain backlogs will eventually ease, compressing infrastructure and model-tuning costs by 10x over the next three years. This deflation in token unit economics will turn intelligence into a near-frictionless commodity, driving a massive surge in enterprise production usage. 6. Why Avoiding the Application Layer Is Essential for Platform Focus Platform defensibility requires strict focus on multi-chip agility without creating vertical hardware or software dependencies. By refusing to move up into the application layer, infrastructure platforms avoid competing with their own ecosystem and maximize their value in specialized model orchestration. 7. Biggest Lesson From Working With Jensen Huang on Leadership Leadership in hyper-velocity markets is defined by rapid judgment, not executive privilege. Because critical information degrades as it moves through layers of corporate hierarchy, leaders must stay close to ground-level technical details to maintain execution speed and avoid flawed strategic calls.