my gpt 6 astra reflections: first, it achieves a record shattering 41.4% in our automation bench, the prior record holder was fable 5.1 at 31.4%. second, its personality doesn't feel fried. its writing is usually very coherent and non-performative. much more pleasant. 1/?

2nd AutomationBench update of the day🔥 We ran Gemini 3.8 Flash across 600+ real business tasks, and found it's Google's best Flash model yet for Ops and Marketing workflows Available soon in Zapier

okay, so astra just set a new high on automationbench: 41.4% of business workflows completed correctly. so how is it better? and what is automationbench? well, we measure that by putting the model in simulated apps, giving it a job, and checking the resulting records and messages with code. the hard part is building a test that catches "updated the wrong customer" or "ignored the hold" while still letting the agent figure out its own way to do the work. more or less, you build an small environment for an agent. start with a known set of crm records, emails, spreadsheet rows, calendar events, etc. give the agent tools that behave like those apps' apis. underneath, it's structured state: sending an email adds a message to the sent mailbox, updating a contact changes a field on that record. nothing goes to an actual customer. each run starts with its own copy of the environment, so one model's changes don't affect another's. then you give it a task. say the spreadsheet has verified job titles and you want the crm brought up to date. update the contacts, mark the spreadsheet rows reconciled, and send the summary to the audit inbox specified in the company's procedure. the agent has to find the procedure and the data itself. it gets a keyword search over api documentation and a tool for making requests with a method, url, and body. there's a budget of 50 turns, and it can make several tool calls in a turn. you can make this quite hard without changing that basic request. put two people with the same name in different accounts. leave an outdated title in one source. mark a row verified, then put a later message in the inbox saying that title is disputed and shouldn't be changed yet. put another record under a security freeze. now the agent has to work out which records the request actually applies to. "update everyone in the spreadsheet" won't do it. the critical information need to be discoverable, though. if the correct answer depends on a fact we never put in the environment, that's our bug. same if the api can't retrieve the relevant message, or the instructions contradict each other without a way to tell which one governs. making a task difficult is easy. making it difficult for a reason that tells you something about the model takes significantly more work. scoring is ordinary, deterministic code that inspects the final state. for the contact task, the checks are along these lines: the eligible contacts have the expected titles, their spreadsheet rows are marked reconciled, and the audit inbox received the required summary. but you also check that the held contacts kept their original titles and statuses. otherwise a model could update everyone and pass just because the two intended updates happened somewhere in there. that's an actual difference we saw between astra and sol. astra updated two eligible contacts; sol updated four, including two it should have left alone. sol had even retrieved the security-freeze message for one of them before making the change. a grader that only checked whether the requested updates happened would miss that failure. crucially -- you have to check the unwanted changes too. the same applies to messages. checking that the right person received an email isn't enough if the agent also sent it to twenty wrong people. these negative checks are part of the task, not an optional safety score off to the side. every scored check has to pass for the workflow to count as complete. we keep partial credit to help inspect failures, but 41.4% means complete tasks, not the average fraction of steps the model got right. we don't prescribe the whole sequence of calls. the agent might find the data a different way, batch some updates, or correct an earlier mistake. if it leaves the required state behind, that should pass. this also means the benchmark only sees what its checks cover. a deterministic grader can be consistently wrong! we audit the tasks and use separate hint-assisted runs to help check that they're solvable; the hints aren't part of the scored runs. the simulator and the grader both need scrutiny when a model fails. the public task set is available for experimentation. the headline scores use a separate, harder held-out set, so don't mix those numbers. a few model developers (meta, etc) are reporting public set scores which aren't verified by us. and none of these scores tell you that the model will succeed on that percentage of work in your company. they tell you how often it completed this particular set of workflows under these conditions. we're actively working on automationbench 2, something that will be significantly trickier...

GPT 6 Astra is here. We ran the numbers on AutomationBench: It's the highest score we've ever recorded. Clean sweep across every domain. Scores 41.4% at Max effort. For context, no model had cleared 40% before today (GPT-5.6-Sol scored 28.8%) 𝗕𝗲𝘀𝘁 𝗳𝗶𝘁 𝗳𝗼𝗿: reconciliation, deal review prep, vendor scorecards, anything where touching the wrong record is expensive. 𝗪𝗲𝗮𝗸𝗲𝗿 𝗳𝗼𝗿: outbound comms where the guidance is scattered. Operations and support are its strongest domains. HR is its weakest, same as every model we test (still the new high score, though) Its edge is arithmetic across messy sources. Finding the policy doc, the logged correction, the exception rule, etc. Example 1: rebalance a quarterly media budget from last quarter's actuals, with finance adjustments and channel eligibility rules buried in email. Both models produced a budget and landed on the same total. Astra found the adjustments, so every per-channel number was right. Sol's looked finished and had the splits wrong. Example 2: answer and log 15 integration inquiries using a reply standard stored in a doc. Astra searched, could not find the standard, and stopped. Zero replies sent. Sol did not find it either, took its best shot at all 15, and earned partial credit. Those examples highlight how these two models make tradeoffs... Astra will not guess. When the instructions exist and it can find them, it finishes the whole job. When it cannot, it pauses the work instead of improvising. Crazy week for LLM releases after a few quiet ones. Astra isn't available to the public yet, but should be soon. We run every new model through @Zapier's AutomationBench, 657 of the hardest workflows we have, across finance, HR, marketing, operations, sales, and support. See every model and every score here:

All hail @zapier 🙌 We have been using Zapier for years & it has saved me SO much time! If you are a #lawyer# or #smallbiz# owner looking to start automation, check out this video!

4 in 5 leaders say their teams are working around AI rules. Shadow IT applies to shadow AI too. @Zapier surveyed 548 directors, VPs, and C-suite leaders. In companies with 500 to 4,999 people, 81% report shadow AI. Over 5,000 people? That drops to only 64%. Better enforcement, or something else? The bigger orgs have more layers between the exec filling out a survey and the person actually building the workflow. So maybe that's lower visibility showing up as a better score? Playing devil's advocate, maybe the big orgs did invest earlier and it worked. I don't know for sure, and I'd want to see how they measure it before I’d believe that. Shadow AI shows up for the same reason every time: the approved path is slower than the workaround. 3 weeks and a ticket to get AI access, and people will find their own way. Data here:

WSJ just wrote about our Slack transparency leaderboard. It measures what % of our messages are public vs hidden in DMs. I'm at 79%. Our CMO is at 98% (coming for you, Dan) An agent can only use what it can read. On a customer call last week, I had an agent build a deck from 60 days of Slack messages about that account. It finished during the call because the messages were public and parsable. Public-by-default has limits. HR issues, legal issues, and sensitive conversations stay private. Brainstorming, work decisions, and project updates belong in channels when they can. That gives people and agents a foundation to build on later. @Zapier's leaderboard is just how we make that habit visible.