tl;dr -- jev is a very useful paradigm which has been hiding in plain sight. also, pour one out for log probs disappearing from so many new closed inference apis... 😢 prompting llms with max_tokens: 1 w/ log probs for fast, parallelized judgments may be an old pre-structured outputs trick gaining new life, but i'll admit i'm rather surprised at how flexible and useful this pattern appears. there's a lot more depth here than i expected -- the demos are quite impressive! back in the golden days of early llms circa 2023, i recall experimenting with fine tuning oai's ada to output either a 1 or a 0 token, and inspecting the log probs to roughly gauge confidence. at zapier we were exploring this around field mapping -- matching output fields from a prior step to input fields in another. seems like training a base model specifically to return calibrated probabilities takes that quite a bit further. it's so cheap that you can ask tons of speculative questions and just throw away the answers. add a hierarchy of follow-up questions and you can really tune the plinko machine. quantity truly has a quality of its own. with short questions and shared context, the rough math works out to ~60 field judgments per penny with fine-tuned ada versus potentially ~11k with jev batching -- amazing! what other interesting paradigms are lurking in plain sight?

okay, so astra just set a new high on automationbench: 41.4% of business workflows completed correctly. so how is it better? and what is automationbench? well, we measure that by putting the model in simulated apps, giving it a job, and checking the resulting records and messages with code. the hard part is building a test that catches "updated the wrong customer" or "ignored the hold" while still letting the agent figure out its own way to do the work. more or less, you build an small environment for an agent. start with a known set of crm records, emails, spreadsheet rows, calendar events, etc. give the agent tools that behave like those apps' apis. underneath, it's structured state: sending an email adds a message to the sent mailbox, updating a contact changes a field on that record. nothing goes to an actual customer. each run starts with its own copy of the environment, so one model's changes don't affect another's. then you give it a task. say the spreadsheet has verified job titles and you want the crm brought up to date. update the contacts, mark the spreadsheet rows reconciled, and send the summary to the audit inbox specified in the company's procedure. the agent has to find the procedure and the data itself. it gets a keyword search over api documentation and a tool for making requests with a method, url, and body. there's a budget of 50 turns, and it can make several tool calls in a turn. you can make this quite hard without changing that basic request. put two people with the same name in different accounts. leave an outdated title in one source. mark a row verified, then put a later message in the inbox saying that title is disputed and shouldn't be changed yet. put another record under a security freeze. now the agent has to work out which records the request actually applies to. "update everyone in the spreadsheet" won't do it. the critical information need to be discoverable, though. if the correct answer depends on a fact we never put in the environment, that's our bug. same if the api can't retrieve the relevant message, or the instructions contradict each other without a way to tell which one governs. making a task difficult is easy. making it difficult for a reason that tells you something about the model takes significantly more work. scoring is ordinary, deterministic code that inspects the final state. for the contact task, the checks are along these lines: the eligible contacts have the expected titles, their spreadsheet rows are marked reconciled, and the audit inbox received the required summary. but you also check that the held contacts kept their original titles and statuses. otherwise a model could update everyone and pass just because the two intended updates happened somewhere in there. that's an actual difference we saw between astra and sol. astra updated two eligible contacts; sol updated four, including two it should have left alone. sol had even retrieved the security-freeze message for one of them before making the change. a grader that only checked whether the requested updates happened would miss that failure. crucially -- you have to check the unwanted changes too. the same applies to messages. checking that the right person received an email isn't enough if the agent also sent it to twenty wrong people. these negative checks are part of the task, not an optional safety score off to the side. every scored check has to pass for the workflow to count as complete. we keep partial credit to help inspect failures, but 41.4% means complete tasks, not the average fraction of steps the model got right. we don't prescribe the whole sequence of calls. the agent might find the data a different way, batch some updates, or correct an earlier mistake. if it leaves the required state behind, that should pass. this also means the benchmark only sees what its checks cover. a deterministic grader can be consistently wrong! we audit the tasks and use separate hint-assisted runs to help check that they're solvable; the hints aren't part of the scored runs. the simulator and the grader both need scrutiny when a model fails. the public task set is available for experimentation. the headline scores use a separate, harder held-out set, so don't mix those numbers. a few model developers (meta, etc) are reporting public set scores which aren't verified by us. and none of these scores tell you that the model will succeed on that percentage of work in your company. they tell you how often it completed this particular set of workflows under these conditions. we're actively working on automationbench 2, something that will be significantly trickier...

my gpt 6 astra reflections: first, it achieves a record shattering 41.4% in our automation bench, the prior record holder was fable 5.1 at 31.4%. second, its personality doesn't feel fried. its writing is usually very coherent and non-performative. much more pleasant. 1/?