A ranking of takes on embedded third party evaluation this past week, from worst to best: [contentless sneer against outgroup] METR bad because [huge Sankey diagram]. fact checking? timelines of relative investments that make causal sense? determining which numbers are large or small fractions of other numbers? sounds like some EA bullshit to me. just look at this diagram with all those curved lines, you can SEE the nest of snakes. METR bad because [an actual specific chain of actors who have some pairwise relationship to each other that ends with someone it would be bad for them to have strong COIs with, e.g. "METR once received funding from an org who once received funding from a person who once gave funding to a now-frontier AI company"]. It is beneath my dignity to explain how this actually affects the decisionmaking of METR; you should vaguely perform a mood affiliation and keep scrolling. Don't think too hard. guys guys guys you HAVE to embed my company/organization into the labs. I have been an unwavering supporter of external embedded auditors since 7 minutes ago when I saw this essay taking off. Look at this thing we published once that sort of looks like AI safety if you squint! Fear not, I can assure you that I have never done anything altruistic in my life and if I had I would have been ineffective at it. I observe that [politicized actor] has had [bad take]. Let me use this correct observation to score points for my side and politicize the situation further. === zero point: takes beyond this line are better than logging off === Dear [person with insane take], here is an earnest explanation of why you are wrong. METR is bad/problematic because [an actually reasonable concern, like greater cultural overlap with Anthropic than OAI or the pressure for individual employees to be on good enough terms with labs that they could later get hired], with no further suggestions for what to do about this issue. Tweets which simultaneously acknowledge that (1) more social independence from labs would be good and (2) almost everyone competent is socially connected to labs, even if they don't propose solutions. [literally any take that engages with object level assessment of AI companies done by a third party org like METR, SecureBio, Guidelight, Redwood, Nightingale, etc] === current discourse frontier: takes beyond this line are better than anyone has yet posted === I have [actually reasonable concern] with METR/Redwood/etc. To address this concern, we should do [concrete proposal that makes any sense and would solve the problem]. I am founding a new third party evaluation org / pivoting my existing org to do more third party evaluation. We're doing [thing which is both (1) not identical to METR (2) remotely useful for assessing AI companies]. Our initial work will be on [concrete specific thing that would help]. METR has problem X. We should instead use preexisting organization Y, which does not have problem X and has comparable expertise at assessing loss-of-control risks and misalignment incidents inside AI companies, for instance [past work they've done of similar quality]. (this would be an amazing take but it is impossible to post because no such org exists)

There are two things you need to do to be successful. You have to come up with the best decisions, and you have to have the courage to make them. The real problem for most people is that they think the best decisions are only in their heads. The most important thing I learned was to take those ideas, put them out there, and stress test them. Whatever success I’ve had in life has had more to do with my knowing how to deal with my not knowing than anything I know. That taught me how to take in what others have, and that has been a key thing. #RayDalio# #Principles# #DecisionMaking# #SuccessMindset#

TL;DR A decision-native model called Jeva arrives, purpose-built for "pick the right one" tasks like tool selection and retrieval. Reordering candidates never changes its answer, and its cache-friendly design cuts latency by up to 27.98x. Title: Jeva: System One Models from First Principles URL: Points 🎯 Purpose-built for typed decisions like tool selection, intent classification, and document retrieval, with no-answer (Null) detection built in from the start 🔀 Fixes the classic problem of candidate reordering flipping the decision: Jeva's flip rate across reorderings is a flat 0.00% 📊 Hits 92.05% on BANKING77 and 98.66% AUROC on null detection, both well ahead of comparison models ⚡ Caches and reuses per-candidate computation, cutting latency by up to 27.98x when candidates are fully reused (2,102ms to 75.1ms) 📈 Holds 69.88% accuracy as candidate sets scale from 32 to 2048, running 2.3x faster than a recursive selection baseline 🧩 Achieves set-dependent decisions with just a single attention block, without needing a K-by-K interaction matrix For the unglamorous but frequent job of picking among tools or RAG candidates, a model designed around caching from the ground up looks genuinely useful. #LLM# #DecisionMaking#

TL;DR A decision-native model called Jeva arrives, purpose-built for "pick the right one" tasks like tool selection and retrieval. Reordering candidates never changes its answer, and its cache-friendly design cuts latency by up to 27.98x. Title: Jeva: System One Models from First Principles URL: Points 🎯 Purpose-built for typed decisions like tool selection, intent classification, and document retrieval, with no-answer (Null) detection built in from the start 🔀 Fixes the classic problem of candidate reordering flipping the decision: Jeva's flip rate across reorderings is a flat 0.00% 📊 Hits 92.05% on BANKING77 and 98.66% AUROC on null detection, both well ahead of comparison models ⚡ Caches and reuses per-candidate computation, cutting latency by up to 27.98x when candidates are fully reused (2,102ms to 75.1ms) 📈 Holds 69.88% accuracy as candidate sets scale from 32 to 2048, running 2.3x faster than a recursive selection baseline 🧩 Achieves set-dependent decisions with just a single attention block, without needing a K-by-K interaction matrix For the unglamorous but frequent job of picking among tools or RAG candidates, a model designed around caching from the ground up looks genuinely useful. #LLM# #DecisionMaking#

If an LLM agent can neatly explain why a strategy is "safe," does that actually prove it understands it? Even in auctions and matching markets where telling the truth is provably the best move, LLM agents still drift from optimal decisions. A team from Harvard and MIT tested whether changing the interface or adding textual scaffolds could fix this — and the results were striking. Simply switching a second-price auction to an ascending clock format pushed Gemma's mean deviation from -5.3 to a near-perfect -0.3. In matching markets, just telling the model "your offer only determines whether you get rejected" cut the error rate from 4.2% to 0.2%. 🔍 But the real twist comes next. In the payoff-safety intervention that substantially improved bidding, none of the 600 treated plans even mentioned the stated safety rationale. Meanwhile, belief prompts that asked the model to reason about opponents pushed opponent-modeling language from 2% to 43% — while the actual bids stayed completely flat. Title: Engineering Simplicity: Simple Mechanism Interfaces Steer LLM Agents URL: In other words, what a model can articulate and what it actually does are nearly separate things. The practical lesson lands hard: judging an agent's understanding by its explanations alone isn't enough — you have to check the actual decisions too. #LLMAgents# #DecisionMaking#

decision-making is moving from humans to machines, and most of the time you can't see it happen AI picks what you see on social media and whether you get hired. even when you make the call yourself, the AI you used for research may have nudged you we need to trace those decisions. on @GenLayer every one is onchain: who voted, for what, and why

# Decision Points for Embedding AI Agents in Enterprise Systems # Prompt Change Control 🎯 The Hook A prompt change looks like "just tweaking some wording" — but it is actually a configuration change that can fundamentally alter system behavior. You have CI/CD and code reviews for deployments, but can you trace who changed which prompt, when, and why? If not, you are running production without change management on one of your most critical system components. The realistic approach is not uniform strictness — it is tiered control based on production impact 🔑 📋 Overview Prompt change control is the dial governing how much approval, testing, and version management you require for changes to system prompts and tool definitions. Stricter settings improve reproducibility and safety but slow development velocity. Looser settings enable rapid iteration and experimentation but increase the risk of unintended changes reaching production. The key insight is that different prompt components carry different risk levels, so control levels should vary accordingly. Role definitions in system prompts and few-shot examples demand completely different governance 📋 🔍 Decision Points This dial branches into three tiers based on production impact: High impact (customer-facing, financial, legal) → Strict: code review + 100% eval pass + 2 approvers + canary deploy Medium impact (internal operations, broad scope) → Standard: code review + 95% eval pass + 1 approver Low impact (internal experiments, limited release) → Lightweight: self-review + basic eval pass + change log Further, control policies should vary by prompt component: System prompt (role, constraints) → Low change frequency but high risk. Manage strictly as a design decision Tool definitions (names, descriptions, schemas) → Directly affects tool selection accuracy. Strict control required Few-shot examples → Medium risk. Standard control with eval quality verification Context injection templates → High change frequency, low-to-medium risk. Lightweight to standard Output format instructions → Low risk, but verify alignment with downstream systems ⚡ 💡 Key Details Five components of a control framework: Version control — Manage prompts in Git just like code. Enables diff visualization and history tracking. If you cannot trace "who changed what and when," future decision-making becomes impossible. Always record the reason for each change. Eval gate — Make passing the existing eval suite a deployment prerequisite for any prompt change. Measure eval regression detection rate (the percentage of prompt-induced quality drops caught before deployment) and add missed cases to strengthen the eval suite. Approval process — Reviewer tiers scaled to impact level. Strict tier requires peer engineer plus tech lead. Changes to financial thresholds (like refund limits) add legal review. Canary deploy — Apply the new prompt to a subset of traffic (roughly 10%) and compare quality metrics over 24 hours before full rollout. This can share infrastructure with A/B testing systems. Rollback procedure — The ability to instantly revert to the previous version when issues arise. Git revert integrated with the deployment pipeline is the baseline 🔬 Five metrics to track: prompt change frequency (per environment), change-to-deploy lead time (target 1-3 business days for strict, within hours for lightweight), incident count caused by prompt changes, eval regression detection rate, and rollback rate 📈 ⚖️ Trade-offs Prioritizing development speed (relaxed) enables rapid prompt improvement and fast A/B test cycles. But "who changed what when" becomes untraceable, making root cause analysis during incidents extremely difficult. Unintended changes can slip into production causing quality degradation or security issues. If system prompt role definitions change without anyone noticing, the blast radius covers every single response 😩 Prioritizing reproducibility and safety (strict) makes all changes traceable and simplifies incident root cause identification. But the approval process becomes a bottleneck, extending improvement lead times. When even small improvements require heavy process, team motivation drops and you end up with "I want to fix this prompt but it is too much hassle" — the exact opposite of what you intended ⚠️ The solution is clear: do not lower the control level itself. Instead, shorten change lead time through eval automation and parallel approval workflows. And explicitly separate control levels between experimental and production environments — never trade production safety for experimentation speed. 🛠️ Use Cases Zendesk customer support agent: System prompt changes require PM plus engineering lead approval. Refund limit wording changes add legal review. Full rollout only after eval pass plus canary (10% traffic for 24 hours). Change lead time is 1-3 business days 📞 Slack internal experiment bot: Developers can change prompts via self-review. Basic eval pass is mandatory but approval process is skipped. Change logs are automatically recorded for incident tracing. Change lead time is within hours 💬 Salesforce sales support agent: Tool definition additions and changes require code review. Prompt changes affecting deal stage classification logic need tech lead approval. Large-scale prompt changes are rolled out incrementally to limit blast radius 🎯 Practical tip: Start with strict-leaning controls and relax gradually as the team matures and evals become comprehensive. When a prompt change causes an incident, add that case to your eval suite to automate prevention. And never execute large prompt changes (like rewriting role definitions) all at once — incremental rollout makes impact identification far easier 💪 #AIAgents# #EnterpriseArchitecture#

Intern-Decision scales multimodal structured decision-making across 0.8B, 2B, and 4B models. 🤖 🏆 The three models average 79.38, 84.68, and 90.02 across seven decision benchmarks. Intern-Decision-4B surpasses Jev at 88.74 while achieving better probability calibration. ⚡ Reported mean latency is 33.98 ms, 33.28 ms, and 44.16 ms respectively, compared with 109.70 ms for Jev in the same local HF setup. 🧠 One forward pass answers multiple choice, score, and yes/no questions with typed JSON and calibrated probability distributions. 🖼️ Text states and up to eight images enable routing, tool selection, scoring, and multimodal workflow control. 📜 Apache 2.0. Applicable Qwen3.5 upstream notices must be retained.