A ranking of takes on embedded third party evaluation this past week, from worst to best: [contentless sneer against outgroup] METR bad because [huge Sankey diagram]. fact checking? timelines of relative investments that make causal sense? determining which numbers are large or small fractions of other numbers? sounds like some EA bullshit to me. just look at this diagram with all those curved lines, you can SEE the nest of snakes. METR bad because [an actual specific chain of actors who have some pairwise relationship to each other that ends with someone it would be bad for them to have strong COIs with, e.g. "METR once received funding from an org who once received funding from a person who once gave funding to a now-frontier AI company"]. It is beneath my dignity to explain how this actually affects the decisionmaking of METR; you should vaguely perform a mood affiliation and keep scrolling. Don't think too hard. guys guys guys you HAVE to embed my company/organization into the labs. I have been an unwavering supporter of external embedded auditors since 7 minutes ago when I saw this essay taking off. Look at this thing we published once that sort of looks like AI safety if you squint! Fear not, I can assure you that I have never done anything altruistic in my life and if I had I would have been ineffective at it. I observe that [politicized actor] has had [bad take]. Let me use this correct observation to score points for my side and politicize the situation further. === zero point: takes beyond this line are better than logging off === Dear [person with insane take], here is an earnest explanation of why you are wrong. METR is bad/problematic because [an actually reasonable concern, like greater cultural overlap with Anthropic than OAI or the pressure for individual employees to be on good enough terms with labs that they could later get hired], with no further suggestions for what to do about this issue. Tweets which simultaneously acknowledge that (1) more social independence from labs would be good and (2) almost everyone competent is socially connected to labs, even if they don't propose solutions. [literally any take that engages with object level assessment of AI companies done by a third party org like METR, SecureBio, Guidelight, Redwood, Nightingale, etc] === current discourse frontier: takes beyond this line are better than anyone has yet posted === I have [actually reasonable concern] with METR/Redwood/etc. To address this concern, we should do [concrete proposal that makes any sense and would solve the problem]. I am founding a new third party evaluation org / pivoting my existing org to do more third party evaluation. We're doing [thing which is both (1) not identical to METR (2) remotely useful for assessing AI companies]. Our initial work will be on [concrete specific thing that would help]. METR has problem X. We should instead use preexisting organization Y, which does not have problem X and has comparable expertise at assessing loss-of-control risks and misalignment incidents inside AI companies, for instance [past work they've done of similar quality]. (this would be an amazing take but it is impossible to post because no such org exists)

There are two things you need to do to be successful. You have to come up with the best decisions, and you have to have the courage to make them. The real problem for most people is that they think the best decisions are only in their heads. The most important thing I learned was to take those ideas, put them out there, and stress test them. Whatever success I’ve had in life has had more to do with my knowing how to deal with my not knowing than anything I know. That taught me how to take in what others have, and that has been a key thing. #RayDalio# #Principles# #DecisionMaking# #SuccessMindset#

# Decision Points for Embedding AI Agents in Enterprise Systems # Prompt Change Control 🎯 The Hook A prompt change looks like "just tweaking some wording" — but it is actually a configuration change that can fundamentally alter system behavior. You have CI/CD and code reviews for deployments, but can you trace who changed which prompt, when, and why? If not, you are running production without change management on one of your most critical system components. The realistic approach is not uniform strictness — it is tiered control based on production impact 🔑 📋 Overview Prompt change control is the dial governing how much approval, testing, and version management you require for changes to system prompts and tool definitions. Stricter settings improve reproducibility and safety but slow development velocity. Looser settings enable rapid iteration and experimentation but increase the risk of unintended changes reaching production. The key insight is that different prompt components carry different risk levels, so control levels should vary accordingly. Role definitions in system prompts and few-shot examples demand completely different governance 📋 🔍 Decision Points This dial branches into three tiers based on production impact: High impact (customer-facing, financial, legal) → Strict: code review + 100% eval pass + 2 approvers + canary deploy Medium impact (internal operations, broad scope) → Standard: code review + 95% eval pass + 1 approver Low impact (internal experiments, limited release) → Lightweight: self-review + basic eval pass + change log Further, control policies should vary by prompt component: System prompt (role, constraints) → Low change frequency but high risk. Manage strictly as a design decision Tool definitions (names, descriptions, schemas) → Directly affects tool selection accuracy. Strict control required Few-shot examples → Medium risk. Standard control with eval quality verification Context injection templates → High change frequency, low-to-medium risk. Lightweight to standard Output format instructions → Low risk, but verify alignment with downstream systems ⚡ 💡 Key Details Five components of a control framework: Version control — Manage prompts in Git just like code. Enables diff visualization and history tracking. If you cannot trace "who changed what and when," future decision-making becomes impossible. Always record the reason for each change. Eval gate — Make passing the existing eval suite a deployment prerequisite for any prompt change. Measure eval regression detection rate (the percentage of prompt-induced quality drops caught before deployment) and add missed cases to strengthen the eval suite. Approval process — Reviewer tiers scaled to impact level. Strict tier requires peer engineer plus tech lead. Changes to financial thresholds (like refund limits) add legal review. Canary deploy — Apply the new prompt to a subset of traffic (roughly 10%) and compare quality metrics over 24 hours before full rollout. This can share infrastructure with A/B testing systems. Rollback procedure — The ability to instantly revert to the previous version when issues arise. Git revert integrated with the deployment pipeline is the baseline 🔬 Five metrics to track: prompt change frequency (per environment), change-to-deploy lead time (target 1-3 business days for strict, within hours for lightweight), incident count caused by prompt changes, eval regression detection rate, and rollback rate 📈 ⚖️ Trade-offs Prioritizing development speed (relaxed) enables rapid prompt improvement and fast A/B test cycles. But "who changed what when" becomes untraceable, making root cause analysis during incidents extremely difficult. Unintended changes can slip into production causing quality degradation or security issues. If system prompt role definitions change without anyone noticing, the blast radius covers every single response 😩 Prioritizing reproducibility and safety (strict) makes all changes traceable and simplifies incident root cause identification. But the approval process becomes a bottleneck, extending improvement lead times. When even small improvements require heavy process, team motivation drops and you end up with "I want to fix this prompt but it is too much hassle" — the exact opposite of what you intended ⚠️ The solution is clear: do not lower the control level itself. Instead, shorten change lead time through eval automation and parallel approval workflows. And explicitly separate control levels between experimental and production environments — never trade production safety for experimentation speed. 🛠️ Use Cases Zendesk customer support agent: System prompt changes require PM plus engineering lead approval. Refund limit wording changes add legal review. Full rollout only after eval pass plus canary (10% traffic for 24 hours). Change lead time is 1-3 business days 📞 Slack internal experiment bot: Developers can change prompts via self-review. Basic eval pass is mandatory but approval process is skipped. Change logs are automatically recorded for incident tracing. Change lead time is within hours 💬 Salesforce sales support agent: Tool definition additions and changes require code review. Prompt changes affecting deal stage classification logic need tech lead approval. Large-scale prompt changes are rolled out incrementally to limit blast radius 🎯 Practical tip: Start with strict-leaning controls and relax gradually as the team matures and evals become comprehensive. When a prompt change causes an incident, add that case to your eval suite to automate prevention. And never execute large prompt changes (like rewriting role definitions) all at once — incremental rollout makes impact identification far easier 💪 #AIAgents# #EnterpriseArchitecture#

Looking to refine your decision-making skills in decoding HER2 mutations and overexpression in NSCLC? Explore our free, CME-accredited case scenarios to unpick the latest clinical trial data and emerging treatment regimens for HER2 NSCLC:

From @WSJopinion: AI will advance better human decision-making but don’t expect politics to let AI self-agency move forward at anything like the speed now envisioned, writes Holman Jenkins

🥠 How Fortune Makes a Decision? Behind every transaction on Fortune lies a four-layer decision-making process that operates in milliseconds: 1️⃣ Real-time data: The market speaks first. Live price feeds, on-chain activity, and market signals flow continuously. 2️⃣ Intelligence: Fortune AI listens. Raw data is processed and interpreted, transforming "noise" into meaningful signals. 3️⃣ Multi-model: Fortune processes decisions through various independent models before execution. 4️⃣ Execution: The final decision is made and carried out with speed and certainty. This is what distinguishes robust infrastructure from mere guesswork. Every transaction on Fortune is not just matched, but validated. 👇 Learn more here: #FortuneProtocol# #FortuneAI# #PredictionMarket#

When AI embeds itself in decision-making, who oversees its influence? Right now: nobody. We vows to build and open-source the methodologies, datasets and benchmarks that change that. Nothing stays behind closed doors. Helps us fund the research: