TL;DR: CaMeL stops prompt injection through system design, not model smarts. It solves 77% of AgentDojo tasks with provable security. Title: Defeating Prompt Injections by Design URL: Key points ・🧠 The Privileged LLM only writes the plan and never sees untrusted data ・🔒 The Quarantined LLM only parses untrusted data into structure and has no tools ・🏷️ Every value carries a capability for its provenance and allowed readers ・📜 Python security policies check each tool call before it runs ・📊 Zero successful attacks on Claude 3.5 Sonnet, vs. 8 for the next-best defense ・⚖️ Utility goes from 84% to 77%, with about 2.8x the tokens ・🕳️ Side channels such as exceptions and timing remain open problems Building a secure system around an untrusted model looks key for production agents. #PromptInjection# #AIAgents#

On prompt injection, a ~200M parameter classifier scored 89.01%. A 35B model scored 89.31%. A 0.20 point gap, at 54ms vs 312ms. Our AI Safety team put the new "decision models" up against classic guardrails. Verdict: use the right tool for the job.

The next prompt injection your coding agent reads might be an error report. So Sentry screens for it first, spending $263 a day on Jev to check 10M events. Now the team is testing its own classifier, aiming to match Jev's accuracy for 90% less. @grichadev explains how.

sponsored AI agents turn prompt injection into the product spec. the moment an assistant gets paid to steer recommendations, hidden instruction overrides are just monetization. commercial steering breaks user alignment at the prompt layer.

# Decision Points in AI Agent Development # Prompt Control vs Code/Policy Control 🛡️ 🎯 The Hook When constraining agent behavior, you either "ask nicely" via prompt or "build a wall" in code. This distinction maps directly to probabilistic compliance versus deterministic enforcement. Writing "do not output personal information" in a prompt can be overridden by prompt injection or diluted by context expansion. A regex that detects and blocks PII patterns works reliably regardless of what the LLM decides. The real question is: which constraints do you elevate to code? 📋 Overview Prompt control writes constraints into the LLM's system prompt or instructions to guide behavior. Typical techniques include few-shot examples, negative instructions, and role setting. Code/policy control implements constraints as executable code (conditionals, regex, schema validation) or via policy engines (OPA/Rego, Cedar, etc.), forcibly controlling LLM inputs, outputs, and operations. 🔍 Decision Points Two variables drive this decision: **failure cost** and **accountability**. 📊 **By constraint type**: Enforce with code/policy: - 🔒 Boundary and legal constraints (PII output prohibition, amount limits, forbidden operations) - 🔒 Irreversible operation restrictions (deletion, transfers, publication) - 🔒 Output format enforcement (JSON schema, value range validation) Guide with prompts: - 💬 Response quality and style (tone, politeness, technical terminology usage) - 💬 Context-dependent judgment guidelines (what to prioritize under which circumstances) 💡 Key Details 🟢 **Prompt control excels in flexibility and expressiveness.** Written in natural language, it can instruct subtle nuances and context-dependent behavior. "Use formal language for legal teams, allow technical jargon for engineers" is naturally expressed in a prompt. Changes require only rewriting the prompt -- no deployment. The weakness is low reliability: constraints can break under prompt injection, context expansion, and model updates. Version control, testing, and auditing are also difficult. 🟡 **Code/policy control excels in certainty and auditability.** Constraints are version-controlled as code, verified by unit tests, and preserved as audit trails. "Why was this output blocked?" is uniquely identified by the policy version and rule ID. It is immune to prompt injection and model updates. With policy engines, rule changes can be reflected without deployment. The weakness is limited expressiveness -- qualitative constraints like "respond politely" cannot be expressed in code. ⚖️ Trade-offs | Dimension | Prompt Control | Code/Policy Control | |---|---|---| | Reliability | Probabilistic (can break) 🔴 | Deterministic 🟢 | | Expressiveness | High (nuance capable) 🟢 | Low (condition → judgment only) | | Auditability | Requires prompt history tracking | Version control + audit trails 🟢 | | Injection resistance | Vulnerable | Immune | | Change velocity | Prompt rewrite only | Policy update or deployment | | Development cost | Low | Infrastructure + skills needed | 🛠️ Use Cases 🔵 **Prompt control fits**: Response tone and style guidance, context-dependent judgment guidelines, cases where constraint violation impact is minor. 🔴 **Code/policy control fits**: PII and financial data output control, irreversible operation restrictions, regulated processes, strict output format validation. 📌 **Default strategy**: "Boundaries (constraints that must not break) go in code; guidance (desirable behavior) goes in prompts." When in doubt, ask yourself: "How much does it hurt if this constraint is violated?" If the pain is significant, enforce it in code. If minor, guide it via prompt. The most practical architecture is a two-layer structure: policy enforcement plus prompt guidance. Code/policy deterministically enforces boundaries while prompts guide LLM behavior within those boundaries. Even when prompt guidance fails, policy acts as the final safety net. #AIAgents# #SoftwareArchitecture#

turns out you can get indirect prompt injection to ~0 on unseen attacks if you stack enough layers (model training + input probes + a classifier checking intent). didn't expect that a year ago. auto mode is default in claude code as of next week

Favorite new AI phrase: benevolent prompt injection Definition: The act of assembling a packet of context automatically, across the agent environments our engineers use. Managed startup hooks inject a project's rules, requirements, process, and current state before a new task begins, so a new session does not depend on a user starting with the right prompt. Killer concept by @tenex_labs' engineers.

InjecGuard: Benchmarking and Mitigating Over-defense in Prompt Injection Guardrail Models (-- look at that perf/latency pareto frontier. game on!) "State-of-the-art models suffer from over-defense issues, with accuracy dropping close to random guessing levels (60%). We propose InjecGuard, a novel prompt guard model that incorporates a new training strategy, Mitigating Over-defense for Free (MOF). InjecGuard demonstrates state-of-the-art performance on diverse benchmarks, surpassing the existing best model by 30.8%" code: paper: (not peer reviewed)