# Practices for Embedding AI Agents in Software # Critic/Judge & Sampling-Aggregation 🎯 The Hook Asking an LLM to verify its own output is like asking the author to review their own paper. Separate generation from verification, and you structurally eliminate self-confirmation bias. 🔥 The Problem LLMs produce probabilistic outputs and hallucinate plausible-sounding falsehoods. When the same model and prompt handle both generation and verification, the biases carry over -- the model tends to affirm its own work. Ask the LLM that drafted a legal document "is this correct?" and it will likely say yes, even when it shouldn't. 💡 The Pattern Combine two complementary techniques. First, Critic/Judge: verify outputs using a separate model, separate prompt, or deterministic code to break self-confirmation bias. Second, Sampling-Aggregation: generate N candidates from the same input and select the best via scoring or majority vote, using probabilistic variance to your advantage. Together, this becomes "generate N candidates, have the Judge score each, pick the highest." Since cost scales linearly with N, apply this selectively to high-value, high-risk paths. ✅ When to Use Use when: - Incorrect output flowing downstream causes real harm (contracts, medical summaries, financial reports) - The economic value of a single request justifies the extra generation cost - Objective quality criteria exist (accuracy, consistency, schema compliance) Don't use when: - Minor errors are tolerable in casual conversation -- guardrails suffice - Latency budget is extremely tight (sub-second) - Evaluation criteria are subjective and hard to quantify (e.g., "creativity") ⚠️ Pitfalls - The Judge can share the Generator's blind spots. Same model family, same knowledge, same class of errors missed. Use a different model family or embed deterministic checks in the Judge - Increasing N adds cost linearly but with diminishing returns. The jump from N=1 to N=3 is significant; from N=3 to N=5, much less so. Start with N=3 - Specify explicit evaluation criteria in the Judge prompt. "Is this correct?" yields vague judgments. Provide concrete axes: factual accuracy, logical consistency, schema compliance 🔧 Implementation Approach - Separate the Generator and Judge into distinct models or distinct prompts, structurally preventing self-confirmation bias - Run N candidate generations in parallel to minimize the latency impact of sampling - Define the Judge's output as structured data (score plus reasoning), and implement the aggregation logic (best-score selection, majority vote) in deterministic code - Design a fallback path for when all candidates fall below the Judge's quality threshold (re-generation cap, human escalation) - Embed deterministic checks (schema validation, value-range checks, database lookups) as part of the Judge pipeline so verification does not rely solely on LLM judgment #AIAgents# #SoftwareArchitecture#