11 LLM eval methods AI engineers must know: (bookmark this) The tricky part about LLM evaluation is that there is no single metric that tells you whether a system is good. The right evaluation method depends on what you are trying to measure. You may want to compare an output against a known answer, judge whether it is semantically correct, inspect how an agent behaved across multiple steps, or block unsafe outputs before they reach the user. A useful way to organize them is by what they are actually evaluating. 𝗥𝗲𝗳𝗲𝗿𝗲𝗻𝗰𝗲-𝗯𝗮𝘀𝗲𝗱 𝗲𝘃𝗮𝗹𝘀, 𝘄𝗵𝗲𝗻 𝗴𝗿𝗼𝘂𝗻𝗱 𝘁𝗿𝘂𝘁𝗵 𝗲𝘅𝗶𝘀𝘁𝘀 → BLEU measures n-gram precision, checking how much of the generated text overlaps with the reference. → ROUGE focuses more on recall, measuring how much of the reference content appears in the output. → BERTScore compares contextual embeddings instead of exact words, so semantically similar answers can still score well. 𝗝𝘂𝗱𝗴𝗲-𝗯𝗮𝘀𝗲𝗱 𝗲𝘃𝗮𝗹𝘀, 𝘄𝗵𝗲𝗻 𝗴𝗿𝗼𝘂𝗻𝗱 𝘁𝗿𝘂𝘁𝗵 𝗶𝘀 𝘂𝗻𝗮𝘃𝗮𝗶𝗹𝗮𝗯𝗹𝗲 𝗼𝗿 𝗶𝗻𝗰𝗼𝗺𝗽𝗹𝗲𝘁𝗲 → G-Eval uses an LLM to score an output against defined criteria. → LLM-as-Judge gives a model a rubric and asks it to score or compare outputs. → LLM Juries run multiple independent judges and aggregate their verdicts, reducing dependence on a single evaluator. 𝗛𝘂𝗺𝗮𝗻 𝗮𝗻𝗱 𝗱𝗲𝘁𝗲𝗿𝗺𝗶𝗻𝗶𝘀𝘁𝗶𝗰 𝗲𝘃𝗮𝗹𝘀 → Human Evaluation relies on people to score outputs across dimensions such as correctness, relevance, and helpfulness. → DAG-based Evaluation encodes evaluation as deterministic decision logic, routing outputs through explicit checks until a final verdict is reached. 𝗘𝘃𝗮𝗹𝘀 𝗯𝘂𝗶𝗹𝘁 𝗳𝗼𝗿 𝗮𝗴𝗲𝗻𝘁𝘀 → Trajectory Accuracy evaluates the sequence of actions an agent took, not just the final answer. → Multi-turn Evaluation scores behavior across an entire conversation, including consistency, memory, and coherence across turns. 𝗘𝘃𝗮𝗹𝘀 𝘁𝗵𝗮𝘁 𝗿𝘂𝗻 𝗮𝘀 𝗴𝗮𝘁𝗲𝘀 → Safety Evaluation checks for things such as toxicity, bias, sensitive-data leakage, and policy violations before an output is accepted. The important part is that these methods are complementary. A production agent may need reference-based evals for correctness, judge-based evals for subjective quality, trajectory evals for tool use, and safety evals before anything reaches the user. If you are building this evaluation layer, Comet’s Opik brings tracing, debugging, test suites, and agent evals into one open-source stack. Check it out on GitHub: (don't forget to star 🌟) I also wrote a deeper breakdown of how evaluation fits into a production observability and self-repair loop. The full article is quoted below.