If you use LLM-as-judge, this one is for you. (bookmark it) Most teams validate their agent's outputs by calling a frontier model as the judge. It works, until it doesn't. Three problems stack up fast: → Cost: you're hitting a frontier API on every turn, every tool call, every response. In production that burns millions. → Latency: bigger models, remote calls, slow reasoning on every check. → Blind spots: frontier models don't actually know your domain. In finance, insurance, or healthcare, they miss the keywords and principles your work depends on. So I walk through a different approach: train your own small LLM judge. Instead of a giant model, you start with a small one and let the system generate the training data for you. It decomposes your domain, samples synthetic examples, runs them through a debate arena where judges reach consensus, then trains on the refined set. The result is a judge that's cheaper, faster, and more accurate on your data than Gemini, Claude, or GPT, with an OpenAI-compatible endpoint you can even deploy on-prem. I show the whole thing end to end, using a Claude Code plugin and a web interface, with a real insurance RAG grounding evaluator as the example. You can get the plugin here: Here's the full breakdown: 00:00 - Intro 00:12 - Three problems with using frontier LLMs as judges 01:05 - A different approach: train your own small judge 01:31 - How it works (synthetic data and a debate arena) 02:50 - Installing the Claude Code plugin 04:03 - Defining your task with /eval 04:34 - Example: an insurance RAG grounding evaluator 05:51 - Kicking it off and giving early feedback 06:26 - Choosing labels, domain, and strictness 08:30 - The web interface and dashboard 09:52 - Bringing your own example data (optional) 10:26 - The finished model: endpoint, accuracy, and speed 11:16 - Control, on-prem deployment, and interpretability 11:57 - Benchmarks vs frontier models and the GitHub repo 12:30 - Outro I worked with the @pluraiAI team on this. Thanks for sponsoring the video.