LLM judges in production: sample them, don't trust them blindly

Sooner or later, every team running agents in production asks a model to grade another model. It is the obvious move. Human review doesn't scale, exact-match checks don't work on open-ended answers, and a well-prompted judge can read a transcript and say "this answered the question" in seconds.

LLM-as-judge is genuinely useful. It is also a measurement instrument with its own biases, its own drift and its own bill. Treat it like one: sample it, calibrate it, keep its output separate from hard facts, and keep an eye on what it costs.

What a judge is good for

A judge earns its place when the thing you care about is real but hard to check with code:

• Did the answer address what the user actually asked?
• Did the agent stay within policy, tone or scope?
• Is a summary faithful to the source it was given?
• Did a multi-step run take a sensible path, or wander?

For these, a judge gives you a cheap, repeatable first pass over far more runs than any person could read. That is valuable. It is still a first pass.

Sample, don't grade everything

Running a judge on every production run is tempting and rarely necessary. A judge call often costs a meaningful fraction of the run it is grading, sometimes more if you use a stronger model to judge a cheaper one. Grading everything can quietly add a large share to your agent's bill for a signal you would get almost as well from a sample.

A random sample, sized so the numbers are stable for the decisions you need to make, is usually enough to see a trend. Two adjustments make sampling work better:

• Stratify by workflow. A uniform sample over all traffic will be dominated by your busiest path. Sample each workflow on its own so the quiet ones get graded too.
• Oversample what is interesting. Runs that errored, retried, ran long or got negative user feedback are worth grading at a higher rate. Just keep that slice labeled separately so it doesn't skew the random-sample trend.

Always report the sample alongside the score: how many runs were graded, out of how many, over what window. A pass rate without its denominator invites over-reading.

Calibrate against human labels

A judge's score only means something once you know how often it agrees with a person who understands the task. Before you trust a judge, and periodically after:

1. Have someone who knows the domain label a set of real production runs.
2. Run the judge on the same runs.
3. Look at where they disagree, not just the overall agreement rate.

The disagreements are where you learn. Maybe the judge passes answers that sound confident but are wrong. Maybe it fails correct answers that are terse. Either way, you either fix the rubric and prompt or you learn to read the judge's number with that error in mind.

Know the common failure modes

LLM judges have well-documented biases. A few to plan for:

• Position bias. When asked to compare two answers, judges tend to favor one position, often whichever comes first. If you do pairwise comparisons, run both orderings and only count a preference that holds in both.
• Verbosity bias. Judges tend to prefer longer, more elaborate answers even when a short one is better. If your agent gets wordier after a prompt change, a judge may reward it for the wrong reason.
• Self-preference. A judge can rate output from its own model family more kindly. Where you can, judge with a different model from the one that produced the answer, and check with your human labels whether it matters for your task.
• Drift when the judge changes. Upgrading the judge model, or a provider updating a model behind the same name, can shift scores overnight with no change in your agent. Pin the judge version, record it next to every score, and re-run your labeled set before switching. A trend line that crosses a judge change is two trend lines.
• Judging what is easy instead of what matters. It is much easier to ask "is this polite and well-formatted?" than "did this resolve the customer's problem?". Rubrics drift toward what a model can check from the text in front of it. Write rubrics from the outcome you care about backwards, and be honest when a judge can't see enough to answer.

Keep judgments apart from hard outcomes

A judge's verdict is an opinion. "The refund was issued", "the pull request merged" and "the user did not come back with the same question" are facts. Both belong in your picture of an agent, but they shouldn't be averaged together or treated as interchangeable.

When they disagree, that is useful. A judge that scores runs highly while the success rate falls is telling you something about your rubric, or about a gap between what reads well and what works. Fold them into one number and you lose that signal.

This is also why a judge makes a weak hard gate on its own. Blocking a deploy or paging someone on a sampled, biased, drifting score will produce false alarms and false comfort in roughly equal measure. Use it to tell you where to look, and let hard outcomes and human review decide.

Watch the judge's own cost

A judge is another model call in production, and it deserves the same scrutiny as your agent's own calls. Track its spend per workflow next to the agent's spend. Watch for the quiet failure where someone raises the sampling rate, or switches to a bigger judge model, and grading starts to cost more than the work it grades. If the signal hasn't changed a decision in a while, sample less.

A short checklist

• Sample per workflow, and report the denominator.
• Calibrate against human labels, and keep the labeled set.
• Pin and record the judge version, and re-check before changing it.
• Run pairwise comparisons both ways round.
• Write rubrics from the outcome back, not from what is easy to check.
• Show judge scores next to hard outcomes, never blended with them.
• Track what judging costs.

Where Morse fits

At Morse we treat sampled quality signals, including judge scores, as their own lines in the performance brief: next to spend, labeled outcomes and production-success trends, each with its source and coverage, and never fused into one number. If that sounds useful, you can join the waitlist.