How to Evaluate Agent Output When There Is No Single Right Answer

Exact Match Fails on Useful Work

Ask an agent to classify a form and you may have a correct label. Ask it to draft an exception memo, investigate a supplier issue, or recommend a response and several answers could work. They won't use the same words. They may not choose the same path.

That doesn't make evaluation subjective in the lazy sense. The business still cares whether the work is supported, safe to use, and appropriate for the next decision. Those obligations can be tested.

The mistake is grading open-ended work as if similarity to a reference answer were quality. A reference can illustrate one good response. It can't cover every valid argument or ordering of evidence.

Our position is that evaluation should begin with the consequence of using the output. I’d change my mind if surface similarity reliably caught harmful omissions and unsupported conclusions. It rarely can, because two texts can look alike while differing on the fact that matters.

Define the Job After the Output

An agent's response usually feeds another action. A research brief informs a selection. A draft message sets an expectation with a customer. An investigation summary tells an operator whether to intervene.

Write down what the next person or system must be able to do. Then define the output obligations from that need. If a brief supports vendor selection, it may need traceable evidence and a clear statement of unknowns. Elegant prose is secondary.

This reframing keeps the rubric close to operating value. “Good analysis” is hard to score. “Every exclusion is tied to a stated requirement and a cited source” is inspectable.

Don't overpack the rubric. Criteria that never affect a release or workflow decision create scoring noise. Keep the obligations that protect the outcome, and handle style with a separate check when it genuinely matters.

Use Anchors, Not Adjectives

Rubrics fail when they use words such as excellent or mostly correct without showing what those grades mean. Reviewers fill the gaps with personal taste. A grading model does the same, just faster.

Build anchors from real cases. For each criterion, show an acceptable output and a defect that should block use. Include a borderline case that must go to review. Explain the difference in terms of the work, not the wording.

Consider source support. A passing answer might cite the governing record for every consequential claim. A blocking defect might rely on a secondary note when the primary record disagrees. The borderline case could involve a primary record that is clearly stale.

Anchors also make change visible. If the business decides stale records may be used with a warning, update the example and rerun prior cases. The rubric becomes a versioned expression of operating policy.

Separate Defects That Have Different Consequences

A single quality score is seductive. It turns a complicated review into one line on a dashboard. It also lets strong writing offset a policy violation.

Keep hard constraints separate from graded judgment. A missing required disclosure may be a fail regardless of the rest of the output. Evidence quality can use a scale if minor variation is acceptable. Tone might be sampled rather than scored on every run.

The same principle applies to aggregate reporting. Don't average away rare defects with large consequences. Report them by type and inspect whether they cluster around a source, action, or request pattern.

This doesn't mean every dimension needs a separate program. Use the few distinctions that change what you do next. A formatting error may trigger repair. An invented claim should block the action and start an investigation.

Human Disagreement Is Signal

Teams often treat reviewer agreement as a target to maximize. Some disagreement is expected when the task permits judgment. The useful question is why the reviewers split.

Maybe the policy has two plausible readings. Perhaps the source record is incomplete. Sometimes one reviewer notices a downstream consequence the rubric never mentioned.

Adjudicate those cases with a domain owner. Record the reason for the final decision and decide whether it represents a reusable distinction. If so, add it to the rubric and evaluation set. If it's truly unique, don't distort the whole framework to make one example easy.

Agreement without inspection can be worse. Reviewers may share the same bad shortcut or skim a long output in the same way. Periodic blind review against known cases helps detect that drift.

Model Graders Need Their Own Evaluation

Using a model to grade agent output can expand coverage. It can also create a closed loop where one model's preference becomes the definition of quality.

Give the grader a narrow criterion and the evidence needed to apply it. Require a structured judgment with a cited reason. Compare that judgment against trained humans on a maintained set, including cases where earlier graders failed.

Watch for position and verbosity bias. A grader may prefer longer responses or accept a conclusion because it appears near a familiar phrase. Swap answer order where comparisons are involved. Test concise valid work against polished work with a hidden defect.

Don't ask the grader for a mystical overall score. Ask whether a specific obligation was met. That makes errors easier to diagnose and lets the team replace one weak check without rebuilding the program.

Evaluate the Trace When the Path Matters

Sometimes the final answer looks fine even though the agent took an unsafe path. It may have queried a prohibited source, exposed data to the wrong service, or attempted the same action twice before succeeding.

Output evaluation won't catch those failures. Add trace checks for permissions, action sequence, and source use when they affect trust. Keep them distinct from judgments about the final work product.

This is especially important for an internal agent workforce. A manager wouldn't evaluate an employee only by reading the final email if the job involved moving money or changing access. The operating record matters.

Trace review doesn't require storing hidden reasoning. Store observable events: requests, sources, policy decisions, proposed actions, approvals, and resulting state. Those are the facts an investigator can use.

Let Production Failures Rewrite the Test

An evaluation set built before release reflects imagined risk. Production supplies the cases nobody predicted. Every meaningful failure should answer a question: which test would have caught this before the action?

Add the case at the right layer. If the agent missed a clause, it may belong in output evaluation. If the source API returned an old version, test the integration. If a reviewer approved without seeing the consequence, repair the gate.

Over time, the evaluation program becomes a record of what the organization has learned about the work. That is more valuable than a static benchmark. Open-ended answers don't need one perfect reference. They need clear obligations and a team willing to sharpen them when reality disagrees.