Most agent evals I see are an LLM judge answering one question: "Is this answer correct? yes/no." Then a dashboard says 87%. Then it goes to production and breaks in ways the 87% never showed.
I've been building retrieval and evaluation for regulatory compliance, where a wrong answer isn't a bad vibe, it's someone failing an audit. A few things I learned the hard way.
Hybrid RAG is necessary, not sufficient
The standard upgrade path is right: BM25 for exact terms, dense vectors for meaning, metadata filters, and Reciprocal Rank Fusion to merge them. On normal docs, this gets you most of the way.
Compliance docs aren't normal docs. Where it still breaks:
- Cross-references. "Subject to the requirements in section 12.3(b)." The chunk you retrieved says almost nothing on its own; the meaning lives three documents away.
- Exceptions and negations. The rule is on page 4. The exception that applies to your case is on page 61. Both are "relevant," and retrieval has no idea one cancels the other.
- Amendments. A newer document silently supersedes an older one. Both score high. Only one is true.
- Applicability. The right answer depends on the type, size or date of the thing being asked about. Same question, different correct answers.
- Tables. Thresholds and limits live in tables that chunkers shred.
You don't fix these with a better embedding model. You fix them with structure: metadata that knows what supersedes what, resolving references before answering, and splitting the question into "what applies here" and "what does it say."
Why yes/no evals lie
A binary judge hides everything interesting. "Mostly right but cited the wrong section" and "confidently wrong" both become a single bit. You can't see drift, you can't rank, you can't tell which step failed.
What I do instead:
Score, confidence, reasoning. The judge returns a structured verdict: a score on a defined scale, how confident it is, and why. The reasoning is gold during human review, because you see how the judge thinks, not just what it decided.
from pydantic import BaseModel, Field
from pydantic_ai import Agent
class Verdict(BaseModel):
score: int = Field(ge=1, le=5, description="1 = wrong or unsupported, 5 = correct and fully grounded")
confidence: float = Field(ge=0, le=1)
grounded: bool = Field(description="Every claim is supported by the cited text")
reasoning: str
failed_step: str | None = Field(default=None, description="retrieval | applicability | answer | citation")
judge = Agent(
"openai:gpt-5-mini",
output_type=Verdict,
instructions=JUDGE_PROMPT, # rubric + edge cases from human review
)
verdict = (await judge.run(render_case(question, retrieved, answer))).output
Pydantic-AI makes this pleasant: typed output, validation, retries when the model returns junk, and the same code works across providers so swapping the judge model is one string.
Deterministic checks first. Before any LLM judges anything, code checks what code can check. Does every cited section exist? Is the cited text actually in the retrieved chunks? Do the numbers in the answer match the source? These are cheap, never flaky, and catch a surprising amount.
Small steps, separate evals. Retrieval gets its own metrics (did the right sections show up in the top k?). Then applicability. Then the answer. Then citations. When the end-to-end score drops, I know which step dropped instead of staring at one number.
Humans tune the judge. Every week, review a sample by hand. Wherever I disagree with the judge, that case goes into the rubric as an edge case. The judge prompt is a living document, and most of its value came from those disagreements.
Tune reasoning effort and pick models with data. More thinking isn't always better for a judge; sometimes it overthinks a clear case into a wrong one. I test a few effort levels on the labeled set. And for picking models, I start from the Artificial Analysis intelligence index against cost, then confirm on my own cases. Benchmarks narrow the list; your data decides.
The short version
- Hybrid retrieval is the floor. Structure is what gets you past it.
- Never grade with a single bit. Score + confidence + reasoning.
- Let code check everything code can check.
- Evaluate each step on its own.
- Your judge is only as good as the disagreements you've fed back into it.