How to evaluate AI agents and RAG systems before production breaks them

Have you built an AI system that worked flawlessly in testing, then fell apart the moment real users found the paths you did not expect?

That is the central problem in agent and RAG evaluation. A system can look excellent in a sandbox, pass a thin set of happy-path prompts, and still fail in subtle, expensive ways: by retrieving the wrong policy, skipping a required tool call, losing state, or silently acting on the wrong customer record.

The goal is not to prove that your model can generate a good answer. It is to prove that the workflow behaves acceptably when the answer, the path, or the surrounding systems get difficult.

Run the Agent & RAG Reliability Audit to see which evaluation gaps are most likely in your workflow.

When an AI failure becomes a business failure

Unreliable LLM systems can cost reputation, customers, time, revenue, and—in high-stakes contexts—create legal exposure. The Air Canada chatbot case is a useful reminder: a customer relied on a bot’s incorrect statement about bereavement fares, and the company remained accountable for the information it presented.

The lesson is not “never deploy a chatbot.” It is that a confident, fluent output is not evidence of a controlled system.

Why agents and RAG are harder to evaluate than one model call

Testing a single LLM response can be relatively straightforward: use known questions, compare an output to a rubric or a gold answer, and check whether it follows instructions.

Agents and RAG pipelines add dynamic paths:

• Retrieval may return irrelevant, incomplete, or stale context.
• An agent may use the wrong tool, retry endlessly, or skip a required action.
• State can desynchronise across steps and lose important context.
• Several paths may produce an acceptable answer, but only some of those paths are safe or auditable.

That means final-answer quality is only one signal. You also need to evaluate the evidence retrieved, tool use, state transitions, approval gates, and recovery behaviour.

The failure modes to look for

1. Agentic execution failures

An infinite action loop is a familiar example: a tool call fails, the agent repeats it, and your token budget disappears. Type mismatches can make a tool fail or an application crash. State or memory loss can make an agent answer a later request without the constraints it learned earlier in the workflow.

2. Retrieval failures

RAG is not automatically grounded. Poor source data, chunking, metadata, routing, or retrieval can produce an answer that sounds credible but rests on irrelevant evidence. Large context windows also create a “lost in the middle” problem, where important instructions or retrieved evidence are ignored.

3. Silent failures

The most damaging failures often return a successful status code. An agent might update the wrong record, send an email to the wrong person, or complete a task with the wrong data. The workflow appears to work until a power user finds the incident.

Why passing evals can still fail in production

The most common reason is that a team evaluates the final answer but not the path that produced it. A weather agent can give the right forecast after hallucinating it instead of calling the approved weather API. The answer looks correct; the system is still unreliable.

This is the wrong-path-to-right-answer problem. For agents, “correct” and “acceptable” are not always the same thing.

The evaluation flywheel

Evaluation should be a production loop, not a one-time gate.

1. Build a custom evaluation dataset

Start with your workflow and your failure cost, not a generic benchmark. Curate 20–25 realistic cases: known edge cases, high-value requests, adversarial inputs, and examples that should be escalated rather than answered.

2. Turn production traces into test cases

If the system is live, traces are your best source of realistic data. Convert failures, overrides, confusing outputs, and surprising successes into versioned cases. Your dataset should evolve as users and dependencies change.

3. Use a small, deliberate metric set

Do not begin with twenty metrics. Start with the few that map to your risk:

4. Run evaluation against representative inputs

Use the tools that fit your stack—such as DeepEval, Ragas, LangSmith, or a focused in-house harness—but keep the baseline explainable. A low score should be traceable to a source, a step, or a behaviour your team can change.

5. Put evals in the delivery pipeline

Prompts, tools, retrieval settings, and models are production changes. Run the right checks on pull requests and before releases. If a model swap reduces faithfulness or changes tool use, the release should surface that evidence before a customer does.

6. Feed traces back into the dataset

Production changes. Models drift, APIs fail, documents change, and users discover new paths. A static evaluation suite becomes a comfort blanket. The useful loop is: trace → review → new test case → release gate → trace.

The vibe-check trap

Four patterns quietly weaken most evaluation efforts:

1. Doing too much too early: huge synthetic datasets and too many metrics before the core risks are defined.
2. Testing only happy paths: predictable questions are not a release test.
3. Relying on subjective prompts in a playground instead of a repeatable workflow.
4. Keeping the same stale evaluation dataset while production changes beneath it.

A practical starting point

If you need a place to start this week, do this:

1. Pick one workflow where a bad answer or action has a real cost.
2. Collect 20 representative cases, including edge cases and “must escalate” examples.
3. Define three to four evaluation signals tied to that workflow.
4. Capture traces for each failure and unexpected path.
5. Add the smallest useful release gate before changing the workflow again.

That is enough to move from “it looked good in the demo” to a system your team can inspect, improve, and defend.

Useful tools and references

• LangSmith evaluation documentation
• DeepEval introduction
• AI Engineering by Chip Huyen, Evaluation Methodology
• Prompt Engineering for LLMs by John Berryman and Albert Ziegler, Evaluating LLM Applications

Need help making this operational?

If a live or near-live workflow is failing, the next step is not more prompt guessing. Run the Agent & RAG Reliability Audit, or book a 30-minute Fit Call if the risk is already urgent.