Why "It Works on My Prompt" Isn't Enough: Intro to LLM & RAG Evals

Why "It Works on My Prompt" Isn't Enough: Intro to LLM & RAG Evals

If I handed you a search engine and you only ever tested it with the query "banana" and it always returned a banana, would you ship it? No. You would test a hundred queries, a thousand, across edge cases, bad spellings, empty results, synonyms. And yet a staggering number of LLM and RAG applications are shipped after the developer runs their one golden prompt, sees a good answer, and concludes the system works. I have done it myself, and so has every engineer who built one. The habit is understandable and seductive, because the model always answers, and when it answers your test case well, it feels like success.

The problem is that a single prompt tells you almost nothing about the system. LLM outputs are distributions, not points. Run the same prompt twice and the model gives different answers with different quality. A RAG pipeline has retrieval, ranking, generation, a dozen places for failure, and a system that nails one query can completely miss the next. The only way to know it works is to evaluate it systematically, on a set of cases, with defined metrics, and to keep evaluating as you change things so you can tell whether an edit helped or hurt.

This article is the honest introduction to evals: what to measure, what the common tools do, and the mistakes that make evals worthless or worse than nothing. I am not going to hand you a universal metric, because there is not one. I am going to teach you to build the evaluation that actually matches your problem.

The trap of the demo prompt#

Start by refusing to trust your demo prompt, and understand why you cannot trust it, mechanically. When you test a RAG system with your one question, you are almost certainly testing it on the question you designed the system around, the question whose answer you already know. You are not testing the system, you are confirming it can do the thing you deliberately built it to do, which carries no information. It would be genuinely surprising if it failed.

The instant the question leaves your designed set, the system is being tested for real, and that is when retrieval stops landing the right chunk, when a synonym breaks the embedding match, when the model confidently cites a document it did not retrieve. Your demo prompt tests none of that. It is confirmation bias wearing a prompt.

The fix is attitude before tooling: acknowledge that any single human test proves nothing, and that you need a dataset and a score. The dataset liberates you from your own certainty, and the score lets you compare versions. Without both, you are not evaluating, you are vibing, and vibes do not survive contact with production traffic.

What you actually need to measure#

The hardest part of evaluation is deciding what "good" means for your specific system, because LLM quality has no single number. But there is a structure to it, and once you know the categories, the task becomes filling in which ones matter for you. For a RAG system there are roughly four things to measure, plus two that apply to any LLM application.

The first pair is about retrieval. Does the system fetch the right documents at all, and does it rank the right one on top? These are precision and recall applied to retrieval, and they are separate from generation quality. You can have perfect retrieval and a bad answer, or bad retrieval and surprisingly good generation, and you need to know which, because they have different fixes. Retrieval metrics answer "did the right evidence even reach the model?"

The second pair is about generation. Faithfulness asks whether the answer is grounded in the retrieved context or whether the model invented facts, and this is the one most directly tied to RAG. It is the difference between "the answer is consistent with what the model saw" and "the model made things up." Answer relevance asks whether the answer actually addresses the question, separate from whether it is true. A model can be coherent and relevant and still be wrong.

There are also cross-cutting concerns: latency and cost, and for agent systems, whether the agent used the right tools in the right order. If your system answers great but takes thirty seconds and four dollars per query, it may be functionally unusable. Measurement is not only about correctness.

The shape of an eval harness#

An evaluation is a loop: you have a set of test cases, you run the system on each, you score each output, and you aggregate. The two inputs you must build carefully are the test cases and the scoring. The test set is the crux, and it deserves more effort than the scoring, because a mediocre score on a great dataset beats a great score on a garbage dataset.

Your test set should have three qualities. It should be representative, drawn from the real distribution of questions your users ask, not only the ones you thought of. It should include adversarial cases, hard ones designed to break retrieval or trigger hallucination, near-duplicates, out-of-scope queries, questions whose answer is in one distractor document. And it should separate slices you care about, so you can see if retrieval fails on new products even when it succeeds on old ones. If you do not slice, an overall score of 80 percent can hide that a specific category is at 20 percent.

test_cases = [
    {
        "question": "Can I use the loyalty credit toward non-flight purchases?",
        "expected_contexts": ["doc_412", "doc_413"],
        "expected_answer_contains": ["no", "flights only"],
        "slice": "policy",
    },
    {
        "question": "What is the cancellation window for group bookings?",
        "expected_contexts": ["doc_255"],
        "expected_answer_contains": ["72 hours"],
        "slice": "booking",
    },
]

A case records the question, what context it should retrieve, and what a correct answer should contain. That is a minimal but honest harness. The expected_contexts lets you score retrieval; the expected_answer_contains lets you score generation without needing a human on every case.

Scoring: when a judge is an LLM#

Once you have the cases, you need scores, and this is where the field splits. The old-school way is a set of deterministic rules: does the retrieved set intersect the expected contexts, does the answer contain the expected substring, does the latency stay under a threshold. These are cheap, fast, reproducible, and they are exactly right for mechanical properties. You should measure everything mechanical this way.

But faithfulness and answer relevance have no mechanical formulation. You cannot write a regex for "is this summary faithful to a document." This is where LLM-as-a-judge comes in, and it is the standard approach for a good reason: it is the only scalable way to score open-ended text, and modern models agree with human raters well enough to be a practical substitute. The trap is doing it sloppily. A judge prompt that says "rate this 1 to 5" with no rubric produces noisy, useless scores.

The discipline of a good judge: give it a precise rubric, a definition of each score, and do not ask it to judge multiple things at once. Faithfulness and relevance are different judgments and want different judges, because folding them into one rating conflates a hallucination with a non-sequitur.

from langsmith import evaluate, Client
from langchain_openai import ChatOpenAI

RUBRIC = """
You evaluate a RAG answer for faithfulness on a 1-5 scale.

A faithful answer is fully supported by the provided context with no invented
facts. It may be brief or incomplete, that is a relevance issue, not this one.

Score 1 if the answer states facts that contradict the context.
Score 5 if every stated fact is directly supported.
Return only the integer.
"""

judge = ChatOpenAI(model="gpt-4o").bind(
    response_format={"type": "json_object"},
)

def faithfulness_scorer(run, example):
    prompt = f"""Context:
{example.inputs['context']}

Claimed answer:
{run.outputs['answer']}

{RUBRIC}"""
    score = judge.invoke(prompt)
    return {"key": "faithfulness", "score": int(score["score"])}

results = evaluate(
    lambda inp: my_rag(inp["question"]),
    data=test_cases,
    evaluators=[faithfulness_scorer],
)

A few practical rules make LLM judges usable. Use the strongest model you can afford for judging, because a bad judge corrupts your data and you will spend a week chasing phantom regressions. Keep the rubric operational, define what each number means in behavior, not in feelings. And run a small human spot-check to confirm the judge is not systematically off. LLM judges are not magic, they drift and they have preferences, but they are the best scalable tool we have, and used with a rubric they are genuinely useful.

Metrics that lie#

The reason I keep pushing on methodology is that bad eval engineering produces numbers that lie to you, and a lying number is worse than no number, because it gives you false confidence to ship. Here are the specific ways metrics lie.

Aggregate-only scores hide category failure. The average hides that new-product questions fail while old ones pass. Always slice. Popularity metrics lie: so-called "top-k hit rate" looks strong but ignores whether the model actually used the right chunk given a noisy top-k. A retrieval metric that counts a hit only if the right chunk is in the top 1 is stricter and more honest for many systems, but teams pick top-5 because it looks better. The number should match the real failure mode, not the flattering one.

The judge itself can lie. If your rubric is vague, your judge rates everything a 4, and you have built a system that "always works" at 80 percent while real behavior is unmeasured. If your judge has an order or length bias, it will prefer whichever answer is longer. Cadence of "5s across the board" is a red flag that your rubric is too soft, not that your system is perfect.

When to start, and how to keep it from rotting#

Teams often ask when to invest in evals, and the honest answer is: before you change anything you care about, and before you put the system in front of users at scale. The exact moment is fuzzy, you do not need a thousand-case golden set on your first prototype, but you need enough to see regressions, because the moment you start tuning retrieval or rewriting prompts, you are blind without a baseline. The baseline is the whole point.

Start small: twenty to fifty well-constructed cases, two or three metrics that match your failure modes, one LLM judge with a hard rubric. That is enough to catch the grossest regressions. Grow the set as you learn what your users really ask, and treat the eval as a live artifact, because a dataset written once about a product that has changed is measuring a ghost. When you add a document type or change the chunking, your old cases may no longer be valid, so revise them.

The other thing that keeps an eval honest is running it as part of every change. A prompt tweak that improves one slice and degrades three others is not an improvement, it is a move of the failure, and you can only see that if the eval runs automatically and you compare to the baseline. Evals are not a one-time certification, they are a regression harness, and they only earn their keep through repetition.

The honest conclusion#

The reason "it works on my prompt" is not enough is that your prompt is not your users' distribution. It is one point in a huge space, and the model that answers it well is like a search engine that passed a single test query. Evaluation replaces your anecdote with evidence, and the discipline of building one, a representative dataset, mechanical metrics for mechanical properties, a rubric-bound judge for open-ended quality, sliced so failures are visible, is what separates a demo from a product.

The uncomfortable truth that every engineer eventually hits: your system is worse than you think, and the eval is what tells you exactly where. That is not a bad outcome, it is the point. The eval that reveals your retrieval fails on new products is the best news you will get all week, because it hands you a precise target instead of a vague sense that things are fine. Measure early, measure honestly, and let the numbers, not the golden answer, be the thing you trust.