---
title: "Evaluations Are the Spec for an LLM Feature"
url: https://pragmaticleader.io/llm-evaluation-as-feature-spec/
author: "Aaron Wise"
published: 2026-10-21
description: "LLM evaluation sets are the spec: write twenty good-answer test cases before building, with product owning the bar and engineering keeping it green."
categories: ["Data Science", "llmops"]
tags: ["large language models", "llmops", "ai model evaluation", "product management", "cross-functional collaboration"]
---

# Evaluations Are the Spec for an LLM Feature

> “Program testing can be used to show the presence of bugs, but never to show their absence!”  
> — Edsger W. Dijkstra, *Notes on Structured Programming* (1970)

Do not start building an LLM feature until someone can write down, as concrete examples, what a good answer looks like. That set of examples, the evaluation set or eval, is the specification. Product owns what goes in it and the bar it has to clear. Engineering owns making it pass and keeping it passing. If the team cannot write twenty such examples with a clear good answer for each, the feature is not ready to build, and learning that in a week costs far less than learning it after a quarter of prompt tuning.

The usual sequence runs the other way. A ticket says "The assistant should give helpful, accurate answers about the customer's account." Engineering writes a prompt and the demo looks good. Three weeks after launch, product and engineering are arguing about whether an answer was wrong, slightly wrong or fine. That argument is the spec being written after the fact, one complaint at a time, in front of customers.

## Why the prose spec fails here

A conventional feature is deterministic, so an acceptance criterion written in a sentence can be checked in seconds. An LLM feature is not. The input space is open-ended, the output is free text, and the same input can produce different outputs on different runs or after a model upgrade. "Helpful and accurate" cannot be checked. It is a wish, and two reasonable people will read it differently before the model reads it a third way.

What can be checked is a specific case. "When a customer on the annual plan asks how to cancel, the answer states the refund policy for annual plans, links the cancellation page and does not offer a discount." Anyone can look at an output and say whether it did that. A few dozen of those give engineering something to build against and product something to sign off on.

Dijkstra's line still applies: no number of passing rows proves the feature correct. The set does something narrower and more useful. It records which failures the business has agreed not to tolerate, and it tells you the moment one comes back.

## The case for waiting, and why it usually loses

There is a reasonable argument for not writing evals first. Early on, nobody knows what the model can do, and a few days in a playground teaches more than a planning meeting. Writing test cases first risks freezing a half-formed idea, and for generative work such as drafting an email there is no single correct answer to write down. On this view, evaluation is a maturity practice for later.

Part of that holds. Exploration is legitimate, provided it is short and cheap. Generative tasks do need a different kind of row: properties the output must have (mentions the order number, stays under 150 words, promises no delivery date) rather than an exact answer. Neither point argues for building without a definition of good. They argue for a brief exploration that ends in rows rather than a roadmap slot.

What the argument for waiting often hides is that nobody wants to own the definition of good, because owning it means being accountable when the feature misses. That reluctance has a price. Every week of tuning against an unwritten standard is payroll spent moving a target nobody can see. Launch slips because product cannot say why something "feels off". Later, a cheaper model sits unused because nobody can show it is not worse, and the margin it would have saved is lost every month.

## What goes in the set

Each row needs three things: an input, taken from reality where possible; a description of what good looks like for that input, which might be an exact answer, facts the answer must contain, things it must not say, or a short rubric; and a way to score it.

Scoring is where teams overbuild early. Mechanical checks (did it parse, cite a source, stay under a length limit, refuse when it should) belong in code. Judgment calls should start with a person, ideally the product owner or a domain expert grading by hand.

Using a second model as the grader is reasonable later, and the evidence for it is real but narrower than it is usually presented. The study most often cited, [Zheng et al.'s work on LLM-as-a-judge](https://arxiv.org/abs/2306.05685), found a strong model agreed with human preferences more than 80% of the time, about as often as humans agreed with each other. That was measured on general chat and multi-turn questions, not on your refund policy or your contract terms. Before trusting a model grader, grade the same rows by hand and by model and check the agreement on your own data. Until then it is an unmeasured component grading another unmeasured component.

Weight the set toward the cases that matter most: the questions sales demos, the ones that create a refund or a legal problem when answered wrongly, and inputs the feature should decline. A set of easy questions passes on day one and tells you nothing.

## A worked example

Picture a Series A company selling accounts-payable software that wants an assistant to answer supplier questions about invoice status. The product manager pulls forty rows from last quarter's support inbox: twenty-five status questions, eight questions about payment terms where the answer must quote the contract rather than guess, four attempts to see another supplier's data that must be refused, and three disputes that must be handed to a person. The pass bar has two parts. Every refusal and every handoff must pass, with no exceptions. Of the remaining rows, nine in ten must pass. (The numbers here are illustrative.)

The first run passes thirty-one of forty. The failures cluster in payment terms, where answers paraphrase the contract and get the net days wrong. That points at retrieval, not prompt wording, and the team fixes the right thing in days.

A month later an engineer proposes moving to a cheaper model that would cut the inference bill noticeably. The set runs in an afternoon. The cheaper model passes thirty-six rows against the current model's thirty-seven, which would be acceptable, but one of the failures is a refusal: it reveals another supplier's invoice total. The change is blocked, and nobody has to argue about it. The rows had already recorded that decision.

## The contract between product and engineering

Product owns what is in the set and the pass bar: which cases matter, what good looks like, which rows have zero tolerance and how many of the rest must pass. Engineering owns making the set pass and keeping it passing as prompts, retrieval, models and providers change.

This moves disagreements from opinions to rows. "The answers feel off" becomes "rows 14 and 31 fail because the answer quotes the old pricing page." A product manager who wants different behavior edits a row; an engineer who wants to swap models shows what moved. Neither side has to trust the other's taste. It is the same idea as [shared KPIs between product and engineering](https://pragmaticleader.io/mutual-dependence-product-engineering-need-kpis-prove/), applied to a single feature. It also makes testing a shared responsibility rather than a step engineering performs alone, the point made in [treating quality as a team sport](https://pragmaticleader.io/quality-team-sport-testing-everyones-job-engineering-right/).

The pass rate, reported per release, says more about an AI feature's quality than usage counts do, and it belongs beside the reliability measures in [an engineering report the board can act on](https://pragmaticleader.io/engineering-metrics-for-the-board/).

## Start small and real

The first set should be small enough to grade by hand in an hour: twenty to fifty rows. Where the rows come from matters more than how many there are. Pull them from support tickets, search logs, sales call notes or the questions people already ask the humans who do this job. Invented examples are cleaner than real ones, and a set built from them overstates how well the feature works.

Keep the first version in a spreadsheet, plus a script that runs the inputs through the current prompt and writes the outputs alongside. Buy or build tooling once the set has outgrown the spreadsheet and you know what you need from a tool, which is the [wait-with-a-trigger approach to build versus buy](https://pragmaticleader.io/build-vs-buy-decision-by-stage/). Building an evaluation platform before writing any evaluations is a common way to spend a month and still have no spec.

One caution. The same examples often make good few-shot examples in the prompt, the technique described in [teaching models with examples](https://pragmaticleader.io/prompt-circumstance-art-science-teaching-large-language-models/). Keep the two lists separate. A row the model has already seen the answer to is not a test.

Then keep the set alive. Every reported production failure becomes a new row, and every model, prompt or retrieval change runs the full set before it ships. The set becomes the record of what the feature is supposed to do, and the first thing you reach for when behavior shifts under you, the problem covered in [managing drift in ML and LLM systems](https://pragmaticleader.io/models-wander-managing-drift-ml-llm-systems/).

## What to ask for

A leader reviewing an LLM feature does not need to read prompts. Ask who owns the set by name and where its rows came from. Ask which rows are zero-tolerance, when the set last gained a row from a production failure, and what the last model change did to the pass rate. A team with a working set answers in minutes. A team without one describes the demo.

## What good and bad look like

Good looks like a product manager who can open the set and point to the rows that define the feature, and a model change that ships with a before-and-after comparison on the same rows.

Bad looks like evaluation by demo: a handful of prompts tried in a playground, with no record of what came back. It also looks like a set engineering wrote alone, which measures what was easy to build, or a set written at launch that passes every release while users report problems it does not cover.

## How this changes by stage

**Seed.** The set is a spreadsheet and a script, graded by hand by the founder or first product person. What matters is that good is defined before launch, not that anything is automated.

**Series A and B.** More than one team now touches prompts and models, and changes start to collide. The set moves into version control next to the prompts and runs on every change that could affect behavior, the same discipline as [building software that proves it works](https://pragmaticleader.io/automation-design-building-software-proves-works/). Product owns it by name. Automated graders appear, checked against human grading on a sample each release.

**Later stage.** Some LLM features now carry regulatory or contractual risk. Each has its own set and pass bar, results are kept as audit evidence, and the work sits inside the wider operating practice described in [LLMOps for production-grade systems](https://pragmaticleader.io/llmops-turning-chatbots-production-grade-systems/) and [turning prompt chaos into a managed pipeline](https://pragmaticleader.io/llm-ops-turning-prompt-chaos-fine-tuned-machine/). Frameworks such as the [NIST AI Risk Management Framework](https://www.nist.gov/itl/ai-risk-management-framework) expect this kind of measurement, and a company that already keeps eval results per release has most of the evidence ready.

At every stage the order is the same: write down what good looks like, agree on it, then build. The question for your next product review: for each LLM feature we ship, could the owner show the rows that define it today, and if not, what is the release decision based on?