---
title: "Measure the Boring Baseline Before You Build With an LLM"
url: https://pragmaticleader.io/llm-baseline-before-you-build/
author: "Aaron Wise"
published: 2026-11-09
description: "LLM baseline first: measure the rule, lookup or small model on the same test, and fix in advance the margin the LLM must beat before you fund it."
categories: ["Data Science"]
tags: ["machine learning", "large language models", "model evaluation", "data-driven decision making", "product strategy"]
---

# Measure the Boring Baseline Before You Build With an LLM

> “Rule #1: Don’t be afraid to launch a product without machine learning.”  
> — Martin Zinkevich, [*Rules of Machine Learning*](https://developers.google.com/machine-learning/guides/rules-of-ml)

Before funding an LLM feature, require a number for the boring alternative: the rule, the lookup, the small classifier or the current human process, measured on the same test as the model. Then require the LLM to beat it by a margin set in advance, large enough to pay for what the model adds beyond its bill. If nobody has measured the boring option, the proposal is not ready for a decision, however good the demo looks.

Picture a Series B company whose support team wants an LLM to route incoming tickets to the right queue. A prototype takes a week. It reads each ticket, picks a queue and gets most of them right in the demo. The project gets a roadmap slot.

Nobody asked how well the current setup already does. The ticket form has a "what do you need help with" dropdown, and a few keyword rules written years ago catch the rest. Had someone counted, they might have found the dropdown and rules already route most tickets correctly, at no per-ticket cost, in a millisecond, with failures anyone can debug by reading the rule. The LLM might still be better. But the demo compared it with nothing, and nothing was never the alternative.

## Why the demo is the wrong comparison

The usual reasoning runs like this: the model handles the task, it took a week, and models only get better, so building on one is a bet on the future. Each part is true, and the decision still does not follow from them. A demo answers "can a model do this?" The decision needs "is a model the best way to do this here?", and those are different questions with different evidence.

A demo is also a biased sample. The examples are chosen by the person who built it, usually the clean and typical cases, and nobody records which ones failed. A baseline measured on a few hundred real inputs and an LLM shown on a dozen hand-picked ones are not being compared at all. The fix is the same test for both, built from real history, the approach set out in [treating evaluations as the spec for an LLM feature](https://pragmaticleader.io/llm-evaluation-as-feature-spec/).

## The case for skipping the baseline

The opposing case deserves a fair hearing. A baseline takes time, and an LLM prompt can work in a day where a trained classifier needs a labeled dataset first. Time to a working version is real value, especially when you are still learning whether anyone wants the feature. And some problems really are language problems with a long tail of phrasing that no rule set will cover. Measuring a keyword rule against them is ceremony.

That argument is right about speed and right about some problems. It does not hold for the commonest case, a team that already has an existing process, historical data and real volume. There, the baseline usually takes a day or two, because it already exists or is a few lines of code. Skipping it saves almost nothing and removes the only reference point that tells leadership whether the new spend buys anything.

## Where the boring option tends to win

Some problems look like language problems and are really lookup, rule or classification problems.

When the answer comes from a fixed set of labels and you have labeled history, a small classifier trained on that history is often about as accurate as an LLM, far cheaper per prediction and easier to retrain. Ticket routing, lead scoring, fraud flags and document type detection usually fall here, and tools that automate model training have made this cheaper than it used to be (see [AutoML for building ML solutions](https://pragmaticleader.io/automl-demystified-smart-way-build-ml-solutions/)).

When the input has a consistent structure, a parser extracts fields reliably. Invoices from a known set of vendors, log lines and form submissions do not need a model that reads them like prose. When the answer lives in a database, a query is the right tool: an LLM asked for a customer's renewal date should at most translate the question into a lookup, not guess.

When the logic is a business rule, such as eligibility, pricing or entitlement, write the rule. Rules are auditable and deterministic, and they change when the business decides they should, not when a provider ships a model update. When the output is a number, such as a forecast, classical statistical methods have decades of practice behind them and are far easier to validate, the discipline covered in [finding real insights without bias or overfitting](https://pragmaticleader.io/crunching-numbers-uncover-real-insights-without-bias-overfitting/).

LLMs earn their place where those options run out: open-ended language with a long tail of phrasing, tasks that require generating text rather than choosing from options, summarizing or comparing long unstructured documents, and problems with plenty of examples but no labels.

## The margin the LLM has to clear

Beating the baseline is not enough. The LLM has to beat it by enough to cover what it adds to the business beyond accuracy. Its per-request price multiplies across traffic, which is why the comparison needs [the cost per answer](https://pragmaticleader.io/llm-cost-per-query/) beside the accuracy figure. It adds a provider dependency, prompt and model versioning, an eval run on every change, and a component whose behavior can shift when the provider updates it. A rule set does none of that.

This is not a new observation. Sculley and colleagues' paper on [hidden technical debt in machine learning systems](https://papers.nips.cc/paper_files/paper/2015/hash/86df7dcfd896fcaf2674f757a2463eba-Abstract.html) argued in 2015 that the model code is a small part of a production ML system, and that the surrounding configuration, data dependencies and monitoring are where the ongoing cost sits. An LLM behind an API moves some of that cost to the provider, but not the evaluation, the monitoring or the dependency.

Set the margin before anyone sees results, in business terms. "The LLM must route at least ten points more tickets correctly than the current rules, at a cost below what the misroutes cost us in handling time." Writing it first stops the bar from moving to fit whichever option the team has grown attached to.

## A worked comparison

Return to the routing example, with illustrative numbers. The team pulls 500 past tickets with the queue each should have gone to. The existing dropdown and keyword rules route 74% correctly. A logistic regression on ticket text, built in two days from three years of labeled history, routes 85%. The LLM routes 89%.

On accuracy alone the LLM wins. Put the other columns beside it and the picture changes. The classifier costs a fraction of a cent per thousand tickets and answers in milliseconds. The LLM costs a few cents per ticket and adds a second or more. Four points of accuracy on the ticket volume of a mid-sized support team might be a few dozen misroutes a week, each costing minutes of an agent's time. If those minutes cost less than the LLM bill plus its upkeep, the classifier is the better product, and the team has the numbers to show it.

The finding that usually matters most is where the 15% of classifier misses fall. If they concentrate in a few ambiguous categories, that is where the LLM belongs.

## The answer is often both

Comparing the two often leads to a design that uses each where it is strong. Rules or a classifier handle the cases they handle confidently, and the LLM gets the residue they cannot place. That keeps cost and latency down for the bulk of traffic and puts the model's flexibility where it is needed.

Another pattern is using the LLM to label data, checking a sample of its labels by hand, and training a small model on the result. The LLM's cost is paid once during labeling, not on every production request. For the wider question of when to build on a model at all, the [guide to building or buying LLM features](https://digitalfoundry.tech/guides/build-vs-buy-llm-features/) covers the vendor side, and the same stage logic applies as in [making build, buy or wait calls at each stage](https://pragmaticleader.io/build-vs-buy-decision-by-stage/).

## What good and bad look like

Good looks like an LLM proposal that includes a baseline number, a cost difference and a margin agreed in advance. "Current rules route 74%; the LLM routes 89%; here is the monthly cost difference and what the misroutes cost today." Leadership can weigh a real trade-off. It also passes the tests in [what earns an AI feature a model](https://pragmaticleader.io/ai-feature-prioritization/): a measurable problem, data you have and a way to evaluate.

Bad looks like a proposal that compares the LLM with nothing, or with a straw man nobody would ship. It also looks like a team that measured the baseline, found it nearly as good, and built the LLM anyway because the project was already announced. That is the sunk-cost pattern that has repeated through every AI wave (see [lessons from AI's failures and false starts](https://pragmaticleader.io/sunk-cost-intelligence-lessons-ais-failures-false-starts/)).

## How this changes by stage

**Seed.** With no labeled data and no existing process to measure, the LLM may well be the fastest way to learn whether anyone wants the feature. That is a sound reason to use one. Log the inputs and outputs, because they become the labeled data for a cheaper baseline later.

**Series A and B.** You have history and volume, and this is where the comparison pays off most. Per-request cost now multiplies across real traffic, and the labeled data to train a simpler model probably exists already in tickets, CRM fields or past decisions.

**Later stage.** The baseline is often a model trained from LLM-generated labels, and the question shifts from "LLM or not" to which requests deserve the expensive path. The discipline is the same as right-sizing any part of a system (see [engineering just enough](https://pragmaticleader.io/goldilocks-platform-engineer-just-enough-not-much/)): use the simplest thing that meets the bar, measure it, and change when the numbers say to.

The question for your next roadmap review is direct: for each LLM project we are funding, what was the boring option's score on the same test, and what margin did we agree the model had to beat it by?