---
title: "AI Features Without a Feature Factory"
url: https://pragmaticleader.io/ai-feature-prioritization/
author: "Aaron Wise"
published: 2026-11-05
description: "AI feature prioritization: fund a model only when there is a measurable user problem, data you can use and a way to evaluate it. Otherwise it is output."
categories: ["Product Management"]
tags: ["artificial intelligence", "product strategy", "model evaluation", "feature factory", "shifting from output to outcome"]
---

# AI Features Without a Feature Factory

> “For a successful technology, reality must take precedence over public relations, for nature cannot be fooled.”  
> — Richard P. Feynman, Appendix F to the Challenger commission report (1986)

Fund an AI feature when three things are in place: a user problem you can measure, data you already have the right to use, and a way to evaluate the output before and after launch. If any one is missing, the feature is not ready, however good the demo looks. That is the test. It is the same test as for any other feature, applied with more discipline, because AI features are unusually cheap to demo and unusually expensive to make reliable.

The usual reasoning runs differently. “Add AI” arrives from the board, from a competitor’s launch, from a sales team that lost a deal to someone with a chatbot. It names a technology rather than a problem, and success is measured by whether something shipped. The logic is that the market is moving, customers expect it, and a company without visible AI looks behind. Some of that is true. But a request framed around a technology is the same shiny-object request [that dilutes product strategy](https://pragmaticleader.io/shiny-object-syndrome-focus-dilution-derails-product-strategy/) in every other form, and it produces the same result: a lot of output and no way to tell whether any of it helped.

The answer is not to refuse. Some AI features are among the most valuable things a product team can ship right now. The answer is to hold them to the question every request should face: what outcome changes, and how will we know?

## What earns a model

The first condition is a user problem you can measure. Not “help users write better,” but “support agents spend most of their time on tickets that a past answer already resolved,” or “customers abandon onboarding at the step where they map their data fields.” A measurable problem gives you a baseline before any model is involved, and a baseline is what lets you tell later whether the feature helped. This is the shift from output to outcome that [product teams that escape the feature factory](https://pragmaticleader.io/beyond-feature-factory-building-products-matter-not-just-features/) have already made for everything else.

The second is data you have, and the right to use it. A model that drafts support replies needs past tickets and resolutions. A model that maps data fields needs examples of correct mappings. If the data doesn’t exist, is scattered across systems no one trusts, or was collected under terms that don’t allow this use, the feature is a data project first (see [the trust stack for AI](https://pragmaticleader.io/trust-stack-building-ai-can-rely-data-deployment/)). That can still be worth doing, but it should be planned and funded as one. An honest [AI readiness check](https://digitalfoundry.tech/ai-readiness-scorecard/) answers this question before a team commits a quarter to it.

The third is a way to evaluate it. Before the feature ships, the team should be able to say what a good output looks like and check a meaningful sample of outputs against that. In practice this means a set of real examples with known good answers, a way to score the model against them every time the prompt or model changes, and a metric in production that tracks whether users accept, edit or ignore what the model produces. The argument that [the evaluation set is the spec](https://pragmaticleader.io/llm-evaluation-as-feature-spec/) for an LLM feature makes the point directly: if you cannot write the test cases, you do not yet know what you are building. Without evaluation, every change is a guess and every complaint is an anecdote.

If one of the three is missing, the feature isn’t ready. It might be ready next quarter, after the data work or the evaluation set exists. It might never be ready, because the problem it addresses isn’t one users have.

## The case for shipping first

The opposing view deserves a fair hearing. Its strongest form goes like this: nobody knows yet which AI features users will value, so the fastest way to learn is to ship several and watch. Evaluation sets take time to build. Competitors are moving. A team that waits for perfect measurement will ship nothing while others learn in public.

Part of that is right. Exploration has value, and a small, contained experiment can be the cheapest way to find out whether a problem is worth solving. That being said, an experiment is only learning if it can produce an answer. Shipping a feature with no baseline, no success metric and no owner is not an experiment. It is a launch with the measurement left out. The fix costs little: decide the metric and the kill criterion before the first prompt is written, ship to a slice of users, and look at the number on a set date. That is still fast, and unlike the alternative it tells you something.

It is also worth asking how the evidence for “customers expect AI” was gathered. A lost deal where the buyer mentioned a competitor’s assistant is one data point, and the deals where it never came up are rarely counted. Before a board mandate turns into a roadmap, ask what share of recent won and lost deals actually turned on an AI capability.

## Start with the boring baseline

A surprising number of “add AI” requests are solved well by something simpler: better search, a rules engine, a template, a smarter default. Building the simple version first is useful even when you expect to replace it. It gives you a baseline to beat, it ships quickly, and sometimes it is good enough that the model never earns its cost.

Cost matters more here than for most features. A model call has a per-use price that scales with adoption, so a successful AI feature can carry an inference bill that the pricing never accounted for. Know [the cost per answer](https://pragmaticleader.io/llm-cost-per-query/) and who pays for it before launch, not after the first invoice, and decide whether the plan that includes the feature charges for it, which is one more reason [engineering needs to be in on pricing](https://pragmaticleader.io/saas-pricing-decisions-engineering/). The running cost does not stop at inference either. The authors of [“Hidden Technical Debt in Machine Learning Systems”](https://papers.nips.cc/paper/2015/hash/86df7dcfd896fcaf2674f757a2463eba-Abstract.html) argue that quick ML wins commonly carry large ongoing maintenance costs from data dependencies, feedback loops and changes in the outside world. A model-backed feature is a system you keep running, and its quality moves as [models and inputs drift](https://pragmaticleader.io/models-wander-managing-drift-ml-llm-systems/).

## What bad looks like

Picture a Series B product team given a mandate to “ship AI this quarter.” By the end of the quarter there is a summarize button on four screens, an assistant panel, and an AI-generated description field. The launch is counted as a success because the features shipped. Six months later no one can say whether the summaries are accurate, because nothing measures it. Each feature has its own prompt, written by whoever built it, and none has an owner. Usage of three of the four is close to zero. The inference costs are real, and so is the maintenance.

The business cost is larger than the invoice. A quarter of roadmap went to features nobody uses. Customers who tried the summaries and found errors now discount the product’s other AI claims. Sales has a slide about AI that the product cannot fully back up. This is a feature factory with a new raw material, and it fails the same way [every feature factory does](https://pragmaticleader.io/feature-factories-shiny-objects-navigating-product-management-anti-patterns/): output counted, outcome assumed.

## What good looks like

Picture the same team taking a different route. It picks one workflow where users already spend a lot of time and where the product has years of data: drafting replies to support tickets. It measures the current baseline: median handling time, the share of tickets reopened, the share resolved on first reply. It builds an evaluation set from a few hundred real tickets with the replies that resolved them, before writing the first prompt. It ships to a slice of agents with a clear success metric (handling time down, reopen rate flat or better) and a kill criterion agreed in advance, so that a feature that doesn’t move the metric gets removed rather than defended.

Three months later the team can show the number that moved, the cost per drafted reply, and the share of drafts agents sent with no edits. That is the conversation the board actually wanted. It also leaves the team with reusable parts: a logging path for model inputs and outputs, an evaluation suite and a habit of setting kill criteria, all of which make the second AI feature cheaper than the first.

The kill criterion matters because AI work has a strong pull toward sunk cost. A demo that almost works invites one more iteration, then another. As the post on [AI false starts](https://pragmaticleader.io/sunk-cost-intelligence-lessons-ais-failures-false-starts/) argues, these efforts tend to fail on organizational and economic grounds rather than technical ones, and a criterion written before the work starts is the cheapest defense.

## What changes by stage

**Seed.** AI may be the product itself, not a feature of it. Then the evaluation set is the product spec: it is the clearest statement of what “working” means. Use hosted models and spend your effort on the data and the evaluation, not on training anything.

**Series A to Series B.** With an established product and customers, the risk is the one above: many thin AI features, none owned and none measured. One well-evaluated feature in a high-value workflow beats several unmeasured ones. This is also the stage to set shared basics, such as one way to log model inputs and outputs, one place evaluation sets live, and one owner per feature. The inference line will start to matter to finance here, so report it as a unit cost alongside the outcome it buys, the approach in [explaining cloud and AI spend to the CFO](https://pragmaticleader.io/explain-cloud-ai-costs-to-cfo/).

**Later stage.** Governance and procurement enter: customer questions about where data goes, policies on which models can be used, and contract terms about AI use. Frameworks such as the [NIST AI Risk Management Framework](https://www.nist.gov/itl/ai-risk-management-framework) start to appear in security questionnaires and enterprise contracts. The test for any single feature doesn’t change, but the cost of shipping one without an evaluation story goes up, because enterprise customers will ask for it.

## The question to ask

When the next “add AI” request arrives, ask what you would ask of every request: which user problem, which metric, and how will we know it worked? If the answers include data you have and a way to evaluate the output, the model has earned its place. If they don’t, you have found the work that has to come first.

For the next planning meeting, a harder version: of the AI features already in the product, which ones could the owner show a moved metric for today, and what is the plan for the rest?