---
title: "Analytics for People Decisions Without False Precision"
url: https://pragmaticleader.io/people-analytics-without-false-precision/
author: "Aaron Wise"
published: 2026-11-17
description: "People analytics for hiring and career decisions: make the criteria explicit, track the funnel, and let scores inform judgment instead of replacing it."
categories: ["Data Science", "Leadership & Management"]
tags: ["data-driven decision making", "hiring strategy", "cognitive bias", "structured interviews", "decision-making"]
---

# Analytics for People Decisions Without False Precision

> “Essentially, all models are wrong, but some are useful.”  
> — George E. P. Box and Norman R. Draper, *Empirical Model-Building and Response Surfaces*

Numbers help people decisions when they force the criteria into the open. They hurt when they hide the judgment behind a decimal point. A candidate scored 7.4 against another at 7.1 has not been shown to be better. Someone made a series of judgments, converted them to numbers, weighted them and added them up, and the precision of the result says nothing about the quality of the judgments underneath.

That does not mean the answer is to go back to instinct. Unstructured judgment in hiring has a long record of inconsistency and bias. The useful position sits between the two: use data and scoring to make criteria explicit and comparable, treat every score as an input to a decision a person owns, and be honest about how much the numbers can actually tell you.

## Where data helps

Data earns its place in people decisions in two ways. The first is measuring the process. A hiring funnel has stages, conversion rates and time in each stage, exactly like a sales funnel. Tracking them shows where good candidates drop out, which sources produce hires, and which interviewers reject far more often than their peers. A person running their own job search can use the same measures to see whether the problem is outreach, positioning or closing. None of this requires judging individuals by number. It is operational data, and it is reliably useful.

The second is making assessment criteria explicit. Asking interviewers to score candidates on defined criteria, against a written description of what each score means, does more for decision quality than the scores themselves. It forces the team to agree in advance on what the role needs. It makes interviewers gather evidence for each criterion rather than forming an overall impression in the first ten minutes. And it makes disagreement visible: two interviewers who scored the same candidate a 2 and a 5 on system design saw different things, and the debrief can find out what. The same logic applies when a senior leader scores opportunities against what they want from their next role. Writing the criteria down is most of the value.

There is good research behind structure. Decades of work in industrial psychology have found that structured interviews, with consistent questions and scoring guides, predict job performance better than unstructured conversations. The pattern is reflected in how [a structured hiring process](https://pragmaticleader.io/art-science-hiring-building-talent-engines-scale/) is built. The finding is about structure and consistency, though, not about any particular number.

## Where false precision creeps in

The trouble starts when scores are treated as measurements rather than summaries of judgment. Several things go wrong.

**The scale is not what it looks like.** A five-point rating looks like an interval scale, where the gap between 3 and 4 equals the gap between 4 and 5. It rarely is. Interviewers use the scale differently: some never give a 5, some give 4 to anyone competent. Averaging across interviewers mixes those habits into a number that looks objective and is not.

**The weights are invented.** A scorecard that weights technical depth at 40%, communication at 25% and leadership at 35% looks rigorous. Usually nobody can say where the weights came from. They express a view about the role, which is fine, but presenting them as if they were derived from data gives them authority they have not earned. Scoring models for roadmaps, such as [the ICE method for prioritization](https://pragmaticleader.io/ice-method-cool-way-prioritize-product-roadmap/), have the same weakness, and it matters more when the thing being ranked is a person.

**Small differences are noise.** With three or four interviewers and subjective scores, the difference between a 7.4 and a 7.1 is well within the variation you would get by changing one interviewer. Ranking candidates on differences that small is choosing at random with extra steps. The same caution applies to any analysis with small samples, a theme covered in [finding real insight without bias or overfitting](https://pragmaticleader.io/crunching-numbers-uncover-real-insights-without-bias-overfitting/).

**What gets scored is what is easy to score.** Criteria that are easy to assess in an interview, such as fluency in a technical discussion, tend to crowd out criteria that matter more and are harder to observe, such as how someone behaves when a project goes badly. A scorecard can make the team more confident while measuring the wrong things.

The deeper risk is a change in who owns the decision. When the number is on the screen, it becomes the default. A hiring manager who prefers the 7.1 candidate now has to argue against the data, and many will not. The decision has moved from a person who can be asked to explain it to a spreadsheet that cannot.

## Bias does not disappear when you add numbers

A common claim is that scoring removes bias from hiring. It reduces some kinds and preserves others. Structure does help: consistent questions and defined criteria leave less room for interviewers to favor people who remind them of themselves. That being said, if the criteria themselves encode a bias, scoring applies it consistently and at scale.

The evidence that bias operates well before the interview is strong. In a well-known field experiment, Bertrand and Mullainathan sent otherwise comparable resumes to employers with names that signaled different races, and found that [resumes with White-sounding names received substantially more callbacks](https://www.nber.org/papers/w9873). Any scoring system trained on, or calibrated against, historical decisions inherits whatever bias those decisions carried. Automated screening tools that learn from past hires are the clearest case. The broader issue of whether automated systems can be trusted with consequential judgments is discussed in [engineering trust into the AI stack](https://pragmaticleader.io/terms-algorithmic-endearment-engineering-trust-ai-stack/).

In the United States, there is also a legal frame. The federal [Uniform Guidelines on Employee Selection Procedures](https://www.ecfr.gov/current/title-29/subtitle-B/chapter-XIV/part-1607) treat a selection rate for any group below four-fifths of the rate for the highest group as general evidence of adverse impact. That applies whether the selection step is an interview, a test or a score. If a scoring step is filtering out one group disproportionately, the company needs to know, and the only way to know is to measure outcomes by group. This is one place where more data, not less, is the right response.

## Keeping a human decision on top of the score

The practical question is how to get the benefits of structure without handing the decision to the number. A few habits do most of the work.

Set criteria and definitions before you see candidates. Write what a strong, adequate and weak answer looks like for each criterion. Criteria written after the interviews tend to be rationalizations of a decision already made.

Score independently, then discuss. Interviewers should submit scores and written evidence before the debrief, so the most senior or most confident voice in the room does not set everyone else’s view, and so the last candidate seen does not quietly become the benchmark, a form of [recency bias](https://pragmaticleader.io/recency-bias-exposed-escaping-recency-bias-business/) that debriefs are prone to. The debrief is for examining disagreements, not averaging them.

Require written reasoning for the decision. The final call should be explained in a few sentences tied to the criteria, by a named decision maker. “Candidate A scored higher” is not a reason. “Candidate A showed stronger evidence on the two criteria the team agreed matter most for this role, and the gap on communication is coachable” is.

Treat scores as ranges, not points. If two candidates are within the noise, say so, and decide on the factors the scores do not capture. That is where judgment is meant to operate.

Review outcomes. Six or twelve months later, compare how hires performed with how they scored. With small numbers, this will not produce statistics, but it will show whether particular criteria or interviewers predict anything. Without the review, the scorecard never gets better.

The balance between quantitative and qualitative evidence is the subject of [when to trust soft versus hard data](https://pragmaticleader.io/gut-feel-vs-cold-facts-trust-soft-hard-data/), and people decisions are the hardest case of that question, because the most important qualities are the least measurable.

## A worked hypothetical

Picture a Series B company hiring a staff engineer. Four interviewers score two finalists on five criteria. Candidate A averages 3.9, candidate B averages 3.7. The hiring manager prefers B and feels unable to say so.

Looking at the detail changes the picture. A’s advantage comes almost entirely from one interviewer who gave all 5s. On the two criteria the team had agreed mattered most, technical judgment under ambiguity and mentoring others, B scored higher with three of four interviewers. The averages were hiding the information the team said it cared about.

The right outcome is not that B wins because the hiring manager liked B. It is that the decision is made on the criteria the team agreed in advance, with the reasoning written down. The numbers made that conversation possible. They should not have ended it.

## How it changes by stage

At a seed company hiring a handful of people a year, elaborate scoring is overhead. What matters is writing down the criteria for each role and asking every interviewer for evidence against them. A simple shared document is enough. Watch for the founder bias problem: early teams often hire people like themselves and call it culture fit, which [the post on spotting fallacies](https://pragmaticleader.io/trust-verify-spotting-fallacies-lead-off-cliff/) would recognize as confirmation bias with a hiring budget.

From Series B onward, hiring volume makes funnel metrics and consistent scorecards worth the effort. This is also when calibration matters: training interviewers on the scale, reviewing interviewers whose scores are consistently out of line, and checking selection rates by group.

At scale, the pressure to automate screening grows, and so does the risk. Any automated scoring should be audited for adverse impact, explainable to the candidate and the hiring manager, and overridable by a person who owns the decision. If nobody can explain why the system rejected someone, nobody should be relying on it.

The question for your next hiring review: when your team last chose between two close finalists, could the decision maker explain the choice in terms of criteria agreed before the interviews, or did the higher average decide it?
