Skip to content
The Pragmatic Leader

Data EngineeringEngineering

The First Data Platform: What a 30-Person Company Actually Needs

Aaron Wise · · 8 min read

Four hand tools laid out on a wooden plank: pliers, a screwdriver, side cutters and a hammer

“A complex system that works is invariably found to have evolved from a simple system that worked.”
— John Gall, Systemantics

A 30-person company needs four things from its first data platform: one warehouse, one managed ingestion tool, transformations written in SQL and kept in version control, and one BI tool. It needs one person who owns all four. It does not need anything else yet, and each extra component you add before a specific signal calls for it is a cost you pay in attention more than in licenses.

That recommendation runs against two instincts you will meet at this size. The first comes from engineering: build the platform once, properly, for the company you expect to become. The second comes from everyone else: the spreadsheets work well enough, so why spend anything at all? Both deserve an answer, because both are reasonable from where the person is standing, and both usually end in the same place, which is a board meeting that opens with an argument about whose revenue number is right.

Start from the questions, not the architecture

Picture a 30-person company a year past its Series A. The CEO wants a revenue number that matches what finance reports to the board. Sales wants to know which closed deals started as product trials. An engineer who has been reading about lakehouses and streaming returns with an architecture diagram that has eleven boxes on it.

Before anyone debates the boxes, list the questions. At this size they are few and stable: revenue and how it moved, pipeline and conversion, product usage, churn and its causes, perhaps support load. Most are asked weekly or monthly. A dashboard that refreshes overnight answers nearly all of them. The data is small, too: a handful of SaaS systems (CRM, billing, support, product analytics) plus the product's own database. Any modern cloud warehouse handles that volume without tuning.

The hard problems at this stage are not technical. They are agreeing what "active customer" means, getting the billing system and the CRM to reconcile, and deciding who settles disputes about a definition. No amount of infrastructure solves those. If nobody has yet done the listening work of finding out who is asking what, do that first; starting a data program from a blank slate covers it. A platform designed before the questions are known is an expensive guess.

The case for building bigger, heard fairly

The engineer with the eleven-box diagram is not wrong about everything. Migrations are painful. Moving from one warehouse to another, or rewriting transformation logic from a BI tool's calculated fields into SQL, costs real weeks. If the company is going to need streaming or a catalog in two years, the argument goes, it is cheaper to put them in now than to retrofit them under pressure.

The argument fails on two counts. The first is that nobody can say which of those components the company will need. An enterprise sales motion needs audit logs; embedded analytics needs low-latency ingestion; neither path needs both on day one. Paying for all of them now is insurance against every possible future, and the premium comes out of the roadmap. This is the trap described in architecting for evolution instead of future-proofing: build so the next step is cheap, not so the next step is already taken.

The second is that the migrations people fear are mostly avoidable with the small platform, provided two habits hold. If every transformation lives in reviewed SQL, moving warehouses is a matter of translating dialects, not rediscovering business logic. If every source comes in through a managed connector, swapping the destination is configuration. The expensive migration is not small-to-large. It is undocumented-to-anything.

The four pieces

One warehouse. Every source lands in one place and every report reads from it. Which warehouse matters far less than having exactly one. The managed cloud warehouses all fit, and at this size a well-run Postgres instance can too. The lakehouse debate is worth having later, once you have files and workloads a warehouse handles badly; at 30 people you almost certainly do not.

One ingestion tool. Use managed connectors for the SaaS sources and for replicating the product database. Writing and maintaining your own API extractors is the fastest way to turn your one data person into a full-time integrations engineer. This is the clearest case for the default in buy by default, build where you differ: nobody chooses your company because of how well it copies rows out of a CRM.

Transformations in SQL, in version control. This is the piece teams most often skip and most regret skipping. Raw source tables are not reporting tables. Something has to join the CRM to billing, remove duplicate customers and define revenue. Put that logic in SQL files in a repository, reviewed like code and run on a schedule. dbt is the common tool here, and its guide to structuring a project (staging, intermediate and mart layers) is a sensible default to adopt rather than invent. SQL matters because the people who will read and fix this logic are analysts and engineers alike, and both can read it.

One BI tool. Dashboards and ad hoc exploration for the rest of the company, with business logic kept out of it. A revenue calculation defined inside a chart's calculated field is a definition nobody can review, test or reuse, and you will end up with a slightly different one in every chart. The BI tool should mostly display tables the transformation layer has already shaped.

Scheduling comes along for free: the ingestion tool runs on its own schedule and the transformations run after it. A separate orchestrator can wait.

What it costs, in business terms

The cost conversation usually focuses on licenses: the ingestion tool priced on volume, the warehouse on compute and storage, the BI tool per seat. Those are worth watching, but they are rarely the binding constraint.

The cost that matters is attention. At this stage "the data team" is one person, often an analyst or an engineer doing it part time. Every component added is one more thing to upgrade, debug and secure. A useful test: one person can keep the platform running in a few hours a week, and someone else could take over from their documentation if they left. If the design fails that test, it is too big for the company, however reasonable each piece looks on its own.

There is also a hiring cost. A small, clean platform with reviewed SQL is easy to hand to a first data engineer or a contractor. An elaborate one built by a single enthusiast is hard to hand to anyone, and it tends to leave with its author. That is the same generalist-to-specialist transition every early team goes through, and the platform should make it easier, not harder.

What to leave out on purpose

Streaming ingestion, a data lake alongside the warehouse, a data catalog, a standalone orchestrator, a feature store, reverse ETL, a semantic layer product. Each solves a real problem. None solves a problem a 30-person company has yet.

Leaving them out is not neglect if it is written down. A short note listing what was excluded, and the signal that would bring each one in, turns "we never got round to it" into "we decided not to yet", and it gives the next person a starting point instead of an archaeology project.

What good looks like, and what bad looks like

Good: there is one definition of revenue, it lives in a reviewed SQL model, and it matches finance to within a known, explained difference. A new business question is answered in a day or two because the clean tables already exist. The person who built it can take a two-week holiday and the dashboards still refresh. Metric definitions that leadership relies on carry an owner, the same discipline that keeps board reporting honest.

Bad: five tools and one maintainer. The board metric is computed one way in the BI tool and another in a spreadsheet, and the meeting starts with an argument about which is right. Nobody can explain why last quarter's number changed. Some of that is data debt you can pay down; most of it is avoidable by keeping the platform small enough to keep clean.

Signals that it is time for more

Stopping at four pieces is a decision with an expiry date. These are the signals that it has arrived, with the addition each one usually justifies.

  • A product feature or an operational process needs data fresher than overnight. A dashboard someone would like hourly does not count; an in-app usage view for customers does. That is when near-real-time ingestion earns its cost.
  • Data starts feeding the product: customer-facing analytics, a model, a retrieval index for an AI feature. From that point the pipeline is production software and needs tests, monitoring and an on-call owner. An AI readiness assessment is a reasonable way to check the data is ready before the feature depends on it.
  • More people write transformations than one reviewer can keep consistent. That is the time for automated tests on models, written data contracts with the teams that own the sources, and possibly a catalog.
  • Full rebuilds of the transformation layer become slow or expensive enough that people avoid running them. That calls for incremental models, designed so they can still be safely re-run.
  • A customer, auditor or regulator asks who can see which data. Column-level access controls, handling of personal data and audit logs move from optional to required, ideally through governance that fits how people already work.
  • Pipelines need to depend on each other across tools in ways the built-in schedulers cannot express. Now a dedicated orchestrator pays for itself.

Notice what is not on the list: the company raising a round, a competitor announcing a data platform, or a new hire who used a bigger stack somewhere else.

How the call changes by stage

Seed. You may not need even the four pieces. A read replica of the product database, a BI tool pointed at it and the billing system's own reports will answer most questions. Build the warehouse when you first need to join two sources, and not before.

Series A, 20 to 50 people. The four pieces are the right size. This is when good habits are cheapest to establish: logic in version control, one definition per metric, one named owner. Retrofitting those habits onto a sprawling setup at Series B costs far more than starting with them.

Series B. Two or three of the signals above usually show up, and the first dedicated data engineer becomes a reasonable hire. That person inherits a small, clean platform with reviewed SQL and a written list of deferred decisions, which lets them spend their first quarter adding what the business now needs instead of untangling what it never did.

Later stage. The platform becomes a product in its own right, with internal customers, service levels and a roadmap. The four pieces stay at its center; governance, access and cost allocation grow around them.

The question to take into your next leadership meeting: which of our recurring business questions can we answer from one trusted number today, and for the ones we cannot, is the gap a missing tool or a missing definition?

Sign up for Newsletter