Scaling Data Platforms, Part 1: Why Foundations Beat Heroics

Every data team eventually hits the wall where heroics, the 2am pipeline rescue, the one engineer who "just knows", stop scaling. This series is about the boring foundations that make heroics unnecessary.

The problem with ad-hoc pipelines

Ad-hoc pipelines optimize for the first delivery, not the hundredth. They accumulate silent coupling: a column rename here, a cron job there, and suddenly nobody can change anything without breaking three dashboards.

• No contracts between producers and consumers

• Schema drift discovered in production

• Backfills that take days and nobody trusts

• Ownership that lives in #data-help instead of code

A foundations checklist

1. Declare schemas as code and version them

2. Make every dataset idempotent and re-runnable

3. Add data quality gates before, not after, publish

4. Assign a named owner to every table

If your recovery plan is a person, you do not have a recovery plan.

Start with schema contracts. Here is the shape we standardize on, see the full spec in the OpenLineage docs:

dataset: orders_enriched
owner: data-platform
schema:
 - name: order_id
 type: string
 nullable: false
 - name: total_cents
 type: integer
sla:
 freshness_minutes: 30

Next up: Part 2 tackles streaming ingestion with Kafka, where foundations matter even more.