Scaling Data Platforms, Part 3: Observability You'll Actually Use

Dashboards nobody opens are just expensive wallpaper. Part 3 is about signals that change behavior.

Instrument the four questions

1. Is the data fresh?

2. Is the data complete?

3. Is the data correct?

4. Who is affected if it is not?

Export freshness as a first-class metric so you can alert on data_freshness_seconds directly.

groups:
 - name: data-freshness
 rules:
 - alert: OrdersStale
 expr: data_freshness_seconds{dataset="orders_enriched"} > 1800
 for: 10m
 labels: { severity: page }
 annotations:
 summary: orders_enriched is stale

Find the gaps with SQL

A cheap completeness check that catches most incidents:

SELECT date_trunc('hour', created_at) AS hr,
 count(*) AS rows
FROM orders_enriched
WHERE created_at > now() - interval '24 hours'
GROUP BY 1
HAVING count(*) < 100 -- expected floor per hour
ORDER BY 1;

The best alert is one an on-call engineer thanks you for.

Make it human

• Every alert links to a runbook

• Every dataset has an owner in the page

• Silence noisy alerts ruthlessly, trust is the asset

That wraps the series. Foundations, streaming, and observability: the unglamorous work that lets you sleep.