Scaling Data Platforms, Part 2: Streaming Ingestion with Kafka

Batch gets you correctness; streaming gets you freshness. In Part 2 we wire up a Kafka ingestion path that is safe to operate at 3am.

Create a topic with sane defaults

Set partitions and retention deliberately, defaults are rarely right for production.

kafka-topics.sh --create \
 --topic orders.v1 \
 --partitions 12 \
 --replication-factor 3 \
 --config retention.ms=604800000 \
 --config min.insync.replicas=2

A consumer that will not lose data

Disable auto-commit and commit after the side effect succeeds, not before.

import { Kafka } from 'kafkajs';

const kafka = new Kafka({ brokers: ['broker:9092'] });
const consumer = kafka.consumer({ groupId: 'orders-sink' });

await consumer.connect();
await consumer.subscribe({ topic: 'orders.v1', fromBeginning: false });

await consumer.run({
 autoCommit: false,
 eachMessage: async ({ topic, partition, message }) => {
 await writeToWarehouse(message.value); // do the work first
 await consumer.commitOffsets([{ topic, partition,
 offset: (Number(message.offset) + 1).toString() }]);
 },
});

The pieces fit together like this:

Operational guardrails

• Alert on consumer lag, not just broker health

• Use idempotent writes so replays are safe

• Keep a dead-letter topic for poison messages

A stream you cannot replay is a stream you cannot trust.

Next up: Part 3 makes all of this observable.