From 200 rows to 40 million: streaming generation at scale

The same generator produces a 200-row preview for a unit test and an 800-million-row dataset for a load test. The interesting engineering question is: how do you build hundreds of millions of interrelated rows without needing hundreds of gigabytes of RAM? The answer is streaming — and a small trick for keeping the analytics honest along the way.

The memory problem

The naive approach holds every table in memory, then computes the downstream analytics from the full tables. That's fine at 200 orders. At 40 million orders — with order lines, shipments, shipment lines, and tracking events multiplying on top — the transactional tables alone won't fit. So instead of materializing everything at once, the pipeline generates the big tables in chunks, writes each chunk to disk as it's finalized, and frees the memory before moving on. Peak memory stays bounded no matter how many orders you ask for.

The analytics problem streaming creates

Here's the catch: a lot of the value is in the derived analytics — demand signals, forecast accuracy, customer lifetime value, DC throughput — and those normally need to scan the full transactional tables. If the full tables never exist in memory at once, how do you compute a customer's yearly revenue or an item's monthly demand?

You don't reload the data — you accumulate exactly the summaries the downstream needs as each chunk streams past, and never look back.

A streaming aggregator watches the chunks go by and keeps compact running statistics: item × month shipped quantities, customer × month revenue, DC × month throughput, carrier performance, and so on. When the transactional generation finishes, those aggregates — kilobytes, not gigabytes — feed the analytics modules. The heavy tables were never all in memory; the summaries were built incrementally.

Honest about the trade-offs

Streaming isn't free, and we don't pretend it is. A few metrics that need exact per-order attribution — certain promotional-lift and demand-signal breakdowns — are computed as well-founded estimates in streaming mode rather than exact per-line rollups, and the documentation says exactly which ones and why. At the dataset level — totals, averages, distributions — the streaming and non-streaming paths agree. The philosophy is to be precise about where an approximation lives instead of hiding it.

Write-through and reproducibility

Two more pieces make it production-grade. Write-through persists each table the moment it's finalized, so memory is freed continuously rather than at the end. And the whole thing stays deterministic: per-chunk random seeding is derived from the run seed and the chunk index, so a given seed and chunk size always produce identical output — the same dataset, byte for byte, every time. That's what lets a benchmark or a bug report reference "the seed-10250 dataset" and mean something exact.

The result

One codebase, one command, and a dial that goes from a laptop-sized preview to a warehouse-stressing production run — with the analytics computed correctly at both ends. Scale stops being a rewrite and becomes a parameter.