Real-time or batch? Choosing the right data architecture
Streaming is not automatically better than batch. A practical way to decide how fresh your data really needs to be, and to avoid paying for latency you never use.
Real-time has a gravitational pull. It sounds modern, it demos well, and no one ever got criticised in a meeting for wanting data faster. But streaming architectures cost more to build, more to run and considerably more to operate, and a great deal of the streaming infrastructure in the world is delivering data in seconds to decisions that are made once a day.
Start from the decision, not the technology
The only question that matters is: how stale can this data be before the decision it feeds gets worse? Answer that honestly, per use case, and the architecture usually chooses itself. A fraud check needs milliseconds. A daily revenue report does not need anything faster than daily. Most things sit somewhere sensible in between.
Nobody was ever criticised for asking for data faster, which is exactly why so much money is spent on latency no decision ever uses.
What real-time actually costs
Streaming is not just a different tool; it is a different operational commitment. State, ordering, exactly-once processing, backfills, replay, and 24/7 on-call, all of it becomes your problem the moment "overnight" is no longer good enough. That cost can be entirely worth paying. It should be a decision, not a default.
- For each dataset, name the decision it serves and the freshness that decision needs.
- Default to batch unless a real decision genuinely needs fresher data.
- Reserve streaming for cases where staleness measurably degrades the outcome.
- Consider micro-batch or incremental processing as a pragmatic middle ground.
- Design so a batch pipeline can be promoted to streaming later without a rewrite.
The pragmatic middle
The choice is rarely as stark as it sounds. Micro-batch and incremental processing deliver "fresh enough" for many use cases at a fraction of the operational weight of true streaming. Modern lakehouse tooling makes it possible to start batch and tighten latency later, provided you designed for that from the start.
When we design a data platform, we grade each flow by the freshness its decisions actually require, spend the complexity budget only where it earns its keep, and keep the architecture able to move from batch toward real-time as genuine needs emerge, not as a reflex.
