Building a Real-Time Data Pipeline Architecture That Doesn't Collapse
Build a resilient real-time data pipeline architecture. This guide covers patterns, tech stacks, and a modernization roadmap for senior engineering leaders.
On this page
- Overview
- Why Your Batch ETL Is a Modernization Bottleneck
- Batch vs. Real-Time Pipeline Decision Matrix
- Assessing Your Architectural Debt
- Anatomy of a Modern Real-Time Pipeline
- Stream Ingestion and Messaging
- Stream Processing
- Real-Time Datastore and Serving Layer
- Choosing Your Architectural Pattern: Lambda vs. Kappa
- The Case for Kappa Architecture
- When Lambda Still Makes Sense
- Change Data Capture as a Modernization Starting Point
- Achieving Operational Excellence in Real-Time Systems
- Building for Fault Tolerance
- The Observability Mandate
- Real-Time Pipeline Observability Checklist
- Your Phased Modernization Roadmap
- Step 1: Secure a High-Impact Pilot Project
- Step 2: Define and Measure Success with KPIs
- Step 3: Assemble Your Core Team and Select Partners
- Step 4: Make the Build vs. Buy Decision
- Next Steps: Address Your Immediate Technical Hurdles
- How Do You Handle Schema Evolution in a Live Pipeline?
- What Is the Best Strategy for Reprocessing Historical Data?
Legacy batch ETL pipelines often hold back modern, AI-driven applications. The ‘store-then-analyze’ model, built for historical reporting, introduces data latency that makes real-time fraud detection, dynamic pricing, and responsive supply chains hard to achieve. Data that is hours old is of little use for immediate decisions.
The limitation is architectural, not a matter of tuning. Batch jobs create engineered data downtime. A pipeline failure discovered hours late means your dashboards and AI models operate on stale, untrustworthy data, capping your business's ability to respond and introducing significant operational risk. Migrating from batch to real-time can cut data latency from hours to seconds or less, but it requires a fundamental architectural shift, not just faster batch jobs.
Why Your Batch ETL Is a Modernization Bottleneck
Modernization means replacing the batch model with continuous processing, not speeding up batch jobs. The value of data decays rapidly; a real-time data pipeline architecture aims to process data and make it actionable as it is generated.
- Operational Agility: Real-time pipelines enable immediate response. A fraudulent transaction can be blocked in milliseconds instead of being discovered hours later after the money is gone.
- Enhanced Customer Experience: Systems can react to user behavior within seconds, offering relevant recommendations or proactive support while the customer is still there.
- AI and ML Efficacy: Machine learning models fed by real-time data streams can make more timely predictions. For features that depend on fresh signals, a model using day-old data is a step behind reality.
In our view, the fragility of legacy systems shows in how long problems go unnoticed. A batch pipeline that fails overnight may not be noticed until the morning reports are wrong, and AI-enabled applications that depend on fresh data make that delay harder to tolerate.
Before committing to modernizing your data platform, the trade-offs between sticking with batch processing and migrating to a real-time architecture must be clear. This matrix compares the two on the attributes that directly affect business outcomes.
Batch vs. Real-Time Pipeline Decision Matrix
| Attribute | Batch Architecture (Legacy) | Real-Time Architecture (Modern) |
|---|---|---|
| Data Latency | Hours to days | Typically sub-second to seconds |
| Business Use Cases | Historical reporting, BI dashboards | Fraud detection, personalization, IoT monitoring, dynamic pricing |
| Decision-Making | Strategic (long-term trends) | Tactical & Operational (immediate action) |
| Failure Impact | Often high. Hours of reprocessing may be required. | Often lower. Recovery from the last checkpoint limits data loss. |
| Infrastructure Cost | Lower compute cost (runs periodically) | Higher compute cost (always on) |
| Complexity | Simpler logic, complex orchestration | Complex state management, simpler orchestration |
The values are typical ranges and our judgement, not measurements. The comparison clarifies that while batch processing has a role for non-urgent tasks, it is a poor fit for any application that depends on fresh data.
Assessing Your Architectural Debt
Answering "yes" to any of the following questions may indicate you are accumulating architectural debt that will constrain future growth:
- High Data Latency: Does it take hours for operational data to reach your analytics and AI models?
- Slow Incident Response: When a supply chain disruption occurs, are your teams forced to react using outdated data?
- Inability to Support New Use Cases: Are you unable to build features like live inventory tracking or instant personalization because your data platform cannot handle them?
Answering these questions is the first step toward a successful data modernization initiative.
Anatomy of a Modern Real-Time Pipeline
A real-time data pipeline is a high-performance system in which every component has to keep up with a tight latency target, often under a second. A failure in any one stage can turn the promise of fresh data into another broken SLA.
Batch systems force you to wait. A real-time architecture feeds AI and applications directly, turning data into action the moment it’s generated. It needs the following components.
Stream Ingestion and Messaging
This is the "first mile" where data freshness is won or lost. The goal is to capture events from sources like application logs, IoT sensors, or database change streams instantly. For unstructured sources, AI-powered data extraction engines impose order at the entry point.
Captured events are handed off to an event streaming platform. This is the backbone of your architecture, decoupling producers from consumers. Durable log retention and partitioned topics let consumers replay or catch up after a failure instead of dropping events when a downstream service is unavailable.
- Ingestion Tools: Kafka Connect or Fluentd act as reliable collectors, funneling data into the pipeline.
- Messaging Platforms: Apache Kafka is the most widely used platform for high-throughput, fault-tolerant event streaming. Apache Pulsar is a strong alternative, offering built-in multi-tenancy and tiered storage.
The trade-off is control versus cost. Self-hosting Kafka provides ultimate control but brings significant operational overhead. Managed services like Confluent Cloud take on much of this overhead for a premium.
Stream Processing
Events arrive as raw payloads from the messaging platform. A stream processing engine applies logic on the fly: enriching fields, calculating fraud scores, or aggregating metrics before data reaches the serving layer.
The stream processor computes results (enriched fields, fraud scores, aggregates) while the data is still moving, rather than only moving it quickly.
Your choice of engine dictates the pipeline's latency and power. Apache Flink is built for stateful, low-latency stream processing and supports exactly-once state consistency, which makes it a common choice for use cases demanding exactly-once processing guarantees. Spark Structured Streaming processes streams as micro-batches by default, with end-to-end latencies as low as 100 milliseconds according to the Spark documentation, which suits tasks that can tolerate some added latency.
Real-Time Datastore and Serving Layer
Processed data requires a home, but a traditional batch-oriented data warehouse is often the wrong address. Warehouses are built for large-scale analytical queries, not the rapid-fire, low-latency lookups that real-time applications demand.
This is the job of a real-time datastore, engineered to absorb a constant firehose of incoming data and serve queries in sub-second time.
| Datastore Type | Key Characteristics | Common Use Cases | Example Tools |
|---|---|---|---|
| In-Memory | Very low latency, in-memory storage | Caching, session stores, leaderboards | Redis, Hazelcast |
| Columnar Analytics | High-throughput ingestion, fast analytical queries | Real-time dashboards, operational monitoring | Apache Druid, ClickHouse, Apache Pinot |
| Search-Based | Powerful text search and aggregations | Log analytics, application performance monitoring (APM) | Elasticsearch, OpenSearch |
This final layer serves the processed data to a live dashboard, microservice, or alerting system.
Choosing Your Architectural Pattern: Lambda vs. Kappa
The choice between Lambda and Kappa is a foundational decision that locks in operational costs and engineering complexity for years. Get this wrong, and you commit your team to chasing data discrepancies between two systems that should have been one.
The Lambda architecture delivers both fast, real-time views and comprehensive batch views by running two separate data paths in parallel. In 2014, Jay Kreps argued in Questioning the Lambda Architecture that once a stream processor can reliably replay history from a log, the separate batch layer becomes redundant. In our view, the dual-path system can be a sign of technical debt, forcing engineers to build and maintain two sets of logic: one for the "speed layer" (streaming) and another for the "batch layer." This duplication taxes productivity and is a common source of bugs where real-time dashboards and quarterly reports disagree.
The Case for Kappa Architecture
The Kappa architecture eliminates this duality. Its core principle is to use a single, unified stream processing engine for everything. Real-time dashboards, historical reprocessing, and batch-style analytics all run through one pipeline. This is enabled by mature tools like Apache Flink and durable event logs like Apache Kafka.
In a Kappa architecture, your streaming system is your batch system. There is no batch layer. Reprocessing historical data simply means replaying events from your durable message queue through the same stream processing logic.
For any new real-time data pipeline architecture, this unified model reduces complexity by eliminating the second codebase. Maintenance becomes easier, and the risk of real-time and batch views drifting apart is reduced. For most new projects, we would start with Kappa.
When Lambda Still Makes Sense
Although Kappa is simpler, Lambda remains a pragmatic choice when constrained by legacy systems or existing team skills.
- Existing Batch Infrastructure: If your company depends on a massive data lake or warehouse with years of established batch ETL, a "rip-and-replace" is not feasible. Lambda allows you to add a real-time speed layer without disrupting the existing batch ecosystem.
- Team Skill Set Mismatch: If your data team is composed of SQL experts skilled in tools like dbt but lacks deep stream processing expertise, Lambda provides a smoother on-ramp. They can continue using SQL for the batch layer while a smaller team builds out the streaming path.
This is relevant for modernization projects replacing legacy ETL with an ELT approach, often using dbt for SQL-based transformations in a lakehouse. In our view this can cut transformation overhead, and formats like Delta Lake, which describes itself as one format to unify ETL, data warehouse, and ML in a lakehouse, can help unify batch and streaming.
Change Data Capture as a Modernization Starting Point
Change Data Capture (CDC) is a pattern for enabling a real-time architecture. CDC tools like Debezium tap into the transaction logs of legacy databases (e.g., Oracle, PostgreSQL) and stream every INSERT, UPDATE, and DELETE as an event into your message queue.
This turns a monolithic database from a data silo into a live event source without modifying the legacy application's code. It is a non-invasive, high-impact first step for many enterprise modernization efforts, feeding directly into either a Lambda or Kappa architecture. The choice then depends on whether you process this CDC stream alongside an existing batch system (Lambda) or make it the single source of truth for a new, unified pipeline (Kappa).
Achieving Operational Excellence in Real-Time Systems
A real-time data pipeline in production demands constant operational discipline to deliver its low-latency promise. A pipeline is only as valuable as it is trustworthy, and without operational discipline you are just generating bad data faster. The aim is a system that anticipates and survives the failures inherent in distributed computing, through checkpointed state, controlled backpressure, and replayable event logs.
Building for Fault Tolerance
Fault tolerance is required of any production-ready real-time system. You must design with the assumption that components will fail, networks will partition, and load will spike.
A critical strategy is stateful checkpointing. Stream processing engines like Apache Flink periodically save their internal state (e.g., windowed aggregations) to durable storage. If a job fails, Flink resumes from a checkpoint by restoring operator state and replaying records from that point, which avoids data loss and limits reprocessing time.
Managing backpressure is also required. When a downstream system cannot keep up, a well-designed pipeline automatically slows down data ingestion at the source to prevent overwhelming the slower component and causing cascading failures.
The goal is exactly-once processing semantics: each event affects the pipeline's state once and only once, even with failures and retries. Flink's checkpointing provides this for state inside the job. Whether delivery is exactly-once end to end also depends on the source and sink connectors, and each connector's documentation states what it guarantees. This matters most for financial transactions or any system where accuracy is absolute.
The Observability Mandate
For a real-time pipeline, observability extends beyond CPU and memory to metrics that map directly to the health and performance of the data flow. Ignoring this leads to "silent failures," where the pipeline appears healthy but is delivering stale or incomplete data for hours.
Track end-to-end latency, consumer lag, and schema drift alongside host metrics so you catch stale or partial data while dashboards still look green. Thinking about AI Architecture for Longevity from day one helps keep your pipelines manageable for years, not just fast. Our DevOps & Platform Modernization hub covers the wider tooling and processes around observability.
The following checklist is a starting point for assessing the operational readiness of your pipeline.
Real-Time Pipeline Observability Checklist
| Metric Category | Key Metrics to Track | Example Thresholds | Tooling Examples |
|---|---|---|---|
| Data Freshness | End-to-End Latency | <1 second for critical paths | OpenTelemetry, custom timestamps |
| Throughput | Events Processed Per Second | Varies by use case; watch for sudden drops | Prometheus, Grafana, Datadog |
| Consumer Health | Consumer Lag (e.g., Kafka Lag) | <1000 messages; should not grow over time | Burrow, Kafka-specific monitoring tools |
| System Correctness | Watermark Gaps, Late Events | Monitor count of late-arriving data | Flink/Spark UI, custom metrics |
| Schema Integrity | Schema Drift Alerts | Zero unexpected changes | Confluent Schema Registry, Apicurio |
| Resource Usage | Backpressure Events, Checkpoint Failures | Zero sustained backpressure; <1% failure rate | Flink Dashboard, Prometheus alerts |
The thresholds are starting points we suggest, not standards, so set yours from your own SLAs. This checklist is a core part of your design and testing phases. In our view, if you cannot measure these metrics, you do not yet have a production-ready system.
Your Phased Modernization Roadmap
A real-time data pipeline architecture needs a concrete execution plan. Avoid a "big bang" migration, which tends to produce budget overruns and project failure. A phased approach builds momentum and reduces risk.
Step 1: Secure a High-Impact Pilot Project
Your first move is to identify a high-impact, low-risk pilot project. The pilot proves the value of real-time data to skeptical stakeholders.
Good candidates for a pilot are self-contained and deliver obvious value:
- A real-time dashboard for a critical function like order fulfillment.
- An MVP fraud detection system that flags transactions for manual review.
- A live inventory tracking system for one high-value product line.
Step 2: Define and Measure Success with KPIs
Once you have a pilot, define success with clear, measurable targets tied to a business outcome (latency cut, fraud loss reduction, inventory accuracy).
Success metrics must be specific. Aim for targets like "reduce fraud-related losses by 15% within Q3 by cutting detection latency from 10 minutes to under 500 milliseconds." This forces a direct link between technical execution and financial impact.
These metrics anchor the project, justify the investment, and guide architectural trade-offs.
Step 3: Assemble Your Core Team and Select Partners
With a defined project and clear goals, assemble a small, dedicated team with a mix of skills: data engineers with streaming expertise (e.g., Flink or Kafka Streams), backend developers, and a product owner who understands the business problem.
Follow these implementation practices: asynchronous communication, idempotent consumers, and state management for late-arriving data. This discipline reduces the risk of pipelines that fail silently or produce duplicates. You can find more best practices for data integration on Domo.com.
Step 4: Make the Build vs. Buy Decision
The next decision is whether to build a custom platform or buy a managed service. This is a strategic choice that dictates your team's focus and long-term operational burden. The "build" route offers total control but carries the complexity of running stateful, distributed systems like Kafka and Flink at production scale, which in our judgement is the main project risk.
The decision to build a real-time platform from scratch is effectively a decision to become a data infrastructure company. If that is not your core business, you are likely better off buying.
Use this framework to guide the decision. As a rule of thumb, lean toward buying if you answer "yes" to more than two of these questions:
- Speed to Market: Is a solution in under three months more valuable than total infrastructure control?
- Core Competency: Is your engineering team's mission to build customer products, not manage data plumbing?
- Talent Scarcity: Do you lack senior-level, in-house experts with deep knowledge of streaming technologies?
- Operational Overhead: Do you want to avoid the 24/7 pager duty of managing a complex distributed system?
Managed services like Confluent Cloud, Tinybird, or Decodable (whose site now says it has been acquired by Redis) let your team focus on business logic, not infrastructure. In our judgement, this path trades some control for lower operational risk and faster implementation, while cost predictability depends on the vendor's pricing model. Streaming ingestion in warehouses (Snowpipe Streaming and BigQuery's Storage Write API) and SQL stream processing such as ksqlDB are other options on the buy side.
Next Steps: Address Your Immediate Technical Hurdles
Production is messier than theory. The sections below cover schema evolution and historical reprocessing after launch.
How Do You Handle Schema Evolution in a Live Pipeline?
Changing a data schema mid-flight without breaking consumers is a primary challenge. A reliable method is a centralized schema registry, like those from Confluent or Apicurio. It acts as the single source of truth for your data's structure.
When a producer attempts to send a message with a new schema, it is first validated against the registry. This is where you enforce compatibility rules, and BACKWARD compatibility is the default in Confluent Schema Registry, which prefers it for Kafka because you can rewind consumers to the start of a topic.
- Backward Compatibility: Consumers using the new schema can read data written with the previous schema. Confluent's guidance is to upgrade consumers before producers start writing the new schema. For Avro, this typically means you can add optional fields (with defaults) or delete fields.
- Forward Compatibility: The reverse. Consumers still using the old schema can read data written with the new one, so producers can be upgraded first. Confluent notes this mode is harder to work with, because you have to anticipate future changes.
- Full Compatibility: The schema is both backward and forward compatible, which is often overkill.
Enforcing these rules programmatically can stop incompatible "poison pill" messages at the source, which makes it much easier to evolve schemas without downtime.
What Is the Best Strategy for Reprocessing Historical Data?
Sooner or later, you will update business logic and need to re-run it against historical data. In a Kappa architecture, this process is simple because there is no separate batch layer to manage.
The strategy is simple: treat your historical data as just another stream. Do not touch your live job. Instead, deploy a new, parallel instance of your stream processing application.
Configure this new job to read from the beginning of your event log (e.g., offset 0 in a Kafka topic). It will churn through all historical events using the new logic, writing its corrected output to a new table or topic.
Once this reprocessing job catches up to the live stream, perform an atomic switch, pointing downstream applications to the new, corrected data source. Then, shut down the old job and its output. This pattern is repeatable and avoids maintaining two different codebases for batch and streaming.
Chief Analyst, Software Modernization Intelligence · 10+ years B2B market research
Published: ·Updated: