Article
Batch ETL vs real-time data replication: how to choose
2026-07-29 · /data-architecture-and-performance/batch-etl-vs-real-time-data-replication/
Data integration architectures are changing as enterprises move beyond rigid, schedule-based data pipelines. While traditional batch ETL and log-based Change Data Capture (CDC) address different operational needs, modern data platforms like Gluesync now unify both approaches, enabling teams to execute real-time replication or federated querying through a single integration framework.
Data integration strategy summary
Traditional batch ETL relies on scheduled SQL queries that risk source database performance and introduce data lag. Log-based CDC offers sub-45ms real-time streaming without query polling. With the evolution of unified data platforms, organizations no longer need to compromise: they can deploy log-based CDC for continuous replication or leverage federated query engines (such as Gluesync Query Forge) for on-demand cross-database analytics without moving raw data.
How traditional batch ETL works
Periodic query-based extraction
Batch ETL executes scheduled SQL queries every few minutes, hours, or days. Incremental jobs usually rely on timestamp columns or primary keys, while some workloads still require full table scans to identify changes. As databases grow, extraction windows become longer and increasingly difficult to fit within maintenance schedules.
Impact on the source database
Every extraction query competes directly with production workloads for infrastructure resources. Repeated polling generates additional CPU usage, memory consumption, disk I/O, and network traffic. Depending on the database engine, heavy extraction jobs can trigger row or table locking and query contention.
Why batch ETL creates stale data
Batch processing introduces an inherent delay between a committed transaction and its availability downstream. Depending on the schedule, data may remain outdated for hours or days, creating data drift and forcing operational decisions on historical snapshots rather than current business events.
How log-based CDC works
Reading transaction logs instead of tables
Log-based CDC captures changes by parsing the database transaction log, where the core engine records every committed transaction, rather than querying application tables. Examples include:
- PostgreSQL: Write-Ahead Log (WAL)
- Oracle: Redo Logs
- SQL Server: Transaction Log
- MongoDB: Oplog and Change Streams
- IBM i: Journal (QSQJRN)
- MySQL: Binary Log (Binlog)
Because ongoing change capture parses these existing logs, it avoids recurring SELECT queries against live business tables.
Continuous replication and sub-45ms latency
Instead of waiting for scheduled extraction windows, committed changes are parsed in commit order and converted into a continuous stream of inserts, updates and deletes. Real-time engines leverage log-based CDC to achieve sub-45 ms latency under supported environments, allowing operational systems, analytics platforms and downstream applications to consume transactional changes immediately after commit.
Minimal source-database overhead
Log-based CDC performs no recurring polling, requires no tracking indexes on production tables and avoids full table scans during replication. However, system architects must account for log retention, storage allocation, and network transmission overhead.
Technical comparison matrix
| Evaluation criteria | Traditional batch ETL | Log-based CDC replication |
|---|---|---|
| Architecture approach | Scheduled query polling and periodic table extraction | Passive transaction log parsing and continuous streaming |
| Latency | Hours or days depending on schedule | Sub-45ms latency under supported conditions |
| Impact on source database | CPU spikes, increased I/O, memory pressure and query contention | Minimal source-database overhead; no recurring extraction queries |
| Schema evolution handling | Schema modifications may break extraction pipelines | Automated DDL propagation where supported |
| Compute & network costs | Large recurring transfers and expensive extraction workloads | Incremental streaming of changed records, reducing peak network bursts |
| Failover & consistency | Risk of partial loads and duplicate processing after failures | Checkpoints (LSN, SCN, offsets) support resumable processing |
Operational trade-offs of log-based CDC
While log-based CDC addresses the latency and performance penalties of batch ETL, it introduces specific engineering considerations:
- Log management and storage: Transaction logs must be retained long enough for the CDC engine to process them. High write volumes or replication halts can cause log accumulation.
- Initial snapshot complexity: Replicating large multi-terabyte databases requires an initial historical load before streaming CDC deltas.
- Target idempotency: Downstream targets must handle retries and failover events cleanly using durable LSN or SCN checkpoint positions.
Beyond binary choices: the unified data integration approach
Modern enterprise integration is no longer a binary choice between pure batch ETL and continuous streaming. Emerging data governance frameworks treat replication, automation, and querying as unified capabilities within a single toolbox.
When continuous physical movement of data is required, log-based CDC streams transaction logs to target data warehouses or event hubs. However, when analytical systems or operational dashboards only need on-demand access to distributed data, physical data movement can be avoided altogether.
For example, platforms like Gluesync (v2.2.10+) combine native real-time log replication with federated SQL capabilities like Query Forge. Through a unified JDBC endpoint, engineers can execute cross-database SQL queries across isolated core hubs without building or maintaining traditional extraction pipelines. This allows architectures to blend low-latency log streaming with zero-copy federated queries under a single operational interface.
Decision framework for data architects
Choose batch or federated querying if:
- Data access is ad-hoc or query frequency is low.
- Source systems prohibit direct transaction log access or lack CDC APIs.
- You need to join remote datasets on demand without establishing physical replication pipelines (using federated engines like Query Forge).
Choose log-based CDC if:
- Operational decisions rely on sub-second data freshness (e.g., fraud detection, real-time inventory, microservice synchronization).
- Production databases suffer performance degradation from recurring SELECT polling or full table scans.
- You are feeding cloud data platforms (Snowflake, BigQuery, Databricks) that benefit from continuous incremental delta updates.
To explore supported database topologies, query pushdown behavior and setup guides for Query Forge, consult the official Gluesync Documentation.
Frequently asked questions
Is log-based CDC faster than batch ETL?
Yes. Batch ETL relies on scheduled extraction windows, introducing latencies ranging from minutes to days. Log-based CDC captures committed changes directly from transaction logs as they occur, enabling sub-45ms continuous streaming.
How does Query Forge differ from physical replication?
Physical replication uses the Gluesync CDC engine to copy data continuously from source to target. Query Forge acts as a federated SQL endpoint, executing remote queries directly against agents without moving underlying datasets.
Does log-based CDC eliminate source database impact?
Log-based CDC significantly reduces source overhead compared to batch ETL. By reading transaction logs rather than executing polling SELECT queries or full table scans, it eliminates query competition on live application tables.
Can log-based CDC handle schema changes automatically?
Many modern CDC engines capture and propagate supported Data Definition Language (DDL) modifications automatically, reducing manual pipeline maintenance during schema evolution.