Tag Archives: Change Data Capture

Operational Edge Projection + CDC Write-Back: A Minimal Pattern for Low-Latency Authority Without Breaking Temporal Truth

This article introduces a practical operational extension to the UK FS SCD2 Bronze/Silver architecture. Rather than forcing operational applications to interact directly with the lakehouse, governed Silver current-state views are projected into a low-latency SQL database for operational consumption. Business decisions are made and stored locally, captured through SQL Change Data Capture (CDC), landed append-only into Bronze, and subsequently reconciled into governed Silver. The pattern preserves temporal correctness, auditability, replayability, and regulatory defensibility while providing the transactional performance and simplicity required by operational systems. It separates operational authority from institutional memory, allowing each layer to optimise for its intended purpose without compromising governance.

Continue reading

Event-Driven CDC to Correct SCD2 Bronze in 2025–2026

Broken history often stays hidden until remediation or skilled-person reviews. Why? Event-driven Change Data Capture fundamentally changes how history behaves in a data platform. When Financial Services organisations move from batch ingestion to streaming CDC, long-standing SCD2 assumptions quietly break — often without immediate symptoms. Late, duplicated, partial, or out-of-order events can silently corrupt Bronze history and undermine regulatory confidence. This article sets out what “correct” SCD2 means in a streaming world, why most implementations fail, and how to design Bronze pipelines that remain temporally accurate, replayable, and defensible under PRA/FCA scrutiny in 2025–2026.

Continue reading