Tag Archives: UK FS SCD2 Bronze

Advanced SCD2 Optimisation Techniques for Mature Data Platforms

Leave a Reply

Advanced SCD2 optimisation techniques are essential for mature Financial Services data platforms, where historical accuracy, regulatory traceability, and scale demands exceed the limits of basic SCD2 patterns. Attribute-level SCD2 significantly reduces storage and computation by tracking changes per column rather than per row. Hybrid SCD2 pipelines, combining lightweight delta logs with periodic MERGEs into the main Bronze table, minimise write amplification and improve reliability. Hash-based and probabilistic change detection eliminate unnecessary updates and accelerate temporal comparison at scale. Together, these techniques enable high-performance, audit-grade SCD2 in platforms such as Databricks, Snowflake, BigQuery, Iceberg, and Hudi, supporting the long-term data lineage and reconstruction needs of regulated UK Financial Services institutions.

Continue reading →

Scaling the SCD2 Bronze Layer: Practical Strategies for Financial Services

Leave a Reply

A rapidly growing SCD2 Bronze layer is an expected outcome of implementing full historical tracking across financial services data platforms, where high-frequency attribute changes, noisy upstream systems, and long regulatory retention periods contribute to rapid data growth. This article outlines practical engineering strategies to keep SCD2 Bronze efficient and cost-effective, including effective partitioning, change-suppression logic, hot–warm–cold tiering, compaction and lifecycle optimisation, windowed processing, metadata offloading, and upstream data contracts. Advanced approaches, such as attribute-level SCD2, hybrid SCD2/delta-merge patterns, and hash-based change detection, further enhance scalability for mature platforms. The piece concludes that SCD2 growth is not a failure but a natural result of robust governance. With the right architecture and operational discipline, organisations can maintain an auditable, scalable, and regulator-aligned Bronze layer ready for analytics, AI, and Data Mesh.

Continue reading →

Using SCD2 in the Bronze Layer with a Non-SCD2 Silver Layer: A Modern Data Architecture Pattern for UK Financial Services

Leave a Reply

UK Financial Services firms increasingly implement SCD2 history in the Bronze layer while providing simplified, non-SCD2 current-state views in the Silver layer. This pattern preserves full historical auditability for FCA/PRA compliance and regulatory forensics, while delivering cleaner, faster, easier-to-use datasets for analytics, BI, and data science. It separates “truth” from “insight,” improves governance, supports Data Mesh models, reduces duplicated logic, and enables deterministic rebuilds across the lakehouse. In regulated UK Financial Services today, it is the only pattern I have seen that satisfies the full, real-world constraint set with no material trade-offs.

Continue reading →

WTF Is SCD? A Practical Guide to Slowly Changing Dimensions

Leave a Reply

Slowly Changing Dimensions (SCDs) are how data systems manage attributes that evolve without constantly rewriting history. They determine whether you keep only the latest value, preserve full historical versions, or maintain a limited snapshot of changes. The classic SCD types (0–3, plus hybrids) define different behaviours… from never updating values, to overwriting them, to keeping every version with timestamps. The real purpose of SCDs is to make an explicit choice about how truth should behave in your analytics: what should remain fixed, what should update, and what historical context matters. Modern data platforms make tracking changes easy, but they don’t make the design decisions for you. SCDs are ultimately the backbone of reliable, temporal, reality-preserving analytics.

Continue reading →