Tag Archives: Data Mesh

From SCD2 Bronze to a Non-SCD Silver Layer in Snowflake

This article explains a best-practice Snowflake pattern for transforming an SCD2 Bronze layer into a non-SCD Silver layer that exposes clean, current-state data. By retaining full historical truth in Bronze and using Streams, Tasks, and incremental MERGE logic, organisations can efficiently materialise one-row-per-entity Silver tables optimised for analytics. The approach simplifies governance, reduces cost, and delivers predictable performance for BI, ML, and regulatory reporting, while preserving complete auditability required in highly regulated financial services environments.

Continue reading

Using SCD2 in the Bronze Layer with a Non-SCD2 Silver Layer: A Modern Data Architecture Pattern for UK Financial Services

UK Financial Services firms increasingly implement SCD2 history in the Bronze layer while providing simplified, non-SCD2 current-state views in the Silver layer. This pattern preserves full historical auditability for FCA/PRA compliance and regulatory forensics, while delivering cleaner, faster, easier-to-use datasets for analytics, BI, and data science. It separates “truth” from “insight,” improves governance, supports Data Mesh models, reduces duplicated logic, and enables deterministic rebuilds across the lakehouse. In regulated UK Financial Services today, it is the only pattern I have seen that satisfies the full, real-world constraint set with no material trade-offs.

Continue reading

MapReduce: A 20-Year Retrospective on How Jeffrey Dean and Sanjay Ghemawat Revolutionised Data Processing

This article provides a retrospective on the 20th anniversary of Jeffrey Dean and Sanjay Ghemawat’s seminal paper, “MapReduce: Simplified Data Processing on Large Clusters”. It explores the paper’s lasting impact on data processing, its influence on the development of big data technologies like Hadoop, and its broader implications for industries ranging from digital advertising to healthcare. The article also looks ahead to future trends in data processing, including stream processing and AI, emphasising how MapReduce’s principles will continue to shape the future of distributed computing.

Continue reading