Tag Archives: streams and tasks

Managing a Rapidly Growing SCD2 Bronze Layer on Snowflake: Best Practices and Architectural Guidance

Slowly Changing Dimension Type 2 (SCD2) patterns are widely used in Snowflake-based Financial Services platforms to preserve full historical change for regulatory, analytical, and audit purposes. However, Snowflake’s architecture differs fundamentally from file-oriented lakehouse systems, requiring distinct design and operational choices. This article provides practical, production-focused guidance for operating large-scale SCD2 Bronze layers on Snowflake. It explains how to use Streams, Tasks, micro-partition behaviour, batching strategies, and cost-aware configuration to ensure predictable performance, controlled spend, and long-term readiness for analytics and AI workloads in regulated environments.

Continue reading

From SCD2 Bronze to a Non-SCD Silver Layer in Snowflake

This article explains a best-practice Snowflake pattern for transforming an SCD2 Bronze layer into a non-SCD Silver layer that exposes clean, current-state data. By retaining full historical truth in Bronze and using Streams, Tasks, and incremental MERGE logic, organisations can efficiently materialise one-row-per-entity Silver tables optimised for analytics. The approach simplifies governance, reduces cost, and delivers predictable performance for BI, ML, and regulatory reporting, while preserving complete auditability required in highly regulated financial services environments.

Continue reading