UK Financial Services firms increasingly implement SCD2 history in the Bronze layer while providing simplified, non-SCD2 current-state views in the Silver layer. This pattern preserves full historical auditability for FCA/PRA compliance and regulatory forensics, while delivering cleaner, faster, easier-to-use datasets for analytics, BI, and data science. It separates “truth” from “insight,” improves governance, supports Data Mesh models, reduces duplicated logic, and enables deterministic rebuilds across the lakehouse. In regulated UK Financial Services today, it is the only pattern I have seen that satisfies the full, real-world constraint set with no material trade-offs.
Executive Summary (TL;DR)
UK Financial Services organisations increasingly adopt a “SCD2 in Bronze, simplified non-SCD2 in Silver” data architecture pattern to meet regulatory demands while enabling fast, clean analytics.
- Bronze Layer = Full SCD2 History
The Bronze layer preserves complete, immutable historical records for audit, risk, FCA/PRA compliance, SMCR accountability, AML/KYC investigations, fraud analysis, and customer remediation. It becomes the authoritative source of temporal truth. - Silver Layer = Clean, Current, Simplified Views
Analysts, BI developers, and data scientists consume deduplicated, latest-record datasets without SCD2 complexity. This improves performance, usability, and stability. - Why This Matters Now
No other publicly documented pattern simultaneously delivers regulatory-grade point-in-time reconstruction, deterministic rebuildability, domain-oriented governance, and high-performance analytics without compromise. By applying SCD2 once, and only once, in Bronze, the architecture eliminates conflicting versions of history, removes duplicated temporal logic, and creates a clean separation between “what we knew and when” (Bronze) and “what we need now” (Silver). - Outcome
The pattern is rapidly becoming the de facto standard across UK banks, insurers, and asset managers because it is the only known approach that satisfies the full set of real-world regulatory, operational, and architectural constraints at the same time.
Contents
1. Introduction
Financial Services organisations in the UK must operate under exceptionally demanding regulatory and governance environments. The FCA, PRA, SMCR, Consumer Duty, Operational Resilience, and multiple reporting obligations (IFRS, Basel frameworks, AML/KYC oversight) all require the ability to reconstruct historical data precisely as it existed at any point in time.
At the same time, data platforms must support:
- real-time analytics
- quants and actuaries working east/west in safe sandboxes
- operational and management reporting
- machine learning and feature pipelines
- modern data product operating models
- governed, secure self-service
These competing pressures require a layered architecture in which Bronze carries the full temporal record, while Silver provides the clean, current-state view required for analytics and operational use.
The now-dominant pattern is:
Store full SCD2 history in the Bronze layer. Present a clean, simplified current-state view in the Silver layer.
This article explains how and why this pattern works, and lays the foundation for the higher-order Gold and Platinum layers described later in the series. Gold represents business-contextualised data, KPIs, MI datasets, and domain-specific derived logic.
Platinum represents the enterprise semantic layer, a conceptual model that unifies meaning across domains, systems, and analytical viewpoints.
2. Understanding the Pattern
Before exploring the technical mechanics or the regulatory drivers behind this architecture, it is essential to understand the foundational pattern itself. Bronze and Silver layers exist for fundamentally different purposes: Bronze retains complete historical fidelity, while Silver offers a streamlined present-state view. Understanding this separation of roles and why each layer must behave differently sets the stage for all downstream architectural choices, from governance and analytics to domain modelling and operational integration.
2.1 Bronze Layer: SCD2 as the Authoritative Audit-Complete Record
The Bronze layer has evolved far beyond its old role as a raw landing zone.
In modern UK FS architectures, Bronze is the canonical historical truth.
Bronze with SCD2 provides:
- Full historical lineage
- Effective-dating (from/to timestamps)
- Surrogate keys for temporal versioning
- Immutable audit artefacts
- Point-in-time reconstruction
- Historical replay for quants/actuaries
- Evidence for AML/KYC investigations
- Customer remediation forensic trails
In short:
Bronze answers: “What did we know, and when did we know it?”
2.2 Silver Layer: A Non-SCD2 Simplification
The Silver layer typically serves analytical teams, operational BI, or data science. These users increasingly require:
- Current-state datasets
- Easily joinable dimensions
- High performance
- Lower cognitive load
- Schema stability
Thus the Silver layer exposes flattened, deduplicated, non-SCD2 views derived from the Bronze SCD2 tables.
These might be:
- Latest-record-only views
- Valid-time snapshots (e.g., “as of yesterday”)
- Soft-SCD0 or SCD1-equivalent simplified layers
- Narrowed, domain-specific tables aligned with Data Mesh or Data Products
3. Why This Pattern Works in UK Financial Services
With the pattern established, the next step is exploring why it has become the dominant approach across UK Financial Services. The answer lies at the intersection of regulation, operational efficiency, and architectural clarity. No other sector combines such intense regulatory scrutiny with such high analytical demand. The Bronze/Silver separation directly addresses FS-specific challenges around lineage, audit, risk modelling, data reuse, and domain ownership. Understanding these drivers reveals why this model is not merely fashionable but structurally necessary.
3.1 Regulatory Alignment
Financial Services firms must maintain robust historical data.
The Bronze layer with SCD2 satisfies:
- FCA Handbook SYSC requires accurate record-keeping
- PRA policies demand historical tracing of risk-related data.
- SMCR accountability regimes require defensible lineage.
- AML/KYC lineage
- FSCS reviews, customer complaints, and remediation depend on historical truth.
- Operational Resilience (aka scenario replay)
Silver satisfies:
- management information
- business intelligence
- exposure calculations
- credit/risk analytics
- ML pipelines
- operational systems consuming current-state data
By implementing SCD2 in Bronze, organisations effectively create a regulatory-ready immutable audit column store, enabling:
- Point-in-time reconstructions
- Evidence trails for regulatory queries
- Detection of data tampering
- Retrospective scenario modelling
3.2 Operational Efficiency
Applying SCD2 logic at the raw ingestion point:
- Simplifies and ensures consistency across all downstream models
- Reduces duplicative SCD2 implementations by teams, creating one authoritative temporal model
- Centralises change detection logic
- Allows deterministic rebuilds of Silver/Gold layers
- Simplifies CDC (Change Data Capture) integration with source systems
- improves performance in Silver and beyond
3.3 Architectural Separation of Concerns
Separation prevents entangling:
- historical logic
- business logic
- modelling rules
- operational schema designs
Bronze = source-aligned historical truth.
Silver = consumption-aligned present truth.
Silver-layer simplification allows:
- Data scientists to get clean, current data without SCD2 noise
- BI developers to avoid complex time-travel joins
- Domain teams to create analytic models aligned to business concepts
- Faster query performance and reduced cost in cloud warehouses/lakehouses
Meanwhile, Bronze remains the authoritative data lineage backbone.
3.4 Supports Data Mesh & Data Governance
Modern FS institutions increasingly adopt domain-oriented architectures. This pattern enables:
- domain ownership in Silver and Gold
- federated governance
- a clear separation between ingestion (Bronze) and refinement (Silver)
- lineage that governance teams can introspect
- compliance-safe reproducibility
The Silver layer becomes a stable, domain-specific interface.
4. How the Pattern Works Technically
Once the rationale is clear, the practical question becomes: How does this actually work?
The Bronze/Silver pattern is simple in principle but requires careful execution in production. This section walks through the technical flow, from ingestion to SCD2 processing in Bronze, to the creation of simplified, consumption-ready Silver tables, as well as examples and platform considerations. Understanding these mechanics ensures the pattern is implemented consistently, predictably, and with the integrity required in Financial Services.
4.1 Data Flow Overview
- Raw Layer: Ingestion from upstream systems (API, database CDC, messaging, batch).
- Base Layer: Normalise/standardise raw structures without applying business logic.
- Bronze Layer: Implement SCD2
- Add effective-from and effective-to timestamps
- Generate surrogate keys
- Maintain “current” flags
- Apply CDC logic (Inserts, Updates, Deletes)
- Silver Layer: Materialised or View-Based Simplification
- Latest-record-only views
- Filtered time windows
- Domain projection
- De-normalisation or enrichment
- De-duplication and simplification
- Materialisation or “live” views
- Gold Layer (Business Context): Business KPIs, MI, Risk Models, Regulatory Reports
- Platinum (Conceptual Meaning): Enterprise conceptual model, semantic reconciliation.
Please note: Gold/Platinum layers are separate articles, but the Bronze/Silver relationship underpins them.
4.2 Example: Customer Dimension
Bronze – SCD2 (Full History)
| Customer_SK | Customer_ID | Name | Address | Effective_From | Effective_To | Is_Current |
|---|---|---|---|---|---|---|
| 101 | C001 | Alice Lee | 12 King St, London | 2021-01-01 | 2022-02-12 | N |
| 102 | C001 | Alice Lee | 14 Bishopsgate, EC2 | 2022-02-13 | 9999-12-31 | Y |
Silver – Non-SCD2 Latest View
| Customer_ID | Name | Address |
|---|---|---|
| C001 | Alice Lee | 14 Bishopsgate, EC2 |
Consumers avoid SCD2 complexity while the enterprise retains full history: Silver presents only the latest valid record, while Bronze maintains every historical version.
Please note: Real customer dimensions in banks often contain 150–400 attributes, sourced from multiple onboarding, servicing, credit, AML, and product systems. SCD2 must reconcile conflicting updates from these systems. A common pattern is to use a hash-by-attribute or attribute priority rule to decide which update “wins” when multiple systems report changes within the same effective window. Without these rules, SCD2 pipelines can generate duplicate versions or incorrect temporal ordering.
Example of conflicting update resolution:
- Core banking updates Customer Address at 12:01
- CRM updates Customer Address at 12:02
- KYC system updates Customer Residency at 12:05
Bronze SCD2 must:
- preserve all three changes
- ensure they appear in correct temporal order
- avoid accidentally overwriting or duplicating earlier versions
This is why hash-based change detection + deterministic ordering rules are essential in FS SCD2 pipelines.
4.3 Platform-Agnostic Implementation
This pattern is common in:
- Databricks Lakehouse (Delta Lake)
- Snowflake with Streams/Tasks
- BigQuery with Change Data Capture patterns
- AWS Iceberg/Hudi based platforms
- Azure Fabric/Synapse pipelines
5. Benefits Specific to UK FS Use Cases
With the architecture and mechanics in hand, it becomes easier to see how this pattern unlocks value across real, high-impact Financial Services use cases. Different functions within banks, insurers, and payment institutions rely on either historical or current-state truth: AML/KYC teams need lineage; risk and actuarial models need accuracy; remediation teams need reconstruction; operational systems need clarity. This section ties the architecture to the everyday needs of FS organisations, demonstrating why Bronze and Silver are indispensable in practice, not just in theory.
5.1 Financial Crime & AML/KYC
Financial Crime and AML/KYC functions rely heavily on the ability to understand how customer identity, risk attributes, and behavioural factors have evolved. AML is fundamentally a temporal investigation discipline; CDD, EDD, and ongoing monitoring all depend on accurate historical trails.
The Bronze SCD2 layer provides the backbone for:
- KYC attribute evolution across lifecycle stages
- changes in identity (address, email, phone, residency)
- entity resolution and merging/splitting events
- source-of-funds and wealth re-analysis
- suspicious activity investigations and case reviews
- historical risk-rating transitions
Silver supports front-line analytics by exposing:
- clean current-state profiles for screening
- the latest customer attributes for transaction monitoring
- enriched identifiers for ML-based anomaly detection
- simplified inputs for risk scoring engines
Together, Bronze enables forensic reconstruction and regulatory defensibility, while Silver powers high-performance monitoring and screening.
5.2 Risk & Regulatory Reporting
Risk, finance, and regulatory reporting functions require absolute consistency over time. Many critical models, Basel RWA, IRRBB, IFRS9 impairments, stress testing, ICAAP/ILAAP projections, depend on the ability to replay input data exactly as it was known at the calculation date.
Bronze SCD2 ensures:
- full temporal lineage for point-in-time model execution
- the ability to reproduce past runs deterministically
- stable inputs for backtesting, challenger models, and re-performance
- transparent evidence for PRA examinations and s166 reviews
Silver provides:
- the clean, current-state datasets needed for BAU risk reporting
- simplified tables that support aggregation under BCBS 239
- operational clarity for credit risk, liquidity risk, and capital teams
The combination ensures both historical defensibility and current accuracy, core requirements for regulated FS institutions.
5.3 Customer Remediation & Complaints
Customer remediation programmes, PPI, pension transfers, motor finance, interest-rate hedging, and emerging Consumer Duty reviews, all require firms to reconstruct what they knew about the customer at the moment advice was given or a decision was made.
Bronze SCD2 supports:
- reconstruction of customer attributes as they existed on the relevant date
- replay of eligibility, affordability, and suitability assessments
- validation of decision logic for fairness and accuracy
- calculation of customer redress with full historical context
Silver supports operational teams by providing:
- a clean present-state view for customer contact operations
- simplified attributes for triage, segmentation, and MI reporting
Bronze delivers the evidence; Silver supports efficient operational execution.
5.4 Fraud Investigations
Fraud investigations, covering card fraud, APP fraud, account takeover, mule activity, and digital channel anomalies, depend on the ability to reconstruct behaviour and identity attributes over time.
Bronze SCD2 supports:
- historical identity changes
- device, channel, and behavioural shifts
- credential and contact detail evolution
- linked account or relationship structures
Silver supports:
- current-state attributes for real-time scoring
- simplified inputs for anomaly detection and ML models
- clean lookup tables for operational fraud engines
Bronze provides the temporal evidence required for investigation and regulatory defence, while Silver provides the current, high-performance view required for detection and frontline decisioning.
5.5 Conduct Risk, Past Business Reviews & Consumer Duty
Past business reviews, such as motor finance remediation, historical pension transfer assessments, vulnerable customer reviews, and Consumer Duty look-backs, depend critically on the ability to reconstruct exactly what customer status, risk profile, facts, or advice context the institution held at a specific point in time.
Bronze SCD2 enables:
- reconstruction of customer financial profile “as known on date X”
- validation of which suitability assessment that was applied
- replay of advice or decision logic using historical attributes
- defensible responses to Skilled Person Section 166 reviews
- accurate customer redress calculations
Silver supports day-to-day dashboards, but Bronze is essential for any regulatory investigation or conduct risk programme where historical truth ensures fairness, compliance, and defensibility.
6. Risks and Mitigations
No architectural pattern is free of trade-offs. While the Bronze/Silver model provides strong benefits, it also introduces new engineering, operational, and governance challenges if not implemented thoughtfully. This section outlines the most common risks, such as uncontrolled Bronze growth, consumer misuse of historical tables, and overly complex SCD2 logic, and provides practical mitigations. The goal is to ensure the architecture remains sustainable, predictable, and compliant as it scales.
| Risk | Mitigation |
|---|---|
| Bronze grows rapidly due to SCD2 | Partitioning, clustering, pruning, and cold storage tiering |
| Consumers accidentally access Bronze directly | Access governance, schema permissions, and catalogue masking |
| SCD2 logic becomes overly complex | Centralised patterns, automated change detection, CI/CD enforcement |
| Excessive rebuild time for Silver | Incremental refresh, materialised views, and change-only pipelines |
| Race Conditions exhibit when overuse of “Is Current” fields | Standardise on EffectiveTo = '9999-12-31' for determining current records, treat boolean current flags as convenience fields only. |
7. Summary: Bronze Preserves History, Silver Delivers Clarity
Implementing SCD2 in the Bronze layer while exposing a non-SCD2 view in the Silver layer has become a leading architecture pattern in UK Financial Services. It elegantly balances:
- Strong regulatory and audit requirements
- The need for clean, simple analytical datasets
- The flexibility of modern lakehouse platforms
- Data Mesh-aligned domain ownership
- High performance and operational clarity
This layered approach ensures Bronze provides audit-grade historical fidelity, while Silver exposes a simplified present-state view, forming a stable foundation for any modern, governed, regulated data platform.
Finally, our Bronze/Silver separation also creates the foundation for the advanced Gold and Platinum layers that follow, where business context and enterprise semantics live.