Skip to content

Horkan

a blog by Wayne Horkan

  • Home
  • About
  • How To Read This Site
  • Neurodiversity
  • Garden Centre
  • Horkan Genealogy

AI Agents for Temporal Repair, Metadata Enrichment, and Regulatory Automation

Leave a Reply

This article explains where AI agents genuinely add value in regulated Financial Services data platforms — and where they become indefensible. It distinguishes “useful” agents from regulator-defensible ones, showing how agents can safely detect temporal defects, propose repairs, enrich metadata, and draft regulatory artefacts without bypassing governance. Using practical patterns and real failure modes, it defines a strict operating model — detect, propose, approve, apply — that preserves auditability, replayability, and SMCR accountability while delivering operational leverage that survives PRA/FCA scrutiny.

Executive Summary (TL;DR)

Where agents are genuinely useful in regulated Financial Services — and where they become indefensible. AI agents can be transformative in Financial Services data platforms — not because they “think”, but because they can orchestrate repeatable work: detecting temporal defects, proposing repairs, enriching metadata, and drafting regulator-facing artefacts.

But regulated environments impose a hard constraint:

An agent must be auditable, replayable, permissioned, and approval-gated — or it is not fit for purpose.

This article lays out:

  • the difference between useful vs regulator-defensible agents
  • agent patterns that actually work for temporal platforms (SCD2 + precedence + PIT)
  • the guardrails and approval workflows that stop agents becoming uncontrolled change-makers
  • two anonymised “war story” patterns you will recognise immediately

The goal is not “agentic hype”. The goal is operational leverage that survives PRA/FCA scrutiny and SMCR accountability.

Part of the “land it early, manage it early” series on SCD2-driven Bronze architectures for regulated Financial Services. Governed agents for temporal repair and regulatory drafting, for platform engineers, AI leads, and risk teams who need agents that preserve auditability. This article delivers guardrails to make AI a force multiplier for compliance.

Contents

Table of Contents
  • Executive Summary (TL;DR)
  • Contents
  • 1. Useful vs Regulator-Defensible Agents
    • 1.1 A “useful” agent
    • 1.2 A “regulator-defensible” agent
  • 2. The Operating Model: “Detect, Propose, Approve, Apply”
  • 3. Agent Pattern: Late-Arriving Higher-Precedence Data (The Nightmare Scenario)
    • 3.1 What the agent should do (and what it must not do)
    • 3.2 A working architecture
    • 3.3 Why this survives scrutiny
  • 4. Agent Pattern: Precedence Drift Detection
    • 4.1 What drift looks like in practice
    • 4.2 What the agent does
    • 4.3 The output that matters
  • 5. Agent Pattern: Automatic Lineage Gap Filling
    • 5.1 What the agent should do
    • 5.2 Why this is valuable
  • 6. Guardrails and Approval Workflows (Non-Negotiable)
    • 6.1 Guardrail 1: No direct writes to Bronze truth
    • 6.2 Guardrail 2: Every agent action has an interaction log
    • 6.3 Guardrail 3: Risk-tiered approval
    • 6.4 Guardrail 4: Deterministic execution
  • 7. Two Anonymised War Stories
    • War story A: The temporal repair agent that saved an audit
    • War story B: The “helpful agent” that audit blocked immediately
  • 8. Conclusion: Where Agents Fit in a Mature Temporal Platform

1. Useful vs Regulator-Defensible Agents

Most conversations about AI agents in Financial Services collapse too many ideas into one word. “Agent” is used to describe everything from a scheduled script to a semi-autonomous system making changes to production data. In regulated environments, that lack of precision is dangerous. Before discussing patterns or architectures, we need to separate agents that are merely useful from those that are defensible under regulatory scrutiny. The distinction is not philosophical — it is operational, evidential, and enforceable.

1.1 A “useful” agent

A useful agent:

  • reads some data
  • makes a decision
  • writes an update
  • tells you it fixed something

This works in demos. It fails in Financial Services because it lacks an evidence chain.

1.2 A “regulator-defensible” agent

A regulator-defensible agent:

  • has a strictly bounded mandate
  • operates only through versioned workflows
  • produces a change proposal (not an untracked change)
  • requires explicit approval for any write affecting regulated truth
  • emits audit artefacts: what it saw, what it proposed, what was approved, what was applied
  • supports replay: “show me what the agent saw when it recommended this repair”

If your agent cannot explain itself through logs and provenance, it’s not an agent — it’s an ungoverned automation.

2. The Operating Model: “Detect, Propose, Approve, Apply”

Once you accept that agents cannot be autonomous actors in regulated systems, the next question is how they should be allowed to operate. The answer is not to neuter them entirely, but to embed them inside an explicit operating model that preserves accountability, traceability, and control. In practice, this looks far less like “agentic intelligence” and far more like a disciplined workflow.

All agent actions must be replayable against the same inputs to reproduce the same proposal.

Before patterns, the core principle:

Agents should be treated like junior operators in a controlled environment: they can detect issues, draft repairs, and stage changes — but the platform remains accountable.

A practical lifecycle:

  1. Detect: agent identifies an anomaly, breach, drift, or gap
  2. Propose: agent generates a repair plan with impacted keys / time windows
  3. Approve: human (or policy engine) approves based on risk class
  4. Apply: deterministic pipeline executes the repair
  5. Evidence: immutable log records the full chain

Every stage in this lifecycle must be replayable: given the same inputs, the agent must produce the same proposal, and the platform must be able to reproduce the decision trail on demand.

This model maps cleanly onto SMCR: the agent proposes; the organisation remains accountable for applying.

3. Agent Pattern: Late-Arriving Higher-Precedence Data (The Nightmare Scenario)

If you have ever run a real SCD2 Bronze layer with multiple upstream sources, you already know that late-arriving, higher-precedence data is not an edge case — it is inevitable. These scenarios are operationally painful because they require retroactive corrections to temporal truth, recomputation of PIT views, and careful handling to avoid audit failure. They are also exactly the kind of work that humans handle inconsistently under pressure.

If you’ve built a real SCD2 Bronze layer with multi-source precedence, this will be familiar:

  • lower precedence source sets an attribute
  • days/weeks later, higher precedence source delivers a correction
  • the platform must retroactively rewrite temporal history
  • downstream PIT views must be recomputed for the affected interval
  • everything must remain audit-defensible

3.1 What the agent should do (and what it must not do)

The agent should:

  • detect that a higher-precedence record has arrived late
  • identify impacted business keys and time ranges
  • generate a repair proposal (“rebuild PIT window for customer_id X from t1→t2”)
  • stage the plan, with evidence of inputs and rules used

The agent must not:

  • directly mutate Bronze history without controlled pipelines
  • “merge fixes” opportunistically
  • change precedence rules on the fly

3.2 A working architecture

Inputs the agent watches:

  • delta_log / CDC arrivals (new change events)
  • precedence metadata table (rank, rule version, effective windows)
  • temporal repair backlog table

Agent output:

  • a row in temporal_repair_requests:
    • request_id
    • business_key (or set)
    • affected_from, affected_to
    • reason_code (late higher-precedence arrival)
    • source_system
    • precedence_rule_version
    • proposed_actions (e.g., rebuild PIT slice; re-run merge for window)
    • risk_class (high if conduct/AML domains)
    • evidence_links (lineage pointers)

Apply step (non-agent):

  • a deterministic temporal repair job executes:
    • rebuild SCD2 segments
    • recompute PIT materialisations
    • emit reconciliation report
    • update lineage

3.3 Why this survives scrutiny

Because your “repair” is:

  • staged
  • approved
  • executed by a deterministic process
  • fully logged and replayable

The agent provides operational speed without becoming an unaccountable actor.

4. Agent Pattern: Precedence Drift Detection

Precedence frameworks are usually designed carefully and documented clearly. Unfortunately, production systems have a habit of quietly drifting away from design intent. Over time, small changes in pipelines, timestamps, or source behaviour can undermine deterministic ordering without triggering obvious failures. By the time a problem surfaces, the damage is already systemic.

Precedence rules look stable on paper. In production they drift, because:

  • systems change behaviour
  • feeds arrive late
  • “temporary overrides” become permanent
  • a pipeline change quietly flips a tie-break rule

Note on Bi-Temporal Semantics:
Where bi-temporal facts are used, drift detection must respect both valid-time and system-knowledge time, preserving when a fact was true and when the organisation knew it, without collapsing history into a single corrected state.

4.1 What drift looks like in practice

Examples:

  • CRM starts overwriting legal name because a field mapping changed
  • KYC risk rating should win for 180 days, but the override window isn’t being enforced
  • an ingestion timestamp replaces source timestamp, breaking deterministic ordering

4.2 What the agent does

Daily (or intra-day) the agent:

  • samples merged outputs per domain
  • compares “winning source” outcomes to the precedence policy
  • flags violations with a concrete evidence set:
    • business key
    • attribute name / group
    • expected winner vs actual winner
    • timestamps and ranks
    • first observed date/time
    • suspected cause (pipeline version change, new source feed, etc.)

4.3 The output that matters

The agent should produce a reconciliation report and open a governed issue:

  • “Precedence breach rate for AML_RISK_SCORE exceeded threshold”
  • “N customers affected”
  • “First occurrence aligned to pipeline release X”

This is gold in audit conversations because it shows active controls and monitoring.

5. Agent Pattern: Automatic Lineage Gap Filling

Most Financial Services organisations now have lineage tooling in place — and most discover that lineage is never complete. Gaps emerge through ad-hoc queries, notebooks, one-off scripts, or tooling that doesn’t emit metadata correctly. Left untreated, these gaps become a liability during audit and regulatory reviews, even when the underlying data is sound.

Most FS firms now use lineage tooling. Nearly all discover gaps:

  • notebook jobs without proper lineage emission
  • ad-hoc SQL executed outside pipeline frameworks
  • vector index builds that aren’t linked to data snapshots
  • “little scripts” that become de facto production

5.1 What the agent should do

When lineage is missing or incomplete, the agent can:

  • infer likely dependencies from:
    • query logs
    • DDL history
    • job metadata
    • table access patterns
  • propose a lineage patch:
    • “table X depends on tables A, B, C via job run Y”
  • stage the patch for approval
  • write back to a lineage registry only after sign-off

5.2 Why this is valuable

Not because inferred lineage is perfect — it won’t be.
Because it turns “unknown gaps” into “known gaps with evidence”, and gives governance teams a way to remediate systematically.

In regulated environments, “we don’t know” is worse than “we know this is incomplete and here’s our remediation trail”.

Regulatory Automation Boundary:

  • The same pattern applies to regulator-facing artefacts.
  • Agents may draft narratives, evidence packs, and reconciliation summaries (e.g. for audits, ICAAP/ILAAP inputs, or remediation reporting), but they do not submit, attest, or approve regulatory outputs.
  • Final accountability and submission remain human responsibilities.

6. Guardrails and Approval Workflows (Non-Negotiable)

The difference between a controlled agent and a career-ending one is not sophistication — it is constraint. Agents become dangerous not when they are powerful, but when they are allowed to act without boundaries, evidence, or approval. In regulated Financial Services, these boundaries cannot be “best practice” or “recommended”. They must be explicit, enforceable, and designed into the platform.

Agents that write directly to “truth” are where careers go to die.

6.1 Guardrail 1: No direct writes to Bronze truth

Agents may write to:

  • staging tables
  • proposals
  • tickets
  • reports
  • metadata suggestions

They do not write to:

  • Bronze SCD2 tables
  • precedence metadata tables
  • PIT outputs
    without a controlled pipeline and approval.

6.2 Guardrail 2: Every agent action has an interaction log

For every detection and proposal you log:

  • inputs referenced (lineage pointers)
  • rule versions used
  • proposed changes
  • risk classification
  • who approved
  • what job applied it
  • post-apply validation results
  • “model_version / prompt_template” for LLM-specific traceability

This is the same evidence posture you need for RAG — applied to agent actions.

6.3 Guardrail 3: Risk-tiered approval

A pragmatic approval model:

  • Low-risk (metadata suggestions, glossary tags, documentation drafts): auto-approve or lightweight approval
  • Medium-risk (pipeline warnings, minor schema drift handling): data engineering approval
  • High-risk (e.g. customer identity, AML risk scores, conduct flags, affordability attributes): formal approval, logged under the accountable owner (SMCR mapping)

6.4 Guardrail 4: Deterministic execution

Agents propose. Pipelines execute.

The execution path must be:

  • versioned
  • tested
  • repeatable
  • re-runnable

“Agent decided to update record X” is not an execution standard.

7. Two Anonymised War Stories

Architectural principles are easiest to agree with in the abstract. Their real value becomes clear only when you see how they behave under pressure — during audits, regulatory reviews, and remediation programmes. The following examples are anonymised, but they are not hypothetical. Variants of both appear repeatedly across large FS organisations.

These are intentionally framed as patterns rather than gossip. You’ll recognise them.

War story A: The temporal repair agent that saved an audit

A large FS organisation had recurring late-arriving KYC corrections. Historically, these fixes were done manually and inconsistently. During review preparation, they implemented an agent that:

  • detected late higher-precedence arrivals
  • produced repair proposals with impacted keys and time windows
  • required sign-off from the accountable domain owner
  • ran a deterministic temporal rebuild job
  • produced a reconciliation report showing “before vs after” and rule versions

Outcome: they could show a regulator not only the repaired state, but the control mechanism that ensured repairs were governed and repeatable. The agent didn’t “fix history”; it enforced an operating model for fixing history.

War story B: The “helpful agent” that audit blocked immediately

Another firm built an agent that directly:

  • edited reference data mappings
  • patched SCD2 records when anomalies were found
  • retried failed merges by changing effective dating logic

It improved metrics quickly. It also produced:

  • no reproducible change logs
  • no approval chain
  • no stable rule versioning
  • no consistent evidence trail

Audit’s view was simple: “This is an uncontrolled actor modifying regulated records.”
The programme was stopped, and the platform team had to unwind changes with limited traceability.

The lesson: in FS, agents must increase control, not bypass it.

8. Conclusion: Where Agents Fit in a Mature Temporal Platform

Agents are not a shortcut around good platform design, and they are not a substitute for governance. They are only effective when layered on top of systems that already have clear temporal semantics, deterministic pipelines, and strong lineage. In that context, agents become accelerators — not risks.

Agents are not a replacement for good engineering. They are a force multiplier for mature platforms that already have:

  • SCD2 Bronze with clear temporal semantics
  • precedence frameworks that are versioned and explicit
  • PIT reconstruction patterns
  • strong lineage and logging

In that context, agents can:

  • detect drift faster
  • propose repairs with better evidence
  • reduce manual toil
  • generate regulator-facing artefacts reliably
  • keep the platform healthy without heroics

The winning posture is simple:

Agents propose. Humans approve. Deterministic pipelines apply. Evidence is immutable.

That is what “agentic” looks like when you intend to survive PRA/FCA scrutiny — and still ship value.

This entry was posted in article and tagged AI Agents, AI Governance, Auditability, Data Governance, Data Lineage, FCA, financial services, Metadata Management, Operational Resilience, Point-in-Time Processing, PRA, Regulated Data Platforms, Regulatory Automation, SCD2, SMCR, Temporal Data, UK FS SCD2 Bronze on December 31, 2025 by Wayne Horkan.

Post navigation

← Integrating AI and LLMs into Regulated Financial Services Data Platforms

Posts on Page

  • AI Agents for Temporal Repair, Metadata Enrichment, and Regulatory Automation



Recent Posts

  • AI Agents for Temporal Repair, Metadata Enrichment, and Regulatory Automation
  • Integrating AI and LLMs into Regulated Financial Services Data Platforms
  • Ontological Desynchronisation: From Birthgaps and Behavioural Sinks to Algorithmic Capture
  • Foundational Architecture Decisions in a Financial Services Data Platform
  • Time, Consistency, and Freshness in a Financial Services Data Platform

On This Day

  • Nothing has ever happened on this day. Ever.

Recent Comments

  1. Wayne H on 20211230 Update
  2. Wayne H on My Years at Sun Microsystems: From Dream Job to Oracle Redundancy
  3. Scott Fairbanks on My Years at Sun Microsystems: From Dream Job to Oracle Redundancy
  4. Wayne H on Curiosity, Cats, and Rabies: A Thought from Tenerife
  5. Lim on Curiosity, Cats, and Rabies: A Thought from Tenerife

Archives

  • December 2025 (31)
  • November 2025 (1)
  • October 2025 (15)
  • September 2025 (21)
  • August 2025 (8)
  • July 2025 (20)
  • June 2025 (28)
  • May 2025 (36)
  • April 2025 (30)
  • March 2025 (20)
  • February 2025 (22)
  • January 2025 (36)
  • December 2024 (39)
  • November 2024 (30)
  • October 2024 (20)
  • September 2024 (5)
  • August 2024 (24)
  • July 2024 (55)
  • June 2024 (10)
  • May 2024 (2)
  • April 2024 (1)
  • March 2024 (4)
  • February 2024 (12)
  • January 2024 (5)
  • December 2023 (7)
  • November 2023 (9)
  • October 2023 (57)
  • September 2023 (132)
  • August 2023 (27)
  • July 2023 (9)
  • June 2023 (10)
  • May 2023 (4)
  • November 2022 (1)
  • May 2022 (1)
  • December 2021 (6)
  • April 2017 (1)
  • September 2015 (2)
  • September 2009 (7)
  • August 2009 (3)
  • July 2009 (16)
  • June 2009 (3)
  • May 2009 (13)
  • April 2009 (6)
  • March 2009 (17)
  • February 2009 (31)
  • January 2009 (37)
  • December 2008 (9)
  • November 2008 (17)
  • October 2008 (7)
  • September 2008 (14)
  • August 2008 (19)
  • July 2008 (1)
  • June 2008 (1)
  • May 2008 (5)
  • April 2008 (5)
  • March 2008 (19)
  • February 2008 (9)
  • January 2008 (10)
  • December 2007 (1)
  • November 2007 (5)
  • October 2007 (4)
  • September 2007 (5)
  • August 2007 (7)
  • July 2007 (3)
  • June 2007 (20)
  • May 2007 (12)
  • April 2007 (19)

Categories

  • article (365)
  • blog (93)
  • blog-post (42)
  • chess (24)
  • facial-recognition (2)
  • film (6)
  • history-of-cheltenham (11)
  • home (34)
  • irish-unification (1)
  • language (1)
  • life (19)
  • link (88)
  • management-consulting (6)
  • music (7)
  • n4s-learning (37)
  • neurodiversity (37)
  • open-source (2)
  • personality-types (21)
  • programming (2)
  • quotation (1)
  • rip (1)
  • search (8)
  • site-design (1)
  • site-update (4)
  • social-media (5)
  • social-technology (2)
  • socio-technical (2)
  • socio-technicalogical (0)
  • talks (2)
  • tech (98)
  • trading (9)
  • uncategorized (160)
  • virtual-book (1)
  • west-midlands-cyber-hub (1)
  • wordpress (4)
  • work (23)

Social

  • Wayne Horkan
  • Wayne Horkan

Meta

  • Log in
  • Entries feed
  • Comments feed
  • WordPress.org

Tags

AI and cybersecurity AI Safety bollocks CIISec critical infrastructure cyber cyber academia cyber clusters cyber commercialisation cyber communities cyber education Cyber Governance Cyber Hubs cyber inclusion cyber infrastructure Cyber Innovation cyber investment cyber leadership cyber policy cyber procurement cyber regulation Cyber Resilience cyber risk Cyber Runway Cyber Runway Scale cyber scaleups cybersecurity Cyber Skills cyber standards Cyber Startups defence cyber Digital Resilience DSIT national cyber strategy NCSC ncsc-for-startups neurodiversity public sector cyber Secure by Design sun-microsystems-blog UKCSC UK cyber strategy UK cyber workforce UK government cyber UK tech ecosystem women in cybersecurity

Breadcrumb

Home » AI Agents for Temporal Repair, Metadata Enrichment, and Regulatory Automation

Popular

Categories

  • article
  • blog
  • blog-post
  • chess
  • facial-recognition
  • film
  • history-of-cheltenham
  • home
  • irish-unification
  • language
  • life
  • link
  • management-consulting
  • music
  • n4s-learning
  • neurodiversity
  • open-source
  • personality-types
  • programming
  • quotation
  • rip
  • search
  • site-design
  • site-update
  • social-media
  • social-technology
  • socio-technical
  • talks
  • tech
  • trading
  • uncategorized
  • virtual-book
  • west-midlands-cyber-hub
  • wordpress
  • work

Links

Proudly powered by WordPress