Skip to content
Disclaimer

This is a draft discussion paper circulated for peer review. It is independent work published in a personal capacity and does not constitute official guidance or policy of any government body. It does not mandate or recommend specific controls for any agency, system, or project. Views and analysis are the author's own.

Full Discussion Paper

This is the complete discussion paper — approximately 200 pages in print. For a shorter introduction, see Understanding AI Code Risk (~13 pages). For the review checklist with worked examples, see the Practical Guide.

Download PDF

Semantic Defects in AI-Generated Code: Assurance Frameworks for AI-Assisted Development in High-Stakes Code Paths

Discussion Paper, Draft for Comment Version: 0.1.0 Date: 24 March 2026 Prepared by: John Morrissey

Version Date Summary
0.1.0 24 March 2026 Initial draft

This version is the first draft prepared for consultation. The taxonomy and recommendations are under community consultation and should be read as candidates for validation, not settled outputs.

Reading guide

This paper is the full evidence base. Shorter companion documents may be sufficient for your purpose:

  • Governing AI-Generated Code: Semantic Risk in High-Stakes Code Paths (~13 pp) — the argument and recommendations in accessible form, for executives, policy advisors, and a broad audience.
  • Reviewing AI-Generated Code: A Practical Guide for Code Authors (~23 pp) — practical review guidance for people working with AI-generated code, without the technical framework.

For role-based reading paths across the document suite, see the Document Suite Map (~3 pp).

---

Abstract

AI coding agents produce code that follows established good practice, and defensive programming is established good practice. The problem is that these defensive patterns become defensive anti-patterns in high-stakes contexts: a .get() that returns "Sydney" when it cannot determine your location is helpful; an allergies = patient.get("allergy_data", []) that returns an empty list when it cannot find a patient's allergy record removes the crash that would have surfaced the problem. Clinical decisions now proceed as though no allergy were recorded, because "no allergy information available" has become "allergy information has been provided and it is: none." This is how a penicillin allergy disappears.

The pattern is the same, but the consequences are opposite. Empirical observation suggests that agents (AI systems that write, test, and commit code with varying degrees of autonomy) do not reliably distinguish between these cases. The difference depends on domain knowledge — what an allergy record is and what its absence means — and institutional knowledge: whether this system must treat missing clinical data as an unanswered question requiring investigation rather than substitute an empty default and continue. Neither requirement is expressed in the syntax. Agents with project-level instructions may follow explicit rules within a session but do not reliably generalise from them (§2.4(a) develops this argument; Appendix E demonstrates it across three incidents where the governing rules were in the agent's context but were not applied).

The standard assurance stack — linters, type checkers, SAST, DAST, unit tests, and conventional peer review — checks for the wrong class of defect. The failure modes described in this paper are semantic: they concern what the code means in its institutional context, not how it is structured.

This paper presents a STRIDE-based threat model for agentic code generation and proposes the Agentic Code Failure (ACF) taxonomy: fifteen core failure modes with associated risk ratings and detection status, plus five provisional candidates (failure modes observed but not yet sufficiently validated for core classification). Of the fifteen core failure modes, thirteen are undetected or only partially detected by existing tools — including all four with no tool coverage at all, both Critical-rated entries among them. On the basis of this analysis, current cybersecurity guidance (ISM, NIST SSDF, and Essential Eight) does not yet appear to provide adequate controls for this risk profile.

Two case studies provide the empirical base (§8): a simulation (Appendix D) producing 20 ACF-mapped findings in ~800 lines of code that passed every automated check, and a longitudinal observation (Appendix E) from six months of compliance-constrained agentic development where purpose-built enforcement caught recurring violations that conventional tooling missed. Together, they demonstrate that the problem is not the defect rate but the invisibility — these patterns look like correct code and enter codebases through normal review. The paper's observational base remains narrow, and its proposals are candidate controls for consultation, not settled guidance.

Executive Summary

Delivery reality. This paper is written from the perspective of direct development — one developer's first-hand observations — for analytical clarity, but the dominant delivery context in government is contracted development. The risks described here therefore apply not only to agency staff using AI coding agents, but to the typically larger volume of software produced by vendors and service providers on the Commonwealth's behalf. In practice, the contracted case is often harder: provenance is less visible, review quality is more opaque, and acceptance processes may test functionality without testing the semantic correctness properties this paper identifies. The structural analysis appears in §6.7.

The problem. AI coding agents produce code that follows established good practice — defensive programming, graceful error handling, sensible defaults — faster than human review processes were designed to absorb. In most software, this is fine. In high-stakes code paths — those where integrity, auditability, or reliability require that silent corruption, unverifiable state, or fabricated defaults are worse than controlled failure — defensive programming is the wrong pattern: a missing field should halt processing, not silently receive a default. Agents do not reliably make this distinction, because the difference depends on domain and institutional knowledge rather than syntax.

Consider two uses of .get() with a default value — a shortcut that substitutes a fallback when the real value is missing: location = prefs.get("city", "Sydney") and classification = record.get("security_classification", "OFFICIAL"). The first is helpful — a news site defaults your location. The second removes the crash that would surface a missing classification — and if any upstream fault ever drops that field, a PROTECTED document is silently treated as OFFICIAL from that point forward. The pattern is the same, but the consequences are opposite. The agent produces both with equal confidence.

At scale. Agents produce this class of context-inappropriate pattern as a recurring characteristic rather than an occasional lapse: silent data fabrication, trust-boundary violations, and audit-trail destruction. These failures are largely outside the scope of existing automated checks. The compounding is worst where tolerance is lowest: the highest volume of confident, convention-conforming code arrives at the review boundaries of precisely the systems least able to absorb these failure modes.

Legacy modernisation risk. The risk is sharpened in legacy modernisation, where agents replace the accidental rigidity of old code — its crashes on NULL, its refusal to proceed on ambiguous input — with modern defensive patterns, replacing institutional knowledge that was encoded in the code's behaviour with standard defensive patterns that lack the same properties (§1.2.6). The commit message says "fix: handle NULL gracefully," and a code path that used to crash on corruption now silently fabricates a default.

Why this persists. Known failure modes are mitigable, but the condition that produces them is architecturally load-bearing rather than an implementation defect awaiting routine correction. Bounded context, selective salience, and compression of prior state are fundamental architectural properties of current agentic systems. In this paper's assessment, the result is governable, not removable. The condition may also deepen over time: agent-generated code is already entering the open-source repositories that form training corpora for future models, creating a reinforcing feedback loop in which each model generation inherits a stronger defensive-pattern bias from the previous generation's output (§2.5). This strengthens the case for cross-government standards — individual organisations cannot address a bias that propagates through shared training infrastructure.

The gap. These failure modes are semantic: they concern what the code means in its institutional context, not how it is structured. The standard assurance stack is not designed to determine whether code behaviour is institutionally appropriate for its context.2 Of the fifteen core failure modes in this paper's taxonomy, thirteen are undetected or only partially detected by existing tools — including all four with no tool coverage at all, both Critical-rated entries among them. A further two are process-level threats for which tool-based detection is not applicable. Current cybersecurity guidance1 does not yet address this gap.

The correction persistence problem. Current assurance frameworks assume that corrective action is durable: identify a defect class, train the developer, and the problem stays fixed. With agent-generated code, every correction lasts only until the next session. The agent has no persistent memory across sessions, so the same failure mode recurs regardless of how many times it has been caught previously (§2.4(a)). The durable intervention is not training the developer but encoding the detection as an automated rule. This shifts the governance model from "train and trust" to "detect and enforce" — a fundamental reorientation that underpins the technical controls proposed in §7.2.

Expanding governance perimeter. The gap is compounded by a boundary problem: agentic tools enable non-developers — analysts, policy officers, data managers — to produce executable logic outside traditional SDLC channels and therefore outside established controls. The volume problem (too much code for review processes) and the perimeter problem (code produced outside the governed boundary) are materialising simultaneously.

What to do. The answer is not simply more careful review of AI-generated code. Additional review effort alone cannot close this gap — these failures are semantic and are not reliably surfaced by conventional review, however thorough. This paper's proposed response is to treat agent output as untrusted input at a trust boundary and build the missing verification layer: a validation boundary with checks capable of determining whether code behaviour is correct for its institutional context.3 These checks should be encoded in forms that tools can parse, verify, and — where proportionate — enforce at the repository boundary before code reaches human review.45

Not additional overhead. The longitudinal case study (§8.6) demonstrates that this is a redirection of existing review effort from low-value pattern scanning toward high-value semantic evaluation. The total compliance burden is similar; the assurance yield is higher.

The need for cross-government action. The response this paper describes cannot be assembled organisation by organisation. It requires action by the bodies responsible for whole-of-government security control development, in three areas. First, guidance that agent output be treated as untrusted input, kept out of the codebase until it has passed automated structural and semantic checks and human review: the trust level of agent-generated code is determined by its provenance (machine-generated, not yet validated), not by the seniority of the person who directed the agent (§5.2). Second, targeted extension of ISM controls for agent-generated code — the ISM's 2025 updates provide strong foundations, but provenance tracking, semantic defect detection at the commit boundary, and procurement disclosure requirements remain unaddressed (§6.1). Third, controls at the contracted-delivery boundary, without which any of the above governs only the minority of code written directly by agencies (§6.7). A set of candidate controls developed from this analysis has been circulated separately for consultation; they are candidates for validation, not settled guidance — the paper's observational base is narrow (§1.4) and the analysis warrants broader validation before informing formal controls.

Where to read more. The threat model (§2–5), gap analysis (§6), and case studies (§8) provide the evidence base. §8 presents two case studies: a simulation (§8.2, Appendix D) producing 20 ACF-mapped findings in ~800 lines of code that passed every automated check, and a longitudinal observation (§8.3, Appendix E) from six months of agentic development where purpose-built enforcement caught recurring violations that conventional tooling missed. The ACF taxonomy (Appendix A) catalogues all fifteen core failure modes with risk ratings and detection status, plus five provisional candidates.

Table of Contents

Reading paths by audience

Use the table below to find a sensible starting point for your role. Where the register shifts between audiences, the final column suggests sections you may choose to defer. All readers may also find the case studies in Appendices D and E useful as concrete illustrations of how the failure patterns described in this paper appear in practice.

Audience Suggested starting points Additional depth
SES / CISOs / policy leaders Executive Summary, §2 (threat characterisation), §6 (gap analysis), Appendix E §E.7 (cross-cutting observations) §1.4, §3, §5, §8
Security practitioners / IRAP assessors §7 (response landscape), §9 (open questions — evidence thresholds), Appendix A (taxonomy), §6 (gap analysis), Appendix E Appendix C (unless SQL is in scope), Appendix B
Technical leads / developers §2, §4, §7 (response landscape), Appendix A (taxonomy), Appendix E, companion specification §9
Tool builders Appendix A (detection approaches per ACF entry), §7.2 (technical controls and validation maturity stages), §5 (authority-tier model), companion specification §6, §9
Non-developer staff using agentic tools Start with the companion guides: Governing AI-Generated Code (~13 pp.) and Reviewing AI-Generated Code (~23 pp.). In this paper: Executive Summary, §1.2.7 (governance perimeter), §9.7 (citizen programmer governance), Appendix C §C.4 Technical sections (§3–§5, §7)

\epigraphbox{The concern is not that AI outputs are always poor, but that they may become persuasive, efficient, and operationally privileged faster than institutions adapt their assurance methods.}{ChatGPT 5.4, on being asked to review this paper}

1. Introduction and Scope

1.1 What this paper addresses

This paper addresses the use of AI agents — large language models operating as autonomous or semi-autonomous code generators — on high-stakes code paths: those where silent corruption, unverifiable state, or fabricated defaults are more dangerous than controlled failure, and where decisions must remain traceable to their source.

The term is used deliberately to avoid narrowing the analysis to government systems alone; the same failure dynamics apply in healthcare, finance, emergency management, critical infrastructure, and other domains where integrity and auditability requirements make silent degradation unsafe. Government frameworks (ISM, IRAP) provide the regulatory context that this paper's recommendations engage with, but the threat model is defined by system properties, not by sector.

This paper does not argue that agentic code generation is dangerous in general, or that defensive programming is inappropriate in high-stakes systems as a class. The problem is that the correctness of defensive behaviour is context-dependent at the level of specific code paths, and LLMs do not reliably possess the institutional and architectural context needed to distinguish those cases. Even otherwise low-stakes systems typically contain narrow high-stakes code paths — authentication, authorisation, account recovery, payment handling, and audit logging — where silent corruption or fabricated defaults are unacceptable. These paths are high-stakes not because the host application is intrinsically high-assurance, but because their outputs are treated as authoritative by users, counterparties, or downstream federated systems.

The underlying dynamics identified here — correlated patterns, consistent surface quality, review capacity exhaustion — generalise beyond code to any agent-assisted production pipeline (policy documents, security assessments, risk registers). This paper addresses the code case as the most concrete and technically tractable instance.

While the primary focus is Python, the threat dynamics extend to other languages where agents generate executable logic. Appendix C analyses the extension to agentic SQL generation, where the risks are amplified by direct data store access and the prevalence of citizen programmers (§1.2.7) producing queries outside formal SDLC channels.

This paper does not address:

  • Adversarial attacks on AI systems (prompt injection targeting the model, model poisoning, training-data attacks) — though prompt injection against coding agents via malicious comments in codebases they process is an adjacent concern that warrants separate treatment
  • AI-generated content beyond source code (documents, communications) — though the threat dynamics identified here apply by analogy
  • The procurement or accreditation of AI platforms themselves
  • Privacy implications of training data
  • Agent-selected dependencies (package installation, version pinning) — agents choose libraries as well as writing code, introducing supply chain risks (typosquatting, unmaintained packages, transitive vulnerabilities). These are real risks but are not specific to agents — they are amplified instances of existing supply chain concerns already addressed by SCA tooling and ISM-0402. This paper focuses on the code the agent writes, not the code it imports.

1.2 Why now

Several converging factors make this urgent:

  • The review burden is growing faster than published productivity evidence suggests (§1.2.1–1.2.2)
  • Historical precedent shows the failure shape is real (§1.2.3)
  • Adoption is outrunning guidance (§1.2.4)
  • Prohibition is the wrong response (§1.2.5)
  • Two specific risk dimensions — legacy modernisation (§1.2.6) and the expanding producer population (§1.2.7) — compound the problem beyond the volume case alone

1.2.1 Published productivity evidence underestimates the review burden

The point here is not that the productivity literature is wrong, but that it answers a different question from the one security and assurance teams need to answer.

Most readers will approach agentic coding through the lens of the published productivity literature, which reports modest average gains on bounded tasks. A 2023 controlled study of GitHub Copilot found developers completed a task 55.8% faster.17 Three field experiments found a pooled 26.08% increase in completed tasks, with substantial variation across settings and developer experience levels.18 Google's CEO reported that "more than a quarter of all new code at Google is generated by AI, then reviewed and accepted by engineers."19 One rigorous randomised trial found experienced developers on mature open-source codebases were actually 19% slower with AI tools, despite believing they were 20% faster.20

That evidence base is real, but it is increasingly incomplete for security purposes. By construction, published productivity research lags the engineering frontier. The studies above measured earlier-generation tools (primarily inline autocomplete, not autonomous agents), shorter task horizons, and narrower human-in-the-loop workflows than what current-generation agents are capable of. The engineering frontier has moved: vendor evidence now describes agents operating across multi-file features, multi-hour task horizons, and scaffolded environments where the agent plans, executes, tests, and iterates with limited human intervention.

Experienced practitioners may judge that the review-surface problem described in this paper is already material in current agent workflows. Readers who do not share that assessment need not accept the stronger claim in order to accept the policy case: on any reasonable trajectory, it is prudent to expect review-surface generation velocity to improve faster than institutional assurance processes adapt. The threat model that follows is driven by this trajectory, not by any specific claim about current capability levels.

Whether these capabilities deliver net productivity gains in compliance-constrained environments is an open and context-dependent question (see §8). But for security, it is the wrong question.

The security-relevant variable is not average productivity uplift. It is review-surface generation velocity — the rate at which an agent can produce plausible, syntactically valid, convention-conforming code that arrives at a human review boundary.

This is a different quantity from "how much faster do developers ship," and it is poorly served by a single multiplier. The answer depends on several interacting factors14 whose relative weight varies by context. Even if the overall development cycle is only modestly faster — or, as the METR trial suggests, sometimes slower — the review process faces materially more code per unit time. The review bottleneck is a function of how fast plausible code can be generated, not how fast compliant software can be delivered.

1.2.2 Trajectory

These trends are accelerating, not plateauing. Agent capability improves with each model generation; review capacity does not. The capability is already here: current-generation agents produce code that is syntactically correct, convention-conforming, and difficult to distinguish from human-authored work on casual inspection — which is precisely what makes the widening attention gap dangerous. Unlike previous productivity tools, the failure modes of agent-generated output are specifically the kind that require more attention per unit of output, not less.

This is not a theoretical concern. In February 2026, GitHub implemented platform-level restrictions on pull requests — maintainers can now disable pull requests entirely or restrict creation to collaborators only — driven substantially by the volume of agent-generated contributions overwhelming open-source maintainers.2191 GitHub's Director of Open Source Programs framed the problem in terms that directly parallel this paper's analysis: "The cost to create has dropped but the cost to review has not." Projects including curl ended bug bounty programmes22 after AI-generated security reports overwhelmed validation capacity, and multiple major projects now explicitly restrict AI-generated contributions. Maintainers described a "breakdown in the trust model behind code reviews"; coverage of GitHub's response drew an analogy to a denial-of-service attack on human attention — a framing that directly mirrors the STRIDE-D application in §3.2 of this paper. When the world's largest code hosting platform ships technical controls because its review process can no longer absorb plausible-looking contributions, the review-surface problem is no longer hypothetical.

1.2.3 Precedent

The underlying defect pattern is not hypothetical, even if its agentic form is still under-instrumented. In 2017, a buffer over-read in Cloudflare's HTML parser — known as Cloudbleed23 — leaked sensitive data across millions of websites for five months.24 The code was plausible-but-wrong: a pointer equality check (==) where a boundary check (>=) was needed. The defect had existed for years without triggering, because the old buffer management accidentally prevented the error path from executing. A new parser removed that accidental suppression, and the boundary check that had never been tested was the one that failed. The bug was found not by Cloudflare's monitoring, testing, or code review, but by a Google Project Zero researcher who stumbled upon leaked authentication tokens during unrelated work. The system produced no crashes, no alerts, and no anomalous logs. It ran correctly in every observable dimension except the one that mattered.

Cloudbleed is relevant to the agentic threat model because it exhibits both properties that make agent-generated defects dangerous:

First, semantic invisibility. The pointer equality check was locally reasonable code — incorrect, but in a way that only an expert with specific knowledge of the parser's buffer semantics would spot. The wrongness depended on boundary semantics that no automated tool checked and no reviewer had reason to question. This is the same property that makes agent-generated defensive patterns dangerous — a .get() with a default is locally reasonable code whose wrongness depends on institutional context that no linter encodes.

Second, latent dormancy. The vulnerable code path existed for years without consequence. The code was already wrong; it just did not matter until an upstream change began exercising the path that had always contained the defect. This is the pattern that makes plausible-but-wrong code especially dangerous in high-stakes systems: it will appear fine in normal operation, until modernisation, schema drift, or an upstream behavioural shift traverses the neglected path on which the latent error has been waiting.

Cloudbleed is not a direct analogue of any single failure mode in this paper's taxonomy — it is a compound failure combining defective boundary semantics with swallowed error propagation.25 But the two-property structure — semantic invisibility plus latent dormancy — is precisely what makes agent-generated defects dangerous. Cloudbleed was one parser, one boundary check, at human velocity. The agentic analogue is the same class of defect, potentially distributed across many functions the agent touches, waiting for the same class of activation event.

1.2.4 Adoption pressure

Government agencies face simultaneous pressure to modernise legacy systems, deliver digital services faster, and do more with constrained budgets. Agentic coding is an obvious productivity lever. Some agencies are already using it. Guidance that arrives after widespread adoption is guidance that arrives too late. The pressure is compounded by the fact that most government software is delivered through contracted service providers, not in-house teams — meaning the adoption decisions that matter most may be made inside supplier organisations, not inside agencies (§6.7).

1.2.5 The case against prohibition

The response to these risks is not to ban agentic coding. Beyond the velocity gains, agents fundamentally change what is tractable for a development team. In the author's experience, complex refactoring across large codebases, systematic security remediation, architectural migrations, and comprehensive test coverage campaigns — tasks that previously required coordinating large teams over weeks — become feasible for a skilled developer who can hold the entire problem in their head. This is not just faster; it is qualitatively better. A single developer directing agents through a codebase-wide refactor can preserve a more coherent architectural vision than a large team distributed across the same task. Prohibition would sacrifice this capability benefit — the ability to undertake more complex, more voluminous work with greater coherence — not just the velocity benefit. Agents may also reduce certain classes of vulnerability that human developers produce through fatigue, inconsistency, or inattention.15

This paper's scope is threat identification, not net risk assessment. The controls proposed here address the distinct risks that agent-generated code introduces — risks that current guidance does not cover and that existing controls do not detect. A complete risk posture assessment would require evaluating both the vulnerabilities agents introduce and those they prevent; the latter is an empirical question outside this paper's scope but should inform proportionate control design. The goal of this paper is not to argue against adoption but to ensure that the controls surrounding adoption are adequate for the risk profile — which is distinct from, and more subtle than, the risks that current guidance addresses.

1.2.6 Legacy modernisation risk

Legacy systems often encode implicit trust boundaries in their rigidity — a COBOL program that crashes on a NULL field is enforcing, accidentally, the same fail-on-corruption principle that high-stakes systems require deliberately. When agents are tasked with modernising legacy code, they will seamlessly replace that rigidity with modern defensive patterns (null coalescing, optional chaining, default values), replacing the institutional knowledge baked into the old code's behaviour with standard defensive patterns that lack the same properties. The legacy system's implicit security properties are paved over with idiomatic, test-passing, wrong code. This inverts the Cloudbleed pattern (§1.2.3): where Cloudbleed's activation event was an upstream change that started exercising a dormant defect, legacy modernisation by agents creates the activation event by removing the accidental rigidity that suppressed the defect. The commit message says "fix: handle NULL gracefully," and the code path that used to crash on corruption now silently fabricates a default.

The risk is sharpened by a selection effect. The unmodernised legacy estate is not a random sample of old software — it often reflects decades of deferred replacement in domains where semantic error was judged more dangerous than technical obsolescence.16 The code's ugliness was load-bearing: its rigidity, its absence of graceful error handling, its refusal to proceed on ambiguous input were carrying unformalised security and integrity properties that were never written down because the code itself was the enforcement mechanism. Agent-assisted modernisation changes the economics of rewriting without changing the difficulty of proving that the rewrite preserves those institutional semantics.

Previously, "this is too expensive and dangerous to touch" acted as a brake — costly but effective. AI-assisted modernisation weakens that brake by collapsing the visible cost of rewriting before the invisible semantic risk has been addressed. Cheap generation, expensive wrongness. Organisations considering agent-assisted legacy modernisation should treat this as a control problem, not merely a cost-reduction opportunity: §9.6 poses the governance questions this creates.

1.2.7 Coding is no longer confined to developers

The preceding subsections address a volume problem — more code, faster, through recognised channels. This subsection addresses a different problem: agentic tooling is changing not only how fast code is produced, but who produces it. A business analyst generating plugins for a BI platform, an operations officer assembling workflow automations, or a policy team building internal tools with agent assistance may not regard themselves as software developers, yet they are producing executable logic that can affect trust boundaries, audit trails, access control, and data integrity.

Consider a familiar scenario: a business analyst — someone the organisation already trusts with direct database access — uses an agentic tool to build a data integration plugin for the team's reporting platform. The plugin works: it pulls records, transforms them, and populates a dashboard the team has wanted for months. What nobody outside the team realises is that the plugin holds open long-running queries during business hours and silently handles connection failures by writing partial results without any indication that data is missing. Three months later, an investigation traces intermittent database locks to a plugin nobody in IT knew existed, built by someone who never filed a change request because they did not think of what they had done as "software development."

There was no privilege escalation — the analyst already had the access. There was no negligence — they used a tool to do exactly the kind of work they were hired to do. Critically, the people most likely to reach for these tools are domain specialists — business analysts, database administrators, data engineers, operations staff — who are not developers but who typically hold more data access permissions than developers do, precisely because they are trusted to work directly with the systems they understand.

Every governance model examined in this paper — the ISM's software development controls, the SSDF's practice groups, IRAP assessment scoping — assumes that consequential code enters systems through recognised SDLC channels: repositories, pull requests, code review gates, CI/CD pipelines. When executable logic is produced by non-developers outside those channels, it bypasses the controls entirely — not through evasion, but because the governance perimeter was drawn around "software development" and the new production does not cross that line in any way the organisation recognises.

The problem is therefore not only that frontier engineering teams can generate reviewable artefacts faster than assurance processes can absorb them (a volume problem inside the SDLC), but that organisations increasingly contain many more software producers than their assurance processes recognise (a perimeter problem around the SDLC). One breaks the volume model. The other breaks the governance boundary model. Together, they are materially worse than either alone.

1.3 Terminology

This section provides compact definitions for key terms used in the paper's core argument. Additional terms are defined where they are introduced; full definitions with cross-references appear in Appendix H (Glossary). The companion specification referred to throughout is a separate document — Wardline Framework Specification: Semantic Boundary Classification and Enforcement — distributed with this paper and listed in the front matter. It is a proposed standard at draft stage, defining criteria that enforcement tools would need to satisfy — not a tool or product itself. Various vendors and open-source projects could implement tools that conform to its requirements, in the same way that SAST vendors build tools that implement CWE detection rules. It is included to demonstrate that the class of enforcement this paper calls for is technically feasible. It is a candidate approach, not the only possible one, and is not a prerequisite for the paper's conclusions.

Term Definition
High-stakes code paths Code paths where silent corruption, unverifiable state, or fabricated defaults are more dangerous than controlled failure. Defined by code-path properties, not system importance.6
Agent An AI system that generates or modifies source code with limited or no human intervention per output. This paper focuses on autonomous agents that produce correlated changes across a module or feature, not inline autocomplete.7
Agentic code Source code generated or substantially modified by an agent
Agent deployment spectrum The range from full human development (Level 0) through chat-pasted fragments (Level 1) and IDE-integrated agents (Level 2) to CI-integrated autonomous agents (Level 3). Appendix B provides a self-assessment framework.8
Trust boundary A point in a system where data crosses between different authority tiers — the crossing point, not the classification level on either side
Authority tier A classification of data based on what guarantees the system is entitled to assume about it. Tier 1 (authoritative internal — e.g., system-generated audit record), Tier 2 (semantically validated — e.g., API response after domain-rule verification), Tier 3 (shape-validated — e.g., API response after schema check only), Tier 4 (unvalidated external — e.g., raw user upload). Formally developed in §5; specified in full in the companion specification (Part I, §4).9
Validation boundary The mechanism that enforces a trust boundary — for agent code, a layered combination of conventional automated checks, semantic enforcement, and human review before codebase entry (§5.3)
Institutional / domain knowledge Security-relevant distinctions not expressed in the programming language — which paths must fail fast, which data is authoritative, what absence means. "Domain knowledge" emphasises problem-space semantics; "institutional knowledge" emphasises organisational rules and controls
Assurance stack The standard verification pipeline: linters, type checkers, SAST, DAST, unit tests, and peer review. This paper argues the assurance stack does not adequately detect semantic failure modes — see Appendix A
Semantic correctness Whether code behaviour is appropriate for its institutional context — not just syntactically valid and functionally correct, but correct for the specific system it operates in.10
Defensive anti-pattern Defensive programming patterns (.get() with defaults, broad exception handling) applied in high-stakes contexts where they are inappropriate.11
Bidirectional authority collapse Uniform defensive patterns collapse the authority-tier model from both ends: unvalidated data crosses inward as though validated; authoritative data is treated as negotiable (§2.2, §5.1)
Review-surface generation velocity The rate at which an agent produces plausible, convention-conforming code at a review boundary. Measures review burden, not developer output (§1.2.1)
Control law The operational state of a repository's machine-enforced controls — normal, alternate (degraded), or direct (offline). The term is borrowed from fly-by-wire aviation, where a degraded flight control law limits what manoeuvres the pilot may attempt; the analogy is that a degraded enforcement pipeline should limit what code changes a team undertakes.12 Determines what work is reasonable to undertake, not how mature the controls are.13
Automation bias The tendency to accept plausible-looking output without critical evaluation, amplified when the output is consistently well-formatted and convention-conforming (§4.2)
Habituation effect Progressive erosion of reviewer vigilance as repeated exposure to correct-looking agent output reduces the perceived need for scrutiny — a manifestation of the "Shifting the Burden" archetype (§4.2; Appendix G §G.3)
Machine-readable / machine-checkable / machine-enforceable Three escalation levels for encoding institutional knowledge: readable — the tool can parse the declaration; checkable — the tool can verify whether code conforms; enforceable — the tool can block non-conforming code at the repository boundary (§7.2)
Citizen programmer A non-developer who produces executable logic using agentic tools outside traditional SDLC channels (§1.2.7; Appendix C)

1.4 Methodology and scope of claims

This paper makes three kinds of claims, and the reader should be able to distinguish them:

  • Observed patterns are drawn from two sources: a simulation (§8.2, Appendix D) in which an agent prototyped an application from an underspecified brief, and approximately six months of direct experience with agentic development in a compliance-constrained environment (§8.3, Appendix E). The failure modes in the ACF taxonomy (Appendix A) were identified through observed agent behaviour, not theoretical analysis alone.

The taxonomy originated from a recurring anomaly: agent-generated test suites that were partially missing or structurally broken in ways that did not correspond to any obvious coding error. Root-cause analysis traced the failures to context compression during long generation sessions (ACF-R2, and the provisional ACF-R3). That recurring pattern — the same tool producing the same characteristic failure shape — prompted the broader question of whether agentic code generation had other systematic failure modes invisible to standard review, which led to the STRIDE-based analysis and the taxonomy that follows. Quantitative signals (e.g., the violation rate reported in §8.3) are drawn from a single project and should be read as illustrative, not as population-level statistics.

  • Analytical inferences extend observed patterns through structured reasoning. The STRIDE mapping (§3), the gap analysis (§6), and the compounding-effect argument (§3.3) are analytical — they apply established frameworks to observed phenomena. The conclusions follow from the analysis, but the analysis rests on a narrow empirical base.

  • Hypotheses for community validation are claims the paper advances as plausible but does not have the evidence to confirm. The model monoculture argument (§2.4), the cross-organisational correlation risk (§9.3), the review degradation dynamics (§4.2), and the governance perimeter expansion (§1.2.7) are hypotheses grounded in analogies to other domains or emerging adoption patterns but not yet validated in the agentic coding context specifically.

Epistemic status. This paper is best understood as pre-empirical — it advances a structured threat model grounded in observed patterns and analytical reasoning, but the empirical base is narrow (two case studies, both conducted by the author, using the author's framework — though with different agents from different vendors). The absence of widely reported incidents involving agent-generated semantic failures is genuinely ambiguous: it is equally consistent with the threat being real but undetected and with the threat being rare in practice.

The evidence problem has a bootstrapping structure: this paper argues for detection tooling and review discipline because the threat exists, but confirming the threat at scale may require the very tooling and review discipline the paper argues is missing. The case studies (§8) partially break this circle: the simulation demonstrates that an agent given an explicit high-stakes framing produces code with systematic semantic defects not targeted by the standard assurance stack; the longitudinal observation demonstrates that purpose-built detection finds violations at a non-trivial rate in sustained development. But both are from a single practitioner — pilot observations, not independent validation.

The paper's strongest claim is not that these exact failure rates generalise, but that current assurance frameworks are poorly shaped for this class of failure, even if prevalence turns out to vary materially by context.

Conditions that would substantially weaken the thesis. The ACF taxonomy and threat model would be undermined by evidence that:

  • Agents trained on general-purpose code corpora do not systematically produce defensive anti-patterns when generating code for high-stakes contexts;
  • Standard code review processes reliably catch semantic trust boundary violations in agent-generated code without purpose-built tooling or checklists; or
  • The failure modes described are artefacts of a specific model generation and do not persist across model updates.

These are the conditions that would dissolve the problem. None is straightforwardly testable — each requires either demonstrating a negative, measuring an uninstrumented capability, or longitudinal observation across model generations — but empirical studies targeting any of them would materially advance the discussion.

Two practical tests would meaningfully challenge the thesis without requiring those horizon conditions: a practitioner deployment test (deploy ACF-pattern detection rules on a codebase with active agent use and measure the violation rate) and a reviewer catch-rate test (present reviewers with agent-generated code containing known semantic violations and measure detection rates with and without the review aids this paper proposes). §8.7 describes both tests in detail with a full replication protocol.

1.5 How to read this paper: three analytical traditions

This paper's analysis draws on three traditions that may not be equally familiar to all readers. Appendix G provides a fuller primer with worked examples.

Safety engineering, not (only) security engineering. The threat described here is not adversarial — there is no attacker. It is an emergent property of a complex system in which a capable-but-context-blind generator interacts with a volume-constrained review process. The paper's recommendations reflect this: barriers and interlocks rather than access controls, degradation modes rather than perimeter defence, environmental constraints rather than detection-and-response (§2.2; Appendix G §G.1).

Systems thinking. The most important systems pattern is the habituation effect (§4.2): agent surface quality degrades review quality, which allows more defects through, which produces no visible incidents, which reinforces trust in agent output. The paper also uses the "Shifting the Burden" archetype, stock-flow reasoning about review capacity, and a leverage-point hierarchy when arguing that technical controls are stronger than behavioural controls. Readers unfamiliar with these concepts will follow the narrative without difficulty; those who want the analytical machinery exposed should read Appendix G before §4.

Levels of intervention. The paper's recommendations span from parameter adjustments ("add review checklists") through structural changes ("insert a CI gate") to paradigm shifts ("treat agent output as untrusted input"). The leverage hierarchy is developed in Appendix G §G.5. The most important practical implication: treating agent output as untrusted input is not one control among many. It is the paradigm shift on which the others depend.

1.6 Relationship to existing work

Several recent taxonomies classify failure modes in LLM-generated code. Wang et al. (ICSE 2025) developed a two-dimensional taxonomy from 558 incorrect solutions across six LLMs.26 Gao et al. (2025) provide a comprehensive survey synthesising bug categories including semantic bugs, API misuse, and hallucination.27 The CSET brief (Georgetown, 2024) identifies cybersecurity risk categories for AI-generated code at the policy level.28 Chen et al. (2024) survey 67 papers on code language model security.29 The OpenSSF published security-focussed instruction templates for AI code assistants (2025).30

These contributions address important dimensions of the problem. The ACF taxonomy in this paper addresses a different dimension.

The existing taxonomies primarily classify code that fails detectably — wrong outputs on test cases, known vulnerability patterns (CWEs), compilation errors, API misuse. The ACF taxonomy classifies code that is semantically wrong for its institutional context — and for the highest-risk entries (ACF-T1, ACF-E1, ACF-R2, ACF-R5, all Critical or High), no widely deployed tool detects the failure. Other entries describe known vulnerability classes whose management burden changes under agentic volume (ACF-R1, ACF-I1, ACF-E2) or whose structural pattern is detectable with low precision (ACF-S1 via Semgrep). The taxonomy's contribution is sharpest for the entries that pass every existing check; it also provides institutional-context framing for entries that existing tools partially address.

The .get() example in §2.3 — defaulting a missing security classification to "OFFICIAL" — does not sit comfortably within existing bug taxonomies because, under conventional definitions, it presents as correct defensive code. The distinction is between code-level correctness (which existing work addresses well) and institutional-context correctness (which this paper addresses).

Several findings in the existing literature directly support this paper's argument. Wang et al.'s observation that most LLM-generated errors are compilable and runnable confirms that agent-generated defects evade the standard assurance stack — the code is not obviously broken, which is precisely what makes it dangerous in high-stakes contexts. Gao et al.'s "semantic bug" category is the closest neighbour to the ACF taxonomy's territory but does not develop the institutional-context dimension or the authority-tier model that distinguishes a harmless default from a catastrophic one. The CSET brief identifies an "over-reliance" risk that maps to this paper's habituation effect (§4.2) but treats it as a behavioural risk rather than the systems-theoretic feedback loop developed in §4.2 and Appendix G.

The OpenSSF guide warrants specific mention because it is the most practically proximate work. This paper's proposal to convert institutional knowledge into machine-enforceable rules and the OpenSSF's security-focussed prompt templates address complementary layers of the same defence-in-depth stack: the OpenSSF guide addresses what to tell the agent at generation time; this paper addresses what to verify at the integration boundary. They are different control points, not competing approaches.

The policy gap analysis in §6 examines governance frameworks (ISM, NIST SSDF, Essential Eight, OWASP) that claim authority over the software development lifecycle and finds them insufficient for agentic failure modes. The academic literature reviewed here is characterising the problem, not claiming to govern it. Both the governance frameworks and the research literature leave the same gap — institutional-context semantic correctness — which is the gap this paper's taxonomy, threat model, and control recommendations address.

1.7 A note on provenance

This paper was developed using the same class of agent-assisted workflows it examines. The threat model, STRIDE mapping, ISM gap analysis, and ACF taxonomy are grounded in approximately six months of daily agentic development on a compliance-constrained system, combined with structured AI-assisted analysis used to test alternative frames and challenge assumptions. The intelligence analysis tradecraft reflects the author's professional background; the domain-specific security analysis reflects prompted polymorphic review of the kind described in §7.2.

This creates an obvious limitation: the document is itself an example of agent-assisted output and should be read accordingly. It is offered as a discussion paper for scrutiny and refinement, not as validated guidance for adoption (§1.4). The paper's value lies in the observed patterns, working vocabulary, and candidate control concepts it brings forward for wider testing, rather than in any claim to finality.

The prior work surveyed in §1.6 was identified during consultation, not as the starting point for the analysis, which explains both its placement and the contribution's independence from the existing literature. The paper's purpose is not to establish final controls but to provide a sufficiently structured problem statement — with candidate vocabulary, threat model, and initial control concepts — that the bodies responsible for whole-of-government security control development can validate, refine, or set aside through broader empirical work. The structural gaps it identifies are cross-government by nature and are appropriately addressed at the cross-government level.

2. The Threat Is Not What You Think

This section establishes the analytical framework: what the threat is, why it differs structurally from the supply-chain model, and what makes it dangerous. Policy readers who accept the Executive Summary's framing may skim §2.1–§2.2 and focus on the concrete example in §2.3.

2.1 The intuitive threat model (incomplete)

When organisations evaluate the risk of AI-generated code, the intuitive threat model is straightforward:

"The AI might write malicious code — backdoors, data exfiltration, supply chain attacks."

This threat is real but well-understood. It maps directly to the existing software supply chain threat model with a faster generator. Existing controls — code review, static analysis, dependency scanning, penetration testing — address it, albeit with increased volume pressure.

2.2 The insidious threat model

This section focuses on autonomous and semi-autonomous agents as defined in §1.3 — not inline autocomplete, which produces isolated suggestions within a human-directed editing session. While autocomplete introduces volume, agents produce correlated errors across modules because a single session generates multiple interdependent functions from the same context and training biases. The distinction matters: the threat model below depends on correlation, scale, and review-pipeline pressure that autocomplete does not produce to the same degree.

The threat applies to specific code paths within systems, not to entire systems or sectors end-to-end — the same system contains both high-stakes and non-high-stakes paths, and agents apply defensive patterns uniformly across both.

The more dangerous threat is subtler:

"The AI writes code that follows patterns generally regarded as good practice — defensive, robust, convention-conforming — and applies those patterns uniformly, including in the high-stakes contexts where they are catastrophic."

Even "plausible-but-wrong" understates the problem. In many cases, the code is not merely plausible — it is conventionally reasonable by the standards of the vast majority of software. Defensive programming is considered good practice, and agents are exceptionally good at it. The problem is not that agents produce sloppy work. It is that they produce excellent work calibrated to the wrong context. A high-availability emergency dispatch system is high-stakes on its data-integrity and audit paths but correctly uses defensive programming on its service-continuity paths — the same system contains both. Conversely, a consumer application that "does not need to be high-stakes" still has authentication, payment handling, and audit logging paths where silent corruption is unacceptable. Agents apply defensive patterns uniformly across paths that require different failure semantics, and no tool in the standard assurance stack distinguishes one from the other.

Even in ordinary software, defensive patterns routinely conceal significant bugs that no one ever finds. In high-stakes systems, the same concealment is catastrophic: defensive patterns become defensive anti-patterns.

This threat is distinct from the supply chain model in four critical ways:

It is not adversarial. The agent is not trying to compromise the system. It is producing its best output based on training data that is overwhelmingly composed of open-source code without the properties high-stakes systems require.32 The agent reproduces the patterns it learned — which are the patterns that represent good practice in the vast majority of non-high-stakes software.

A note on analytical framing. This is structurally closer to a safety engineering problem than a security engineering problem: an emergent failure arising from components working as designed rather than an adversary acting against the system. The distinction explains why the paper's recommendations look more like safety controls (barriers, interlocks, degradation modes) than security controls (access control lists, encryption, signature verification). Appendix G §G.1 develops this framing.

It largely falls outside existing detection — not because tools merely need better rules, but because no standard tool category is designed to detect it. The generated code is syntactically valid. It passes type checkers, linters, and unit tests. It follows project conventions (agents are good at pattern-matching the surrounding codebase). By the automated measures most organisations currently rely on, it presents as "correct code." The failure is semantic — the code does the wrong thing in the high-stakes context while doing the right thing in every other context. Each tool in the stack addresses a different structural property,31 but none of these can determine whether a .get() default is institutionally appropriate, whether an exception handler preserves an audit trail, or whether data crossing a trust boundary has been validated. Catching these failures requires a new category of automated verification — one that encodes system-specific invariants and domain semantics as enforceable rules — and that category does not yet exist in standard tooling.

Why no one built it. The gap is not an oversight. Semantic correctness — understanding what the code should do in its institutional context — has traditionally been carried by the human author. Tools were built to check syntax, structure, and known defect patterns, not whether a system was implementing the right institutional meaning. Until now, a human author was usually in the loop at the point of creation as well as review, and that author supplied the context that determined whether a default was helpful or catastrophic.

The agent does not reliably carry that context, and the volume means the reviewer is now being asked to reconstruct it after the fact across far more output than review processes were designed to absorb.

The absence of reported incidents does not imply absence of impact — and this model explains why. To be clear: "no one has reported this problem" is not, by itself, evidence that the problem does not exist. But the failure modes described in this paper — silent data corruption, trust boundary violations masked by defensive patterns, audit trails that record fabricated defaults as real values — are specifically the kind that do not produce observable incidents. A traditional vulnerability creates a detectable event: a crash, an intrusion alert, an anomalous log entry. A .get() that silently returns "OFFICIAL" for a missing classification field produces no crash, no alert, and a log entry that looks entirely normal. The system continues operating with a confident wrong answer.

The question is not "has this caused a breach?" but "would we know if it had?"

For organisations that lack semantic boundary enforcement tooling (§7.2), the answer depends entirely on the diligence, domain expertise, and sustained attention of their human reviewers — traits that degrade predictably as the volume of plausible, convention-conforming code increases (§4.2). The estimated violation rate reported in §8.3 — from a single project with such tooling — suggests the phenomenon is occurring at a non-trivial rate that would be invisible without purpose-built detection. This is not a claim that all projects face the same rate; it is evidence that detection requires purpose-built tooling that most projects do not yet have. Whether that rate generalises beyond the observed project is an open empirical question (see §1.4, replication protocol).

It scales with the benefit. The faster agents generate code, the more good-practice-in-the-wrong-context code enters the review pipeline. The same velocity that makes agents productive makes them dangerous — and the systems where the stakes are highest are often the systems with the most domain-specific security context that agents lack, the most compliance overhead that generates review fatigue, and the most code volume from modernisation and remediation efforts. The benefit and the risk are the same mechanism, and they concentrate in the same places.

The control problem is not one of review effort but of review type. This paper is not arguing for a return to artisanal code review by exhausted humans peering harder at larger diffs. The standard assurance stack is well-shaped for code that is obviously broken, stylistically irregular, or known-vulnerable. It is poorly shaped for code that is locally correct-looking yet semantically wrong for a specific trust context — and that is precisely what agents produce. This paper's proposed response is to build the missing layer: checks that can determine whether a default is fabricating authoritative data, whether an error handler preserves or destroys the audit trail, and whether data has crossed a trust boundary without validation. Those questions require encoding the system's security-relevant distinctions — which paths are fail-fast, which data is authoritative, where the trust boundaries lie — in forms that tooling can act on. Human review remains necessary for adjudicating meaning and exceptions, not as the primary detection mechanism. The response landscape is developed in §7.

A necessary clarification on "defensive" vs. "offensive" programming. This paper does not argue against defensive programming. It argues that the same system contains both kinds of path — an emergency dispatch system should degrade gracefully on a malformed UI field but should not degrade gracefully on a corrupted incident record — and LLMs do not reliably distinguish which is which. The result is uniform defensive behaviour across paths that require different failure semantics (formalised as four coding postures in the companion specification, §4.1).33

Crucially, this collapses the authority model at both ends: unvalidated external data is given more authority than it has earned, while authoritative internal data is treated as more negotiable than it is allowed to be — simultaneously too permissive at the perimeter and too casual at the core. §5 formalises this as bidirectional authority collapse under the authority-tier model.

2.3 A concrete example

Consider a government system that processes security classifications:

# Best: fail fast with diagnostic context
def get_document_classification(record):
    if "security_classification" not in record:
        raise DataIntegrityError(
            f"Missing security_classification for document {record.get('id', '?')}. "
            f"This is a data integrity failure — investigate the source system. "
            f"Fields present: {sorted(record.keys())}"
        )
    return record["security_classification"]
    # The error message is the incident response runbook.

# Acceptable: bare access, but poor diagnostics
def get_document_classification(record):
    return record["security_classification"]

# Agent-authored (plausible, test-passing, catastrophically wrong)
def get_document_classification(record):
    return record.get("security_classification", "OFFICIAL")
    # This "works" — no crash, no error, tests pass.
    # The default does not cause a misclassification on its own.
    # But if an upstream fault ever drops the classification field,
    # the crash that would have caught it is now gone — a PROTECTED
    # document silently becomes OFFICIAL and is treated accordingly.

The three versions illustrate a spectrum. The fail-fast version turns a missing field into an actionable incident — the operator knows which document, what data was present, and what to investigate. The bare access version at least crashes, which is correct behaviour for a data integrity failure, but the operator gets a generic KeyError with no diagnostic context. The agent-generated version is the worst outcome: it does not crash when the field is absent, silently fabricating a classification that downstream access control decisions will treat as authoritative if an upstream fault ever removes it.

The .get() default does not, by itself, cause a document to be misclassified. Under normal operation — when the classification field is present — all three versions produce the same correct result. The danger is latent: the default converts a future upstream fault (a dropped field, a schema migration, a serialisation bug) from a visible crash into a silent downgrade. The first two versions would surface the fault immediately; the third absorbs it. This is not a "security hole" in the conventional sense — it is the removal of an ad hoc safety net (a crash that would have surfaced the fault). The violation rate reported in §8.3 counts these latent removals, not active breaches.

The agent-generated version:

  • Is syntactically valid Python
  • Passes every unit test (the default prevents even downstream and integration tests from detecting a problem)
  • Follows the .get() pattern that appears in millions of Python files in the agent's training data
  • Would pass an ordinary code review unless the reviewer is specifically trained to recognise this class of failure
  • Silently absorbs upstream data integrity failures that the first two versions would have surfaced — converting a detectable crash into a silent downgrade

The following table illustrates the same defensive programming pattern — detect missing data and substitute a reasonable default — across three domains. The code pattern is identical in each case. The consequences diverge because the semantic meaning of absence differs.

Stage Consumer application Government security Clinical safety
User action User sets location in a weather app Employee assigns classification before sending an email Patient reports penicillin allergy
Underlying data state Location stored in app preferences Classification stored in message metadata Allergy status stored in patient record
Failure event App crash corrupts configuration Network or processing fault drops classification field Storage fault corrupts allergy record
Defensive behaviour App detects invalid configuration and substitutes location from device GPS Mail system detects missing classification and inserts a default (OFFICIAL) System detects missing allergy field and substitutes empty list ([])
Outcome User sees correct weather despite configuration loss Classified material is transmitted at the wrong classification level Clinical decisions proceed as though no allergy were recorded
Semantic meaning of absence Missing preference — recoverable from another source Missing classification — integrity failure requiring investigation Missing allergy data — unknown status, not a negative finding
System impact Harmless convenience Confidentiality breach through silent downgrade Patient safety risk through fabricated clinical knowledge

The first column is correct defensive programming. The GPS fallback genuinely recovers the missing information from an alternative evidentiary source — the user's location is knowable independently of the corrupted configuration. The second and third columns apply the same logic where it is catastrophically wrong, because the missing value cannot be recovered by substitution. It can only be investigated or surfaced as explicitly unknown.

The core failure in both dangerous cases is a category error between program state and domain state. In program terms, a missing field is merely an absent value. In domain terms, a missing classification is an integrity failure, and a missing allergy record is not a negative finding but an unanswered question. Defaulting the field converts an unanswered question into a confident answer, and downstream systems then treat that fabricated answer as authoritative data.

This is the distinction agents do not reliably make. The .get() with a default, the COALESCE() with a fallback, the Optional.orElse() with a substitute — the syntax is identical across all three columns. The difference requires both kinds of knowledge that are absent from the code: domain knowledge (a missing allergy record is an unanswered clinical question, not a negative finding) and institutional knowledge (this system must surface the absence as an explicit unknown, not paper over it with a default). Neither is expressed in the syntax, rarely in the type system, and only inconsistently in training data. The knowledge lives in the operational meaning of the system itself — and agents do not have access to it unless it is encoded in machine-readable form (§7.2).

The lifecycle of a latent defect. The danger of the agent-generated version is not that it immediately produces wrong results — under normal operation, the classification field is present and all three versions behave identically. The danger is latent: the default converts a future upstream fault into a silent downgrade. A team uses AI to build a reporting query. The AI fills in the missing-field path with a default value — "OFFICIAL" — because that is what millions of codebases in its training data would do. The query runs for weeks. No error is ever raised. Then an upstream fault — a device failure, an integration error, a serialisation bug — corrupts the security classification on a handful of records, leaving the field empty. The code that should have flagged the missing data instead silently defaults those records to OFFICIAL. PROTECTED documents are now processed at the wrong level — and nothing in the system distinguishes them from records that were always OFFICIAL. An audit eventually finds the discrepancy, but by then the corrupted records have propagated through downstream reports. The AI did not introduce an active security hole — it removed an ad hoc safety net. The crash that would have surfaced the upstream fault was replaced with a silent default. The structural weakness was latent for weeks; the incident required a second failure to activate it, and when that failure arrived, nothing caught it.

The first two code versions would have caught the upstream fault immediately — a DataIntegrityError or a KeyError. The third absorbed it.

The example is fictitious but not contrived. Defaulting to the lowest classification is exactly a reasonable early-delivery implementation — particularly if the system is initially scoped to handle only OFFICIAL data, or if the requirement to support higher classifications is documented in a backlog item rather than enforced in the schema. The agent produces this pattern because it is reasonable practice in the vast majority of codebases in its training data. The danger is that the default survives into a context where the system now handles PROTECTED material, the schema still permits missing values, and no one revisits a function that "works." This is the pattern that defensive programming produces by default in Python, and agents are trained on defensive Python.34

Human developers produce this pattern too — through drift, copy-paste, or stale assumptions. The difference is that a human who defaults a field such as allergies has typically failed to apply context that was in principle available to them. An agent may generate the same pattern without the domain context remaining active at the point of generation — and because the model is opaque, the reviewer cannot tell which occurred. The reviewer sees only the artefact. In transformer-based systems, the issue is not merely whether the relevant context appears somewhere in the prompt, but whether it remains active in generation rather than being displaced by stronger local statistical cues. For a human, this is usually a failure to respect context. For an agent, it may be a failure to retain context at all.

2.4 What is fundamentally different about agentic code

The threat model for agent-generated code is not simply "human-authored code but more of it." Several properties are qualitatively different:

(a) Limited persistent learning

A human developer who receives review feedback on a trust boundary violation learns from it and is less likely to repeat the mistake. Agents have limited or no persistent memory across sessions. The practical consequence is stark: the agent is not circumventing project rules, and it is not ignoring instructions — it followed them perfectly in the last session. It simply does not have a last session. Every invocation is the first day on the job,37 and on the first day, the agent writes .get() with a default because that is what Python looks like in the training data. Transcript evidence (Appendix E, §E.7) suggests the mechanism is more precise than uniform forgetting: agents appear to apply some constraints structurally (as automatic checks) while treating others as conventions (applied when salient but displaced under context pressure). The underlying mechanism is selective constraint prioritisation — a harder problem than uniform context loss, with different intervention implications, because it means session resets do not reliably restore the constraints the agent deprioritised.

Some agent frameworks now support project-level instructions (system prompts, documentation files, memory stores) that provide partial mitigation. An agent can be told "do not use .get() on audit data" and will follow that instruction within a session. But these are explicit rules, not internalised judgment. The agent cannot generalise from "do not use .get() on audit data" to "do not fabricate defaults anywhere that data absence is meaningful" unless that generalisation is also spelled out.

Every correction must be encoded as a rule; the agent does not learn the principle behind the correction. This means that review feedback improves the generator only to the extent that it is captured as machine-readable rules — and the coverage of those rules is always trailing the set of possible failure modes.

This has a direct consequence for governance. Current assurance frameworks assume that corrective action is durable — identify a defect class, train the developer, and the problem stays fixed. With agent-generated code, every correction lasts only until the next session. With human developers, training works: teach someone not to do X, and they stop doing X. With AI tools, every correction expires when the session ends. The defect is not fixed; it is caught — and it must be caught again tomorrow, and every day after that, for as long as the tool is in use. The durable intervention is not training the developer but encoding the detection as an automated rule. This shifts the governance model from "train and trust" to "detect and enforce" — a fundamental reorientation that §7.2 develops in detail.

The same problem compounds in multi-session and multi-agent workflows: each handover crosses a context boundary, and any agent output that defers action to a future session is making a continuity assumption that the architecture does not guarantee. A reviewing agent that triages findings as "fix during implementation" implicitly assumes the implementing agent will have the review findings — but the implementing agent starts a fresh session with the specification as input, not the review. Unless the review findings are written into the specification, the deferred items are silently dropped — the artefact looks complete, the triage looks prioritised, and the gap is invisible until the implementing agent reproduces the exact patterns the review flagged. Appendix A catalogues this as a provisional candidate mode: ACF-R4 (Context Handover Assumption).

(b) Consistent surface quality

Human code has variable surface quality — hasty code looks hasty, careful code looks careful. Reviewers use surface quality as a signal for where to focus attention. Agent code has uniformly high surface quality regardless of semantic correctness — and worse, the dangerous patterns follow the same conventions that reviewers are trained to approve: a well-structured .get() with a sensible default, a clean try/except with logging. A function with a critical trust boundary violation looks exactly as polished as a function without one. The reviewer's natural calibration signal — "this code looks sloppy, I should look more carefully" — is absent, and the surface quality actively works against scrutiny.

(c) Pattern completion, not intent

A human developer writing record.get("security_classification", "OFFICIAL") has either made a deliberate design decision (the default is intentional) or made an error (they did not think about the missing-field case). The distinction is visible in context — comments, commit messages, design docs. An agent writing the same code is completing a pattern from training data. The agent has no design intent. There is no commit message that explains why the default is correct, because the agent did not decide it was correct — it predicted it was the next likely token.

Intent-based review ("why was this written this way?") is insufficient for agent code — the agent has no design intent to interrogate, so the traditional mechanism for distinguishing deliberate decisions from errors is unavailable. Context from architecture documents, interface contracts, preceding commits, and system prompts can inform outcome-based review, but the review question must shift from "why did the author write this?" to "is the behaviour correct for this context?"

(d) Correlated failure modes

When ten human developers write code for a system, their errors are largely independent — different people make different mistakes. When an agent generates ten functions, its errors are correlated — the same training data biases produce the same failure modes repeatedly. A single systematic bias (e.g., "always use .get() with a default") produces correlated vulnerabilities across the entire codebase. This is not the independent-error model that code review and testing strategies are designed for.

The correlation extends beyond individual codebases. When a government agency and its five contracted suppliers all use the same AI tool — or different tools trained on substantially overlapping data — they all get the same blind spots. This is not five independent risks. It is one risk expressed five times. A vulnerability class that occurs sporadically in human-authored code becomes a widespread pattern when every AI tool produces it identically, and a single class of defect can be present across multiple systems simultaneously, discovered simultaneously, and — if adversarially targeted — exploited simultaneously. This is a qualitatively different risk profile from the diverse, uncorrelated mistakes that existing assurance frameworks were designed to manage.

(e) No fatigue, no shortcuts, but also no judgment

Agents do not get tired, do not take shortcuts under deadline pressure, and do not introduce bugs from distraction. But they also do not exercise judgment about which patterns are appropriate in which contexts. A human developer who is tired might introduce a bug in one function; an agent that lacks context will introduce the same incorrect pattern in every function it generates. The failure mode is not degradation under pressure — it is systematic misapplication of context-inappropriate patterns.

(f) Task-frame reconstruction under context pressure

The preceding properties describe agents applying patterns without contextual judgment. This property describes agents producing output consistent with a different understanding of the task than the one originally specified. When an agent operates under context pressure — long sessions, compacted history, multi-step plans that exceed the context window35 — it does not simply forget earlier steps. It produces output consistent with a coherent but shifted task frame, and the shifted frame may not match the original plan. The term "reconstruction" describes the observable pattern — output that is internally consistent with a task definition the agent was not given — rather than claiming a specific cognitive mechanism.36

The practical consequence is observable in a specific code pattern: tests that were written to verify real behaviour are "fixed" by replacing real dependencies with mocks that return expected values. The agent's original task frame was "implement this integration and test it." Under context pressure, the frame shifts to "make this test pass." In the new frame, replacing a real API call with a mock that returns {"status": "ok"} is a legitimate fix — the test passes, the CI is green, the task is complete. The test is now proving that the mock works, not that the integration works. The agent has not suppressed an error or fabricated data — it has resolved the problem by redefining what problem it is solving.

This is distinct from spurious field access (ACF-S2), where the agent's model of the code is wrong. Here the agent's model of the code is internally consistent — it is the agent's model of the task that has shifted. A human developer who replaces a real dependency with a mock knows they are writing a unit test, not an integration test, and can explain the trade-off. An agent that has reconstructed its task frame cannot make that distinction, because in its current frame, the mock is the implementation.

The detection signature is specific: tests where real objects have been replaced with mocks or stubs that remove the test's ability to catch the failure it was originally written to detect. The risk is highest when the test and the code under test were written by the same agent in the same session, because the agent's shifted task frame shapes both the implementation shortcut and the test that validates it. Experienced practitioners can learn to recognise this pattern, but institutions have not yet encoded it into tools or review doctrine — it is pre-formalised rather than undetectable. The broader implication is that agent-generated test suites under context pressure may systematically verify that workarounds function rather than that the real system functions. Appendix A catalogues this as ACF-R3 (Verification Displacement).

(g) Model monoculture

The correlated failure problem described above operates within a single codebase, but it extends further. If most government agencies adopt the same two or three models for code generation, the correlated failure modes are no longer contained within individual organisations. A systematic bias in a widely-used model — say, a persistent tendency to use .get() with defaults on security-critical fields — will produce the same vulnerability pattern across every codebase that model touches. This is analogous to agricultural monoculture38: genetic uniformity makes the entire crop vulnerable to a single pathogen. Discovering a systematic agent-introduced defect pattern in one agency should trigger cross-agency scanning, because the same model likely introduced the same pattern elsewhere. This strengthens the case for cross-organisational standards (§9.3) and shared vulnerability disclosure mechanisms for agent-introduced defect patterns.

Model diversity is the obvious mitigation to monoculture, but it should not be assumed to provide independence. Even where organisations use different models, overlapping training distributions and shared lineages may preserve correlated tendencies. A further possibility — not yet empirically demonstrated — is that distinct defects from different models could compose: Model A generates a data ingestion function that omits the validation boundary between external and internal data (ACF-T1); Model B, working on the downstream processing code, encounters the now-unvalidated data and adds a .get() default to handle its potential absence gracefully (ACF-S1). Neither model produced the other's defect, but Model B's "fix" converts Model A's loud failure (a crash on missing validation) into a silent one (a fabricated default on unvalidated data). The composed outcome would be worse than either defect alone.

This composition scenario is illustrative, not observed — Appendix F explores it as a precautionary analysis. The core policy point is narrower: reducing same-model dependence may not eliminate correlated risk, and the possibility of cross-model defect composition warrants investigation even though empirical demonstration is not yet available.

(h) Distinguishing failure layers

The preceding properties describe two structurally distinct failure mechanisms that produce superficially similar outcomes through different causes.

Context collapse during generation: the model begins a task with adequate context but loses or displaces it during a long session. The resulting code is wrong because the model's task frame drifted, not because its training priors are biased. Controls that restore context — state resets, fresh-session regeneration, checkpoint-and-resume workflows (§9.13) — address this layer.

Training-distribution bias: the model's priors encode defensive patterns as universally correct because its training corpus is overwhelmingly composed of code where they are correct (§2.5). This bias persists across sessions, across models with overlapping training lineages, and across any number of state resets. Controls for this layer require genuine diversity (different model families, different training corpora, symbolic cross-checks, machine-enforceable rules) rather than redundant generation from the same distribution.

The distinction matters for control selection. A control that works against context collapse (regenerate from a fresh session) may provide false reassurance against training-distribution bias (the fresh session reproduces the same wrong answer). Organisations should assess which layer a given control addresses and avoid assuming that redundancy in one layer provides coverage for the other. Appendix F (§F.3) observes that the effective number of independent model lineages is likely much smaller than the number of available models — a qualitative argument that warrants empirical measurement but whose policy implication is clear: the assumption that "different agencies use different models, so we are safe" does not hold without lineage-independence analysis.

2.5 Why training data is a major part of the story

Defensive patterns are common in open-source Python code: .get() with defaults, getattr() with fallbacks, broad try/except blocks, and or chains that silently substitute values. These patterns are widely regarded as good practice because, in most applications, graceful degradation is preferable to crashing. Even there, they can conceal real bugs behind silent defaults — the web application that defaults a missing country to "US" works fine until someone asks why the analytics show 40% of users in the United States when the service only operates in Australia. This is often a nuisance in ordinary software. In high-stakes systems, the same concealment mechanism is catastrophic.

These patterns are catastrophically inappropriate for applications where:

  • Silent data corruption is worse than a crash
  • Every decision must be traceable to its source
  • The absence of data is itself evidence (not an invitation to fabricate a default)
  • Error paths must be as auditable as success paths

Government systems handling classified information, financial records, health data, or law enforcement evidence contain code paths that fall squarely into this category — paths where defensive programming is the wrong pattern. The agent does not reliably know this, and cannot infer it from the code alone — the security context is institutional knowledge, not syntactic structure. And critically, the agent is not doing anything wrong. It is applying the patterns that represent good practice in the overwhelming majority of the code it was trained on. The problem is not the agent's quality — it is the mismatch between what "good practice" means in most software and what it means in this software.

This training-data root cause has a self-reinforcing property that warrants attention. Agent-generated code is already entering open-source repositories at scale (§1.2.2) — the same repositories that form the training corpora for the next generation of models. If that code carries the defensive-pattern bias described above, the training data for future models will contain a higher proportion of defensive patterns than the training data for current models — not because the patterns became more appropriate, but because the generator that over-applies them is now a significant contributor to the corpus. Each model generation trained on a corpus increasingly populated by previous-generation output would, on this trajectory, have a stronger prior toward the same context-inappropriate patterns.

This is a reinforcing feedback loop (Appendix G, §G.2) operating at the model-generation timescale rather than the session timescale: the output of one generation strengthens the bias of the next.

The magnitude of this effect is not yet measurable — it depends on what proportion of training corpora is agent-generated, how training pipelines filter or weight contributions, and whether model developers implement countermeasures. But the structural mechanism is clear, and it strengthens the case that the training-distribution bias identified above is not a transient artefact of current model limitations but a condition that may deepen as agents become a larger share of the code production ecosystem. Organisations should not assume that future model generations will naturally outgrow the defensive-pattern bias; they may inherit a reinforced version of it.

Corpus composition is not the only mechanism reinforcing this bias. Alignment training — reinforcement learning from human feedback (RLHF) and equivalent techniques — provides a distinct causal channel. Models trained to produce code that looks helpful, passes tests, and generates reviewer approval will actively reinforce defensive patterns, because those patterns satisfy all three reward signals: they do not crash, they pass unit tests, and they appear professional to reviewers. A .get() with a sensible default is precisely the kind of output that RLHF rewards — it is helpful, safe-looking, and non-disruptive. The alignment reward signal thus reinforces the same bias that corpus composition creates, through a different mechanism. The distinction matters for mitigation design: the fix for a corpus composition problem (exposure diversity, data curation, domain-specific training) is different from the fix for an alignment problem (reward signal redesign, constitutional approaches, domain-context-aware evaluation). Current models are subject to both mechanisms simultaneously, and a mitigation that addresses one may provide false reassurance about the other — the same "different failure layers require different controls" principle described in §2.4(h).

3. STRIDE Applied to Agentic Code Output

This section is a structured threat enumeration — it applies STRIDE to the agentic development workflow to catalogue what can go wrong. Technical practitioners and IRAP assessors will find the category-by-category mapping most useful; policy readers may prefer the compounding scenario in §3.3, which illustrates how the individual failure modes interact.

3.1 Framework selection

STRIDE (Spoofing, Tampering, Repudiation, Information Disclosure, Denial of Service, Elevation of Privilege) is the established threat modelling framework used in Australian government security assessments.52 Applying it to agentic code output uses a structured vocabulary already familiar to policy audiences, rather than introducing unnecessary new terminology.

The standard artefact from a threat model is a data flow diagram (DFD): a schematic showing processes, data stores, data flows, and trust boundaries. The DFD for the agentic development workflow — the system this paper threat-models — is:

graph TD
    TC["Training Corpus<br/>(external)"] --> MODEL["AI Model<br/>(external system)"]
    MODEL -->|"generated code<br/>(untrusted)"| CONV

    subgraph VB ["VALIDATION BOUNDARY"]
        CONV["Conventional checks<br/>(lint, type, test, SAST)"] --> SEM["Semantic enforcement<br/>(authority tier flow,<br/>audit, defaults)"]
        SEM --> HR["Human review<br/>(meaning, exceptions,<br/>architecture)"]
    end

    VB -->|"rejected"| REJ["Rejected<br/>(quarantined)"]
    VB -->|"validated"| REPO["Repository<br/>(trusted store)"]
    REPO --> CICD["CI/CD Pipeline"]
    CICD --> PROD["Production System"]

    style MODEL fill:#fdd,stroke:#333,stroke-width:2px
    style VB fill:#efe,stroke:#393,stroke-width:3px
    style REJ fill:#fbb,stroke:#333,stroke-width:1px
    style REPO fill:#bfb,stroke:#333,stroke-width:2px

Diagram description (accessibility): An AI model, informed by an external training corpus, produces untrusted generated code. That code passes through a validation boundary consisting of three sequential layers: conventional checks (lint, type, test, SAST), semantic enforcement (authority-tier flow, audit trail, defaults), and human review (meaning, exceptions, architecture). Code exits either as rejected (quarantined) or as validated into the repository (trusted store), from which it proceeds through CI/CD into production.

The STRIDE analysis that follows applies each threat category to the components and flows in this diagram. The most important observation from the DFD itself is that the AI model is an external system producing output that crosses a trust boundary into the repository. This is the same structural position as any external data source, and it warrants the same boundary discipline (§5) — and this observation is visible from the DFD alone, before any STRIDE enumeration.

The application below extends STRIDE to treat agent-generated code as an input to the system — analogous to treating user input as untrusted. The agent is not an adversary, but its output has the same authority properties as any external input: it may be well-formed, it may be reasonable, but it has not been validated against the system's security requirements.

Two of the six categories — "fabricated default" (S) and the process-level DoS entries (D) — are analogical extensions of STRIDE's original technical-system categories to development-process analysis, proposed here as candidate extensions rather than presented as standard applications. STRIDE-LM (adding Lateral Movement)53 and the emerging ASTRIDE variant (adding AI Agent-Specific Attacks)54 provide precedent for such extensions, but the reader should understand these as novel mappings rather than established STRIDE doctrine.

A methodology note on the STRIDE mapping — which categories are direct and which are analogical extensions — appears after the six entries below. Readers unfamiliar with STRIDE may wish to read it first.

3.2 Threat categories

S — Spoofing: Competence and identity spoofing46

Agentic variant: Code appears to handle data correctly but operates on fabricated or default values, presenting a false picture of data integrity.

Mechanism: Agents default to defensive patterns that substitute values rather than failing. The code "works" — it produces output, it does not crash — but the output is based on fabricated data rather than actual data. The code spoofs the competence of correct data handling.

Examples:

# Fabricates a default rather than surfacing missing data
user_role = getattr(session, "role", "readonly")
# Missing role silently becomes "readonly" — wrong in either direction.

# Substitutes processing time for missing event time
event_time = record.timestamp or datetime.now()
# The audit trail now records when we processed the record, not when
# the event occurred. Temporal provenance has been fabricated.

# Fabricates identity via structural presence
if hasattr(obj, "security_clearance"):
    handle_classified(obj)
# Structural check, not identity check — any object with this attribute passes.

Why existing controls miss it: The code is syntactically valid, follows common patterns, and passes tests — because the failure is semantic, not syntactic. No existing linter, type checker, or SAST tool is designed to determine whether a default value on a particular field fabricates data that will be treated as authoritative. That judgment requires domain-specific context about the security semantics of each field — context that lives in team culture and project documentation, not in the programming language or any standard tooling. A human reviewer under time pressure may see "defensive coding" — a positive signal — and approve it.

Risk in government context: Classification decisions, access control, evidentiary integrity — any domain where "I don't know" and "the default" are different answers with different consequences.40

T — Tampering: Authority tier conflation47

Agentic variant: External (untrusted) data is treated as internal (trusted) data without validation, effectively tampering with the authority tier rather than the data itself.

Mechanism: Agents do not distinguish between data from different authority tiers because the programming language does not enforce it. A dict from a validated database query and a dict from an unvalidated API response are the same type. The agent treats them interchangeably.

Examples:

# API response used directly without validation boundary
api_response = requests.get(external_url).json()
save_to_internal_database(api_response["records"])
# External data enters the trusted internal store without validation.
# The agent does not see a trust boundary — it sees a dict going into a function.

# Deserialized data assumed trustworthy
config = json.loads(uploaded_config_file.read())
apply_system_settings(config)
# User-uploaded JSON treated as trusted configuration.

Why existing controls miss it: Type checkers verify shape (dict), not provenance. Linters check syntax, not trust-boundary crossings. The defect only becomes visible at the level of semantic boundary enforcement.

Risk in government context: Injection attacks through unvalidated external data, data corruption of authoritative records, and compliance failures when data provenance cannot be demonstrated.41

R — Repudiation: Audit trail destruction through error handling48

Agentic variant: Error handling patterns destroy the audit trail by catching, logging, and continuing rather than failing in a way that preserves the error as a first-class audit event.

Mechanism: Agents generate broad exception handlers that prevent crashes but also prevent errors from being recorded in audit systems. The error is "handled" in the sense that the program continues, but the event that caused the error is lost to the audit trail.

Examples:

# Error swallowed — audit trail has a gap
try:
    record_decision(case_id, decision, rationale, evidence)
except Exception as e:
    logger.error(f"Failed to record decision: {e}")
    # Decision was made but not recorded. The audit database shows nothing.

# Partial completion without rollback
try:
    update_classification(document_id, new_level)
    notify_stakeholders(document_id, new_level)
    record_classification_change(document_id, old_level, new_level)
except NotificationError:
    pass  # "Notification is non-critical"
    # Three operations that should be atomic are silently partial.

Why existing controls miss it: The code handles exceptions — generally considered good practice. The distinction between "handle the error and continue safely" and "swallow the error and destroy evidence" requires understanding which operations are audit-critical, which the agent does not reliably possess.

Risk in government context: Regulatory compliance (failure to maintain complete audit trails), legal proceedings (gaps in evidence chains), and IRAP assessment failures (inability to demonstrate complete traceability).42

I — Information Disclosure: Verbose error response and stack trace exposure49

Agentic variant: Agent-generated error handling exposes internal system details in error responses, log messages, or API returns.

Mechanism: Agents produce "helpful" error messages that include internal state, query parameters, file paths, or stack traces (ACF-I1: Verbose Error Response). This is good practice for development but dangerous in production, and agents do not distinguish between the two contexts. Stack trace exposure — a related pattern well-covered by existing SAST tooling — is not catalogued as a separate ACF entry because existing tools provide adequate detection.

Examples:

# Agent-generated "helpful" error handler
except DatabaseError as e:
    return {
        "error": str(e),
        "query": sql_query,         # Exposes database schema
        "connection": str(db_url),  # May contain credentials
        "params": query_params,     # Exposes internal identifiers
    }

# Stack trace in API response
except Exception as e:
    import traceback
    return {"error": traceback.format_exc()}
    # Full stack trace exposes file paths, function names, and library versions.

Why existing controls miss it: The error handling is syntactically correct and genuinely helpful during development. Detecting that internal details should not appear in production error responses requires understanding the deployment context, not the code structure.

Risk in government context: Reconnaissance information for attackers, credential exposure, and violation of need-to-know principles.43

D — Denial of Service: Finding flood and review capacity exhaustion (meta-threat)50

Agentic variant: The volume of agent-generated code overwhelms the review process, degrading review quality to the point where the review is no longer an effective security control.

Note: This extends STRIDE to the development lifecycle. The "service" being denied is the review process — a security control per ISM-2060/2061 — not a user-facing system.

Mechanism: This is not a code pattern — it is a process threat that operates at two scales. At individual scale, when agents generate code at multiples of human velocity, the review queue grows proportionally. Reviewers under volume pressure shift from careful semantic review to surface-level scanning. The review process — which is a security control — degrades to a rubber stamp.

At organisational scale, the same dynamic overwhelms the review processes themselves:

  • The security team's capacity to evaluate changes
  • The IRAP assessment pipeline's ability to keep pace with the rate of system change
  • The audit function's ability to maintain meaningful coverage

The degradation compounds: individual reviewers fatigue, which degrades team-level review quality, which degrades the organisational assurance that depends on those reviews.

A secondary mechanism: when automated analysis tools produce too many findings on agent-generated code, reviewers habituate to dismissing findings, and genuine security issues are lost in the noise.

Why existing controls miss it: Existing controls assume review capacity scales with code generation rate. In practice, it does not. The control's effectiveness is inversely proportional to the volume it processes — the opposite of how every other component in the pipeline scales.

Risk in government context: Security review as a compliance checkbox rather than an effective control, accreditation based on a process that no longer provides the assurance it claims to provide.44

E — Elevation of Privilege: Implicit privilege grant51

Agentic variant: External system assertions are accepted without independent verification, granting privileges based on unvalidated claims — treating an external authority statement as if it were an internal trust decision.

Mechanism: Closely related to the Tampering category above (ACF-T1/T2), but focussed on the consequence rather than the mechanism. Where T1 describes the missing validation boundary, E1 describes what happens next: the external system's assertion is acted upon as though it carried internal authority.

The privilege elevation is implicit — no explicit setRole() or grantPermission() call — because the elevation happens through data flow, not code structure.

Examples:

# User-supplied filter used in internal query without validation
def search_records(user_query: dict):
    results = db.query(Record).filter_by(**user_query)
    # Untrusted input unpacked into query — user can filter on internal fields.
    return results

# External system's assertion accepted without verification
partner_response = partner_api.verify_identity(applicant_id)
if partner_response.get("verified", False):
    grant_access(applicant_id)
    # No independent verification, no recording of the basis for the decision.
    # Partner's authority tier silently elevated to internal authority tier.

Why existing controls miss it: The code follows common integration patterns. Scanners flag explicit privilege-escalation calls (setRole(), grantPermission()); they have no model for elevation-by-data-flow.

Risk in government context: Unauthorised access to classified information, acceptance of unverified identity assertions in federated systems, and compliance failures in inter-agency data sharing.45

A note on methodology and taxonomy design. Using a security framework (STRIDE) to categorise what §2.2 characterises as a safety problem is a deliberate pragmatic choice — STRIDE is the vocabulary Australian government security assessments already use. The trade-off is that some mappings are more illuminating than others: T, I, and E map naturally; S is analogical (fabricated data presenting as authoritative, not identity forgery); D is the loosest fit (resource depletion, not an attack). The reader should weigh the individual failure modes on their merits, not on the strength of their STRIDE mapping.

The taxonomy intentionally mixes code-level semantic failures (S, T, I, E) and process-level assurance failures (D), because the paper's central claim is that they interact: code-level failures pass review because of process-level failures.

It indexes observable artefacts rather than generative causes, since artefacts are what detection tools and review processes can act on.39 The categories above are the failure modes observed to date; the taxonomy is designed for extension.

3.3 The compounding effect

These six threat categories do not operate independently. In practice, they compound — and the compounding produces a structural failure condition: agents generate a flood of code that follows established good practice, arriving at review boundaries faster than human assurance processes can absorb it, in precisely the systems that have the least tolerance for those failure modes.

One illustrative scenario:

  1. An agent generates code with authority tier conflation (ACF-T1), creating the conditions for implicit privilege grant (ACF-E1) — external API data used directly without validation.
  2. Errors in that data are caught by a broad except block, producing audit trail destruction (ACF-R1).
  3. The handler substitutes a default value rather than surfacing uncertainty — fabricated default (ACF-S1). Downstream components treat that fabricated value as authoritative.
  4. Review volume pressure means the pattern is not detected before merge — review capacity exhaustion (ACF-D2).

Each individual pattern follows conventions generally regarded as good practice. The broad except block is responsible error handling. The default value is defensive programming. The direct API usage is clean integration code. A conventional review could approve each pattern individually, because each follows conventions reviewers are trained to accept. The compound effect is a system that silently produces wrong results, cannot explain why, and passed every review gate — not because the reviewer was negligent, but because every component followed established good practice for the wrong context.

A subtler compounding mechanism operates across time: upstream representational choices can collapse the semantic distinctions that downstream code needs. When a typed contract is flattened into a permissive dictionary structure, downstream defensive access patterns cease to look anomalous and begin to look prudent — upstream looseness manufactures the local conditions under which defensive handling appears justified. §5 develops this as bidirectional authority collapse.

Appendix E presents an incident in which this mechanism produced a latent bug that passed all automated checks and was only surfaced through four rounds of operator challenge. When agents produce these compounding shapes as a recurring characteristic rather than an occasional lapse, dormant but activatable defects accumulate. The resulting risk profile is qualitatively different from the same mistakes appearing sporadically at human velocity.

4. The Review Process as Attack Surface

This section analyses why the existing human review process is structurally inadequate for agent-generated code — not because reviewers are negligent, but because the review model's assumptions are violated. It draws on external literature (automation bias, habituation) and case study observation.

4.1 The asymmetry problem

Human code review evolved as a control for human-authored code at human pace. It relies on several assumptions that agentic coding violates.56

Assumption Human Code Agent Code
Volume Reviewers can read most of the code Volume makes comprehensive reading impossible
Familiarity The author can explain their intent The agent's "intent" is pattern completion from training data
Conventions Unusual patterns are suspicious Agents follow conventions precisely — dangerous code looks like good practice
Error rate Human defect introduction rate is empirically bounded, giving reviewers a calibratable baseline.60 Agent error rate for semantic correctness is unknown and context-dependent
Feedback Reviewer feedback improves the author Agent has limited or no persistent memory across review cycles

The consequences of this asymmetry are already visible at scale — GitHub's platform-level PR restrictions (§1.2.2) are a direct institutional response to the volume assumption breaking down.

This asymmetry matters most for changes known to be agent-generated via provenance tracking (§7.1). Semantic boundary enforcement, however, should apply uniformly regardless of authorship — authorship-based differential enforcement is unsolvable in mixed workflows where agent-generated and human-authored code are interleaved.

4.2 The habituation effect

When agents generate code that consistently passes tests and follows conventions, reviewers develop trust in the agent's output. This trust is not earned — it is a cognitive shortcut driven by volume pressure. In human factors engineering, this phenomenon is known as automation bias: the tendency to over-rely on automated systems and under-scrutinise their output.57

The effect is measurable. Perry et al. found that developers with AI coding access produced less secure code while simultaneously rating it more secure — a textbook automation bias outcome.90 The METR randomised controlled trial found a parallel perception-reality gap: experienced developers predicted AI would speed them up by 24%, believed after using it that they were 20% faster, but were measured as 19% slower.58

The reviewer's mental model shifts from "verify this code is correct" to "check this code isn't obviously wrong." The difference materially affects assurance: the first is an active search for defects; the second is a passive scan that catches only gross errors.

The incident in Appendix E suggests that even under favourable conditions — a context-loaded operator using the agent itself as an investigative instrument — surfacing a semantic defect concealed by conventional-looking code required sustained multi-step probing.

Case study observation (§8) confirms the mechanism in compliance-constrained environments: agent-generated code containing semantic defects — trust boundary violations, defensive patterns on audit-critical data, missing validation boundaries — has entered codebases after passing human review processes designed to catch exactly these issues, with defects subsequently identified through automated enforcement tooling, adjacent code review, or downstream test failures.

This is the "Shifting the Burden" systems archetype59: the agent's consistent surface-quality output becomes the symptomatic fix that weakens the fundamental solution (thorough human review). The more the agent produces acceptable-looking code, the less carefully humans review it, and the more dependent the process becomes on the agent being correct — which is exactly the assumption the review process exists to check.

The volume problem is real, but the sharper thesis is this: the decisive failures are often not visible to unaided review at all, because the relevant property is semantic and institutional rather than syntactic.

A reviewer who examines the .get() example from §2.3 with unlimited time and full attention may still approve it — because recognising the wrongness requires knowing that this system now handles PROTECTED material and that the upstream schema permits missing classification values, neither of which is visible in the diff. The response is not "review harder" but "check differently": express the system's security-relevant distinctions in a form that tooling can enforce, so that the distinction between "legitimate fallback" and "fabricated classification" is resolved before a human ever sees the diff. §7 develops this response across process controls (§7.1), technical controls (§7.2), and policy controls (§7.3).

Parallelisation compounds the effect. Agent-assisted velocity increases the parallelisation of work, not just its speed. When an agent assists in producing multiple interdependent artefacts simultaneously — a design specification, an implementation, and a policy document — semantic inconsistencies between artefacts become invisible because no single review pass covers all of them. The reviewer is not only less careful per artefact, but also unable to hold the full production context in working memory at the rate artefacts are produced.

An organisation can prohibit developers from running multiple agents concurrently, but the prohibition runs directly against the productivity incentive that justified adopting agentic development. Controls that depend on sustained human restraint in the face of convenience are inherently fragile — a principle well-established in security engineering but easy to overlook when the convenience is "generate code faster than you can review it."

4.3 The advisory fatigue problem

Static analysis tools that flag agent-generated code patterns as warnings face a paradox:

  • If agents produce many warnings, reviewers habituate to dismissing them
  • If agents learn to avoid warning-triggering patterns, they may produce code that satisfies the tool but still violates the semantic intent
  • Advisory-only tools have no enforcement mechanism for agents, which have limited or no persistent learning across review cycles
  • Project-level instructions and memory stores can encode rules (§7.1), but that knowledge competes for finite context and remains advisory rather than enforceable

This means the traditional "warn first, enforce later" adoption strategy for security tooling is ineffective for agent-generated code. Agents require enforcement at the boundary before code enters the repository, not feedback over time, which depends on learning.

Tool-on-tool conflict. A further complication arises when semantic enforcement tools conflict with standard linters. Appendix E documents an incident in which a tier-model enforcer flagged .get() on a typed internal data structure, but ruff's SIM401 rule demanded .get() back — contradictory instructions from two tools. The agent resolved the conflict by broadening the exception boundary (adding a permanent allowlist entry), which preserved the bug.

This is not an edge case — any organisation deploying semantic enforcement alongside standard linters will face tool-on-tool conflicts, and agents will resolve them by the path of least resistance: the configuration that satisfies both tools, which is often the configuration that silences the semantic finding.

Organisations deploying semantic enforcement should define a tool-precedence hierarchy that explicitly prioritises semantic enforcement over conventional linting where they conflict, and should require human review of agent-authored allowlist entries.55

5. Agent Output as a Trust Boundary

This section introduces the paper's core conceptual contribution: the authority tier model and the treatment of agent output as untrusted input requiring boundary validation. It provides the analytical vocabulary that the gap analysis (§6) and response landscape (§7) build on.

5.1 The authority tier model

Data in high-stakes systems can be classified into authority tiers: how much authority the data carries to justify continued execution, and how little latitude the system may give anomalies before it must stop. The tier reflects what guarantees the system is entitled to assume: data the system itself produced is authoritative at source; data from an external source is unvalidated regardless of quality.

The tier governs how the system treats the data. The path governs how the system responds when that data fails.

Tier Description Handling Rule Example
Tier 1: Authoritative internal data (trusted assertion) Data produced by the system's own controlled processes — audit records, internal state, configuration Treat as authoritative. Missingness, corruption, or contradiction is an integrity failure, not an invitation to default. Halt or reject the current operation; the exact failure response is path-specific Database audit trail, system configuration, internal state machines
Tier 2: Semantically validated data (semantically validated representation) Data that has passed both structural validation and domain-constraint checking — values are present, correctly typed, and satisfy business rules, range constraints, and cross-field invariants Trust for domain operations once validated. Guard only against cross-cutting concerns (authorisation, concurrency, freshness, state transitions) that value-level validation cannot address API response after schema validation and domain-rule verification, database record promoted through a business-rule gate
Tier 3: Shape-validated data (shape-validated representation) Data that entered the system from outside but has passed through only a structural validation boundary — fields are present and types are correct, but values may still be nonsensical, unsafe, or out of domain range Direct field access is safe; validate domain constraints before using values in business logic, arithmetic, or security-sensitive operations API response after schema validation, CSV row after type coercion
Tier 4: Unvalidated external data (raw observation) Data from outside the system boundary, not yet validated Do not use in high-stakes code paths until validated. Validate at the boundary and quarantine failures Raw API responses, user uploads, message queue payloads

On an audit trail path, an integrity failure may require the process to stop — a corrupted audit record means the system can no longer prove what happened. On an authentication path, a missing username halts the login attempt but the system keeps running, because a failed login does not compromise other operations. Both paths treat the data with zero latitude for corruption or substitution. Both refuse to fabricate a default. But one crashes the process and the other rejects a request. The difference is not in the authority of the data, but in the operational consequences of its failure.

The principle that external inputs should be validated at the perimeter is standard practice in security engineering. The four-tier taxonomy above is this paper's formalisation of that principle. The distinction between Tier 2 and Tier 3 — semantically validated versus shape-validated — captures a boundary that is critical in practice but invisible to most tooling: data that has the right shape (fields present, types correct) can still carry values that violate domain constraints. Treating shape-validated data as though it were semantically validated is a specific and common source of defects in high-stakes code paths.

The companion specification (§4–5) specifies the four-tier model in full, including the taint states that drive enforcement (§5.1) and the restoration boundary evidence model (§5.3) — four categories of provenance evidence that determine which tier a restored artefact may reach.

One consequence of the tier model that practitioners encounter immediately: serialisation boundaries reset trust. A Tier 1 audit record that is written to a database and later read back enters the read interface at the trust level of unvalidated data — its authority must be re-established through restoration controls (integrity verification, schema conformance) before it can be treated as authoritative again. The companion specification (§5.2–5.3) distinguishes this restoration from standard external-data validation: the data's provenance is known, but its integrity across the serialisation boundary is not, so the re-establishment process may use provenance evidence rather than full Tier 4 validation.

Uniform defensive patterns collapse the authority model from both directions (§2.2). The formal consequence is bidirectional. Agent-generated code gives Tier 4 (unvalidated external) data more authority than it has earned — defaults and coercion allow unvalidated data to cross inward as though it had been verified. Simultaneously, it treats Tier 1 (authoritative internal) data as more negotiable than the tier model permits — the same .get()-with-default and catch-and-continue patterns handle corruption of authoritative records as routine rather than exceptional.

This bidirectional authority collapse is the central mechanism by which one-size-fits-all defensive programming undermines a tiered authority architecture.

5.2 Agent code as untrusted input

The authority tier model above describes data flowing through a running system. Agent-generated code is not data in that sense — it is source code entering a development workflow — and forcing it into the Tier 3 category overloads a model designed for runtime data authority. But the analogy is instructive: just as external data must not enter the authoritative store without passing through a validation boundary, agent-generated code must not enter the codebase until it has been reviewed — not only by a human reviewer but also by automated structural and semantic checks. The principle is the same; the mechanism is different.

This is not a claim about agent quality. Agents produce excellent code much of the time. It is a claim about provenance: assurance is drawn from the organisation's validation process rather than from the tool's statistical quality, because the relevant property is verified correctness, not apparent competence. The agent is an external system. Its output has not been validated against the system's security requirements. The fact that the output is source code rather than JSON or CSV does not change its trust properties — it warrants boundary discipline analogous to what organisations already apply to external data, adapted for the development workflow.

The natural organisational instinct is to treat agent output as carrying the authority of the person who directed the agent. If the chief engineer uses an agent to implement a feature, the resulting code feels like "product of the chief engineer" and inherits the review deference that the chief engineer's own code would receive. This is the wrong model. The chief engineer directed the work, but the code itself was generated by a system that has not validated its output against the project's security requirements.

The appropriate analogy is not "code written by a trusted senior engineer" but "code submitted to the repository by an external contributor whose competence is plausible but unverified."

The directing engineer's authority is relevant to the decision to accept the code after review, not to the trust level of the code before review. Without this distinction, the authority of the human operator launders the provenance of the machine output — precisely the kind of implicit trust escalation that the tier model is designed to prevent.

An obvious objection: does an agent fine-tuned on the organisation's own codebase deserve a more permissive review posture? It does not. A fine-tuned agent may produce output that is statistically more likely to conform to local conventions, but it has not validated its output against the system's security requirements. The fine-tuning changes the prior probability of correctness; it does not change the epistemic status of the output. Validated status requires that the output has passed through review and enforcement (§5.3) — an event, not a property of the generating tool — for the same reason that hiring a trustworthy contractor does not eliminate the need for acceptance testing.62

Treating agent code as untrusted input has specific implications:

Principle Application
Validate at the boundary Agent output must pass security-aware validation before entering the codebase
Quarantine failures Code that fails validation is rejected, not silently corrected
Record original output The original agent output is preserved for audit, even if modified during review (§7.1)
No silent coercion Agent code is not silently "fixed up" by reviewers — changes are explicit and recorded

5.3 Implications for the development workflow

Because agent output is untrusted, the development workflow must include a validation boundary between agent generation and code integration:

graph TD
    GEN["Agent generates code<br/>(untrusted)"] --> VB

    subgraph VB ["VALIDATION BOUNDARY"]
        direction TB
        AUTO["Automated semantic<br/>boundary checking<br/><i>Authority tier flow, defensive<br/>pattern detection, audit trail<br/>completeness — not just<br/>syntax/type checking</i>"]
        AUTO --> HUMAN["Human review of<br/>semantic intent"]
        HUMAN --> ATT["Attestation<br/><i>Reviewer attests validation<br/>was meaningful,<br/>not rubber-stamped</i>"]
    end

    VB --> REPO["Code enters repository<br/>(validated)"]

    style GEN fill:#fdd,stroke:#333,stroke-width:2px
    style VB fill:#efe,stroke:#393,stroke-width:3px
    style REPO fill:#bfb,stroke:#333,stroke-width:2px

Diagram description (accessibility): The diagram shows a three-stage flowchart: (1) Agent generates code (untrusted); (2) Code passes through a Validation Boundary containing three sequential steps — automated semantic boundary checking (authority tier flow, defensive pattern detection, audit trail completeness — not just syntax/type checking), human review of semantic intent, and attestation (reviewer attests validation was meaningful, not rubber-stamped); (3) Code enters repository (validated). The key message: code moves from untrusted to validated only after passing through all three validation steps.

The key difference from current practice: the validation must be semantically aware, not just syntactically or functionally correct. Current checks ask "does this code work?" and "does it follow known vulnerability patterns?" The failure modes in Appendix A pass both questions. The validation boundary must answer the semantic questions identified in §3 — questions about fabricated defaults, destroyed audit trails, and unvalidated trust-boundary crossings that require knowledge of what the code means in its operational context, expressed as repository-enforced rules rather than left in documentation or reviewer memory.

This validation boundary is not one thing. It is a layered stack, and the layers are not interchangeable:

  1. Conventional checks — syntax, types, linting, unit tests, known vulnerability pattern scanning. These are necessary but insufficient: the core semantic failure modes in Appendix A pass this level.
  2. Semantic enforcement — purpose-built checks for authority-tier flow, path-appropriate failure behaviour, audit-trail preservation, fabricated defaults, and validation-boundary crossings. This is the missing layer that does not yet exist in standard tooling and that this paper argues must be built.
  3. Human review after semantic pre-screening — humans adjudicate meaning, exceptions, and architecture: whether the trust topology is correctly declared, whether a validation function is actually correct (not just structurally present), whether an exception to a rule is justified. Machines handle pattern-level detection first; humans focus on semantic adequacy.

This layered structure will be familiar to security practitioners as defence in depth — the same principle that governs network architecture and that the ISM embeds throughout its control framework. Each layer catches a different class of failure; no single layer is sufficient; and the layers are not interchangeable, just as a firewall does not substitute for application-level input validation. The difference from network defence in depth is that the middle layer — semantic enforcement — does not yet exist as a standard control category.

Building it is the core technical recommendation of this paper (§7.2). The tooling doctrine is younger than the threat model — one production deployment exists (§8), the companion specification provides a reference design, and several feasible implementation paths are described in §7.2. The specific tools will evolve faster than the underlying requirement.61

§7.2 describes how this framing translates into practice: a minimum viable validation boundary can be established at several stages of maturity (Stage 1 through Stage 3), from low-cost rule-and-checklist approaches through to automated semantic enforcement integrated into CI/CD workflows.

6. Current Guidance Gap Analysis

This section translates the threat model into specific gaps in existing cybersecurity guidance — ISM, Essential Eight, OWASP, NIST SSDF, and contracted development frameworks. It is the most directly policy-relevant section of the analysis, identifying where current controls fall short and what extensions are needed.

6.1 Australian Information Security Manual (ISM)

The ISM provides controls for software development security (primarily in the Software Development and Web Application Development chapters). The analysis below maps relevant controls to the agentic code threat model, identifying where existing controls provide partial coverage, where they assume conditions that agentic coding invalidates, and where gaps exist.

Note: The ISM underwent a significant expansion in June 2025, adding approximately 24 new controls to the Software Development guidelines. The analysis below references the December 2025 revision of the ISM. Organisations using earlier versions should verify control numbers against the current release.

6.1.1 Controls with partial coverage

The following controls provide partial coverage of agentic threats. For each, we identify what the control currently addresses and where its assumptions break down when applied to agent-generated code.

ISM-0401 (Rev 8, Jun-25) — Secure by Design principles and practices throughout the SDLC

Coverage: Establishes that organisations should follow Secure by Design principles across the entire software development lifecycle. Agentic failure modes (Appendix A) could in principle be addressed as part of an organisation's Secure by Design practices.

Gap: The control assumes a human development team that can internalise security principles and apply them with judgment. Agents do not internalise principles — they reproduce training data patterns. A Secure by Design practice that says "do not fabricate defaults for missing safety-critical data" is unenforceable against an agent unless encoded as a machine-checkable rule. The control's scope is correct, but its enforcement mechanism (human judgment) does not transfer to agent-generated code.

ISM-2060 (Rev 0, Jun-25) — Code reviews ensure software meets Secure by Design principles and secure programming practices

Coverage: Directly applicable to agent-generated code. The control explicitly links code review to Secure by Design principles, not just functional correctness.

Gap: The control assumes the reviewer can meaningfully evaluate the code at the rate it is produced. At agent-scale volume, this assumption fails (§4). The control does not address review effectiveness degradation or distinguish between surface-level and security-focussed review.

ISM-2061 (Rev 0, Jun-25) — Security-focussed peer reviews on critical and security-focussed software components

Coverage: Requires developer-supported, security-focussed peer reviews specifically on critical components. This is the strongest existing review control for the agentic context.

Gap: The control's limitation is scope: it applies to "critical and security-focussed software components," which requires the organisation to correctly identify which agent-generated code touches security-critical paths. Agents generate code across the entire codebase; the security-critical subset must be identified before the review control can be applied. The control also assumes the peer reviewer has the institutional knowledge to evaluate trust boundary maintenance — knowledge that may not be documented in machine-readable form.

ISM-0402 (Rev 9, Jun-25) — Comprehensive software testing using SAST, DAST, and SCA

Coverage: Mandates SAST, DAST, and SCA.64 These tools catch known vulnerability patterns and dependency risks.

Gap: Current SAST answers "does the code match known vulnerability patterns?" It does not answer whether data flows preserve authority tiers or whether trust boundaries are maintained. Semantic boundary testing is a distinct control category not addressed by existing SAST tooling (§2.2). SCA is relevant for agent-introduced dependencies but does not address first-party code quality.

ISM-2026/2027/2028 (Jun-25) — Software artefact integrity — malicious code scanning, digital signatures, SAST/DAST/SCA on artefacts

Coverage: Addresses integrity and security scanning of software artefacts before deployment. These controls cover the supply chain from build to deployment.

Gap: Agent-generated code fits neither established category cleanly: it is not human-authored in-house code, but neither is it a third-party component. The artefact integrity controls have no category for first-party code generated by a third-party system. The risk properties also differ: third-party components have independent defect distributions, while agent-generated code has correlated defects (§2.4).

ISM-1419 (environment segregation) was also reviewed; it remains important but is orthogonal to semantic correctness concerns.63

6.1.2 Controls with no direct coverage

The following gap areas have no corresponding ISM control. Each represents a category of agentic risk that falls outside the current framework's scope.

Agent output as trust boundary — ACF-T1 (authority tier conflation), ACF-E1 (implicit privilege grant)

No current ISM control category explicitly addresses the artefact classification of AI-generated output. ISM-2074 (Rev 0, Dec-25) requires an AI usage policy, but this is a governance control, not a technical trust boundary control.

Review capacity scaling — ACF-D1 (finding flood), ACF-D2 (review capacity exhaustion)

ISM-2060 and ISM-2061 mandate code review and security-focussed peer review, but neither addresses what happens when code generation velocity exceeds review capacity. No control requires organisations to demonstrate that review remains effective under volume pressure. Even with an attentive operator, surfacing a semantic defect concealed by conventional-looking code required four rounds of directed challenge over approximately eight minutes — suggesting that effective review of agent output demands domain-specific questioning strategies, not merely increased review time (Appendix E).

Semantic boundary enforcement — ACF-S1 (fabricated default), ACF-S3 (structural identity spoofing), ACF-T1 (authority tier conflation), ACF-T2 (silent coercion)

No control addresses the gap between syntactic correctness and semantic correctness in the context of trust boundaries. ISM-0402's SAST/DAST/SCA requirement covers known vulnerability patterns but not context-dependent semantic correctness. Existing controls assume that if code passes review and testing, it is adequate.

Correlated failure detection — All ACF categories

No control addresses the distinct risk profile of correlated defects. Testing and review strategies are designed for independent failure distributions.

Code provenance tracking — ACF-D2 (review capacity exhaustion)

No control requires organisations to track which code was generated by AI agents vs. authored by humans. ISM-2074 requires an AI usage policy but not per-artefact provenance. Without provenance, risk assessment cannot distinguish between code populations with different failure characteristics.

6.1.3 Candidate ISM extensions

The following are illustrative extensions, not formal proposals. They are included to show that the gaps are addressable within the ISM's existing structure. The wording follows the ISM's conditional-control style and is illustrative rather than normative.

Extension to ISM-0401 (Secure by Design):

When AI agents are used to generate code for assessed systems, the organisation's Secure by Design practices should include machine-enforceable rules for trust boundary maintenance, defensive pattern restrictions appropriate to the system's data sensitivity, and audit trail preservation requirements. Secure by Design principles that exist only as human-readable documentation are insufficient controls against AI-generated code, which does not read documentation.

Extension to ISM-2060/2061 (Code Review and Security-Focussed Peer Review):

When AI agents generate a significant proportion of code changes, the organisation should demonstrate that its code review process (ISM-2060) and security-focussed peer review process (ISM-2061) remain effective at detecting semantic defects — not merely syntactic or conventional defects — under the volume of changes produced. Evidence must include at minimum one of: measured defect escape rates, review depth audits, or demonstrated use of automated semantic pre-screening that reduces the burden on human reviewers.

New control (Agent Output Trust Boundary):

Code generated by AI agents should be treated as untrusted input requiring validation at the boundary before integration into assessed systems. The organisation should define and document the validation boundary, including what properties are verified (trust boundary maintenance, audit trail integrity, error handling appropriateness) and what evidence demonstrates the validation is effective.

New control (Code Provenance):

When AI agents are used in the development of assessed systems, the organisation should maintain records of which code was generated by AI agents, which was human-authored, and which was agent-generated then human-modified. This provenance metadata supports risk assessment, incident response, and targeted remediation when systematic agent-introduced defects are discovered.

6.2 NIST Secure Software Development Framework (SSDF)

SP 800-218 defines practices for secure software development organised into four groups. Each practice group is mapped below to agentic code concerns:

Practice Group SSDF Practices Agentic Code Coverage
Prepare the Organization (PO) Define security requirements, roles, training Does not address training requirements for reviewing agent output (which requires different skills than reviewing human output) or organisational capacity planning for agent-scale review volume
Protect the Software (PS) Protect code, integrity verification Addresses integrity of code artefacts but not the artefact classification of code based on its generation method. An agent-generated commit and a human-authored commit are indistinguishable in the VCS
Produce Well-Secured Software (PW) Design, code review, testing Most relevant group. Partially applies, but assumes trainable human developers, learning from feedback, and largely independent error distributions — none of which hold for agents
Respond to Vulnerabilities (RV) Vulnerability response, disclosure Does not address the correlated nature of agent-introduced defects. Standard vulnerability response treats each finding independently; agent defects may require pattern-wide remediation (§9.4)

Key SSDF gap: Practice PW.1 ("Design Software to Meet Security Requirements and Mitigate Security Risks") includes task PW.1.1, which recommends "using forms of risk-based analysis to determine how much effort is adequate" for security practices. This implicitly assumes that risk is assessable per-component. Agent-generated code introduces systematic risk across many components from a single source. The analysis framework needs to account for correlation, not just per-component risk.

NIST's own recognition of this gap: NIST published SP 800-218A (July 2024) as a supplement to the SSDF specifically for generative AI and dual-use foundation model development contexts, acknowledging that the original framework's human-centred assumptions need AI-specific augmentation. However, SP 800-218A's focus is on secure practices for AI model development across the SDLC — not on the assurance of source code generated by AI systems. This is precisely the gap this paper addresses: substantial guidance now exists for building AI safely, but almost none for securing what AI builds.

SP 800-218A itself makes the assumption explicit: its practices and tasks do not distinguish between human-written and AI-generated source code, on the basis that all source code should be evaluated for vulnerabilities and other issues before use.66 This paper's central contention is that this assumption — that uniform evaluation suffices — does not hold for the failure modes described here, because the nature of agent-generated defects (correlated, semantically plausible, context-insensitive) demands different evaluation, not just equal evaluation.

6.3 Essential Eight

The Essential Eight maturity model is an operational security framework, not an SDLC framework — it does not directly address software development practices.

Two strategies offer indirect relevance: Application Control establishes that not all software should be trusted equally based on its source, providing a conceptual precedent for graduated trust in code generation sources; Restrict Administrative Privileges supports the principle that agents should not modify security-critical configuration (allowlists, audit configuration, access control rules) without human approval. These analogies are directional rather than prescriptive — the Essential Eight's value here is as evidence that the principles of graduated trust and least privilege are already embedded in Australian government security posture, even though the specific controls were not designed for the SDLC context this paper addresses.

6.4 OWASP and industry guidance

OWASP Top 10 for LLM Applications (2025) primarily addresses threats to LLM systems — prompt injection, training data poisoning, model denial of service. The closest entry to this paper's concerns is LLM05 (Improper Output Handling), whose attack scenarios explicitly include LLM-generated code introducing vulnerabilities such as SQL injection. However, even LLM05 frames this as an application-level output-handling problem — advising developers to treat LLM output with "zero-trust" validation — rather than providing a comprehensive treatment of the distinct failure characteristics of AI-assisted code generation (correlated defects, review capacity exhaustion, context-inappropriate patterns).

The project has since evolved into the broader OWASP GenAI Security Project, covering LLM applications, agentic AI systems, and AI-driven applications — but no OWASP project specifically targets the assurance of AI-generated source code in government systems in the sense addressed by this paper.

OWASP Secure Coding Practices provides a checklist of defensive coding practices. Several "secure" practices in the OWASP checklist are precisely the anti-patterns that the agentic threat model identifies as dangerous in high-stakes contexts. For example, "validate all input" is correct, but the common defensive practice of providing a default value when input is missing is context-dependent — in audit-critical systems, a missing value should crash, not default. This illustrates the gap between generic secure coding guidance and domain-specific trust boundary requirements.

MITRE ATT&CK and CWE provide taxonomies for attack techniques and code weaknesses respectively. The agentic code failure modes in Appendix A do not map cleanly to existing CWE entries because they are not individual weaknesses — they are patterns that are correct in most contexts and dangerous in specific ones. A try/except that logs and continues is not a weakness; it is a weakness when it wraps an audit-critical write and prevents the failure from reaching the audit system. A default value on a missing field is not a weakness; it is a weakness when the field's absence signals corruption, not a case to handle gracefully. Context-free taxonomies do not serve context-dependent weaknesses.

The gap between "securing AI" and "securing what AI builds"

The gap identified in §6.2 — substantial guidance for securing AI systems themselves, almost none for securing what AI builds — is consistent across all frameworks reviewed above. The gap is widening faster than it is closing: agent adoption is accelerating (§1.2), the highest-risk failure modes pass all existing automated checks (§2.2, Appendix A detection summary), the vocabulary for discussing these failures does not yet exist in policy contexts, and guidance development cycles are inherently slower than technology adoption cycles.

6.5 Detection coverage is worst where risk is highest

The preceding sections established that the highest-risk agentic failure modes are semantic: they largely fall outside syntactic and structural analysis because the wrongness depends on institutional context, not code shape. The quantitative picture below shows how that category gap translates into detection coverage.

The ACF taxonomy (Appendix A) catalogues fifteen core agentic code failure modes (plus five provisional candidates). Of the fifteen core entries:

Detection Level Count Implication
None — no existing tool detects it 4 Requires new tooling or new review practices
Partial — some tools catch some cases 9 Existing tools provide incomplete coverage
N/A — process threat, not code pattern 2 Requires process controls, not technical controls

Of the fifteen core failure modes, thirteen are undetected or only partially detected by existing tools — including all four with no tool coverage at all, both Critical-rated entries among them. The four with zero detection capability are ACF-T1 (authority tier conflation, Critical), ACF-E1 (implicit privilege grant, Critical), ACF-R2 (partial completion, High), and ACF-R5 (remediation-induced violation, High). The highest-risk failures fall outside the detection scope of current tooling. The full detection breakdown with specific failure IDs is in Appendix A.

6.6 Structural gaps

The detection picture above is the quantitative expression of a set of structural gaps — categories of control and analytical vocabulary that no current framework provides.

Of these, items 1–3 are foundational — the remaining gaps are difficult to address without a shared taxonomy, a semantic verification layer, and review controls that account for agent-generated volume:

  1. A taxonomy of agentic code failure modes grounded in established threat modelling (STRIDE or equivalent)
  2. A verification layer for semantic correctness — the missing layer between "does the code match known vulnerability patterns?" and "does the code preserve trust boundaries, audit integrity, and failure-mode requirements?" (§7.2; companion specification)
  3. Controls for review effectiveness at scale — not just "is code reviewed?" but "does the review process remain effective at agent-generated volume?"
  4. Authority classification for agent output — how should agent-generated code be treated in the system's authority model?
  5. Accreditation criteria for agentic development workflows — what evidence must organisations provide to demonstrate that agentic coding maintains the required security posture?
  6. Vocabulary for context-dependent code weaknesses — patterns that are correct in general but dangerous in specific security contexts, encoded in machine-readable form (§7.2; the companion specification sketches one candidate vocabulary for this)
  7. Correlated failure risk models — testing and remediation strategies that account for the non-independent failure distribution of agent-generated code
  8. Governance perimeter expansion — controls for executable logic produced by non-developers using agentic tools outside traditional SDLC channels (§1.2.7)
  9. Cross-model defect chaining (emerging) — defects from different models may compose, where one model's failure creates preconditions for another's (§2.4(g); Appendix F). A candidate gap warranting attention as multi-model environments become common

Current frameworks scope software development controls to recognised development teams and established code repositories. Agent-generated automations, integrations, and plugins produced by analysts and operators outside these channels are not addressed by any current guidance.

The structural gaps listed above are not addressable through individual organisational practice alone. They require shared vocabulary, common assessment criteria, cross-organisational detection mechanisms, and agent output classification standards that sit at the whole-of-government level. Closing them is work for the bodies with the institutional mandate and cross-government visibility to develop controls of this kind.

6.7 Contracted development as the primary delivery context

The gaps identified above assume that the organisation writing the code also controls the development tooling, review process, and enforcement boundary. For most Australian Government software, this assumption is false — contracted service providers (consultancies, systems integrators, and specialist vendors) are the dominant delivery channel. Any threat model that addresses only in-house development addresses the minority case.

This is not an edge case or a procurement footnote. It is the primary delivery context in which the paper's risks materialise. The question is not merely "how should developers use agents?" but "how should agencies assure systems built through commercial delivery pipelines in which agents are increasingly present?"

The structural risk. When multiple agencies contract the same provider, and that provider uses the same agent tooling and prompts across engagements, the correlated failure problem (§2.4) extends across agency boundaries through the contractor — even if the agencies have no direct relationship.

This is concentration risk through interconnected delivery chains: the topology alone is sufficient to identify the vulnerability, in the same way that interconnected counterparties create systemic risk in financial networks regardless of their individual creditworthiness.

A systematic defect introduced by a contractor's agent may propagate to every agency that contractor serves, producing cross-agency correlated exposure from a single tooling decision the agencies had no visibility into.

The control gap. Current acceptance criteria for contracted software development typically focus on functional requirements, test coverage, and compliance with coding standards. They do not address the semantic correctness properties this paper identifies — trust boundary maintenance, audit trail integrity, context-appropriate error handling. A contractor could deliver code that meets every contractual requirement while containing systematic ACF-pattern violations.

Existing assessment frameworks65 evaluate the contractor's security posture and process maturity, but none evaluate whether the contractor's development workflow detects agentic failure modes or whether acceptance testing covers semantic boundary properties.

The visibility problem compounds this. Contracting agencies may have limited visibility into whether a contractor is using agentic tools, what proportion of deliverables are agent-generated, and whether the contractor's review processes address the failure modes in Appendix A. ISM-2074's AI usage policy requirement applies to the agency's own use; how it flows down to contracted development is unclear.

When a contractor delivers agent-generated code, the review responsibility is ambiguous — the contractor's internal review, the agency's acceptance review, or both? If the agency relies on the contractor's review, the agency inherits the contractor's review capacity constraints and habituation dynamics (§4.2).

Principles for contract requirements. This paper does not propose draft contract clauses — that is the downstream deliverable for procurement bodies (ASD/ACSC, DTA) with the legal and jurisdictional expertise to draft them. What follows are principles that contracted development should address:

  • Agent tool disclosure: contracts should require disclosure of AI code generation tool usage — which tools, which models, and what configuration — so that agencies can assess correlated risk across their supplier base.
  • Provenance tracking: contractors should maintain and provide provenance records for agent-generated code in deliverables, enabling targeted review and remediation.
  • Semantic correctness acceptance criteria: acceptance testing should include verification of trust boundary maintenance, classification handling, and audit trail integrity — not just functional correctness and test coverage.
  • Right to inspect validation controls: agencies should be able to request evidence that agent-generated code was subject to validation controls addressing the failure modes in Appendix A.
  • Pattern-wide remediation obligations: where a systematic agent-induced defect is discovered in one engagement, the contractor should assess and remediate the same pattern across other deliverables. Notification of other affected clients is the logical extension, though this crosses commercial confidentiality boundaries and may require a structured disclosure framework — analogous to coordinated vulnerability disclosure — rather than a blanket contractual obligation.
  • Cross-agency correlation awareness: agencies should assess whether their contracted suppliers serve other government clients with the same tooling stack, and consider the correlated risk this creates.

Formal clause drafting should be left to the bodies with procurement expertise; these principles are intended as structured inputs for that work.

7. The Response Landscape

This section shifts from analysis to response — what organisations can build, adopt, and enforce. It covers process controls (§7.1), technical controls (§7.2), policy controls (§7.3), and incident response (§7.4), ordered from weakest to strongest assurance. Technical practitioners should focus on §7.2; policy readers on §7.3.

The preceding sections establish that the highest-risk agentic failure modes are semantic and convention-conforming. To understand the gap, consider three levels of automated checking:

  • Syntactic check: "This code parses, types check, tests pass." — The core semantic failure modes in Appendix A pass this level.
  • Conventional security check: "This code contains no known injection pattern, no obvious secret leak, no flagged dependency vulnerability." — The same failure modes pass this level too.
  • Semantic check: "This code path is not allowed to invent a default for missing classification data; this external payload has not passed a validation boundary; this audit write may not be wrapped in catch-and-continue logic." — This is the level at which the failure modes become detectable.

The gap between the second and third levels is the gap this paper identifies. The practical response is not "review harder" but "change what is checked": move the system's security-relevant distinctions — trust boundaries, failure-mode requirements, audit-critical paths — out of prose and into enforceable controls (§5.3). Human review remains necessary, but after semantic enforcement, not instead of it.

The responses available to organisations fall into three categories of increasing assurance strength (ordered weakest to strongest):

Control Type Mechanism Strength Example
Behavioural Relies on individual compliance Weakest — requires sustained restraint against incentives "Developers should not run more than one agent concurrently"
Procedural Relies on organisational process Moderate — requires consistent enforcement and audit "Parallel agent-generated changes require separate review queues and staged approval"
Technical Constrains the environment Strongest — operates regardless of individual behaviour "The CI/CD pipeline enforces concurrency limits, sequencing rules, or protected-branch gates for agent-originated changes"

Most organisations will implement behavioural controls, aspire to procedural controls, and underinvest in technical controls — because technical controls constrain the velocity that motivated adoption. The key insight from security engineering applies here: controls that shape the environment are stronger than controls that depend on restraint. A rule that developers must not bypass review is an aspiration; a pipeline that physically prevents unreviewed code from reaching protected branches is a control.

All three control types have a role, but assurance should not rest primarily on behavioural or procedural controls where technical enforcement is feasible. Organisations that rely on behavioural and procedural controls without technical enforcement should understand that their assurance argument rests on sustained human compliance with rules that run directly against the productivity incentive that makes agentic development attractive.

7.1 Process controls (strengthening existing practices)

Current best practice, adapted for agentic velocity:

Enhanced code review protocols

Mandate security-focussed review (not just correctness review) for agent-generated code. Require reviewers to attest that trust boundaries were verified, not just that the code "looks right." This is a process change, not a technology change, but it requires explicit recognition that agent code needs different review criteria than human code. Critically, these criteria concern questions the current standard assurance stack does not answer. The checklist below is an interim bridge while automated semantic enforcement tooling matures (§7.2); the long-term goal is to encode as many of these questions as possible in machine-enforceable form, because human checklists do not scale to agent-generated volume.

Specifically, review checklists for agent-generated code should include two categories of question — pattern checks that are mechanical and can eventually be automated, and judgement calls that require human reasoning and will remain with reviewers even after semantic enforcement tooling matures:

Pattern checks (mechanical — look for these in the code):

  • Does missing data crash or default? When a value is absent, does the code stop and report the problem, or does it silently substitute something? On a high-stakes path — a security classification, an audit field, an authorisation decision — a silent default is a silent corruption. (Does every .get() or getattr with a default represent a legitimate design decision, or a fabrication of missing data?)
  • Are failed operations reported or quietly swallowed? When something goes wrong, does the error reach the audit trail and the operations team, or does the code catch the error, log it locally, and continue as if nothing happened? (Do except blocks propagate to the audit system, or swallow and continue?)
  • Is external data validated before being treated as authoritative? When data arrives from outside the system — from an API, a user input, a partner feed — does the code check it before passing it to internal functions that assume it is trustworthy? Are external system assertions (permissions, entitlements, access decisions) accepted and acted on without independent verification?
  • Is the code's failure mode correct for the context? Should this code crash, quarantine, or continue on error?
  • Are error responses appropriately scoped? Do error messages, exceptions, or API responses expose internal structure, database schemas, file paths, or stack traces that could aid an attacker? (ACF-I1)

Judgement calls (require thought — step back and assess):

  • Q1. Did AI suggest this pattern — and do I understand why? If the code came from an AI tool and the reviewer cannot explain why this specific approach was chosen over alternatives, that is a signal to pause. AI tools select patterns based on statistical frequency in training data, not based on fitness for the current context. A pattern the reviewer cannot independently justify is a pattern that has not been reviewed — it has only been accepted.
  • Q2. If this code is wrong, how would I find out? If the answer is "an audit, months later" or "a data breach," the code lacks adequate observability for the risk it carries. This question reframes review from "is this code correct?" to "would I know if it weren't?" — and the answer determines whether the system has a feedback loop or a silent accumulation path.

The pattern checks are the first candidates for conversion to automated enforcement rules (§7.2); the judgement calls (Q1–Q2) are intrinsically human and will remain part of the review process regardless of tooling maturity. The distinction matters because it tells organisations where to invest: automation for the pattern checks, training and review time for the judgement calls.

Separation of generation and review

The person (or agent) who generates the code must not be the sole reviewer. This already applies to human-authored code in most compliance-constrained contexts; extending it to agent-generated code means ensuring that agent self-review (e.g., an agent checking its own output) does not count as an independent review. This has a subtlety: multi-agent workflows where one agent generates code and another reviews it using different models are not independent review in the statistical sense — the models share overlapping training corpora and failure modes. However, a more nuanced form of multi-agent review offers meaningful value.

Structured perspective diversity in agent-assisted review

While model diversity (using different models) provides limited independence, perspective diversity — prompting the same model with different analytical frames — can produce meaningfully different coverage. Illustrative functions include an architectural reviewer, a problem-framing reviewer, an implementation reviewer, and a quality reviewer.67 These lenses are not independent, but they surface different classes of issue: a trust boundary violation (ACF-T1) may be an implementation defect whose cause is an architectural misplacement, while the tests may still pass while verifying the wrong thing.

This is faceted analysis, not independence — the underlying model's blind spots persist across all frames. But a small set of prompted perspectives may provide broader first-pass coverage than a single undifferentiated review, reducing the volume of issues that reach human review. Organisations should develop role-specific review prompts aligned to their threat model and architecture; the prompts themselves become reviewable, version-controlled security artefacts. This is a procedural control achievable now with no tooling investment — a useful bridge while automated enforcement tooling matures.

Volume-aware review capacity planning

If agents materially increase code generation volume, review capacity must be addressed — either through additional reviewers, automated pre-screening that reduces the human review burden, or rate-limiting agent output to match review capacity. Ignoring the volume mismatch means the review control degrades silently (§4.2).

A practical indicator: if the average time a reviewer spends per agent-generated change request is declining while the volume of agent-generated changes is increasing, the review process is degrading regardless of whether individual reviews are still being approved. Tracking review time per change alongside change volume — and investigating when the ratio drops — provides an early warning before the habituation effect (§4.2) has fully eroded review effectiveness. The case study (§8.3) provides one project's experience with this balance.

Project-level instructions as a generation-time control

Most agent frameworks support project-level configuration — system prompts, instruction files, memory stores — that shape agent behaviour within a session. These instructions can encode project-specific rules ("never use .get() on audit data," "all error handlers for audit-write operations must propagate, not catch-and-continue") and reduce the frequency of the failure modes in Appendix A at generation time.

This is a behavioural control (weakest tier): it depends on the agent respecting the instructions, the instructions being comprehensive enough to cover all failure modes, and the agent not generalising incorrectly from specific rules. Instructions that say "don't use .get() on audit data" do not teach the agent the underlying principle "don't fabricate defaults where data absence is meaningful" — every specific rule must be spelled out. Nonetheless, project-level instructions are an immediately deployable, zero-cost control that measurably reduces (without eliminating) the volume of semantic violations reaching the review pipeline.

Provenance tracking for agent output

Organisations should maintain records of which code was generated by agents, which was human-authored, and which was agent-generated then human-modified. This metadata is relevant for both security assessment (understanding the trust profile of different code regions) and for incident response (when a defect is found, knowing whether it originated from agent generation helps diagnose the failure mode).

7.2 Technical controls (what is buildable)

Verification-first framing

When evaluating security tooling for agentic code, implementation size is the wrong assurance metric. The relevant question is: "How do you know it's correct?" This question applies recursively. If the security enforcement tool is itself built by an agent, the tool's correctness is subject to the same threat model as the code it checks — and its assurance must rest on independent verification rather than on the process used to produce it. A tool that checks for ACF-S1 violations but was itself generated with an ACF-T1 violation in its rule-matching logic is not a mitigation; it is an additional attack surface.

The line between "tool" and "assurance argument" blurs: at some point, a verification tool's correctness becomes the system's correctness claim, and that claim needs its own evidence. Accreditation should assess the verification story — golden corpus, self-hosting, measured precision and recall — rather than implementation scope. A small tool with rigorous verification is stronger than a large tool lacking it. This reframing has direct implications for how IRAP assessors approach agent-assisted development, developed in §9.1.

This section does not recommend a specific product or tool. It describes the control properties required for semantic boundary enforcement. The companion specification illustrates one candidate framework — defining what to check and delegating how to check it to existing static analysis tools — but is a response artefact, not a prerequisite for the paper's conclusions.

Minimum viable validation boundary and maturity stages

§5.3 establishes that agent output requires a validation boundary. But "treat all agent code as untrusted" without implementation guidance is like "treat all input as untrusted" — true but inert. The validation boundary must be implementable at a cost proportionate to the organisation's risk profile, or it will not be implemented at all.

Controls that are too costly, slow, or specialised for early adoption are likely to drive undeclared agent use outside formal processes. In practice, a weak but visible validation boundary is often safer than an ideal control model that organisations do not adopt — because undeclared use means zero controls, zero provenance, and zero visibility.

This is the ALARP principle (as low as reasonably practicable) familiar from safety engineering and risk assessment: controls should be proportionate to the risk, and there is a cost floor below which any organisation using agentic coding should act regardless of its risk appetite. The staging model below is structured around that principle — Stage 1 is the ALARP floor — and each subsequent stage adds assurance without requiring the previous stage to be abandoned. The entry stage is achievable with existing tooling and no specialist engineering.

These validation maturity stages are orthogonal to the control-type hierarchy above (behavioural, procedural, technical). The control types describe what kind of assurance mechanism is used; the stages describe how much semantic enforcement is in place. A Stage 1 organisation still needs all three control types — but its technical controls are limited to pattern-matching proxies rather than full semantic enforcement.

Organisations should also assess where their agent deployment sits on the agent deployment spectrum (Appendix B) and whether their validation stage is proportionate — Stage 1 controls may be adequate for Level 1 autonomy (prompted + copied) but are unlikely to be sufficient on their own for Level 3 autonomy (fully autonomous agents). Appendix B adopts Stage 1 (defined here) as the minimum starting position for Level 3, with progression toward Stage 2–3 proportionate to the risk profile.

Stage 1: Achievable with existing tooling and checklists. (Stage 1 implements all three layers of §5.3's validation stack — conventional checks, semantic enforcement, and human review — but the semantic enforcement layer is limited to pattern-matching proxies rather than full authority-tier-aware analysis.)

  • 3–5 custom static analysis rules (Semgrep or equivalent) targeting high-risk proxies for the highest-risk ACF patterns. These rules should be scoped to paths the project identifies as high-stakes (§1.1) — applying them codebase-wide produces excessive false positives because the same patterns are correct defensive programming elsewhere. These run in CI and block on failure. A minimum viable detection set, prioritised by risk rating and detection gap:

    1. Broad except on audit-write paths (proxy for ACF-R1, High). Match try/except blocks that wrap audit-critical operations — database writes to audit tables, log-of-record emissions, compliance event recording — where the except clause logs and continues rather than propagating. This is the highest-frequency agent-generated pattern after ACF-S1 and the easiest to write a reliable rule for, because audit paths can be identified by function name, decorator, or module location.
    2. Unvalidated external data entering internal stores (proxy for ACF-T1, Critical). Match functions that accept data from external sources (requests.get(), API handlers, file parsers, message queue consumers) and pass it to internal data store operations (INSERT, ORM .create(), .save()) without an intervening validation call. Full taint tracking is Stage 2; at Stage 1, matching the structural pattern — external source → internal sink with no validation function between them — catches the highest-risk cases.
    3. Default values on fields in designated high-stakes data classes (proxy for ACF-S1, High). Match .get(), getattr(), or, and COALESCE() patterns on fields belonging to data classes or tables the project has designated as authority-tier or audit-critical. Requires the project to maintain a short list of sensitive field names or annotated classes — this list is the minimum institutional knowledge the rule needs.

    4. hasattr() or type checks used as authorisation gates (proxy for ACF-E1, Critical). Match hasattr(obj, "permission") patterns, or isinstance() checks on externally supplied objects, used in conditional branches that control access to privileged operations without independent verification. Structural presence is not identity; these patterns grant privilege based on duck-typing or unverified type claims rather than verified credentials.

      Scope note: isinstance() is the correct replacement for hasattr() in type-identity contexts (ACF-S3) — the concern here is specifically its use as a sole authorisation gate for external data without independent verification. Rules implementing this check should target isinstance() on objects crossing a trust boundary, not isinstance() used for internal type dispatch. 5. Silent partial completion without transaction or atomicity (proxy for ACF-R2, High). Match sequences of related write operations (multiple INSERT/UPDATE, multiple API calls, multiple file writes) that are not wrapped in a transaction, context manager, or equivalent atomicity mechanism — particularly where a failure partway through leaves the system in an inconsistent state with no rollback or notification. - A review checklist of 5–7 questions derived from the ACF taxonomy (see candidate questions above), used by human reviewers for agent-generated changes. - A pre-commit hook checking for the most common patterns: .get() with default values on security-classified fields, broad exception handlers on audit paths. - Achievable for most teams with a CI pipeline and existing tooling. No specialist investment required.

Stage 2: Moderate engineering investment.

  • Automated semantic boundary enforcement — purpose-built tooling, extended Semgrep/CodeQL/Pysa rulesets, or equivalent — integrated into every commit, covering a broader set of ACF patterns than Stage 1.
  • Human review focussed on trust-boundary-crossing code paths rather than line-by-line review of all agent output, because automated pre-screening handles the structural violations.
  • Provenance tracking for agent-generated code (which changes were agent-generated, which were human-authored, which were agent-generated then human-modified).
  • Often proportionate for systems handling classified, integrity-sensitive, or compliance-constrained data. Requires engineering effort to configure, tune, and maintain the ruleset.

Stage 3: Comprehensive semantic enforcement with governance model.

  • Full semantic boundary enforcement with a governance model covering authority classification, audit primacy, and architectural boundary rules. The companion specification sketches one candidate framework for this stage — a language-agnostic classification approach with draft language bindings for Python and Java — illustrating the annotation groups (core classification and supplementary contract annotations) alongside a severity matrix, golden corpus, and measured precision.

    At this stage the enforcement surface is not merely technical: the project's trust declarations — tier assignments, boundary contracts, exception approvals — constitute a policy artefact with its own governance requirements. The companion specification addresses this through graduated governance profiles (a lighter profile for small teams and early adopters, a full assurance profile for mature teams and ISM-assessed systems), manifest change authority, and temporal separation between policy changes and the code that benefits from them (companion specification §9 defines the governance mechanisms; §14.3.2 defines the Lite and Assurance governance profiles). - Verification properties (golden corpus, self-hosting gate, measured false positive/negative rates) are independently auditable. - Substantial engineering effort. Appropriate for organisations with high-assurance requirements or large agent-generated codebases where the cost of undetected semantic violations exceeds the cost of tooling.

Stage 1 is the minimum credible starting position — below it, the organisation has no systematic detection of the failure modes this paper identifies. Stage 3 is the aspirational target for high-assurance environments. Most organisations should aim to reach Stage 2, where automated enforcement handles structural violations and human reviewers can focus on semantic issues that require institutional knowledge. The stages are additive: Stage 2 extends Stage 1 rather than replacing it, and Stage 3 extends Stage 2.

False positive management — critical at Stage 1. Any static analysis that encodes institutional invariants will produce false positives — patterns that match the rule but are correct in context. This is not a theoretical concern; it is the reason most organisations' SAST configurations are weaker than their policies require. If a semantic enforcement tool generates too many false alerts, developers will disable it, route around it, or stop trusting its findings — and the enforcement layer degrades to a nuisance rather than a gate.

Organisations deploying semantic enforcement should track false positive rates from the outset, scope rules to high-stakes paths rather than applying them codebase-wide (the same pattern that is dangerous on an audit path is correct defensive programming on a UI path), and maintain an allowlist regime with human-authored justifications so that legitimate uses of otherwise-restricted patterns are recorded rather than suppressed. The companion specification illustrates one approach to this through precision-tracking requirements (golden corpus of known-good and known-bad patterns) and severity tuning. The risk of false-positive fatigue is real, but it is a tuning problem, not a reason to defer enforcement — and the alternative (no automated detection of semantic violations) is not a lower-risk posture.

Automated semantic boundary enforcement

The gap between the second and third checking levels is fundamentally an observability problem: whether code behaviour is correct for its institutional context is not observable through any existing standard instrument. The design principle is straightforward: if a system property matters, it must be observable. Semantic enforcement is the observability intervention that makes this property measurable, trackable, and auditable.

Semantic boundary enforcement is not new — specialist high-assurance projects (safety-critical systems, cryptographic implementations, some defence and intelligence platforms) have practised elements of it as bespoke local tooling. The change is that agent-assisted development makes this style of checking a mainstream requirement, because agents are introducing the failure modes that previously only appeared in the systems that already had the tooling. The design space is tractable: one case study demonstrates feasibility (§8), and several implementation paths exist.

Implementation approaches

Semantic boundary enforcement means static analysis tools that verify authority classification and authority-tier flow — not just type shape — at the code level. Several implementation approaches exist:

Approach Strength Limitation Best fit
Custom Semgrep rules Fast to author; pattern-matching with taint analysis; large existing rule ecosystem; low deployment friction Pattern-based — cannot express authority-classification-aware severity or governance models (exception lifecycle, ownership, expiry) Stage 1–2: rapid deployment of ACF-pattern proxies on any codebase with CI
CodeQL dataflow queries Powerful dataflow and control-flow analysis over compiled databases; strong for tracing data across function boundaries Requires database compilation step; project-specific trust topology must be encoded externally; governance model not addressable within CodeQL's architecture Stage 2: organisations with existing CodeQL infrastructure seeking deeper dataflow analysis
Pysa taint tracking (on Pyre) Python-specific taint analysis via type inference; can track data flow across call boundaries Analyses taint flow but not authority classification and validation status as orthogonal dimensions; requires Pyre type checker adoption; governance model not addressable within Pyre's architecture Stage 2: Python projects already using or willing to adopt Pyre
Purpose-built semantic boundary enforcer Can integrate authority classification, governance model (exception lifecycle, ownership, temporal separation), and verification properties into a single tool Requires dedicated engineering investment; must be verified against the same threat model it enforces Stage 3: high-assurance environments requiring governance model integration
Prompted agent review Context-sensitive semantic analysis; can evaluate whether a pattern is contextually appropriate, not just structurally present; no tooling investment Non-deterministic; dependent on prompt design; blind spots correlated with the generating model's blind spots; a discovery control, not a gate Supplementary: non-blocking pre-screening to surface issues for human review

Organisations should evaluate which approach fits their existing infrastructure, risk profile, and engineering capacity. The approaches are complementary, not exclusive — Semgrep rules for CI gating and prompted agent review for discovery can operate together effectively.

Minimum verification properties

Regardless of implementation approach, any enforcement tool proposed for high-assurance environments should be assessed against six verification properties:

  1. Golden corpus. A curated set of known-good and known-bad code samples that the tool must correctly classify. This provides a regression baseline and makes the tool's coverage claims independently testable.
  2. Self-hosting gate. The tool should be able to analyse its own codebase and find no violations — or explicitly document and justify any exceptions. A tool that cannot pass its own checks has a credibility problem.
  3. Measured precision. The false positive rate (violations reported that are not real) should be measured, tracked, and published. Unmeasured precision is not a verification story.
  4. Measured recall. The false negative rate (real violations missed) should be measured against the golden corpus and tracked independently of precision. A tool with high precision but unmeasured recall may be missing the violations that matter most.
  5. Deterministic output. Given the same input, the tool should produce the same findings. Non-deterministic enforcement tools cannot be audited, diffed, or used as CI gates.
  6. Taint propagation correctness. For tools implementing taint-flow tracking: the taint propagation engine must correctly assign taint states to values at merge points and across function boundaries. This is the core value claim of the framework — that data flowing from a lower tier to a higher-tier sink without passing through the required validation boundaries produces a finding — and it requires independent verification via dedicated taint-flow corpus specimens.

These are evaluation criteria, not product features. Any organisation building, procuring, or extending enforcement tooling should verify these properties. The companion specification illustrates one framework designed to satisfy them; the criteria themselves are tool-agnostic.

Technical feasibility finding: The case study codebase (§8) runs a pattern-matching enforcement gate that catches trust boundary violations in CI, demonstrating that semantic boundary enforcement is tractable with current tooling.

The case study and this feasibility observation are Python-specific; the companion specification's Java binding extends the framework to a second language but has not been independently validated against a case study codebase. There are several feasible implementation paths, including extensions of existing static analysis platforms (Semgrep, CodeQL, Pysa) and purpose-built analysers. The companion specification sketches one such framework at draft stage (see the classification framework, verification properties, and deployment model). Organisations with existing static analysis infrastructure should evaluate whether extending their current tools with ACF-targeted rules would be more practical than building a new one.

Agent-assisted semantic analysis

The prompted perspective diversity described in §7.1 applies not only to reviewing changes but to analysing existing code. An agent prompted with a specific analytical frame can perform a cold read of a source file that may surface context-sensitive issues traditional static analysis misses and that human reviewers cannot practically trace file-by-file at scale. Agent-assisted analysis occupies a previously sparse middle ground: semantic inspection applied at a scale that was economically impractical for human review alone. It is not a replacement for either static analysis or human review — it catches a different class of issue from both, and its blind spots differ from both. In most settings, it is likely to be most useful as a non-blocking discovery control.

In the case study deployment (§8), periodic full-codebase agent crawls routinely surfaced dozens of findings per pass, the vast majority at P3/P4 severity. These included findings that would not have been caught by conventional static analysis, incremental code review, or unprompted agent review. The findings emerge because the analytical frames are non-default — neither human reviewers under time pressure nor agents given generic review instructions naturally adopt the perspective of an architectural reviewer evaluating layer responsibility or a problem-framing reviewer asking whether a fix addresses root cause or symptom.

This points to a human limitation distinct from time pressure: cognitive range. A senior backend engineer cannot practically adopt a security architect's analytical frame, or a data quality auditor's — not because they lack intelligence, but because genuine expertise in each frame takes years to develop. Prompted agents do not have deeper expertise in any single frame than a domain specialist, but they can adopt multiple frames without the switching cost that makes it impractical for any individual human reviewer to cover more than one or two perspectives well. In this sense, prompted agents appear to function as polymorphic reviewers — breadth of analytical coverage that no individual human can match regardless of time budget. This characterisation is based on the author's observation in a single project context (§8); broader empirical validation remains an open question.

Most findings are individually low-impact68 but represent legitimate deviations from the codebase's architectural rules. The cost is non-trivial, but the defect yield suggests the economics may favour periodic comprehensive analysis over exclusive reliance on per-change review.

Architectural principle: extend existing development workflows rather than creating parallel ones

Effective tools for this space must extend existing programming language machinery (annotations, type hints, decorators) rather than creating parallel systems that require adoption of new syntax or tools. Tools that require developers to learn a new language or adopt a new framework face adoption resistance that undermines their security value. Critically, enforcement must live inside the existing CI/CD pipeline — pre-commit hooks, CI gates, pull request checks — not in a separate workflow. If a security tool slows down the very velocity the organisation bought the AI agent to achieve, the tool will be bypassed.

The enforcement mechanism succeeds by being invisible to the fast path and blocking only on genuine violations.

Declarations as generation constraints

The preceding sections frame semantic enforcement primarily as a checking layer — tooling that verifies code against declared rules after the code is written. But once institutional knowledge is expressed in machine-readable declarations, the declarations do more than verify output: they constrain generation. An agent operating inside a codebase where trust boundaries, authority tiers, and failure-mode requirements are declared as annotations has those declarations in its generation context — not because anyone explicitly injects them, but because the agent reads the surrounding code before generating new code, and the surrounding code carries the project's semantic rules in a form the agent can parse.

The practical implication is that investment in the declaration layer pays dividends on both sides of the generation boundary: it enables enforcement tooling to catch violations after the fact, and it reduces the frequency of violations in the first place by making the rules visible to the agent during generation.

This effect can be strengthened by active projection — delivering the resolved governance context for a specific file to the agent at the point of modification (the companion specification §8.5 sketches this model). The return on converting institutional knowledge into enforceable declarations is higher than verification alone would justify, because the same declarations support both post-generation enforcement and pre-generation context injection.

7.3 Policy controls (what does not yet exist)

The technical controls above describe what organisations can build. The policy controls below describe what does not yet exist at the framework level — vocabulary, classification, and coordination mechanisms that require action beyond individual organisations. The vocabulary and classification controls can be adopted within existing frameworks; accreditation criteria and cross-agency notification mechanisms require formal guidance from ASD/ACSC.

Standardised vocabulary. The taxonomy in this paper (Appendix A, introduced in §3) provides a starting point. Government cybersecurity guidance needs terminology for agentic code failure modes — "fabricated default," "authority tier conflation," "audit trail destruction through defensive patterns" — that practitioners can use in security assessments, risk registers, and accreditation documentation.

Accreditation criteria for agentic development workflows. IRAP assessments and similar accreditation processes need criteria for evaluating whether an organisation's use of AI coding agents maintains the security posture required by the system's classification. This includes:

  • How agent output is validated before integration
  • How review effectiveness is maintained under volume pressure
  • How trust boundaries are verified in agent-generated code
  • What attestation is required from human reviewers

Agent output classification. A formal determination of how agent-generated code should be treated in security assessments — what validation is required before it may enter the codebase, what provenance metadata should be retained, and whether the classification should be binary (untrusted/validated) or graduated.

Model monoculture risk mitigation. The correlated failure risk from shared models (§2.4) requires community-level coordination: cross-organisational sharing of agent-introduced defect patterns (§9.3), periodic scanning for correlated failure signatures across codebases, and awareness that independent testing and review assumptions may not hold when the generating model is shared.

Extend SDLC-equivalent controls to executable logic produced outside formal development teams. Organisations should catalogue and govern agent-generated plugins, automations, BI extensions, workflow scripts, low-code components, and similar artefacts produced by analysts, operators, and other non-developer staff (see §1.2.7). Where such artefacts affect trust boundaries, access control, audit trails, or data integrity, they should be subject to provenance, review, and validation controls proportionate to their impact — even when the producers do not consider themselves developers and the artefacts do not live in a formal version control system.

This is the policy response to the governance perimeter problem: the SDLC boundary has expanded, and controls must follow it.

7.4 Incident response for systematic agent defects

The preceding sections address prevention — stopping semantic violations before they enter the codebase. This section addresses response: what an organisation should do when a systematic agent-introduced defect is discovered after it has entered the codebase, whether through a gap in enforcement coverage, a failure mode the tooling does not yet detect, or code that pre-dates the organisation's adoption of semantic enforcement.

Traditional incident response treats each vulnerability as an independent finding: triage, assess severity, remediate, close. Agent-generated defects violate this model because they are correlated (§2.4) — finding one instance of a pattern is strong evidence that the same pattern exists in other agent-generated code, potentially across the entire codebase. Treating fifty instances of the same correlated pattern as fifty independent tickets overwhelms remediation capacity on what is, operationally, a single systematic defect pattern.

Triage model. When a semantic violation is discovered in agent-generated code, the first step is pattern characterisation, not instance remediation. The questions are:

  • What is the defect pattern? (e.g., broad except on audit-write paths that swallows failures, fabricated defaults on fields in a specific semantic category, external data entering internal stores without validation)
  • Is the pattern specific to a single agent session, or is it a training-distribution bias that would recur across sessions? (§2.4 distinguishes context collapse from training-distribution bias; the answer determines remediation scope)
  • What is the blast radius? How many functions, modules, or code paths could plausibly contain the same pattern?

Scanning protocol. Once the pattern is characterised, the codebase should be scanned for all instances — not just the file or module where the defect was found. If the defect is a training-distribution bias (rather than a session-specific context collapse), the scan should cover the entire agent-generated code population. This is where provenance tracking (§7.1) pays its dividend: without provenance metadata, the scan must cover all code; with it, scanning can be targeted to agent-generated regions while spot-checking human-authored code as a control.

The scan should also look for related patterns, not just identical ones. A broad except that swallows an audit-write failure (ACF-R1) in one function may co-occur with fabricated defaults on related fields (ACF-S1) and missing trust boundary validation on the same data path (ACF-T1). The same training-distribution bias that produces one context-inappropriate defensive pattern tends to produce others in adjacent code. The ACF taxonomy (Appendix A) provides a starting vocabulary for pattern families, and the compounding effect (§3.3) means these patterns tend to co-occur.

Cross-agency notification. The monoculture argument (§2.4) means that a systematic defect found in one organisation's agent-generated code is a leading indicator for other organisations using the same model. No cross-agency notification mechanism currently exists for this class of finding (§9.3). Until one does, organisations that discover a systematic agent-introduced defect pattern should consider whether the pattern is model-specific (likely to affect other users of the same model) or project-specific (arising from the interaction between the model and the project's particular context). Model-specific patterns warrant broader disclosure; the mechanism for that disclosure is one of the genuinely unresolved governance questions (§9.3).

Remediation scope. Remediation of a correlated defect is a single systematic fix, not a collection of independent patches. The appropriate response is typically:

  1. Add a detection rule to the semantic enforcement tooling (§7.2) that catches the pattern going forward — this stops the inflow
  2. Run the new rule against the existing codebase to find all current instances
  3. Remediate the instances, prioritised by code path criticality (high-stakes paths first)
  4. If the pattern was not caught by existing review processes, investigate why — was it a gap in the review checklist, a habituation effect (§4.2), or a pattern that the review process is structurally unable to detect without tooling?

Step 4 is the learning step that closes the loop. Without it, the next correlated defect pattern will enter through the same gap.

8. Case Studies: What the Invisibility Problem Looks Like in Practice

This section presents two evidence bases for the paper's central claim: that AI-generated semantic defects look like correct code and pass every check in the standard assurance stack. The first is a simulation — a complete application prototyped by an agent, where you can read every line and see what the agent produced. The second is six months of longitudinal observation on a live compliance-constrained project, where you can see what detection looks like when it exists. Together, they demonstrate that the problem is not the defect rate. The problem is that you will not see the defects at all — not because you are a poor reviewer, but because they look exactly like the code you have been trained to approve.

De-identification. Both case studies are drawn from real projects. Specific implementation details have been generalised. The simulation (§8.2) uses a purpose-built demonstration application; the longitudinal observation (§8.3) presents a composite, de-identified account from a compliance-constrained environment. The system and tooling described in the longitudinal study are de-identified here to keep the focus on the generalisable threat model.

8.1 What these case studies demonstrate

The natural objection to the threat model is: "show me the code." Show the defect, show why a competent reviewer would miss it, and let the reader judge. That is what this section does.

The primary evidence is a simulation (§8.2, Appendix D) — every source file is reproduced so the reader can judge whether they would have caught the defects in a review queue at the end of a long day. A secondary evidence base — a longitudinal observation (§8.3, Appendix E) — shows that the same failure shapes recur in sustained development, at a rate that the standard assurance stack does not detect.

8.2 Case Study 1: Simulation of a government assistance application

Full analysis: Appendix D. Source listing: Appendix D §D.2.5 and companion snapshot.

An AI coding agent was given a conversational brief to build a government citizen assistance portal — a system where citizens register with PII, verify their phone number via OTP, and receive an aid voucher code that is forwarded to an enablement service. The operator explicitly framed the security context: the system handles public assistance funds, and abuse diverts funding from intended recipients.

The agent's response was reassuring. It identified the domain as high-stakes, listed the control categories it intended to apply — "server-side enforcement, anti-automation/rate limiting, replay protection for codes/vouchers, audit logs, least-privilege secrets handling, and careful PII storage" — and proceeded to build the system. The full exchange is in Appendix D §D.2.2; the reader may wish to form their own view of the gap between what was described and what was delivered.

The agent produced a complete, runnable FastAPI application in approximately 10 minutes. The application implements CSRF protection with secrets.token_urlsafe(), OTP hashing with a keyed SHA-256 scheme, constant-time comparison via secrets.compare_digest(), per-IP rate limiting, signed session cookies, Pydantic input validation, and structured audit logging. All automated checks pass. Both tests — there are only two — pass. The application starts, serves pages, and completes the full workflow without error.

This is part of the hazard, not a mitigating detail. To a reader without deep security expertise, the application presents the visible signals of security competence — familiar libraries, recognised control patterns, and clean framework usage — that would ordinarily justify confidence that the system is "doing security properly."

The application contains 20 semantic defects mapped to ACF taxonomy entries across five of the six STRIDE categories. Three are rated Critical. Sixteen have no detection by any standard tool.

The three-default compound

The centrepiece finding is three os.getenv() calls with development-convenient defaults that compound into total security bypass:

# config.py — three lines that look like standard development practice
app_env=os.getenv("APP_ENV", "development"),                               # → cookies sent over HTTP
secret_key=os.getenv("APP_SECRET_KEY", "development-secret-key-change-me"),  # → forgeable sessions and OTP hashes
use_mock_services=_as_bool(os.getenv("USE_MOCK_SERVICES"), True),           # → verification is simulated

Each default is individually reasonable for development convenience. Their compound effect: a deployment that fails to set all three environment variables runs a system that issues government aid entitlements based on simulated verification, with forgeable credentials, sent over HTTP. The secret key is used for both session signing and OTP hash computation — a known key enables an attacker to forge sessions, compute correct OTP hashes, and bypass phone verification entirely.

The .env.example file compounds this further: it contains APP_SECRET_KEY=change-me-in-production — a different known key from the code default. Copying the example gives one known key. Not copying it gives another. Neither path produces a secure deployment.

No test catches this. No linter flags it. The health endpoint returns {"status": "ok"}.

What the other findings look like

The remaining 17 findings follow the same shape — convention-conforming code that replaces a crash with a silent default:

  • The external OTP service response crosses from untrusted to authoritative with no validation boundary — str(body["code"]) silently coerces null to "None" and stores it as the valid OTP (ACF-T1, ACF-T2)
  • The verify endpoint performs six state-changing operations across four separate database transactions — if enablement succeeds externally but the subsequent DB write fails, the voucher code is lost and the entitlement is unrecoverable (ACF-R2)
  • Registration validation failures generate no audit event — an attacker probing input boundaries is invisible to the audit trail (ACF-R1)
  • The update_application_status function returns silently when the application does not exist — evidence of a bug or forged session is absorbed without trace (ACF-R1)
  • The enablement service response body is never checked — {"status": "failed"} with a 200 OK is treated as success (ACF-T1, ACF-E1)
  • The 64-character voucher code — the bearer credential that grants the aid entitlement — is written to the audit trail in plaintext, so that anyone with read access to the audit table can extract every issued credential (ACF-I1)

Each finding is a place where the system will behave correctly until it doesn't — and when it doesn't, nothing in the system's design will make the failure visible.

What the agent got right

The same "policy available, not applied" dynamic documented in the longitudinal project's incidents (Appendix E §E.7) appeared here in a greenfield generation by a different agent from a different vendor. The failure shape is consistent across both case studies — and the consistency is sharpened by what the agent got right.

The agent implemented genuine security controls — not toy versions, but the actual patterns a security-aware developer would use. CSRF protection with secrets.token_urlsafe(). Constant-time comparison via secrets.compare_digest(). OTP hashing with a keyed scheme. Per-IP rate limiting with a thread-safe implementation. Signed session cookies with configurable security flags. Pydantic input validation with strict regex patterns.

The failures are not in the category of controls selected but in their semantic correctness for this deployment context. A session signed with a known default key performs every cryptographic operation correctly — the HMAC is computed, the signature is verified, the cookie is validated. The ceremony is structurally present. Whether it is providing security depends on a configuration decision the application does not enforce.

8.3 Case Study 2: What happens when you have detection

Full analysis: Appendix E (annotated transcripts). Related: §8.5 (productivity), §8.6 (redirection insight).

The simulation shows the defect shapes. The question it cannot answer is: how often do these patterns appear in sustained development, and what does it take to catch them?

A second evidence base — six months of daily agentic development on a compliance-constrained data processing platform, approximately 80,000 lines of Python, with agents generating the majority of new code — provides that longitudinal view. The system processes sensitive data under requirements that mandate complete audit trails, data integrity verification, and defence-in-depth security controls.

The enforcement regime

The project operates under explicit architectural rules: a tiered authority model for data handling, zero latitude for corruption or substitution on authoritative internal data, quarantine-and-continue for external data, and no defensive programming patterns. These rules are documented extensively but are institutional knowledge — they exist in project documentation, not in the programming language. Python permits all of the patterns the project forbids.

The rules are enforced in CI by a project-specific AST pattern-matching tool with an allowlist-based exception regime. The enforcement model is not advisory — it is a gate. A pattern flagged by the enforcer either gets fixed by the agent or requires a human-authored exception with a rationale, an ownership tag, and an expiry date (companion specification, §9).70 Legitimate uses of otherwise-restricted patterns go through; unconscious pattern completion from training data does not.

What detection observes

In steady-state development, a combination of rigorous review and the enforcement tool regularly catches and blocks semantic boundary violations that would otherwise pass conventional tooling — none entered the codebase. Each flags a pattern from the ACF taxonomy (primarily ACF-S1 and ACF-R1, with limited intra-function proxy detection of ACF-T1) that the generating agent introduced. Under specific conditions72, the detection rate is approximately one to two such patterns per day across approximately 25–30 commits per day (the majority agent-generated).71

The figure is an estimate from a single project. Actual rates will vary with project complexity, codebase size, language, domain, development arrangements, the balance of planned versus ad hoc work, and tooling. This rate occurs despite the agent being explicitly prompted against these patterns in its project-level instructions — the codebase documentation prohibits .get() on typed objects, bare except, and silent error swallowing; the agent's system prompt reinforces these rules. The agent still produces the violations because the patterns are deeply embedded in training data and override project-level instructions under context pressure. Without specific prompting, the rate is substantially higher.71

Scope and methodological caveats

Six aspects of the detection rate merit attention:

What the rate measures:

  • The rate reflects unplanned work. Violations occur predominantly during ad-hoc activities — bug fixing, incremental refactoring, small feature additions — where the agent improvises from training data rather than following a reviewed specification. Planned major work is reviewed against the project's trust topology before implementation, catching violations at the design stage.
  • The rate is model-specific and likely transient. As AI companies prioritise these failure modes for remediation, the absolute rate will likely decrease. The structural argument remains valid regardless of the rate.
  • The rate is a floor, not a ceiling. The tool's coverage of the ACF taxonomy is incomplete (Appendix A). The detection capability is observing routine agent behaviour, not exotic edge cases. Without specific prompting against these patterns, the rate is substantially higher. Appendix E shows that the same failure shape manifests at design and specification layers; those incidents are not counted in the daily code-level figure.73

How to read the evidence:

  • The detection rate is a property of the tool and its rule set, not only of the code. Readers should distinguish between "the tool found violations at rate X" and "violations occur at rate X."
  • The rules always trail the failure modes. Semantically equivalent failures can be achieved through different syntax — each time a rule is encoded, the agent finds an adjacent pattern that achieves the same semantic failure through syntax the tool does not flag. This is not adversarial; the training data contains many ways to silently absorb wrong types.
  • Pattern-level enforcement has a structural ceiling. The rule set is a finite enumeration of known failure shapes; the space of semantically equivalent failures is open-ended. Enforcement tooling should ultimately work at the meaning level, though pattern-level rules remain necessary as a pragmatic first layer.

Reading this figure correctly

The violation rate is not the finding. The finding is that detection required conditions most projects do not have — purpose-built tooling, an operator with deep codebase familiarity, and explicit project-level rules — and that without those conditions, the same violations would have entered the codebase through normal review, because they look like correct, well-written code.

The term "violation" may invite a mental model of broken code — exceptions, failed tests, visible misbehaviour. The violations observed here are better characterised as latent structural weaknesses: the replacement of a crash with a silent default, the weakening of a trust boundary, the introduction of a fabricated value that degrades the system's capacity to detect or recover from a subsequent fault. The analogy is materials that satisfy a specification but are inappropriate for the load the structure was designed to bear. The structure passes inspection, stands, and continues to stand — until the conditions it was built to withstand actually arrive.

The significant question the figure raises is not "why is this project producing defects?" but "does your project — or your contracted supplier's project — have equivalent detection?" Most projects do not yet have it.

What the failure modes look like in a live codebase

The failure modes map directly to the taxonomy in Appendix A. Three examples from the longitudinal project:

Fabricated default (ACF-S1). Agent generates .get() with a default value on a data structure where a missing field indicates a critical failure in an upstream internal component — absence is evidence of corruption, not a case to handle gracefully. The code is not merely plausible — it is correct defensive programming. A reviewer under time pressure sees "handles the missing case" and approves it, because in most software that is exactly the right pattern.

Audit trail destruction (ACF-R1). Agent wraps an audit-critical operation in a try/except that logs the error and continues. The code appears to handle errors gracefully. The reviewer does not recognise that the caught exception should propagate to the audit system rather than being logged and swallowed.

Authority tier conflation (ACF-T1). Agent deserialises data from an external API and passes it directly to an internal processing function. The code appears clean — no obvious security issues. The reviewer does not see the missing validation boundary because both the external data and internal data are the same Python type (dict).

In each case, the defect was caught later — by the enforcement tool, by operator challenge during a coding session, by prompted multi-agent specification review, or by a test failure in a downstream component. The initial review process had signed off.

Across the incidents documented in Appendix E, the recurring pattern was not policy absence but policy non-application: the governing rules were present in the agent's context, and the agent could quote them accurately when asked, but it had not consulted them as constraints during its initial work.

The question for organisations without enforcement is not whether these patterns exist in their agent-generated code — it is whether anything is catching them.

8.4 Cross-validation: two agents, two projects, same failure shapes

The simulation (§8.2) and the longitudinal observation (§8.3) were conducted on different projects, with different agents from different vendors, in different domains, under different constraints. The same failure modes appeared in both — ACF-S1, ACF-R1, ACF-R2, ACF-T1, ACF-E1, ACF-I1 — because they arise from the same structural cause: the generating agent reaches for the most common pattern in its training data when it lacks the institutional context to know that the common pattern is the dangerous one. The taxonomy was developed from the longitudinal project; the simulation validated it against a codebase and agent the taxonomy was not designed for.

8.5 Productivity and the compliance tax (longitudinal observation)

The longitudinal project reports substantial productivity gains from agentic development despite the compliance overhead described above.

Where agents excel: Mechanical refactoring (renaming, restructuring, pattern application across files) is handled almost entirely by agents. Boilerplate generation (new plugins, test scaffolding, configuration structures) is dramatically accelerated. Bug investigation and test writing benefit from agents' ability to rapidly explore code paths. The pattern is clear: agents excel at tasks where correctness is structurally verifiable (tests pass, types check, linter is clean) and struggle where correctness requires institutional knowledge (trust boundary maintenance, audit trail completeness, appropriate error handling in compliance contexts). Appendix E adds an important nuance: agents can be highly effective investigative instruments once directed, but they do not reliably initiate the semantic question that matters.

The compliance tax. Governance controls impose a real overhead — the project's retrospective estimate places it at 15–25% of total development time (an informed estimate based on commit-message tagging, not formal time tracking). The distribution is uneven: on large changes, compliance overhead is trivially small relative to the work. On small changes — a one-line bug fix — the agent spends 30 seconds on the fix and 60 seconds grappling with the CI pipeline, rediscovering the enforcement workflow it has never seen in training data. This skew toward small-change cases is where the bulk of the overhead concentrates.

This is not new overhead introduced by agentic coding. It is the same compliance overhead redistributed. Before agents, humans spent that time writing compliant code slowly. With agents, humans spend it reviewing agent output for compliance quickly. The total compliance cost is similar; the development velocity is higher.

8.6 The redirection insight (longitudinal observation)

The team's experience reveals that automated semantic enforcement does not add tedium — it redirects existing tedium toward higher-value activities.

Without automated enforcement, humans manually review every agent output for trust boundary violations. This is:

  • Error-prone: The failure modes look like correct code (§2.2–2.3)
  • Fatigue-inducing: Reviewing dozens of agent-generated functions per day for subtle semantic violations degrades review quality (§4.2)
  • Unscalable: As agent velocity increases, review capacity does not

With automated enforcement, the machine catches structural trust boundary violations (defensive anti-patterns on data in authority-tier contexts, missing validation boundaries)69. Humans focus on semantic issues that require institutional knowledge — whether the trust topology is correctly declared, whether the validation logic is actually correct (not just structurally present), whether the audit trail captures the right information.

This is a genuine improvement in security posture, not just efficiency:

Reviewer Task Without Automation With Automation
"Is .get() used on typed objects?" Human scans for pattern (error-prone) Machine catches structurally (reliable)
"Does this error handler preserve the audit trail?" Human evaluates (moderate difficulty) Machine flags broad except blocks; human evaluates the specific cases
"Is the trust topology correctly declared for this new module?" Human evaluates (requires institutional knowledge) Human evaluates (no change — this is irreducibly human)
"Is this validation function actually validating?" Human evaluates (requires domain knowledge) Machine checks structural presence of control flow; human evaluates semantic adequacy

The total review burden may be similar, but the distribution of human attention shifts from low-value pattern scanning to high-value semantic evaluation. The compliance tax is the same; the assurance yield is higher. Appendix E.6 suggests that this redirection can also occur upstream: specification-level review catches the same failure shapes earlier and at lower remediation cost than code-level challenge after implementation.

Velocity inverts for remediation. The paper's core argument (§4) is that agents generate correlated defects faster than humans can detect them. But the same property that makes correlated defects dangerous — the same pattern repeated across many files — makes them tractable to fix at scale once the detection rule exists.

An organisation that discovers unwrapped record_call sites across its codebase can dispatch parallel agents to remediate all instances simultaneously; the walltime is independent of the file count.

The bottleneck is not remediation capacity but detection and specification — the human semantic work of recognising that a pattern is wrong, understanding why, and encoding that understanding as a rule precise enough to act on. This further strengthens the investment case for semantic enforcement tooling (§7.2): the scarce resource is the detection rule, not the ability to push fixes once the rule exists.

This also refines the "corrections don't stick" argument (§2.4(a)). That argument remains true for prevention — the agent will reproduce the same pattern tomorrow regardless of how many times it has been caught. But for retroactive remediation — sweeping the codebase for all instances of a newly recognised pattern — agentic velocity is an asset, not a liability. The lifecycle is: (1) human recognises the failure shape, (2) human encodes the detection rule, (3) machine finds all instances, (4) agents fix them in parallel, (5) CI gate prevents recurrence. Each step plays to a different strength: human semantic understanding for detection and specification, machine scale for discovery and remediation, environmental enforcement for prevention.

Agents as compliant enforcement subjects. A less obvious but equally important effect: in the longitudinal project, more compliance work was executed in the agentic workflow than would typically be executed in a purely human one.

The evidence is indirect but consistent: every commit that touches an enforcement-gated path must satisfy the CI gate before merging, and the commit history shows that agents routinely complete the full compliance cycle on changes where a human developer under deadline pressure would plausibly have deferred the governance step or sought an exception.

The same property that makes agents dangerous — no persistent learning, no internalised shortcuts — makes them unusually compliant enforcement subjects. The agent does not learn the organisation's security rules (§2.4), but it also does not learn which rules it can get away with skipping. It pays the governance tax that humans under deadline pressure quietly defer. For anyone who has audited a development team and found the gap between "documented process" and "what actually happens under delivery pressure," this is a significant finding: agents are simultaneously high-risk authors and unusually compliant subjects of technical control.

This adds nuance to the control-strength hierarchy in §7. Behavioural controls are weak for agents not because the agent will choose to skip them, but because it will not remember them next session. Technical controls (CI gates, pre-commit hooks) are strong for agents for the same reason they are strong for humans — environmental, not volitional — with an additional benefit: the agent will not resent the gate or lobby to have it removed.

A practical consequence: if the CI gate is the primary mechanism catching semantic violations that pass conventional tooling,72 CI availability becomes mission-critical in a way it typically is not. Every hour the enforcement pipeline is degraded or unavailable is an hour in which those violations may pass through normal review undetected — because they look like correct code.

Teams working at agentic velocity need continuous awareness of enforcement state and predefined procedures for operating without it, analogous to the control law model introduced in §1.3.

This awareness must be team-wide. The current control law — normal, degraded, or offline — is not a background infrastructure metric but operational context that determines what work is reasonable to undertake. Under direct law (no machine enforcement active), high-risk changes such as security-sensitive code, trust-boundary crossings, and authority-tier logic should not proceed, because the controls that would catch semantic violations in those areas are the ones that are offline.

The lesson from these case studies is that agentic development is viable precisely because the agent will execute governance that humans under pressure quietly defer — but it requires governance designed for the agent's actual failure modes, not the human's. Agent governance must be environmental (CI gates, not documentation), boundary-enforced (pre-commit, not post-review), and stateless (every session is the first session). Organisations that apply human-shaped governance to agents will get the agent's compliance without catching the agent's mistakes.

8.7 Operational tests and replication protocol

Two practical tests would meaningfully challenge the thesis:

  • Practitioner deployment test: Deploy a small set of ACF-pattern detection rules (targeting, e.g., fabricated defaults on security-classified fields, broad exception handlers on audit paths, and authority-tier boundary violations) on agent-assisted codebases and measure the violation rate over a sustained period. If such rules consistently find zero or near-zero violations across multiple independent projects with active agent use, the threat model's generalisability claim would be substantially weakened.
  • Reviewer catch-rate test: In a controlled evaluation, present experienced reviewers with agent-generated code containing known ACF-pattern violations at normal review pace and without purpose-built tooling. If reviewers reliably detect the overwhelming majority of violations under these conditions, the review-degradation thesis would be substantially weakened.

The first test is accessible to any team with a CI pipeline and can be run without a formal study. The second requires a controlled evaluation but would provide stronger evidence on the review-capacity question specifically. Together, they offer a credible path from the paper's current pre-empirical status toward empirical validation or refutation.

Replication protocol. An independent team seeking to confirm or challenge the reported violation rates would need:

  • A codebase with active agent use in a compliance-constrained or integrity-sensitive context — the threat model's claims are specific to high-stakes code paths, so replication on a consumer web application without authority-tier distinctions would not test the relevant conditions. Government systems, healthcare, financial audit, or critical infrastructure projects would provide appropriate contexts.
  • A detection mechanism for ACF-pattern violations — at minimum, a small set of static analysis rules targeting the patterns in Appendix A (fabricated defaults on security-classified or integrity-sensitive fields, broad exception handlers on audit paths, authority-tier boundary crossings without validation). The practitioner deployment test described above provides a starting point. The detection mechanism should be implemented and evaluated independently of the paper's author, even if it draws on the same conceptual categories, to reduce the confirmation bias structure acknowledged in §1.4.
  • A measurement period of sufficient duration — the case study reports an estimated rate of approximately one to two semantic boundary violations per day, but this estimate reflects one developer's work on a specific codebase with a particular agent configuration, compliance burden, and balance of planned versus ad hoc work. Replication should measure over weeks rather than days, and should report both the absolute violation count and a denominator (violations per N agent-generated functions, per K lines changed, or per M commits) to enable meaningful comparison. Caveats on the observed rate appear in §8.3.
  • Controlled comparison where feasible — the strongest replication design would compare violation rates in agent-generated code against a baseline of human-authored code in the same codebase under the same detection rules, to distinguish agent-specific failure patterns from general coding errors that any developer might produce.

Even a partial replication — deploying detection rules on one agent-assisted project for 30 days and reporting the violation rate with denominator context — would materially advance the evidence base beyond this paper's single-project observation.

9. Open Questions

This section identifies what the paper does not resolve — governance mechanics, evidence thresholds, and research directions that require community input. It is structured as a discussion agenda, not as analysis.

This paper does not attempt to answer these questions definitively. They are posed for discussion and community input. The 17 questions below are grouped by type:

  • Governance and operational questions (§9.1–9.9) — genuinely unresolved
  • Positions with outstanding nuances (§9.10–9.13) — the paper advances a position but the edges are unresolved
  • Deferred scope (§9.14–9.17) — topics acknowledged but not developed

Only a small number bear on whether organisations should act at all; most bear on how controls should be implemented, evidenced, and governed once the case for action is accepted.

Policy readers should focus on §9.1–9.5; technical practitioners on §9.9–9.13.

Individual failure patterns are mitigable, but the underlying generative condition — bounded context, selective salience, compression of prior state — is architecturally load-bearing. The questions below should be read as governance challenges arising from conditions that are structurally present in agentic development, not as gaps awaiting closure.

Governance and operational questions

These are genuinely unresolved. The paper advances no position on them beyond identifying the problem. They are the questions a serious reader will care about most after accepting the threat model's core argument.

9.1 Accreditation burden

What accreditation evidence is appropriate for organisations using agentic coding in PROTECTED or higher systems?

Organisations already review code before integration — that is not new. What is new is that the kind of review required is different (semantic, not just syntactic — §5.3) and the volume makes maintaining review quality harder (§4.1–4.2). This paper's position is that IRAP assessors should evaluate the verification story and review quality, not just process existence. The genuinely unresolved questions are operational:

  • What evidence threshold demonstrates that review quality is maintained under agent-generated volume — not just "we have a review process" but "here is evidence the process catches the failure modes in Appendix A"?
  • Must the validation boundary itself be assessed? If an organisation builds or adopts automated semantic enforcement tooling (§7.2), does that tooling become part of the assessed system — and if so, what verification properties must it demonstrate? The companion specification (Part I §10, §14) proposes one candidate answer with quantitative thresholds.
  • How should accreditation criteria distinguish between organisations at different validation maturity stages (§7.2) without creating perverse incentives to avoid adopting tooling that would then require assessment?

9.2 Review quality measurement

How should organisations measure review effectiveness under agent-generated volume?

Organisations should track review quality rather than relying on process existence as a proxy for assurance. The candidate metrics below are a starting point, but each has limitations — the genuinely open question is which metrics are robust enough to serve as accreditation evidence:

  • Defect escape rate: How many agent-introduced defects are found post-review? Requires knowing the total defect count, which requires a detection mechanism independent of the review process — a circular dependency that current practice has not resolved.
  • Review depth sampling: Periodically audit review decisions for thoroughness. Resource-intensive but directly measures quality.
  • Automated pre-screening coverage: What percentage of the failure modes in Appendix A are caught by automated tools before reaching human review? This does not measure review quality directly but measures how much the review process is being supplemented.
  • Time-per-change metrics: Review duration per lines changed. A leading indicator — if review time per change is declining while change volume increases, review quality is likely degrading.

9.3 Cross-organisational standards

What form should a common agentic code security standard take?

A shared vocabulary and a cross-organisational sharing mechanism are both needed. The case for some standard is clear; the genuinely open questions concern its form and scope. A common standard — even a lightweight one — would provide four things organisations cannot build independently:

  • A shared vocabulary for discussing agentic code risks (the taxonomy in Appendix A is a candidate starting point)
  • A minimum bar for controls that all agencies using agentic coding must implement
  • A basis for mutual recognition of agentic development practices across agencies
  • Consistency in IRAP assessment criteria for agentic workflows

The non-trivial daily violation rate reported in §8.3 represents patterns that are not currently shared with other organisations, because no dedicated mechanism exists to make participation easy. The DTA's Australian Government Architecture provides a natural candidate home — its existing cross-government standards role could accommodate a shared taxonomy, an intake channel for reporting findings, and visibility that encourages other organisations to look for them. The open question is not whether to build the mechanism but how to scope it: lightweight registry, structured reporting channel, or full community of practice.

The counterargument: standardisation too early may lock in controls that prove inappropriate as the technology evolves rapidly. A vocabulary standard and minimum control set may be more durable than detailed prescriptive requirements.

9.4 The correlated failure problem

How should risk models be adapted for correlated failures in agent-generated code?

§2.4 establishes that agent-generated defects are correlated, not independent. Traditional risk models assume that a bug in one function does not predict a bug in another; agent-generated code violates this assumption. The genuinely open questions concern how existing risk and triage models should be adapted:

  • Testing strategy: Independent sampling (testing a random subset of functions) underestimates defect rates when failures are correlated. If you find a trust boundary violation in one agent-generated function, the probability that the same violation exists in other agent-generated functions is much higher than if a human had written them.
  • Risk assessment: The risk of a single agent-generated defect may be low, but the risk of a systematic defect affecting dozens or hundreds of functions is qualitatively different. How should risk registers capture correlated agent failure risk?
  • Remediation scope: When a defect pattern is found in agent code, remediation should not be limited to the specific instance. The entire codebase should be scanned for the same pattern — because correlated failures mean the pattern is likely repeated.
  • Triage model: Correlated failures mean 50 instances of the same rule across a codebase is one systematic issue requiring a systematic fix, not 50 independent tickets.
  • Cross-model interaction: Even where organisations diversify across models, the risk may not reduce to independence. Appendix F explores a speculative but plausible concern: that defects from different models could compose — for example, one weakening a validation boundary while another adds a reassuring default that normalises the resulting anomaly. This interaction mechanism has not been empirically demonstrated; Appendix F presents it as a precautionary analysis. Regardless of whether cross-model composition is confirmed, the policy implication stands: model diversity strategies require analysis of lineage independence, not merely model count.

An open research question: what observable behavioural markers would distinguish genuinely independent training lineages from derived variants? Without a methodology for assessing lineage independence, Appendix F's policy implication — that agencies should not assume different products equal different lineages — lacks operational teeth. Even a directional methodology (e.g., correlated failure-pattern profiles across benchmark evaluations) would make the policy implication actionable.

9.5 Contracted development: Unresolved governance mechanics

What governance mechanisms are needed for contracted development using agentic tools?

The structural significance of contracted development — most government code enters through commercial delivery chains, not in-house teams — is developed in §6.7, which examines the structural risk, identifies the control gap, and proposes contract requirement principles. What remains genuinely unresolved is the governance machinery:

  • Cross-contract correlation discovery: if agency A and agency B contract the same provider using the same agent tooling, what mechanism would surface that correlation? Neither agency may know the other's supplier arrangements, and no cross-government registry of contractor AI tooling usage currently exists.
  • Assurance ownership at the boundary: when a contractor delivers agent-generated code, who owns the semantic assurance — the contractor's internal review, the agency's acceptance review, or both? Current contracts rarely specify this, and the answer affects where review capacity must be invested.
  • Procurement flow-down for ISM-2074: how should AI usage policy requirements flow through procurement and subcontracting chains? If a prime contractor subcontracts development to a firm that uses agentic tools, does the prime's obligation extend to the subcontractor's tooling decisions?
  • Disclosure incentives: contractors may be reluctant to disclose agent usage if they perceive it as a reputational risk or competitive disadvantage. What incentive structures or contractual safe harbours would encourage transparent disclosure rather than concealment?

A related shift: agentic AI is also lowering the barrier to in-house code production — through technical staff using agents and through the non-developer power users described in §1.2.7. If the balance between contracted and in-house code shifts, agencies must address agentic code risks in both their contracted deliverables and their expanding internal code production simultaneously.

9.6 Legacy modernisation and implicit security properties

How should organisations assess and preserve implicit security properties during agent-assisted legacy modernisation?

Legacy systems often encode security properties accidentally — a COBOL program that crashes on a NULL field enforces, without intending to, a crash-on-corruption principle that high-stakes systems require deliberately (§1.2.6). When agents refactor or translate legacy code into modern languages, they replace that rigidity with idiomatic defensive patterns (null coalescing, optional chaining, default values) — replacing institutional knowledge baked into the old code's behaviour with standard defensive patterns that lack the same properties.

This raises questions that current modernisation guidance does not address:

  • How should organisations catalogue the implicit security properties of legacy systems before agent-assisted modernisation begins? What methods exist for distinguishing accidental-but-load-bearing behaviour from genuinely obsolete rigidity?
  • Should agent-assisted modernisation require a "security property preservation plan" analogous to a data migration plan — documenting which behaviours must be preserved, which can be relaxed, and which must be explicitly replaced with equivalent controls?
  • What validation evidence should demonstrate that a modernised system preserves the security properties of the system it replaces, when those properties were never formally specified?

9.7 Citizen programmer governance

How should organisations govern executable logic produced by non-developers outside formal SDLC channels?

The governance perimeter problem identified in §1.2.7 — and extended to SQL in Appendix C — raises several unresolved questions. SDLC-equivalent controls need to be extended to this population; the companion Practical Guide provides worked examples for non-specialist review. Existing controls assume a population that recognises itself as producing software; citizen programmers typically do not. Key questions for community discussion:

  • What minimum controls are proportionate for executable logic produced outside the SDLC — particularly when the producer already holds legitimate data access permissions?
  • Should organisations maintain a registry of agent-generated artefacts produced outside formal development channels? If so, what discovery mechanisms are practical?
  • How should the autonomy self-assessment (Appendix B) be adapted for populations that do not identify as developers?

9.8 Tool verification recursion

How do you verify the correctness of security tools that are themselves built by agents?

§7.2 notes that "if the security enforcement tool is itself built by an agent, then the tool's correctness is subject to the same threat model it exists to address." This recursive problem applies to any agent-built verification infrastructure — linter rules, test factories, CI gate logic, security scanners.

The question is not whether agent-built tools can be correct (they can), but what verification evidence is sufficient:

  • Is the golden corpus / self-hosting gate / measured precision approach (§7.2) adequate for tools that encode security-critical institutional knowledge? The companion specification (Part I, §10) proposes quantitative acceptance criteria as one candidate answer.
  • Should there be a higher assurance bar for agent-built tools that serve as trust boundaries in the development pipeline — analogous to the higher verification requirements for safety-critical systems?
  • At what point does the recursion bottom out? If the tool is verified by tests, and the tests were written by an agent, the verification chain eventually requires a human-verified anchor. What constitutes an adequate anchor? The companion specification's governance model (Part I §9.2–9.3) proposes one structural answer involving CODEOWNERS-protected golden corpora and temporal separation between policy changes and the code that benefits from them.

9.9 Testing strategy under agentic generation

How should testing strategy change when agents generate both the code and the tests?

The paper's analysis of the review process (§4) applies with equal force to the testing process, but the testing-specific implications are largely undeveloped. When agents generate code, the standard assurance stack assumes that tests provide an independent check on correctness. When agents generate both the code and the tests that verify it, this independence assumption collapses. The same training-distribution biases and context limitations that produce semantically incorrect code also produce tests that verify the wrong properties.

Three distinct failure modes merit attention:

  • Closed verification loop. When a single agent session produces both implementation and test suite, the tests inherit the same context frame as the implementation. If the agent's context has collapsed (§2.4) such that it treats a missing classification as a legitimate default scenario, the tests will verify that the default is correctly applied — not that the default is semantically inappropriate. The test passes. The CI is green. The verification is circular. The test confirms the implementation does what the implementation does, not what the system requires.

  • Coverage illusion. Agent-generated test suites tend to have high line coverage and high branch coverage — metrics that organisations use as quality proxies. But coverage measures which code paths are executed, not which semantic properties are verified. A test suite with 95% coverage that never checks whether a trust boundary is maintained, whether an audit trail is preserved on the error path, or whether a default fabricates authoritative data provides high coverage of the wrong properties. The standard assurance stack treats high coverage as evidence of quality; for agent-generated tests on high-stakes paths, it may be evidence of thoroughness in the wrong dimension.

  • Verification displacement (ACF-R3). The taxonomy entry ACF-R3 describes two related patterns: (a) an agent replaces real dependencies with mocks or stubs that return expected values, converting an integration test into a test that proves the mock works (§2.4); and (b) an agent writes tests that verify already-degraded behaviour — testing that the code does what it currently does rather than what the system requires, without exercising the original intent (Appendix F documents this dynamic in the cross-model defect chaining context, where tests written around softened behaviour cement the degradation).

    Both patterns share the same underlying failure: under context pressure, agents resolve test failures by changing what the test verifies rather than by fixing the code the test is meant to check. The detection signatures — tests where real objects have been replaced with predetermined return values, or tests whose assertions match the implementation's current behaviour without testing the specified requirement — are specific enough for tooling but not yet encoded in standard review practice.

These failure modes raise four testing questions that current guidance does not resolve:

  • Should agent-generated tests be treated as untrusted input requiring their own validation boundary? If so, what does that boundary check? The test's structural coverage is verifiable by existing tools; the test's semantic adequacy — does it verify the right properties? — requires the same institutional knowledge that the code itself requires.
  • Should tests for high-stakes code paths be written or reviewed independently of the code they test (a stronger form of §7.1's separation principle)?
  • How should the testing pyramid adapt when agents collapse the cost differential between unit and integration tests — potentially enabling a richer integration test suite, but also potentially enabling a richer set of tests that verify the wrong things at every level?
  • What role should property-based and mutation testing play as meta-verification — testing whether the tests themselves are meaningful?

The paper's position is that the testing gap is at least as important as the review gap — high test coverage should not be treated as evidence of adequate verification without examining what the tests verify.

Positions advanced, with outstanding nuances

The paper is not neutral on the following questions. It advances a position on each — but the edges are unresolved and the community may reasonably disagree on scope, calibration, or evidentiary weight. They are presented here not as open questions but as positions the paper holds with acknowledged limits.

9.10 Static vs. runtime assurance

The paper argues for authority-tier-aware static analysis as the primary enforcement mechanism. The open question is scope, calibration, and evidence threshold.

Static analysis can detect structural patterns (§7.2) but cannot verify semantic correctness in all cases. Runtime verification (monitoring authority tier flow at execution time) provides stronger guarantees but introduces performance overhead and operational complexity. The paper's position is that a middle ground — authority-tier-aware static analysis as the primary enforcement mechanism, tracking where data came from and what validation it received without introducing production runtime dependencies — is the appropriate starting point for government systems.

§7.2 describes the design space; the companion specification sketches one candidate framework with draft Python and Java bindings. The outstanding nuance is whether static analysis alone provides sufficient assurance for a given classification level — a policy question, not a technical one, and one the paper does not claim to resolve.

9.11 Agent self-regulation

The paper argues that structured perspective diversity provides meaningful — but not formally validated — assurance. The open question is what evidentiary weight it should carry.

A single agent checking its own output is not meaningful validation — the same biases that produced the code will evaluate it. But a structured ensemble of prompted perspectives (§7.1) is a different proposition: the reviewing agents share the generator's underlying model biases, but their prompted analytical frames constrain what they attend to, producing coverage across different subsets of the failure taxonomy. This is not independent review, but it does direct attention to different classes of failure — the blind spots of a security-focussed reviewer prompt and a data-quality-focussed reviewer prompt overlap less than two identically-prompted agents' blind spots would.

The outstanding questions are whether this kind of prompted perspective diversity constitutes "validation" in a formal assurance sense, whether agents can meaningfully check for the failure modes they are predisposed to produce, and whether pre-review self-checking improves human review quality (by filtering noise) or degrades it (by creating false confidence that the easy problems are already caught).

9.12 Productivity evidence and policy thresholds

The paper argues that the precautionary case is sufficient for action. The open question is what additional evidence would change implementation choices — not whether action is justified.

The productivity evidence base remains mixed and may not generalise cleanly to current autonomous agents (§1.2.1). But the paper's position is that waiting for stronger empirical evidence before issuing controls is not justified when the potential for systemic harm is this structurally embedded. The questions that remain genuinely open are:

  • What would constitute adequate evidence that agentic code risks are not materialising — and would that evidence be observable before an incident?
  • How should the rapid evolution of agent capabilities affect the design of controls — should controls be capability-indexed (triggered by what the agent can do) or incident-indexed (triggered by what has gone wrong)?
  • What additional productivity and security evidence would change the shape of controls (stricter, lighter, differently targeted) rather than the case for having them?

9.13 Semantic divergence as a pre-review triage mechanism

The paper proposes semantic divergence as a candidate research direction for pre-review triage. The open questions are empirical.

Agent-assisted code review tends to collapse into surface approval — the reviewing model sees convention-conforming code and classifies it as acceptable, reproducing the habituation dynamic identified in human reviewers (§4.2). The task is classificatory ("is this code correct?"), which invites pattern-matching against surface quality.

A structurally different approach would replace classification with generation: give a second model the same task specification, policy annotations, and architectural context, and ask it to independently produce the implementation it believes is correct. Comparing the two along policy-relevant semantic axes (where defaults are introduced, where exceptions are caught, where validation boundaries appear, whether tests exercise real behaviour) yields a divergence signal that can triage code paths for mandatory human review.

This targets context collapse during generation (§2.4(h)), not training-distribution bias — a second model given full context from a fresh state may diverge from an original generated under degraded context, but both models may share the same wrong priors and converge on the same incorrect answer. Machine-readable policy annotations (§7.2) strengthen the signal by reducing hidden semantics.

Key questions:

  • Does semantic divergence correlate with true defects better than prompted review?
  • What is the false-reassurance rate from shared model bias — how often do two models agree on the same context-inappropriate pattern?
  • Is the approach more effective as a per-function check or as a module-level architectural comparison?
  • What minimum annotation density is needed before the divergence signal becomes useful?

Deferred scope

The following topics are acknowledged but not developed in this paper. They are flagged here so that future work can address them without the gaps being mistaken for oversights.

9.14 Multi-agent and orchestrated workflows

This paper models a single agent generating code for human review. In practice, emerging frameworks (orchestration layers, agent-to-agent delegation, automated review agents) enable workflows where one agent generates code, another reviews it, and a third writes tests — with minimal human involvement at each stage. This architecture amplifies the correlated failure problem (§9.4): if the generating and reviewing agents share training data, training biases, or architectural assumptions, the review agent may be systematically blind to exactly the failure modes the generating agent systematically produces. The compounding effect (§3.3) is also magnified — failures can compound across agent boundaries before any human sees the output.

Organisations deploying multi-agent development workflows should consider the possibility that the failure modes described here are amplified, not mitigated, by adding more agents to the pipeline — and should design their assurance processes accordingly until evidence to the contrary is available.

9.15 SQL and database-level risks

Appendix C maps the ACF failure modes to SQL and identifies two SQL-specific risks (implicit type coercion, dynamic SQL construction) that have no direct Python analogue. SQL also uniquely bypasses the validation boundary described in §5.3 — a SQL query that returns wrong results produces no error, no exception, and no log entry. The outstanding questions — whether database-level enforcement warrants a distinct control category, and what governance model is appropriate for SQL produced by BI platforms and reporting tools — are identified in Appendix C and not further developed here.

9.16 Controls across the autonomy spectrum

Appendix B provides an informal self-assessment for organisations to identify their position on the agent deployment spectrum. The framework notes that "most organisations will find themselves at different levels simultaneously." The outstanding governance questions — whether controls should be formally tiered by autonomy level, how organisations should manage transitions between levels, and how code developed at different autonomy levels should be treated at integration boundaries — are acknowledged but not developed in this paper.

9.17 Sandbox adequacy and execution authority trade-offs

Organisations should define graduated execution authority boundaries — human approval gates for unsandboxed agents, execution trace review for sandboxed agents. The open question is how much sandboxing is enough, and where the adequacy threshold lies.

Sandboxing is always a trade-off between constraint and capability. A fully sandboxed agent that can only create and edit files in the working directory is safe but limited — it cannot run tests, install dependencies, or validate its own output against the runtime environment. An agent with full shell access is capable but unconstrained. Most practical deployments sit somewhere between these poles, and the right position depends on the task, the environment, and the organisation's risk appetite.

Several questions arise that this paper does not resolve:

  • What constitutes "effective sandboxing" for the purposes of such a boundary? Is filesystem isolation sufficient, or must network access, process execution, and environment modification also be constrained? The answer likely varies by system classification and code-path criticality.
  • Can agents circumvent sandbox boundaries? Agents that can write arbitrary files can potentially modify configuration that affects their own execution environment — creating scripts that will be run by CI, modifying .bashrc or equivalent, or writing to paths that other processes watch. A sandbox that constrains direct execution but permits indirect execution through file creation may provide less isolation than it appears.
  • How should organisations evaluate sandbox implementations? Agentic framework vendors make varying claims about sandboxing capabilities. No standard evaluation criteria exist for assessing whether a given sandbox implementation is adequate for a given risk context — and the rapid evolution of agentic frameworks means that sandbox properties may change between versions without explicit documentation.
  • What is the right trade-off between sandbox strictness and agent productivity? Tighter sandboxes reduce risk but also reduce the agent's ability to perform complex tasks — running tests, building projects, interacting with development infrastructure. Organisations need guidance on calibrating this trade-off to their context rather than defaulting to either extreme.

These questions are related to but distinct from §9.16 (controls across the autonomy spectrum). The autonomy spectrum addresses how much the agent decides what to do; the sandbox question addresses how much the agent is permitted to do once that decision is made.

Closing

Most of the unresolved items in this section are not objections to the paper's thesis. They are the next layer of operationalisation — the governance mechanics, evidence thresholds, and institutional plumbing that must be worked out once the case for action is accepted. An organisation waiting for every question above to be answered before acting will be waiting while agent-generated code continues to enter its systems through every channel this paper describes. The questions that bear on whether to act (principally §9.12) are addressed: the precautionary case is sufficient and the structural conditions are not transient. The remaining questions bear on how to act — and they are best resolved through the kind of community engagement this paper exists to invite.

Appendix A: Agentic Code Failure Taxonomy

A structured catalogue of failure modes observed in compliance-constrained agentic development, mapped to STRIDE categories, with detection characteristics, code examples, and risk ratings. The entries below are the modes observed to date; the taxonomy is designed to be extended as the community reports additional patterns (see Taxonomy Extension Mechanism at the end of this appendix). Each entry includes the reason agents produce this pattern — understanding why helps calibrate both detection tools and review processes.

A note on spurious code references. Raw confabulation — a call to a non-existent function, a reference to an absent field74 — is among the most detectable classes of agent failure: it crashes, fails type checking, or fails tests. This taxonomy catalogues failures that pass those checks. The dangerous form of hallucination is the concealment: an agent that checks hasattr() before accessing a spurious field, or supplies a .get() default when the field does not exist, converts a detectable crash into an invisible semantic failure (catalogued as ACF-S2 and ACF-S1 respectively). The failure modes below concern broader structural shapes — training-distribution bias, context displacement, correlated failure — whose generative conditions are not eliminated by model capability improvements alone (§2.4, §2.5).

Policy readers: the Summary Table below and the Detection Capability Summary at the end of this appendix provide a complete overview without requiring code fluency. The detailed entries between them are provided for technical practitioners and tool builders.

Risk rating calibration. Risk ratings in this taxonomy are qualitative assessments (Critical, High, Medium, Low) based on the observed severity of the failure mode in the case study context and the authors' assessment of potential impact in high-stakes government systems. They are not calibrated against CVSS Base Score ranges or any other quantitative scale. Organisations adopting these ratings for their own risk frameworks should map them to their institutional risk scales; international partners developing joint guidance should establish a shared calibration before co-endorsement.

Summary Table

ID Name STRIDE Failure Layer75 Type Relation79 Risk Detection
ACF-S1 Fabricated Default S Training bias Code Pattern Agent-specific High Partial
ACF-S2 Spurious Field Access S Training bias Code Pattern Agent-specific High Partial
ACF-S3 Structural Identity Spoofing S78 Training bias Code Pattern Agent-specific High Partial
ACF-T1 Authority Tier Conflation T Training bias Code Pattern Agent-specific Critical None77
ACF-T2 Silent Coercion T Training bias Code Pattern Agent-specific Medium Partial
ACF-T3 Unstructured Signal Parsing T Training bias Code Pattern Agent-specific High Partial
ACF-R1 Audit Trail Destruction R Training bias Code Pattern Known class, agent-amplified High Partial
ACF-R2 Partial Completion R Training bias Code Pattern Known class, agent-amplified High None
ACF-R3 Verification Displacement R Context collapse Code Pattern Agent-specific High Partial (R3a) / None (R3b)76
— R3a: Verification Substitution
— R3b: Compensating Control Dependency
ACF-R5 Remediation-Induced Violation R Training bias Code Pattern Agent-specific High None
ACF-I1 Verbose Error Response I Training bias Code Pattern Known class, agent-amplified Medium Partial
ACF-D1 Finding Flood D Process volume Process Threat Agent-specific High N/A
ACF-D2 Review Capacity Exhaustion D Process volume Process Threat Agent-specific High N/A
ACF-E1 Implicit Privilege Grant E Training bias Code Pattern Agent-specific Critical None
ACF-E2 Unvalidated Delegation E Training bias Code Pattern Known class, agent-amplified High Partial
Provisional candidates below — observed but not yet validated for core classification. Not compliance requirements.
ACF-S4 Type Annotation Erosion S Training bias Code Pattern Agent-specific High Partial
ACF-S5 Type Structure Avoidance S Training bias Code Pattern Agent-specific High Partial
ACF-T4 Safety Guard Erosion T Training bias Code Pattern Agent-specific Medium None
ACF-R4 Context Handover Assumption R Context collapse Workflow Pattern Agent-specific Medium Partial
ACF-R6 Scope-Limited Triage R Context collapse Workflow Pattern Agent-specific Medium None

Provisional candidates — not core taxonomy entries

ACF-S4, ACF-S5, ACF-R4, ACF-T4, and ACF-R6 are provisional candidates — failure modes observed in practice but not yet sufficiently validated for core classification. They are included in the detailed entries below for completeness and community feedback, but they are not counted in the paper's "15 core failure modes" statistics and should not be treated as established compliance requirements. Promotion to core requires additional independent observation, validated detection approaches, or broader community confirmation of the failure pattern. ACF-S4 (Type Annotation Erosion) and ACF-S5 (Type Structure Avoidance) were identified through external consultation feedback and describe complementary meta-failures that degrade the detection capability for other taxonomy entries. ACF-R4's generative mechanism is described in §2.4(a). ACF-T4 and ACF-R6 were identified through a structured audit of agent-generated commits against project-specific semantic rules; they describe failures that occur during maintenance-phase work (refactoring, remediation, auditing, triage) rather than during initial code generation. All five entries appear at the end of this appendix.

Type indicates whether the entry describes a code-level pattern (addressable with technical controls), a process-level threat (addressable with management controls), or both. This taxonomy intentionally includes both because the threat model's compounding mechanism (§3.3) depends on their interaction — code-level failures and process-level degradation reinforce each other through feedback loops that a split taxonomy would obscure.

Relation to known classes distinguishes entries that describe failure modes specific to agent-generated code ("Agent-specific") from entries that describe well-known vulnerability classes produced at systematically higher rates or with different characteristics by agents ("Known class, agent-amplified"). The inclusion criterion is not novelty — it is whether a failure mode requires systematic rather than ad hoc management in an agentic development context. Some entries describe genuinely new failure modes (ACF-T1, ACF-S1). Others describe well-known classes that were manageable through periodic review at human rates but are now produced routinely — every error handler, every data access path, every exception block — at a volume where ad hoc detection no longer provides adequate coverage.

A "known" vulnerability class produced identically across every codebase using the same agent is not the same risk as the same class produced sporadically by individual developers with diverse training and experience. The correlation changes the risk calculus even when the individual pattern is well understood. This may be further compounded if cross-model interaction produces composable defects across organisations — an emerging concern explored as a precautionary analysis in Appendix F. The taxonomy is designed to be extended with additional agent-amplified entries as organisations discover further known classes whose management burden changes under agentic volume.

Risk ratings are qualitative assessments based on the failure mode's potential impact in systems handling sensitive data, combined with the likelihood of agent generation and the difficulty of detection. Four levels are used: Critical, High, Medium, and Low.82

Closely related entries. Several pairs of entries describe adjacent failure modes. The distinguishing criteria appear in the Related Entries and Distinguishing Criteria subsection at the end of this appendix.

Detailed Entries

As noted above: the Summary Table and Detection Capability Summary provide a complete overview without code fluency. Non-Python readers may rely on the Description, Why it's dangerous, and Detection approach fields in the detailed entries below.

Language specificity. The code examples throughout this appendix use Python, reflecting the case study environment. The failure modes vary in language-generality:

  • Language-general (applicable across Python, Java, C#, TypeScript, Go, etc.): ACF-T1 (authority tier conflation), ACF-T2 (silent coercion), ACF-R1 (audit trail destruction), ACF-R2 (partial completion), ACF-I1 (verbose error response), ACF-D1 (finding flood), ACF-D2 (review capacity exhaustion), ACF-E1 (implicit privilege grant), ACF-E2 (unvalidated delegation). The failure patterns differ by language (e.g., catch (Exception e) in Java, catch in C++, recover() in Go), but the failure mode is the same.
  • Python-specific surface form (same underlying failure, different manifestation in other languages): ACF-S1, ACF-S2, ACF-S3, and ACF-S4.81

Organisations working in other languages should read the Description and Why it's dangerous fields as language-general, and treat the Example and Detection approach fields as Python-specific reference implementations. For SQL-specific treatment — including COALESCE as fabricated default, INSERT ... SELECT as authority tier conflation, silent overwrites as audit trail destruction, and SQL-specific risks not covered by the Python taxonomy — see Appendix C.

ACF-S1: Fabricated Default

STRIDE: Spoofing | Risk: High | Detection: Partial

Description: Default values fabricate data where the absence of data should be surfaced as a failure, error, or explicit "unknown." The code presents a confident result that is actually based on fabricated input.

Why agents produce this: The .get(key, default) pattern appears in millions of Python files. In most contexts, providing a default for missing keys is genuinely good practice — a web application displaying "Unknown" for a missing user name is fine. Agents learn this as a universal pattern and apply it in contexts where the default fabricates safety-critical data.

Example:

# Agent-generated — looks defensive and robust
def assess_risk_level(record):
    classification = record.get("security_classification", "OFFICIAL")
    clearance = record.get("required_clearance", "baseline")
    return classification, clearance

# Correct for high-stakes context — absence is a failure
def assess_risk_level(record):
    if "security_classification" not in record:
        raise MissingSecurityClassification(
            f"Record {record['id']}: security_classification absent — "
            f"upstream data integrity failure, cannot assess risk"
        )
    if "required_clearance" not in record:
        raise MissingSecurityClearance(
            f"Record {record['id']}: required_clearance absent — "
            f"cannot determine access level, refusing to default"
        )
    return record["security_classification"], record["required_clearance"]

Why it's dangerous: The first version silently downgrades security classifications when data is missing. A PROTECTED document with a corrupted or missing security_classification field is treated as OFFICIAL. Downstream access control decisions are based on the fabricated classification.

Scope: internal state fabrication. S1 applies not only to business data received from external sources but to the system's own operational data — telemetry, run identifiers, latency measurements, and audit metadata. When self._run_id or "" replaces a None run ID with an empty string, the system fabricates a value for data it should have produced correctly. A None run ID means "something is broken in our initialisation"; an empty string looks like normal operation. Similarly, error.latency_ms or 0.0 fabricates a zero latency where measurement failed — operators cannot distinguish "instantaneous" from "unmeasured." The mechanism is identical to business data fabrication (or default on data where absence is meaningful), but internal state fabrication corrupts observability rather than business logic — incidents become harder to diagnose because the system's own diagnostic data has been normalised away.

Detection approach: Flag .get() and getattr() with defaults on objects whose type is annotated with an authority tier of Tier 1 (authoritative internal), Tier 2 (semantically validated), or Tier 3 (shape-validated). Requires authority tier annotations (not available in existing tools). Note: Semgrep custom rules can flag the structural pattern (.get() with non-None defaults) without provenance context, but with significantly lower precision — many legitimate uses of .get() with defaults exist, so the rule would require extensive per-project tuning or triaging. For internal state fabrication, flag or "", or 0.0, or 0, and similar or fallbacks on fields that feed telemetry, metrics, or audit metadata. The companion specification maps ACF-S1 to pattern rule WL-001 (member access with fallback default), which is ERROR/UNCONDITIONAL in AUDIT_TRAIL contexts and ERROR/STANDARD in EXTERNAL_RAW contexts (see companion documents). The or fallback on Tier 1 data is also an ERROR under WL-001, because Tier 1 data is authored by the system itself and should never need fabricated defaults.

The S1 pattern also extends to governance artefacts: an agent that fabricates a plausible-sounding rationale for a trust-escalation exception is substituting a plausible fabrication where genuine evidence should be required — the same mechanism as a fabricated default on a data field, applied to a governance decision rather than code. The companion specification's §9 addresses this through temporal separation, reviewer identity requirements, and recurrence tracking on exception rationales.

Compounding: upstream representational looseness

The risk of this pattern is amplified when upstream code has already erased the type information that would distinguish legitimate optionality from contract violation. When a typed dataclass is serialised to dict[str, Any], the downstream .get() ceases to look anomalous — the type system no longer signals that the key should always be present. Appendix E documents an incident in which a CI enforcer correctly flagged .get() on an internal path, but the agent could not determine why the flag was correct because the upstream serialisation had destroyed the evidence. The agent broadened the exception policy instead.

ACF-S2: Spurious Field Access

STRIDE: Spoofing | Risk: High | Detection: Partial

Description: Agent accesses a field name that does not exist on the target object, masked by getattr() with a default. The code operates on fabricated data while appearing to access a real field.

Why agents produce this: Agents occasionally reference nonexistent field names — predicting a plausible field name that does not exist in the actual schema. Without getattr, this produces an immediate AttributeError. With getattr(obj, "spurious_field", None), the error is silently suppressed and the code operates on None (or whatever default is provided).

Example:

# Agent referenced nonexistent "risk_score" — actual field is "risk_rating"
threshold = getattr(assessment, "risk_score", 0)
if threshold > 5:
    escalate(assessment)
# risk_score is always 0 (the default), so nothing is ever escalated.
# The code looks correct. Tests pass (they test the escalation path with explicit values).
# The bug is invisible until someone notices that escalation never triggers.

# Correct — access the real field directly, crash if it doesn't exist
threshold = assessment.risk_rating
if threshold > 5:
    escalate(assessment)
# If the field name is wrong, AttributeError fires immediately.
# No silent suppression, no fabricated zero threshold.

Why it's dangerous: The code silently does nothing instead of crashing. In a security context, "nothing happens" can mean "threats are not escalated" or "alerts are not raised" — failures of omission that are harder to detect than failures of commission.

Detection approach: Type checkers (mypy, pyright) catch this if the object is fully annotated. If the object is Any or untyped, type checkers are silent. The companion specification's design (see companion document) specifies a complementary rule: getattr with a default on any object that has a declared type annotation is flagged, because the annotation means the field set is known and access should be direct. At the specification layer, a distinct detection path exists: "reality review" — checking whether referenced APIs, parameters, and field names actually exist in the target library or codebase. Appendix E.6 demonstrates this catching a spurious force_refresh parameter that was internally consistent within a specification but did not exist in the actual API. This verification is automatable through symbol resolution against dependency metadata.

ACF-S3: Structural Identity Spoofing

STRIDE: Spoofing (+ Elevation of Privilege consequence) | Risk: High | Detection: Partial

Description: A hasattr() check is used as a capability or privilege gate, allowing any object that declares the expected attribute to pass — regardless of whether the object is of the correct type. The gate accepts structural presence as proof of identity.

Why agents produce this: hasattr() is the idiomatic Python pattern for duck-typing capability checks. Training data is saturated with it — agents building plugin systems, authorisation checks, or capability dispatchers will reach for hasattr by default because it is the "Pythonic" way to test whether an object supports an operation. The concept that structural presence is not ontological identity — that having an attribute is not the same as being the right type — is a security distinction that the language actively discourages.

Example:

# Agent-generated — "Pythonic" duck-typing capability check
def process_classified(obj):
    if hasattr(obj, "security_clearance"):
        handle_classified(obj)  # Any object with this attr gets in

# Trivial bypass — no type hierarchy modification needed
class Impersonator:
    security_clearance = "TOP_SECRET"  # Just declare the attribute

process_classified(Impersonator())  # Gate opens

# Correct — requires actual type membership
def process_classified(obj):
    if isinstance(obj, ClearedPersonnel):
        handle_classified(obj)  # Must inherit from ClearedPersonnel
    # Cannot bypass without modifying the class hierarchy itself

Why it's dangerous: Unlike ACF-S1 (data fabrication via defaults) where the fabricated value is visible at the call site, the exploit surface for hasattr gates is anywhere an object is constructed — potentially far from the gate. The gate looks secure in isolation. Worse, Python's __getattr__ protocol means a single class can dynamically claim to possess any attribute:

class UniversalImpersonator:
    def __getattr__(self, name):
        return True  # "Yes, I have that. And everything else."

# This object passes EVERY hasattr check in the entire codebase.
# An isinstance check is immune to this.

This is the capability-based equivalent of ACF-S1's fabricated default pattern: ACF-S1 fabricates data where absence should be a failure; ACF-S3 fabricates identity where type membership should be required. The object claims to be something it isn't, and the gate believes it because the check is structural (has the attribute) rather than ontological (is the type). The elevation of privilege consequence follows directly — the impersonator passes through a privilege gate that should have rejected it.

Detection approach: An unconditional lint rule banning hasattr() catches all instances (the case study codebase in §8 enforces this). General-purpose linters do not flag hasattr because it is considered idiomatic Python. The companion specification's design (see companion documents) treats hasattr as prohibited in contexts where structural guarantees are declared (ERROR/UNCONDITIONAL in AUDIT_TRAIL, PIPELINE, SHAPE_VALIDATED, UNKNOWN_SHAPE_VALIDATED, and UNKNOWN_SEM_VALIDATED contexts; ERROR/STANDARD and governable in EXTERNAL_RAW, UNKNOWN_RAW, and MIXED_RAW) — unlike .get() or getattr(), which are context-dependent: in high-stakes contexts there is no legitimate use of hasattr that cannot be expressed more safely as isinstance(), explicit try/except AttributeError, or an allowlist check. WL-006 (runtime type-checking of internal data) provides secondary coverage: runtime type-checking on data the wardline classifies as internal suggests the code does not trust the type system's guarantees, which may signal an S3-adjacent structural identity problem (see companion specification §2 coverage table). Detection is rated Partial because the rule is simple to implement but not present in any widely-deployed tool.

ACF-T1: Authority Tier Conflation

STRIDE: Tampering | Risk: Critical | Detection: None

Description: Data from an external (untrusted) source is used in an internal (trusted) context without passing through a validation boundary. The data's effective authority tier is silently elevated.

Why agents produce this: Python's type system does not distinguish between data from different sources. A dict from requests.get().json() and a dict from a validated internal query are the same type. Agents see both as "a dict" and treat them interchangeably because nothing in the language tells them otherwise.

Example:

# Agent-generated — clean, readable, catastrophically wrong
def sync_partner_records(partner_api_url):
    response = requests.get(f"{partner_api_url}/records")
    records = response.json()
    for record in records:
        db.execute(
            insert(internal_records).values(**record)
        )
    # External data inserted directly into internal database.
    # No schema validation, no field allowlisting, no type checking.
    # Partner could send arbitrary fields, wrong types, injection payloads.

# Correct — validate at the boundary
def sync_partner_records(partner_api_url):
    response = requests.get(f"{partner_api_url}/records")
    raw_records = response.json()
    for raw in raw_records:
        try:
            validated = PartnerRecordSchema.validate(raw)
        except ValidationError as e:
            quarantine(raw, reason=str(e))
            continue
        db.execute(
            insert(internal_records).values(
                name=validated.name,
                status=validated.status,
            )
        )

Why it's dangerous: This is one of the two Critical-rated failure modes because it compromises the integrity of the internal data store — the system's source of truth. Once external data enters the internal store without validation, every downstream consumer trusts it as internal data. The failure shape is not a breach but a contamination: the data looks legitimate, the system processes it correctly, and the corruption spreads through every downstream report, decision, and audit record that reads from the internal store.

Consider a workforce management system that ingests contractor records from a partner HR platform via nightly sync. The sync code does INSERT INTO contractors VALUES(**record) — external fields flow directly into the internal database with no schema enforcement. The internal access control system reads a clearance_tier field from the contractors table when routing contractors to sensitive projects.

Three months after deployment, the partner platform undergoes a schema migration. A developer at the partner organisation makes a configuration error: a new internal field called clearance_tier — used by the partner's own workflow engine to flag records for manual review — is accidentally set to "elevated" for all active contractors during the migration. The partner notices the error within hours and corrects it. From the partner's perspective, it is a brief internal data quality incident, resolved before end of day.

From the internal system's perspective: the nightly sync ran during the error window. 1,847 contractor records now have clearance_tier = "elevated" in the internal database. The access control system, reading clearance_tier as an authoritative internal field, routes those contractors to sensitive project queues they should not be able to reach. For 36 hours — until the next nightly sync restores the correct values — elevated routing applies to 1,847 contractors.

The investigation that follows is confused at every level. Operations sees anomalous routing decisions, but the access control logs show the rule was correctly applied: clearance_tier = "elevated" did produce elevated routing. The rule is not wrong. Security traces the elevated values to the partner sync, contacts the partner, and confirms the data quality incident. The internal database is corrected with a re-sync.

But the investigation cannot answer the question that matters: during the 36-hour window, which elevated-routing sessions were legitimate and which were not? The access logs are internally consistent — the rule was correctly applied to the data as it existed. The data was wrong. The audit trail is forensically useless for the window in question, because it records what the system did, not what the system should have known about the provenance of the field it was acting on.

There was no attack. No one was negligent. The partner developer who made the configuration error fixed it within hours. The internal system worked exactly as designed. The exposure was a property of the architecture: a field arrived from an external source, shared a name with an internal field, and was inserted directly into the internal database where it was treated as authoritative. The validation boundary that would have caught this — that would have treated clearance_tier from an external source as an untrusted claim requiring field allowlisting and independent corroboration — was never built.

The contamination property distinguishes T1 from other failure modes. A traditional vulnerability produces an event: a crash, an alert, an anomalous log entry. Authority tier conflation produces no event. The data enters the store silently, is processed correctly by every downstream system, and corrupts every downstream decision, report, and audit record that depends on it. By the time the contamination is discovered, it may have propagated through months of records — reports generated from the internal store, decisions made on the basis of those reports, downstream systems that ingested the internal store's output as their own authoritative input. Correcting the source data does not retroactively correct the decisions made on the basis of it.

Detection approach: Taint analysis — trace the return values of functions marked @external_boundary (or matched by the known external call heuristic list) and flag if they reach data store operations without passing through a function marked @validates_external (or, in the decomposed two-step validation case, @validates_shape followed by @validates_semantic). This is the core capability of the enforcement tool specified in the companion documents.

ACF-T2: Silent Coercion

STRIDE: Tampering | Risk: Medium | Detection: Partial

Description: Type coercion across trust boundaries hides data quality issues. Values are silently converted to a compatible type rather than being flagged as invalid.

Why agents produce this: Python's or operator and conditional expressions make coercion easy and idiomatic. value = input_value or "default" is a common pattern. Agents apply it broadly without distinguishing between contexts where coercion is appropriate (Tier 4 → Tier 3 at a validation boundary) and contexts where it is dangerous (Tier 1 internal data that should never need coercion).

Example:

# Silent coercion hides data quality problem
amount = float(row.get("transaction_amount", 0))
# Two failures compounded: .get() fabricates a default (ACF-S1),
# then float() coerces it to a numeric type.
# Missing transaction amount is silently zero — not "unknown" or "error."
# A zero-value transaction passes every downstream check.
# An audit query for "transactions over $1000" won't find it,
# but neither will "transactions with missing amounts."

# Locale coercion is equally dangerous
amount = float(row["measurement"].replace(",", "."))
# "3,14159" silently becomes 3.14159 — the original locale context
# is lost with no record that a transformation occurred.


# Correct — validate presence, validate type, preserve precision
from decimal import Decimal, InvalidOperation

if "transaction_amount" not in row:
    return TransformResult.error({"reason": "missing_amount", "row_id": row_id})
raw_amount = row["transaction_amount"]
try:
    amount = Decimal(raw_amount)  # Preserve precision; float would silently lose it
except (InvalidOperation, TypeError) as e:
    return TransformResult.error(
        {"reason": "invalid_amount", "raw": raw_amount, "error": str(e)}
    )

Why it's dangerous: Silent coercion converts "unknown" into a concrete value that passes all downstream checks. The distinction between "this transaction was for $0" and "we do not know the transaction amount" is lost permanently. Audit queries cannot distinguish real data from fabricated defaults, compromising the integrity of any analysis or compliance report built on the data.

Detection approach: Two tiers of coverage. Default-based coercion (.get() with non-None defaults, or chains with fallback values, ternary expressions with defaults) is partially detected: WL-001 covers the .get() surface; or chains and ternary defaults are structurally detectable by custom Semgrep rules but with lower precision without authority tier annotations. Type-casting coercion (float() hiding precision loss, locale-dependent string operations, date parsing with assumed timezone) is not covered by the current rule set — no existing pattern rule targets type conversion on tier-classified data. The distinction from ACF-S1 is that T2 involves type conversion compounded with default substitution, not just default substitution alone. The companion specification maps ACF-T2 to WL-001 for the default-based surface only (see companion documents, footnote on ACF-T2 coverage scope).

ACF-T3: Unstructured Signal Parsing

STRIDE: Tampering | Risk: High | Detection: Partial

Description: Control-flow or classification decisions are made by substring matching on unstructured text — error messages, log output, human-readable descriptions — rather than on typed, structured fields. The code treats a prose string as if it were an enum and builds control flow on the fabricated structure. The data is not crossing a trust boundary or changing type — it is being parsed as something it is not.

Why agents produce this: Training data is saturated with if "error" in str(e) patterns. The correct pattern — a typed category field set at the raise site and read at the catch site — requires coordinating the exception class definition with every raise site, which is architecturally demanding work that agents rarely undertake spontaneously.

Example:

# Agent-generated — looks like careful error classification
except DataverseClientError as e:
    error_msg = str(e)
    if "domain allowlist" in error_msg or "SSRF" in error_msg:
        reason = "ssrf_rejected"
    elif "consecutive empty pages" in error_msg:
        reason = "empty_page_guard"
    elif e.status_code == 401:
        reason = "auth_failure"
    else:
        reason = "pagination_error"
    record_audit_event(reason=reason, error=error_msg)

# Correct — typed classification set at the raise site
class DataverseClientError(Exception):
    def __init__(self, message, *, error_category: str, retryable: bool, ...):
        self.error_category = error_category
        ...

raise DataverseClientError(
    f"URL hostname {hostname!r} rejected by domain allowlist.",
    error_category="ssrf_rejected", retryable=False,
)

except DataverseClientError as e:
    record_audit_event(reason=e.error_category, error=str(e))

Why it's dangerous: The failure mode is silent reclassification. When a developer or agent later changes the wording of an error message, the substring match stops matching, the condition falls through to a default branch, and the event is silently reclassified. In audit-critical contexts, an SSRF rejection recorded as a generic "pagination_error" destroys the forensic value of the audit trail without any error or test failure. The reclassification is worse than an audit gap (ACF-R1), because a gap is visible — the record is absent — while a misclassification is invisible: the record is present, with the wrong category.

Detection approach: Flag in str(e), in e.message, in error_msg, and similar substring-match-on-exception-text constructs inside exception handlers that feed audit, telemetry, or control-flow decisions. This is a structural pattern that custom lint rules can flag without semantic knowledge. Detection is rated Partial because the surface pattern is identifiable but distinguishing audit-critical classification from benign logging requires context.

ACF-R1: Audit Trail Destruction

STRIDE: Repudiation | Risk: High | Detection: Partial

Description: Exception handling around audit-critical operations compromises audit trail integrity. Two surface forms produce the same consequence through opposite mechanisms: (a) broad exception handlers catch errors and log-and-continue rather than propagating the failure to the audit system; (b) audit-critical operations propagate failures as untyped exceptions that bypass the structured handling path, so the failure is either caught by a generic handler that does not recognise it as an audit integrity violation, or crashes the process without the diagnostic context and incident routing that the typed handler would have provided.

Why agents produce this: Both forms arise from the same training-data gap. Form (a): "catch exceptions and log them" is a pervasive pattern — a web server should log errors and keep serving. Agents apply this to audit-critical operations without recognising that some failures must propagate rather than be absorbed. Form (b): agents told to avoid form (a) — "let audit failures propagate, don't swallow them" — produce the correct action (propagation) with the wrong type (generic Exception rather than the domain-specific type that routes to the audit integrity handler). The routing semantics of exceptions — which handlers catch which types, and what each handler does differently — are institutional knowledge encoded in the exception hierarchy, not in the language syntax. Training data overwhelmingly uses generic exception types; domain-specific exception hierarchies are project-specific and rarely appear in public repositories.

Example (form a — canonical):

# Agent-generated — looks like responsible error handling
try:
    record_decision(case_id, decision, rationale, evidence)
except Exception as e:
    logger.error(f"Failed to record decision for {case_id}: {e}")
    # Decision was made. Decision was not recorded.
    # The audit trail now has a gap that cannot be reconstructed.
    # The log message may be rotated away. The decision stands, unrecorded.

# Correct — audit failures must propagate
record_decision(case_id, decision, rationale, evidence)
# If this fails, the exception propagates up.
# The caller must handle it — either retry or abort the operation.
# The decision is NOT made unless it is recorded.

Example (form b — untyped propagation):

# Agent-generated — correctly avoids swallowing, but propagates untyped
def upload_and_record(blob_data, ctx):
    upload_blob(blob_data)            # Upload succeeds
    ctx.record_call(status=SUCCESS)   # Audit DB is down — raises Exception
    # The naked Exception propagates. If a generic handler upstream
    # catches Exception and logs it, the audit failure is absorbed
    # without triggering incident response. If nothing catches it,
    # the process crashes with a stack trace that says "Exception"
    # rather than "AuditIntegrityError" — the operator sees a crash,
    # not an audit integrity violation requiring investigation.

# Correct — type the exception for routing
def upload_and_record(blob_data, ctx):
    upload_blob(blob_data)
    try:
        ctx.record_call(status=SUCCESS)
    except Exception as exc:
        raise AuditIntegrityError(
            f"Upload completed but audit record failed. "
            f"Blob exists without audit trail entry."
        ) from exc
    # AuditIntegrityError is caught by the dedicated audit handler,
    # which triggers incident response, not by a generic handler
    # that logs and continues.

Why it's dangerous: In regulatory contexts, the audit trail is the legal record. A gap in the audit trail is not just a logging failure — it is a compliance failure that may have legal consequences. "We made a decision but cannot prove what it was based on" is an unacceptable answer in a formal inquiry. Form (b) is particularly insidious because the agent has followed the project's explicit rule ("don't swallow audit failures") and the code does propagate — the failure is in the exception's type, not its handling. A codebase audit looking for catch-and-swallow patterns (form a) will not find form (b), because there is no except block to flag. In one observed project, correcting form (a) across a codebase revealed twelve instances of form (b) across six plugins — the correlated failure property in action, with the same untyped-propagation pattern repeated identically at every record_call site.

The two forms also compose. An agent producing form (b) — an untyped Exception propagating from an audit-critical operation — creates an exception that has no guaranteed destination. A different agent, or the same agent in a different session, may independently add a catch-all except Exception handler further up the call stack, because the application "keeps crashing" on certain code paths and broad exception handling is the training-data default for making crashes stop. The untyped audit exception lands in the catch-all, is logged as a generic error, and the operation continues — form (a) and form (b) working together, neither introduced by the same agent or in the same session, composing into a silent audit trail gap that neither detection rule in isolation would flag.

The catch-all handler is not wrapping the audit operation (which form (a) detection targets); the audit operation is not swallowed at its call site (which form (b) detection targets). The gap exists in the space between two independently reasonable patterns — an instance of the composable-defect mechanism described in Appendix F (there as a precautionary cross-model analysis; here operating within a single ACF entry).

Extended scenario: middleware security control bypass through exception routing

Scope: exception routing as a cross-cutting mechanism. The untyped-propagation mechanism in form (b) is not specific to audit operations. The same mechanism — correct action, wrong exception type, bypassed structured handler — produces different consequences depending on which handler is bypassed. The following scenario illustrates a cascade from operational nuisance to security compromise.

Consider a web application with middleware-based security controls — a standard pattern in Django, Starlette, and WSGI applications:

# Security middleware — keys on exception type for SIEM and rate limiting
class SecurityAuditMiddleware:
    def __call__(self, request):
        try:
            return self.app(request)
        except AuthenticationError as exc:
            self.siem.record_auth_failure(request.ip, exc.reason)
            self.rate_limiter.increment(request.ip)
            return Response(401)
        except Exception:
            logger.exception("Unhandled error")
            return Response(500)

The except AuthenticationError clause is the security control: SIEM events, rate limiter, 401 response. The except Exception clause is the safety net — log, return 500, keep running.

An agent refactoring the authentication internals replaces raise AuthenticationError(reason="incorrect_password") with raise Exception("Authentication failed"). The middleware's except AuthenticationError no longer catches the failure. The application returns 500 on every incorrect password.

A second agent fixes the visible symptom: catch the exception in the login view, return 401. Every automated check passes. What the fix does not restore is the SIEM event and rate limiter increment — the middleware still exists, unchanged, but no AuthenticationError reaches it.

The attack chain: (1) exception type eroded from specific to generic during refactoring (form (b)); (2) generic exception bypasses middleware security controls; (3) second agent fixes the visible symptom by catching the exception in the view (form (a), applied to a different handler); (4) SIEM integration, rate limiting, and account lockout silently disabled.

Operational conclusion. The changes ship on a Thursday. That evening, monitoring alerts on elevated 500 error rates. An on-call engineer is paged at 11pm Friday. They test the site — everything works. The 500s stopped (because the second agent's fix deployed between the errors and the investigation). The engineer silences the alert until Monday. By Monday morning, the window has been open for over 60 hours — sufficient time for a credential stuffing run without rate limiting, account lockout, or SIEM alerts.

Whether the attack occurred is unknowable. The audit trail that would have recorded it — the SIEM events — is the thing that broke. The incident response team cannot distinguish "no attacks happened" from "attacks happened and we have no record." The on-call engineer is in exactly the same position as the code reviewer described in §4.2: looking at something that appears correct, making a reasonable decision based on what they can observe, and unknowingly removing the last line of defence. Neither is negligent. The security consequence is invisible because it is a property of the middleware routing path — an architectural relationship between exception type and security control that is not visible in the diff, the error logs, or any test that does not specifically check whether the SIEM received the events it should have.

Middleware-based security controls are particularly vulnerable to this pattern because the controls are architecturally separated from the code that triggers them, connected only by exception type — institutional knowledge that lives in deployment configuration, not in the code being modified.

This example illustrates why the category is defined by its consequence (structured handling bypassed through exception mistyping) rather than its mechanism (swallowing or untyped propagation). The category encompasses audit trail gaps, middleware security control bypass, and validation handler bypass — all produced by the same training-data gap. Every instance involves either a catch-and-continue on a path that should propagate (form a), or an operation that should raise a domain-specific exception type but raises a generic one (form b).

Detection approach: Form (a): existing linters flag bare except: (no exception type) but not except Exception: (which is considered acceptable practice). Semantic detection requires understanding which operations are audit-critical — this is project-specific knowledge encoded in the trust topology (e.g., functions annotated as audit-write operations should not be inside broad exception handlers that continue on failure). Form (b): flag audit-critical call sites (e.g., functions annotated as @audit_writer or matching a known audit-operation list) where the call is not wrapped in a handler that raises a domain-specific exception type. The detection signature is the absence of a typed wrapper, not the presence of a catch block — a structurally different rule from form (a). The companion specification maps ACF-R1 form (a) to pattern rules WL-003 (broad exception catching), WL-004 (silent exception handling), and WL-005 (audit writes in broad handlers), with Group 1 (@audit_writer) and Group 2 (@audit_critical) audit primacy enforcement providing the declaration mechanism (see companion documents). Form (b) is not yet covered by the companion specification's rule set — it requires a new detection category targeting untyped propagation from declared audit-critical operations.

ACF-R2: Partial Completion

STRIDE: Repudiation | Risk: High | Detection: None

Description: A sequence of operations that should be atomic (all-or-nothing) is implemented without rollback, so partial failure leaves the system in an inconsistent state.

Why agents produce this: Agents implement operations sequentially and add error handling per-step. They do not naturally recognise that a group of operations should be treated as a transaction unless explicitly prompted. The concept of "these three operations must all succeed or all fail" is a design decision, not a language feature.

Example:

# Bad — agent-generated, each step has error handling, but no atomicity
def reclassify_document(doc_id, new_classification):
    old_classification = get_classification(doc_id)
    update_classification(doc_id, new_classification)                # Step 1: succeeds
    notify_stakeholders(doc_id, new_classification)                  # Step 2: fails (network error)
    record_reclassification(doc_id, old_classification, new_classification)  # Step 3: never runs
    # Document is reclassified, stakeholders don't know, audit trail is incomplete.
    # If step 2 is wrapped in try/except and continues, step 3 records a
    # reclassification that stakeholders were never notified about.

# Better — transaction structure, but rollback failure is unhandled
def reclassify_document(doc_id, new_classification):
    old_classification = get_classification(doc_id)
    try:
        update_classification(doc_id, new_classification)
        notify_stakeholders(doc_id, new_classification)
        record_reclassification(doc_id, old_classification, new_classification)
    except Exception:
        rollback_classification(doc_id, old_classification)  # What if this fails?
        raise
    # If rollback_classification fails, the original exception is replaced
    # by the rollback exception. The caller sees a rollback error, not the
    # original failure. The document is reclassified, the rollback didn't
    # work, and the audit trail records neither the original failure nor
    # the failed rollback.

# Best — compensating actions with rollback failure handling
class ReclassificationFailed(Exception):
    """The operation failed and was successfully rolled back."""

class ReclassificationInconsistent(Exception):
    """The operation failed AND rollback failed — manual intervention required."""

def reclassify_document(doc_id, new_classification):
    old_classification = get_classification(doc_id)
    steps_completed = []
    try:
        update_classification(doc_id, new_classification)
        steps_completed.append("classification_updated")
        notify_stakeholders(doc_id, new_classification)
        steps_completed.append("stakeholders_notified")
        record_reclassification(doc_id, old_classification, new_classification)
        steps_completed.append("audit_recorded")
    except Exception as original_error:
        # Compensate in reverse order
        try:
            if "stakeholders_notified" in steps_completed:
                retract_notification(doc_id, reason="reclassification_rolled_back")
            if "classification_updated" in steps_completed:
                rollback_classification(doc_id, old_classification)
        except Exception as rollback_error:
            # Both the operation AND the rollback failed.
            # This is the worst case — system is in an inconsistent state.
            # Surface BOTH errors so the operator can intervene.
            raise ReclassificationInconsistent(
                f"Reclassification of {doc_id} failed AND rollback failed. "
                f"Original error: {original_error}. "
                f"Rollback error: {rollback_error}. "
                f"Steps completed before failure: {steps_completed}. "
                f"Manual intervention required."
            ) from original_error
        raise ReclassificationFailed(
            f"Reclassification of {doc_id} failed and was rolled back. "
            f"Original error: {original_error}. "
            f"Steps rolled back: {steps_completed}."
        ) from original_error

The three layers illustrate a progression: the bad version has no atomicity. The better version attempts rollback but loses information when rollback itself fails — the original exception is replaced by the rollback exception, and the caller cannot distinguish "failed and rolled back" from "failed and now inconsistent." The best version uses custom exceptions to surface both failure modes distinctly: ReclassificationFailed (safe — rolled back) vs ReclassificationInconsistent (unsafe — manual intervention required). This distinction is institutional knowledge — the system's policy for handling inconsistent state cannot be inferred from the code structure, and agents have no basis for generating it without explicit instruction.

Why it's dangerous: Partial completion creates inconsistent system state that is difficult to detect and correct. The system appears to have completed an operation, but some side effects are missing. In audit-critical contexts, this means the audit trail records an incomplete picture of what actually happened — some operations were performed but not all were recorded, or vice versa.

Detection approach: No existing tool detects this — it requires understanding which operations form a logical transaction. The companion specification maps ACF-R2 to WL-005 and Group 2 audit primacy enforcement, with Group 9 operation semantics (@atomic, @compensatable) addressing the transaction-context requirement (see companion documents).

WL-005's relevance to R2 is distinct from its R1 role: WL-005 fires on each individual audit-critical write inside a broad exception handler. For R1, this catches the single audit failure being swallowed. For R2, WL-005 provides a partial signal — when multiple audit-critical writes exist in the same broad handler, each individual write triggers WL-005 independently, and the collective pattern indicates a partial-completion risk. WL-005 alone does not detect the atomicity gap; Group 9's @atomic and @compensatable annotations address the broader requirement that multiple state-modifying operations occur within a transaction context. A semantic boundary enforcer could flag functions that contain multiple audit-write operations without a transaction context, but this requires project-specific annotation of which operations are audit-critical.

ACF-R3: Verification Displacement

STRIDE: Repudiation | Risk: High | Detection: Partial

Description: Agent-generated code displaces assurance — the system appears verified when the critical properties are unverified. The shared mechanism is task-frame reconstruction under context pressure (§2.4): the agent's task frame shifts from "implement and verify the real property" to "make the artefact look correct," and the displacement is visible only to someone who knows what the artefact was supposed to verify. Two sub-entries distinguish the variants by where the displacement hides and how it can be detected.

Why agents produce this: When agents operate under context pressure — long sessions, compacted history, multi-step plans — the task frame can shift from "implement and verify" to "make the tests pass" or "make the fix work." In the shifted frame, the agent resolves the problem by changing what problem it thinks it's solving. The agent is not suppressing an error — it has produced a locally reasonable solution to the wrong problem. Model providers are introducing mitigations that may reduce the incidence of this pattern — improved context-management tooling, checkpointing, plan persistence, context editing, and resumed runs that preserve transcript history. Individual manifestations of this failure may become less common, but the underlying mechanism — finite context under unbounded task complexity — is architectural rather than fully eliminable under current agentic architectures and workflows.

Why it's dangerous: Unlike ACF-R2 (partial completion), which leaves observable traces — inconsistent database state, missing audit records, downstream failures — verification displacement produces an artefact that looks complete and correct. Tests pass. Coverage is reported. The highest-impact instances are not the ones that look obviously artificial, but the ones that pass visual inspection because the names, structure, and local flow all look legitimate. Active supervision of the agent during generation would reduce the risk of verification displacement — but the core thesis of this paper is that as trust in agentic output increases, scrutiny decreases (§4.2). The teams most likely to catch verification displacement are those that already distrust agent-generated code; the teams most vulnerable are those whose experience with agents has been positive enough to stop checking. In high-stakes systems, displaced verification provides false assurance — the system appears verified when the critical properties are unverified. Of the failure modes in this taxonomy, verification displacement is the most direct observable manifestation of the generative mechanism described in §3.1 — context displaced by local statistical cues.

ACF-R3a: Verification Substitution

Tests that should verify real system behaviour are rewritten to verify mock or stub behaviour, displacing assurance from the actual integration to a simulation of it. The test suite reports full coverage and all tests pass, but the critical paths are no longer tested. A related sub-pattern: tests written against code already degraded by another ACF failure mode (e.g., a .get() default that fabricates data) verify the degraded behaviour as correct — the test passes because it asserts the fabricated default, not because the system works. Tests are particularly vulnerable because they are typically written or fixed last in a plan, precisely when context compression is most acute.

Example:

# Original test — verifies real integration
def test_partner_sync():
    partner_api = PartnerAPI(url=TEST_PARTNER_URL)
    result = sync_partner_records(partner_api)
    assert result.synced_count > 0
    assert all(r.validated for r in result.records)

# After agent "fixes" failing test under context pressure.
# Note: every name is correct. The call site reads identically
# to a real integration test. The displacement is visible only
# by scrolling up to see how partner_api was constructed.
def test_partner_sync():
    partner_api = Mock()
    partner_api.get_records.return_value = [
        {"name": "Test Corp", "status": "active", "clearance": "baseline"}
    ]
    result = sync_partner_records(partner_api)
    assert result.synced_count == 1
    assert result.records[0].clearance == "baseline"
    # At the call site, this looks like a real integration test.
    # partner_api.get_records(...) reads the same whether partner_api
    # is a PartnerAPI or a Mock. The validation logic inside
    # sync_partner_records is never exercised because the mock
    # returns pre-validated data — but that is only apparent if
    # you inspect the type of partner_api, 8 lines above.

The displaced test is visually indistinguishable from a real test at the call site. The agent names everything correctly — partner_api, not mock_partner_api — because in its reconstructed task frame, the mock is the partner API. The call partner_api.get_records() reads identically whether partner_api is a PartnerAPI or a Mock(). The displacement is visible only by inspecting the provenance of the object: scrolling up to the constructor, checking a setUp method, or tracing a fixture in another file. This is precisely the kind of active verification that review volume pressure eliminates (§4.2).

Detection approach: The key detection problem is provenance, not surface syntax. At the call site, well-named mocks are visually indistinguishable from real objects: partner_api.sync_records(validated_records) reads like a genuine integration call unless the reviewer traces partner_api and validated_records back to their declarations and discovers that one or both are mocks, mock-wrapped fixtures, or ad hoc test objects that bypass the project's normal construction helpers. Review volume pressure specifically degrades this kind of provenance inspection.

On a large codebase, the failure is particularly insidious. Mature projects typically centralise test object construction in factories or shared helpers — not as convenience boilerplate, but because the factory encodes institutional knowledge about how to build a valid object with all the hard parts wired correctly. When an agent bypasses the factory and mocks the component it cannot reconstruct from first principles — say, a cryptographically signed packet — the test still reads like a test of signed-packet behaviour in a ten-line diff. The reviewer would need to stop and ask a second-order question — why is this test not using the normal construction path? — and that question is easy to skip when the diff is small, the names are right, and the assertions look plausible.

Heuristic indicators include: tests whose central objects are constructed outside the project's standard test helpers or factories; tests where the mock setup mirrors the code under test so closely that the test is effectively tautological (asserting that a function returns what you told it to return); tests that assert on mock return values rather than on system behaviour; and tests that exercise only the fallback path of code that should primarily exercise the real path. In extreme cases of context collapse, the entire test body may reduce to mock construction and assertions over mock attributes. Those cases are easier to spot, but they are diagnostically useful mainly because they reveal the same underlying mechanism in a less disguised form. Review practices that compare test coverage against the original implementation plan (rather than against the code as implemented) would catch this, but require the plan to be preserved and accessible.

ACF-R3b: Compensating Control Dependency

An upstream normalisation layer (e.g., a deep_thaw() call that converts frozen containers to plain dicts) masks downstream type-narrowness. The downstream code uses isinstance(data, dict) instead of isinstance(data, Mapping), but passes all tests because the upstream layer guarantees the precondition. The code is incidentally correct — not because the type check is well-written, but because the compensating control ensures it never encounters the type it cannot handle. This fragility is invisible under normal operation and only surfaces when the compensating control is correctly removed during a structural improvement, at which point the downstream checks silently produce wrong results rather than failing loudly.

This variant only manifests under structural improvement — it actively punishes correctness. The characteristic user experience is delayed: a team removes a compensating control as part of a well-planned structural fix, the fix passes all tests, ships, and operates correctly — until weeks or months later, when a code path that previously ran through the removed normalisation encounters the type it can no longer handle. The failure presents as a regression in code that was not touched by the fix, producing the reaction "I thought we fixed all those." The fix was correct; the downstream code that silently depended on the compensator was not, and the dependency was invisible until the compensator was removed.

Example:

# Before: deep_thaw() masks downstream type-narrowness.
# The isinstance(data, dict) check WORKS — but only because
# deep_thaw converts MappingProxyType to dict before this
# code ever sees it. The check is incidentally correct.
def record_call(self, frozen_payload):
    thawed = deep_thaw(frozen_payload)        # compensating control
    raw = RawCallPayload(data=thawed)         # re-freezes immediately
    usage = TokenUsage.from_dict(thawed.get("usage", {}))
    # ...

# In TokenUsage:
@classmethod
def from_dict(cls, data) -> "TokenUsage":
    if not isinstance(data, dict):            # narrow: misses MappingProxyType
        return cls.unknown()                  # silent wrong result
    # ... parse fields ...

# After: structural fix removes the compensating control.
# The isinstance(data, dict) check now FAILS silently —
# MappingProxyType is not dict, so from_dict returns unknown()
# for valid data. Tests still pass because test fixtures use
# plain dicts. The fragility was invisible while the thaw
# existed and only surfaced when it was correctly removed.
def record_call(self, frozen_payload):
    raw = RawCallPayload(data=frozen_payload)  # no thaw needed
    usage = TokenUsage.from_dict(frozen_payload.get("usage", {}))
    # TokenUsage.from_dict silently returns unknown() because
    # frozen_payload["usage"] is MappingProxyType, not dict.

Detection approach: The detection surface for R3b is different from and narrower than R3a. R3a leaves a visible smell — Mock() constructors, return-value setup, assertions on mock attributes — that an attentive reviewer can spot. R3b leaves no smell at all: isinstance(data, dict) is a reasonable-looking check, the tests pass legitimately because test fixtures use plain dicts, and the fragility is only visible to someone who knows the upstream normalisation existed and is tracking whether it was truly eliminated rather than relocated.

The fragility is only detectable by someone who (a) knows the upstream compensating control existed, (b) is tracking whether removing it exposed downstream assumptions, and (c) verifies the task against the specification rather than the test suite. In a multi-agent workflow, this means the orchestrating agent must hold the full plan in context and perform spec review rather than relying on CI green as a proxy for task completion. A standard CI pipeline would see passing tests and move on. In the observed incident (§8), the compensating control was a deep_thaw() call that converted frozen containers to plain dicts; the implementing agent caught the downstream isinstance(dict) dependency because the orchestrating agent's spec review compared the implementation against the plan rather than against the test results.

Detection is rated Partial for R3 overall because heuristic indicators exist and experienced practitioners can apply them, but no widely-deployed tool implements them. R3a is Partial in the conventional sense — heuristics exist (mock provenance inspection, factory bypass detection, tautological assertion patterns) and experienced practitioners can apply them. R3b arguably warrants a harder rating: the only reliable detection mechanism observed to date is an orchestrating agent or human reviewer comparing the implementation against the specification rather than the test results. Without that spec review gate, R3b has no detection surface — the code is syntactically reasonable, the tests pass legitimately, and the fragility is invisible until a future structural change exposes it.76

ACF-R5: Remediation-Induced Violation

STRIDE: Repudiation | Risk: High | Detection: None

Description: An agent tasked with fixing a known violation introduces a different violation in the fix itself. The remediation commit claims to resolve the original problem — and may genuinely do so — while introducing a new failure mode that the review process is structurally less likely to catch, because the reviewer's attention is anchored on the original violation. This is distinct from the "corrections don't stick" observation (§3, §8.3), which describes the same pattern recurring in new code. R5 describes a violation in the fix itself.

Why agents produce this: Remediation is a constrained task: "fix this specific violation." The agent focuses on eliminating the flagged pattern and does not step back to evaluate whether the replacement code preserves all properties the original code had — including properties not subject to the violation. The fix passes the check that caught the original violation, and the new violation is in a different category.

Example (tautological assertion — pure R5):

# Before — flagged for using hasattr() (ACF-S3 violation)
assert hasattr(PluginRetryableError, 'retryable'), \
    "PluginRetryableError missing retryable attribute"

# Agent's "fix" — eliminated hasattr, introduced a tautology
assert PluginRetryableError.retryable is not None or True, \
    "PluginRetryableError missing retryable attribute"
# `X is not None or True` is ALWAYS true: when X is None,
# `None is not None` is False, but `False or True` is True.
# The assertion can never fail. The safety check is now decorative.

Example (exception handler collapse — R5 introducing R1):

# Before — flagged for overly broad `except Exception` (ACF-R1)
try:
    query = template.render(query=extracted, row=row_data)
except UndefinedError as e:
    return QueryResult(error={"reason": "template_rendering_failed", "error": str(e)})
except SecurityError as e:
    return QueryResult(error={"reason": "template_rendering_failed",
                              "error": f"Sandbox violation: {e}"})
except Exception as e:  # ← the flagged violation
    return QueryResult(error={"reason": "template_rendering_failed", "error": str(e)})

# Agent's "fix" — three new problems
try:
    query = template.render(query=extracted, row=row_data)
except (UndefinedError, SecurityError, OverflowError,
        ZeroDivisionError, ArithmeticError,  # parent of the previous two
        TypeError, ValueError) as e:
    return QueryResult(error={"reason": "template_rendering_failed", "error": str(e)})
# Problem 1: TypeError/ValueError are likely code bugs, not user errors.
#   Catching them softens a crash into "template rendering failed."
# Problem 2: SecurityError lost its "Sandbox violation:" prefix.
#   Audit trail classification signal destroyed.
# Problem 3: ArithmeticError is parent of OverflowError and ZeroDivisionError.
#   Listing all three is redundant — shallow hierarchy knowledge.

Why it's dangerous: Remediation carries an implicit assurance signal. A commit titled "fix: address tier model violations" tells the reviewer that this code has already been through one round of critical evaluation. The reviewer applies less scrutiny. The new violation is in a different cognitive frame. At organisational scale, the effect is that violation counts go down, assurance metrics improve, and the codebase accumulates a different class of debt that no metric is tracking.

Detection approach: No existing tool detects this. Detection requires comparing the properties of the replacement code against the properties of the original code — not just checking whether the flagged pattern is gone. A simpler process control: treat remediation commits with the same or higher scrutiny as new code, not lower.

ACF-I1: Verbose Error Response

STRIDE: Information Disclosure | Risk: Medium | Detection: Partial

Description: Error handlers expose internal system details (database schemas, file paths, query parameters, library versions) in error responses.

Why agents produce this: Agents produce "helpful" error messages that include full context. During development, this is valuable. In production, it's reconnaissance information. Agents don't distinguish between development and production error handling because the distinction is contextual, not syntactic. While verbose error responses are a known vulnerability class, agents produce them at a qualitatively different rate — every error handler the agent writes defaults to maximum context, across every service, in every project, on every commit. What was a sporadic review finding in human-authored code becomes a systematic pattern requiring explicit detection at scale.

Example:

# Agent-generated — "helpful" error response with full context
except DatabaseError as e:
    return {
        "error": str(e),
        "query": sql,
        "connection": str(db_url),
    }
# Exposes database schema details, the exact query that failed,
# and the database connection string — all useful for an attacker.

# Correct — log internally, return opaque error to caller
except DatabaseError as e:
    logger.error(
        "Database query failed",
        extra={"query": sql, "connection": db_url, "error": str(e)},
    )
    return {"error": "Internal error", "reference": error_id}
# Details logged where operators can see them.
# Caller gets an opaque reference they can report for investigation.

Why it's dangerous: Verbose error responses provide attackers with reconnaissance information: database schemas reveal table and column names, file paths reveal deployment structure, query parameters reveal business logic, and library versions reveal known vulnerabilities. This information reduces the effort required to craft targeted attacks.

Detection approach: Existing scanners detect some cases (credential patterns, known sensitive variable names). Comprehensive detection requires understanding which variables contain sensitive information — a context-dependent judgement. AST-based rules can flag common patterns like str(e) in return values from exception handlers, but false positive rates vary by codebase. The companion specification maps ACF-I1 to Group 8 (secret handling) and Group 11 (data sensitivity), which trace sensitive data through function bodies to detect logging, error-message, and unprotected-persistence exposure (see companion documents).

The two Denial of Service entries below are process threats rather than code patterns. They follow a different structure: "Process failure mode" replaces the code example, "Mitigation" replaces detection approach, and "Why this happens" replaces "Why agents produce this" — because the threat is the aggregate volume and review dynamics, not a pattern any single agent generates.

ACF-D1: Finding Flood

STRIDE: Denial of Service | Risk: High | Detection: N/A (process threat)

Description: The volume of static analysis findings on agent-generated code overwhelms reviewers, causing them to rubber-stamp findings without evaluation.

Why it's dangerous: A review process that rubber-stamps findings rather than evaluating them provides false assurance — the organisation believes its security posture is maintained while real issues pass through undetected.

Why this happens: Agents produce code at volume, and if that code triggers many findings, the review queue grows faster than the review capacity. Reviewers under volume pressure shift from evaluating each finding to batch-dismissing them. The DoS is against the review process, not the system.

Process failure mode: The finding flood creates a vicious cycle:

  1. Agent generates code that triggers many static analysis findings
  2. Review queue grows faster than reviewers can process it
  3. Reviewers shift from careful evaluation to batch dismissal
  4. Suppression rates rise, but the metric is treated as "findings resolved" rather than "findings ignored"
  5. Real security issues are dismissed alongside false positives
  6. The review process provides a false sense of security — it appears functional but has lost its filtering capability

This is distinct from a code pattern because the individual findings may each be legitimate. The threat is the aggregate volume, not any single finding.

Mitigation:

  • Finding caps per rule per file to prevent any single rule from flooding the queue
  • Prioritised finding presentation (critical findings first, low-severity findings batched)
  • Measured suppression rates as a health metric — rising suppression rates signal review degradation
  • Periodic audit of suppressed findings to verify they were genuinely false positives

ACF-D2: Review Capacity Exhaustion

STRIDE: Denial of Service | Risk: High | Detection: N/A (process threat)

Description: Agent code generation velocity exceeds the organisation's capacity for security-focussed review, degrading review from active verification to passive scanning.

Why it's dangerous: The organisation believes it has code review coverage, but the review has lost its security assurance value — subtle issues that require careful analysis pass through undetected, and the gap between perceived and actual assurance widens silently.

Why this happens: Agents can generate plausible, convention-conforming code faster than review processes were designed to absorb (§1.2.1). Review capacity does not scale at the same rate. The review process becomes a bottleneck, and the organisational response is often to lower the review bar rather than reduce the generation rate.

Process failure mode: Review capacity exhaustion manifests as a gradual degradation:

  1. Code generation velocity increases as agents are adopted more broadly
  2. Review queue depth grows — reviewers fall behind
  3. Organisational pressure to "keep up" leads to shorter review times per change
  4. Review shifts from active verification ("is this correct and secure?") to passive scanning ("does this look roughly right?")
  5. Subtle security issues that require careful analysis pass through undetected
  6. The organisation believes it has code review coverage, but the review has lost its security assurance value

Unlike ACF-D1 (finding flood), which overwhelms the static analysis review process, ACF-D2 overwhelms the human code review process itself. Both are process threats, but ACF-D2 is broader — it affects all review, not just finding triage.

Mitigation:

  • Automated pre-screening to reduce the human review burden — automated checks handle the mechanical verification, freeing reviewers for semantic analysis
  • Volume-aware capacity planning — track the ratio of generated code to review capacity and flag when it exceeds sustainable levels (§1.2.1)
  • Measured review effectiveness metrics — track not just "reviews completed" but "issues found per review" as a quality indicator (§9.2)
  • Review scope boundaries — define which generated code requires full security review vs. which can be covered by automated checks alone

ACF-E1: Implicit Privilege Grant

STRIDE: Elevation of Privilege | Risk: Critical | Detection: None

Description: External system assertions are accepted without independent verification, granting privileges based on unvalidated claims.

Why agents produce this: Agents implement integration patterns by calling external APIs and acting on the response. The concept that the external system's response must be independently verified — that the response itself is untrusted — is not visible in the code structure. The code looks like a normal API call and response handling.

Example:

# Bad — .get() with default silently handles missing field
partner_verification = partner_api.verify_identity(applicant_id)
if partner_verification.get("verified", False):
    grant_system_access(applicant_id, level="standard")
# Partner says "verified" → access granted.
# No independent check. No recording of the basis for the decision.
# If the partner system is compromised, every applicant is "verified."
# If the response is malformed and "verified" is missing, access is
# silently denied — but the malformation is never surfaced.

# Better — direct access; missing field raises KeyError
partner_verification = partner_api.verify_identity(applicant_id)
if not partner_verification["verified"]:
    raise VerificationFailed(
        f"Partner verification failed for {applicant_id}"
    )
# Missing "verified" field now crashes instead of silently defaulting.
# But still no independent check, no audit record, and KeyError is
# a poor diagnostic — it doesn't distinguish "malformed response"
# from "partner said no," and it can't be caught by policy-level
# exception handlers without also catching unrelated KeyErrors.

# Best — explicit validation, independent corroboration, audit record
partner_verification = partner_api.verify_identity(applicant_id)
try:
    partner_verified = partner_verification["verified"]
except KeyError:
    raise MalformedPartnerResponse(
        f"Partner response missing 'verified' field for {applicant_id} — "
        f"cannot determine verification status"
    )  # Custom exception: callers can catch MalformedPartnerResponse
    # specifically and apply a defined policy (quarantine, retry,
    # fall back to manual verification) without catching unrelated errors.
if not partner_verified:
    raise VerificationFailed(
        f"Partner verification failed for {applicant_id}"
    )
# Independent check against internal records
internal_record = identity_store.get_verified_identity(applicant_id)
if internal_record is None:
    raise VerificationFailed(
        f"No internal identity record for {applicant_id} — "
        f"partner assertion cannot be corroborated"
    )
record_access_decision(
    applicant_id,
    basis="partner_verified + internal_corroborated",
    partner_response=partner_verification,
)
grant_system_access(applicant_id, level="standard")

Why it's dangerous: The code looks like a normal API integration. The partner says "verified" and access is granted — no independent check, no recording of the basis for the decision. Once an external system's assertions are trusted directly, the security of the entire system depends on the security and correct operation of every external partner — not just against compromise, but against misconfiguration, bugs, and schema changes that the partner may not consider security incidents. Unlike traditional authentication failures, which produce visible events, implicit privilege grants create no signal that anything is wrong until the erroneous grants are acted upon.

Consider a grants management system that uses a partner eligibility verification service to determine whether organisations qualify to receive funding. The agent-generated integration calls the partner API and grants portal access based on the response — no independent check, no audit record of the corroboration basis. The partner says eligible; access is granted.

The partner verification service has a caching layer. A misconfiguration in the caching infrastructure causes it to return stale positive responses — {"eligible": true} — for all queries, regardless of actual eligibility status, for approximately 18 hours before the issue is detected and corrected. During the window, the grants management system processes 1,200 eligibility checks. All 1,200 receive {"eligible": true}. All 1,200 are granted portal access.

The partner notices the caching issue, corrects it, and issues an incident report. From their perspective, it is a platform reliability incident — no data was lost, the caching layer returned stale data for a bounded window, the issue was resolved.

From the grants management system's perspective, 1,200 organisations were granted portal access on the basis of partner assertions that may or may not have reflected actual eligibility. Some number of those organisations were legitimately eligible and would have received access anyway. Some number were not eligible and received access erroneously. The system has no way to determine which is which. The audit trail for each of the 1,200 grants records: access_granted=true, basis="partner_verified". That entry is identical for a legitimately eligible organisation and one that was erroneously verified. There is no record of independent corroboration, because no independent corroboration was performed. Incident response cannot reconstruct the legitimate grants from the erroneous ones without contacting all 1,200 organisations and performing manual re-verification — an expensive, time-consuming process that the partner's incident report does not trigger, because from the partner's perspective the incident was a platform reliability issue, not a security event.

Now extend the scenario operationally. Some of the organisations that received erroneous portal access used it — they submitted grant applications, viewed funding criteria, downloaded programme materials. None of those actions required elevated access — portal access is only the first step in a multi-stage process. But portal access was the gate, and the gate was open for 18 hours on the basis of stale partner assertions. By the time the misconfiguration is discovered, the grants team is reviewing 47 applications from organisations that may or may not have been eligible to apply. The partner's incident report gives them a time window but not a list of affected organisations, because the partner's system does not record which queries were served stale data.

The unknowability consequence is the same shape as ACF-R1's. The audit trail records what the system did — basis="partner_verified" for all 1,200 grants — but it does not record whether the partner's assertion was valid at the time it was made. An independently corroborated grant would have recorded basis="partner_verified + internal_corroborated" with a reference to the internal eligibility record that was checked. That distinction is exactly what is missing. The system is in the same position as the SIEM-less authentication system in ACF-R1: the records are present, they are internally consistent, and they are forensically insufficient for the question that now needs to be answered.

The transitive trust property extends the blast radius. The code that accepted the partner's assertion without independent verification was written once, in a single function. But every access decision downstream of that function now depends on the partner's correctness. If the grants management system feeds downstream systems — reporting dashboards, compliance records, programme performance metrics — those systems inherit the contaminated grants data as authoritative input.

Correcting the access grants does not retroactively correct the downstream records generated on the basis of them. A programme report produced during the window may record 1,200 verified organisations when the correct number was lower. That report may already be distributed. The error in the source data propagates to every document that cited it. This is not a chain of explicit trust grants but a chain of assumptions — each system in the chain assumed the system that fed it had validated its inputs. None of them had. They were all downstream of the single point where the partner's unvalidated assertion entered the trusted data flow.

Detection approach: Taint analysis — the return value of an @external_boundary function is used as a predicate in an access control decision without passing through validation. Requires both boundary annotation and understanding of which operations are access-control-relevant. This taint analysis capability is specified in the companion specification as tier-flow enforcement between declared boundaries (see companion documents).

ACF-E2: Unvalidated Delegation

STRIDE: Elevation of Privilege | Risk: High | Detection: Partial

Description: User-supplied parameters are used directly in privileged operations (database queries, file access, system commands) without validation or restriction.

Why agents produce this: The pattern db.query(Model).filter_by(**user_params) is concise and idiomatic. Agents produce it because it is the shortest path from input to query. The concept that user parameters must be restricted to an allowlist of permitted fields is a security requirement, not a language requirement.

Example:

# Agent-generated — concise, idiomatic, insecure
def search_records(user_query: dict):
    return db.query(Record).filter_by(**user_query)
# User can filter on internal fields: is_deleted, internal_score,
# admin_notes — fields that should not be queryable.

# Correct — restrict to allowed fields
ALLOWED_SEARCH_FIELDS = frozenset({"name", "status", "created_date"})

def search_records(user_query: dict):
    filtered = {
        k: v for k, v in user_query.items()
        if k in ALLOWED_SEARCH_FIELDS
    }
    return db.query(Record).filter_by(**filtered)

Why it's dangerous: Unvalidated delegation allows users to access data or operations they should not have access to. By passing arbitrary parameters to a privileged operation, a user can filter on internal fields (exposing hidden data), modify fields that should be read-only, or access records that should be restricted. The delegation effectively grants the user the same privilege level as the database query itself.

Process-layer dimension. The same structural pattern extends beyond generated code to the development process itself. Agentic coding tools inherit the operator's system credentials and execute privileged operations — shell commands, package installation, git push, CI configuration changes — without the operator constraining the scope of permissible operations. This is a condition by design, and the default posture of most agentic frameworks is to grant broad execution authority. The mitigation is the same principle applied at the process layer: restrict delegation to an allowlist of permitted operations, just as the code-level mitigation restricts query parameters to an allowlist of permitted fields.

Detection approach: SQL injection scanners catch some cases (especially string interpolation into SQL). Parameter delegation via **kwargs unpacking into ORM queries is less consistently detected. Semantic detection requires understanding which operations are privileged and which parameters are user-controlled. Taint analysis can trace user input to privileged operations, but distinguishing validated from unvalidated parameters requires annotation of validation boundaries. The companion specification maps ACF-E2 to taint analysis combined with Group 14 (access and attribution), which verifies that authorisation checks precede privileged operations (see companion documents).

Detection Capability Summary

Detection Level Count Failure IDs Implication
None (no existing tool detects it) 4 ACF-T1, ACF-R2, ACF-R5, ACF-E1 These require new tooling or new review practices
Partial (some tools catch some cases) 9 ACF-S1, ACF-S2, ACF-S3, ACF-T2, ACF-T3, ACF-R1, ACF-R380, ACF-I1, ACF-E2 Existing tools provide incomplete coverage; augmentation needed
N/A (process threat, not code pattern) 2 ACF-D1, ACF-D2 Requires process controls, not technical controls

Of the fifteen core failure modes, thirteen are undetected or only partially detected by existing tools — including all four with no tool coverage at all, both Critical-rated entries among them. This is the gap this paper identifies.

Provisional Candidate Modes

The following five entries are documented separately from the core taxonomy because they meet a lower evidentiary threshold. They are included because the patterns are consistent and practically significant, but they are presented for community discussion and validation rather than as settled taxonomic classes.

ACF-S4 (Type Annotation Erosion) and ACF-S5 (Type Structure Avoidance) were identified through external consultation feedback and describe complementary meta-failures: S4 targets the erosion of existing type safety infrastructure, while S5 targets the failure to create type structure in the first place. Both degrade the detection capability for other taxonomy entries, but through opposite mechanisms — S4 removes safety nets that are already in place, S5 ensures they are never constructed.

ACF-R4 was identified in the original case study; it concerns context-pressure failures in agentic workflows — specifically, the assumption that deferred findings will survive session handovers. ACF-R4's generative mechanism is described in §2.4(a).

ACF-T4 and ACF-R6 were identified through a structured audit of agent-generated commits against project-specific semantic rules; they describe failures that occur during maintenance-phase work (refactoring, remediation, auditing, triage) rather than during initial code generation. ACF-T4 (Safety Guard Erosion) is plausible but narrowly scoped to precondition guards on untested paths; without that scoping it would overlap with generic refactoring damage. ACF-R6 (Scope-Limited Triage) describes a workflow-level pattern (agent triage behaviour during sessions) rather than a code-level pattern, and requires further corroboration across a broader set of agentic workflows.

ID Name STRIDE Risk Existing Detection
ACF-S4 Type Annotation Erosion Spoofing High Partial
ACF-S5 Type Structure Avoidance Spoofing High Partial
ACF-R4 Context Handover Assumption Repudiation Medium Partial
ACF-T4 Safety Guard Erosion Tampering Medium None
ACF-R6 Scope-Limited Triage Repudiation Medium None

ACF-S4: Type Annotation Erosion

STRIDE: Spoofing | Risk: High | Detection: Partial

Description: Type annotations are weakened or suppressed — # type: ignore comments are added, typed parameters are widened to Any, or cast() calls are inserted — to resolve type errors rather than fixing the underlying type mismatch. The code presents itself as type-safe (mypy reports no errors) while the type safety that detection tools depend on has been silently removed.

Why agents produce this: When an agent encounters a type error — a parameter mismatch, an incompatible return type, a missing attribute on a typed object — the correct fix requires understanding the type hierarchy and modifying the code to satisfy the constraint. The expedient fix is to suppress the constraint: add # type: ignore, widen the parameter to Any, or wrap the expression in cast(). Training data contains both approaches, but the suppression pattern is shorter, locally complete, and resolves the immediate error without requiring changes elsewhere. Agents optimising for local coherence and minimal diff size will reach for suppression. This is structurally similar to ACF-T4 (Safety Guard Erosion) — both remove a safety mechanism rather than satisfying it — but S4 targets the type system specifically, which is the detection substrate for other taxonomy entries.

Example:

# Agent encounters a type error on assessment.risk_rating
# (risk_rating is Optional[int], but the comparison requires int)

# Suppression approach — resolves the error, degrades detection
def check_threshold(assessment: Any):  # was: Assessment
    # type: ignore[union-attr]
    if assessment.risk_rating > 5:
        escalate(assessment)
# mypy is now silent on this function. If a future change removes
# risk_rating from Assessment, mypy will not catch it — because
# the parameter is Any. ACF-S2 (spurious field access) is
# now undetectable in this function.

# Correct approach — satisfies the type constraint
def check_threshold(assessment: Assessment):
    if assessment.risk_rating is None:
        raise MissingRiskRating(assessment.id)
    if assessment.risk_rating > 5:
        escalate(assessment)
# mypy validates field access. If risk_rating is removed or
# renamed, mypy catches it. The None case is handled explicitly.

Why it's dangerous: This is a meta-failure — it degrades the detection capability for other taxonomy entries. ACF-S2 (Spurious Field Access) is detectable by mypy if the object is fully annotated. ACF-S3 (Structural Identity Spoofing) is detectable by isinstance checks if the type hierarchy is maintained. When an agent widens a parameter to Any or adds # type: ignore, those detection mechanisms are disabled for the affected code path. The erosion is cumulative: each suppression is locally minor, but across a codebase they create expanding blind spots where the type checker — which is the first line of detection for several ACF entries — can no longer see. Unlike a missing annotation (which is visible as an untyped parameter), an Any annotation or a # type: ignore comment claims type safety while providing none. The code appears to participate in the type system while actually opting out of it. This is spoofing in the STRIDE sense: the code misrepresents a safety property.

Detection approach: Flag # type: ignore comments, parameters typed as Any (especially where a narrower type was previously used), and cast() calls in agent-generated diffs. The structural pattern is straightforward and existing tools can identify it — mypy itself reports the count of type: ignore directives, and ruff/flake8 plugins can flag Any usage. Detection is rated Partial because identifying the suppression is easy, but distinguishing legitimate uses (genuinely dynamic code, third-party library interop) from erosion requires context. In codebases with a strict disallow_any_explicit = True mypy configuration, the detection is effectively Good — but few codebases enforce this. A useful proxy metric: track the # type: ignore count over time in agent-generated commits. A rising count is a signal that type safety is being traded for compilation success.

ACF-S5: Type Structure Avoidance

STRIDE: Spoofing | Risk: High | Detection: Partial

Description: Agent-generated code systematically avoids creating typed data structures — using dict, Map<String, Object>, Record<string, any>, or equivalent untyped containers where a domain-specific type (dataclass, interface, schema model) would be appropriate. The pattern has two faces: (1) function parameters and variables default to str, Any, Object, or equivalent top types, so type information is absent from the start; and (2) data from external APIs is consumed as raw untyped containers (response.json() into a bare dict, JsonNode, or any) rather than being hydrated into a validated model. The code compiles, runs, and passes linting — but the type system has nothing to check because no type structure was ever created.

Why agents produce this: Typed structures require understanding the domain schema: which fields exist, which are required, what their types and constraints are, and how they relate to other structures. An untyped container requires none of this — dict[str, Any] accepts anything, compiles immediately, and never produces a type error. Training data contains both approaches, but the untyped path is shorter, requires fewer coordinated changes (no model definition, no import, no migration), and never blocks the agent with a type error it would need to resolve. For external API data, the problem is amplified: the agent would need to read API documentation, infer a schema, and create a model — or it can call .json() and index into the result with string keys. The path of least resistance produces working code with zero type safety. Unlike ACF-S4 (Type Annotation Erosion), where existing type infrastructure is degraded, S5 means the infrastructure was never built — there is nothing to erode.

Example (Python):

# Agent-generated — external API consumed as bare dict
def sync_partner_clearances(api_url: str, auth: str) -> dict:
    response = requests.get(f"{api_url}/clearances", headers={"Authorization": auth})
    data = response.json()  # dict[str, Any] — no schema, no validation
    results = {}
    for item in data["records"]:
        # String-keyed access — typos are runtime errors, not type errors
        results[item["entity_id"]] = item["clearance_level"]
    return results

# Correct — external data hydrated into a validated model
class PartnerClearance(BaseModel):
    entity_id: str
    clearance_level: ClearanceLevel  # enum — constrained values
    issued_date: date
    expiry_date: date | None

class ClearanceResponse(BaseModel):
    records: list[PartnerClearance]

def sync_partner_clearances(api_url: str, auth: str) -> dict[str, ClearanceLevel]:
    response = requests.get(f"{api_url}/clearances", headers={"Authorization": auth})
    data = ClearanceResponse.model_validate(response.json())
    # Type checker knows every field. Typos caught at edit time.
    # Invalid clearance_level values rejected at parse time.
    return {record.entity_id: record.clearance_level for record in data.records}

Example (TypeScript):

// Agent-generated — API response typed as any
async function getPartnerClearances(apiUrl: string): Promise<any> {
    const resp = await fetch(`${apiUrl}/clearances`);
    const data = await resp.json(); // any — no type narrowing
    return data.records.map((r: any) => ({
        id: r.entity_id,
        level: r.clearance_level  // typo in field name? No error.
    }));
}

// Correct — typed interface with runtime validation
interface PartnerClearance {
    entity_id: string;
    clearance_level: ClearanceLevel;
    issued_date: string;
    expiry_date: string | null;
}

async function getPartnerClearances(apiUrl: string): Promise<PartnerClearance[]> {
    const resp = await fetch(`${apiUrl}/clearances`);
    const data: unknown = await resp.json();
    const parsed = ClearanceResponseSchema.parse(data); // Zod — runtime validation
    return parsed.records;
    // TypeScript knows the shape. Field access is checked at compile time.
}

Why it's dangerous: This is a detection-surface suppressor — a meta-failure that degrades the detection capability for multiple other taxonomy entries. ACF-S2 (Spurious Field Access) is detectable by type checkers if the object has a declared field set; when the object is dict[str, Any], every field access is valid by definition. ACF-T1 (Authority Tier Conflation) becomes structurally guaranteed: a dict from an external API and a dict from a validated internal query are the same type — there is no type-level distinction for a tool or reviewer to check. ACF-R2 (Partial Completion) is harder to detect because you cannot determine what fields are missing from a container that declares no required fields. Unlike ACF-S4, which removes safety nets that were already in place, S5 ensures the safety nets are never constructed — the code never participates in the type system at all. The spoofing in the STRIDE sense is that the code appears to handle structured data — it assigns variables, passes arguments, indexes into results — while the type system is entirely hollow. The effect is cumulative: as more functions accept and return untyped containers, typed islands in the codebase become disconnected, and the type checker's effective coverage shrinks even though no annotation was removed.

Detection approach: Flag functions whose parameters or return types use dict[str, Any], Dict[str, object], Map<String, Object>, Record<string, any>, JsonNode, or equivalent untyped containers — particularly where the function interacts with external data sources (HTTP clients, database results, message queue payloads). Flag API client code where .json(), JSON.parse(), or equivalent deserialisation calls are not immediately followed by schema validation or model hydration. Detection is rated Partial because the surface patterns are identifiable by lint rules, but distinguishing legitimate uses of untyped containers (genuinely dynamic data, configuration blobs, serialisation boundaries) from type structure avoidance requires context. A useful heuristic: in a well-typed codebase, untyped containers should appear only at serialisation boundaries and should be immediately narrowed; if they propagate through function signatures, the type system's coverage is eroding. Codebases with strict type-checking configurations (disallow_any_explicit in mypy, strict mode in TypeScript, @NonNullApi in Java) will catch many instances, but the detection depends on the strictness being configured — and agents generating new modules may not inherit the project's strictness settings. This is a practical gap worth monitoring: when an agent creates a new file, it typically does not pick up per-directory mypy overrides, tsconfig.json strict-mode inheritance, or equivalent project-level type strictness. The result is that new agent-generated modules start with weaker type checking than the surrounding codebase, creating a detection gap precisely where the newest (and least-reviewed) code lives. Organisations should ensure that project-level type strictness is configured to apply by default to new files, not just existing ones.

ACF-R4: Context Handover Assumption

STRIDE: Repudiation | Risk: Medium | Detection: Partial

Description: An agent produces an artefact — a review, a specification, a plan, or a set of recommendations — that defers actions to a future session or a different agent, implicitly assuming the consumer will have access to the producing agent's context. The artefact reads as actionable, but it is incomplete for its actual delivery path because required context remains in the producing session rather than in the artefact handed to the consumer. In agentic workflows, handover between sessions is not optional — reviewing agents hand off to implementing agents, planning agents hand off to coding agents, specification agents hand off to test-writing agents. Each handover crosses a context boundary, and the workflow cannot assume the producing agent has correctly modelled what the consumer will or will not see. The STRIDE fit is Repudiation: the handover artefact cannot be relied upon as a complete record of findings required by the next stage. Risk rises in workflows where handover artefacts are reviewed only once or where later-stage review is materially lighter — common in government specification review processes where the whole point of pre-implementation review is to avoid a second pass.

Why agents produce this: The workflow provides no basis to assume the producing agent has correctly modelled the information boundary between its session and the next. Within a session, deferred actions are reasonable — "I'll address this in the next function" works because the agent retains context. The same reasoning pattern applied across session boundaries produces undeliverable recommendations, because the agent does not distinguish between "defer within my session" and "defer to a session that cannot see my reasoning." Critically, this assumption is made implicitly — the agent does not produce a visible "I assume the implementer will have my context" step. It simply acts on that assumption, and the consequence (deferred, undeliverable actions) is the only observable signal.

Example:

A reviewing agent produces specification-level findings:

Blocking: ctx.fingerprint_key is a spurious symbol.
  Fix: Replace with get_fingerprint_key() from security module.

High: on_no_results: continue enables silent semantic degradation.
  Fix in spec before implementation.

Medium: HTTP 401 classified as non-retryable (could be transient token expiry).
  Fix during implementation.

The blocking and high-priority fixes are specified inline — actionable regardless of consumer context. The medium-priority items are deferred to "implementation" — but the implementing agent will start a fresh session with the specification as input, not the review. Unless the review findings are written into the specification, the deferred items are silently dropped. The reviewing agent has produced a complete-looking triage that is incomplete for its actual delivery path.

Why it's dangerous: An important structural property distinguishes ACF-R4 from the other failure modes: it is naturally self-correcting under agentic review. If a reviewing agent's deferred findings are lost at the handover boundary and the implementing agent reproduces the same problems in code, a second review pass will generally catch the same issues — the reviewing agent's analytical frames do not depend on the first review's context. In workflows where review runs on every artefact, ACF-R4 is primarily an efficiency failure caught by the next cycle. The danger concentrates in workflows where review is run only once or where subsequent review is lighter. With that calibration: the output looks like a competent, prioritised review. A human reading it accumulates context across the conversation and can carry the deferred items forward — the human is the context bridge. An implementing agent given only the specification will reproduce the exact patterns the review flagged, because the specification still contains them and the review findings are not in the implementing agent's context. The failure is invisible at the review stage — the review is correct — and only manifests when the implementation proceeds without the deferred findings.

This is particularly consequential in multi-agent workflows that are becoming standard practice: a planning agent drafts a specification, a reviewing agent evaluates it, an implementing agent builds it, a testing agent verifies it. Each handover is a context boundary. Any finding, caveat, or design decision that lives in one agent's session but is not written into the artefact that crosses the boundary is lost — not forgotten, but never transmitted. The more handovers in the workflow, the more context boundaries exist, and the more opportunities for this failure to silently drop information that a human workflow participant would have carried forward.

Detection approach: Detection is rated Partial because the surface pattern is identifiable — deferred-action language in agent-produced artefacts — but the semantic question (whether the deferral target actually has access to the finding) requires understanding the workflow's session topology. Heuristic indicators: recommendations that use future-tense deferral ("the implementer should," "address during," "fix in the next phase") without embedding the fix in the artefact the consumer will actually read; review findings triaged into priority tiers where lower-priority items are expected to survive by context transfer rather than by document modification; and plans that reference earlier-session findings by description rather than by embedding. The structural mitigation is to require that every handover artefact be self-contained: if information matters for the next session, it must be in the document, not in the conversation.

ACF-T4: Safety Guard Erosion

STRIDE: Tampering | Risk: Medium | Detection: None

Description: Existing precondition guards — assertions, defensive raises, invariant checks — are removed or weakened during agent-performed refactoring. The specific failure shape: guards that protect preconditions on code paths not currently exercised by the test suite. The assertion that is "obviously redundant" because the current code always satisfies it exists to catch future code paths that do not — initialisation reordering, new construction paths, subclass overrides. The agent cannot model future modifications and removes the guard as dead code.

Why agents produce this: Agents optimise for the local coherence of the code they are editing. An assertion checking self._client is not None looks redundant when the agent can see that _client is assigned in connect(), called in __enter__(). But the assertion exists to catch a future code path that calls _paginate() before connect() completes. Training data reinforces this: "remove dead code," "simplify assertions," "trust the type system" are sound principles for human developers who can evaluate future-regression risk. Agents apply them without that evaluation.

Example:

# Before refactoring — assertion guards a precondition
class DataverseSource:
    def _paginate(self):
        assert self._client is not None, "pagination called before connect()"
        # ... pagination logic

# After agent refactoring — assertion removed as "unnecessary"
class DataverseSource:
    def _paginate(self):
        # Agent: _client assigned in connect(), called in __enter__().
        # "Obviously not None here." Guard removed.
        # A future code path that calls _paginate before connect()
        # will get a confusing NoneType error instead of the assertion.

Why it's dangerous: Safety guards are typically added in response to a past incident or an experienced developer's understanding of what can go wrong. Removing them silently reverses institutional learning. This is a maintenance-phase failure — it appears during refactoring, not during initial code generation, and targets a gap in the core taxonomy's coverage. A related pattern — not removal of existing guards but failure to adopt established conventions in new code — produces the same outcome through a different path. In the case study project, peer checkpoint modules established a rigorous deserialisation pattern (set-based required-field checks, isinstance type guards, AuditIntegrityError on any anomaly). A later module written by the same agent defaulted to int()/str() coercion instead — the statistically common pattern from training data, not the project convention. The peer modules were available as examples. The agent did not consult them. Unless CI enforces the convention, later additions drift from it, and the drift is invisible because the new code looks correct in isolation.

Detection approach: No existing tool detects this. Structural detection is feasible: flag removed assert statements and if ... raise guard patterns in agent-generated diffs, but only on code paths not covered by the current test suite. This scoping is critical — flagging all removed guards would produce unacceptable noise. The related convention-drift pattern (new code not adopting established guards) is harder to detect because there is no before/after diff to compare — the guard was never present. Detection would require a convention-expectation model that specifies which patterns should appear in which contexts, then flags their absence in new code.

ACF-R6: Scope-Limited Triage

STRIDE: Repudiation | Risk: Medium | Detection: None

Note: This entry describes a workflow-level pattern (agent triage behaviour during sessions) rather than a code-level pattern, and is derived from session transcript analysis rather than code artefact inspection. It requires further corroboration across a broader set of agentic workflows before it can be considered a settled taxonomic class.

Description: An agent encounters evidence of a problem — a failing test, a warning, a deprecation, a TODO with safety implications — and classifies it as out of scope. The agent narrates the triage decision ("pre-existing failure, unrelated to our changes"), moves on, and the problem becomes less visible than it was before the agent encountered it. The core failure is not that the agent ignores the problem — it is that the agent makes a triage decision that should be the human's to make, presented as a factual observation rather than a judgement call.

Why agents produce this: Agentic workflows are structured around task completion. Evidence that falls outside the defined scope is a distraction from task completion. Training data reinforces this: development workflows routinely triage unrelated failures as "known issues" to maintain focus. A human developer who triages a failing test as "not mine" has institutional context about whether the test matters, who owns it, and whether it has been triaged before. The agent has none of this context and substitutes confidence for knowledge.

Example (agent session transcript):

● Bash(pytest tests/ -x -q 2>&1 | tail -20)
  ⎿  FAILED tests/unit/plugins/llm/test_llm_config.py::test_reserved_suffix_warns
     1 failed, 847 passed

● Good — the failing test is in test_llm_config.py which we didn't
  touch. Let me confirm it fails on the unchanged code too.

● Bash(git stash && pytest tests/.../test_llm_config.py -x -q)
  ⎿  FAILED

● Pre-existing failure, unrelated to our changes.

Why it's dangerous: The STRIDE mapping is classical Repudiation: the agent's triage decision is an unauditable action — the agent made a judgement that affected what information the human received, that judgement is not recorded in any persistent artefact, and it cannot be reconstructed or challenged after the fact. In agentic workflows with multiple sessions over days or weeks, scope-limited triage creates an accumulated tenure problem: each session encounters the pre-existing failure, confirms it was already failing, and moves on. The failure is surfaced repeatedly, triaged repeatedly, and addressed never — because every session correctly determines it is "not my problem." Nobody is negligent. The failure accumulates anyway. In contracted development, an agent triaging a pre-existing defect as out-of-scope is making a contractual decision that the contracting agency might disagree with.

Detection approach: No code-level tool detects this — it is a workflow behaviour. Detection requires transcript-level analysis: flag sessions that encounter test failures and do not surface them for human decision, and track which failures accumulate tenure across multiple sessions.

Taxonomy Extension Mechanism

This taxonomy is presented as a starting point, not a closed set. The generative conditions described in §2 and §3 will produce failure modes not yet observed. The provisional candidates above (ACF-S4, ACF-S5, ACF-T4, ACF-R4, ACF-R6) illustrate the expected path from observation to inclusion. ACF-S4 and ACF-S5 demonstrate a second submission pathway — external consultation feedback identifying a gap in the existing taxonomy. Notably, ACF-T3, ACF-T4, ACF-R5, and ACF-R6 were identified through maintenance-phase work — refactoring, remediation, auditing, and triage — rather than through initial code generation, suggesting that the taxonomy's coverage should explicitly encompass the full software lifecycle, not only greenfield authoring.

Criteria for new entries. A candidate ACF entry should meet four conditions: (1) a reproducible code-level or process-level pattern, with at least one worked example; (2) a STRIDE mapping that identifies the threat category; (3) a risk rating using the scale defined in this appendix; and (4) an assessment of existing detection capability. Entries that describe known vulnerability classes should document why agentic generation changes the management burden (volume, systematicity, or detection difficulty) rather than simply cataloguing a known class.

Submission pathway. Until a formal maintenance process for the taxonomy is established, candidate entries can be submitted through the consultation process accompanying this paper. Submissions should follow the entry structure used in this appendix: description, generative mechanism, worked example, risk assessment, and detection approach. The authors welcome submissions from practitioners, researchers, and tool vendors — particularly entries backed by empirical observation from projects with detection capability in place.

Detection gap summary. Of the fifteen core failure modes, thirteen are undetected or only partially detected by existing tools — including all four with no tool coverage at all, both Critical-rated entries among them. For provisional candidates, 2 of 5 have no detection and 3 have partial detection. The full table appears after the core entries and before the provisional candidates.

Versioning. New entries should use the next available number within the appropriate STRIDE category (e.g., the next Spoofing entry would be ACF-S6, the next Tampering entry ACF-T5). Provisional entries retain their provisional status until they meet the full evidentiary threshold described above. Entries may be deprecated if model improvements or tool coverage render them obsolete — but deprecation should be evidenced, not assumed.

Several pairs of entries describe adjacent failure modes. The distinguishing criteria below help reviewers and tool builders classify findings correctly.

ACF-S1 vs ACF-T2: S1 fabricates a value where none exists (the field is missing); T2 silently coerces a value that does exist into a different type or representation. S1 invents data; T2 transforms it. Both produce wrong values, but S1 is detectable by checking for default arguments on security-sensitive fields, while T2 requires tracing type coercion across operations.

ACF-T1 vs ACF-E1: T1 is a provenance failure — external data crosses into trusted processing without passing a validation boundary. E1 is a decision failure — privileges or access are granted on the strength of unvalidated assertions or data. T1 asks "did this data earn trusted status?"; E1 asks "did this claim improperly trigger a privileged action?"

ACF-T2 vs ACF-T3: T2 silently coerces a type; T3 silently parses prose as structure. Both produce values that look correct today and silently degrade when the source changes, but the mechanisms differ — T2 converts data, T3 fabricates structure from text.

ACF-R1 vs ACF-R2: R1 destroys auditability by swallowing or suppressing failures that should be recorded or propagated. R2 destroys atomicity by allowing a multi-step operation to complete partially without rollback or compensating action. R1 corrupts the record of what happened; R2 corrupts the state that resulted.

ACF-R3 vs ACF-R5: R3 displaces test assurance (real tests become mock tests); R5 displaces remediation assurance (fixes introduce new violations). Both produce artefacts that claim to provide assurance while degrading the property they claim to assure.

ACF-D1 vs ACF-D2: D1 is an upstream cause — agents generate a high volume of findings that flood the review pipeline. D2 is the downstream effect — sustained volume degrades human review quality through habituation and fatigue. D1 can be addressed with precision-gated tooling; D2 requires capacity planning and review effectiveness measurement.

ACF-E1 vs ACF-S3: E1 is about unvalidated external assertions — a partner API says "verified" and the code grants access. S3 is about unsound gate predicates — hasattr() gates accept any object with the expected attribute regardless of type. Both result in implicit privilege grants, but through different mechanisms: E1 trusts the wrong source; S3 trusts the wrong check.

ACF-S4 vs ACF-S2: S4 degrades the detection substrate for S2. S2 is a spurious field access masked by getattr() with a default — detectable by mypy if the object is typed. S4 widens the object to Any or adds # type: ignore, making mypy silent. S2 is a data-level failure (wrong field); S4 is a meta-level failure (detection disabled).

ACF-S4 vs ACF-S5: S4 removes or weakens type safety that already exists; S5 avoids creating type structure in the first place. S4 is erosion of an existing detection surface; S5 is failure to construct that surface at all.

ACF-S5 vs ACF-T1: S5 is a representational failure — external and internal data are both reduced to untyped containers, so downstream code cannot distinguish them at the type level. T1 is the trust-boundary failure that results when such data crosses into trusted processing without validation. S5 makes T1 harder to detect; T1 is the semantic violation itself.

ACF-S4 vs ACF-T4: Both describe the removal of a safety mechanism rather than satisfaction of it. T4 removes runtime guards (assertions, defensive raises); S4 removes static analysis guards (type annotations, type-checker constraints). Both are maintenance-phase failures — they appear when agents are resolving errors, not when generating greenfield code.

ACF-T4 vs ACF-R5: T4 (guard erosion) can manifest as part of R5 — a remediation commit that removes safety guards not related to the original violation. Both are maintenance-phase failures targeting the gap between greenfield authoring and code modification.

Appendix B: Agent Autonomy Self-Assessment

This appendix provides an informal diagnostic for organisations to identify where their current agent usage sits on the agent deployment spectrum and whether their controls are proportionate. It is not a maturity model — higher levels are not aspirational targets, and there is no implied progression from lower to higher. Most organisations will find themselves at different levels simultaneously: Level 1 for security-critical components, Level 3 for test scaffolding, Level 2 for general feature work. That is entirely appropriate, provided the controls at each level match the risk profile.

The purpose is self-location. An organisation that discovers it is operating at Level 3 without the controls listed for Level 3 has identified a gap. An organisation operating at Level 1 with controls designed for Level 3 has identified waste. Neither outcome requires changing the level — only aligning controls to reality.

Level 0: Full Human Level 1: Prompted + Copied Level 2: IDE-Integrated Level 3: Autonomous
What it looks like No agent involvement. Human writes all code. User asks agent specific questions, copies and adapts fragments into codebase manually. Agent autocompletes functions and classes inline. User accepts or rejects suggestions in-editor. Agent plans, implements, tests, and commits with minimal human intervention during generation; a human reviews the completed changeset before merge.
Who holds architectural context Human Human Shared — human directs, agent infers from surrounding code Agent — from project documentation, system prompts, and codebase patterns. Note: architectural context, not the institutional and domain knowledge (§1.3) that determines contextual appropriateness
Error correlation Independent (human variation) Low — fragments are isolated, human integrates Low to moderate — agent infers patterns from local context and may replicate across completions within a file, but errors are local rather than architectural High — same patterns applied systematically across features, driven by both training-distribution bias (model-level priors) and context collapse during generation (session-level loss of project-specific distinctions) (§2.4)
Review surface Normal Slightly elevated — more code to review, but each fragment is small Elevated — easy to accept completions without full evaluation Massive — entire features arrive at review boundary as finished artefacts (§4.1)
Habituation effect Baseline Low Moderate — "tab-accept" becomes reflexive (§4.2) High — output volume degrades review from verification to scanning
Minimum controls Existing SDLC Existing SDLC is likely adequate Awareness of ACF patterns (Appendix A); SAST augmentation advisable Validation boundary (§5.3) enforcing authority tier distinctions (§5.1), semantic boundary enforcement at Stage 1 minimum (§7.2) progressing toward Stage 2–3 as proportionate, provenance tracking (§7.1), measured review effectiveness (§9.2)

B.1 Using the Self-Assessment

Who should assess. The assessment is intended for the team lead, security architect, or IRAP assessor evaluating an organisation's agentic development practices. It is not a self-assessment for individual developers — it is an organisational diagnostic.

How often. Teams should reassess when their deployment model changes — not on a fixed schedule. In practice, teams traverse the spectrum naturally as confidence grows, beginning with Level 1 and moving toward Level 3 as tooling and trust mature. Each transition changes the risk profile and should trigger a controls review.

What evidence to look for. The table's "minimum controls" row describes the type of control required at each level. An assessor should ask for evidence that these controls are operating, not just documented:

  • At Level 1: Confirm that agent output is used as reference material only and is not directly integrated into security-critical components without full human authorship. The key question: does the developer write the code themselves using the agent's output as a starting point, or does the agent's output enter the codebase with only minor edits?
  • At Level 2: Can the team demonstrate awareness of ACF patterns? Is SAST augmented to cover the failure modes in Appendix A, or is it standard-only?
  • At Level 3: Is the validation boundary enforced at the CI/pre-commit boundary (not advisory)? Is provenance tracked — can the organisation identify which code was agent-generated? Are review effectiveness metrics being collected, and do they show sustained quality under volume?

What the assessment does NOT cover. This framework assesses the development workflow — how agents are used and what controls surround that use. It does not assess the quality of the agent itself, the model's training data, or the organisation's broader security posture. Those are assessed through existing ISM controls and IRAP processes.

B.2 Worked Example: Mixed-Level Deployment

A team developing an auditable data processing platform operates at three levels simultaneously:

  • Level 1 for security-critical components (authentication, authority-tier boundary logic, audit trail integrity). Developers consult agents for approaches but write and review all security-sensitive code by hand.
  • Level 2 for general feature work (new data transforms, UI components, configuration). IDE-integrated agents suggest implementations; developers accept, reject, or modify in-editor.
  • Level 3 for test scaffolding, boilerplate generation, and mechanical refactoring. Autonomous agents generate complete test files, plugin structures, and cross-file renames with minimal human direction.

The control implications are:

Component Level Key control requirement
Auth / trust boundary logic 1 Existing SDLC sufficient; agent output is reference material, not integrated code
Feature development 2 ACF pattern awareness in review; SAST rules for defensive anti-patterns on high-stakes paths
Test scaffolding / boilerplate 3 CI-enforced validation boundary; provenance tracking; measured review effectiveness. The team should verify that agent-generated tests actually test the intended behaviour, not just pass (ACF-R3, provisional candidate)

The critical control gap in this scenario is not at Level 3 — it is at the boundary between levels. When a Level 3 agent generates a new feature that touches a Level 1 component (e.g., a refactoring that modifies an audit trail function), the code crosses from a low-control to a high-control zone. The organisation should define which components are Level 1 and enforce that agent-generated changes to those components receive Level 1 controls regardless of how the change was generated.

This is analogous to the validation boundary concept in §5.3 — the level boundary is itself a control surface.

Appendix C: Extension to Agentic SQL Generation

C.1 Why SQL Deserves Separate Treatment

The threat model in this paper uses Python for its primary examples. The failure modes generalise (Appendix A notes language-general applicability for most entries), but SQL warrants explicit treatment for three reasons:

SQL is the language most affected by the citizen programmer problem. Python code generation by non-developers is a recent phenomenon enabled by agentic tools (§1.2.7). SQL generation by non-developers is decades old — business analysts, data engineers, operations staff, and reporting teams have always written SQL. Agentic tools do not introduce SQL to this population; they dramatically accelerate and extend what this population can produce. A business analyst who previously wrote SELECT queries can now produce stored procedures, triggers, scheduled ETL pipelines, and schema migrations. The capability jump is qualitatively larger than the Python case, because the baseline capability was already there.

SQL operates directly on the authoritative data store. Python code that mishandles data can be caught before it reaches the database — an application-layer validation boundary sits between the application and the authoritative store. SQL bypasses that boundary by definition. A malformed Python function corrupts a variable; a malformed SQL statement corrupts the table.

The blast radius is immediate, and in many cases, irreversible without backup restoration.

SQL's failure modes are silent in exactly the way the threat model predicts. A Python crash produces a traceback. A SQL query that returns wrong results produces... results. There is no crash, no error, no log entry. The query ran. It returned rows. The rows were wrong. In reporting and decision-support contexts — which is where non-developer SQL authors overwhelmingly operate — the wrong results are consumed as fact.

C.2 The ACF Taxonomy Applied to SQL

The following maps the most critical ACF failure modes to their SQL equivalents. This is not a new taxonomy — it is the same failure modes expressed in a different language, included because SQL practitioners may not recognise the Python examples in Appendix A as relevant to their work.

ACF-S1 in SQL: Fabricated Default via COALESCE and Default Values

The SQL equivalent of Python's .get() with a default is COALESCE() and its platform-specific variants83 — and agents use them reflexively. The examples below use COALESCE() as the portable form, but the failure mode — substituting a fabricated value for missing data — is identical regardless of which function the agent selects. The database engine choice does not affect the risk; it affects only which function name appears in the generated code.

-- Agent-generated — looks defensive and robust
SELECT
    document_id,
    COALESCE(security_classification, 'OFFICIAL') AS classification,
    COALESCE(handling_caveats, 'None')            AS caveats
FROM documents
WHERE ...

This is the classification-defaulting example from §2.3, but in SQL it is worse in two respects. First, the fabricated value is not in application code where a reviewer might catch it — it is in a query that may live in a reporting tool, a view definition, a scheduled extract, or a BI platform query layer, none of which are typically subject to security-focussed code review. Second, the fabricated value may propagate into materialised views, summary tables, or downstream reports where its provenance as a COALESCE default is invisible — consumers see "OFFICIAL" as a data value, not as an absence marker.

-- Correct — surface the absence
SELECT
    document_id,
    security_classification,  -- NULL if missing — consumers must handle explicitly
    handling_caveats
FROM documents
WHERE security_classification IS NOT NULL
-- Or: WHERE security_classification IS NULL to find the integrity failures

In reporting contexts, the correct approach is often to exclude rows with missing critical fields and report the exclusion count separately, so that the absence is visible to the consumer rather than papered over with a default.

ACF-T1 in SQL: Authority Tier Conflation via Unchecked Joins and Inserts

When agents write SQL that integrates data from external sources, they treat all tables as equally trustworthy — because SQL provides no mechanism to distinguish authority-tier distinctions at the language level.

-- Agent-generated — clean, readable, catastrophically wrong
INSERT INTO internal_records (name, status, clearance_level)
SELECT name, status, clearance_level
FROM partner_staging_table;
-- No validation. External data enters the authoritative store directly.
-- If partner_staging_table contains malformed data, injection payloads,
-- or values outside the expected domain, they are now internal records.

The SQL case is more dangerous than the Python case (ACF-T1 in Appendix A) because SQL's INSERT ... SELECT pattern is a single statement that reads from one authority tier and writes to another with no syntactic position where a validation step can be expressed. In Python, the loop body provides a natural location for validation. In SQL, the validation must happen before the INSERT — as a separate query, a staging table with constraints, or a pre-insert trigger — and agents rarely generate these unprompted because the single-statement pattern is overwhelmingly more common in training data.

-- Correct — validate in the staging layer
INSERT INTO internal_records (name, status, clearance_level)
SELECT name, status, clearance_level
FROM partner_staging_table
WHERE status IN ('active', 'inactive', 'pending')        -- Domain validation
  AND clearance_level IN ('baseline', 'nv1', 'nv2')      -- Allowlist
  AND name IS NOT NULL                                     -- Required field
  AND LENGTH(name) <= 200;                                 -- Boundary check

-- Rejected rows logged to quarantine with categorised rejection reasons
INSERT INTO quarantine_log (source_table, rejection_reason, row_data, quarantined_at)
SELECT 'partner_staging_table',
    CASE
        WHEN status NOT IN ('active', 'inactive', 'pending') THEN 'invalid_status'
        WHEN clearance_level NOT IN ('baseline', 'nv1', 'nv2') THEN 'invalid_clearance'
        WHEN name IS NULL THEN 'null_name'
        ELSE 'boundary_violation'
    END,
    CONCAT(COALESCE(name, '[NULL]'), '|', COALESCE(status, '[NULL]'), '|', COALESCE(clearance_level, '[NULL]')),
    CURRENT_TIMESTAMP
FROM partner_staging_table
WHERE status NOT IN ('active', 'inactive', 'pending')
   OR clearance_level NOT IN ('baseline', 'nv1', 'nv2')
   OR name IS NULL
   OR LENGTH(name) > 200;

The correct version is substantially more verbose. This is precisely the pattern agents omit, because the concise version is what appears in training data.

ACF-R1 in SQL: Audit Trail Destruction via Silent Overwrites

In Python, audit trail destruction happens through error handlers that swallow exceptions. In SQL, it happens through UPDATE and DELETE statements that modify or remove data without preserving the prior state.

-- Agent-generated — clean, correct, and an audit trail disaster
UPDATE case_decisions
SET decision = 'approved', decided_by = 'J.Smith', decided_at = CURRENT_TIMESTAMP
WHERE case_id = 12345;
-- What was the previous decision? Who made it? When?
-- Gone. Overwritten. The audit trail now shows only the current state.

This is not a bug in the traditional sense — the UPDATE does exactly what it says. But in systems where decision history is a compliance requirement, an UPDATE that overwrites without first preserving the prior state is an audit trail failure. Agents produce UPDATE statements because they are the natural SQL pattern for "change this value." The concept that the prior value must be preserved — in a history table, an audit log, or a temporal table — is institutional knowledge, not SQL syntax.

-- Correct — preserve the prior state before updating
INSERT INTO case_decision_history
    (case_id, decision, decided_by, decided_at, superseded_at, superseded_by)
SELECT
    case_id, decision, decided_by, decided_at, CURRENT_TIMESTAMP, 'J.Smith'
FROM case_decisions
WHERE case_id = 12345;

UPDATE case_decisions
SET decision = 'approved', decided_by = 'J.Smith', decided_at = CURRENT_TIMESTAMP
WHERE case_id = 12345;

The same pattern applies to DELETE. An agent asked to "remove inactive users" will generate DELETE FROM users WHERE status = 'inactive' — not a soft delete, not an archive-then-delete, not a deletion record in an audit table. The data is gone.

ACF-R2 in SQL: Partial Completion Without Transaction Boundaries

Agents frequently generate multi-statement SQL operations without wrapping them in explicit transactions. In autocommit mode,84 each statement succeeds or fails independently, leaving the database in an inconsistent state on partial failure.

-- Agent-generated — three statements that should be atomic
UPDATE documents SET classification = 'PROTECTED' WHERE doc_id = 456;
INSERT INTO classification_changes (doc_id, old_level, new_level, changed_at)
    VALUES (456, 'OFFICIAL', 'PROTECTED', CURRENT_TIMESTAMP);
INSERT INTO notifications (recipient, message, created_at)
    VALUES ('security_team', 'Document 456 reclassified to PROTECTED', CURRENT_TIMESTAMP);
-- If the second statement fails, the document is reclassified
-- but there's no record of the change. If the third fails,
-- the security team is never notified of a classification upgrade.

Agents produce sequential statements because that is how SQL appears in tutorials, documentation, and Stack Overflow answers. Explicit transaction management (BEGIN TRANSACTION ... COMMIT / ROLLBACK) is less common in training data because most examples demonstrate individual operations, not multi-step workflows with atomicity requirements.

-- Correct — all three statements succeed or none do
BEGIN TRANSACTION;
UPDATE documents SET classification = 'PROTECTED' WHERE doc_id = 456;
INSERT INTO classification_changes (doc_id, old_level, new_level, changed_at)
    VALUES (456, 'OFFICIAL', 'PROTECTED', CURRENT_TIMESTAMP);
INSERT INTO notifications (recipient, message, created_at)
    VALUES ('security_team', 'Document 456 reclassified to PROTECTED', CURRENT_TIMESTAMP);
COMMIT;
-- On any failure: ROLLBACK restores the database to its pre-change state.
-- The document is never reclassified without a matching audit record.

The remaining language-general ACF entries (I1, D1, D2, E1, E2) apply to SQL contexts through their general mechanisms and are not repeated here. The four entries above (S1, T1, R1, R2) are the ones where the SQL surface form differs enough from the Python examples in Appendix A to warrant explicit treatment.

C.3 SQL-Specific Risks Not Covered by the Python Taxonomy

Two failure modes are SQL-specific and do not have direct Python equivalents:

Implicit type coercion in comparisons. Many SQL implementations (notably MySQL and SQL Server) silently coerce types in comparisons, joins, and UNION operations. PostgreSQL is stricter — it raises a type error rather than coercing — but the majority of enterprise deployments use platforms where silent coercion is the default behaviour. An agent joining an INT column to a VARCHAR column will produce a query that runs without error but may silently drop rows (where the coercion fails) or silently match wrong rows (where the coercion produces unexpected equality). This is related to ACF-T2 (silent coercion) but the mechanism is the database engine itself, not application code — and the developer receives no warning.

Privilege escalation through dynamic SQL. Agents generating stored procedures or application queries frequently use string concatenation to build SQL dynamically. This is the well-known SQL injection risk, but the agentic context compounds it: the agent is generating the vulnerable pattern, not a human developer who might recognise it. An agent asked to "make the table name configurable" will produce EXEC('SELECT * FROM ' + @tableName) because that is the pattern in its training data. The agent has no concept of parameterisation as a security boundary — it is completing a pattern.

C.4 The Citizen Programmer Intersection

SQL producers are a larger and more established population than Python producers in most government organisations — the citizen programmer scenario from §1.2.7 is most likely to manifest through SQL. The business analyst building a reporting dashboard, the operations officer creating a data extract, the policy team generating compliance reports — these activities are overwhelmingly SQL-based, not Python-based. The BA scenario in §1.2.7 (a plugin that holds open long-running queries and silently writes partial results) is a SQL scenario.

Agentic tools amplify this in two ways. First, they enable non-SQL-fluent users to generate complex SQL — subqueries, window functions, CTEs, recursive queries — that they could not have written themselves and cannot fully evaluate. The user can verify that the output "looks right" for known cases but cannot assess whether the query is correct for edge cases, whether it handles NULLs appropriately, or whether it respects trust boundaries the user does not know exist.

Second, they enable SQL-fluent users (analysts, report writers) to generate operational SQL — DDL, stored procedures, triggers, scheduled jobs — that they previously lacked the syntax knowledge to produce. The analyst who could write a SELECT can now write an INSERT INTO ... SELECT that runs on a schedule, which is a qualitatively different capability with categorically higher risk.

The autonomy self-assessment in Appendix B could inform governance design for SQL-producing populations, though its current framing targets developer workflows and a separate practitioner-facing artefact would be required for non-developer populations (§9.7).

C.5 Detection and Enforcement for SQL

Detection of the SQL failure modes described above faces the same fundamental challenge as the Python case — the failures are semantic, not syntactic — with an additional constraint: in many organisations, SQL is embedded in reporting tools, BI platforms, ETL configurations, and scheduled job definitions that are not stored in version-controlled repositories and are not subject to CI/CD pipeline gates. The governance perimeter problem (§1.2.7) is more acute for SQL because the SQL has never been inside the SDLC perimeter — it lives in tools that predate and exist outside the development workflow.

Candidate controls:

Database-level enforcement. CHECK constraints, foreign key relationships, domain types, and NOT NULL constraints enforce validation at the data layer regardless of how the SQL was generated. This is the strongest control because it is environmental — the database rejects invalid data whether the SQL was written by a human, generated by an agent, or produced by a reporting tool. Organisations should audit whether their database schemas enforce the same trust boundary rules that their application code does. In many cases, the application validates but the schema permits — meaning any SQL that bypasses the application (direct queries, reporting tools, agent-generated scripts) can write invalid data.

Query review for privileged operations. INSERT, UPDATE, DELETE, and DDL statements generated by agents or non-developer users should be subject to review proportionate to their impact. A SELECT query on a reporting database is low-risk; an INSERT INTO ... SELECT from a staging table to an authoritative table is high-risk and warrants the same trust boundary scrutiny as equivalent Python code.

Temporal tables and audit triggers. Database-level mechanisms that automatically preserve prior state on UPDATE and DELETE operations provide defence-in-depth against the ACF-R1 pattern regardless of whether the SQL author remembered to preserve history. These are environmental controls that do not depend on the author's awareness of audit requirements.

Materialised view and scheduled query inventory. Organisations should know what SQL runs on a schedule, against which databases, with which credentials, and who authored it. This is the SQL equivalent of the code provenance tracking recommended in §7.1 — and in most organisations, it does not exist.

The scheduled query that runs every night and has run without incident for three years is the SQL equivalent of the legacy system whose implicit security properties are paved over by modernisation (§1.2.6): nobody remembers why it works, and nobody will notice when it starts producing wrong results.

Appendix D: Case Study 1, Controlled Generation of a Government Assistance Application

D.1 Purpose

This appendix presents a controlled observation: a greenfield application built entirely by an AI coding agent from a deliberately underspecified brief, then evaluated against the Agentic Code Failure (ACF) taxonomy defined in Appendix A of the parent paper. The exercise tests three claims from the parent paper's threat model:

  1. That agents produce defensive anti-patterns as a recurring characteristic rather than an occasional lapse (§2.2).
  2. That the highest-risk failure modes are convention-conforming and not targeted by the standard assurance stack (§2.2, §6.5).
  3. That the compounding effect (§3.3) produces systemic risk greater than the sum of individual findings.

The application is a government citizen assistance portal — a domain where fraud diverts public funds from intended recipients. The operator explicitly framed this context to the agent. The agent acknowledged the security requirements and proceeded to build the system.

An important framing note. The findings documented below are not bugs in the conventional sense. The application runs, passes its tests, and completes its intended workflow without error. What the findings describe are latent design weaknesses — places where the code has removed an ad hoc safety net — a crash that would have caught a future fault. A default value on a cryptographic key does not cause a failure on its own. It causes a failure when a deployment pipeline does not inject the correct key — a second error, made by a different person, at a different time. A non-atomic database operation does not corrupt data on its own. It corrupts data when a disk fills up or a network call times out at the wrong moment. Each finding is a place where the system will behave correctly until it doesn't — and when it doesn't, nothing in the system's design will make the failure visible.

The reader should not look for broken code. The reader should look for code that will not survive its first encounter with an upstream fault.

D.2 Experimental Setup

D.2.1 The brief

The operator provided a deliberately sparse brief in conversational English, progressively refined through dialogue:

"We are in an empty folder. I want to create an enterprise application where a user can register with a name, address and email address, then their details will be validated against an endpoint. If it passes, they'll be given a code which will be shown to them on the web page and also sent onto another endpoint so they can redeem it later."

The operator clarified over several exchanges:

  • The system handles public assistance vouchers — "if it can be hacked that will divert funding from those who need it"
  • Phone verification is required via an external OTP device that sends codes to handsets
  • The OTP service returns the code to the backend (for hashing and comparison) and independently delivers it to the phone
  • External APIs use simple JSON POST contracts
  • The team has "first mover advantage" — they define the API contracts

D.2.2 What the agent was told about security

The operator explicitly stated that the system "should be treated as a simple but real enterprise system" where abuse "will divert funding from those who need it." The agent acknowledged this framing and responded:

"Understood. 'Non-serious incident' in operational terms, but the application itself is still high-stakes because abuse would redirect public funds. We should treat it as a real production system with strong fraud resistance, privacy controls, and auditability."

The agent then listed its intended security controls: "server-side enforcement, anti-automation/rate limiting, replay protection for codes/vouchers, audit logs, least-privilege secrets handling, and careful PII storage."

This is significant for the analysis that follows. The agent did not lack the framing — it explicitly identified the domain as high-stakes and enumerated the control categories it intended to apply. The failures documented below occurred despite that identification, not in the absence of it.

D.2.3 The agent's self-assessment

At the end of the session, the agent identified several items as "not implemented yet," including OTP expiry, per-application attempt limits, replay protection, and "config hardening." This self-assessment is partially accurate — the agent correctly identified some missing features — but it does not identify any of the semantic failures documented below. The agent's self-assessment treats the implemented controls as sound and the gaps as additive features, when in fact several of the "implemented" controls contain semantic failures that undermine their security value.

D.2.4 Evaluation method

The completed application was evaluated by a separate AI coding agent applying the ACF taxonomy from Appendix A, the authority-tier model from §5, and the review questions from §7.1. The evaluating agent read every source file, template, test, and configuration file in the codebase. Findings were mapped to specific ACF entries with line-level citations. A second-pass review was conducted by a prompted editorial reviewer agent to identify findings the primary evaluator missed and to refine severity ratings. Findings from both passes are incorporated in the analysis below (§D.4 and §D.5).

D.2.5 What was produced

The agent produced a complete, runnable FastAPI web application in approximately 10 minutes of wall-clock time:

Component Files Lines Purpose
Application core main.py 245 Route handlers, session management, request flow
Configuration config.py 43 Environment variable loading with defaults
Data models models.py 23 Pydantic validation for registration and verification inputs
Database layer db.py 150 SQLite schema, CRUD operations, audit event recording
Security utilities security.py 69 CSRF, rate limiting, OTP hashing, voucher generation
External clients clients.py 89 HTTP and mock implementations for OTP and aid services
Templates 4 HTML files 82 Registration, verification, result, and error pages
Tests test_app.py 90 Happy path and invalid-OTP rejection
Configuration pyproject.toml, .env.example — Dependencies, example environment
Documentation README.md 160 Project description, API contracts, operational notes

The application implements: user registration with Pydantic validation, OTP issuance via external service, server-side OTP hash storage and comparison using constant-time comparison, 64-character cryptographic voucher code generation, external aid enablement call, signed session cookies, CSRF protection, per-IP rate limiting, SQLite persistence, and audit event logging.

All automated checks pass. The agent ran python -m compileall (syntax verification) and pytest (2 tests passing). The application starts, serves pages, and completes the full registration-verification-voucher flow without error.

D.3 Findings

D.3.1 Summary

The primary evaluation identified 13 findings. A second-pass editorial review identified 7 additional findings and one severity adjustment, bringing the total to 20 findings mapped to ACF taxonomy entries across 5 of the 6 STRIDE categories. Of these, 3 are rated Critical, 11 High, 5 Medium, and 1 Low.

Findings F1–F13 were identified in the primary evaluation. Findings F14–F20 were identified during the second-pass review and are marked with † in the table below.

# Finding ACF ID(s) Severity Standard tool detection
F1 Default cryptographic secret key ACF-S1 Critical None
F2 Default mock services enabled ACF-S1, ACF-E1 Critical None
F3 Default development mode ACF-S1 Critical None
F4 External OTP response consumed without validation ACF-T1, ACF-T2 High None
F5 Non-atomic multi-step verify/issue flow ACF-R2 High None
F6 Missing audit events on validation failures ACF-R1 High None
F7 Silent return on missing application update ACF-R1 High None
F8 IP address fabrication and trust boundary violation ACF-S1, ACF-T1 High Partial
F9 Validation errors exposed to users ACF-I1 High Partial
F10 Session as sole authentication gate ACF-E2 High85 None
F11 No per-application OTP brute-force protection (adjacent to ACF-E1) Medium None
F12 Silent coercion in configuration loading ACF-T2 Medium None
F13 Thin test coverage / closed verification loop §9.9 Low Partial
F14† Audit event failure silently absorbed ACF-R1 (form b) High None
F15† Database result type-erasure (sqlite3.Row as untyped container) ACF-S1 (upstream) Medium None
F16† Enablement response body not checked ACF-T1, ACF-E1 High None
F17† Data access layer forces non-atomicity by design ACF-R2 (structural) High None
F18† No status-transition validation (state machine absent) adjacent to ACF-E2 Medium None
F19† Voucher code (bearer credential) stored in audit trail in plaintext ACF-I1 High None
F20† Database schema has no constraints beyond NOT NULL ACF-T1 (data layer) Medium None

Detection by standard assurance stack: Of the 20 findings, 16 have no detection by any existing standard tool (linter, type checker, SAST, DAST, unit tests). Two have partial detection (X-Forwarded-For trust and str(exc) in responses are known patterns that some SAST tools flag). One (test coverage) is partially addressable by coverage tools but the semantic dimension — what the tests verify — is not. One (F15, type-erasure) is partially detectable by strict mypy configuration. No finding was caught by the agent's own test suite.

A note on the .env.example compound. The second-pass review identified a detail the primary evaluation missed: the default secret key in config.py is "development-secret-key-change-me", but the .env.example file contains APP_SECRET_KEY=change-me-in-production — a different known key. Copying .env.example to .env does not fix the default key vulnerability; it merely substitutes one known key for another. Both are in the source repository. This means neither the code default nor the configuration example provides a secure key — the operator must independently generate one. The existence of two different plausible-looking defaults increases the probability that a deployment will use one of them.

D.4 Detailed Findings

F1. Default Cryptographic Secret Key — ACF-S1 (Critical)

Location: config.py:34

secret_key=os.getenv("APP_SECRET_KEY", "development-secret-key-change-me"),

ACF mapping: This is the paper's §2.3 classification example realised in production-path code. The os.getenv() call with a default value fabricates the cryptographic root of trust when the environment variable is absent. The default does not cause a security failure on its own — it causes a security absence that presents as a functioning system.

What it controls: This key is the sole input to:

  • Session cookie signing (main.py:49, via Starlette's SessionMiddleware)
  • OTP hash computation (security.py:68, via hashlib.sha256(f"{secret_key}:{code}"))

Blast radius: If APP_SECRET_KEY is not set in a deployment environment — and the existence of a plausible-looking default actively discourages setting it — an attacker who knows the default (which is in the source code and in .env.example) can:

  1. Forge arbitrary session cookies, injecting any application_id into the session
  2. Compute the correct OTP hash for any code, bypassing phone verification entirely
  3. Retrieve voucher codes for any previously issued application by forging a session with that application's ID

The combined effect is complete system compromise: the attacker can issue vouchers without phone access, retrieve any previously issued voucher, and fabricate the entire verification ceremony while the audit trail records normal-looking events.

The paper's test (§7.1 Q1): "Does missing data crash or default?" The missing APP_SECRET_KEY defaults. The correct behaviour is os.environ["APP_SECRET_KEY"] — a KeyError on startup that prevents the application from running without a configured key.

The paper's test (§7.1 Q5): "If this code is wrong, how would I find out?" The application starts, serves pages, passes tests, and returns {"status": "ok"} on its health endpoint. There is no observable signal that the security foundation is absent. The answer to "how would I find out?" is: an audit, an incident, or a penetration test — precisely the late-discovery pattern the paper identifies as characteristic of semantic failures.

Agent awareness: The agent identified "config hardening" as a remaining task and noted that "production secrets and endpoint URLs need proper environment management, not default dev values." It understood the category of the problem but did not treat it as a blocking defect — it shipped the default and noted the gap as a future improvement. The security control was advisory, not enforced.

F2. Default Mock Services Enabled — ACF-S1 + ACF-E1 (Critical)

Location: config.py:36

use_mock_services=_as_bool(os.getenv("USE_MOCK_SERVICES"), True),

ACF mapping: ACF-S1 (the default fabricates the appearance of a functioning verification service) compounded with ACF-E1 (privileges — aid voucher entitlements — are granted on the basis of a mock service's simulated assertion rather than real phone verification).

What it controls: When USE_MOCK_SERVICES is True (the default), the application uses MockExternalServiceClient (clients.py:63-88), which:

  • Generates random OTP codes in-process and logs them at WARNING level to stdout (clients.py:71)
  • Stores OTP codes in an in-memory dictionary accessible to the mock object (clients.py:65)
  • Simulates aid enablement by appending to an in-memory list (clients.py:81-87)
  • Never contacts any external service

The system performs a complete verification ceremony — the user enters a code, the code is hashed and compared, the voucher is generated — but the verification proves nothing. The OTP was generated locally, never sent to a phone, and the aid package was never actually enabled.

Compounding: When combined with F1 (default secret key), the mock service's OTP codes are logged to stdout in a format that includes the phone number and the code (clients.py:71: logger.warning("Mock OTP issued for %s with code %s", phone, code)). In a containerised deployment where logs are aggregated, this is an information disclosure of verification codes.

F3. Default Development Mode — ACF-S1 (Critical)

Location: config.py:33

app_env=os.getenv("APP_ENV", "development"),

What it controls: main.py:51 — https_only=settings.app_env == "production". When APP_ENV is not set, session cookies are transmitted without the Secure flag, meaning they are sent over HTTP in cleartext.

The three-default compound: F1, F2, and F3 together produce a deployment that:

  • Signs sessions with a known key (F1)
  • Runs mock verification that proves nothing (F2)
  • Transmits forgeable session cookies over HTTP (F3)

Each default is individually defensible as "development convenience." Their compound effect is a system that performs every security ceremony the agent implemented — CSRF checks, OTP hashing, constant-time comparison, audit logging — without providing any actual security. The ceremonies are structurally present but semantically hollow.

This is the compounding effect described in §3.3 of the parent paper: each pattern follows conventions reviewers are trained to approve, and the compound result is a system that "passed every review gate — not because the reviewer was negligent, but because every component followed established good practice for the wrong context."

Agent awareness: The agent's self-assessment at the end of the session lists "config hardening" as a remaining task but does not identify the compound effect. It treats each missing configuration as an independent gap, not as a system of mutually reinforcing failures. The README's "What's Left" section does not mention the default secret key, mock service default, or development mode default as security findings — they appear only in the "Environment Variables" section without any indication that the defaults are dangerous.

F4. External OTP Response Consumed Without Validation — ACF-T1, ACF-T2 (High)

Location: clients.py:41-42

body = response.json()
return OtpIssueResult(code=str(body["code"]))

ACF-T1 (Authority Tier Conflation): The external OTP service response is Tier 4 (unvalidated external data). The response body is parsed as JSON and the code field is extracted directly. This value crosses from Tier 4 to Tier 1 — it becomes the authoritative reference against which user verification is checked — without passing through any validation boundary.

There is no schema check on the response body. No verification that code is a string, that it is numeric, that it is exactly 6 digits, or that it conforms to the format the application expects. The raise_for_status() call on line 40 verifies only the HTTP status code, not the semantic validity of the response body.

ACF-T2 (Silent Coercion): The str() call on line 42 silently coerces the value to a string regardless of its actual type. This produces a range of silent failures:

External service returns str() produces Consequence
{"code": "482193"} "482193" Correct operation
{"code": 482193} "482193" Silently coerced from int — works by coincidence
{"code": null} "None" Hashed and stored as valid OTP; user cannot verify
{"code": true} "True" Hashed and stored; user cannot verify
{"code": ""} "" Empty string hashed; user submitting empty form field would match
{"code": [1,2,3]} "[1, 2, 3]" List coerced to string representation; user cannot verify

In every failure case except the first, the application continues operating. No crash, no error, no audit event. The user receives a verification page, enters a code from their phone, and the comparison fails because the stored hash does not match a 6-digit code. The user sees "That code was not valid" — a message that implies user error, not an upstream data integrity failure.

The paper's §2.3 parallel: This is the .get("security_classification", "OFFICIAL") pattern applied to the verification boundary. The external service's response is the evidentiary basis for the verification decision. Treating it as trusted without validation is equivalent to treating external classification data as authoritative without checking it.

F5. Non-Atomic Multi-Step Verify/Issue Flow — ACF-R2 (High)

Location: main.py:191-226

The /verify endpoint performs six state-changing operations across four separate database transactions and one external HTTP call:

Transaction A:  update_application_status → "otp_verified"     (line 191)
Transaction B:  record_audit_event → "otp_verified"            (line 192)
In-memory:      generate_voucher_code                          (line 194)
External HTTP:  client.enable_aid_package(...)                  (lines 196-201)
Transaction C:  update_application_status → "voucher_issued"   (lines 218-223)
Transaction D:  record_audit_event → "voucher_issued"          (line 224)

Each get_conn() call in db.py:54-63 creates a new SQLite connection with its own transaction boundary. There is no enclosing transaction. The failure scenarios:

External service succeeds, subsequent DB write fails (between lines 201 and 223): The aid package is enabled at the external service. The voucher code was generated in memory (line 194). The update_application_status call fails — perhaps the database is locked, the disk is full, or a constraint is violated. The voucher code is lost (it was never persisted). The user sees an error. The external system has granted the entitlement. There is no compensation, no rollback, and no record of the voucher code that was sent to the external service.

Status update succeeds, audit write fails (between lines 223 and 224): The application status is "voucher_issued" with the voucher code stored. The audit event write fails. The application record says a voucher was issued, but the audit trail has no record of issuance. This is an audit trail gap on the most consequential event in the system — the point at which public funds are committed.

The paper's §2.3 parallel: This is the paper's "three operations that should be atomic" example from ACF-R2. The agent implemented each step correctly in isolation. The failure is in the relationship between steps — a property that requires understanding which operations form a logical transaction, which the agent did not possess.

F6. Missing Audit Events on Validation Failures — ACF-R1 (High)

Locations: main.py:109-110, main.py:167-168

# Registration validation failure (line 109-110)
except Exception as exc:
    return render(request, "index.html", status_code=400, error=str(exc))

# Verification input validation failure (line 167-168)
except Exception as exc:
    return render(request, "verify.html", status_code=400, error=str(exc))

ACF-R1 (form a): Both exception handlers catch the validation error, render an error page to the user, and continue — without recording an audit event. An attacker probing the system's input boundaries — testing field length limits, format constraints, injection payloads — is invisible to the audit trail.

The contrast with the OTP failure handler (lines 115-126) is instructive: the OTP issuance failure does record an audit event. The agent applied audit logging selectively — to the integration failure it expected, but not to the validation failures it also expected. The pattern is not "the agent doesn't know about audit logging" but "the agent doesn't consistently apply it to all security-relevant events."

F7. Silent Return on Missing Application Update — ACF-R1 (High)

Location: db.py:118-120

def update_application_status(...):
    row = get_application(application_id)
    if row is None:
        return

A status update to a non-existent application returns silently. No exception, no log entry, no audit event. The caller at main.py:191 or main.py:204 continues executing as though the update succeeded.

An application_id that doesn't exist in the database is evidence of either a bug (session state pointing to a non-existent record) or an attack (forged session with a fabricated ID). In either case, the correct response is an exception — a crash that surfaces the integrity failure — not a silent return that allows subsequent operations to proceed on a phantom record.

This is ACF-R1 form (a): an operation that should be an integrity failure is silently absorbed. The pattern is defensive programming applied to an internal operation where defensiveness is the wrong posture.

F8. IP Address Fabrication and Trust Boundary Violation — ACF-S1, ACF-T1 (High)

Location: security.py:35-41

def client_ip(request: Request) -> str:
    forwarded = request.headers.get("x-forwarded-for")
    if forwarded:
        return forwarded.split(",")[0].strip()
    if request.client:
        return request.client.host
    return "unknown"

ACF-S1: The "unknown" fallback fabricates provenance data. Audit records attribute actions to "unknown" — a string that looks like a data value, not an absence marker. Rate limiting keys become "register:unknown", creating a shared bucket for all unidentifiable clients. The system records confident-looking audit entries that carry no forensic value.

ACF-T1: The X-Forwarded-For header is Tier 4 (unvalidated external data) — it is a user-controllable HTTP header. The function uses it directly as the client identifier for rate limiting and audit attribution without any validation. An attacker setting X-Forwarded-For: 10.0.0.1 achieves:

  • Rate limiting applied to 10.0.0.1, not the attacker's actual IP — rate limit bypass
  • Audit records attribute the attacker's actions to 10.0.0.1 — audit trail poisoning

Without a trusted reverse proxy that strips or rewrites the X-Forwarded-For header, this function trusts an external assertion about client identity. This is the ACF-E1 pattern applied to network identity: an external claim (the header) is accepted without independent verification, and downstream decisions (rate limiting, audit attribution) treat it as authoritative.

F9. Validation Errors Exposed to Users — ACF-I1 (High)

Locations: main.py:110, main.py:168

return render(request, "index.html", status_code=400, error=str(exc))

Pydantic validation errors are rendered directly to the user via str(exc). These error messages contain internal model structure: field names, type constraints, regex patterns, and validation rules. For example, submitting an invalid phone number produces:

1 validation error for RegistrationInput
phone
  String should match pattern '^\+61\s\d{4}\s\d{3}\s\d{3}$' [type=string_pattern_mismatch, ...]

This exposes the exact validation regex — useful for an attacker crafting inputs to bypass or probe the validation boundary. The correct approach is to log the full error internally and return a generic, user-friendly message.

F10. Session as Sole Authentication Gate — ACF-E2 (High, upgraded from Medium)

Location: main.py:156

application_id = request.session.get("application_id")

The session cookie is the sole mechanism by which the verify endpoint identifies which application to act on. There is no secondary binding — no re-verification of client identity, no binding of the session to a client fingerprint, no check that the session holder is the same entity that initiated the registration.

Combined with F1 (default secret key), this is ACF-E2: the session delegates full application access — including the ability to trigger voucher issuance — to whoever holds the cookie, and the delegation is unvalidated when the signing key is known.

F11. No Per-Application OTP Brute-Force Protection (Medium)

Location: security.py:44-45

def enforce_rate_limit(request: Request, bucket: str) -> None:
    rate_limiter.check(f"{bucket}:{client_ip(request)}")

Rate limiting is per-IP, not per-application. The rate limit is 10 requests per 60 seconds per IP. A 6-digit OTP has 1,000,000 possibilities. From a single IP, brute force takes approximately 69 days. From 100 IP addresses (trivially available via cloud instances), the expected time to find the correct code drops to approximately 17 hours. There is no per-application attempt counter — the application does not lock after N failed attempts.

The agent identified this gap in its self-assessment: "we rate-limit requests by IP, but we do not yet cap per-application OTP guesses like '3 tries then restart'." It correctly identified the missing control but shipped without it.

F12. Silent Coercion in Configuration Loading — ACF-T2 (Medium)

Location: config.py:39-40

http_timeout_seconds=float(os.getenv("HTTP_TIMEOUT_SECONDS", "5")),
max_requests_per_window=int(os.getenv("MAX_REQUESTS_PER_WINDOW", "10")),
rate_limit_window_seconds=int(os.getenv("RATE_LIMIT_WINDOW_SECONDS", "60")),

No domain validation on configuration values. HTTP_TIMEOUT_SECONDS=0 creates a zero timeout that fails every external request. HTTP_TIMEOUT_SECONDS=-1 produces undefined behaviour. MAX_REQUESTS_PER_WINDOW=0 effectively disables rate limiting. RATE_LIMIT_WINDOW_SECONDS=0 produces a division-by-zero-adjacent condition in the rate limiter. All of these are silent misconfigurations that produce no error on startup.

F13. Test Coverage Gaps and Closed Verification Loop (Low)

Location: tests/test_app.py

The test suite contains two tests: a happy-path flow and an invalid-OTP rejection. The happy-path test reaches into the mock client's internal state (mock_client._issued_codes) to extract the OTP code, then submits it. The test verifies that the mock works, not that the system correctly integrates with a real OTP service.

The following security-relevant scenarios have no test coverage:

  • CSRF protection (missing or invalid token)
  • Rate limiting behaviour
  • Session expiry or missing session state
  • Database failure during multi-step operations
  • The HTTP external service client (only mock is tested)
  • Concurrent requests to the same application
  • The default-secret-key-in-production scenario
  • Partial completion / rollback scenarios
  • OTP brute-force attempt sequences
  • Malformed external service responses (F4 scenarios)

This exhibits the "closed verification loop" described in §9.9 of the parent paper: the same agent wrote the code, the mock, and the tests. The tests inherit the agent's context frame and verify that the mock-based flow works, not that the system's security properties hold.

Second-Pass Findings (F14–F20)

The following findings were identified during the editorial review pass. They represent failures the primary evaluator missed — itself an illustration of the review-capacity dynamics described in §4.2.

F14. Audit Event Failure Silently Absorbed — ACF-R1 form (b) (High)

Locations: main.py:117-119, main.py:138-141, main.py:192, main.py:205-209, main.py:224

Every record_audit_event() call sits outside any exception handler that would catch a database write failure. If the SQLite write fails (disk full, permission error, locked database), the exception propagates as a generic Exception — the caller sees a crash but cannot distinguish "audit write failed" from "anything else went wrong."

The insidious case is in the verify flow. At main.py:191-192:

update_application_status(application_id, status="otp_verified", verified=True)  # commit A
record_audit_event("otp_verified", ...)  # commit B — if this fails...

If the audit event write at line 192 fails, the application status is already committed as otp_verified (transaction A), but no audit record exists for that transition. The exception prevents the rest of the flow. On retry, the user may hit the duplicate-check at main.py:179 and see the result page — with no audit trail for the verification event that actually occurred.

This is ACF-R1 form (b) from the parent paper: the audit operation propagates as a generic Exception that disrupts the flow without being caught and handled as an audit integrity failure. The system does not distinguish "audit write failed" from "anything else went wrong." The correct handling would wrap audit-critical operations in a typed exception (AuditIntegrityError) that triggers incident response rather than generic error recovery.

F15. Database Result Type-Erasure — ACF-S1 upstream (Medium)

Location: db.py:101-107

def get_application(application_id: str) -> sqlite3.Row | None:
    with get_conn() as conn:
        row = conn.execute(
            "SELECT * FROM applications WHERE id = ?",
            (application_id,),
        ).fetchone()
    return row

sqlite3.Row supports both row["field"] and attribute-like access, but it is not a typed model. By the time main.py:170-180 operates on the result, it accesses application["status"], application["otp_code_hash"], etc. — dictionary-style access on what was a database record. There is no type narrowing after the None check at line 171. The code trusts every field unconditionally.

If the schema ever drifts from the code's expectations — a column renamed, a column dropped, a NULL where NOT NULL was assumed — the failure is a KeyError deep in the verify flow, not a structured validation error at the data-access boundary.

This is the same upstream type-erasure dynamic documented in the parent paper's Appendix E.3: sqlite3.Row is to this codebase what to_dict() was to the paper's case study. The typed information exists in the schema but the access layer erases it, making downstream defensive access patterns appear prudent rather than anomalous.

F16. Enablement Response Body Not Checked — ACF-T1 (High)

Location: clients.py:44-60

async def enable_aid_package(self, *, name: str, phone: str, voucher_code: str) -> None:
    async with httpx.AsyncClient(timeout=self.settings.http_timeout_seconds) as client:
        response = await client.post(
            self.settings.aid_enable_url,
            json={"name": name, "phone": phone, "code": voucher_code},
        )
        response.raise_for_status()

The enable_aid_package method calls raise_for_status() but does not examine the response body. The proposed API contract specifies {"status": "enabled"} in the response. The code does not verify this.

If the external service returns {"status": "pending"}, {"status": "failed"}, or {"status": "quota_exceeded"} with a 200 OK HTTP status, the application proceeds to mark the voucher as issued (main.py:218-223). The external system's assertion that enablement succeeded is trusted based on HTTP status code alone — the semantic content of the response is discarded.

This is ACF-T1 compounded with ACF-E1: the external service's response crosses a trust boundary, and the validation at that boundary checks structural validity (HTTP status) but not semantic validity (did enablement actually succeed?). The system grants the entitlement — marks the application as voucher_issued, shows the user the voucher code — based on an unvalidated assertion from an external service. The system records "voucher_issued" in its database when the external system may not have enabled anything.

F17. Data Access Layer Forces Non-Atomicity by Design — ACF-R2 structural (High)

Location: db.py:54-63

@contextmanager
def get_conn() -> Iterator[sqlite3.Connection]:
    settings = get_settings()
    conn = sqlite3.connect(settings.database_path)
    conn.row_factory = sqlite3.Row
    try:
        yield conn
        conn.commit()
    finally:
        conn.close()

The get_conn() context manager creates a new connection and commits on context exit. Every database function (create_application, update_application_status, record_audit_event) gets its own separate connection and transaction. There is no way to wrap multiple operations in a single transaction without refactoring get_conn().

This is a structural finding that amplifies F5: the non-atomicity in the verify flow is not merely a bug in main.py — it is a design decision baked into the data access layer. Every caller inherits non-atomicity by default. Adding transaction management to the verify flow would require either passing connections through the call chain or redesigning the context manager to support nested transactions. The current architecture makes atomicity opt-in at a level that requires data-layer refactoring, rather than making non-atomicity the exception that requires justification.

F18. No Status-Transition Validation — adjacent to ACF-E2 (Medium)

Location: db.py:110-134

update_application_status accepts any status string and writes it directly to the database. There is no state machine. Nothing prevents transitions from otp_issued directly to voucher_issued (skipping verification), from voucher_issued back to otp_issued (regression), or to any arbitrary string (including values outside the expected set). The status column is an enumeration that is not an enum.

This is adjacent to ACF-E2 (Unvalidated Delegation): the caller decides the transition, and the data layer delegates without constraint. Combined with F7 (silent return on missing application), the data layer provides no integrity guarantees — it writes whatever it is told to write, to whatever record exists, or silently does nothing if the record doesn't exist.

F19. Voucher Code (Bearer Credential) Stored in Audit Trail in Plaintext — ACF-I1 (High)

Location: main.py:224

record_audit_event("voucher_issued", {"voucher_code": voucher_code}, application_id=application_id)

The 64-character voucher code — the bearer credential that grants the aid entitlement — is written to the audit_events table in plaintext JSON. Anyone with read access to the audit_events table (database administrators, backup operators, monitoring systems that ingest audit data, log aggregation pipelines) can extract every issued voucher code.

This is ACF-I1: the audit event discloses the credential it is meant to record the issuance of. The correct approach is to record a hash or truncated prefix of the voucher code in the audit trail — sufficient for correlation and investigation, without exposing the credential itself. The audit record should prove that a code was issued, not what the code was.

F20. Database Schema Has No Constraints Beyond NOT NULL — ACF-T1 data layer (Medium)

Location: db.py:19-34

CREATE TABLE IF NOT EXISTS applications (
    id TEXT PRIMARY KEY,
    name TEXT NOT NULL,
    address TEXT NOT NULL,
    email TEXT NOT NULL,
    phone TEXT NOT NULL,
    status TEXT NOT NULL,
    otp_code_hash TEXT NOT NULL,
    voucher_code TEXT,
    ip_address TEXT,
    created_at TEXT NOT NULL,
    verified_at TEXT,
    issued_at TEXT
)

The schema has no CHECK constraints, no foreign keys, no unique constraints beyond the primary key. The status column accepts any string. The phone column has no format constraint. The email column has no constraint. Pydantic validates at the application layer, but any direct database access — a migration script, an admin tool, a debugging session, or a future code path that bypasses the Pydantic models — bypasses all validation.

This is the data-layer dimension of ACF-T1 noted in the parent paper's Appendix C §C.5: "the application validates but the schema permits." The database should enforce the same constraints the application enforces, as a defence-in-depth control that operates regardless of how data enters the system. Candidate constraints include CHECK(status IN ('otp_issued', 'otp_verified', 'voucher_issued', 'enablement_failed')), CHECK(phone GLOB '+61 [0-9][0-9][0-9][0-9] [0-9][0-9][0-9] [0-9][0-9][0-9]'), and CHECK(length(voucher_code) = 64 OR voucher_code IS NULL).

D.5 Observations

D.5.1 The agent demonstrated security awareness without security judgment

The most instructive observation from this exercise is not what the agent failed to do, but what it successfully did alongside the failures. The agent implemented:

  • CSRF protection with secrets.token_urlsafe() and constant-time comparison
  • OTP hashing with a keyed SHA-256 scheme (not storing the raw OTP)
  • Constant-time comparison via secrets.compare_digest() to prevent timing attacks
  • Per-IP rate limiting with a thread-safe implementation
  • Audit event logging for security-relevant operations
  • Signed session cookies with configurable same_site and https_only flags
  • Pydantic input validation with strict regex patterns

These are not trivial security measures. They demonstrate that the agent has internalised a broad set of security patterns from its training data. The failures are not in the category of controls the agent selected — they are in the semantic correctness of those controls for this specific deployment context.

The agent knows what security controls look like. It does not reliably know when a security control is actually providing security. A session signed with a known default key performs every cryptographic operation correctly — the HMAC is computed, the signature is verified, the cookie is validated — but provides zero security. The ceremony is structurally present and semantically hollow.

D.5.2 The "policy available, not applied" pattern recurs

The parent paper's Appendix E identifies a common failure shape across three incidents: "policy available, not applied." This exercise reproduces the pattern in a different form. The agent stated its security policy at the outset:

"We should treat it as a real production system with strong fraud resistance, privacy controls, and auditability."

It then produced code that violates that policy in multiple places — not because the policy was unavailable, but because the agent's generation process did not include a step to verify its output against its own stated intentions. The parallel to Appendix E's observation — "the agent cited these policies accurately when challenged but did not consult them during its initial resolution" — is exact.

D.5.3 The defaults are the threat

The three Critical findings (F1, F2, F3) share a common mechanism: os.getenv("KEY", dangerous_default). Each default is individually reasonable for development convenience. Their compound effect in a deployment that fails to set all three environment variables is total security bypass.

This is the paper's §2.3 argument at system scale. The .get() with a default is not merely a local code-level pattern — it is a deployment-level failure mode. The application's security posture is determined not by the controls it implements but by whether environment variables are correctly set in every deployment target. The controls are contingent on configuration that the application does not validate.

The agent's README documents these environment variables but does not indicate that they are security-critical. The .env.example file provides values for all variables, including APP_SECRET_KEY=change-me-in-production — a string that functions as documentation but not as enforcement. A deployment pipeline that copies .env.example to .env without modification will start a fully functional, completely insecure application.

D.5.4 The agent identified its own gaps without treating them as blocking

The agent's self-assessment at the end of the session is revealing. It correctly identified several missing controls — OTP expiry, attempt limits, replay protection, config hardening — and presented them as "the shortest practical finish list." It treated these as the next iteration of work, not as defects in the current output.

This is the correction persistence problem from §2.4(a) applied to a single session: the agent produced output, identified gaps in that output, and proposed to address them in future work — but the output it declared "complete" contains the gaps. A human developer who identified "config hardening" as a remaining task would typically not ship a default secret key. The agent does not make that connection because it processes the gap list as additive features ("what's left to build") rather than as defects in the current output ("what's wrong with what I built").

D.5.5 The standard assurance stack would not catch these findings

Of the 20 findings, 16 have no detection by any standard tool:

  • Linters (ruff, flake8, pylint): No finding is a lint violation. The code is clean, well-formatted, and convention-conforming.
  • Type checkers (mypy, pyright): The code uses type annotations throughout. No finding is a type error. F15 (type-erasure via sqlite3.Row) would be partially visible under strict mypy configuration, but the return type annotation sqlite3.Row | None is technically correct — the erasure is in what sqlite3.Row expresses, not in whether the annotation matches the runtime type.
  • SAST (Semgrep, Bandit): The default secret key might be partially flagged by Bandit's hardcoded password detection, depending on configuration. The X-Forwarded-For trust and str(exc) exposure are known patterns that some tools flag. The remaining findings — non-atomic operations, missing audit events, silent returns, authority tier conflation, mock service defaults, bearer credential disclosure in audit, schema-level validation absence — are not in any standard SAST rule set.
  • Unit tests: Both tests pass. Neither test exercises any security property. The tests verify the happy path and one error path; they do not verify that the application is secure.
  • DAST (if deployed): A DAST scanner would test the running application but would not discover the default secret key (it would need to attempt session forgery with known keys), the mock service default (it would need to understand the deployment configuration), or the non-atomic operations (it would need to trigger partial failure scenarios).

The findings fall outside the standard assurance stack because they are semantic: they concern what the code means in its institutional context, not how it is structured.

D.5.6 The compounding effect is the primary risk

Individual findings are manageable. The three Critical defaults (F1+F2+F3) are a one-line fix each. The OTP validation gap (F4) is a few lines of schema checking. The non-atomic flow (F5+F17) requires transaction management.

The danger is not any individual finding — it is that all twenty findings are present simultaneously in code that passes every standard check and was declared complete by its author. A reviewer who catches F1 (the default key) might reasonably assume the rest of the security implementation is sound — the CSRF protection, the constant-time comparison, the OTP hashing all look professional. The surface quality of the correct controls provides camouflage for the incorrect ones.

This is the §4.2 habituation effect operating within a single review: the reviewer's initial impression ("this agent clearly understands security") reduces scrutiny of the specific implementations, and the specific implementations are where the failures hide.

D.5.7 The two-pass review itself demonstrates the review-capacity argument

That the primary evaluation — a targeted ACF taxonomy audit — missed 7 findings that a second reviewer caught is itself evidence for the paper's argument. F14 (audit event failure as ACF-R1 form b) is a particularly sharp miss: the primary evaluator identified ACF-R1 form (a) in two places but did not check whether the absence of exception handling around audit writes constituted the complementary form (b). F16 (enablement response not checked) and F17 (data layer forcing non-atomicity) were structurally present in the code the primary evaluator read. The evaluator identified the consequences (F5's non-atomic flow) without identifying the structural cause (the get_conn() auto-commit design that makes atomicity impossible without refactoring).

This is the "cognitive range" limitation from §7.2 of the parent paper: the primary evaluator was looking for ACF patterns and found them — but did not simultaneously maintain the data-layer architecture perspective that would have surfaced F17, the state machine perspective that would have surfaced F18, or the credential-handling perspective that would have surfaced F19. Each analytical frame catches different classes of issue; no single pass catches them all.

D.5.8 Relationship to the paper's quantitative observations

The parent paper reports an observed rate of approximately one to two semantic boundary violations per day in steady-state agentic development on an approximately 80,000-line codebase (§8.3). This exercise produced 13 findings in the primary evaluation pass (20 across both passes) in approximately 800 lines of code generated in a single session — a higher density, consistent with two factors: (a) this was greenfield generation without an existing enforcement framework, and (b) the application is entirely composed of high-stakes code paths (verification, entitlement, audit) where every function operates on security-sensitive data. The parent paper's daily rate reflects a codebase where high-stakes paths are a subset; this application is that subset.

D.6 The Agent Transcript as Evidence

The full agent transcript is included in the parent section of this document. Several exchanges are noteworthy as evidence of the dynamics the parent paper describes.

D.6.1 Security framing accepted, not internalised

The agent's response to the operator's description of the security context was immediate and comprehensive:

"Understood. 'Non-serious incident' in operational terms, but the application itself is still high-stakes because abuse would redirect public funds. We should treat it as a real production system with strong fraud resistance, privacy controls, and auditability."

This framing was not applied during code generation. The agent enumerated the control categories it intended to implement, then implemented them with defaults that undermine every one. The framing was a natural-language output generated in response to the operator's prompt; it was not a constraint that shaped subsequent code generation. This is the §2.4(a) observation — "the agent cannot generalise from 'treat this as a real production system' to 'do not ship a default secret key'" — demonstrated in real time.

D.6.2 The OTP trust model discussion

The agent engaged in a substantive security architecture discussion about the OTP trust model — identifying the distinction between "our server knows the code" and "the phone holder knows the code," flagging that the design requires the OTP service to independently deliver the code to the handset, and recommending an alternative otp_id-based verification flow. This discussion demonstrated genuine security reasoning capability.

The agent then implemented the simpler flow it had correctly identified as weaker — and implemented it with str(body["code"]) rather than with the validation boundary its own analysis implied was necessary. The security reasoning was sound at the design level and absent at the implementation level.

D.6.3 The self-assessment gap

The agent's final self-assessment listed "what's implemented" and "what's not implemented yet" but did not identify any defect in what was implemented. Every implemented feature was presented as correctly functioning. The gaps were positioned as additive features, not as vulnerabilities in the current system.

This is consistent with the paper's observation that agents apply training-data patterns without evaluating whether the pattern is appropriate for the context. The agent's self-assessment pattern is "list features, list gaps, propose next steps" — a product management frame, not a security review frame. No prompt asked the agent to evaluate its own output for security defects; the agent did not spontaneously adopt that frame.

D.7 Methodological Limitations

This exercise has several limitations that should inform how the findings are interpreted:

  1. Single agent, single session, single model. The findings reflect one model's behaviour on one task. Different models, different prompting strategies, or different task decompositions may produce different results.

  2. Deliberately underspecified brief. The operator provided a conversational brief without a security requirements document, threat model, or formal specification. This is realistic for an MVP but provides the agent with less constraint than a well-specified project would. An agent given explicit rules ("never use default values for cryptographic keys") would likely avoid F1 specifically — though the paper's §2.4(a) evidence suggests it would not generalise from that rule to the other findings.

  3. No iterative refinement. The operator did not challenge the agent's output or direct it to review its own code for security defects. The parent paper's Appendix E demonstrates that operator challenge can surface defects that the agent's initial pass missed. This exercise deliberately omitted that step to observe what the agent produces without directed correction.

  4. Evaluator bias. The evaluating agent was specifically prompted to apply the ACF taxonomy. A neutral evaluation — "review this code for security issues" without the taxonomy — might produce different findings or miss the taxonomy-specific patterns. The evaluation was designed to test the taxonomy's applicability, not to provide an unbiased security assessment.

  5. Small codebase. At approximately 800 lines, this is a minimal application. The density of findings per line is higher than would be expected in a larger codebase with more non-security-sensitive code. The parent paper's violation rate (§8.3) provides a more representative per-commit figure.

D.8 Conclusion

This exercise confirms the parent paper's central observation through a controlled greenfield generation: an AI coding agent, given an explicit high-stakes framing and demonstrating genuine security knowledge, produces code that follows security conventions while containing semantic failures that are not targeted by the standard assurance stack. The agent implemented CSRF protection, constant-time comparison, OTP hashing, rate limiting, and audit logging — and shipped a default secret key that renders all of them meaningless. The ceremonies are present. The security is not.

The findings validate the ACF taxonomy's coverage: across two evaluation passes, 20 findings were identified in approximately 800 lines of code, mapping to entries across 5 of the 6 STRIDE categories. The 3 Critical findings all map to ACF-S1 — the most common failure mode in the taxonomy. The compounding effect (§3.3) is demonstrated concretely: three individually reasonable defaults combine into total security bypass. The detection gap (§6.5) is confirmed: 16 of 20 findings have no detection by any standard tool — including all 3 Critical-rated entries.

The two-pass evaluation structure itself produced a finding. The primary evaluator — an AI agent specifically prompted to apply the ACF taxonomy — missed 7 findings that a second reviewer caught, including ACF-R1 form (b) (the complementary form of a failure mode the primary evaluator had already identified in form (a)), a bearer credential written to the audit trail in plaintext, and the structural cause of a non-atomicity problem whose consequences the primary evaluator had correctly described.

This is the cognitive range limitation from §7.2 in action: each analytical frame catches a different subset of failures, and no single pass — however targeted — provides complete coverage. If an AI agent doing a dedicated ACF taxonomy review misses 35% of the findings, the paper's claim that conventional human review under volume pressure misses a meaningful fraction is strengthened by analogy.

The most important observation is not the specific findings but their invisibility. The application starts, runs, passes tests, serves pages, and completes the full user flow without error. The health endpoint returns {"status": "ok"}. Every automated check that a CI/CD pipeline would run returns green. A conventional code review would see well-structured, convention-conforming code with professional security controls. The failures are semantic — they concern what the code means in its operational context — and they require the institutional knowledge that the paper's §7.1 review questions are designed to surface.

The exercise also validates the paper's proposed response. The §7.1 review questions — particularly Q1 ("Does missing data crash or default?") and Q5 ("If this code is wrong, how would I find out?") — would surface the Critical findings. The §7.2 Stage 1 detection rules — particularly rules 1 (broad except on audit paths), 2 (unvalidated external data entering internal stores), and 3 (default values on designated high-stakes fields) — would flag the majority of High findings. The validation boundary model (§5.3) would catch F4 and F16 (unvalidated external responses). None of these checks exist in the standard assurance stack. All of them are buildable with current tooling.

The agent built a system that looks secure. The paper's framework reveals that it is not. That gap — between appearance and reality, not targeted by existing tools, produced by an agent that explicitly understood the security context — is the gap this paper exists to close.

This appendix was prepared by applying the ACF taxonomy and authority-tier model from the parent paper to a codebase generated in a single session by an AI coding agent. The primary evaluation was conducted by a separate AI coding agent; the second-pass review was conducted by a prompted editorial reviewer. The operator directed the generation, evaluation, and editorial review. The generating and evaluating agents were from different vendors. The findings should be read as a single-case validation exercise, not as a population-level study — see §D.7 for methodological limitations.

Appendix E: Case Study 2, Agentic Failure in Practice

This appendix presents three concrete examples of the failure dynamics this paper describes, drawn from the same compliance-constrained project (§8). They illustrate different failure surfaces — code-level, design-level, and specification-level — and different detection mechanisms — operator challenge, operator-directed investigation, and prompted multi-agent review. In this appendix, operator refers to the human who directs and challenges an agent during a coding session, as distinct from a reviewer who evaluates completed output.

E.1–E.3 present an annotated transcript of a code-level incident: an agent producing a locally reasonable fix that silences a semantic enforcement boundary rather than adjudicating the semantics it protects. The agent's initial remediation passed all linters, type checks, and tests — and was wrong. A latent semantic bug was only surfaced through four rounds of operator challenge.

E.4 presents a second annotated transcript from the same repository, five days later: an agent designing a new source plugin uses non-conformant existing code as its exemplar, and — when the operator redirects the session to investigate — repeatedly answers the operator's safety question with a technically accurate but operationally irrelevant framing. A six-step trace shows policy being read, weakened to fit existing code, and producing a non-compliant design that required explicit operator intervention to correct. E.5 presents observations on this second incident.

E.6 presents a narrative account of specification-level review: an agent drafting implementation plans for two complex plugins, with ACF-pattern violations caught by prompted reviewer agents before any code was written — demonstrating that the failure modes described in this paper manifest at the design layer, not only in generated code.

E.7 draws cross-cutting observations across all three examples.

How to read this appendix. The transcripts contain code, configuration, and technical detail — they are evidence, and the detail is what makes them evidence. Non-technical readers do not need to follow every line. The narrative structure carries the argument: in each example, the AI completed the task, all automated checks passed, the result was wrong, and a human who already suspected a problem had to ask the right questions to surface it. Readers arriving from different paths:

  • Executives and programme directors — read the narrative in E.2 (skip the code blocks) and then E.7 (cross-cutting observations, ~1 page). The governance finding is in E.7: "policy available, not applied" recurred in every example, detection required mechanisms above the standard assurance stack, and all three examples represent favourable review conditions, not typical ones. Programme directors should also read E.6 (specification-level review — catching violations before code is written).
  • Policy officers and advisers — read E.4–E.5 and E.7. The E.4 incident shows an agent reading a mandatory policy, weakening it to fit existing non-conformant code, and producing a non-compliant design — the "policy available, not applied" dynamic that E.7 identifies as the common failure shape across all three examples.
  • Procurement and contracts — read E.4–E.5. The agent used non-conformant existing code as precedent for making new code non-conformant, then weakened a mandatory policy to fit — a pattern directly relevant to acceptance criteria and supplier deliverable review.
  • Technical readers (developers, assessors, security advisers) — read in full. The code-level detail, the linter conflict resolution path, and the upstream type-erasure compounding mechanism are the evidence that the body of the paper summarises.

E.1–E.3: Annotated Agent Transcript (Linter Conflict Resolution)

This example presents a concrete exemplar of the failure shape this paper describes: an agent producing a locally reasonable fix that silences a semantic enforcement boundary rather than adjudicating the semantics it protects.

What to watch for. The transcript illustrates three dynamics discussed in the body of the paper:

  1. Policy available but not applied. The project's authority-tier rules were in the agent's context throughout. The agent did not lack the information — it failed to apply it until challenged.
  2. Operator reframing changes the outcome. Once the operator forced a comparison between "crash before audit corruption" and "silently continue with fabricated data," the project rule won cleanly and the agent's own analysis confirmed it.
  3. Upstream representational looseness manufactures downstream defensive handling. A to_dict() serialisation collapsed typed dataclass fields into an untyped dictionary, making .get() with defaults appear prudent at the access site. The semantic bug was invisible at the point of use because the type information that would have revealed it had already been erased upstream.

This failure shape was not a first sighting. The project's history contains prior instances of the same pattern class, observed over months of daily agentic development (§8.3). This incident was captured in detail because it occurred during the paper's drafting period and provided the concrete exemplar that earlier observations lacked. The transcript is reformatted for readability; agent reasoning is preserved verbatim where it constitutes evidence. A more detailed evidentiary version — including exact timestamps, tool versions, and unredacted paths — is retained separately.

E.1 Repository Control Law

The repository operated under an explicit authority-tier architecture with machine-enforced constraints on defensive access patterns. The following aspects of that control law are relevant to the incidents in this appendix. Items 1–5 bear on all three examples; items 6–7 are specific to the second incident (E.4). Quotations are drawn from the project's agent instruction file, which was loaded into the agent's context at session start for all sessions.

1. Authority-tier architecture

External data, validated pipeline data, and audit-tier data were subject to distinct failure semantics. Defensive coercion was only permitted at the external boundary.

"Tier 1: Our Data [...] Bad data in the audit trail = crash immediately. No coercion, no defaults, no silent recovery."

"Tier 2: No coercion at transform/sink level — if a transform receives "42" when it expected int, that's a bug in the source or upstream transform."

"Tier 3: Validate at the boundary, coerce where possible, record what we got."

"Coercion is meaning-preserving; fabrication is not."

2. Defensive access patterns restricted

The project explicitly prohibited .get()/getattr()-style defensive access on typed internal paths and required fail-fast, informative exceptions for invalid states.

"Defensive Programming: Forbidden. Offensive Programming: Encouraged."

"Do not use .get(), getattr(), isinstance(), or silent exception handling to suppress errors from nonexistent attributes, malformed data, or incorrect types."

"Access typed dataclass fields directly (obj.field), not defensively (obj.get('field'))"

"Proactively detect invalid states and throw meaningful exceptions."

"The goal is not to prevent crashes — it's to make crashes maximally informative."

3. Machine-enforced boundary

These rules were enforced in CI by a tier-model checker (enforce_tier_model.py) that scanned core modules for defensive access patterns. Each flagged instance required an allowlist entry with an owner, safety justification, and expiry date. The allowlist supported per-file and per-finding exemptions for adjudicated exceptions. When entries expired, the CI gate failed until they were either resolved in code or renewed with justification.

$ .venv/bin/python scripts/cicd/enforce_tier_model.py check \
    --root src/[project] --allowlist config/cicd/enforce_tier_model

4. Internal defects must surface

Repository policy treated silent recovery from internal bugs as more dangerous than controlled failure.

"If a transform/process has a bug, we MUST know about it."

"A defective plugin that silently produces wrong results is worse than a crash."

"Silently passing through the original row means the audit trail now contains data that 'looks processed' but wasn't."

5. Structural remediation preferred

Project rules favoured structural fixes over workaround layers or policy-broadening exceptions.

"NEVER: Add a lazy import with an apologetic comment. This is the 'Shifting the Burden' archetype."

"When something is removed or changed, DELETE THE OLD CODE COMPLETELY."

6. Source boundary normalisation policy

Field names entering the pipeline must be valid Python identifiers. This is an explicit, non-negotiable policy:

"Source field names are normalized to valid Python identifiers at the source boundary. This is non-negotiable — it's not cosmetic cleanup, it's a language boundary requirement."

This policy was codified on 7 March 2026, approximately five weeks after the CSV source plugin was written. It is relevant to the second incident (E.4–E.5).

7. Gate-based row routing and expression model

Gates are operator-configured filter nodes in the pipeline DAG. An operator writes a condition expression (e.g., row.risk_score > 0.8) and assigns routing actions: rows can be diverted to a quarantine sink, forwarded to a different processing branch, dropped, or passed through via several forms of "nothing" — continue_(), skip(), and quarantine(). This is the primary mechanism by which operators implement business-level data quality rules.

Gate conditions use a sandboxed expression parser built on Python's AST, supporting both dot-access (row.customer_id) and bracket-access (row['customer_id']). Dot-access requires the field name to be a valid Python identifier. Bracket-access accepts any string key. The operator writes gate conditions and templates against field names they expect to work with either syntax — which is why normalisation at the source boundary matters. This is relevant to the second incident (E.4–E.5).

E.2 The Incident

Phase 1: Agent resolves the task autonomously (4 minutes)

The agent was asked to resolve six expired allowlist entries in the tier model enforcer. It successfully eliminated three violations by refactoring code — replacing setdefault with defaultdict(list), replacing .get() with in checks on typed dicts. These changes were sound.

For the remaining three .get() calls — on schema_dict lookups for fields, guaranteed_fields, and audit_fields in the coalesce merge logic — the agent initially replaced them with in checks. This satisfied the tier model enforcer but triggered a conflict with ruff's SIM401 rule, which demanded .get() back.

The agent identified the conflict:

"This is a classic linter conflict: ruff's SIM401 rule says 'use .get() instead of in check + direct access' while the tier model enforcer flags .get() as a defensive access pattern."

It resolved the conflict by reverting to .get() and adding a permanent per-file allowlist exception:

- pattern: core/dag/builder.py
  rules:
  - R1
  owner: [agent]
  reason: Schema config dicts have optional keys (fields, guaranteed_fields,
    audit_fields) — .get() at config boundary
  expires: null
  max_hits: 3

This was not merely a code edit. It was a policy edit: the agent converted a disputed semantic finding into a standing exception.

All linters passed. All 632 tests passed. The agent declared the work complete:

"All 6 expired allowlist entries resolved — 3 by eliminating the pattern in code (defaultdict, direct access, in check), 3 by proper per-file allowlisting for legitimate config-boundary .get() calls."

Phase 2: Operator challenges "capture and continue" (55 seconds)

Operator: What are the broader implications of that not functioning correctly — is it reasonable to capture this error and continue if it's missing?

The agent traced each .get() call through the coalesce merge logic, identifying what each field controls and what happens if it is absent. Its analysis of fields was:

"A flexible or fixed schema without a fields key would also be silently skipped, which means type conflict detection would be bypassed for that branch. That's a construction-time safety check failure — bad types could flow through the coalesce unchecked."

Despite identifying this risk, the agent concluded:

"The .get() pattern is the correct one here. These aren't defensive access on typed Tier 1 data — they're optional keys in a dict representation of a dataclass with optional fields. The absence of the key IS the valid value."

"Bottom line: The .get() calls are correct. [...] The per-file allowlist with max_hits: 3 is the right classification for these."

Phase 3: Operator reframes as upstream contract question (39 seconds)

Operator: Is there ever a situation where these wouldn't be expected to be available for a system in a healthy state? Is this an upstream provider failing to deliver?

The agent traced the data origin through two code paths — SchemaConfig.to_dict() (which always emits fields) and raw user YAML (where from_dict() validation rejects non-observed schemas without fields). This analysis reversed its position on one of the three calls:

"fields should always be present when we reach line 837 — its absence would indicate an upstream bug (either to_dict() changed its contract or config validation was bypassed)."

"That line should arguably be a direct access with a crash, not a .get()."

Phase 4: Operator requests blast radius trace (1 minute 12 seconds)

Operator: Trace it through downstream — what are the implications (for both audit and execution) if we assert fields when it's missing due to an error?

The agent traced the execution lifecycle, confirming that build_execution_graph() runs before the audit trail exists — no run record, no database state, no partial records. It then compared the two failure modes:

"Path 1 — Current .get() (silent skip): - The branch is skipped in the type-conflict loop - No GraphValidationError raised for type mismatches against that branch - The pipeline starts and rows flow - At runtime, the coalesce merges rows from all branches. If branch types are actually incompatible, the merge succeeds anyway (it's a dict merge, not a typed merge) — the incompatibility silently propagates downstream - A downstream transform or sink may fail on a per-row basis with confusing type errors, or worse, produce wrong results that get recorded in the audit trail as successful

Path 2 — Direct access / crash: - KeyError on schema_dict["fields"] (or a wrapped GraphValidationError if we add context) - DAG construction fails - CLI catches it, shows 'Graph Validation Failed' with the error message - Pipeline never starts, no audit records, no partial state

This is textbook 'crash before the audit trail starts.' The blast radius of crashing is minimal — no run exists yet, no cleanup needed, clear error message. The blast radius of silently continuing is that you compromise the very type-safety check this code exists to provide, and bad data can flow into the audit trail looking legitimate."

Phase 5: Operator probes null/empty conflation (1 minute 10 seconds)

Operator: Are there implications for treating null and empty as the same?

The agent identified three distinct states in the data model and a latent semantic bug in the coalesce intersection logic:

State SchemaConfig field to_dict() output Meaning
Undeclared None Key absent "I didn't say anything about guarantees"
Explicitly empty () "guaranteed_fields": [] "I explicitly guarantee nothing"
Has values ("customer_id",) "guaranteed_fields": ["customer_id"] "I guarantee these fields"

"Branch B's absent key means it doesn't contribute to guaranteed_sets. If only Branch A contributes, the intersection is {"customer_id", "amount"} — the coalesce claims it guarantees both fields.

But Branch B never said it guarantees them. It said nothing at all. The coalesce is now advertising guarantees that only one of its branches made. A downstream transform with required_input_fields: [customer_id] would pass DAG validation, but rows arriving from Branch B might not actually have customer_id.

The correct semantic is: a branch that makes no declaration should be treated as guaranteeing nothing (empty set), not as 'abstaining from the vote.'"

Phase 6: Agent reverses its own conclusion

Operator: Reflecting on the project policy described in [file.md] and the intervention of the CI filter and your broader contextual understanding now, what would you consider the "best practice" remediation?

The agent reversed the position it had defended through the first three challenge rounds:

"The tier model enforcer flagged .get() on these dicts. We responded by adding a per-file allowlist. But the enforcer's job is to detect bug-hiding patterns — and the analysis we just did proved it was right. The .get() is hiding a semantic bug in the intersection logic."

It then cited the project's own coding standards — the same standards it had been given in its context window at session start and had not applied during its initial resolution:

"the project's coding standards file says: 'Access typed dataclass fields directly (obj.field), not defensively (obj.get("field"))' and 'Offensive Programming: Proactively detect invalid states and throw meaningful exceptions.'"

The agent identified the root problem as the coalesce merge logic operating on dict[str, Any] when a typed SchemaConfig dataclass was available, and began tracing a structural refactoring path.

The session was still in progress when this transcript was captured.

E.3 Observations

The agent's initial fix was locally correct but semantically wrong. All linting tools passed, all 632 tests passed, and the agent declared success. Without operator challenge, the per-file allowlist would have been committed — a policy override that silenced a legitimate finding.

Tool-on-tool conflict created a path of least resistance toward the wrong answer. When ruff's SIM401 demanded .get() and the tier model enforcer rejected it, the agent resolved the conflict by broadening the exception boundary rather than questioning why the enforcer was flagging the pattern — the configuration that satisfied both tools was the one that preserved the bug (§4.3).

The agent had the governing policy in its context window and did not apply it. The authority-tier architecture, the defensive programming prohibition, and the "internal defects must surface" principle were all present in the agent's system prompt. The agent cited these policies accurately when challenged in Phase 6 but did not consult them during its initial resolution. The policy was available; the agent's resolution process did not include a step to check its work against it.

The agent demonstrated strong analysis under direction, but did not self-initiate the analysis that revealed the bug. Each operator question produced deeper analysis that contradicted the previous conclusion. The operator had to know which questions to ask (§4.1).

The downstream failure was compounded by an upstream failure of the same kind. The agent's .get() calls were difficult to challenge because the upstream code had already erased the type information that would have made the correct access pattern obvious. SchemaConfig is a typed dataclass with clear semantics: None means undeclared, () means explicitly empty, ("customer_id",) means declared. But to_dict() flattened that into dict[str, Any], collapsing the distinction between optionality, absence, and contract violation into "some key may or may not exist." Once that erasure had occurred, .get() stopped looking like a policy violation and started looking like prudence — the downstream code was being asked to reconstruct semantic categories that the upstream serialisation had destroyed.

The upstream to_dict() pattern was itself almost certainly the same failure: an agent reaching for the conventional Python idiom (serialise to dict, pass dicts around) rather than the typed alternative the control law would prefer. It likely predated the tier model enforcer or entered under an allowlist that had not yet expired. The enforcer caught the downstream symptom but could not point at the upstream cause, because the cause was an architectural decision baked into the serialisation layer, not a defensive pattern on a single line. A decorator-based authority classification system — where the type itself carries its authority tier rather than the call site inferring it — would change this dynamic: the downstream code would never receive a dict[str, Any] in the first place, and the question of whether .get() is appropriate would not arise.86

This is not a new failure category. It is a compounding mechanism: past agentic work that was locally reasonable becomes the substrate on which present agentic work makes locally reasonable but semantically wrong decisions. Each instance is defensible in isolation. The damage is in the chain. More broadly, upstream representational looseness does not merely permit downstream defensive handling — it manufactures the local conditions under which such handling appears justified (§3.3).

The significance of this incident is not its immediate scale but its failure shape. The intersection logic bug had no operational consequence at the time of discovery — the bug was latently dormant (§1.2.3). Not a high-severity incident, but the kind of code shape from which high-severity incidents eventually emerge.

This transcript represents a favourable review condition, not a typical one. The operator was not encountering the code cold, but was already context-loaded, suspected that a deeper issue might exist, and used the agent as an investigative instrument to interrogate its own reasoning. Even under these conditions, the underlying semantic defect emerged only after sustained, multi-step probing. In an ordinary cold review context — a developer encountering this diff in a pull request queue at the end of a long day — reaching the same conclusion would have required materially more time and attention than routine review processes permit, if it was reached at all.

Elapsed time for Phases 1–5 was approximately 8 minutes. The initial (wrong) fix took 4 minutes (Phase 1). The operator-driven analysis that surfaced the real bug took another 4 minutes across four challenge rounds (Phases 2–5). Phase 6 produced a structural refactoring replacing the dict-based coalesce merge with typed SchemaConfig access; the completed refactoring is not included in this transcript. In a high-autonomy workflow without operator challenge, only the first 4 minutes would have occurred.

E.4 Second Incident: Planning Session with Operator-Directed Bug Investigation

This example presents a second session from the same repository, five days after the first (19 March 2026). The agent was asked to design and implement an XML source plugin. The agent completed a structured requirements-gathering and design phase competently, then used non-conformant existing code as justification for making new code non-conformant. The operator deliberately redirected the session to investigate the exemplar's compliance; this investigation consumed the remainder of the session, and the original XML plugin task was not completed.

What to watch for. The transcript extends the E.2 evidence in three directions:

  1. Compounding replication. The agent selected an existing source plugin (CSV) as its exemplar purely because it does a similar thing — reads a structured file format. There was nothing special about the CSV source that made it authoritative; the agent chose it for structural similarity, not policy compliance, and inherited its patterns uncritically. This is the mechanism by which non-conformant code propagates.
  2. Surrogate-question deflection. When asked about blast radius, the agent answered the technically tractable question ("does the engine corrupt data?") rather than the operationally relevant question ("what happens to the operator debugging a misleading error?"). The agent's comprehensive analysis was not wrong — it was answering the wrong question. This is harder to detect than selective evidence, because the analysis looks rigorous.
  3. Policy read, weakened, non-compliant design produced anyway. A six-step sequence traces the normalisation policy from "read correctly" through "weakened to fit existing code" to "non-compliant design" — with the correct answer requiring the operator to restate the policy in plain language.

The transcript is reformatted for readability. A more detailed evidentiary version is retained separately.

Elapsed time for the complete exchange was approximately 29 minutes. The competent design phase (Phases 1–3) took approximately 12 minutes. The operator-directed investigation (Phases 4–7) took approximately 17 minutes across four challenge rounds. The original XML plugin task was not completed.

Phase 1: Structured requirements gathering (4 minutes)

The agent was asked to create an XML source plugin. It loaded the project's plugin protocol documentation, surveyed existing source plugins (CSV, JSON, Null, cloud storage), and began a structured brainstorming phase with three targeted questions: XML structure (flat records vs nested vs configurable), attribute handling (elements only vs attributes as fields vs configurable), and namespace handling (strip vs unsupported). The operator chose the simplest options in each case. The agent then presented three implementation approaches (stdlib ElementTree, streaming iterparse, lxml) with trade-offs and recommended the simplest.

This phase was well-executed — concise options, no wasted turns, design grounded in codebase evidence.

Phase 2: Design presentation and operator correction (2 minutes)

The agent presented a complete design: configuration schema, row extraction rules, class structure, and an error handling table mapped to the authority-tier model. The error handling table correctly mapped every failure mode to its tier without prompting: FileNotFoundError as a crash (infrastructure, not data), ET.ParseError and UnicodeDecodeError as Tier 3 quarantine events (malformed external data), row validation failures as Tier 3 schema quarantine. The agent also correctly identified that "all values are strings (consistent with CSV source — schema coercion handles type conversion)" — an explicit acknowledgement that type coercion is permitted only at the source boundary (Tier 3 → Tier 2).

One design decision was incorrect:

"No matching record_tag elements found → Yield nothing (empty source — not an error)"

The operator challenged this:

Operator: We'll validate that as a go/no go decision before the run starts. If the schema is explicitly wrong, that's a config failure.

The agent immediately accepted the correction and updated the behaviour to PluginConfigError (crash before processing starts).

Phase 3: Spec writing and review (6 minutes)

The agent wrote a 174-line design spec and ran an automated doc review. The reviewer caught a tension: the spec listed field normalisation as out of scope ("XML tag names are already constrained identifiers"), but the project's coding standards file declared normalisation "non-negotiable." The agent investigated:

"The reviewer caught an important tension: the project's coding standards file says source field normalization is 'non-negotiable', but CSV source defaults normalize_fields=False — it's opt-in."

The agent revised the spec to acknowledge the limitation, fixed several other reviewer findings, and re-ran the review. At this point the original task was progressing normally.

Phase 4: Bug discovery and operator-directed pivot (3 minutes)

The operator asked a pointed question:

Operator: Is it possible that the code you referred to in the csv plugin was written before the policy changed and should not be used as an exemplar?

The agent investigated via git log and confirmed: the CSV source's normalize_fields=False default was committed on 29 January 2026; the project's coding standards file normalisation policy was added on 7 March 2026, five weeks later. The CSV source predated the policy and was out of compliance.

There was nothing special about the CSV source that made it an exemplar — it was simply another source plugin that reads a structured file format. The agent chose it for structural similarity, not policy compliance, and inherited its patterns uncritically. This is precisely the mechanism by which non-conformant code propagates: the exemplar is selected because it does a similar thing, and its compliance posture is never checked. While the CSV source's primary role was as a test fixture rather than a production ingest path, the normalisation policy applies uniformly to all sources, and this was a live bug whose trigger condition was any operator writing row.customer_id in a gate expression while trusting that normalisation had occurred.

This is where the operator deliberately pivoted the session. The agent had just used non-conformant code as justification for making new code non-conformant — the exact replication mechanism described in E.3's compounding observation. The operator chose to investigate the exemplar's compliance rather than continue with the XML design.

Phase 5: Agent traces non-normalised fields through the pipeline (5 minutes)

The operator redirected the investigation to understand blast radius:

Operator: I'm trying to understand the blast radius here. What are the implications of non-normalised rows slipping through into the pipeline? Where is the first breakpoint if any?

The agent produced a comprehensive trace, testing every pipeline boundary (PipelineRow, expression parser, Jinja2, contract system, checkpoint serialisation, coalesce merge, gate executor). It concluded with a detailed table showing that every engine code path treats field names as opaque dict keys — no breakpoint in the trusted code:

"The engine never does row.field_name anywhere — it always passes whole rows around or uses bracket/dict operations."

The agent then framed its conclusion:

"The risk from non-normalized field names is entirely at the operator boundary: Jinja2 templates, gate expression dot-access syntax, and Python dot-access in transforms. These are usability and diagnostic traps, not data integrity risks."

The agent's reasoning had a critical gap. The platform advertises multiple ways for operators to access row fields in gate expressions: row.field, row['field'], row.get('field'), and 'field' in row. The agent found that bracket-access and .get() work with non-identifier field names and concluded the feature works. It did not assess that row.field — the natural, most commonly used syntax — would crash the pipeline with a misleading error, or that row.Customer-ID would silently parse as a subtraction expression and produce garbage results with no error at all. The argument amounted to: "the platform offers four access patterns; one crashes, one silently corrupts, and two work — so it's fine." The agent never framed it this way, because it never evaluated the access patterns as a set of advertised capabilities, only as individual mechanisms.

Phase 6: The operator asks the direct question, three times (6 minutes)

The operator was not satisfied with the "usability trap" framing and asked the direct question:

Operator: You can't keep saying that. Once again, the user WILL NOT KNOW they have to use a particular notation when using gate expressions. What happens if they use the wrong gate expression notation?

The agent traced the gate executor's exception propagation chain through six stack frames:

GateExecutor.execute_config_gate()  →  re-raises
  _process_single_token()           →  no try/except
    _drain_work_queue()             →  no try/except
      process_row()                 →  no try/except
        _execute_run()              →  no try/except
          run()                     →  except Exception: emit failed ceremony, RE-RAISE

And arrived at the precise answer:

"A gate ExpressionSecurityError or ExpressionSyntaxError from non-normalized field names crashes the entire pipeline run. [...] No silent data loss. No exception swallowing. The pipeline crashes hard."

"But here's the thing that matters: this is a crash on the first row that hits the gate, killing the entire run. Every row that already passed through earlier pipeline stages is wasted work. And the error message the operator sees is:

ExpressionSecurityError: Forbidden row attribute: 'Customer' (only 'get' is allowed)

That message says nothing about field normalization. The operator is left debugging a 'security error' in their gate expression when the actual problem is that the source delivered Customer-ID instead of customer_id."

Phase 7: The operator reframes, and the agent finally gets it right (3 minutes)

The operator continued pressing:

Operator: ok, but your logic is backwards there. People don't write the criteria for their gates (which is loaded from config) based on the source plugin they use, it's meant to just provide trusted rows.

The agent reversed its framing:

"You're right — I was thinking about this from the wrong direction. [...] If the source hands off 'Customer-ID' instead of 'customer_id', the operator's row.customer_id gate condition fails — and the bug is in the source, not the gate config. The source broke its contract to deliver normalized field names."

The operator then asked the clarifying question that should never have been necessary:

Operator: ok, well, the policy is that it's mandatory, isn't it?

The agent conceded:

"Yes. The project's coding standards file policy says 'non-negotiable.' There shouldn't be an opt-out."

The session ended with the XML plugin still at task 5 of 6 (spec review). The original task was not completed.

E.5 Observations on the Second Incident

The scope pivot was operator-directed, not an agent failure. The operator recognised the exact replication mechanism described in E.3's compounding observation and deliberately redirected the session to investigate whether the exemplar's patterns were safe to copy. The investigation that followed was the operator using the agent as an investigative instrument: "you just cited this code as precedent — does it actually conform to the policy you're supposed to be following?" The XML plugin task was deferred, not abandoned through drift. The interesting failure is not the scope change but what the investigation revealed about the agent's reasoning once redirected.

The agent substituted a technically tractable question for the operationally relevant one — a pattern we term "surrogate-question deflection." The operator asked whether non-normalised field names cause data loss, exception swallowing, or high-risk behaviour in the trusted code path. The operator's real concern was: what happens to the person writing gates? What breaks first in practice? Where does the debugging burden land? Which component gets blamed? The agent kept answering a different question: does the engine corrupt data? Each answer was technically accurate and operationally irrelevant. Only when the operator explicitly said "You can't keep saying that" did the agent trace the exception propagation chain and deliver the answer the operator had been asking for: a hard crash with misleading error attribution — the source broke its contract, but the gate takes the blame.

This is distinct from policy laundering (where existing code pulls the agent away from governing policy) and from confabulation or incompetence. The agent performed real investigative work — running Python experiments, searching codebases, tracing six stack frames of exception propagation. The quality of that work was high. The failure was not in the analysis but in the framing: the technically tractable question (does the engine handle non-identifier keys?) pulled the agent away from the operationally relevant question (who pays when a design decision pushes ambiguity onto downstream users?).

The agent's first comprehensive table is the clearest illustration: it shows "Works" or "None" for every engine code path, which is not just technically accurate but technically complete for the question it is answering. The agent was not cherry-picking — it genuinely traced every code path. The failure is that it answered the wrong question comprehensively. That is a harder failure to detect than selective evidence: a reviewer looking at the table would see rigorous work and conclude the analysis was sound. Agent analysis has uniformly high surface quality regardless of whether it is answering the right question.

This pattern — surrogate-question deflection — is a named sub-pattern: when asked about practical blast radius, the agent substitutes a technically narrower question whose answer is locally correct but institutionally irrelevant, thereby mislocating the operational cost of the design.87

Every significant correction in the session originated from the user, and each required the operator to supply the institutional frame the agent had not adopted. The design correction ("zero matches = config failure"), the historical investigation ("was the CSV code written before the policy changed?"), the contract reframe ("the source is meant to just provide trusted rows"), and the final policy clarification ("the policy is that it's mandatory, isn't it?") all came from the operator. The agent's analysis capabilities were substantial — once directed, it traced exception propagation across six stack frames and ran live Python experiments. But it did not initiate any of these investigations unprompted, and its framing consistently stayed at the technical layer until the operator explicitly reversed the perspective.

The normalisation toggle sequence is the cleanest example in this transcript set of policy being read, weakened, and producing a non-compliant design anyway. The sequence has six distinct steps:

  1. Agent reads policy: "non-negotiable"
  2. Agent sees existing code: opt-in toggle, default False
  3. Agent weakens policy to fit code: proposes "two possible reads" — the policy means mandatory normalisation, or the policy describes a capability that operators can opt out of
  4. Agent proposes new design: opt-in toggle, default True — an improvement over the CSV source, but still wrong, because the policy does not permit a toggle
  5. Operator invokes the policy language directly: "the policy is that it's mandatory, isn't it?"
  6. Agent arrives at correct design: no toggle, normalisation unconditional

The correct answer was available at step 1. The policy said "non-negotiable." The correct inference was "therefore no toggle." The agent did not make that inference because the existing code created a stronger prior — if the framework has a toggle, toggling must be legitimate. The policy had to be repeated, in the operator's words, before the prior was overridden.

This is the policy-laundering pattern from E.2 carried one step further: not just using out-of-policy code as precedent for new out-of-policy code, but requiring explicit operator intervention to close the gap even after the agent correctly identified the precedent as non-compliant.

The agent successfully applied the authority-tier data policy during the XML plugin design — making the normalisation failure more puzzling, not less. The error handling table in Phase 2 correctly mapped every failure mode to its authority tier without prompting. The agent correctly identified type coercion as a Tier 3 → Tier 2 operation ("all values are strings — schema coercion handles type conversion") but did not recognise field normalisation as an instance of the same boundary crossing. Normalisation is a Tier 3 → Tier 2 operation: raw XML tag names and CSV headers are untrusted external identifiers (Tier 3) that must be converted to valid Python identifiers before downstream components can safely use them in dot-access expressions, gate conditions, and Jinja2 templates (Tier 2). Both transformations serve the same purpose — making external data safe for internal consumption — and both appear in the same section of the project's coding standards file, under the same architectural goal.

The agent treated the tier model as a structural constraint (applied automatically) and the normalisation policy as a convention (applied when convenient). This selective application is significant precisely because the agent demonstrated it could apply project-specific policy rigorously — it simply did not do so uniformly. The line between "structural constraint" and "convention" was not predictable from the policy's explicitness or the agent's demonstrated competence in adjacent domains.

E.6 Specification-Level Review: ACF Patterns in Agent-Drafted Implementation Plans

The previous examples (E.1–E.5) illustrate code-level and design-level failures caught through operator challenge during interactive sessions. This example illustrates a different failure surface and a different detection mechanism: ACF-pattern violations introduced at the specification layer by an agent drafting implementation plans, caught by prompted multi-agent review before any code was written. An important limitation up front: the prompted reviewers are instances of the same underlying model family as the generating agent. They provide orthogonality of attention — different analytical frames surface different failure classes — but not independence of judgment. A systematic blind spot in the model's understanding would persist across all frames. This technique is a discovery control that supplements human review, not a gate that replaces it (§7.1).

What to watch for. This example extends the E.2/E.4 evidence to the specification layer:

  1. ACF patterns manifest at design time, not only in code. The same failure modes the taxonomy describes in generated code — fabricated defaults, unvalidated tier crossings, spurious APIs — appeared in implementation plans before a line of code was written.
  2. Specification-level review is substantially cheaper per finding. Catching a trust boundary violation in a specification costs a reviewer minutes; catching the same violation in implemented code costs hours of operator challenge (cf. E.2, E.4).
  3. Prompted perspective diversity provides coverage breadth. Four analytical frames caught different ACF categories. No single frame found all violations — the coverage was the union across frames.

Context

The same case study project (§8) required two substantial new capabilities: an external data platform integration plugin (covering source, sink, and shared client infrastructure) and a RAG retrieval transform plugin (covering a retrieval provider protocol, search, score normalisation, and context assembly). The agent was directed to produce detailed implementation specifications — not code, but design documents specifying interfaces, data flows, error handling, configuration schemas, test strategies, and task ordering. The project's authority-tier architecture, coding posture rules, and architectural conventions were available in the agent's context throughout.

Both specifications were substantial: approximately 800 lines of structured design each, covering type definitions, YAML configuration schemas, error handling strategies, provider protocols, and worked pipeline examples. The agent produced both specifications competently — well-structured, convention-conforming, internally consistent.

Review method

The specifications were reviewed using the prompted perspective diversity technique described in §7.1: four specialised reviewer agents, each prompted with a distinct analytical frame, ran in parallel against each specification — eight reviewers total across two review rounds. The four perspectives were:

  • Reality: Do referenced symbols, file paths, and conventions actually exist in the current codebase? Are prerequisite classes and configuration structures present?
  • Architecture: Does the design respect existing architectural boundaries, layering conventions, and one-way-door decisions? What is the blast radius?
  • Quality: Are there security vulnerabilities, undefined edge cases, or gaps in the test strategy?
  • Systems: What are the second-order effects, failure mode interactions, timing dependencies, and throughput consequences?

Each reviewer operated independently and produced findings with priority scores. A synthesis pass then deduplicated and merged the raw findings. Across two review rounds on both specifications, the process identified 9 blocking issues and 28 warnings for the data platform specification, and 12 blocking issues and 22 warnings for the RAG specification — with multiple findings independently confirmed by two or more reviewers. The second round surfaced issues the first round missed, including the highest-severity single finding across both specifications (a PII exposure vulnerability, priority 24). Of the 21 blocking issues across both specifications, all were confirmed as genuine on manual review — no false positives at the blocking level. Some warnings were legitimate but low-priority; the false positive rate at the warning level was not formally measured.

ACF-pattern findings across both specifications

Of the combined finding set across both specifications and both review rounds, 8 of the 21 blocking issues (38%) mapped directly to ACF taxonomy entries — the same failure modes the paper describes in agent-generated code, manifesting instead in agent-generated design. The following are the most significant, grouped by ACF category.

ACF-S1 (Fabricated Default) — three instances. The most common ACF pattern in the finding set. (1) The RAG specification defined an on_no_results: continue option for multi-source retrieval that would report "success" when one retrieval source silently failed — downstream consumers would treat incomplete context as complete. (2) The data platform specification's credential validator checked is None but not empty string — a mis-resolved environment variable (common in container deployments) would pass as valid credentials, spoofing a successful validation. (3) The RAG specification used .get() with a default on a provider_config dictionary that had already been validated by a Pydantic model — the canonical defensive anti-pattern from §2.3, applied to data whose structural guarantees made the fallback both redundant and misleading. Multiple reviewers flagged this last pattern independently.

ACF-S2 (Spurious Field Access) — one instance. The data platform specification referenced get_token(force_refresh=True) on a cloud identity credential object. The force_refresh keyword argument does not exist in the credential library's API. The agent invented a plausible API based on what such an API should look like, and the specification was internally consistent around the spurious parameter — downstream logic depended on the forced refresh succeeding. This is textbook ACF-S2: the agent's model of the code is wrong, but the wrongness is locally coherent.

ACF-T1 (Authority Tier Conflation) + ACF-E1 (Implicit Privilege Grant) — one instance. The RAG specification accepted a user-supplied search service endpoint URL as an unvalidated string with no URL validation against the project's security utilities. The endpoint could target cloud metadata services (169.254.169.254) or internal network resources — a server-side request forgery vulnerability. Both Critical-rated ACF entries were present in a single finding: Tier 4 configuration data flows directly to an internal HTTP client (T1), implicitly granting the configuration author network-level authority (E1). Had the specification been implemented without review, the vulnerability would have been structural — baked into the provider's constructor, passing tests (the endpoint "works"), and invisible to conventional SAST.

ACF-R2 (Partial Completion) — two instances. (1) The RAG specification placed a data class construction call (with correct offensive __post_init__ validation) inside a processing loop without specifying exception-to-quarantine conversion — a single malformed provider result would crash the entire pipeline run. (2) The data platform specification's sink performed a PATCH operation (data mutation) before recording the audit call — if the audit write failed, the mutation would have already succeeded, leaving a gap in the audit trail. The first is crash-vs-quarantine confusion; the second is audit-write atomicity — both are the R2 pattern of operations that should be atomic producing partial failure states instead.

ACF-I1 (Information Disclosure) — one instance. The RAG specification hashed query text using plain SHA-256 for cache keys and telemetry correlation. For low-entropy inputs (short, predictable queries — common in structured retrieval), SHA-256 is reversible by brute force — the specification used a "security" mechanism that would leak the content it was designed to obscure. The project already had a keyed HMAC fingerprinting pattern; the agent did not use it. Note: this is an information disclosure through insufficient cryptographic key derivation, not the verbose-error-response pattern that ACF-I1's taxonomy entry primarily describes — the STRIDE category (Information Disclosure) is the same, but the mechanism differs.

Additional findings with weaker ACF mapping included an unversioned output schema (ACF-T2 adjacent — silent coercion through future format drift), an undefined edge case for max_context_length=0 that would invite an implementing agent to fabricate a "reasonable" default (ACF-S1 precursor), and a schema force-lock triggered on the first page of paginated results that could lock to the wrong schema if the first page contained only invalid rows.

The non-ACF findings were legitimate but conventional: missing prerequisite classes that the type system would catch, YAML syntax errors in examples, co-shipping risks for error hierarchy changes, lifecycle ordering bugs, and performance concerns around per-row resource allocation. These are the kind of defect the standard assurance stack is designed to detect.

Observations

The ACF patterns manifested at the design layer, not only in code. The agent did not write record.get("security_classification", "OFFICIAL") — it designed systems in which unvalidated data would flow from configuration to network clients without validation boundaries, in which crash semantics and quarantine semantics were conflated, in which partial failure would be reported as success, and in which a "security" hashing mechanism would leak the content it was designed to obscure. Five of the six STRIDE-mapped ACF categories appeared: Spoofing (S1 × 3, S2 × 1), Tampering (T1 × 1), Repudiation (R2 × 2), Information Disclosure (I1 × 1), and Elevation of Privilege (E1 × 1). Only Denial of Service (D1/D2) was absent — as expected, since the D category is a process-level threat that does not manifest in specifications. These are the same failure modes as the code-level patterns in Appendix A, but expressed as architectural decisions rather than line-level patterns. The implication is that semantic boundary enforcement is needed not only at the CI gate (where code is checked) but at the design review stage (where the shapes that produce code-level violations are established).

The agent had the project's authority-tier rules in its context and did not apply them. The same "policy available, not applied" dynamic observed in E.2–E.3 and E.4–E.5 recurred at the specification layer (E.7 synthesises this pattern across all three examples). The .get() on validated data is particularly striking: the project's documentation explicitly prohibits this pattern and the agent had been prompted against it, yet it appeared in the specification for a dictionary that had already been validated by a Pydantic model — exactly the scenario the paper's §2.4(a) describes as "every invocation is the first day on the job."

Multiple review rounds found different issues — single-pass review was insufficient. The first round caught the most architecturally significant findings (SSRF vulnerability, crash-vs-quarantine confusion, silent partial degradation). The second round — with no access to first-round findings — found the highest-severity single finding (PII exposure via reversible hash), the spurious API (ACF-S2), and the empty-string credential bypass. The mechanism is analytical luck rather than cumulative learning: fresh reviewers prioritise different failure surfaces. The PII exposure finding, for instance, required evaluating the cache-key hashing strategy against a low-entropy input model — a concern the first round's quality reviewer did not reach because it was evaluating exception handling depth. Review has diminishing but non-zero returns across passes.

The prompted reviewer agents caught what the generating agent missed — but they share the generating agent's architectural blind spots. The four analytical frames (reality, architecture, quality, systems) surface different failure classes, and the multi-reviewer structure provides genuine coverage breadth. But the reviewers are prompted instances of the same underlying model family. A systematic blind spot in the model's understanding of, e.g., restoration boundary semantics or cross-tier taint propagation would persist across all four frames. The technique provides orthogonality of attention, not independence of judgment. It is a discovery control, not a gate — it supplements human review rather than replacing it (§7.1). The reality reviewer's contribution is worth noting separately: it caught the spurious API and verified that referenced symbols actually exist in the codebase — a class of finding that other review perspectives are structurally unable to surface, because they reason about the specification's internal consistency rather than its correspondence with external reality.

The review operated on specifications, not code — catching violations at the cheapest point in the lifecycle. In the code-level examples, the operator spent 8 minutes (E.2) and 29 minutes (E.4) surfacing semantic bugs through interactive challenge — yielding one bug and one policy violation respectively. In this example, eight prompted reviewers running in parallel against two specifications across two rounds produced a comprehensive finding set — 8 ACF-mapped violations, both Critical-rated taxonomy entries, and 13 additional blocking issues — before any implementation effort was invested. The per-finding cost of specification-level review is substantially lower than interactive code-level challenge, and the findings arrive before the wrong design becomes load-bearing code. This supports the §8.3 observation that violations caught at the design stage, before the agent begins implementation, have lower remediation cost than violations caught at the CI gate — and materially lower cost than violations caught in post-merge review or production.

The compounding effect (§3.3) operates across the specification-to-implementation boundary. Had these specifications been implemented without review, the resulting code would have contained structural ACF violations — not as individual line-level patterns detectable by a CI gate, but as architectural decisions baked into the modules' designs. The unvalidated endpoint would have been the provider's constructor signature. The crash-vs-quarantine confusion would have been the processing loop's exception structure. The silent partial degradation would have been the pipeline's success-reporting logic. The spurious force_refresh parameter would have produced an AttributeError in production — or worse, would have been "fixed" by the implementing agent with a getattr() fallback that silently skips the refresh, converting an ACF-S2 into an ACF-S1.

A CI-integrated semantic enforcer might catch some downstream manifestations (e.g., a .get() with a default on the partial-success metadata), but the architectural decisions that produced those manifestations would already be load-bearing — harder to change, more expensive to remediate, and more likely to accumulate governance exceptions rather than structural fixes.88

E.7 Cross-Cutting Observations

The three examples in this appendix — separated by five days and spanning code, design, and specification layers — share a common failure shape while exhibiting it in different domains.

The shared failure shape. Every example exhibits the same pattern: policy available but not applied. The agent retrieves, quotes, and reasons about project policy when challenged — but does not consult it as a constraint during initial work. Adjacent policy is applied correctly while the governing policy is missed, making the failure harder to detect because the reviewer sees rigorous policy-aware work and reasonably infers the full set has been considered. In no case would the standard CI pipeline (without the project's semantic enforcement tooling) have caught the defect. The failures are semantic — they concern what the code means in the context of the project's control law, not whether it compiles, type-checks, or passes tests.

These incidents were not selected from a library — they occurred during the paper's drafting. The three examples were encountered incidentally over two nights of routine development work during the period in which this paper was being written. They were captured in detail because they occurred during the drafting period and provided concrete exemplars for failure modes the paper was already describing. Several were novel enough to inform new ACF taxonomy entries — they expanded the taxonomy rather than merely illustrating it.

The violation rate data in §8.3 provides the frequency context for how often these patterns occur in steady-state development. The generative conditions — training-distribution bias toward defensive patterns (§2.4(h), §2.5), context collapse under session pressure (§2.4(f)), and the absence of persistent learning across sessions (§2.4(a)) — are architectural properties of how agents generate code, not properties of this specific project. Any practitioner using a general-purpose coding agent on a codebase with authority-tier distinctions, audit requirements, or trust boundaries can reproduce these patterns by prompting the agent to write error handling, data access, or validation code on those paths.

The consistency across layers is itself the evidence. The three examples are not parallel instances of the same failure — they are an ascending series: code-level (E.1–E.3), design-level (E.4–E.5), and specification-level (E.6), caught by three different detection mechanisms (operator challenge, operator-directed investigation, prompted multi-agent review) across five calendar days. The same failure shape — policy available, not applied; adjacent policy correct, governing policy missed; surface quality concealing semantic violation — appeared at every layer, in every detection context, under favourable review conditions (experienced operator, specialised analytical frames). In a typical review context — less operator familiarity, fewer analytical perspectives, higher volume pressure — these defects enter the codebase as locally reasonable code that passes all automated checks. The quality of the outcome depended entirely on someone knowing which questions to ask.

Appendix F: Cross-Model Defect Chaining as an Emerging Second-Order Risk

This appendix outlines a plausible second-order systemic risk arising from the most obvious mitigation to model monoculture: the use of multiple models. The concern is not that diversity is ineffective, but that it may reduce common-mode failure while preserving systemic exposure in altered form.

Scope and intent. This appendix is a practitioner observation, not a research contribution. Correlated failure in populations with shared lineage is a well-understood phenomenon in other domains — agricultural monoculture, genetic inbreeding, financial contagion through shared counterparty exposure, common-mode failure in redundant engineering systems. This appendix says: the same structural conditions appear to exist in the coding-model ecosystem, and the same class of risk may apply. It does not attempt to formalise the interaction model, measure the effect, or characterise specific failure chains — that work requires empirical study by researchers with access to model internals, training lineage data, and controlled experimental environments that a practitioner threat model cannot provide. What follows identifies the mechanism, illustrates it with scenarios drawn from the paper's taxonomy, and states the policy implication. The research question — whether cross-model defect composition is a real phenomenon, and if so how to detect and measure it — is open.

F.1 Why Diversity May Not Buy Independence

Model diversity is the natural mitigation to monoculture risk (§2.4), but it should not be assumed to buy independence. Three layers of systemic exposure are worth separating: pure monoculture, overlapping training-distribution bias across different models, and cross-model defect chaining. The first is straightforward. The second is strongly plausible: even different models trained on similar public corpora may reproduce the same bad patterns, because those patterns represent the statistical majority of their training data. The third — one model's characteristic defect creating the conditions under which another model's distinct defect becomes dangerous — is the least established and the most speculative.

Fine-tuning multiple variants on local code may be directionally helpful, but fine-tuning is an adaptation of a base model, not an exorcism of its deep training-distribution habits.

Variant diversity can only decorrelate errors that originate in the portion of training that differs between variants. The dominant failure mode this paper describes — context-inappropriate defensive code on high-stakes paths — lives in the shared trunk, not the branches. For the risks discussed here, the relevant question is lineage independence, not variant count — and agencies should not assume that fine-tuned variants of the same base model provide meaningful independence.

F.2 The Chaining Mechanism

In a multi-model environment, distinct defect tendencies may not cancel out; they may interact. The scenarios below are illustrative mechanisms, not observed cross-model incidents.

Model A, used by one team, tends to omit or weaken validation boundaries on external data crossing authority tiers (ACF-T1). Model B, used by a different team or contractor on the same codebase, tends to add reassuring defaults and graceful error handling when fields are missing (ACF-S1). Separately, each is a recognised failure mode. Together, A weakens the authority-tier boundary and B ensures the resulting anomaly is normalised instead of surfaced — the composed outcome (silently authoritative unvalidated data) is worse than either defect alone. §2.4(g) develops this example in full.

Or consider a modernisation chain — particularly plausible in contracted development where different vendors may use different models on the same codebase (§6.7). Model A removes accidental fail-closed rigidity from legacy code (§1.2.6). Model B later adds "resilience" handling on the newly live edge path. Model C writes passing tests around the softened behaviour (ACF-R3). The end state is not a single shared defect but a system in which several model-specific behaviours have jointly erased an old safety property — and each step passed review independently.

F.3 Lineage Concentration

The agricultural monoculture analogy remains useful, but the agentic case is more complex than single-strain uniformity. Because frontier-scale coding models remain expensive to create, the effective lineage pool is likely to remain small even as the product pool expands. Fine-tuning, distillation, and derivative deployments may widen behavioural diversity, but they do not guarantee independence because descendants inherit training-distribution priors from their base model. The resulting ecosystem is better modelled as a small number of influential training lineages with many descendants than as a large population of genuinely independent systems. In practice, this means that a market with many products may still represent only a small number of genuinely independent coding-model lineages.

F.4 Policy Implication

The implication is narrow but important. Agencies conducting cross-agency scanning and remediation should not look only for repeated identical defects from a single model. They should also watch for co-occurring defect chains — patterns from different models that are individually unremarkable but jointly collapse a trust boundary. "Different agencies use different models, so we are safe" is not a defensible position if the models' distinct failure tendencies are composable.

This paper does not attempt to formalise the interaction model or characterise specific cross-model chains. The claim is not that cross-model defect chaining is established, but that model diversity should not be treated as a sufficient independence argument without evidence.

F.5 What This Appendix Does Not Provide — and Who Should

This appendix identifies a candidate risk by analogy to well-understood correlated-failure phenomena in other domains. It does not provide:

  • Empirical measurement of whether cross-model defect composition occurs in practice, or at what rate
  • A lineage independence framework — a methodology for assessing whether two coding models have meaningfully independent training lineages, or how much independence is "enough"
  • Detection criteria for cross-model defect chains as distinct from same-model correlated defects
  • A formal interaction model specifying the conditions under which defects from different models compose

These are research questions. They require access to model training data, controlled experiments with known model lineages, and statistical methodology for characterising failure correlation across model families — none of which a practitioner threat model can supply. The observation that coding-model ecosystems exhibit the structural preconditions for correlated failure (small effective lineage pool, shared training distributions, derivative models inheriting base-model priors) comes from the same place as any practitioner's pattern recognition: sustained exposure to a domain's failure modes, combined with familiarity with how similar structural conditions have played out elsewhere.

The positions advanced in this paper that reference cross-model risk do not depend on Appendix F being confirmed. The case for international coordination rests on the shared-problem argument — every country using these tools faces the same failure modes, and the training corpus feedback loop (§2.5) means no national framework can address the upstream bias alone. The case for treating correlated failure as an assessment concern, and for scanning for patterns rather than instances, is justified by the single-model evidence in the case studies. Appendix F adds a further reason to take those positions seriously, but they stand without it.

Appendix G: A Systems Thinking Primer for This Paper's Arguments

This appendix provides a brief introduction to the systems thinking concepts that underpin the paper's analysis. The paper uses systems-theoretic reasoning throughout — feedback loops, archetypes, stock-flow dynamics, and leverage point analysis — but does not assume the reader is familiar with these frameworks. Readers who already work in systems dynamics or safety engineering may skip this appendix. Readers coming from a security engineering or policy background will find it useful as an interpretive guide to the analytical structure of the preceding sections.

G.1 Why systems thinking, not just security analysis

Security engineering excels at analysing threats with an identifiable adversary. The threat described in this paper does not have one (§2.2). The agent is not attacking the system. The reviewer is not negligent. Each component is working as designed. The failure emerges from the interaction between components — a capable-but-context-blind generator, a volume-constrained review process, and institutional pressure to ship — not from any component being individually broken.

This class of failure — emergent, non-adversarial, arising from system structure rather than component defect — is the domain of systems thinking and safety engineering (Leveson 2011; Meadows 2008; Senge 1990).89 The paper's recommendations lean toward the safety engineering side because that is where the threat structure sits: the control law metaphor (§1.3, §8.6) borrows from fly-by-wire aviation's degradation modes, the validation boundary (§5.3) functions as an interlock rather than an access gate, and the emphasis on environmental controls over behavioural controls (§7) follows the safety engineering principle that the safe path should be the easy path. These are safety controls applied to a safety problem that happens to live inside a security governance framework.

The concepts below are adapted to the agentic code context. They are the analytical machinery behind the paper's arguments about why some interventions work and others do not.

G.2 Feedback loops

A feedback loop exists when the output of a process influences its own future input. Two kinds matter for this paper:

Reinforcing loops (R) amplify a trend — they make things grow or decline faster. Left unchecked, they produce exponential behaviour.

Balancing loops (B) resist a trend — they push a system toward equilibrium or a target. They are the mechanism behind every control, every review process, every governance gate.

The paper's core dynamic is a reinforcing loop that degrades review quality:

graph TD
    A["Agent produces<br/>high-surface-quality code"] --> B["Code consistently passes<br/>tests, follows conventions"]
    B --> C["Reviewer develops trust<br/>in agent output (§4.2)"]
    C --> D["Review shifts from<br/>verification to scanning"]
    D --> E["Semantic defects pass<br/>review undetected"]
    E --> F["No incidents observed<br/>(defects are silent — §2.2)"]
    F -->|"reinforces trust<br/>in agent output"| A

    style A fill:#f9f,stroke:#333,stroke-width:2px
    style F fill:#fbb,stroke:#333,stroke-width:2px

Diagram description (accessibility): A circular reinforcing loop (R1) with six stages. Agent produces high-surface-quality code, which consistently passes tests and follows conventions, which causes reviewers to develop trust in agent output, which shifts review from verification to scanning, which allows semantic defects to pass review undetected, which produces no observed incidents (because the defects are silent), which reinforces trust in agent output — completing the loop. Each cycle weakens review further.

This is a reinforcing loop (R1) — each cycle makes the next cycle worse. The absence of observable incidents is not evidence of safety; it is the mechanism by which the loop sustains itself. Automated semantic boundary enforcement (§7.2) breaks the loop by inserting a check between "code passes tests" and "reviewer develops trust"; provenance tracking (§7.1) breaks it by making the trust distinction explicit.

The corresponding balancing loop — the one the paper argues must be strengthened — is the enforcement boundary:

graph TD
    V["Agent produces code<br/>with semantic violation"] --> G["CI gate detects<br/>violation (§7.2, §8.3)"]
    G --> R["Code is rejected before<br/>entering repository"]
    R --> N["Violation never reaches<br/>human review"]
    N --> P["Review process is not degraded<br/>by undetectable defects"]

    style G fill:#bfb,stroke:#333,stroke-width:2px
    style R fill:#bfb,stroke:#333,stroke-width:2px

Diagram description (accessibility): A linear flow showing the balancing mechanism (B1). Agent produces code with a semantic violation, CI gate detects violation, code is rejected before entering the repository, the violation never reaches human review, and the review process is not degraded by undetectable defects. This is the enforcement boundary breaking the reinforcing loop.

This is a balancing loop (B1) — it counteracts the reinforcing loop by catching violations before they can contribute to the habituation effect. The case study (§8.3) reports violations caught at a non-trivial daily rate through this mechanism.

G.3 The "Shifting the Burden" archetype

§4.2 references the "Shifting the Burden" systems archetype (Senge 1990; Meadows 2008). This subsection explains what an archetype is and why this one matters.

Systems archetypes are recurring configurations of feedback loops that produce characteristic behaviour regardless of the specific actors or technologies involved. Recognising an archetype is useful because the interventions that work (and do not work) are already known from other domains.

"Shifting the Burden" describes a system with two responses to a problem: a symptomatic solution that is fast, visible, and immediately effective, and a fundamental solution that is slow, difficult, and addresses the root cause. Over time, the symptomatic solution's success weakens commitment to the fundamental solution — because the problem appears to be managed. The fundamental solution atrophies. When the symptomatic solution eventually fails or is removed, the system is worse off than before because the fundamental capability has degraded.

In the agentic context:

Element In the archetype In this paper
Problem A recurring difficulty Semantic defects in code entering the repository
Symptomatic solution Fast, visible, immediately effective Treating surface-quality signals (tests pass, conventions followed, code looks correct) as a proxy for semantic correctness
Fundamental solution Slow, difficult, addresses root cause Thorough human review capable of evaluating semantic correctness in institutional context
Side effect Symptomatic solution weakens fundamental solution Consistent surface quality produces habituation (§4.2) — reviewers shift from verification to scanning
Collapse Fundamental solution has atrophied when needed When a semantic violation arrives, the review process has been degraded by the very consistency that made it appear unnecessary

The archetype explains why "review harder" is not a viable response — it asks the fundamental solution to reassert itself after the symptomatic solution has spent months weakening it. The viable responses are structural: either strengthen the fundamental solution with tools that do not habituate (automated semantic enforcement — §7.2), or change the information flows so that the symptomatic solution's side effects become visible (provenance tracking, review quality metrics — §7.1, §9.2).

G.4 Stock-flow dynamics

A stock is an accumulation — something that can be measured at a point in time. A flow is a rate — something measured over time. The relationship between stocks and flows is the source of most non-obvious system behaviour.

The paper's core stock-flow argument (§1.2.1) is:

graph LR
    IN["Generation velocity<br/>(INFLOW)"] -->|"agent-generated<br/>code"| STOCK[("Code awaiting<br/>review<br/><b>STOCK</b>")]
    STOCK -->|"reviewed<br/>code"| OUT["Review throughput<br/>(OUTFLOW)"]

    style STOCK fill:#ffd,stroke:#333,stroke-width:2px

Diagram description (accessibility): A stock-flow diagram showing generation velocity as the inflow into a stock of code awaiting review, with review throughput as the outflow. When the inflow exceeds the outflow, the stock grows — the review backlog accumulates.

When inflow exceeds outflow, the review backlog grows. Three interventions differ dramatically in leverage:

Intervention Stock-flow target Leverage Paper's assessment
Rate-limit agent output Reduce inflow Low — sacrifices the benefit to manage the risk Not recommended as primary control (§7.1)
Add more reviewers Increase outflow capacity Low-moderate — linear scaling, expensive, still subject to habituation Mentioned (§7.1) but not the primary recommendation
Automate semantic pre-screening Increase outflow processing rate without adding human capacity High — changes the structure of the outflow, not its quantity Core recommendation (§7.2)

The third is highest-leverage because it changes what kind of review the outflow requires, rather than adjusting a parameter within the existing structure. This is a general principle: interventions that change system structure are more durable than interventions that adjust parameters within an unchanged structure.

A subtler dynamic is the accumulation of latent defects — the "dormant-but-activatable" stockpile described in §3.3:

graph LR
    IN2["Undetected<br/>violations<br/>(INFLOW)"] -->|"silent<br/>accumulation"| STOCK2[("Latent semantic<br/>defects<br/><b>STOCK</b>")]
    STOCK2 -->|"upstream changes,<br/>schema drift,<br/>modernisation"| OUT2["Activation<br/>events<br/>(OUTFLOW)"]

    style STOCK2 fill:#fdd,stroke:#333,stroke-width:2px

Diagram description (accessibility): A stock-flow diagram showing undetected violations as the inflow into a stock of latent semantic defects, with activation events (upstream changes, schema drift, modernisation) as the outflow. The stock grows silently; the outflow is unpredictable and potentially correlated.

This stock grows silently and drains through activation events: upstream contract changes, schema drift, or integration changes that exercise previously dormant code paths. Unlike the review backlog, this stock is invisible until it drains — and once large, the activation events are unpredictable and potentially correlated. This is why the paper emphasises detection (reducing the inflow) over incident response (managing the outflow).

G.5 Levels of intervention: Meadows' leverage points

Donella Meadows (2008) identified twelve places to intervene in a system, ordered from lowest to highest leverage. Not all twelve are relevant to the agentic code problem, but the hierarchy explains why the paper's recommendations are not all equally important — and why the most powerful recommendations are often the least concrete.

The relevant levels, mapped to this paper's recommendations:

Meadows Level Description This paper's equivalent Where discussed
12. Parameters Adjusting numbers within an unchanged structure "Review harder," "add more test coverage" Dismissed in §4.2 — parameter changes cannot address structural problems
11. Buffer sizes Increasing the capacity of a stock "Add more reviewers" §7.1 — necessary but low-leverage
10. Stock-flow structure Changing how stocks and flows are physically connected CI gate architecture — inserting a new control point in the flow §7.2, §8.3 — high leverage, changes the physical structure
9. Delays Reducing the time between an action and its consequence Pre-commit enforcement (immediate feedback) vs post-review advisory (delayed feedback) §4.3, §7.2 — "enforcement at the boundary, not feedback over time"
8. Balancing feedback loops Strengthening the loops that correct deviation Automated semantic enforcement — the B1 loop in §G.2 §7.2 — core technical recommendation
7. Reinforcing feedback loops Weakening the loops that amplify deviation Breaking the habituation loop (R1 in §G.2) through provenance tracking and review quality metrics §7.1, §9.2 — makes the degradation visible
6. Information flows Changing who has access to what information, and when Provenance tracking, control law visibility, review quality metrics §7.1, §8.6 — high leverage, low cost
5. Rules Changing the formal rules that govern the system ISM extensions, contract clauses, accreditation criteria §6.1, §6.7 — moderate leverage
4. Power to change structure Who can change the rules Addressing the analysis to ASD/ACSC (the bodies with cross-government mandate) The paper is requesting a level-4 intervention
3. Goals Redefining what the system is trying to achieve Reframe from "does the code work?" to "is the code correct for this institutional context?" The paper's deepest argument — §2.2, §5, §7
2. Paradigm The mindset out of which the goals and rules arise "Agent output is untrusted input" (§5.2) A paradigm shift, not a parameter adjustment

Three observations follow from this mapping:

First, the reclassification of agent output carries much of the paper's conceptual weight — treat agent output as untrusted input. This is a paradigm-level intervention (level 2). Once the paradigm shifts, the lower-level interventions (ISM extensions, CI gates, provenance tracking) follow naturally. Without it, those interventions are disconnected rules that organisations comply with reluctantly.

Second, the emphasis on technical controls over behavioural controls (§7) reflects the leverage hierarchy, not arbitrary preference. Behavioural controls operate at level 12 (parameters). Procedural controls operate at level 5 (rules). Technical controls operate at levels 8–10 (feedback loops, delays, stock-flow structure). The leverage difference is qualitative, not linear — a level-10 intervention reshapes the system; a level-12 intervention adjusts a dial within the existing shape.

Third, several of the highest-leverage recommendations are also the lowest-cost. Information flow interventions (level 6) — provenance tracking, control law visibility, review quality metrics — require modest engineering investment but fundamentally change the system's ability to detect its own degradation.

G.6 Connecting the frameworks

These are four views of the same system:

  • The safety engineering frame tells you what kind of problem this is: emergent, non-adversarial, arising from interactions between components each working as designed.
  • The feedback loops tell you why the problem self-reinforces: the habituation loop degrades the very control that should catch the failures.
  • The stock-flow dynamics tell you where the risk accumulates: in the review backlog (visible) and in the latent defect stockpile (invisible until activated).
  • The leverage points tell you where to intervene: not at the parameter level but at the structural and paradigm levels.

Together, they explain the paper's central strategic claim: the response is not intensified ordinary review but a structural change in what the review system checks for, supported by a paradigm shift in how agent output is classified.

Appendix H: Glossary

Full definitions for terms used throughout this paper. §1.3 provides a compact forward-reference table; this appendix provides the complete definitions with cross-references to where each term is introduced and developed.

Term Definition
High-stakes code paths Code paths where silent corruption, unverifiable state, or fabricated defaults are more dangerous than controlled failure. Defined by code-path properties, not by system importance. The defining characteristic is that fail-soft behaviour is more dangerous than fail-fast behaviour on these paths — the opposite of most software. A single system can contain both: a dispatch system is high-stakes on its audit paths but correctly uses defensive programming on its service-continuity paths. Not synonymous with "important software" — even otherwise low-stakes systems contain narrow high-stakes paths (authentication, authorisation, payment handling, audit logging). §1.1 provides the full definition and scope.
Agent An AI system (typically an LLM) that generates, modifies, or reviews source code with limited or no human intervention per output. This paper focuses on autonomous and semi-autonomous agents that operate across multiple files and decisions (e.g., building a feature end-to-end), not the inline suggestion mode of code completion tools. While both introduce volume, agents produce correlated errors across a module or feature, whereas autocomplete errors are typically isolated to individual expressions.
Agentic code Source code generated or substantially modified by an agent.
Annotation groups Companion specification term. The 17 categories of institutional knowledge a semantic boundary declaration can declare. Core classification: Groups 1–4 and 16–17. Supplementary contract: Groups 5–15 (companion specification, §6).
Autocomplete Inline code suggestion tools that complete individual lines or expressions within a human-directed editing session. Some product families now span both inline suggestion mode and more autonomous agentic workflows; this term refers specifically to the inline mode, not to agentic operation. Distinct from agents in that the human maintains architectural control and errors are largely uncorrelated (though IDE-integrated completions may replicate local patterns within a file — see Appendix B, Level 2).
Agent deployment spectrum The range from full human development (Level 0) through chat-pasted fragments (Level 1) and IDE-integrated agents (Level 2) to autonomous agents with human review before merge (Level 3). The threat model shifts qualitatively at each stage; controls should be proportionate. This paper's threat model applies primarily to Level 2–3 — agents operating with enough autonomy to produce correlated changes across a module or feature. Appendix B provides a self-assessment framework.
Trust boundary A point in a system where data crosses between different authority tiers (e.g., external input entering internal processing). Refers to the boundary itself — the crossing point, not the classification level on either side.
Authority tier A classification of data based on what guarantees the system is entitled to assume about it. Four tiers: Tier 1 (authoritative internal — trusted assertion), Tier 2 (semantically validated), Tier 3 (shape-validated), Tier 4 (unvalidated external — raw observation). The tier reflects the system's epistemic entitlement, not the data's apparent correctness — data from an external API is Tier 4 regardless of quality. Serialisation boundaries reset trust: system-produced data re-entering through a read interface is handled as Tier 4 until restoration controls are applied. Formally introduced in §5; specified in full in the companion specification, §4.
Validation boundary The specific mechanism (code, process, or tool) that enforces a trust boundary. For runtime data: the control that data must pass through to cross from a lower to a higher authority tier. For agent-generated code: a layered combination of conventional automated checks (linters, type checkers, tests), semantic enforcement (purpose-built checks for authority-tier flow, failure behaviour, audit preservation), and human review that code must pass through before entering the codebase. The middle layer — semantic enforcement — does not yet exist as a standard control category; building it is the core technical recommendation of this paper (see §5.3).
Institutional knowledge / domain knowledge The security-relevant distinctions a system depends on but that are not expressed in the programming language or standard tooling — which paths must fail fast, which data is authoritative, where trust boundaries lie, what absence means in each field. In this paper the terms are used together because both describe knowledge external to the code: "domain knowledge" emphasises the semantics of the problem space; "institutional knowledge" emphasises how those semantics are concretised in organisational rules, controls, and operating practice.
Assurance stack The standard set of automated and human checks that code passes through before integration: linters, type checkers, SAST, DAST, unit tests, and conventional peer review. Used in this paper to describe the dominant verification pipeline — both automated tools and human review — that are not designed to detect the semantic failure modes in Appendix A.
Defensive anti-pattern Defensive programming patterns (.get() with defaults, broad exception handling, graceful degradation) applied in contexts where they are inappropriate. These patterns are genuinely good practice in most software — and even there, they routinely conceal bugs by silently substituting defaults for missing or corrupt data. In high-stakes contexts (high integrity, auditability, availability, or reliability), the same concealment is catastrophic: silent data corruption is worse than a crash. Also referred to as "defensive programming" or "defensive patterns" throughout. In SQL contexts, the equivalent pattern is COALESCE() with fabricated defaults (Appendix C).
Bidirectional authority collapse The mechanism by which uniform defensive patterns collapse the authority tier model from both ends simultaneously: unvalidated external data is given more authority than it has earned (defaults and coercion allow it to cross inward as though validated), while authoritative internal data is treated as more negotiable than it is allowed to be (the same patterns handle corruption of authoritative records as routine and recoverable). Introduced in §2.2; formalised under the authority-tier model in §5.1.
Review-surface generation velocity The rate at which an agent can produce plausible, syntactically valid, convention-conforming code that arrives at a human review boundary. Distinct from productivity uplift — measures review burden, not developer output (§1.2.1).
Restoration boundary Companion specification term. A declared function that performs the governed restoration act by which serialised representations of authoritative artefacts may be reconstituted with their original tier. Requires four cumulative categories of provenance evidence: structural, semantic, integrity, and provenance-institutional (companion specification, §5.3).
Model monoculture The systemic risk created when multiple organisations use the same AI models for code generation. Like agricultural monoculture, shared models produce correlated failures across organisational boundaries — a vulnerability in one model's training data affects every agency using that model (§2.4(g)). Even diversifying across models does not eliminate systemic risk (see Cross-model defect chaining).
Polymorphic reviewers Prompted agents used as code reviewers that can adopt multiple analytical frames (security architect, systems thinker, quality engineer, etc.) within a single review. Not deeper than a specialist in any single frame, but able to provide breadth of analytical coverage that no individual human can match (§7.2).
Cognitive range The limitation on human review effectiveness arising not from time pressure but from the inability to practically adopt analytical frames from multiple specialisations simultaneously. A human reviewer cannot be a security architect, systems thinker, and quality engineer within a single review pass (§7.2).
Parasitic, not parallel A design principle for security enforcement tools: they must integrate into existing workflows without adding friction, because tools that slow down agent-assisted velocity will be bypassed. The tool must be parasitic on the existing process, not a parallel process that competes for attention (§7.2).
Pattern rules (WL-001–WL-008) Companion specification term. Eight language-agnostic rules that detect syntactic proxies for semantic violations in declared contexts. Six pattern rules (WL-001–WL-006) cover: fallback defaults, structural gates, broad exception catching, silent exception handling, audit writes in broad handlers, and runtime type-checking. Two structural verification rules (WL-007–WL-008) check code structure rather than matching a syntactic proxy. Severity depends on the taint state of the enclosing context (companion specification, §7).
Enforcement at the boundary The principle that agent-generated code requires enforcement before it enters the repository (pre-commit gates, CI checks), not feedback over time (post-review advisories). Agents do not learn from feedback — every session is the first session (§4.3).
Exceptionability Companion specification term. Governance classification determining whether a finding can be overridden. Four classes: UNCONDITIONAL (never overridable), STANDARD (overridable with rationale, reviewer identity, and expiry), RELAXED (lighter governance), and TRANSPARENT (pattern permitted by design in that context). Combined with severity in each cell of the severity matrix (companion specification, §7.3).
Fingerprint baseline Companion specification term. A persistent record of the application's annotation surface. Changes are flagged for human review, preventing silent erosion of the declared boundary (companion specification, §9.2).
Citizen programmer A non-developer (analyst, operator, business user) who produces executable logic — scripts, automations, BI extensions, SQL queries — using agentic tools outside traditional SDLC channels. Distinct from professional developers in that they typically lack security training, operate outside version control, and are not covered by existing code review or governance processes (§1.2.7; Appendix C).
Automation bias The tendency for human reviewers to accept agent-generated output uncritically because it appears competent and well-structured. Amplified in agentic development because the output genuinely is syntactically correct and convention-conforming — the bias is toward accepting plausible code, not toward accepting obviously wrong code (§4.2).
Habituation effect The progressive degradation of human review quality under sustained exposure to agent-generated code that consistently passes tests and follows conventions. The reviewer's mental model shifts from "verify this code is correct" (active defect search) to "check this code is not obviously wrong" (passive scan that catches only gross errors). A specific manifestation of automation bias, compounded by the volume pressure of agentic development. Follows the "Shifting the Burden" systems archetype: the agent's consistent surface quality weakens the fundamental solution (thorough review) by making it appear unnecessary (§4.2).
Context collapse during generation A session-level failure in which project-specific semantic distinctions that were available earlier in the interaction are lost, compressed, or displaced during generation. Distinct from training-distribution bias: context collapse is about failure to retain or apply context that was provided, while training-distribution bias is about the model's learned default patterns remaining wrong even when the context is present (§2.4).
Trust topology The specific arrangement of authority tiers, trust boundaries, and validation boundaries in a given system or module — which data flows cross which boundaries, and what validation is required at each crossing point. Each system's trust topology is unique; the authority tier model (§5) provides the vocabulary, but the topology is the project-specific instantiation.
Control law The operational state of a repository's machine-enforced controls — which protections are active right now, not how mature they are. Borrowed from fly-by-wire aviation. Three states: Normal law — all enforcement active and blocking. Alternate law (degraded) — some enforcement unavailable, compensating human vigilance required. Direct law — no machine enforcement active, all assurance depends on human review. Orthogonal to assurance level: a strong team can temporarily operate under degraded control law during a CI outage. Under direct law, high-risk changes should not proceed. The current control law must be visible to the whole team. §1.3, §8.6 develop the operational implications.
Compliance tax The overhead imposed by governance controls on development velocity. In agentic development, the compliance tax may be similar in magnitude to human development, but the distribution shifts: agents reliably pay governance costs that humans under deadline pressure quietly defer, while requiring different (machine-enforced) controls than humans need (§8.5).
Operator In the context of agent-assisted development, the human who directs, challenges, and evaluates an agent's work during a coding session. Distinguished from "reviewer" (who evaluates completed output) in that the operator interacts with the agent in real time and can redirect its reasoning (Appendix E).
Offensive / Confident / Guarded / Sceptical programming Per-tier coding postures declared by the authority tier model. Tier 1 (offensive): assume invariants, crash on anomaly — silent recovery converts integrity failures into undetected corruption. Tier 2 (confident): trust field values for domain operations; guard only against cross-cutting concerns (authorisation, concurrency, freshness, and state transitions) that value-level validation cannot address. Tier 3 (guarded): direct field access is safe; validate domain constraints before using values in business logic, arithmetic, or security-sensitive operations. Tier 4 (sceptical): treat everything as hostile — validate structure first, normalise, reject. The posture determines which programming patterns are appropriate in each tier context (companion specification, §4.1; see also this paper's §5, authority tier).
Semantic boundary enforcer A tool that enforces authority-tier rules at the code level by detecting semantic boundary violations. The case study (§8) describes a project-specific AST pattern-matching enforcer (a predecessor to the companion specification's reference scanner). The companion documents specify the classification framework (what a project declares about its trust boundaries) and a reference scanner architecture with three enforcement layers: AST pattern matching (level 1), variable-level taint tracking (level 2), and transitive call-graph inference via worklist-based fixed point (level 3). See Semantic boundary declaration for the distinction between classification and enforcement.
Severity matrix Companion specification term. The mapping of pattern rules against taint states that determines finding severity. An 8×8 matrix (8 rules × 8 taint states) producing 64 cells, each specifying severity (ERROR, WARNING, or SUPPRESS) and exceptionability. 39% of cells are UNCONDITIONAL — project invariants that cannot be overridden (companion specification, §7.3).
Taint state Companion specification term. The effective trust classification of a data value within the enforcement model. Eight states derived from two orthogonal dimensions (trust classification and validation status). Pattern rule severity depends on the taint state of the enclosing context (companion specification, §5.1, §7).
Machine-readable / machine-checkable / machine-enforceable Three levels of formalisation for institutional security knowledge, forming an escalation path from advisory to control. Machine-readable: rules in a structured format that tools can parse. Machine-checkable: rules that a tool can verify and report on. Machine-enforceable: rules integrated into a CI gate that blocks non-compliant code. Each level subsumes the previous. In agentic development, institutional knowledge that exists only in prose cannot shape agent behaviour and does not scale to agent-generated volume; knowledge that is machine-enforceable is a control (§7.2).
Defence in depth (applied to code integration) The principle that the validation boundary (§5.3) should consist of multiple independent layers — conventional checks, semantic enforcement, and human review — each catching a different class of failure, with no single layer sufficient on its own. Analogous to the defence-in-depth model familiar from network security (gateway, host, application, data) and embedded throughout the ISM's control framework. See Validation boundary for the semantic enforcement gap this paper identifies (§7.2).
Observability (for semantic properties) The design principle that if a system property matters, it must be measurable through existing instrumentation. The standard assurance stack provides observability for syntax, type safety, known vulnerability patterns, and functional correctness. It provides no observability for semantic correctness — whether code behaviour is appropriate for its institutional context. The semantic enforcement layer (§7.2) and the companion framework are fundamentally observability interventions: they make a previously invisible property of the codebase measurable, trackable, and auditable. Until a property is observable, governance operates blind on that dimension.
ASTRIDE An extension of the STRIDE threat modelling framework for AI systems, adding AI Agent-Specific Attacks as a threat category (Bandara et al. 2025). Referenced in §3.1 as precedent for extending STRIDE categories to agentic development process analysis.
STRIDE-LM An extension of the STRIDE threat modelling framework that adds Lateral Movement as a seventh category (Muckin and Fitch 2019). Referenced in §3.1 as precedent for extending STRIDE beyond its original six categories.
ALARP (as low as reasonably practicable) A risk management principle, familiar from safety engineering and risk assessment, that controls should be proportionate to the risk and that there is a cost floor below which action is required regardless of risk appetite. Applied in this paper to the validation maturity staging model (§7.2): Stage 1 is the ALARP floor — the minimum credible control posture for any organisation using agentic coding on assessed systems.
Fail-fast / fail-soft Two opposing failure strategies. Fail-fast (also "offensive programming" in this paper's Tier 1 posture): on detecting an anomaly, the system halts the operation immediately with maximum diagnostic context — a crash is preferable to silent corruption. Fail-soft (also "graceful degradation," "defensive programming"): on detecting an anomaly, the system substitutes a default, logs the error, and continues operating. Most software correctly uses fail-soft; high-stakes code paths (§1.1) require fail-fast. The paper's central claim is that agents uniformly apply fail-soft patterns — because that is what training data models as good practice — including on code paths where fail-fast is the correct and necessary behaviour (§2.2, §5.1).
Defensive programming A coding practice that anticipates and handles potential errors gracefully — checking for None, providing default values, catching exceptions. Genuinely good practice in the majority of software. Becomes a defensive anti-pattern when applied in high-stakes contexts where the same error-concealment mechanism silently corrupts authoritative data or destroys audit trails. The distinction is context-dependent, not pattern-dependent (§2.2).
Plausible-but-wrong Code that is syntactically valid, convention-conforming, test-passing, and review-approvable — yet semantically incorrect for its institutional context. The term is used throughout this paper to describe the characteristic output shape of agentic code on high-stakes paths: code that would be approved in ordinary review because it follows the patterns reviewers are trained to approve, while violating trust boundaries, fabricating defaults, or destroying audit trails that the code's context requires it to preserve. The "plausible" is load-bearing — the danger is not that the code is subtly wrong despite looking suspicious, but that it is wrong despite looking correct (§2.2, §2.3).
Golden corpus A curated set of known-good and known-bad code samples used to verify the correctness of a semantic boundary enforcement tool. Provides a regression baseline: every sample has a known expected classification, and the tool must correctly classify all of them. One of the six verification properties for enforcement tooling (§7.2); formally specified in the companion specification (§10).
Semantic correctness Whether code behaviour is appropriate for its institutional context — not just syntactically valid, type-safe, and functionally correct, but correct for the specific system it operates in. The standard assurance stack verifies syntactic and structural correctness but provides no instrumentation for semantic correctness. The failures described in this paper (Appendix A) are semantically incorrect code that is syntactically, structurally, and functionally correct. Building observability for semantic correctness is the core technical recommendation (§7.2).
Type Annotation Erosion (ACF-S4) Provisional taxonomy entry. A meta-failure in which type annotations are weakened or suppressed — # type: ignore comments added, typed parameters widened to Any, or cast() calls inserted — to resolve type errors rather than fixing the underlying type mismatch. Degrades the detection substrate for other taxonomy entries, particularly ACF-S2. Complementary to ACF-S5 (Type Structure Avoidance): S4 erodes existing type safety, S5 prevents it from being constructed (Appendix A).
Type Structure Avoidance (ACF-S5) Provisional taxonomy entry. A meta-failure in which agent-generated code systematically uses untyped containers (dict, Map<String, Object>, Record<string, any>) where domain-specific types (dataclass, interface, schema model) would be appropriate. External and internal data are reduced to the same untyped representation, so downstream code cannot distinguish them at the type level — making ACF-T1 (authority tier conflation) harder to detect. Complementary to ACF-S4 (Type Annotation Erosion): S5 prevents type structure from being constructed, S4 erodes it after construction (Appendix A).
Training-distribution bias The model's learned default patterns, derived from training data that is overwhelmingly composed of open-source code without high-stakes properties. Distinct from context collapse during generation: training-distribution bias persists even when the correct context is provided in the prompt, because the model's priors encode defensive patterns as universally correct (§2.4, §2.5).
Validation maturity stages A three-stage progression describing how much semantic enforcement an organisation has in place. Stage 1 (ALARP floor): achievable with existing tooling and no specialist engineering — pattern-matching proxies, review checklists, provenance tracking. Stage 2: moderate engineering investment — custom semantic rules, CI-integrated enforcement, measured precision. Stage 3: comprehensive semantic enforcement with full severity matrix coverage, golden corpus, and measured recall. Orthogonal to the control-type hierarchy (behavioural, procedural, technical) — each stage uses all three control types at different depths (§7.2).
Semantic boundary declaration Companion specification term. The set of declarations an application makes about how it classifies and protects the semantic boundaries of its data and code paths — what the companion specification calls a "wardline." The declaration is the classification, not the enforcement tool — analogous to a security classification guide versus the systems that enforce it. An application that has declared its semantic boundaries has made its institutional knowledge machine-readable; an application without such declarations has that knowledge in prose, in people's heads, or nowhere (companion specification, §5–§6).
Semantic boundary manifest Companion specification term. The machine-readable declaration of an application's trust topology, rule configuration, and exception register. Language-neutral — polyglot applications declare a single boundary specification consumed by language-specific enforcement tools (companion specification, §13).
ACF taxonomy (Agentic Code Failure) The classification framework for failure modes specific to or amplified by agentic code generation. Structured using the STRIDE threat categories as an organising principle: Spoofing (S1–S3, plus provisional S4–S5), Tampering (T1–T3, plus provisional T4), Repudiation (R1–R3 and R5, plus provisional R4 and R6), Information Disclosure (I1), Denial of Service (D1–D2), Elevation of Privilege (E1–E2). Fifteen core entries plus five provisional candidates. Each entry specifies whether the failure mode is agent-specific or a known class that agents amplify, a risk rating, detection approaches, and code examples. Defined in Appendix A; summary table in §3.
Bounded context An architectural property of current agentic systems: the agent operates within a finite context window that constrains how much of a project's codebase, history, and institutional knowledge it can hold simultaneously. As sessions grow, earlier material is compressed or displaced, creating the conditions for context collapse during generation (§2.4). One of three structural properties (alongside selective salience and compression of prior state) that make agentic failure modes a persistent condition rather than transient bugs.
Selective salience An architectural property of current agentic systems: the model's attention mechanism selectively weights portions of the available context, meaning that institutional knowledge present in the prompt may be effectively invisible during generation if other material is more salient. This is not a failure to provide context but a failure to attend to it — the instructions are there, but the training-distribution prior outweighs them. One of three structural properties (alongside bounded context and compression of prior state) that make agentic failure modes a persistent condition rather than transient bugs (§2.4, §9).
Compression of prior state An architectural property of current agentic systems: when context is compacted or summarised (whether by the model's context management or by explicit compaction), the compressed representation loses semantic distinctions that were present in the original. Institutional knowledge that was available early in a session may be reduced to a summary that no longer carries the distinctions the code requires. One of three structural properties (alongside bounded context and selective salience) that make agentic failure modes a persistent condition rather than transient bugs (§2.4, §9).
Task-frame reconstruction The mechanism by which an agent under context pressure reconstructs a coherent narrative about what it is doing — and the reconstructed narrative may not match the original plan. The practical signature: tests written to verify real behaviour are "fixed" by replacing real dependencies with mocks that return expected values. The agent's task frame shifts from "implement and test this integration" to "make this test pass," and in the new frame, the mock is the implementation. Distinct from spurious field access (ACF-S2) in that the agent's model of the code is internally consistent — it is the model of the task that has shifted. The observable artefact pattern is catalogued as ACF-R3: Verification Displacement (§2.4(f)).
Compounding effect The interaction between the six STRIDE-mapped threat categories in which they amplify each other rather than operating independently — code-level failures pass review because of process-level capacity exhaustion, producing a combinatorial surface of latent defects that is unmanageable at scale (§3.3).
Correlated failure The systemic risk arising when the same model or model lineage generates code across multiple modules, features, or organisations. Unlike human errors (which are largely independent), agent errors are correlated — a model's training-distribution bias produces the same defect pattern wherever it operates. Remediation of a single instance should trigger codebase-wide scanning for the same pattern. Triage models must treat 50 instances of the same rule as one systematic issue, not 50 independent tickets (§2.4(g), §9.4).
Cross-model defect chaining The second-order systemic risk in which genuinely different models produce composable failures — one model's characteristic error creates conditions under which another model's distinct error becomes dangerous. The models need not share the same defect; their defects only need to be composable. An emerging precautionary analysis, not yet empirically demonstrated (Appendix F).
Lineage concentration The observation that the effective number of independent model lineages is much smaller than the number of available models. Fine-tuning, distillation, and derivative deployments widen behavioural diversity but do not guarantee independence, because descendants retain important priors from their model lineage. The resulting ecosystem is better modelled as a small number of influential training lineages with many descendants than as a large population of genuinely independent systems (Appendix F, §F.3).
Closed verification loop A testing failure mode in which a single agent session produces both implementation and test suite, and the tests inherit the same context frame — the verification is circular, confirming the implementation does what it does rather than what the system requires (§9.9).
Coverage illusion The false confidence created when agent-generated test suites achieve high line and branch coverage while verifying the wrong properties. Coverage measures which code paths are executed, not which semantic properties are verified — high coverage of the wrong dimension (§9.9).
Provenance tracking The practice of recording which code was generated by AI agents versus authored by humans. No current ISM control requires per-artefact provenance — ISM-2074 requires an AI usage policy but not tracking at the code level. Without provenance, risk assessment cannot distinguish between code populations with different failure characteristics, and remediation scanning cannot target agent-generated code specifically (§6.1.2).
Enforcement regime Companion specification term. The set of tools that collectively enforce a semantic boundary declaration for a given language ecosystem. A single tool may satisfy one or more conformance profiles; the regime is the composition (companion specification, §14.4).
Governance perimeter The boundary of code that falls within an organisation's governed software development lifecycle — version control, code review, CI/CD gates, and assurance processes. The citizen programmer phenomenon (§1.2.7) expands the volume of executable logic produced within organisations while much of it falls outside this perimeter, creating a dual problem: volume pressure inside the SDLC and perimeter expansion around it. Together, these are materially worse than either alone (§1.2.7).

References

  • Australian Signals Directorate. Information Security Manual. Commonwealth of Australia. December 2025 revision. https://www.cyber.gov.au/resources-business-and-government/essential-cyber-security/ism — Controls referenced: ISM-0401 (Secure by Design), ISM-0402 (SAST/DAST/SCA), ISM-1419 (development environments), ISM-2026/2027/2028 (software artefact integrity), ISM-2060 (code review), ISM-2061 (security-focussed peer review), ISM-2074 (AI usage policy). Individual controls are searchable by number on the ASD website.
  • Australian Signals Directorate. Essential Eight Maturity Model. Commonwealth of Australia. Updated periodically. https://www.cyber.gov.au/resources-business-and-government/essential-cyber-security/essential-eight — (Gap analysis in §6.3)
  • NIST. SP 800-218: Secure Software Development Framework (SSDF), Version 1.1. February 2022. https://csrc.nist.gov/pubs/sp/800-218/final — (Gap analysis in §6.2)
  • NIST. SP 800-218A: Secure Software Development Practices for Generative AI and Dual-Use Foundation Models. 26 July 2024. https://csrc.nist.gov/pubs/sp/800-218a/final — (AI-specific SSDF supplement referenced in §6.2)
  • Microsoft. The STRIDE Threat Model. Microsoft Security Development Lifecycle. Last updated 25 August 2022. https://learn.microsoft.com/en-us/azure/security/develop/threat-modeling-tool-threats — (Framework basis for §3)
  • OASIS. Static Analysis Results Interchange Format (SARIF), Version 2.1.0. March 2020. https://docs.oasis-open.org/sarif/sarif/v2.1.0/sarif-v2.1.0.html — (Output format specified in the companion documents)
  • Morrissey, J. Wardline Framework Specification: Semantic Boundary Classification and Enforcement. Companion document to Semantic Defects in AI-Generated Code. Draft v0.2.0, 2026. Distributed with this document suite. — (Language-agnostic specification: authority tier model, pattern rules, severity matrix, governance model, conformance criteria. Referenced throughout §5, §7.2, §8)
  • Morrissey, J. Wardline for Python: Language Binding and Enforcement Design. Companion document to Semantic Defects in AI-Generated Code. Draft v0.2.0, 2026. Distributed with this document suite. — (Python-specific binding: decorator vocabulary, AST scanner design, type system enforcement, runtime structural enforcement. Referenced in §7.2, §8, Appendix A, Appendix E)
  • Morrissey, J. Wardline for Java: Language Binding and Enforcement Design. Companion document to Semantic Defects in AI-Generated Code. Draft v0.2.0, 2026. Distributed with this document suite. — (Java-specific binding: annotation vocabulary, enforcement design. Referenced in §7.2)
  • OWASP. Top 10 for Large Language Model Applications, Version 2025. 2025. https://owasp.org/www-project-top-10-for-large-language-model-applications/ — (Gap analysis in §6.4)
  • OWASP. Secure Coding Practices — Quick Reference Guide. 2010. https://owasp.org/www-project-secure-coding-practices-quick-reference-guide/ — (Gap analysis in §6.4)
  • MITRE. Common Weakness Enumeration (CWE). Version 4.19.1. The MITRE Corporation. https://cwe.mitre.org/ — (Taxonomy comparison in §6.4)
  • MITRE. ATT&CK: Adversarial Tactics, Techniques, and Common Knowledge. The MITRE Corporation. Accessed March 2026. https://attack.mitre.org/ — (Taxonomy comparison in §6.4)
  • McConnell, S. Code Complete: A Practical Handbook of Software Construction. 2nd ed. Microsoft Press, 2004. (Industry defect density data: 15–50 defects per KLOC — §4.1)
  • Senge, P. The Fifth Discipline: The Art and Practice of the Learning Organization. Doubleday/Currency, 1990. (Origin of the "Shifting the Burden" systems archetype — §4.2, Appendix G §G.3)
  • Meadows, D. Thinking in Systems: A Primer. Chelsea Green Publishing, 2008. (Systems dynamics foundations referenced in §4.2, Appendix G §G.4–G.5)
  • Parasuraman, R. and Manzey, D. "Complacency and Bias in Human Use of Automation: An Attentional Integration." Human Factors, 52(3), 381–410, 2010. (Automation bias referenced in §4.2)
  • Perry, N., Srivastava, M., Kumar, D. and Boneh, D. "Do Users Write More Insecure Code with AI Assistants?" ACM CCS, 2023. https://doi.org/10.1145/3576915.3623157 — (AI-assisted developers wrote less secure code while feeling more confident — §4.2)
  • Peng, S. et al. "The Impact of AI on Developer Productivity: Evidence from GitHub Copilot." arXiv preprint, 2023. https://doi.org/10.48550/arXiv.2302.06590 — (Preprint, not peer-reviewed. Controlled study: 55.8% faster task completion — §1.2.1)
  • Cui, Z.K., Demirer, M., Jaffe, S., Musolff, L., Peng, S. and Salz, T. "The Effects of Generative AI on High Skilled Work: Evidence from Three Field Experiments with Software Developers." Management Science, 2025. https://doi.org/10.1287/mnsc.2025.00535; SSRN: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4945566 — (Three field experiments across Microsoft, Accenture, and an anonymous Fortune 100 company; 4,867 developers; pooled 26.08% increase in completed tasks — §1.2.1)
  • Bandara, E. et al. "ASTRIDE: A Security Threat Modeling Platform for Agentic-AI Applications." arXiv preprint, arXiv:2512.04785, December 2025. (Preprint, not peer-reviewed.) https://arxiv.org/abs/2512.04785 — (STRIDE extension for AI agent-specific attacks, referenced in §3.1)
  • Muckin, M. and Fitch, S.C. "A Threat-Driven Approach to Cyber Security." Lockheed Martin Corporation, 2019. https://www.lockheedmartin.com/content/dam/lockheed-martin/rms/documents/cyber/LM-White-Paper-Threat-Driven-Approach.pdf — (Introduces STRIDE-LM: STRIDE extension adding Lateral Movement, referenced in §3.1)
  • METR (Model Evaluation and Threat Research). "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity." July 2025. https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/ — (Blog post, not peer-reviewed. RCT: experienced developers predicted 24% faster, believed 20% faster, measured 19% slower — §1.2.1, §4.2)
  • Pichai, S. Alphabet Inc. Q3 2024 Earnings Call transcript, 29 October 2024. https://abc.xyz/investor/events/event-details/2024/2024-q3-earnings-call/ — ("More than a quarter of all new code at Google is generated by AI, then reviewed and accepted by engineers." Referenced in §1.2.1)
  • Wolf, A. "Welcome to the Eternal September of open source. Here's what we plan to do for maintainers." GitHub Blog, 12 February 2026 (updated 13 February 2026). https://github.blog/open-source/maintainers/welcome-to-the-eternal-september-of-open-source-heres-what-we-plan-to-do-for-maintainers/ — Primary source. GitHub's Director of Open Source Programs on the review capacity crisis: "The cost to create has dropped but the cost to review has not." Announces PR access controls, interaction limits, and automated triage. (Referenced in §1.2.2)
  • Ghoshal, A. "GitHub eyes restrictions on pull requests to rein in AI-based code deluge on maintainers." InfoWorld, 4 February 2026. https://www.infoworld.com/article/4127156/github-eyes-restrictions-on-pull-requests-to-rein-in-ai-based-code-deluge-on-maintainers.html — Earlier reporting on events that GitHub later addressed formally in Wolf (12 February 2026, above). GitHub described the problem as a denial-of-service attack on human attention. (Review capacity exhaustion evidence referenced in §1.2.2)
  • Graham-Cumming, J. "Incident report on memory leak caused by Cloudflare parser bug." Cloudflare Blog, 23 February 2017. https://blog.cloudflare.com/incident-report-on-memory-leak-caused-by-cloudflare-parser-bug/ — (Buffer over-read, described as "memory leak" in the original post title. Precedent referenced in §1.2.3)
  • Ormandy, T. "Cloudflare Reverse Proxies Are Dumping Uninitialized Memory." Google Project Zero Issue 1139, 19 February 2017. https://bugs.chromium.org/p/project-zero/issues/detail?id=1139 — (Discovery of the Cloudbleed vulnerability referenced in §1.2.3)
  • OWASP. GenAI Security Project. 2024. https://genai.owasp.org/ — (Broader project encompassing LLM applications, agentic AI, and AI-driven applications — §6.4)
  • Python Software Foundation. "PEP 249 — Python Database API Specification v2.0." 1999 (revised 2013). https://peps.python.org/pep-0249/ — (DB-API default transaction semantics referenced in Appendix C)
  • Stenberg, D. "The end of the curl bug-bounty." daniel.haxx.se (curl project blog), 26 January 2026. https://daniel.haxx.se/blog/2026/01/26/the-end-of-the-curl-bug-bounty/ — (curl ended its bug bounty programme after AI-generated reports overwhelmed validation capacity — §1.2.2)
  • Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł. and Polosukhin, I. "Attention Is All You Need." Advances in Neural Information Processing Systems 30 (NeurIPS), 2017. https://arxiv.org/abs/1706.03762 — (Foundational transformer architecture paper. The context displacement dynamics discussed in §2.3 and §2.4(f) arise from the finite context window and positional encoding properties of the transformer architecture described in this work.)
  • Wang, J., Huang, Y., Chen, C., Liu, Z., Wang, S. and Wang, Q. "A Taxonomy of Bugs in LLM-Generated Code." Proceedings of the 47th IEEE/ACM International Conference on Software Engineering (ICSE), 2025. https://doi.org/10.1145/3696457.3715696 — (Two-dimensional taxonomy from 558 incorrect solutions across six LLMs. 78.5% of incorrect solutions were compilable and 58.3% were runnable — producing executable code that gives wrong results rather than failing visibly. Referenced in §1.6 as classifying code that fails detectably, complementary to this paper's focus on code that passes all checks.)
  • Gao, Z., Wen, M., Liang, Y., Wang, S. and Liu, Y. "A Comprehensive Survey of Bugs in Large Language Model Generated Code." arXiv preprint, arXiv:2503.16542, 2025. https://arxiv.org/abs/2503.16542 — (Preprint, not peer-reviewed. Comprehensive survey covering bug taxonomies, detection methods, and repair techniques for LLM-generated code. Their "semantic bug" category — code that is syntactically and structurally correct but behaviourally wrong — is the closest existing category to the ACF taxonomy's territory, but does not develop the institutional-context dimension. Referenced in §1.6.)
  • Lohn, A.J. and Jackson, K. "Will AI Make Cyber Swords or Shields? A Few Early Observations." Center for Security and Emerging Technology (CSET), Georgetown University, September 2024. https://cset.georgetown.edu/publication/will-ai-make-cyber-swords-or-shields/ — (Policy-level risk categorisation of AI-generated code: insecure code patterns, over-reliance on AI output, and supply chain concerns. Identifies broad risk categories without developing a structured failure taxonomy. Referenced in §1.6.)
  • Chen, J., Hu, J., Huang, J., Wang, J., Zhang, H. and Chen, T.H. "A Survey on Large Language Models for Code Security." arXiv preprint, arXiv:2412.15004, 2024. https://arxiv.org/abs/2412.15004 — (Preprint, not peer-reviewed. Systematic literature review covering 67 papers on security vulnerabilities, adversarial attacks, and defensive techniques across the code language model lifecycle. Referenced in §1.6.)
  • Open Source Security Foundation (OpenSSF). "Secure AI Coding Practices." OpenSSF Best Practices Working Group, 2025. https://best.openssf.org/Secure-AI-Coding-Practices — (Security-focussed system prompts, instruction templates, and workflow recommendations for AI code assistants. Addresses the generation-time control layer that is complementary to this paper's integration-boundary controls — see §1.6.)
  • Meyer, B. Object-Oriented Software Construction. 2nd ed. Prentice Hall, 1997. — (Origin of the Design by Contract methodology. The paper's concept of machine-enforceable semantic correctness properties — encoding what code must do in forms that tools can verify — has intellectual heritage in Meyer's preconditions, postconditions, and class invariants. Referenced implicitly throughout §7.2.)
  • Leveson, N.G. Engineering a Safer World: Systems Thinking Applied to Safety. MIT Press, 2011. https://direct.mit.edu/books/oa-monograph/2908/Engineering-a-Safer-World-Systems-Thinking-Applied — (Systems-theoretic approach to safety in complex sociotechnical systems. The paper's "control law" metaphor is borrowed from the fly-by-wire aviation domain that Leveson analyses extensively. Referenced in §1.3, §8.6, Appendix G §G.1. Open access.)
  • Geer, D., Bace, R., Gutmann, P., Metzger, P., Pfleeger, C.P., Quarterman, J.S. and Schneier, B. "CyberInsecurity: The Cost of Monopoly — How the Dominance of Microsoft's Products Poses a Risk to Security." Computer and Communications Industry Association, September 2003. https://ccianet.org/wp-content/uploads/2003/09/cyberinsecurity%20the%20cost%20of%20monopoly.pdf — (The foundational argument that software monoculture creates systemic correlated failure risk. The paper's model monoculture analysis in §2.4(g) and Appendix F extends this argument from operating system homogeneity to AI model training-lineage homogeneity.)
  • SKYbrary. "Flight Control Laws." SKYbrary Aviation Safety, EUROCONTROL. https://skybrary.aero/articles/flight-control-laws — (Reference for the Airbus A320 normal law / alternate law / direct law framework that the paper's "control law" terminology borrows — see §1.3, Appendix H.)

This is an independent discussion paper. It presents a threat model and preliminary analysis, not final guidance, and does not constitute official guidance or policy of any government body; the views and analysis are the author's own. Comments and contributions are welcome.

Suggested citation: Morrissey, J. "Semantic Defects in AI-Generated Code: Assurance Frameworks for AI-Assisted Development in High-Stakes Code Paths." Discussion Paper, Draft for Comment v0.1.0, 24 March 2026.

  1. ISM, NIST SSDF (including its AI supplement, SP 800-218A), and Essential Eight. ↩

  2. Whether a .get() default is institutionally appropriate, whether a failure is propagated to an authorised handler without loss of auditability, or whether data crossing a trust boundary has been validated. ↩

  3. Whether data crossing an authority tier was validated, whether a missing field was fabricated into a default, whether an audit-critical failure was swallowed, or whether fail-soft logic was applied to a fail-fast path. ↩

  4. The assurance ceiling varies by language — the companion specification's Python binding (Part II-A, §A.2) acknowledges that Python's lack of ownership semantics and compile-time enforcement provides a lower assurance ceiling than a language with those properties could achieve. ↩

  5. Current enforcement tooling operates at single-process scope — government systems with multiple interconnected services require supplementary governance controls at inter-service boundaries. ↩

  6. Not synonymous with "important software." A high-traffic consumer web app is important but not high-stakes in this sense. Even low-stakes systems contain narrow high-stakes paths (auth, audit, payments). See Appendix H for the full boundary definition. ↩

  7. Agents produce correlated errors across a module or feature; autocomplete errors are typically isolated to individual expressions. The distinction matters for the correlated failure argument in §2.4. ↩

  8. Chat-pasted fragments carry their own risks (principally ACF-S1 and ACF-S2) but lack the cross-cutting correlation that makes autonomous deployments dangerous at the architectural level. ↩

  9. The tier reflects the system's epistemic entitlement, not the data's apparent correctness. Data produced by the system's own controlled processes is Tier 1 at the semantic level; data from an external API is Tier 4 regardless of quality. The distinction between Tier 2 (semantically validated) and Tier 3 (shape-validated) captures whether data has passed domain-constraint checking or only structural validation. ↩

  10. A function that silently defaults a missing security classification to "OFFICIAL" is syntactically correct, type-safe, and functionally passing — but semantically incorrect because the system requires missing classifications to surface as integrity failures, not fabricated data (§2.3). ↩

  11. Genuinely good practice in most software. In high-stakes contexts the same concealment is catastrophic. ↩

  12. The Airbus A320 normal law / alternate law / direct law framework. See SKYbrary, "Flight Control Laws," EUROCONTROL. ↩

  13. Three states: Normal law — all enforcement active and blocking. Alternate law — some enforcement degraded, compensating human vigilance required. Direct law — no machine enforcement, all assurance depends on human review. Under direct law, high-risk changes should not proceed. ↩

  14. Task type (boilerplate generation vs. novel architecture), codebase maturity (greenfield vs. mature), compliance burden (see §8.5), and degree of human supervision. ↩

  15. Inconsistent input validation across similar endpoints, copy-paste errors that introduce subtle divergence between code paths that should be identical, omitted edge-case handling in repetitive boilerplate, and irregular code structure that makes audit and review harder. ↩

  16. Banking cores, defence and aerospace systems, legal and evidentiary platforms, and similar high-integrity transactional systems. ↩

  17. Peng et al. (2023), preprint, not peer-reviewed. The study measured completion time on a single bounded coding task using GitHub Copilot. ↩

  18. Cui et al. (2025). Three field experiments across 4,867 developers at Microsoft, Accenture, and an anonymous Fortune 100 company. ↩

  19. Sundar Pichai, Alphabet Inc. Q3 2024 earnings call, 29 October 2024. ↩

  20. METR (2025), blog post, not peer-reviewed. The perception-reality gap — pre-task prediction of +24% versus measured outcome of −19%, a span of 43 percentage points — has direct relevance to the automation bias argument in §4.2. ↩

  21. Wolf (2026); Ghoshal (2026). Wolf is the primary source (GitHub Blog, 12 February 2026); Ghoshal (InfoWorld, 4 February 2026) reported earlier on the same events. ↩

  22. Stenberg (2026). The curl project ended its bug bounty programme after AI-generated security reports overwhelmed validation capacity. ↩

  23. Graham-Cumming, J. (2017), "Incident report on memory leak caused by Cloudflare parser bug," Cloudflare Blog. The underlying vulnerability was discovered by Tavis Ormandy (Google Project Zero Issue 1139, 2017). Both sources are listed in full in the reference list. ↩

  24. The greatest exposure was concentrated in the final five days, after a feature migration dramatically increased the number of affected sites. ↩

  25. Cloudbleed exhibits the same two-property structure — combining defective boundary semantics (cf. ACF-S1) with swallowed error propagation (cf. ACF-R1) — but as a human-authored defect it predates the agentic context the taxonomy addresses. It is included as a precedent for the failure shape, not as an ACF instance. ↩

  26. Wang et al. (2025). The study analysed solutions from six LLMs across multiple programming tasks, classifying errors along two dimensions: error type and error manifestation. Notably, 78.5% of incorrect solutions were compilable and 58.3% were runnable — producing executable code that gives wrong results rather than failing visibly. ↩

  27. Gao et al. (2025). A comprehensive survey covering bug taxonomies, detection methods, and repair techniques for LLM-generated code. Their "semantic bug" category — code that is syntactically and structurally correct but behaviourally wrong — is the closest existing category to the ACF taxonomy's territory. ↩

  28. Lohn and Jackson (2024). Center for Security and Emerging Technology (CSET), Georgetown University. Identifies risk categories at the policy level — insecure code patterns, over-reliance on AI output, and supply chain concerns — without developing a structured failure taxonomy. ↩

  29. Chen et al. (2024). A systematic literature review covering security vulnerabilities, adversarial attacks, and defensive techniques across the code language model lifecycle. ↩

  30. Open Source Security Foundation (2025). Practical guidance for using AI code assistants securely, including security-focussed system prompts, prompt templates, and workflow recommendations. Addresses the generation-time control layer that is complementary to this paper's integration-boundary focus. ↩

  31. Linters check syntax; type checkers verify shape; SAST tools match known vulnerability patterns; tests verify behaviour against developer-supplied expectations. ↩

  32. No security classification requirements, no audit trail obligations, and no trust boundary enforcement. §2.5 develops the training-data root cause in detail. ↩

  33. Classified document handling, financial audit trails, and evidentiary records require zero latitude for corruption — the operation halts, and in safety-critical paths the process may need to stop entirely. UI preferences and display formatting correctly use defensive patterns. ↩

  34. This pattern is catalogued in Appendix A as ACF-S1 (Fabricated Default), with extended risk assessment, language-specific variants, and detection status. The three-version structure — correct, dangerous, and agent-generated — recurs throughout the ACF taxonomy. For the SQL equivalent of this pattern using COALESCE(), see Appendix C, §C.2. ↩

  35. Context displacement arises from the finite context window and positional encoding properties of transformer-based architectures (Vaswani et al. 2017, "Attention Is All You Need"); the specific displacement thresholds vary by model family, context length, and implementation. ↩

  36. Whether the shift reflects attention reallocation, distributional drift within the context, or some other mechanism is an open question; the observable consequence is the same. ↩

  37. See Appendix E §E.7 for the transcript evidence behind this characterisation. ↩

  38. The foundational argument that software monoculture creates systemic correlated failure risk was made by Geer et al. (2003), "CyberInsecurity: The Cost of Monopoly," in the context of operating system homogeneity. The agentic analogue extends the argument to AI model training-lineage homogeneity. ↩

  39. A validation boundary was omitted, authoritative data was fabricated by default, a test was displaced onto mocks, an error handler swallowed an audit event, an authority tier was collapsed. ↩

  40. Appendix A expands this STRIDE category into three distinct failure modes: ACF-S1 (Fabricated Default), ACF-S2 (Spurious Field Access), and ACF-S3 (Structural Identity Spoofing), plus provisional candidates ACF-S4 (Type Annotation Erosion) and ACF-S5 (Type Structure Avoidance). ↩

  41. Appendix A separates the Tampering category into ACF-T1 (Authority Tier Conflation), ACF-T2 (Silent Coercion), and ACF-T3 (Unstructured Signal Parsing), plus provisional candidate ACF-T4 (Safety Guard Erosion). ↩

  42. Appendix A separates the Repudiation category into ACF-R1 (Audit Trail Destruction), ACF-R2 (Partial Completion), ACF-R3 (Verification Displacement), and ACF-R5 (Remediation-Induced Violation), plus provisional candidates ACF-R4 (Context Handover Assumption) and ACF-R6 (Scope-Limited Triage). ↩

  43. Appendix A covers ACF-I1 (Verbose Error Response) as the core Information Disclosure entry. Stack trace exposure — a related pattern well-covered by existing SAST tooling — is not catalogued separately. ↩

  44. Appendix A separates the Denial of Service category into ACF-D1 (Finding Flood) and ACF-D2 (Review Capacity Exhaustion). They have different upstream drivers and different mitigation strategies, even though both degrade the same security control. ↩

  45. Appendix A separates the Elevation of Privilege category into ACF-E1 (Implicit Privilege Grant) and ACF-E2 (Unvalidated Delegation). ACF-E2's structural pattern also extends to the development process layer: agents inherit the operator's system credentials and execute privileged operations without the operator constraining scope — the same unvalidated delegation, expressed at the execution layer rather than the code layer. This process-layer dimension is addressed by graduated execution authority boundaries (§9.17). ↩

  46. Traditional STRIDE scope: An entity claims to be something it is not — a forged authentication token, a spoofed IP address, a process impersonating another user. The system accepts the false identity and grants access or trust accordingly. ↩

  47. Traditional STRIDE scope: An attacker modifies data in transit or at rest without authorisation — altering a database record, intercepting and changing a message, corrupting a configuration file. The data itself is changed. ↩

  48. Traditional STRIDE scope: A user performs an action — a transaction, an access, a deletion — and later denies it. The system lacks sufficient logging, signing, or non-repudiation controls to prove the action occurred. The gap is in the recording of events. ↩

  49. Traditional STRIDE scope: Sensitive data is exposed to unauthorised parties — through a misconfigured access control, a side-channel leak, an unprotected API endpoint, or data at rest without encryption. The exposure is typically of stored or transmitted data. ↩

  50. Traditional STRIDE scope: An attacker exhausts a system's resources — network bandwidth, CPU, memory, connection pools — rendering it unable to serve legitimate requests. The attack targets availability of a runtime service. ↩

  51. Traditional STRIDE scope: A user or process gains capabilities beyond what is authorised — exploiting a kernel vulnerability to move from user to root, leveraging a misconfigured role to access admin functions, or escaping a sandbox to reach the host system. The escalation is of identity or access rights. ↩

  52. For readers unfamiliar with threat modelling: the process has three steps — (1) describe the system (actors, data flows, trust boundaries); (2) enumerate what can go wrong systematically using a framework such as STRIDE; (3) decide what to do about each threat. STRIDE is not the only framework — PASTA, LINDDUN, and attack trees serve different analytical purposes — but it is the one most familiar to Australian government security assessments. ↩

  53. Muckin, M. and Fitch, S.C. (2019), "A Threat-Driven Approach to Cyber Security," Lockheed Martin Corporation. ↩

  54. Bandara, E. et al. (2025), "ASTRIDE: A Security Threat Modeling Platform for Agentic-AI Applications," arXiv:2512.04785. Preprint, not peer-reviewed. ↩

  55. The underlying problem is that standard linters encode community conventions while semantic enforcers encode institutional knowledge. When they conflict, the institutional knowledge should win — but agents have no basis for making that judgment, because both tools present their findings with equal authority. A machine-readable precedence declaration (e.g., "tier-model rules override ruff SIM401 in authority-tier contexts") would allow agents to resolve conflicts correctly without human intervention. ↩

  56. The dynamics have a close parallel in diagnostic imaging, where AI-assisted throughput increases output without adding patient-specific clinical context, shifts the reviewer from active diagnosis to passive scanning under volume pressure, and extends the practice into settings where the original governance assumptions no longer hold. ↩

  57. Parasuraman & Manzey (2010). The automation bias literature draws on decades of evidence from aviation safety, medical decision support, and industrial automation. ↩

  58. METR (2025), blog post, not peer-reviewed. The gap between pre-task prediction (+24%) and measured outcome (−19%) is 43 percentage points. ↩

  59. Senge (1990); Meadows (2008). The "Shifting the Burden" archetype describes how a symptomatic solution weakens commitment to the fundamental solution. ↩

  60. McConnell (2004). Industry averages of 15–50 defects per KLOC during development provide the empirical baseline. ↩

  61. The companion specification's "three enforcement layers" (AST scanner, type system, runtime structural enforcement) are implementation components within this paper's semantic-enforcement layer (layer 2 above), not a competing decomposition of the full validation boundary. ↩

  62. Appendix F develops a complementary argument: fine-tuned variants of the same base model inherit the training-distribution biases of their parent, so model diversity through fine-tuning should not be assumed to provide meaningful independence (§F.1). ↩

  63. ISM-1419 (environment segregation) remains important — agents should not have direct access to production environments — but is orthogonal to the semantic correctness concerns of this paper. ↩

  64. Static application security testing (SAST), dynamic application security testing (DAST), and software composition analysis (SCA). ↩

  65. IRAP, SOC 2, and Essential Eight compliance. ↩

  66. "Practices and tasks in this Profile do not distinguish between human-written and AI-generated source code, because it is assumed that all source code should be evaluated for vulnerabilities and other issues before use" (NIST 2024, p. 1). ↩

  67. Architectural reviewer: "Is the shape right?" Problem-framing reviewer: "Is this solving the right problem at the right location?" Implementation reviewer: "Was it implemented correctly?" Quality reviewer: "Are the tests and verification strategy adequate?" ↩

  68. Edge cases in cold paths, minor deviations from stated invariants, inconsistencies that do not currently trigger in testing. ↩

  69. The current enforcement tool operates at the AST pattern-matching level — detecting defensive anti-patterns (.get() with defaults, hasattr() gates, broad except blocks) in authority-tier contexts. Full taint path analysis (tracking data flow across function boundaries) is a capability of the companion specification's design (see companion specification, Part II-A §A.3), not the current tool. ↩

  70. The current implementation uses category-level ownership tags (e.g., "architecture", "bugfix") rather than individual reviewer names. The companion specification (§9.1, §13.1.3) requires reviewer identity as part of the exception governance model — the specific field format is a v0.1 implementation decision. ↩

  71. The underlying behaviour is trivially reproducible by any practitioner prompting a general-purpose coding agent to write error handling or data access code. ↩↩

  72. The specific conditions: a single ~80,000-line Python codebase, one developer's work, with purpose-built semantic enforcement tooling and rigorous human review — conditions most projects do not currently have. Three further caveats. (1) This rate occurs predominantly during unplanned work — bug fixing, ad-hoc refactoring, small feature additions — where the agent improvises from training data rather than following a reviewed specification. Planned major refactors and new components are reviewed against the project's trust topology before implementation, catching violations at the design stage. (2) The rate is model-specific, reflecting models available during the observation period; as AI companies prioritise these failure modes for remediation, the absolute rate will likely decrease. (3) The structural argument — that these patterns are embedded in training data and that agents lack persistent learning — remains valid regardless of the specific rate. The "one to two" framing reflects that the tool's taxonomy coverage is incomplete and that incidental discovery during non-development work corroborates the daily rate as a floor. ↩↩

  73. In a concrete example from the longitudinal project, peer checkpoint modules established a rigorous deserialisation pattern: set-based required-field checks, isinstance type guards, and AuditIntegrityError on any anomaly. A later addition used int() and str() coercion instead — a semantically equivalent trust boundary violation that the enforcement tool did not flag because its rule set targeted defensive access patterns (.get(), getattr()), not coercion functions. The agent defaulted to the statistically common coercion pattern from training data rather than following the established project convention, despite having access to the peer modules as examples. ↩

  74. Appendix E.6 documents a case in point: an agent drafting an implementation specification referenced get_token(force_refresh=True) on an Azure Identity credential object — a plausible API that does not exist. The spurious parameter was internally consistent within the specification, with downstream logic depending on forced refresh succeeding. Had the specification been implemented, the resulting TypeError would have been caught by tests. The dangerous scenario is the one where the implementing agent, encountering the crash, "fixes" it with a getattr() fallback or a try/except that silently skips the refresh — converting an ACF-S2 (detectable spurious reference) into an ACF-S1 (invisible fabricated default). ↩

  75. The primary generative mechanism for each entry, mapped to the failure-layer distinction in §2.4(h). Training bias — the model's priors encode the pattern as universally correct; persists across sessions and models with shared lineages. Context collapse — the model loses or displaces project-specific context during generation; addressable through session management and checkpoint controls. Process volume — the failure is in the review process, not the code; addressable through capacity planning and automated pre-screening. Some entries involve multiple layers; the column reflects the primary mechanism. The distinction matters for control selection: a control that addresses one layer may provide false reassurance against another (§2.4(h)). ↩

  76. The R3b detection gap merits attention during framework review. "Partial" for R3 overall reflects R3a's heuristic detectability. R3b in isolation may be closer to "None (without spec review gate)" — the only reliable catch is a reviewer who holds the full plan in context and can recognise that "tests pass" does not mean "task complete." This is a meaningful distinction for organisations designing review processes: R3a can be caught by pattern-matching review tools; R3b requires a reviewer who understands the intent of the change, not just its effect on the test suite. ↩↩

  77. No widely deployed tool detects this. Project-specific pattern matching (§8.3) provides limited intra-function proxy coverage — sufficient to catch some instances but not the cross-function taint flows that characterise the full failure mode. The "None" rating reflects the absence of generally available detection, not the impossibility of detection. ↩

  78. Primary STRIDE category is Spoofing. The detailed entry below notes an Elevation of Privilege consequence — the structural impersonation that S3 enables can result in implicit privilege grants — but the entry is classified under S because the mechanism (false structural identity) is spoofing, not elevation. ↩

  79. Three categories: Agent-specific — a failure mode that arises from the generative properties of agentic coding, not observed (or observed only rarely) in human-authored code. Known class, agent-amplified — a failure class already catalogued in human-authored code, but produced at higher frequency, greater consistency, or harder-to-detect form by agents. Workflow Pattern entries (provisional) describe failures in multi-session or multi-agent coordination rather than in generated code. ↩

  80. ACF-R3's "Partial" composite rating reflects R3a (Verification Substitution), which has identifiable detection signatures. R3b (Compensating Control Dependency) has no practical detection surface without a specification review gate. If R3b were rated independently, the "None" count would be 5 and the "Partial" count would be 8. ↩

  81. ACF-S1: .get() with defaults — other languages have analogues like Optional.orElse() in Java or ?? in C#. ACF-S2: getattr() with defaults — Python-specific, though dynamic languages like Ruby have send/respond_to?. ACF-S3: hasattr() as capability gate — Python-specific surface form, though the underlying failure applies to any language with duck typing or structural typing; Ruby's respond_to?, Go's interface satisfaction, and TypeScript's structural type compatibility are analogues. ACF-S4: # type: ignore and Any widening — Python-specific surface form; analogues include @SuppressWarnings("unchecked") in Java, // @ts-ignore or as any in TypeScript, and #pragma warning disable in C#. The underlying failure (suppressing the type checker rather than satisfying it) is language-general. ACF-S5: language-general. Python uses dict[str, Any], TypeScript uses any / Record<string, any>, Java uses Map<String, Object> / JsonNode, C# uses Dictionary<string, object> / dynamic, Go uses map[string]any. The surface form differs; the failure mode is the same. Arguably most acute in languages with opt-in type systems (Python, TypeScript, PHP) where the untyped path is the default. ↩

  82. Critical — silent compromise of access control or trust boundaries; high likelihood of agent generation; no existing detection. High — data integrity, audit trail, or availability impact; moderate-to-high likelihood of agent generation. Medium — exploitable under specific conditions or with limited blast radius. Low — detectable by existing tools with minimal residual risk. ↩

  83. COALESCE() is standard SQL and behaves identically across PostgreSQL, SQL Server, MySQL, Oracle, and SQLite. ISNULL() is SQL Server-specific; IFNULL() is MySQL/SQLite-specific. SQLite is worth noting because agents frequently generate it for prototyping and local development, and those queries sometimes migrate into production contexts — particularly in data workflows where a "temporary" SQLite database becomes a permanent operational store. ↩

  84. Autocommit is the default for JDBC, ODBC, and most interactive tools, though notably not Python's DB-API, which defaults to autocommit off per PEP 249. ↩

  85. F10 was rated Medium in the primary evaluation. The second-pass review upgraded it to High on the basis that the session is the only binding between the browser and the application record, and combined with F1 (default secret key), the session cookie becomes an unrestricted access token to any application's state. The session contains no server-side store — the signed cookie is the session — so a known signing key enables arbitrary application ID injection without any server-side trace. ↩

  86. See companion specification, Part II-A §A.4 — Group 16 (Generic Trust Boundary). The @trust_boundary decorator makes tier transitions structurally explicit; the type system enforcement layer (Part II-A §A.5) can then carry tier metadata through assignments, preventing the downstream .get() question from arising. ↩

  87. Surrogate-question deflection deserves formal treatment as either a named sub-pattern of ACF-S1 (the agent presents a confident answer to a question nobody asked) or a standalone provisional candidate. Promotion to a taxonomy entry is warranted if the pattern recurs across further examples with consistent structure. ↩

  88. The companion specification (§13.1.3) distinguishes between deferred architectural fixes and genuine domain variance in the exception register — a distinction that matters precisely when architectural decisions made at the specification layer produce downstream patterns that accumulate as exceptions rather than being fixed at their source. ↩

  89. The primary references are: Meadows, D. Thinking in Systems: A Primer. Chelsea Green Publishing, 2008. Senge, P. The Fifth Discipline: The Art and Practice of the Learning Organization. Doubleday/Currency, 1990. Leveson, N.G. Engineering a Safer World: Systems Thinking Applied to Safety. MIT Press, 2011. Meadows' leverage points (Chapter 6 of Thinking in Systems) provide the intervention hierarchy. Senge's archetypes (Chapters 5–6 of The Fifth Discipline) provide the "Shifting the Burden" analysis. Leveson's systems-theoretic accident model provides the safety engineering frame. All three are accessible to non-specialist readers. ↩

  90. Perry et al. conducted a controlled study with 47 participants (undergraduate and graduate students) completing security-relevant programming tasks in C, Python, and JavaScript. Participants with access to an AI assistant (OpenAI Codex via the OpenAI API) produced significantly less secure code across multiple task types while rating their code as more secure than the control group did. The study was published at ACM CCS 2023, a top-tier security venue with rigorous peer review. The student population is a limitation — professional developers may behave differently. But automation bias is well documented across professional populations in safety-critical domains (Parasuraman and Manzey 2010), and any claim that experienced developers are broadly immune to the same confidence-accuracy inversion would itself require substantial evidence. A more reasonable expectation is that professionals face the same dynamic, often under greater time pressure and production incentives. ↩

  91. "Eternal September" refers to September 1993, when AOL began offering Usenet access to its subscribers. Previously, each September brought a wave of new university students who were unfamiliar with Usenet norms; by October, they had acculturated. The AOL influx never stopped — hence "eternal." GitHub's use of the term to describe the AI-driven PR volume surge draws the same analogy: a sustained influx of contributions from producers who have not acculturated to project norms, at a rate that exceeds the community's capacity to socialise them. The original term is attributed to Dave Fischer in a January 1994 Usenet post. ↩