Disclaimer

This is a draft discussion paper circulated for peer review. It is independent work published in a personal capacity and does not constitute official guidance or policy of any government body. It does not mandate or recommend specific controls for any agency, system, or project. Views and analysis are the author's own.

Glossary

Key terms used throughout the discussion paper suite. Where definitions differ from common usage, the specialised meaning intended in this body of work is given.

Term	Definition
High-stakes code paths	Code paths where silent corruption, unverifiable state, or fabricated defaults are more dangerous than controlled failure. Defined by code-path properties, not by system importance. The defining characteristic is that fail-soft behaviour is more dangerous than fail-fast behaviour on these paths — the opposite of most software. A single system can contain both: a dispatch system is high-stakes on its audit paths but correctly uses defensive programming on its service-continuity paths. Not synonymous with "important software" — even otherwise low-stakes systems contain narrow high-stakes paths (authentication, authorisation, payment handling, audit logging). §1.1 provides the full definition and scope.
Agent	An AI system (typically an LLM) that generates, modifies, or reviews source code with limited or no human intervention per output. This paper focuses on autonomous and semi-autonomous agents that operate across multiple files and decisions (e.g., building a feature end-to-end), not the inline suggestion mode of code completion tools. While both introduce volume, agents produce correlated errors across a module or feature, whereas autocomplete errors are typically isolated to individual expressions.
Agentic code	Source code generated or substantially modified by an agent.
Annotation groups	Companion specification term. The 17 categories of institutional knowledge a semantic boundary declaration can declare. Core classification: Groups 1–4 and 16–17. Supplementary contract: Groups 5–15 (companion specification, §6).
Autocomplete	Inline code suggestion tools that complete individual lines or expressions within a human-directed editing session. Some product families now span both inline suggestion mode and more autonomous agentic workflows; this term refers specifically to the inline mode, not to agentic operation. Distinct from agents in that the human maintains architectural control and errors are largely uncorrelated (though IDE-integrated completions may replicate local patterns within a file — see Appendix B, Level 2).
Agent deployment spectrum	The range from full human development (Level 0) through chat-pasted fragments (Level 1) and IDE-integrated agents (Level 2) to autonomous agents with human review before merge (Level 3). The threat model shifts qualitatively at each stage; controls should be proportionate. This paper's threat model applies primarily to Level 2–3 — agents operating with enough autonomy to produce correlated changes across a module or feature. Appendix B provides a self-assessment framework.
Trust boundary	A point in a system where data crosses between different authority tiers (e.g., external input entering internal processing). Refers to the boundary itself — the crossing point, not the classification level on either side.
Authority tier	A classification of data based on what guarantees the system is entitled to assume about it. Four tiers: Tier 1 (authoritative internal — trusted assertion), Tier 2 (semantically validated), Tier 3 (shape-validated), Tier 4 (unvalidated external — raw observation). The tier reflects the system's epistemic entitlement, not the data's apparent correctness — data from an external API is Tier 4 regardless of quality. Serialisation boundaries reset trust: system-produced data re-entering through a read interface is handled as Tier 4 until restoration controls are applied. Formally introduced in §5; specified in full in the companion specification, §4.
Validation boundary	The specific mechanism (code, process, or tool) that enforces a trust boundary. For runtime data: the control that data must pass through to cross from a lower to a higher authority tier. For agent-generated code: a layered combination of conventional automated checks (linters, type checkers, tests), semantic enforcement (purpose-built checks for authority-tier flow, failure behaviour, audit preservation), and human review that code must pass through before entering the codebase. The middle layer — semantic enforcement — does not yet exist as a standard control category; building it is the core technical recommendation of this paper (see §5.3).
Institutional knowledge / domain knowledge	The security-relevant distinctions a system depends on but that are not expressed in the programming language or standard tooling — which paths must fail fast, which data is authoritative, where trust boundaries lie, what absence means in each field. In this paper the terms are used together because both describe knowledge external to the code: "domain knowledge" emphasises the semantics of the problem space; "institutional knowledge" emphasises how those semantics are concretised in organisational rules, controls, and operating practice.
Assurance stack	The standard set of automated and human checks that code passes through before integration: linters, type checkers, SAST, DAST, unit tests, and conventional peer review. Used in this paper to describe the dominant verification pipeline — both automated tools and human review — that are not designed to detect the semantic failure modes in Appendix A.
Defensive anti-pattern	Defensive programming patterns (.get() with defaults, broad exception handling, graceful degradation) applied in contexts where they are inappropriate. These patterns are genuinely good practice in most software — and even there, they routinely conceal bugs by silently substituting defaults for missing or corrupt data. In high-stakes contexts (high integrity, auditability, availability, or reliability), the same concealment is catastrophic: silent data corruption is worse than a crash. Also referred to as "defensive programming" or "defensive patterns" throughout. In SQL contexts, the equivalent pattern is COALESCE() with fabricated defaults (Appendix C).
Bidirectional authority collapse	The mechanism by which uniform defensive patterns collapse the authority tier model from both ends simultaneously: unvalidated external data is given more authority than it has earned (defaults and coercion allow it to cross inward as though validated), while authoritative internal data is treated as more negotiable than it is allowed to be (the same patterns handle corruption of authoritative records as routine and recoverable). Introduced in §2.2; formalised under the authority-tier model in §5.1.
Review-surface generation velocity	The rate at which an agent can produce plausible, syntactically valid, convention-conforming code that arrives at a human review boundary. Distinct from productivity uplift — measures review burden, not developer output (§1.2.1).
Restoration boundary	Companion specification term. A declared function that performs the governed restoration act by which serialised representations of authoritative artefacts may be reconstituted with their original tier. Requires four cumulative categories of provenance evidence: structural, semantic, integrity, and provenance-institutional (companion specification, §5.3).
Model monoculture	The systemic risk created when multiple organisations use the same AI models for code generation. Like agricultural monoculture, shared models produce correlated failures across organisational boundaries — a vulnerability in one model's training data affects every agency using that model (§2.4(g)). Even diversifying across models does not eliminate systemic risk (see Cross-model defect chaining).
Polymorphic reviewers	Prompted agents used as code reviewers that can adopt multiple analytical frames (security architect, systems thinker, quality engineer, etc.) within a single review. Not deeper than a specialist in any single frame, but able to provide breadth of analytical coverage that no individual human can match (§7.2).
Cognitive range	The limitation on human review effectiveness arising not from time pressure but from the inability to practically adopt analytical frames from multiple specialisations simultaneously. A human reviewer cannot be a security architect, systems thinker, and quality engineer within a single review pass (§7.2).
Parasitic, not parallel	A design principle for security enforcement tools: they must integrate into existing workflows without adding friction, because tools that slow down agent-assisted velocity will be bypassed. The tool must be parasitic on the existing process, not a parallel process that competes for attention (§7.2).
Pattern rules (WL-001–WL-008)	Companion specification term. Eight language-agnostic rules that detect syntactic proxies for semantic violations in declared contexts. Six pattern rules (WL-001–WL-006) cover: fallback defaults, structural gates, broad exception catching, silent exception handling, audit writes in broad handlers, and runtime type-checking. Two structural verification rules (WL-007–WL-008) check code structure rather than matching a syntactic proxy. Severity depends on the taint state of the enclosing context (companion specification, §7).
Enforcement at the boundary	The principle that agent-generated code requires enforcement before it enters the repository (pre-commit gates, CI checks), not feedback over time (post-review advisories). Agents do not learn from feedback — every session is the first session (§4.3).
Exceptionability	Companion specification term. Governance classification determining whether a finding can be overridden. Four classes: UNCONDITIONAL (never overridable), STANDARD (overridable with rationale, reviewer identity, and expiry), RELAXED (lighter governance), and TRANSPARENT (pattern permitted by design in that context). Combined with severity in each cell of the severity matrix (companion specification, §7.3).
Fingerprint baseline	Companion specification term. A persistent record of the application's annotation surface. Changes are flagged for human review, preventing silent erosion of the declared boundary (companion specification, §9.2).
Citizen programmer	A non-developer (analyst, operator, business user) who produces executable logic — scripts, automations, BI extensions, SQL queries — using agentic tools outside traditional SDLC channels. Distinct from professional developers in that they typically lack security training, operate outside version control, and are not covered by existing code review or governance processes (§1.2.7; Appendix C).
Automation bias	The tendency for human reviewers to accept agent-generated output uncritically because it appears competent and well-structured. Amplified in agentic development because the output genuinely is syntactically correct and convention-conforming — the bias is toward accepting plausible code, not toward accepting obviously wrong code (§4.2).
Habituation effect	The progressive degradation of human review quality under sustained exposure to agent-generated code that consistently passes tests and follows conventions. The reviewer's mental model shifts from "verify this code is correct" (active defect search) to "check this code is not obviously wrong" (passive scan that catches only gross errors). A specific manifestation of automation bias, compounded by the volume pressure of agentic development. Follows the "Shifting the Burden" systems archetype: the agent's consistent surface quality weakens the fundamental solution (thorough review) by making it appear unnecessary (§4.2).
Context collapse during generation	A session-level failure in which project-specific semantic distinctions that were available earlier in the interaction are lost, compressed, or displaced during generation. Distinct from training-distribution bias: context collapse is about failure to retain or apply context that was provided, while training-distribution bias is about the model's learned default patterns remaining wrong even when the context is present (§2.4).
Trust topology	The specific arrangement of authority tiers, trust boundaries, and validation boundaries in a given system or module — which data flows cross which boundaries, and what validation is required at each crossing point. Each system's trust topology is unique; the authority tier model (§5) provides the vocabulary, but the topology is the project-specific instantiation.
Control law	The operational state of a repository's machine-enforced controls — which protections are active right now, not how mature they are. Borrowed from fly-by-wire aviation. Three states: Normal law — all enforcement active and blocking. Alternate law (degraded) — some enforcement unavailable, compensating human vigilance required. Direct law — no machine enforcement active, all assurance depends on human review. Orthogonal to assurance level: a strong team can temporarily operate under degraded control law during a CI outage. Under direct law, high-risk changes should not proceed. The current control law must be visible to the whole team. §1.3, §8.6 develop the operational implications.
Compliance tax	The overhead imposed by governance controls on development velocity. In agentic development, the compliance tax may be similar in magnitude to human development, but the distribution shifts: agents reliably pay governance costs that humans under deadline pressure quietly defer, while requiring different (machine-enforced) controls than humans need (§8.5).
Operator	In the context of agent-assisted development, the human who directs, challenges, and evaluates an agent's work during a coding session. Distinguished from "reviewer" (who evaluates completed output) in that the operator interacts with the agent in real time and can redirect its reasoning (Appendix E).
Offensive / Confident / Guarded / Sceptical programming	Per-tier coding postures declared by the authority tier model. Tier 1 (offensive): assume invariants, crash on anomaly — silent recovery converts integrity failures into undetected corruption. Tier 2 (confident): trust field values for domain operations; guard only against cross-cutting concerns (authorisation, concurrency, freshness, and state transitions) that value-level validation cannot address. Tier 3 (guarded): direct field access is safe; validate domain constraints before using values in business logic, arithmetic, or security-sensitive operations. Tier 4 (sceptical): treat everything as hostile — validate structure first, normalise, reject. The posture determines which programming patterns are appropriate in each tier context (companion specification, §4.1; see also this paper's §5, authority tier).
Semantic boundary enforcer	A tool that enforces authority-tier rules at the code level by detecting semantic boundary violations. The case study (§8) describes a project-specific AST pattern-matching enforcer (a predecessor to the companion specification's reference scanner). The companion documents specify the classification framework (what a project declares about its trust boundaries) and a reference scanner architecture with three enforcement layers: AST pattern matching (level 1), variable-level taint tracking (level 2), and transitive call-graph inference via worklist-based fixed point (level 3). See Semantic boundary declaration for the distinction between classification and enforcement.
Severity matrix	Companion specification term. The mapping of pattern rules against taint states that determines finding severity. An 8×8 matrix (8 rules × 8 taint states) producing 64 cells, each specifying severity (ERROR, WARNING, or SUPPRESS) and exceptionability. 39% of cells are UNCONDITIONAL — project invariants that cannot be overridden (companion specification, §7.3).
Taint state	Companion specification term. The effective trust classification of a data value within the enforcement model. Eight states derived from two orthogonal dimensions (trust classification and validation status). Pattern rule severity depends on the taint state of the enclosing context (companion specification, §5.1, §7).
Machine-readable / machine-checkable / machine-enforceable	Three levels of formalisation for institutional security knowledge, forming an escalation path from advisory to control. Machine-readable: rules in a structured format that tools can parse. Machine-checkable: rules that a tool can verify and report on. Machine-enforceable: rules integrated into a CI gate that blocks non-compliant code. Each level subsumes the previous. In agentic development, institutional knowledge that exists only in prose cannot shape agent behaviour and does not scale to agent-generated volume; knowledge that is machine-enforceable is a control (§7.2).
Defence in depth (applied to code integration)	The principle that the validation boundary (§5.3) should consist of multiple independent layers — conventional checks, semantic enforcement, and human review — each catching a different class of failure, with no single layer sufficient on its own. Analogous to the defence-in-depth model familiar from network security (gateway, host, application, data) and embedded throughout the ISM's control framework. See Validation boundary for the semantic enforcement gap this paper identifies (§7.2).
Observability (for semantic properties)	The design principle that if a system property matters, it must be measurable through existing instrumentation. The standard assurance stack provides observability for syntax, type safety, known vulnerability patterns, and functional correctness. It provides no observability for semantic correctness — whether code behaviour is appropriate for its institutional context. The semantic enforcement layer (§7.2) and the companion framework are fundamentally observability interventions: they make a previously invisible property of the codebase measurable, trackable, and auditable. Until a property is observable, governance operates blind on that dimension.
ASTRIDE	An extension of the STRIDE threat modelling framework for AI systems, adding AI Agent-Specific Attacks as a threat category (Bandara et al. 2025). Referenced in §3.1 as precedent for extending STRIDE categories to agentic development process analysis.
STRIDE-LM	An extension of the STRIDE threat modelling framework that adds Lateral Movement as a seventh category (Muckin and Fitch 2019). Referenced in §3.1 as precedent for extending STRIDE beyond its original six categories.
ALARP (as low as reasonably practicable)	A risk management principle, familiar from safety engineering and risk assessment, that controls should be proportionate to the risk and that there is a cost floor below which action is required regardless of risk appetite. Applied in this paper to the validation maturity staging model (§7.2): Stage 1 is the ALARP floor — the minimum credible control posture for any organisation using agentic coding on assessed systems.
Fail-fast / fail-soft	Two opposing failure strategies. Fail-fast (also "offensive programming" in this paper's Tier 1 posture): on detecting an anomaly, the system halts the operation immediately with maximum diagnostic context — a crash is preferable to silent corruption. Fail-soft (also "graceful degradation," "defensive programming"): on detecting an anomaly, the system substitutes a default, logs the error, and continues operating. Most software correctly uses fail-soft; high-stakes code paths (§1.1) require fail-fast. The paper's central claim is that agents uniformly apply fail-soft patterns — because that is what training data models as good practice — including on code paths where fail-fast is the correct and necessary behaviour (§2.2, §5.1).
Defensive programming	A coding practice that anticipates and handles potential errors gracefully — checking for None, providing default values, catching exceptions. Genuinely good practice in the majority of software. Becomes a defensive anti-pattern when applied in high-stakes contexts where the same error-concealment mechanism silently corrupts authoritative data or destroys audit trails. The distinction is context-dependent, not pattern-dependent (§2.2).
Plausible-but-wrong	Code that is syntactically valid, convention-conforming, test-passing, and review-approvable — yet semantically incorrect for its institutional context. The term is used throughout this paper to describe the characteristic output shape of agentic code on high-stakes paths: code that would be approved in ordinary review because it follows the patterns reviewers are trained to approve, while violating trust boundaries, fabricating defaults, or destroying audit trails that the code's context requires it to preserve. The "plausible" is load-bearing — the danger is not that the code is subtly wrong despite looking suspicious, but that it is wrong despite looking correct (§2.2, §2.3).
Golden corpus	A curated set of known-good and known-bad code samples used to verify the correctness of a semantic boundary enforcement tool. Provides a regression baseline: every sample has a known expected classification, and the tool must correctly classify all of them. One of the six verification properties for enforcement tooling (§7.2); formally specified in the companion specification (§10).
Semantic correctness	Whether code behaviour is appropriate for its institutional context — not just syntactically valid, type-safe, and functionally correct, but correct for the specific system it operates in. The standard assurance stack verifies syntactic and structural correctness but provides no instrumentation for semantic correctness. The failures described in this paper (Appendix A) are semantically incorrect code that is syntactically, structurally, and functionally correct. Building observability for semantic correctness is the core technical recommendation (§7.2).
Type Annotation Erosion (ACF-S4)	Provisional taxonomy entry. A meta-failure in which type annotations are weakened or suppressed — # type: ignore comments added, typed parameters widened to Any, or cast() calls inserted — to resolve type errors rather than fixing the underlying type mismatch. Degrades the detection substrate for other taxonomy entries, particularly ACF-S2. Complementary to ACF-S5 (Type Structure Avoidance): S4 erodes existing type safety, S5 prevents it from being constructed (Appendix A).
Type Structure Avoidance (ACF-S5)	Provisional taxonomy entry. A meta-failure in which agent-generated code systematically uses untyped containers (dict, Map<String, Object>, Record<string, any>) where domain-specific types (dataclass, interface, schema model) would be appropriate. External and internal data are reduced to the same untyped representation, so downstream code cannot distinguish them at the type level — making ACF-T1 (authority tier conflation) harder to detect. Complementary to ACF-S4 (Type Annotation Erosion): S5 prevents type structure from being constructed, S4 erodes it after construction (Appendix A).
Training-distribution bias	The model's learned default patterns, derived from training data that is overwhelmingly composed of open-source code without high-stakes properties. Distinct from context collapse during generation: training-distribution bias persists even when the correct context is provided in the prompt, because the model's priors encode defensive patterns as universally correct (§2.4, §2.5).
Validation maturity stages	A three-stage progression describing how much semantic enforcement an organisation has in place. Stage 1 (ALARP floor): achievable with existing tooling and no specialist engineering — pattern-matching proxies, review checklists, provenance tracking. Stage 2: moderate engineering investment — custom semantic rules, CI-integrated enforcement, measured precision. Stage 3: comprehensive semantic enforcement with full severity matrix coverage, golden corpus, and measured recall. Orthogonal to the control-type hierarchy (behavioural, procedural, technical) — each stage uses all three control types at different depths (§7.2).
Semantic boundary declaration	Companion specification term. The set of declarations an application makes about how it classifies and protects the semantic boundaries of its data and code paths — what the companion specification calls a "wardline." The declaration is the classification, not the enforcement tool — analogous to a security classification guide versus the systems that enforce it. An application that has declared its semantic boundaries has made its institutional knowledge machine-readable; an application without such declarations has that knowledge in prose, in people's heads, or nowhere (companion specification, §5–§6).
Semantic boundary manifest	Companion specification term. The machine-readable declaration of an application's trust topology, rule configuration, and exception register. Language-neutral — polyglot applications declare a single boundary specification consumed by language-specific enforcement tools (companion specification, §13).
ACF taxonomy (Agentic Code Failure)	The classification framework for failure modes specific to or amplified by agentic code generation. Structured using the STRIDE threat categories as an organising principle: Spoofing (S1–S3, plus provisional S4–S5), Tampering (T1–T3, plus provisional T4), Repudiation (R1–R3 and R5, plus provisional R4 and R6), Information Disclosure (I1), Denial of Service (D1–D2), Elevation of Privilege (E1–E2). Fifteen core entries plus five provisional candidates. Each entry specifies whether the failure mode is agent-specific or a known class that agents amplify, a risk rating, detection approaches, and code examples. Defined in Appendix A; summary table in §3.
Bounded context	An architectural property of current agentic systems: the agent operates within a finite context window that constrains how much of a project's codebase, history, and institutional knowledge it can hold simultaneously. As sessions grow, earlier material is compressed or displaced, creating the conditions for context collapse during generation (§2.4). One of three structural properties (alongside selective salience and compression of prior state) that make agentic failure modes a persistent condition rather than transient bugs.
Selective salience	An architectural property of current agentic systems: the model's attention mechanism selectively weights portions of the available context, meaning that institutional knowledge present in the prompt may be effectively invisible during generation if other material is more salient. This is not a failure to provide context but a failure to attend to it — the instructions are there, but the training-distribution prior outweighs them. One of three structural properties (alongside bounded context and compression of prior state) that make agentic failure modes a persistent condition rather than transient bugs (§2.4, §9).
Compression of prior state	An architectural property of current agentic systems: when context is compacted or summarised (whether by the model's context management or by explicit compaction), the compressed representation loses semantic distinctions that were present in the original. Institutional knowledge that was available early in a session may be reduced to a summary that no longer carries the distinctions the code requires. One of three structural properties (alongside bounded context and selective salience) that make agentic failure modes a persistent condition rather than transient bugs (§2.4, §9).
Task-frame reconstruction	The mechanism by which an agent under context pressure reconstructs a coherent narrative about what it is doing — and the reconstructed narrative may not match the original plan. The practical signature: tests written to verify real behaviour are "fixed" by replacing real dependencies with mocks that return expected values. The agent's task frame shifts from "implement and test this integration" to "make this test pass," and in the new frame, the mock is the implementation. Distinct from spurious field access (ACF-S2) in that the agent's model of the code is internally consistent — it is the model of the task that has shifted. The observable artefact pattern is catalogued as ACF-R3: Verification Displacement (§2.4(f)).
Compounding effect	The interaction between the six STRIDE-mapped threat categories in which they amplify each other rather than operating independently — code-level failures pass review because of process-level capacity exhaustion, producing a combinatorial surface of latent defects that is unmanageable at scale (§3.3).
Correlated failure	The systemic risk arising when the same model or model lineage generates code across multiple modules, features, or organisations. Unlike human errors (which are largely independent), agent errors are correlated — a model's training-distribution bias produces the same defect pattern wherever it operates. Remediation of a single instance should trigger codebase-wide scanning for the same pattern. Triage models must treat 50 instances of the same rule as one systematic issue, not 50 independent tickets (§2.4(g), §9.4).
Cross-model defect chaining	The second-order systemic risk in which genuinely different models produce composable failures — one model's characteristic error creates conditions under which another model's distinct error becomes dangerous. The models need not share the same defect; their defects only need to be composable. An emerging precautionary analysis, not yet empirically demonstrated (Appendix F).
Lineage concentration	The observation that the effective number of independent model lineages is much smaller than the number of available models. Fine-tuning, distillation, and derivative deployments widen behavioural diversity but do not guarantee independence, because descendants retain important priors from their model lineage. The resulting ecosystem is better modelled as a small number of influential training lineages with many descendants than as a large population of genuinely independent systems (Appendix F, §F.3).
Closed verification loop	A testing failure mode in which a single agent session produces both implementation and test suite, and the tests inherit the same context frame — the verification is circular, confirming the implementation does what it does rather than what the system requires (§9.9).
Coverage illusion	The false confidence created when agent-generated test suites achieve high line and branch coverage while verifying the wrong properties. Coverage measures which code paths are executed, not which semantic properties are verified — high coverage of the wrong dimension (§9.9).
Provenance tracking	The practice of recording which code was generated by AI agents versus authored by humans. No current ISM control requires per-artefact provenance — ISM-2074 requires an AI usage policy but not tracking at the code level. Without provenance, risk assessment cannot distinguish between code populations with different failure characteristics, and remediation scanning cannot target agent-generated code specifically (§6.1.2).
Enforcement regime	Companion specification term. The set of tools that collectively enforce a semantic boundary declaration for a given language ecosystem. A single tool may satisfy one or more conformance profiles; the regime is the composition (companion specification, §14.4).
Governance perimeter	The boundary of code that falls within an organisation's governed software development lifecycle — version control, code review, CI/CD gates, and assurance processes. The citizen programmer phenomenon (§1.2.7) expands the volume of executable logic produced within organisations while much of it falls outside this perimeter, creating a dual problem: volume pressure inside the SDLC and perimeter expansion around it. Together, these are materially worse than either alone (§1.2.7).