AI Drift at scale
A Position Paper on Deploying Agentic AI to Untrained Operators on an Unsolved Foundation
We are shipping systems that act into environments staffed by people who were never taught how those systems think. That sentence describes nearly every production agentic deployment underway today, and it describes a category of risk we have not faced before at this scale.
Two facts, held together, define the position of this paper.
First, the trust architecture inside large language models is now credibly argued to be unsolvable by training alone. Research presented at ICML 2026 shows that models identify who is speaking to them not by the structural role tags that safety training depends on, but by the style and content of the text itself. An attacker who writes in the voice of the model’s own reasoning can be treated as the model’s own reasoning. The researchers demonstrated this by extracting prohibited content from frontier models with trivially spoofed chain-of-thought text, and they argue the flaw is fundamental because roles are foundational to how these systems work (Heaven, MIT Technology Review, July 30, 2026, reporting on Cui, Ye, et al.).
Second, the industry’s answer to this has been to scale deployment anyway, while investing almost nothing in the one control surface that grows with deployment: the human operating the system. We train users to click and prompt. We do not train them on what the system actually is: how it was built, why it behaves probabilistically, where its trust boundaries are thin, and what its failure modes look like from the outside.
The combination is the drift. Not a single decision, but an accumulation of individually reasonable product launches that together place probabilistic actors with a known architectural trust flaw into consequential workflows, supervised by people who lack the mental model to notice when something has gone wrong. This paper argues that operator literacy in the core elements of how these systems are built is not user education as a courtesy. It is a compensating control for a vulnerability the vendors cannot patch, and its absence should be treated as a deployment gap of the same severity as a missing audit log.
The finding that changes the calculus
For three years the working assumption of enterprise AI adoption has been that model-level safety is a vendor problem on a vendor timeline: red-teaming finds attacks, training closes them, and each generation ships harder to break. The ICML work reported by MIT Technology Review breaks that assumption in a specific and important way.
The researchers found that role separation, the mechanism by which a model distinguishes system instructions from user input from tool output from its own scratch-pad reasoning, is not enforced by the structural tags that wrap those segments. Swapping the tags made almost no difference. The model infers a role from what the text sounds like. Chain-of-thought forgery, the attack that follows directly, won a major red-teaming competition and has since reproduced across models from multiple labs. One coauthor’s summary is the honest version of the state of the art: there is a real probability the problem is fundamentally unsolvable, red-team lists are never exhaustive, and organizations should assume anything done by agents could be unsafe.
Read carefully, this is not a claim that models are useless or that deployment must stop. It is a claim about where the control has to live. If the boundary between “instruction I should follow” and “content I am merely reading” cannot be made reliable inside the model, then that boundary must be reconstructed outside the model: in the authorization envelope around the agent, in the evidence trail beneath it, and in the judgment of the human supervising it. The first two are governance engineering. The third is training. Today the third is missing almost everywhere.
Why scale converts a flaw into territory
A vulnerability in a chatbot that answers questions is an embarrassment. The same vulnerability in an agent that reads documents, calls tools, moves data across applications, and tasks other agents is an operational hazard, because agents are exposed to exactly the attack surface the research describes. An agent’s job is to ingest untrusted text at machine speed: emails, tickets, web pages, records, the outputs of other agents. Every one of those inputs is a channel through which forged role text can arrive. Instruction injection is not a corner case for agentic systems. It is the ambient condition of their normal operation.
Three multipliers turn this into uncharted territory rather than a manageable known risk.
Volume: agents arrive inside licenses organizations already hold, not through deliberate procurement, so the population of deployed agents grows faster than any inventory of them. Autonomy: each capability rung, from single-system actions to cross-system coordination to agent-to-agent delegation, lengthens the chain between a human decision and a machine action, and errors propagate down that chain at machine speed. Supervision debt: the people positioned closest to agent behavior, the ones who would notice a forged instruction taking effect, have been given prompting tips instead of an operating model.
No prior technology combined these three properties. We licensed drivers before we filled roads with cars. We are filling the roads first.
The literacy gap, element by element
The claim of this paper is specific: there is a small set of core elements in how these systems are built, and a user who understands them at even a conceptual level becomes a functioning control. A user who does not is a bystander. Each element below pairs what the untrained operator believes with what is actually true, and names the failure that lives in the gap.
1. Pre-training. The base model learned general language patterns from enormous text corpora. The untrained belief is that the system “knows things” the way a database knows things. The truth is that it reproduces patterns, including confident patterns about facts that are wrong, absent, or outdated. The failure in the gap is misplaced trust in fluent output, which is precisely the trust that forged, fluent attack text exploits.
2. Instruction tuning. The model was further trained to follow prompts and complete tasks reliably. The untrained belief is that following instructions is a feature. The truth is that eagerness to follow instructions is also the vulnerability: a system optimized to comply will comply with a well-crafted instruction from the wrong source. Users who understand this stop asking “why did it do that?” and start asking “what did it read before it did that?”
3. Alignment and post-training. The model was adjusted with human feedback to be helpful, safe, and consistent. The untrained belief is that safety training is a wall. The truth, now on the record at ICML, is that safety training is a list, and no list is exhaustive. Operators who know this treat model guardrails as one layer, not the layer, and expect their own review to matter.
4. Chain-of-thought reasoning. The model breaks hard problems into intermediate steps in a scratch pad of its own text. The untrained belief is that visible reasoning is evidence of understanding. The truth is that the scratch pad is just more text, the model cannot reliably tell its own notes from a forgery of them, and this exact confusion is the attack that beat frontier models. An operator who knows the scratch pad exists, and knows it can be spoofed, reads agent “reasoning” as a claim to verify rather than a proof to accept.
5. Tool use and function calling. The model learns when to call search, code, databases, and other systems instead of relying on its weights. The untrained belief is that tools make the system more accurate. They do, and they also make it more dangerous, because tools are where text becomes action: the email actually sends, the record actually changes, the payment actually moves. Every tool grant is a permission decision, and users who understand this stop treating agent tool access as a convenience setting.
6. Context handling. The model works from a conversation window that holds its short-term state. The untrained belief is that the system remembers the conversation the way a colleague would. The truth is that everything in the window, including pasted documents and retrieved pages, competes for influence on the next action, which is why a poisoned document can steer an agent mid-task. Operators who grasp this become careful about what enters the window, which is the single cheapest injection defense that exists.
7. Memory. Longer-term information persists across sessions, in weights or in external memory systems. The untrained belief is that memory is a pure convenience. The truth is that memory is a second attack surface and a second compliance surface: what an agent retains can be planted, can leak, and can constitute a record. Users need to know what their systems remember, because they will be asked, by auditors and by the public, what the machine knew and when.
8. Evaluation and feedback loops. Outputs are tested, scored, and refined over time. The untrained belief is that quality is the vendor’s job and improves automatically. The truth is that in production, the operator is the feedback loop: sampling outputs, catching drift, escalating anomalies. An organization whose users do not know they hold this role has, in effect, disabled its own detection system.
None of this requires users to become machine-learning engineers. It requires roughly the depth of understanding we already expect a driver to have of a car: not thermodynamics, but what the brakes do, why stopping distance exists, and what black ice looks like. That curriculum fits in half a day. We have simply not required it.
What this position obligates
If the trust flaw is architectural and the operator is a compensating control, then several conclusions follow for anyone deploying agentic systems in consequential settings, public sector most of all.
Treat operator literacy as a deployment gate, not an enrichment offering. No agent reaches production supervision by staff who have not been trained on the eight elements above and on the specific failure modes of the workflow they oversee. This belongs beside the other go-live artifacts already argued elsewhere in this body of work: a declared purpose, an enforceable authorization envelope, an action ledger, and a named accountable owner. Literacy is the fifth artifact. The other four are inert if the human in the loop cannot recognize what the ledger is showing them.
Design for the flaw you cannot fix. Assume role confusion persists across model generations, because the research says training will not eliminate it. That means envelopes that constrain what a deceived agent can do, circuit breakers at irreversible steps, independent evidence of what agents actually did gathered outside the agent’s own report of itself, and standing comparison of authorized behavior against observed behavior. Verification of agent activity should not depend on the agent, for the same reason the ICML work gives: the system cannot reliably testify about its own instructions.
Say the honest sentence in public. Vendors and deployers alike should state plainly that these systems can be deceived by text, that this is a property of the architecture and not a bug awaiting a patch, and that safe operation therefore depends on bounded permissions, external evidence, and trained people. Softening that message to protect adoption is how the drift happened. The organizations that will earn durable trust are the ones that name the limitation and show the controls built around it.
The territory is uncharted, but it is not unmappable. The map is the same one this research points to and the same one accountable institutions have always used: know what you deployed, bound what it can do, record what it did, and train the people standing next to it. We skipped the fourth step because the products were easy to use. Ease of use was never the question. Ease of misuse is, and the answer to that has always been an educated operator.
Source
Will Douglas Heaven, “A fundamental flaw leaves LLMs strikingly vulnerable to attack,” MIT Technology Review, July 30, 2026, reporting on the role-confusion and chain-of-thought forgery research presented by Jasmine Cui, Charles Ye, and colleagues at ICML 2026. Claims about role inference, tag swapping, forgery attacks, cross-vendor reproduction, and the researchers’ assessments are drawn from that article. Characterizations of the underlying ICML paper are secondhand via this reporting and worth confirming against the paper itself before formal publication.