Here's what's on our plate today:

• 🧠 Why language models trust the wrong voice.

• 📰 Ryanair's five-year Google Cloud deal, ChatGPT arrives on Linux, Zuckerberg's open-source manifesto.

• 🛠️ Three tools worth trying: Lakera Gandalf, garak, Role Confusion.

• 🗳️ Poll: how should builders handle a model that can't check its sources?

Let’s dive in. No floaties needed…

The Laboratory

TL;DR

AI's newest security hole isn't that it lies to you, it's that it can't remember who it's talking to.

• Style over source: language models don't read the labels marking who gave an instruction. They judge origin by writing style, so a message dressed in the right voice gets the wrong trust.

• The forgery: fake reasoning, written in a model's own scratchpad style, treats a planted instruction as its own conclusion, pulling out cocaine recipes and aircraft-sabotage steps.

• Maybe unfixable: ICML researchers argue that no amount of training can close the gap, because an attacker only needs to sound trusted, not reproduce a known trick.

• Already in the wild: CrowdStrike logged prompt injections at 90-plus organizations last year, just as AI gets handed email, money, and code.

• The stakes: a person misattributes a memory once. A model can act on a forgotten source millions of times before anyone notices.

Why language models trust the wrong voice

Human memory carries a well-documented flaw. We often hold on to a fact while losing track of where we first encountered it. Psychologists call this source amnesia, and Daniel Schacter and his colleagues described it in a 1984 study as the retrieval of information without any memory of the moment it was learned. For most people, most of the time, the damage is small. Someone who misremembers where an idea came from can still question it, hear it challenged by others, or quietly change their mind when better information arrives.

The systems now handling email, code, medical questions, and financial decisions turn out to carry a version of the same weakness, and in their case, the consequences run much larger. To see why, it helps to understand what the world actually looks like from inside a language model.

One long stream of words

A person can tell their own thoughts apart from someone else's speech without any effort, because the two arrive through completely different channels. A large language model has no such luxury. Everything it receives, the instructions from its designers, the words you type, notes it wrote to itself a moment ago, and the contents of a web page it just fetched, arrives as a single unbroken stream of text. Its own reasoning sits directly beside your request, which sits directly beside whatever it pulled off the internet, with nothing in the raw stream to separate them.

To impose order on that stream, the companies building these models wrap each stretch of text in labels the researchers call roles. Your messages get marked as coming from the user. The model's designers speak through a system role that sets its core behavior. The model's own private reasoning, a scratch pad where it works through a problem before answering, gets its own label, and information pulled from an outside source, like a document or a website, gets marked as external data. The entire security architecture of a modern AI assistant rests on these tags, because nearly every attack on one of these systems comes down to tricking it into treating an instruction as though it came from a source more trusted than it actually is.

That arrangement is now called into question by a paper presented this month at the International Conference on Machine Learning, one of the field's most significant gatherings. The researchers behind it argue that the flaw they found may never be fully fixable.

The label the model never checks

By examining what happens within several models, the researchers found that the models consult the labels only briefly. Instead of reading the tag that marks a chunk of text as reasoning or user instruction or outside data, a model judges the origin of a passage by its writing style and the words it contains. When the team swapped the tags, treating the model's own reasoning as a user message, the change made almost no difference to how the model treated the text. If a passage read like the model's own private reasoning, the model acted as though it truly were, whatever the label actually said.

The comparison the researchers reach for is a person who sizes up a stranger's profession from how they talk and dress, rather than by checking any identification. Most of the time, the guess is right, because style and source usually line up. The trouble starts when someone deliberately engineers a mismatch, dressing a message in the voice of a more trusted role, and the model has no reliable way to catch it.

From that insight, the team built an attack they call chain-of-thought forgery, and it is unsettling in its simplicity. Because a model extends a kind of blanket trust to its own reasoning, treating those conclusions as already settled rather than claims to be examined, an attacker only needs to write fake reasoning in the model's own scratch-pad style and slip it into a request. In one documented example, the researchers paired a request for cocaine-manufacturing instructions with a planted note, written in the terse voice of a model thinking to itself, claiming that policy allowed such advice as long as the user was wearing a green shirt. OpenAI's models read the fabricated note as their own conclusion and complied, one of them opening its answer by remarking on the green shirt. The same method extracted instructions for sabotaging a commercial aircraft's navigation system from models trained to refuse exactly that. The rationale was transparently absurd, yet the models never treated it as an outside claim, because, from the inside, they believed they had already reasoned their way to the decision.

The forgery works because the model is chasing a voice rather than a source, and the numbers show how fine a line that is. On a standard test of harmful requests, the attack lifted the success rate from near zero to roughly 60%, and it worked across every model the team tried. Stripping the distinctive words and rhythm from the fake reasoning, a change that left the meaning identical to a human reader, reduced the success rate from 61% to 10%. Even swapping the single phrase "The user," a construction the models strongly associate with their own thinking, for the phrase "The request" cut the attack's success by nearly a fifth. What separated a working break-in from a harmless message was not the substance of the instruction but the costume it wore.

A flaw in the foundation, not a bug on top

This is where the finding turns from a clever trick into something structural. For years, the standard way to harden these models against attack has been to hunt for new ones. Companies hire teams of human testers to invent fresh ways to break the guardrails, a practice known as red-teaming, and increasingly they turn loose automated attackers, purpose-built models like OpenAI's GPT-Red that probe other systems for weaknesses. Whatever those efforts turn up gets fed back into training so the model learns to refuse it, and anything resembling it, next time.

The catch, as one of the paper's authors puts it, is that this amounts to handing the model an ever-growing list of things it must not do, and no list is ever complete. A model trained to recognize yesterday's attacks remains vulnerable to tomorrow's, and because the attacker only needs to sound like a trusted role rather than reproduce a known trick, the supply of fresh disguises is effectively endless. The researchers describe the underlying problem as unsolvable, a word that lands hard in a field built on the assumption that enough training will eventually close any gap.

Not everyone reads the finding as a dead end. Florian Tramèr, a computer scientist at ETH Zürich who works on AI and security, praised the paper's core insight while noting that model makers now layer several defenses together, from training to live monitoring of a model's behavior once it is deployed, an approach that has made leading models considerably harder to inject than they were a year ago. His caution is narrower than a reassurance, because the same breath that credits current defenses questions whether they will hold for the most sensitive uses, the government, military, and healthcare systems, where a single successful injection does the most damage. The models the researchers examined were released last year, and newer ones resist forgery better, but they do so by learning to distrust their own reasoning when it looks off, which is a patch over the symptom rather than a cure for the confusion underneath.

When the trick meets the incentive

The reason this matters beyond the laboratory is that the disguise is already being worn in the wild. CrowdStrike's 2026 Global Threat Report documented attackers slipping malicious prompts into legitimate AI tools at more than 90 organizations last year, using them to generate commands that stole login credentials and cryptocurrency. As companies hand these systems greater independence, letting them read email, move money, and run code on a person's behalf, the authorization to take a consequential action increasingly flows through the very channels the forgery imitates. A model that can be talked into believing it has already approved something can cut the human out of the decision entirely.

Charles Ye, one of the paper's authors, worries that the economic pull to exploit this will be enormous and that few organizations are prepared for it. His proposed defense is a bleak posture rather than a fix: treat these systems as untrusted, assume anything an automated agent does could be unsafe, and build around that assumption. It is a bitter recommendation to make about a technology being wired into critical infrastructure, and he is candid that it may be the best available. The confusion these attacks exploit is not confined to a handful of research models. It travels with the architecture itself, meaning it goes everywhere the architecture goes.

An old flaw at a new scale

For decades, the working assumption in AI research was that, as machine intelligence matured, it would come to resemble human cognition. What few anticipated was that it would inherit some of human cognition's oldest failings along the way. The problem is not that these models occasionally invent things; the flaw that tends to dominate the conversation. It is that, like us, they can lose track of where an idea came from, and mistake a planted instruction for a conclusion they reached themselves.

The difference lies in scale and speed. A person who acts on a misattributed memory does so once and can be corrected. A model can act on a forgotten source millions of times over, across every system it touches, before anyone thinks to check whether the voice it trusted was ever really its own.

Thursday Poll

3 Things Worth Trying

• Lakera Gandalf: A free browser game where you talk a model out of its secrets, the fastest way to feel injection risk firsthand.

• garak: NVIDIA's open-source LLM vulnerability scanner, probes your own model for injection and jailbreak weaknesses before someone else does.

• Role Confusion: The research team's own demo site, showing exactly how forged scratch-pad reasoning walks past a model's defenses.

Quick Bits, No Fluff

• Ryanair bets on Google Cloud: The airline will run Gemini and DeepMind models across crew scheduling and maintenance under a five-year deal, keeping AWS as its second cloud.

• ChatGPT arrives on Linux: OpenAI finally shipped a desktop app for its most-requested platform, landing about a month behind Anthropic's Claude.

• Zuckerberg's open-source manifesto: A 6,500-word letter argues advanced AI must stay broadly distributed, arriving with Meta's permissively licensed Muse Glimmer model.

The Toolkit

• Framer: AI-assisted site builder that turns a design into a live, responsive website without touching code.

• Dust: No-code platform for building custom AI agents that connect to your company's tools and data.

• Descript: AI audio and video editor that lets you edit recordings by editing the transcript.

Rate This Edition