Researchers Identify Core Weakness That Exposes Large Language Models to Manipulation

When we converse, I know which words are mine because I feel my mouth move, Cui explains. In contrast, a language model receives a single, uninterrupted flow of tokens—user prompts intertwine with prior outputs, internal notes, and copied text. “It’s essentially one continuous sheet of tokens,” she notes.

To distinguish speakers, chatbots insert role‑based markers into the token stream. User input is wrapped in tags, while the model’s replies go inside tags. System‑level instructions from designers sit in tags, the model’s own chain‑of‑thought appears in tags, and any material drawn from external sources—such as a web page or another agent—is placed in tags. (Cui notes that these labels correspond to OpenAI’s convention; other providers may use different terminology, but the function remains identical.)

Role tagging forms the basis of current defenses against LLM attacks, since most exploits try to convince the model that an instruction originated from an unauthorized source. For instance, many jailbreaks—where a user coaxes the model into producing disallowed output—succeed by presenting text as if it were or content. Likewise, prompt‑injection attacks, in which an adversary injects new directives, rely on the model treating text as though it were , , or material.

When developers train models to withstand such attacks, they focus on teaching the system to recognize instructions that appear in inappropriate contexts.

But Cui and her team discovered that LLMs struggle considerably to track role boundaries. In a series of experiments probing the internal workings of several models, they observed that the models determine a segment’s role not from its surrounding tags but from its linguistic style and vocabulary.

The experiments showed that exchanging tags—substituting for tags, say—had virtually no effect on the model’s interpretation. When a passage resembled the model’s own chain‑of‑thought, the LLM treated it as genuine chain‑of‑thought content, and the same pattern held for the other role categories.

Weak link

According to the researchers, an attacker need only craft text that mimics a particular role to compromise an LLM. Since role handling is intrinsic to how these models operate, training alone cannot completely eliminate the vulnerability.

“I really like this paper,” comments Florian Tramèr, a computer scientist specializing in LLM security at ETH Zürich. He adds that the attack concept is exceptionally clever.

Source link