AI coding agents can be tricked in under an hour, researchers find

Researchers have shown it takes less than an hour to trick a Fortune 500 company's AI coding agent into pinging a server it controls, not through hacking, but by simply registering a package name that documentation already pointed to.

The vulnerability centres on llms.txt and llms-full.txt, files an increasing number of websites now publish so AI agents can read their documentation more easily.

When researchers scanned 6,214 live domains belonging to defence contractors, Fortune 500 firms and major tech companies, they found 8,265 of these files.

Among them, 120 separate sites referenced code packages or domain names that had never actually been registered.

To investigate whether the flaw was possible to exploit, researchers registered some of the unregistered domain names themselves and offered software that would call home immediately after being installed.

The system belonging to a Fortune 500 company responded within an hour, followed by dozens of other organisations not much later. 

Unregistered domain names could have arisen due to typical reasons, such as abandoned software packages, copying errors, and renaming projects, but the researchers remarked that whatever could be registered by the researchers could be registered by cybercriminals.

This threat is actual in cases where the AI agent has access to run shell or package manager commands and faces such an old-fashioned or erroneous command.

In our tests, Claude, the Codex from OpenAI, and the Hermes from Nous Research turned out to be vulnerable to this erroneous documentation, as experts refer to this vulnerability as slopsquatting.