A new proof-of-concept attack called "Friendly Fire" demonstrates a troubling vulnerability in AI-powered security tools: when asked to scan open-source packages for malicious code, popular AI coding agents can be manipulated into executing the attacker's payload on the analyst's own machine.
The research, published by the AI Now Institute, targets AI coding assistants that have been repurposed as security scanning tools — a common practice among developers looking to leverage large language models for automated code review.
How Friendly Fire Works
The attack is a variant of prompt injection, but with a particularly dangerous twist. When a developer instructs an AI agent to review a potentially malicious package, the attacker embeds hidden instructions within the code or documentation of that package. These instructions hijack the AI's reasoning, redirecting it from "analyze this code" to "execute this code."
The attack chain works as follows:
- Attacker publishes a malicious package to npm, PyPI, or another registry
- Hidden prompt injection is embedded in comments, README files, or code strings
- Developer asks an AI agent to scan the package for threats
- The AI agent reads the injected instructions and interprets them as legitimate task directives
- The agent executes the malicious payload — on the developer's machine, with the developer's permissions
Affected Agents
The researchers confirmed the attack works against several leading AI coding agents:
- Anthropic Claude Code — CLI-based agent for coding tasks
- Google Gemini CLI — command-line AI assistant
- OpenAI Codex / ChatGPT with code interpreter
The vulnerability stems from a fundamental challenge in agentic AI: agents with tool-use capabilities (file system access, shell execution, network calls) cannot reliably distinguish between instructions from their authorized user and instructions embedded in the content they are analyzing.
Why This Is Significant
Traditional static analysis tools read code — they do not execute it in response to instructions embedded within it. AI agents blur this line. When an agent has both the ability to read code and the ability to run shell commands, a sufficiently crafted prompt can bridge the two.
This attack is especially dangerous in CI/CD pipelines and automated security workflows where:
- Agents run with elevated permissions
- There is minimal human oversight of individual tool calls
- The output of one AI agent feeds into the next
Mitigations
Until AI agent frameworks implement stronger sandboxing and instruction-source validation, security teams should:
- Never run AI agents with shell execution capabilities against untrusted code without strict sandboxing (e.g., Docker containers with no network access)
- Treat AI security scans as advisory — not as a substitute for deterministic static analysis tools
- Audit agent tool call logs to detect unexpected command execution
- Apply the principle of least privilege — agents should not have write or execute permissions unless absolutely necessary
- Use separate environments — scan suspicious packages in isolated VMs, not developer workstations
The Broader Implication
Friendly Fire exposes a category of risk that will grow as AI agents take on more autonomous roles in software development and security operations. The same capabilities that make these agents powerful — reading files, running commands, accessing APIs — make them dangerous when their inputs can be manipulated.
As AI-assisted security tooling matures, the industry will need formal standards for agent sandboxing, instruction provenance verification, and human-in-the-loop checkpoints for high-risk operations.
For now: scan untrusted code the old-fashioned way before letting an AI agent anywhere near it.