Agentic AI red teaming is often described as “like pen testing, but for AI.” That framing misleads buyers into expecting the wrong deliverable. Here is what we actually test, what the reports look like, and how to scope an engagement that produces useful findings.
What agentic AI red teaming is not
It is not a prompt injection scavenger hunt. It is not running OWASP LLM Top 10 payloads through a chatbot and screenshotting the responses. It is not asking the model to produce harmful content and reporting that it did.
Those exercises have their place in early LLM safety testing, but they do not test the thing an agentic system actually does: take autonomous action in the real world via tools, APIs, and privileges granted to it by the organization.
What we actually test
1. Tool invocation chains under adversarial input
We identify every tool the agent can call. For each one, we construct inputs — through the user interface, through documents the agent reads, through data returned by one tool and fed to another — designed to coerce the agent into using tools in ways that violate its intended policy. Example: a customer service agent with access to a refund API and a data export API. Can we induce it to export another customer’s data by framing the request as a refund query?
2. Privilege escalation across the agent boundary
Agents often run with service account privileges that exceed any single user’s privileges. We test whether a user interacting with the agent can cause the agent to perform actions on the user’s behalf that the user is not authorized to perform directly. This is the classic confused deputy problem, back in a new form.
3. Indirect prompt injection through retrieved content
Any agent that reads external data — emails, documents, web pages, support tickets — can be influenced by content embedded in that data. We plant injection payloads in each ingestion surface and measure what the agent does with them. A payload in a help desk ticket that causes the agent to escalate access, exfiltrate data, or take an unintended action is a critical finding.
4. Policy bypass through role-play, format, or context manipulation
We test whether safety policies and refusal behaviors hold under a range of adversarial framings. This is less about jailbreaking the model and more about whether the agent’s business policies (what it will and will not do on behalf of users) hold under pressure.
5. Supply chain and model substitution
If the agent uses third-party models, tools, or MCP servers, we test whether these dependencies could be compromised or substituted — and what the blast radius would be if they were.
What the report looks like
An engagement report from us includes:
- Executive summary for non-technical leadership — one page, risk-rated.
- Threat model of the agent system — what we considered in scope, what we did not, and why.
- Findings catalogue, each with: title, CVSS-style severity (adapted for AI contexts), affected tool chain, reproduction steps, business impact, and remediation.
- Prompts, payloads, and traces that produced each finding, so your engineering team can rebuild test cases.
- Architectural recommendations — frequently more valuable than the finding list itself. Agentic systems often have systemic issues (over-permissioned tools, no human-in-the-loop at critical junctions) that a finding-by-finding fix cannot address.
- Retest window — we come back and verify fixes within 60 days of the report at no extra cost.
How to scope the engagement
Three things make an agentic red team scoping conversation productive:
- A written list of tools the agent can call, with the privileges each requires.
- Rules of engagement: what systems are in scope, whether we test production or a staging environment, which users (synthetic or real) we may impersonate, and your data-handling constraints.
- Threat priorities: what outcome you most want to prevent (data exfiltration, unauthorized transactions, reputational misuse). This shapes test prioritization.
What to do this quarter
If you are deploying an agent that holds any production privilege, scope a red team engagement before go-live. Retrofitting controls after an agent is live and integrated is 5–10x more expensive than designing them in. For an agent already in production, start with the three highest-privilege tool chains and expand from there.
The Risk Labs runs agentic AI red teaming engagements for production LLM and agent systems. Scope a testing engagement.