AI Application Penetration Testing

Prompt injection, direct and indirect

• ▸Direct injection: user input overriding system instructions or extracting them
• ▸Indirect injection: instructions hidden in retrieved documents, web pages, emails or file uploads the model reads
• ▸Injection through data the application trusts because it produced it: summaries, tickets, previous turns
• ▸Instruction persistence across a session, and whether a poisoned turn contaminates later ones
• ▸Multi-language and encoding tricks that bypass naive input filtering

Tools, functions and MCP servers

• ▸Whether a tool can be invoked with arguments the user should not be able to supply
• ▸Tool authorisation: does the tool re-check the caller's permissions, or trust the model's intent
• ▸Chained tool calls reaching further than any single call was meant to
• ▸MCP server exposure: authentication, scoping and whether one tenant's server can be reached by another
• ▸Server-side request forgery through tools that fetch URLs on the model's behalf

Retrieval and data boundaries

• ▸Cross-tenant retrieval: whether a query can surface another customer's documents
• ▸Filter enforcement at the vector store rather than in the prompt
• ▸Embedding and index poisoning through user-supplied content
• ▸Sensitive data in retrieved context that the user is authorised to ask about but not to see
• ▸Conversation history and cache leakage between users or sessions

Output handling

• ▸Model output rendered as HTML or Markdown, producing XSS from generated content
• ▸Generated SQL, shell commands or code executed without validation
• ▸Output used in authorisation decisions ("the model said the user is an admin")
• ▸Downstream systems trusting AI-generated content as pre-validated

The ordinary application underneath

• ▸Authorisation on every endpoint the AI feature exposes, per role and per tenant
• ▸API keys for model providers in frontend bundles, logs or error output
• ▸Rate limiting and cost abuse: expensive generation triggered by unauthenticated callers
• ▸Prompt and response logging that stores customer data somewhere it should not be

Is AI security actually different, or is this a rebrand?

Two things are genuinely new. Untrusted text can become instructions, which breaks the old assumption that data and code are separable — and it arrives through content your application fetches, not only through the input box. And the model can act, through tools, with whatever authority you granted them.

Everything else is ordinary application security wearing new vocabulary, and it is where most findings land: authorisation per tenant, secrets handling, output encoding, rate limiting. A team that treats AI security as a novel discipline and skips the boring half ships the same IDOR everyone else does, just with an LLM in front of it.

Can prompt injection be fixed?

Not reliably at the prompt layer, and anyone promising otherwise is selling. Instruction and data share a channel, so filtering raises the cost of an attack without closing the class.

What does work is assuming injection succeeds and constraining the consequences: tools that re-check the caller's authorisation rather than trusting the model, retrieval filtered at the store rather than by instruction, output treated as untrusted input, and human confirmation on actions that move money or data. Then a successful injection produces an odd answer instead of an incident — which is the achievable goal.

What about MCP servers and agents?

They concentrate the risk, because an agent with tools is a system that acts on instructions it received from somewhere. The questions worth asking are unglamorous: which credentials does the server hold, whose authority does a tool call carry, can one tenant reach another tenant's server, and is there an audit trail of what the agent actually did.

The recurring finding is a tool that trusts the model's intent instead of re-checking the user's permissions. That is not an AI flaw; it is a missing authorisation check in a new place, and it is testable the same way any authorisation flaw is.

Do you need access to our model or training data?

No. We test your application: the endpoints, the tools, the retrieval layer and the surrounding authorisation. That is where the exploitable findings are, and it needs nothing but a URL and credentials for each role.

Model-level work — weight extraction, training data reconstruction, alignment evaluation — is a different discipline and out of scope. If your model provider is OpenAI, Anthropic or Google, their infrastructure is their responsibility and we do not test it.

Each finding with severity, the exact prompt or request that triggers it, what it reached, and the control that breaks the chain.

What every tool can do and whose authority it acts with — usually the first document that makes the risk legible to the people who built it.

Poisoned content to injected instruction to tool call to data, documented end to end rather than as three unrelated medium findings.

What closed, what regressed and when, which matters more in AI features because prompts and tools change weekly.

Obligations phase in from 2025-2027 by risk class; security testing evidence supports the risk-management and robustness expectations for higher-risk systems

CC6.1: logical access controls, which now include what your tools and agents can reach

Article 32: regular testing of technical measures, including anything that sends personal data to a model provider

A.12.6: technical vulnerability management, extended to the AI surface

Not a regulation, but the reference taxonomy most reviewers expect you to have mapped against