00Why this class is different
An internal AI assistant feels like a chat box, but underneath it is several systems wearing one interface. It reads from a knowledge base over retrieval, it holds connectors into ticketing, mail, documents and line-of-business systems, it often runs automated workflows triggered by events rather than people, and it does all of this under an identity that frequently carries more access than the person using it. Every one of those properties is an attack surface, and most of them are invisible from the chat window.
The defining risk is simple to state. These systems mix instructions and data in the same channel. A ticket, a document, an email or an uploaded file is data the assistant is meant to process, but the model reads it as language, and language can carry instructions. When the assistant also holds tools and runs without a human in the loop, a line of text inside a support ticket can become an action taken under a privileged identity.
This article sets out the methodology for red-teaming that class of system. It is distilled from assessments of internal AI assistants and LLM-backed workflow applications, fully sanitised, with no client or product named. It is written as a repeatable method, not a single engagement.
01Anatomy of the target
Before attacking, map what you are actually looking at. Most internal assistants and workflow apps are built from the same parts, and the trust boundaries sit between them.
- The interface and orchestration app. A normal web application and API that manages conversations, sessions, uploads and the calls to the model. Ordinary application security applies here in full.
- The model and its system prompt. The instruction hierarchy, the guardrails, and whatever context the app injects (role, user, available tools).
- The retrieval layer (RAG). An index over internal knowledge: documents, wikis, tickets, past conversations. The key question is whether retrieval respects the asking user's permissions, or the service's.
- Tools and connectors. The actions the assistant can take: query a database, create or update a ticket, send mail, call an internal API, fetch a URL. This is where language turns into consequence.
- Workflow automation. Event-triggered flows that chain model calls and tool actions, sometimes with no human approval between trigger and effect.
- Identity and multi-tenancy. Who the assistant acts as, how user and tenant boundaries are enforced, and whether sessions are isolated.
The assistant mixes instructions and data in one channel, then backs them with privileged tools. That is the whole problem, in one sentence.
02Methodology followed
Run as adversary emulation with concrete goals, not a feature checklist. The phases below layer the AI-specific vectors on top of a conventional application, API and identity assessment, because the chat box is still sitting on an ordinary web stack.
P0Scoping & attacker goals
Vectors framed · data exfiltration, unauthorised action, privilege escalation, workflow subversion, abuse
Define what an attacker actually wants from this assistant: read data belonging to another user or tenant, make the assistant perform an action the user is not entitled to, escalate through the assistant's identity, subvert an automated workflow, extract the system prompt or connector secrets, or run up cost. Agree data-handling rules for any real content touched. Every later test traces to one of these goals.
P1Surface & trust-boundary mapping
Vectors mapped · untrusted-input inventory, tool/permission inventory, human-in-the-loop analysis
- Inventory every place untrusted content can reach the model: direct prompts, uploaded files, retrieved documents, tickets, emails, calendar items, web pages the assistant fetches, and prior-conversation memory.
- Inventory every tool and connector and its real capability and blast radius, read versus write, internal versus external, reversible versus not.
- Mark which actions have a human in the loop and which fire autonomously. Autonomous plus untrusted input is the highest-value path.
P2Application & API security
Vectors · broken access control, IDOR on conversations/files, upload abuse, SSRF, rate-limit gaps
- Test the orchestration API the way you would any app: authentication, authorisation, object-level access control on conversations, files and histories.
- Probe file upload, attachment handling and any URL-fetch feature for SSRF and parser abuse.
- Check rate limiting and quotas on the model-facing endpoints, since these gate both abuse and cost.
P3Identity, authorisation & multi-tenancy
Vectors · RAG authorisation bypass, cross-user/tenant leakage, confused deputy, session isolation
- Test whether retrieval enforces the asking user's permissions. Ask for content the user should not see and watch whether the index returns it anyway because it is scoped to the service, not the person.
- Probe cross-user and cross-tenant boundaries: can one user's data, conversations or uploads surface in another's session.
- Look for the confused-deputy pattern: the assistant holds broad access and acts on a low-privilege user's request, becoming a privilege-escalation path by design.
P4Prompt-layer attacks
Vectors · direct injection, jailbreak, system-prompt & config extraction, guardrail bypass
- Attack the instruction hierarchy directly: override the system prompt, defeat guardrails, and extract the system prompt, tool definitions and any embedded configuration or secrets.
- Test refusal and safety behaviour under framing, role-play and encoding, and record how reliably each bypass reproduces rather than treating a single success as proof.
P5Indirect prompt injection through content
Vectors · poisoned documents, tickets, emails and files; RAG poisoning; delayed triggers
- Plant instructions inside content the assistant will later ingest: a support ticket, a knowledge-base document, an email, a shared file, so that processing the item executes the attacker's text.
- Test RAG poisoning: introduce a document into the indexed store that steers answers, overrides policy, or carries exfiltration instructions whenever it is retrieved.
- Test delayed and second-order triggers, where the injection lies dormant until a specific user or workflow pulls the content.
P6Tool & connector abuse
Vectors · excessive agency, missing per-action authorisation, argument injection, tool-chaining, SSRF via tools
- Test excessive agency: can a normal user drive a high-impact action (external mail, record deletion, config change) with no secondary authorisation.
- Test whether prompt injection can reach a tool call, turning a poisoned document into an action rather than just a wrong answer.
- Probe tool-argument injection and tool chaining, where one action's output feeds another to escalate, and URL-fetch tools for SSRF into internal services.
P7Workflow & automation subversion
Vectors · trigger injection, autonomous-action abuse, cross-workflow leakage, loop & cost abuse
- Target flows that run without a human in the loop: an inbound email or ticket that starts a workflow and carries injection straight into an action.
- Test cross-workflow data leakage, where context or output from one automation bleeds into another.
- Test loop and cost abuse: inputs that cause runaway chains of model and tool calls.
P8Output handling & downstream impact
Vectors · rendered-output injection, markdown/link exfiltration, unsanitised output consumed downstream
- Test how the assistant's output is rendered: markdown, HTML and links in a web UI can become stored cross-site scripting or a data-smuggling channel through auto-loaded image and link URLs.
- Test where output is consumed by another system without sanitisation, so the model's text becomes the next component's input.
P9Sensitive-data exposure
Vectors · over-broad retrieval, secret leakage, transcript/log exposure, memory bleed
- Test for over-broad retrieval that returns secrets, keys or personal data embedded in indexed content.
- Check whether transcripts, logs and memory retain sensitive content, and whether that store is itself exposed or under-protected.
P10Supply chain & integrations
Vectors · connector/plugin trust, third-party tool poisoning, model & dependency provenance
- Assess the trust placed in connectors, plugins and third-party tools, including whether a trusted integration can change behaviour after approval.
- Check provenance and change control for models, prompts and dependencies that feed the pipeline.
P11Detection, reproducibility & retest
Vectors validated · abuse monitoring, reliable-vs-one-off measurement, regression suite, fix verification
- Review whether prompt-injection attempts, anomalous tool use and bulk retrieval are logged and alertable, or invisible.
- Because behaviour is non-deterministic, re-run each candidate finding enough times to separate a reliable failure from a lucky one, and record the success rate.
- Capture confirmed payloads as a regression suite so the same injection and authorisation tests re-run against every release, and retest fixes with the owning engineers.
03Impacts this method surfaces
Representative outcomes, one line each, framed as the class of impact the vectors above tend to reveal. All generalised and sanitised.
| Vector | Representative impact | Typical severity |
|---|---|---|
| RAG authorisation bypass | Retrieve another user's restricted documents because retrieval is scoped to the service, not the asking user | Critical |
| Indirect injection to tool call | Instructions hidden in a support ticket make the assistant take an unintended action under its own identity | Critical |
| Excessive tool agency | A normal user triggers a high-impact action with no secondary authorisation | High |
| Autonomous workflow subversion | An inbound email drives an automated flow to execute attacker-controlled instructions with no human in the loop | High |
| Output-channel exfiltration | A poisoned knowledge-base document smuggles data out through an auto-loaded markdown image URL | High |
| System-prompt & config leakage | Direct injection reveals the system prompt, tool definitions and embedded configuration | Medium |
| Over-broad retrieval | Secrets or personal data embedded in indexed content returned in ordinary answers | Medium |
| Object-level access control | Another user's conversations or uploads reachable through the orchestration API | High |
04Defensive priorities
The method keeps pointing at the same handful of leveraged fixes.
- Enforce the user's permissions at retrieval. Scope every RAG query to the asking user and tenant, never to the service identity. This closes the highest-impact class above.
- Treat all ingested content as untrusted. Tickets, documents, emails and files are data, not instructions. Isolate them from the instruction channel and never let their text alone authorise an action.
- Put a human in the loop for consequence. Require explicit, out-of-band authorisation for high-impact or irreversible tool actions, especially inside autonomous workflows.
- Constrain tools to least privilege. Give each tool the narrowest capability the task needs, and apply per-action authorisation rather than a single broad identity.
- Sanitise output. Render model output safely and strip or gate auto-loading links and images that can exfiltrate; sanitise anything a downstream system will consume.
- Log and regression-test. Make prompt-injection and anomalous tool use visible, and keep a standing suite of injection and authorisation tests that runs against every release.