Scoring Vendor LLM Deployments for System Prompt Leakage Detection Maturity

System prompt confidentiality and data exfiltration risk detection in vendor LLM deployments require five layers of assessment, covering prompt hygiene, infrastructure isolation, detection and response, agentic permissions, and continuous monitoring. A classifier alone cannot establish maturity. The strongest assessment asks for evidence across all five dimensions and prevents strong detection scores from concealing structural weaknesses.

Why most vendor LLM deployments have no reliable way to contain system prompt leakage

Yang et al. (2026) found that over 80% of measured real-world LLM-based applications across six major commercial platforms leaked system prompts under realistic adversarial queries. The exposed material frequently included developer identities, third-party service API keys, and instructions about how an assistant should answer. These were not merely laboratory failures. Alibaba and Baidu officially classified the reported issues as medium-severity vulnerabilities after responsible disclosure. The finding makes leakage a concrete assessment problem for buyers, rather than a theoretical concern to acknowledge in a questionnaire.

The underlying mistake is treating the system prompt as a secure container. It is part of the model’s input context, where adversarial queries can target it. OWASP explicitly says that the system prompt should not be considered a secret or used as a security control. That distinction changes what protection means. Even when an attacker cannot recover the exact text, watching how the model behaves can reveal its guardrails and operational logic. A useful maturity score must therefore show both how likely disclosure is and whether the deployment puts anything security-critical within reach.

How does the attack surface expand beyond someone asking for the system prompt?

Application-layer extraction starts with crafted queries that get a model to repeat its instructions. Yang et al. identify “attention drift” as a mechanism, in which query-key alignment bias and softmax amplification progressively pull the model away from defensive constraints. Hardening instructions can therefore weaken over a conversation. Kevin Liu’s extraction of Microsoft Bing Chat’s entire system prompt and hidden codename “Sydney” in 2023 illustrates the direct route. A related failure in an email assistant allowed crafted prompts to expose its system prompt and cause it to execute unwanted prompts, leading to unintended data disclosure.

Infrastructure leakage takes a different route entirely. A 2025 study of KV-cache timing attacks reported a 99–100% attack success rate on shared-cache inference backends, where an adversarial tenant could reconstruct another tenant’s private prompt through timing differences. No adversarial prompting was needed for that attack. The weakness sits in the shared inference infrastructure. A buyer cannot establish protection against this vector by inspecting a prompt filter, and may have no visibility into the backend unless the assessment explicitly asks about it.

Indirect injection enters through material the model reads, including documents, emails, webpages, and tool outputs. The Model Context Protocol ecosystem SoK describes how MCP can turn an LLM from a passive text processor into an active system component with shell-level privileges. EchoLeak in Microsoft 365 Copilot (2025) used a hidden prompt in an ordinary email to steer the assistant into exfiltrating sensitive internal data during routine work. LangChain GmailToolkit CVE-2025-46059 likewise involved data exfiltration through crafted emails, although the supplier disputes arbitrary code execution. These distinct routes explain why a chatbot-focused checklist can underestimate an agentic vendor’s exposure.

Why detection-only scoring overstates a vendor’s actual leakage posture

Detection and resistance are different capabilities, even when a vendor presents them together. Input classifiers, prompt shields, and output filters can show that a vendor has taken steps to defend the system without proving that attacks will fail. Researchers have achieved complete evasion against Azure Prompt Shield and Meta Prompt Guard. That finding concerns a class of control. For buyers and third-party risk management teams, the important question is whether a claimed detection capability has survived bypass testing, rather than whether the vendor can demonstrate a dashboard.

Yang et al. found that existing defenses failed to prevent leakage without degrading usability. Tightening the defense can impair the product, while loosening it leaves room for attacks. More detection is not automatically better protection. An “impossibility-style argument against data-instruction separation defenses” in 2026 peer-reviewed literature also suggests that agents may remain susceptible when defenses rely on separating instructions from data within the same context. That is a reason to question absolute assurances while recognizing that defensive measures can still have value.

The practical resolution is to place detection within a layered defense. OWASP favors enforcing sensitive constraints through systems outside the LLM, because attacks can alter how the model follows its instructions. Authorization should not depend on the model’s judgment. Under that approach, a vendor that removes credentials from prompts and moves authorization into deterministic external systems is more mature than one that applies a sophisticated classifier to an insecure prompt. The assessment must reward that architectural difference, even when the classifier produces the more impressive demonstration.

What the major maturity frameworks actually measure — and where they agree

The SANS AI Security Maturity Model (May 2026) has five stages from Unaware/Ad Hoc through Optimizing/Adaptive and uses a 0–5 evidence scale to score the answers to assessment questions. Unsupported self-reports cannot score above 2. Its Governance Floor Rule prevents overall maturity from exceeding one stage above the Govern pillar, while its Minimum Pillar Rule imposes the same limit relative to the lowest-scoring pillar. The model explicitly places a smaller organization at a documented Stage 2 ahead of an enterprise claiming Stage 3 without supporting documentation.

The OWASP Enterprise Adoption Maturity Model (June 2026) treats the type of deployment and how well it is governed as separate measures. Its two-axis grid spans unmanaged Shadow AI, vendor-embedded assistants, and federated cross-organizational agent networks, producing 45 posture cells. The AT classifications distinguish what an organization runs, so a bounded assistant and an agent with external integrations do not receive the same risk interpretation. AT6 and AT7 at low governance levels are marked critical gap, while AT8, Federated/Cross-Boundary, is marked do-not-deploy below Governance Level 3.

The University of Washington LLM Compliance Framework uses a 30-control library across six risk domains, scored 0–5 and anchored to the NIST SP 800-53A evidence hierarchy. Its consequential boundary is between 2 and 3, separating informal partial implementation from formal, consistently applied controls. A passing threshold therefore has real consequences. Accepting a 2 means accepting a control that has not crossed that boundary. Read together, the frameworks support evidence over assertion, limits on how much strong areas can offset weak ones, and assessment that accounts for deployment scope. They provide precedents for a leakage rubric.

The five dimensions that together constitute leakage detection maturity for a vendor deployment

Dimension 1 is prompt hygiene and credential separation. It asks whether the system prompt contains API keys, credentials, internal architecture, role-based permissions, or fraud-detection logic. This is the assessment’s non-negotiable starting point. OWASP’s baseline is that sensitive functionality belongs in deterministic external systems, while prompts provide general behavioral guidance. The evidence should include architecture documentation or audit findings that show credential management and authorization are outside the LLM context. A high score elsewhere cannot make up for credentials exposed through the prompt.

Dimension 2 is inference infrastructure isolation, which deals with a weakness that application controls cannot fix. The key distinction is whether the backend isolates caches per tenant or uses a shared KV cache. This question belongs with the infrastructure owner. Root salting eliminates cross-principal prefix sharing, but imposes a cache-hit-rate cost that the vendor must be willing to absorb. A documented cache isolation policy supports a score, while an undocumented or “default shared” answer indicates low maturity. Most vendor questionnaires omit this question, leaving the timing side channel outside the assessment altogether.

Dimension 3 is detection and response, assessed within its proper scope. It covers input validation, output filtering, prompt injection classifiers, and behavioral monitoring, supported by published SLAs, documented response timelines, audit trails, and explainable workflows. Putting a control in place does not by itself prove that it works. Evidence of bypass testing is essential to a high score, and classifier claims without it should be capped at the equivalent of 2 in this rubric. OWASP’s 2025 edition recommends defense in depth through input validation, output filtering, privilege restrictions, and human-in-the-loop controls for sensitive operations. Those protections complement one another and the first two dimensions.

Dimension 4 covers the scope of an agentic deployment and how its tool access is governed. It distinguishes a bounded assistant from an agent with RAG and tool access, or an MCP-connected agent with external server integrations. What can the model actually cause to happen? The assessment should check for least-privilege tool permissions and runtime intent verification before a tool runs, and ask whether the threat model covers indirect injection through external content. Under this proposed rubric, MCP-connected agents below governance level 3 should get a do-not-deploy decision, which is stricter than the grid’s critical-gap label for low-governance AT6 deployments. This dimension prevents a simple chatbot questionnaire from approving a system whose permissions make injection far more consequential.

Dimension 5 is continuous monitoring and evidence generation. It assesses whether production monitoring detects emerging threats, behavioral drift, and new injection vectors, while producing audit-ready records of AI asset behavior over time. A static certificate cannot demonstrate adaptive control. The adoption model’s highest governance tier instead requires telemetry-driven operation. Dedicated AI risk intelligence platforms can provide continuous monitoring across a vendor ecosystem where generic governance frameworks and traditional TPRM tools leave this work undone. Their relevant capability is not merely another score, but evidence that remains useful as models, plugins, and agent configurations change after deployment.

How to apply the scoring dimensions in a vendor assessment — and what a low score actually signals

For TPRM, information security, Privacy, and Legal teams, applying the rubric starts with a common evidence standard. Any dimension supported only by a policy assertion is capped at 2, regardless of how sophisticated the claim sounds. The proposed composite score is capped at the lowest dimension, a deliberately stricter limit than the SANS allowance of one stage above the weakest pillar. The purpose is to prevent excellent detection from offsetting an undocumented isolation policy or insecure prompt architecture.

For Dimension 1, teams should request architecture documentation showing external credential and authorization handling, plus third-party-reviewed data flow diagrams. For Dimension 2, they should request tenant cache isolation documentation and any third-party audit of the multi-tenant architecture. Detection evidence must show that the control works once it is installed. Dimension 3 therefore calls for incident response SLAs, logs or reports that cover a set period, and bypass-test results.

Agentic and monitoring evidence need separate checks because they show powers and changes that a prompt review cannot capture. Dimension 4 calls for a tool-permission manifest or least-privilege policy, plus runtime intent verification documentation when MCP or equivalent integrations are used. For Dimension 5, teams should ask for continuous monitoring reports and samples of compliance evidence that cover a review period. These records allow evaluators to distinguish a consistently applied control from a promise about how the system should behave.

A score of 2 or below on prompt hygiene signals structural exposure, regardless of the detection story. The same result for agentic governance warrants a do-not-deploy posture under this rubric when the vendor operates agentic deployments. Neither weakness can be averaged away safely. A low monitoring score means the buyer cannot see what changes after the contract is signed. Internal maturity and external vendor readiness also need separate checks, because good governance by the buyer does not prove that a vendor can provide the controls it claims to have.

The reassessment cadence should reflect that vendor AI does not stand still after approval. Model updates, plugin additions, and agent configuration changes can make a point-in-time assessment stale. Continuous monitoring keeps the scores for the other dimensions up to date between reviews. Formal reassessment should therefore recur, with monitoring evidence showing what has changed since the previous decision. The score remains useful only while its supporting evidence describes the deployment the buyer is actually using.