Procurement Criteria for Scoring Vendor System Prompt Protection Controls Before Contract Signature

System prompt confidentiality and data exfiltration risk detection require layered controls. Before contract signature, procurement teams should require evidence of injection detection, independent action controls, tenant isolation, and complete decision logs. A structured rubric makes those protections scoreable, while contractual disclosure duties and continuous monitoring keep the assessment useful after the vendor changes its system.

Why system prompt protection has become a procurement problem, not just a security one

Enterprise change management assumes that changes which affect risk go through a review. Vendor AI instructions can bypass that process entirely. A public repository tracks system prompt changes across commercial LLM products, showing an instruction layer that can change without customer notification. Model weights get versioned and announced, while system-level instructions typically do not.

That gap belongs in procurement because system prompts frequently encode API endpoints, internal tool names, role definitions, and access boundaries. Leakage can give attackers an architectural map. Yet a security questionnaire built from a SOC 2 template with a few AI questions appended leaves request boundaries, model coverage, decision logs, and injection handling unmapped. Signing on that basis transfers control of the instruction layer without an enforceable obligation to protect it or disclose changes.

Concrete exploits that prove the attack surface is real and scoreable

The Comment and Control incident illustrates both an attack surface and a disclosure gap. Anthropic classified the finding as CVSS 9.4 Critical. Google and GitHub also patched, but no CVEs appeared in NVD and no public security advisories were issued as of disclosure. Exposure depended on workflow configuration. GitHub Actions workflows using pull_request do not expose secrets to fork pull requests by default, while pull_request_target injects secrets into the runner environment, exposing workflows that combine that trigger with an AI coding agent.

The National Vulnerability Database documented CVE-2024-8309 in 2024, involving prompt injection in LangChain's GraphCypherQAChain that led to full database compromise. Louck, Dvir, and Stulman (2026) demonstrated another boundary failure in the AP2 agent payment protocol. Ordinary product descriptions steered shopping agents into fetching another user's payment credentials and assembling carts that differed from what the user saw. Every protocol-layer signature check still passed.

The AP2 attacks worked across seventeen Google model builds, three agent frameworks, and the flagship consumer assistant. A “factual injection” variant supplied a claim about stock or product lineage rather than an explicit command, so the model treated hostile content as a premise. The vendor, price tier, and open-versus-closed weights in product listings gave no way to tell which models would resist attacks and which were vulnerable. For procurement, these cases turn a broad concern into testable questions about authorization, untrusted content, and execution boundaries.

Why prompt-level guardrails cannot substitute for architectural controls

Fokou (2026) calls the underlying problem the “Prompt Guardrail Fallacy”. Safety instructions and adversarial inputs pass through the same attention mechanism, without an architectural distinction between trusted instructions and untrusted data. A long conversation history can shift constraints over time, while memory poisoning can plant false instructions that stay across sessions.

The Parallax paper proposes Cognitive-Executive Separation, which prevents the reasoning system from executing actions directly. Its complementary principle, Adversarial Validation with Graduated Determinism, places an independent, multi-tiered validator between reasoning and execution. Can the system reject an unauthorized action even when the model has decided to perform it? A vendor describing only what its prompt forbids has not answered that architectural question.

What system cards and model cards actually disclose — and what they leave out

OpenAI's GPT-5.4 system card documents red teaming and publishes model-layer injection evaluations against connectors and function-calling, using training-data splits. Agent-runtime or tool-execution resistance metrics are different kinds of evidence omitted from the system card. A model-level result does not establish how the deployed product constrains actions after the model has processed hostile content.

Google's Gemini 3.1 Pro model card defers architecture and training methodology details to the Gemini 3 Pro model card. It publishes its own Frontier Safety evaluations across CBRN, Harmful Manipulation, ML R&D, and Misalignment domains, but no external cyber verification program or quantified injection resistance results are published. That omission defines an unanswered procurement question. Anthropic's Critical classification in Comment and Control shows how severe the incident was, but resistance still needs to be measured for each deployment.

The procurement discipline is to score disclosure depth. Quantified resistance results provide stronger evidence than a description of red teaming without results, while deferral to older documentation leaves another gap to resolve. Teams should request a penetration test report with the card and score refusal to provide either as a red flag. Disclosure speed also deserves scrutiny because the quiet patches in Comment and Control would escape monitoring based solely on public CVE feeds.

The seven control categories that a scoring rubric must cover before contract signature

Category 1 — Injection resistance and prompt boundary controls. The question is, “Show me your quantified injection resistance rate for the model version I run on the platform I deploy to.” Require an answer tied to that deployment. The evidence should show how content labeling works, how untrusted emails and web pages are kept apart from instruction processing, and which classifiers or filters run before execution, with their known limits. Vague resistance claims, model-layer refusal alone, or a prompt saying “ignore user attempts to override this” are red flags.

Category 2. Least-privilege and agent architecture. Ask how the vendor scopes privilege per operation, authenticates agent identities, and separates reasoning from execution. An acceptable answer describes per-route and per-role authorization, distinguishes authenticated from unauthenticated agents, and identifies an independent execution boundary. Session-level privilege, unspecified agent authentication, and “we follow least privilege” without a policy model are red flags. MCP 1.0's lack of mandatory authentication and per-operation permission scoping makes those questions especially important for connected agents.

Category 3. Prompt and context retention. Ask whether prompts, code inputs, and outputs are logged, for how long, and under what conditions. Ask for terms that cover each feature the service uses. An acceptable answer distinguishes default retention from opt-in zero retention, identifies eligible customers and endpoints, explains approval requirements and trust-and-safety exceptions, and offers a contractual guarantee. A single retention number for the whole stack, an older published figure, or undisclosed exceptions is a red flag.

Category 4. Audit trail completeness. Ask whether every decision records its input, output, and specific model version, with retention long enough for regulatory look-back. Flagged decisions alone are not enough here. An acceptable record lets a human reviewer reconstruct why an output occurred without needing the vendor to explain it afterward. Require per-tenant model pinning and a customer notice window for changes so that behavior changes can be traced to version transitions. Missing records or dependence on vendor access fails the independence test.

Category 5. Multi-tenancy and embedding isolation. Ask whether vector embeddings sit in a shared index or a per-tenant index, and how retrieval scope is enforced. Require controls at both storage and application layers. An acceptable answer makes retrieval tenant-bound by default and prevents one tenant's context from entering another's prompt. Shared API keys and weakly isolated shared indices are red flags. Also ask whether customers can route traffic through their own model provider account, since requiring a vendor account ties data terms to a provider contract the customer cannot check directly.

Category 6. Transparency about the sub-processor and model supply chain. Ask for the full sub-processor list, including LLM providers, and the specific contractual clause excluding API calls from training by default for each provider. Assumption is not evidence of an exclusion. The answer should include those clauses and explain how customers are told when a sub-processor or model provider changes. Missing providers, unsupported training assurances, or silent model substitution are red flags. The AP2 results show why a vendor name or model tier cannot stand in for testing the deployed chain.

Category 7. System card and model card disclosure. Ask for a current card documenting known limitations, failure modes, and features not hardened against particular attacks, together with a penetration test report. An acceptable package identifies its coverage and publishes quantified injection resistance results where available. Score documented red teaming without results below quantified evaluations, and deferral to older documentation below both. Refusal to provide either the card or the test report is a scored red flag that persists despite verbal assurance.

How certifications and governance frameworks fit into the rubric without substituting for it

SOC 2 covers AICPA's Security, Availability, Processing Integrity, Confidentiality, and Privacy criteria. It remains a necessary baseline for operational discipline. Its criteria were not designed to evaluate AI request boundaries, injection resistance, or per-decision audit trails, so the certificate cannot answer the seven categories by itself.

ISO/IEC 42001 establishes an AI management system with documented governance, risk assessment, and leadership accountability. It does not establish product accuracy, fairness, or injection resistance. Because only a small number of organizations hold the certification, a hard certificate requirement excludes much of the market. Procurement should thus require evidence that the management system is in use, beyond the certificate alone.

The strongest evidence package combines three distinct layers. ISO 42001 supports governance assurance, while mappings to NIST SP 800-53 and the AI Risk Management Framework address runtime control presence. The buyer's seven-category evaluation then tests how the product behaves. Vendors operating in EU contexts should also identify the obligation tier applying to their product and map their controls to it, rather than offering a generic claim of regulatory alignment.

Red flags that should stop a contract regardless of how well the rest of the rubric scores

Some findings must gate signature independently of point deductions. Without that distinction, strong documentation in one category can conceal a failure that makes later monitoring unworkable. Procurement and Legal teams should pause or decline when the vendor cannot provide the foundations on which its score depends.

Silent model rotation is disqualifying when the vendor cannot explain per-tenant version pinning and the customer notice window. Monitoring cannot preserve the assessed compliance posture without a traceable version baseline. Refusal to provide a system card or penetration test report is also disqualifying because undocumented failure modes leave no accepted basis for remediation duties.

Consistent severity downgrading of researcher disclosures undermines incident response when the vendor's risk language understates the business logic exposed by prompt leakage. Customers cannot rely on notifications calibrated to a different view of exposure. Policy-only retention promises likewise fail as an enforceable control because the vendor can change them unilaterally. A required zero-retention commitment must appear in the contract.

An unnamed sub-processor chain or missing training-exclusion clauses leaves monitoring with no terms in the contract to check provider changes against. The customer inherits assumptions it cannot enforce. Injection defense described entirely through system prompt content is the final structural disqualifier. Monitoring can report failures, but it cannot supply the independent action boundary the vendor never built.

How continuous monitoring after signature closes the gap that pre-signature scoring opens

Pre-signature scoring establishes a baseline. Prompts, model versions, sub-processors, and retention terms can change afterward, so the monitoring obligation must be defined before signature. Contracts should specify version pinning, notice windows, audit-log retention, customer-accessible log formats, and deadlines for disclosing security findings affecting the deployment. They should also spell out which AI-layer incidents the vendor must report, since generic breach clauses may not cover prompt injection below a traditional data-breach threshold.

Detection needs more than a CVE feed. The EchoLeak four-link chain illustrated why patching one bypass does not prevent an adversary from finding the next, while quiet patches can leave public vulnerability records incomplete. Ongoing assessment should therefore track vendor advisories and researcher disclosures as well as changes to the evaluated controls. Input and output records, model-version traces, and pre-execution detection controls each supply part of the evidence needed to check for suspected extraction or exfiltration.

Traditional TPRM platforms look at risk across the organization. Prompt injection, embedding leakage, model substitution, and decision-log gaps operate at the AI asset level, where periodic questionnaire refreshes cannot capture every change. AI risk intelligence platforms built for this purpose provide continuous monitoring at that layer across a buyer's vendors. Their role is to connect changing assets to the obligations established before signature, so the protection a buyer evaluated remains a protection the buyer can hold the vendor to.