Procurement Scoring Architecture for Dual OWASP LLM and NIST AI RMF Vendor Evaluation

AI vendor risk assessment mapped to OWASP LLM Top 10 and NIST AI RMF needs one scoring architecture integrating the two frameworks. OWASP supplies the attack classes, while NIST supplies the accountability, context, testing, and response requirements that turn them into procurement criteria. A defensible score combines both in evidence-backed criteria, applies pass/fail gates before ranking vendors, and stays current through continuous monitoring.

Why standard vendor security reviews fail for LLM-based products

A procurement team can read SOC 2 fluently and still struggle to ask a meaningful question about prompt injection. SOC 2 Type II assesses whether internal controls operate effectively within its scope. It does not establish whether someone can manipulate a model through its input channel. The distinction is a question of scope. A vendor can pass its audits while leaving serious model-layer vulnerabilities unexamined.

The interface itself explains why that gap exists. Conventional SaaS presents a deterministic interface, while an LLM processes natural language and produces outputs shaped by training data, system prompts, retrieval context, and input structure. Each creates opportunities for manipulation that an infrastructure review was not designed to assess. ISO certifications and penetration tests remain useful for the controls they examine, but they cannot substitute for a review of model behavior. Procurement therefore needs additional criteria to supplement its existing toolkit.

What OWASP LLM Top 10 and NIST AI RMF each cover — and where each falls short alone

OWASP LLM Top 10 gives procurement teams a vocabulary for specific model-layer risks. Its 2025 edition, released November 18, 2024, defines attack classes, gives examples, and describes prevention strategies. That makes questions about prompt injection or sensitive information disclosure more precise than a general request for “AI security controls.” OWASP is a nonprofit, community-driven foundation, rather than a vendor authoring its own evaluation standard. But an attack taxonomy cannot run a risk program.

NIST AI RMF supplies the organizational structure that OWASP lacks. Its four functions are GOVERN, which establishes policies and accountability, MAP, which identifies risk in context, MEASURE, which assesses risk, and MANAGE, which prioritizes responses and monitoring. These functions organize responsibility and evidence across a vendor relationship. They do not supply a model-layer attack taxonomy. A team could follow the functions and still never ask how a vendor handles indirect prompt injection.

Neither framework produces a vendor score by itself. OWASP alone leaves ownership and response arrangements unresolved, while NIST alone leaves the technical questions underspecified. Treating them as separate checklists preserves that gap because a policy answer need not connect to a tested control. The 2026 OWASP edition provides explicit mappings to NIST AI RMF and AI 600-1, among other frameworks. That documented bridge supports a common set of procurement criteria integrating security and governance.

How do you map AI vendor risk assessment to OWASP LLM Top 10 and NIST AI RMF?

The unit of assessment should be a procurement criterion that joins a named risk to required evidence. For each selected OWASP risk, GOVERN asks who owns the policy and approves exceptions. MAP asks where the risk surfaces and which components are involved. MEASURE asks what testing demonstrates the control’s effectiveness, while MANAGE asks how incidents and residual risk are handled. This is the crosswalk’s basic organizing principle. Each function examines a different part of the same risk.

For Prompt Injection, GOVERN calls for a policy on untrusted input, and MAP locates where that input reaches the model. Does the vendor tell direct instructions apart from malicious content that comes through retrieval or other indirect paths? OWASP’s 2025 document treats direct and indirect injection as distinct attack types with shared, layered mitigations. MEASURE therefore needs adversarial test evidence covering indirect vectors. MANAGE completes the criterion with a documented production incident path.

Sensitive Information Disclosure follows the same structure but requires different artifacts. GOVERN asks for data minimization rules covering training and inference, while MAP identifies personal, proprietary, or regulated data entering the system. System prompts also need to be checked when they hold business logic, API endpoints, or access secrets. There is no guarantee that they will stay secret forever. MEASURE needs evidence of training data curation and tests of output filtering, and MANAGE needs a procedure for responding to a disclosure.

Supply Chain moves the assessment beyond the vendor’s own code. GOVERN establishes a third-party AI component policy, while MAP requires a dependency graph covering base models, fine-tuning datasets, plugins, frameworks, and embedding APIs. MEASURE asks for a signed AI bill of materials and component verification evidence, including adapter-weight provenance where LoRA or PEFT adapters are used. Upstream dependencies belong inside the review boundary. MANAGE then asks what happens when an upstream component is found to be compromised.

Excessive Agency requires equally specific questions about authority. GOVERN defines the maximum permitted scope of autonomous action, while MAP identifies tools, permissions, and integrations, with a justification for each. OWASP distinguishes excessive functionality, excessive permissions, and excessive autonomy, so a generic answer about guardrails leaves the mapping incomplete. MEASURE asks how least-privilege is tested. It also asks what audit log captures actions. MANAGE asks about the kill switch and conditions requiring human approval.

The same method extends to other technical risks. Data and Model Poisoning connects source verification under MAP to backdoor testing under MEASURE and re-evaluation procedures under MANAGE. For Improper Output Handling, the review links where output goes downstream to the procedures for output validation, encoding, and response. The artifacts change with the risk being assessed. GOVERN draws on policy documents and records of who has each role, MAP on data maps, MEASURE on test results, and MANAGE on incident reports or records of residual-risk acceptance.

Structuring the scoring table: weighting, pass/fail gates, and a 0–1–2 evidence scale

Vendors should pass the pass/fail gates before they get a weighted score. Routing buyer prompts or documents through an unreviewed third-party LLM API can disqualify a vendor regardless of strengths elsewhere. Each gate needs a binary condition and defined evidence, such as a signed inventory of every AI component processing buyer data. TPRM, InfoSec, and Legal should agree these conditions before RFP issuance. Late gates create avoidable problems in negotiation.

For vendors that pass, every criterion uses the same evidence scale. A 0 means absent, refused, or evasive, while a 1 means partial, documented but not demonstrated, or promised for a future release. A 2 requires reviewer-accessible evidence, such as a test report, signed SBOM, audit log sample, or monitoring configuration export. A confident answer is not a demonstrated control.

Weights should vary with how much risk the buyer faces. An agent writing to production systems warrants more emphasis on Excessive Agency than a read-only summarization tool. AI asset discovery, the quality of audits and evidence, and agentic governance should each carry enough weight to affect that score. The proof-of-concept requirement is a crucial checkpoint. Roughly the last fifth of total scoring should come from a reproducible run on buyer-supplied test sets, using the same rubric as the written response.

Supply Chain appears in OWASP and in NIST GOVERN 6, MAP 4, and MANAGE 3, but it should have one row that uses evidence from both frameworks. Data handling also needs one criterion, with sub-criteria for training, inference, and output. Any higher weight should be deliberate and visible. Otherwise, duplicate rows quietly distort the ranking.

TPRM assesses organizational accountability and dependency documentation, InfoSec assesses testing and technical controls, and Legal assesses data handling, disclosure, and residual-risk acceptance. Disagreements require a reconciliation meeting. A technical 0 must remain visible as a failure. A policy score of 2 cannot demonstrate that the missing technical control works.

How agentic AI changes the scoring requirements for Excessive Agency and supply chain

An agentic product makes the distinction between policy and tested behavior more urgent. When a system reads mail, queries databases, and makes API calls, it can compress or remove the human review step between generated text and action. Manipulated input can therefore lead to an irreversible operation. The scoring table needs an explicit agentic-capabilities flag. A positive answer makes agentic criteria mandatory.

MAP must then require an enumerated inventory of tools, APIs, and permissions, with a justification for each permission. MEASURE must include multi-step adversarial scenarios in which injection at one stage propagates to a later tool call. Single-turn tests cannot show that this boundary is protected. MANAGE needs a documented kill switch with trigger conditions, tested recovery procedures, an identified operator, a response window, and an account of preserved or lost state. Agent-to-agent interaction or use of the Model Context Protocol also requires a distinct communication-scope criterion.

Supply chain evidence must expand with the architecture. The inventory needs base model weights and their source, fine-tuning adapters and training data provenance, retrieval sources and integrity checks, and the connectors through which the agent acts. Naming a foundation model provider is not enough. A vendor unable to supply signed adapter hashes or explain plugin vetting scores 1 on that criterion, below a score of 2. This keeps component familiarity from being mistaken for verified provenance.

OWASP Top 10 for Agentic Applications, published December 9, 2025, adds the agentic-specific assessment layer. Its tool misuse entry maps to Excessive Agency and addresses legitimate tools used harmfully. The broader agentic list also addresses risks across multi-step workflows. Both lists should be used when reviewing an agentic vendor. Their shared risks should still feed consolidated criteria rather than become another pair of disconnected checklists.

The shadow-AI problem: extending the scoring architecture to the full vendor population

Even a well-built score fails if the vendor never enters the assessment. AI features now appear inside productivity tools, project management platforms, communication tools, and other SaaS contracts that were not purchased as AI products. These features send the organization’s data through an AI layer, but BAA and DPA reviews often focus on top-tier AI platforms. The embedded long tail remains comparatively difficult to see. Calling only the obvious platforms “AI vendors” leaves the inventory narrower than the exposure.

The intake remedy is to flag every contract with AI features and apply an AI-vendor review gate to flagged contracts. NIST MAP 4 calls for risk mapping of every part of an AI system, including its third-party components. That means knowing which SaaS vendors use AI to process the organization’s data. Initial intake cannot settle the question permanently. A document management platform can add summarization after signature, introducing disclosure and output-handling exposure outside the original review.

NIST MANAGE 3 calls for third-party risk management to keep going within the same architecture. Along with periodic reassessment, a review is therefore needed when a model version changes, sub-processors change, or an incident affects a shared foundation model. AI risk intelligence platforms built for this task fill the monitoring gap by tracking vendor AI assets across the ecosystem, including those of vendors that were not labeled AI at onboarding. Their role is broader visibility over time. That monitoring supplies the changes the scoring process needs to reassess, rather than treating a procurement-time result as permanent.

Making the score defensible to procurement, legal, and regulators

Procurement needs a ranked comparison it can act on. The score shows where a vendor is weak, whether in supply chain documentation, agentic shutdown controls, or another defined criterion, and keeps those differences visible within the total. It also supports a concrete discussion about what the contract must require before go-live. Vendors must still pass the gates even when they score well overall. A promise made in negotiations cannot count as demonstrated evidence just because there is pressure to close the deal.

Legal needs the documents beneath that comparison. GOVERN evidence links policies, who is responsible for what, risk appetite, and standards for exceptions to the representations and warranties the parties are negotiating. Texas’s TRAIGA treats documented NIST alignment as a factor in affirmative defense. That makes the documentation trail consequential when accountability is disputed. For regulatory scrutiny, MANAGE artifacts such as approvals, blocked action logs, incident reports, and residual-risk acceptance records show a functioning process.

The architecture itself also needs version control. OWASP entry numbers can mean different things in different editions, so numbered references in compliance documents should include the year suffix, and each scoring version should name the framework editions it uses. Framework updates require review before rollover. Vendor changes require the same continuing attention. A procurement score is a starting point, and continuous monitoring keeps its evidence and conclusions defensible as models, sub-processors, and features change.