Benchmarking AI Vendor Security Responses Against Peer Disclosures

You need a structured framework to benchmark AI vendor security disclosures against peer responses, because the shared baselines procurement teams used to rely on are disappearing at once. A team that wants to detect whether a vendor's system prompt is vulnerable to extraction, or whether adversarial queries are probing a deployed LLM for exfiltration paths, needs more than a vendor's self-reported policy: it needs criteria that hold up across peers even when those peers define "vulnerability" differently. This piece lays out those criteria, built from how AI vendors actually disclose, delay, and sometimes ignore the risks specific to their systems.

Comparing AI vendor security disclosures

A procurement analyst sits down to compare two AI vendor security responses. Both have disclosure policies. Both publish advisories. The instinct is to line them up side by side, score the volume, score the speed, and pick a winner. That instinct runs into a problem: the assumptions behind it, a common scoring system, a shared disclosure vocabulary, a comparable policy scope, are all unreliable right now, and they are unreliable at the same time.

Start with the number buyers have leaned on for years. Most CVEs published after April 15, 2026 arrive without a NIST-validated CVSS score or CPE list. Many still carry a vendor-supplied score, so the number isn't gone entirely, but the single validated figure that let teams normalize severity across vendors has been stripped out of the majority of new entries, knocking out the first leg of the comparison.

The second leg is the policy vacuum. Many vendors you compare against each other aren't even playing the same game. The same study identified three distinct postures among vendors that do publish something: proactive clarification, silent, and restrictive. Those postures produce disclosures structurally different enough that a raw side-by-side count misleads more than it informs.

Someone might reasonably ask why this moment is different. Speed is the answer. When the volume and velocity of disclosure both double inside eight months, misreading a peer comparison turns a quarterly planning problem into a days-long exposure window.

AI-specific vulnerabilities versus the traditional software flaws disclosure policies were built for

Standard vulnerability disclosure infrastructure was built for deterministic software flaws: a buffer overflow behaves the same way every time it's triggered, and a patch either closes it or doesn't. Neither of those categories fits comfortably inside the taxonomy that most disclosure policies were drafted against.

The Edinburgh study found that jailbreaking and hallucination are the vulnerability types most commonly declared out-of-scope by AI vendors. A vendor can look like it has a mature disclosure program while routing the findings that matter most into a gap with no defined process.

That gap isn't a drafting oversight. So the two responses become structurally incomparable, even though the vendor appears, on paper, to run a disclosure program.

Research on agentic commerce platforms sharpens the point further. No existing CVE taxonomy cleanly captures this class of finding.

The agentic layer adds a further complication. No framework currently specifies who is entitled to those traces or who bears responsibility for disclosing them.

The risk profile compounds on top of the taxonomy problem that generic policies struggle to classify. GTIG found that AI-assisted vulnerability discovery turns up proportionally more medium- and high-risk findings, and more remote code execution outcomes, than the broader CVE ecosystem does. AI-era disclosures skew toward exactly the findings that generic policies handle worst.

If a platform runs continuous assessment of vendor AI deployments, scoring them against a dedicated AI risk management framework instead of a generic disclosure checklist, it can show which vendors have built genuine AI-specific risk architecture and which have just retrofitted a traditional software policy onto an AI product.

The collapse of the shared scoring baseline in vendor benchmarking

Once CVSS scores and CPE lists stop appearing on most new CVE entries, the instrument that let buyers normalize severity across vendors disappears with them. What fills that space is vendor-supplied severity claims, the very thing benchmarking exists to check.

Raw CVE volume, taken without threat-intelligence context, now actively misleads. A vendor whose product touches the Linux kernel will look far more exposed in a raw CVE count than in any view adjusted for actual risk.

Black Kite's 2026 Supply Chain Vulnerability Report puts a number on how wide that gap runs. The space between disclosed volume and actionable risk is the dominant feature of the landscape a benchmarking exercise has to navigate.

The timing has inverted too. The patch lifecycle enterprises have used for years to measure vendor responsiveness runs against the disclosure date, so now it runs backwards. The metric that matters is a vendor's response speed relative to exploitation, not relative to when the advisory went public.

CISA's Known Exploited Vulnerabilities catalog looks like an obvious substitute baseline, and it has grown: Black Kite found KEV additions expanded 32% year-over-year, roughly 20% cumulative catalog growth. KEV is a lagging indicator, useful for triage, but not built to benchmark vendor performance in advance.

The practical effect of all this is a reversal of a long-standing assumption. A vendor that publishes a well-structured advisory carrying its own internal severity rating now looks more transparent than one that publishes a CVSS score derived from an unscored NVD entry. Now, in a landscape where the standardized score often isn't there to begin with, a vendor's willingness to publish and explain its own rating carries more information than the absence of one borrowed from an infrastructure that stopped enriching it.

The breakdown of the 90-day coordinated disclosure window under AI-speed vulnerability discovery

The 90-day coordinated disclosure window has functioned for years as the industry's default benchmark for vendor responsiveness. AI-assisted discovery compresses that timeline to hours in many cases, which turns the 90-day anchor into a comparison point that no longer reflects how fast the underlying work actually moves.

Some vendors have adapted the window to a compressed form. For actively exploited critical vulnerabilities, Anthropic targets a compressed 7-day timeline instead. That dual-track structure, a slower public default paired with a fast track for active exploitation, is a more differentiated standard than most peers publish.

The limits of any embargo are visible in the "Dirty Frag" case. If multiple parties can independently find the same flaw with AI assistance, that kind of break is structurally likely, not a rare accident. An embargo timeline, however carefully negotiated, is aspirational once the underlying research can be replicated by a stranger in an afternoon.

The Cloud Security Alliance's critique goes past timeline length into the structure of the norm itself. They don't say how a timeline should be calculated when vulnerabilities arrive in machine-volume batches. They say nothing about findings that produce no CVE, no advisory, and no public record at all, three conditions that occur routinely in AI-assisted discovery programs.

Anthropic's Project Glasswing illustrates exactly that last gap. A large and growing share went to maintainers under coordinated-disclosure timelines, and unpatched findings were held under cryptographic hash commitments pending fixes, coordinated across more than 40 additional early-access organizations. If a team only counts visible advisories, it will underrate a vendor running a serious coordinated disclosure program and overrate one that simply publishes everything immediately.

Strong AI vendor disclosure across the criteria that differentiate peers

So counting advisories or comparing CVSS averages won't give you a meaningful peer comparison. A benchmarkable AI vendor disclosure posture breaks down into five distinguishable dimensions: policy scope, channel specificity, timeline differentiation, agentic coverage, and data-handling commitments. Vendors that score well across all five are rare, so the comparison needs to happen dimension by dimension.

Policy scope asks whether a vendor's policy explicitly recognizes AI-specific vulnerabilities. With only 18% of AI vendor policies explicitly mentioning AI risk, a buyer's first task is figuring out which of the three Edinburgh postures a given vendor holds. A "silent" vendor's generic policy offers no real basis for comparison on AI-specific findings, no matter how well it reads on traditional categories. It scopes its SaaS platform including FedRAMP GovCloud, specifies out-of-scope categories like denial-of-service, social engineering, and third-party services, defines coordinated disclosure with agreed disclosure dates, and commits to quarterly program review by its security team.

Timeline differentiation asks whether a vendor publishes distinct timelines for critical versus standard findings. Black Kite found that attackers now exploit before disclosure on average, so a vendor's active-exploitation timeline matters more operationally than its standard 90-day commitment.

Agentic and supply chain coverage asks whether a policy addresses third-party AI components and autonomous agent incidents, or stops at the model. The 33 protocol-level vulnerabilities documented in agentic commerce platforms sit entirely outside the scope of a policy written only for model-layer issues. The Vercel and Context.ai case shows how this plays out in practice: a third-party AI tool compromise led to OAuth token harvesting, which pivoted into platform credentials and ended in customer data exfiltration. The benchmarking task extends down the vendor's own vendor stack. Black Kite found that major AI coding assistants and agentic frameworks, including GitHub Copilot, Cursor, and Claude Code, recorded their first high-severity CVEs in 2025, which makes "which agentic infrastructure is your vendor running, and what controls govern it?" a baseline procurement question. Open-source dependencies carry a disproportionate share of discoverable supply chain risk: the LiteLLM compromise, where malicious code was inserted into two PyPI package versions and harvested credentials before detection, is the concrete case for why a vendor's open-source posture belongs in this exercise.

Data-handling and breach-notification commitments ask whether retention and notification timelines are contractually fixed or left to vague language. Retention policy itself can shift without warning: Anthropic's June 2026 mandated 30-day retention policy for traffic on its Fable and Mythos models, which overrode prior zero-data-retention agreements, drew what one description called an "almost allergic reaction" from many companies, with Booz Allen among those restricting employee use of the Fable model for work touching proprietary cybersecurity software. Unilateral retention changes are an operational risk that belongs inside the disclosure benchmarking framework, not filed away as a one-time DPA review.

Because AI vendors so often declare jailbreaking and prompt injection out-of-scope, precisely the categories most relevant to AI-specific harm, enterprises need direct visibility into what each vendor's threat model actually covers and where the edges of that coverage sit. The gap between a vendor's stated disclosure scope and its real exposure to AI-layer attacks is the kind of blind spot that continuous AI risk intelligence monitoring, the category Promptarmor operates in alongside other vendor assessment platforms, is built to catch.

The organizational readiness gap and its weight relative to vendor policy quality

A vendor's disclosure posture, however strong, only produces value if the enterprise receiving it can detect, interpret, and respond to what gets disclosed. Benchmarking vendor policy is a procurement exercise with diminishing returns once that internal capability is missing, and for AI incidents specifically, it usually is.

The trend is moving in the wrong direction on both sides of the relationship. Organizations rating their own AI incident response as "excellent" dropped between 2024 and 2025, and the share reporting "good" responses fell as well. Neither vendors nor buyers are currently positioned to get full value out of a rigorous disclosure framework, which is an argument for building readiness into the benchmarking exercise rather than treating it as a separate project to tackle later.

One might ask whether any external floor exists for AI transparency at all, something buyers can anchor to the way they once anchored to CVSS. It gives buyers one fixed comparison point on model documentation, but it doesn't reach into incident response or disclosure timelines. That's a narrow anchor, but a real one: treat it as a starting point rather than a substitute for the five-dimension framework above.

Running a practical peer benchmarking exercise against these criteria

Turning the framework into something a risk, security, or legal team can actually execute takes four sequenced steps: normalize the comparison units, score each vendor against the five dimensions, map the results against internal readiness, and monitor continuously on an ongoing basis.

Step one normalizes before any comparison happens. Establish whether each vendor's severity ratings are self-supplied or derived from NVD and CVSS data; with CVSS scores now absent from most new CVE entries, self-supplied ratings are common and should be judged on their methodology. Comparing raw counts without adjusting for embargo rates produces an inverted ranking.

Step two scores each vendor against the five dimensions laid out above: policy scope, channel specificity, timeline differentiation, agentic and supply chain coverage, and data-handling commitments. The question for each dimension is specific. Is the breach-notification timeframe contractually fixed, the way Azure OpenAI's 72-hour commitment is, or left to language like "prompt notice"?

Step three maps those scores against the organization's own readiness. A vendor that scores high on agentic coverage is a procurement advantage only if the enterprise has the forensic capability to act on an AI platform incident when one occurs. For vendors running high-embargo programs, confirm the organization has a named contact inside that coordinated disclosure program, an early-access relationship with something like Glasswing, for example, because an embargoed finding offers no operational benefit to a buyer with no channel into it.

Step four makes the exercise continuous. Black Kite's finding that attackers now exploit before public disclosure, by an average of seven days, means a point-in-time assessment can't defend against a timeline measured in days. Traditional TPRM tools and generic security governance platforms weren't built to track prompt injection, agentic connector exfiltration, or protocol-layer vulnerabilities in commerce agents on an ongoing basis. Dedicated AI risk intelligence platforms combine vendor assessment with real-time AI threat detection, Promptarmor among the options enterprises evaluate for this role, and they fill that specific gap and belong in the monitoring architecture alongside the periodic policy review.

The enterprises that get the most out of benchmarking AI vendor disclosures treat the exercise as a continuous intelligence function running year-round, not a checkbox revisited once a year during contract renewal.