SecurityScorecard opens benchmark for AI vendor-risk work and attack resistance

SecurityScorecard has released an open benchmark for testing whether AI models can handle third-party risk work and withstand attacks such as prompt injection. It gives security teams a way to assess both capability and susceptibility before handing an agent work involving vendor data.

Watch Desk analysis

What happened

SiliconANGLE reports that the benchmark covers vendor-risk tasks including exposure triage and fourth-party technology risk. Tests use snapshots of SecurityScorecard’s vendor-risk telemetry, checked by domain experts, with protected answer keys and rotating inputs intended to make memorising the test less useful.

The benchmark also tests resistance to prompt injection, jailbreaks and other attacks. Capability scores are measured against a baseline with confidence intervals; models receive an autonomy-readiness level from L0 to L4. The separate Model Security suite grades attack resistance from A to F. Version 2026.09 is available now, with documentation and registration for a public leaderboard.

Why it matters

Vendor-risk work often relies on questionnaires and documents supplied by the vendors themselves, which can be incomplete or deliberately misleading. The benchmark tests whether models can weigh those claims against underlying telemetry, rather than simply produce a convincing summary. It also recognises that doing the job and resisting manipulation are different tests, both relevant before an agent is given meaningful responsibility.

There is an important distinction between the public and official evaluations. Security teams can run models on the public tier and receive results marked synthetic-calibrated. SecurityScorecard says official expert-verified scores come from its internal tier; enterprises seeking a certified evaluation must approach the company directly. The report does not identify the models tested or publish their scores, so this launch offers a testing framework, not a public ranking of today’s contenders.

Our read

The useful idea is putting capability and attack resistance in the same conversation. A model that handles vendor-risk tasks well but folds under a deceptive input is not ready for autonomy; equally, a secure model that cannot do the work has not earned the keys. The framework is open, but certification remains a separate, company-run step. Teams should treat it as a way to structure evaluation, not as proof that a particular model is safe to deploy.

What to watch

• Which models appear on the public leaderboard, and how their results vary by task.
• Whether the benchmark’s tests and scoring gain input from outside labs and security teams.
• How well results from the public tier correspond with expert-verified evaluations.

Discussion spark: Should organisations require AI agents to pass a combined capability and attack-resistance benchmark before handling vendor-risk work, or is human review the more useful safeguard?

Watch Desk is operated by WittyWires as an independent cross-cutting AI news tracker. It does not speak for the organisations or people it covers.