Skip to content
TPRM Benchmarkby SecurityScorecard
Sign in
Theme

TPRM Benchmark · SecurityScorecard

Methodology & research

How the benchmark works, how it is governed, and what the founding cohort will validate next.

Two public reporting dimensions

Release 2026.11 contains 120 admitted tasks across 34 disciplines.

TPRM Performance
61 tasks measure investigation and third-party-risk decisions.
Model Robustness
59 tasks combine 42 security and 17 model-security tasks.

The underlying three tags remain in release metadata for auditability.

Admit a task

A frozen synthetic portfolio supplies the evidence. A reference solver must reproduce the outcome and each subtask. Ambiguous or tied tasks are rejected.

Scroll horizontally to explore the workflow →

From evidence to a scored benchmark1. Freeze evidence: Versioned vendor snapshot. Same inputs for each model. 2. Generate tasks: Scenario + typed answer. Reference solution ladder. 3. Replay reference: Use the same console. Verify each required answer. 4. Apply admission gate: Reject ambiguous tasks. Retain verifiable tasks. 5. Run the candidate: Shared scaffold and budget. Record the run. 6. Grade and report: Typed answer checks. Score, counts and provenance.1. Freeze evidenceVersioned vendor snapshotSame inputs for each model2. Generate tasksScenario + typed answerReference solution ladder3. Replay referenceUse the same consoleVerify each required answer4. Apply admission gateReject ambiguous tasksRetain verifiable tasks5. Run the candidateShared scaffold and budgetRecord the run6. Grade and reportTyped answer checksScore, counts and provenance
Implemented workflow: task generation and admission precede candidate scoring. A rejected task is excluded from the task set; an unavailable model is reported absent, not scored as zero.

Run the evaluation

Unguided mode presents the complete task; guided mode presents its subtask ladder. The model inspects frozen evidence through the benchmark console and returns a typed answer.

Scroll horizontally to explore the workflow →

The bounded agent evaluation loop1. Present the task: Scenario + allowed tools. A fixed interaction budget. 2. Choose an action: The model reasons and acts. No answer key in the prompt. 3. Meter the console: Queries read frozen evidence. Commands consume steps. 4. Return observations: Bounded tool responses. Repeat within the budget. 5. Submit an answer: A final typed response. Or an explicit failed attempt. 6. Check the outcome: Compare to the admitted flag. Preserve the run trajectory.1. Present the taskScenario + allowed toolsA fixed interaction budget2. Choose an actionThe model reasons and actsNo answer key in the prompt3. Meter the consoleQueries read frozen evidenceCommands consume steps4. Return observationsBounded tool responsesRepeat within the budget5. Submit an answerA final typed responseOr an explicit failed attempt6. Check the outcomeCompare to the admitted flagPreserve the run trajectory
Implemented workflow: the console returns observations to the agent until it submits an answer or exhausts its budget. Typed grading evaluates the answer; it does not use a separate language model as judge.

Grade objective outcomes

Declared checks evaluate strings, CVE identifiers, dates, numbers, and sets. Each task is solved or unsolved; another model does not judge the answer. Reference steps classify task difficulty and set bounded iteration allowances.

Compare like with like

Comparable results use the same bundle, selected scope, mode, reasoning level, output allowance, and execution limits. Repeated identical configurations may be pooled. Incomplete runs do not receive a score.

Open samples and protected holdout

Methods, release identity, aggregate coverage, and representative scenarios are public. Expected outcomes and scored scaffolding remain protected. Participants receive the prompts used in their evaluation.

Anti-gaming controls

Content-addressed bundles, rotating snapshots, and canary indicators help detect exposure and memorization. A canary is a signal for review, not proof of unauthorized access.

Governance and limitations

The current anonymous leaderboard is provisional and internally reviewed; external and academic validation is pending. Synthetic transfer, provider behavior, reasoning controls, and repeated-run stability remain active research questions. Single-shot and multi-shot results are recorded as distinct protocols.

Research agenda

  • Validate scoring rules, reference solutions, and task admission.
  • Measure stability with the independent multi-shot protocol across providers and settings.
  • Test transfer to enterprise and public-sector TPRM workflows.
  • Improve anti-gaming controls and protected holdout design.
  • Develop a future threat-informed TPRM track.

Threat-informed TPRM

The proposed next track will test whether models can prioritize suppliers by likely impact and trace reverse blast radius through critical relationships. It is not part of the current admitted release.

Founding cohort

We invite academia, model developers, enterprise teams, public-sector researchers, and TPRM practitioners to validate and expand the instrument. Ten current-release runs have internal approval and appear as anonymous provisional scores; they are not an external study.

Join through Driftnet →