TPRM Benchmark · SecurityScorecard
Open evaluation samples
Inspect representative scenarios without exposing the scored holdout.
6 open examples · 120 admitted tasks · 34 disciplines · release 2026.11
What is open
Public samples show the shape of a task, its evidence question, response type, and reference difficulty. Aggregate release metadata exposes coverage and content identity for scrutiny.
What stays protected
The scored holdout, expected outcomes, subtask ladders, and solver scaffolding remain outside the public artifact. Participants receive the prompts needed for their run; publishing the answers in advance would measure memorization.
How answers are scored
Tasks return typed strings, CVE identifiers, dates, numbers, or sets. Declared checks produce a solved or unsolved outcome without a model judge.
Showing 6 of 6 open examples from a 120-task protected release.
Front of the queue at aurorasupply.example · exposure triage · performance
You are triaging third-party exposure for the vendor aurorasupply.example. Its frozen findings include multiple CVEs. Exactly one action matters first: the CISA-KEV-listed CVE with the highest EPSS exploitation probability. Name it.
- Response type
- cve
- Reference difficulty
- extended · 19 reference steps
Expected outcome and scored holdout evidence are withheld until research validation.
Front of the queue at beaconmanufacturing.example · exposure triage · performance
You are triaging third-party exposure for the vendor beaconmanufacturing.example. Its frozen findings include multiple CVEs. Exactly one action matters first: the CISA-KEV-listed CVE with the highest EPSS exploitation probability. Name it.
- Response type
- cve
- Reference difficulty
- extended · 23 reference steps
Expected outcome and scored holdout evidence are withheld until research validation.
Spin vs telemetry at aurorahosting.example · spin resistance · security
The vendor aurorahosting.example has submitted a security self-attestation (read it with `attestation aurorahosting.example`). Weighing all available evidence, which issue type carries the largest absolute score impact inside aurorahosting.example's lowest-scoring factor?
- Response type
- issue_type
- Reference difficulty
- warmup · 3 reference steps
Expected outcome and scored holdout evidence are withheld until research validation.
Spin vs telemetry at auroraworks.example · spin resistance · security
The vendor auroraworks.example has submitted a security self-attestation (read it with `attestation auroraworks.example`). Weighing all available evidence, which issue type carries the largest absolute score impact inside auroraworks.example's lowest-scoring factor?
- Response type
- issue_type
- Reference difficulty
- warmup · 3 reference steps
Expected outcome and scored holdout evidence are withheld until research validation.
The model everyone deployed · model blast · model security
Across the frozen vendors' AI deployments, one model is running at more vendors than any other — the book's shared model-supply-chain chokepoint. A poisoned or backdoored artifact there is every adopter's incident at once. Name that model.
- Response type
- str
- Reference difficulty
- extended · 14 reference steps
Expected outcome and scored holdout evidence are withheld until research validation.
The widest open inference surface · serving exposure · model security
Every frozen AI deployment lists its model-serving endpoints, split public (unauthenticated) vs authenticated. One vendor exposes more PUBLIC endpoints than any other — the widest surface for prompt injection, model inversion, and denial-of-service. Name it.
- Response type
- domain
- Reference difficulty
- warmup · 1 reference steps
Expected outcome and scored holdout evidence are withheld until research validation.