CWE-bench v2 opens 100 security-fix tasks across 91 CWE types
Hugging Face
- Security
- Benchmarks
- Agents
- #cwe-bench
- #harbor
- #vulnerability-fixing
- #leaderboard
Collinear AI published CWE-bench v2, a public set of 100 Dockerized tasks in which an agent receives a real open-source repository checkout and an instruction to find and fix a security weakness without breaking functionality. Scoring is programmatic, tasks span 91 CWEs, and results are tracked on a dataset-page leaderboard for open-weight models and a separate Space covering API-only models. The authors report Gemini 4 Argon leading the 11 evaluated models at 67.8% pass@1.
What v2 changes about security-fix evaluation
Collinear AI's CWE-bench v2 publishes 100 tasks, each a self-contained Dockerized environment pairing a checkout of a real open-source repository with a short instruction to find and fix a security weakness while preserving existing behavior. Scoring is deterministic and programmatic through a verifier per task, and the dataset ships a registry.json so the Harbor harness can run the set directly from the Hugging Face repo. The authors say tasks span 91 CWEs, that every task passed checks for realistic instructions and verifier fairness, and that each was reviewed by hand. A held-out set stays private for tracking progress on unseen tasks. The reported leaderboard covers 11 models, with Gemini 4 Argon at 67.8% pass@1 and 81.0% pass@4 ahead of Claude Opus 5.5 at 67.3% pass@1; 95 tasks were solved by at least one model.
Application scenario for a security engineering lead
A security engineering lead considering an agent-assisted remediation workflow could use the public set as an offline screening ground rather than as proof of production readiness. The realistic scenario: an agent is pointed at a repository with a short vulnerability description and must produce a patch that fixes the weakness without breaking the build. Because the benchmark's verifiers check behavior rather than the shape of one specific fix, a team could compare agent configurations on whether the resulting patch passes functional checks, then route only ambiguous cases to a human reviewer. This is a candidate use of the benchmark, not an established result.
How to run a small trial and judge it
A team could start with a fixed subset of tasks and a fixed number of attempts per task, computing mean reward against a baseline of its own current model or manual triage process. Success would mean the agent's pass rate on the subset exceeds that baseline while functional checks stay green; failure checks would include patches that pass the security verifier but break existing tests, or fixes that only mirror the upstream patch. Because the dataset is gated, access must be accepted first, and a token needs to be saved to git's credential helper for Harbor to fetch tasks. Any rollout result should be reviewed by a person before it informs a policy change.
Trade-offs, prerequisites and when to skip it
The set is public, so models may have seen it; the authors keep a held-out set for that reason and ask people to contact them to evaluate on it. The benchmark measures single-instruction fix behavior in Dockerized environments, not full application security review, so it should not be treated as coverage of a codebase's real risk. Cost matters: the authors report DeepSeek V4.1 Flash at 57.0% pass@1 for $0.04 per rollout, while higher-scoring models cost substantially more per rollout. A team without Docker infrastructure, gated-dataset access, or reviewers to audit agent patches should not adopt this as a workflow gate.
Context and limitations
- Based on a short source excerpt; the full article and individual task files were not analyzed.
- Leaderboard numbers are as reported by Collinear AI; no independent verification of the harness or results is described here.
- Pricing figures cited for models are the authors' reported per-rollout costs at publication and may change.
- Based on bounded source text, not independent testing. Practical suggestions are editorial proposals.