Tech

OrcaCyber Zero 1.5 Benchmarks: 95.8 on the Evaluable Subset

The number to read on OrcaRouter‘s OrcaCyber Zero 1.5 benchmarks block is 95.8, and the four words that matter are the ones in brackets after it: CVE-Bench on the evaluable subset [VENDOR].

Subset reporting is normal, defensible, and the single most common way a security benchmark number gets misquoted downstream. This piece is about what the qualifier costs you when you drop it.

What CVE-Bench actually asks

CVE-Bench takes published vulnerabilities and asks a model to do something real with them — identify the vulnerable path in a codebase, propose a patch, or produce an exploit, depending on the subset. The important property for a buyer is that the tasks are grounded in real CVEs rather than invented challenges, so a high score is closer to a statement about production work than a CTF score is.

The second property is that the benchmark contains tasks whose ground truth is contested or unavailable. A CVE whose fix was later found incomplete, or whose reproduction depends on a specific compiler and OS, cannot be graded cleanly. Those tasks get annotated out.

So the “evaluable subset” is the set of tasks the annotation could grade — not the easier half, and not the harder half, though in practice it skews toward the ones with unambiguous fixes.

Why 95.8 is not comparable to a different 95.8

Two vendors can both report 95.8 on CVE-Bench and have measured different things, because the subset boundary depends on the annotation, not on a published constant.

The consequence is narrow but real: the score compares cleanly against another run of the same annotation, and against the vendor’s own earlier version of the same card. It does not compare cleanly against a number from a paper that used a different split.

If you are building a vendor shortlist, the safe move is to record the parenthetical alongside the number. Four words, and they prevent a comparison that would otherwise look exact and be wrong.

The other three rows

Zero 1.5 publishes three more, and each answers a different question.

Cybench 100, under unrestricted agent execution. Perfect, and the parenthetical is again the whole story — the model is driving tools, not answering questions.

HumanEval+ 93.9. The strict variant of HumanEval, which adds tests that catch solutions passing the original set by coincidence. 93.9 on the strict set is a solid function-level number and is directly comparable with other vendors’ HumanEval+.

SWE-bench Pro V2 76.5. Repository-level repair against a harder split than the classic set. This is the lowest of the four rows and the most predictive of day-to-day engineering work.

Read together, the four rows describe a model that acts well, writes functions well, and repairs repositories reasonably. That is a coherent pentest-agent profile rather than a set of scores somebody picked for the slide.

What Zero 1.0 publishes, for contrast

The earlier post-trained card, Zero 1.0, publishes a single row: CyberGym Level 1 pass@1 98.07 [VENDOR]. One benchmark, one number, no subset qualifier.

That makes Zero 1.0’s block easier to quote and harder to interpret. A card with one row tells you nothing about whether the strength is broad or narrow, and CyberGym Level 1 measures detection on a defined task family rather than end-to-end exploitation.

Zero 1.5’s four rows are the more useful document, even though every one of them carries a qualifier. Breadth beats precision when the question is “what kind of model is this”.

The three ways a CVE benchmark is run, and why only one of them predicts your week

The same CVE can be evaluated at three depths, and a buyer should know which one a score refers to.

Detect. The model is shown a snippet and asked whether it is vulnerable, and optionally which CWE it maps to. This is the cheapest form to run and the easiest to score, and it is what most “vulnerability detection” numbers on leaderboards measure.

Locate. The model is given a repository and told which CVE is present, then asked for the file, function or line. This needs a harness that can walk a codebase and a grader that can match locations, which is where most benchmarks lose their tasks to the “not evaluable” bucket.

Repair or exploit. The model produces a patch that must pass the existing test suite, or an input that must actually reach the vulnerable path in a running container. This is the depth a security team actually operates at, and it is the most expensive and least published.

CVE-Bench’s evaluable subset sits in the second and third depths, which is why the subset exists: a task whose container will not build cannot be graded at depth three at all. A 95.8 across that subset is therefore a statement closer to production work than a detection score is — and still one whose task count nobody outside the vendor knows.

See also: Inside the Technology That Helps Modern HVAC Systems Think for Themselves

The denominator problem, stated plainly

Every benchmark score is a fraction, and the denominator is where the argument lives.

A 95.8 over 24 tasks means one missed task. A 95.8 over 960 tasks means 40 missed tasks, and the interval around the number is narrow enough to compare. Same figure, different confidence, and the card prints only the figure.

For a shortlist this rarely changes the ranking. For a go/no-go decision on a single card it changes a great deal, because the question “is this model reliable at repository repair” is answered by the sample size, not by the mean.

If you can only get one more number from a vendor before committing, ask for the task count. It is cheaper for them to provide than a re-run and it settles more than any additional decimal place would.

The serving side, because a benchmark is not a product

Zero 1.5’s telemetry over the seven days to 2026-11-15: 1,901 ms median, 122.2 tokens per second, 0.859% errors, 10.37 billion tokens of traffic, at $4.13 blended [OURS].

The traffic figure is the one that is easy to skip. Ten and a third billion tokens in a week makes Zero 1.5 the second-busiest card in the tier, behind only DeepSeek V4.1 Flash at 32.99B [OURS]. Whatever is being built on this card, it is not a demo.

Text-only, chat-completions only, quality 9/10, ZDR flagged. Those four facts narrow the audience more than any benchmark row does.

The short version

95.8 on CVE-Bench’s evaluable subset, 100 on Cybench under agent execution, 93.9 on HumanEval+, 76.5 on SWE-bench Pro V2. Four rows, all vendor-reported, none reproduced independently.

The subset qualifier is not a hedge and not a weakness. It is the boundary within which the number is true, and quoting 95.8 without it invites a comparison the benchmark cannot support.

Buy this card for the breadth of its evidence and for its traffic, not for any single score. Ten billion tokens a week says more about the product than a 100 on a task set nobody outside the vendor has seen.

Sourcing note: prices, context windows, output ceilings, modalities, quality scores, latency, throughput, error rates, traffic and the parameter/endpoint surface are read from the card’s public model data on 2026-11-15, with seven-day windows ending that day [OURS]. Benchmark rows are OrcaCyber’s own, unreproduced by any third party, and the harness subsets named in the row are quoted with the numbers [VENDOR].

Related Articles

Leave a Reply

Your email address will not be published. Required fields are marked *

Back to top button