Latest

Related Posts

Best Cyber LLM: Eleven Cards Ranked by Evidence, Not by Claim

The honest answer to “which cyber LLM is best” is that four of the eleven cards on OrcaRouter‘s cyber tier publish a security benchmark and seven publish none or something adjacent [OURS]. Any ranking that puts eleven names in order is inventing the missing seven rows. What can be ranked, without invention, is the evidence: what each card has shown, how old the showing is, and whether it describes the card or its parent checkpoint.

That is the ranking below. It is deliberately not a leaderboard — the tier is the one place all eleven live side by side, and it does not sort them either.

Tier one: a row that describes the card

Two cards publish a security benchmark evaluated for the product itself rather than inherited.

OrcaCyber Zero 1.5 reports CVE-Bench 95.8 on the evaluable subset, Cybench 100 under unrestricted agent execution, HumanEval+ 93.9 and SWE-bench Pro V2 76.5 [VENDOR]. It is post-trained, so the rows are about Zero 1.5. Its serving rows are unremarkable in the good sense: 1,901 ms median, 122.2 tokens a second, 0.859% errors, 10.37 billion tokens in the week [OURS].

OrcaCyber Zero 1.0 reports CyberGym Level 1 pass@1 98.07 [VENDOR]. One row, one number, and it is the highest single security score on the shelf. Its traffic is 4.20B tokens, its median 1,936 ms, its error rate 0.183% — the second lowest in the tier [OURS].

The caveat on both is the same and it is not small: no third party has reproduced either block. They are the vendor’s own evaluation, run on the vendor’s harness, with the subset choices stated in the row name.

Tier two: a row inherited from a checkpoint

Four abliterated cards publish benchmark rows, and every one of those blocks was evaluated before the card shipped.

DeepSeek V4.1 Flash publishes six rows dated 2026-09-09 — CyberGym 88.1, SEC-Bench Pro 62.8, ExploitGym 15.3 among them — against a release date of 2026-10-23 [VENDOR]. MiMo V2.6 Flash publishes four dated 2026-09-20, led by CyberGym 95.1 and dragged by SEC-Bench Pro 47.5 [VENDOR]. GLM 5.3 Flash publishes five dated 2026-08-25, including SWE-bench 92 and CyberBench 67.9 [VENDOR]. Qwen3.8 Flash Next publishes five dated 2026-08-25, from GPQA Diamond 91.7 to Toolathlon 73.5 [VENDOR].

The dates are the whole point. These are measurements of the base checkpoints, taken weeks before the refusal-removal edit existed. They tell you the ceiling the abliterated build might reach. They do not tell you what it reaches.

Tier three: no rows

Five cards publish nothing under public benchmarks: Kimi K3.1 Pro, GLM 5.3, Hy4, Ultra 1.0, and — with a different explanation — anything on the tier whose section reads “pending” [OURS].

That is not the same as being bad. Kimi K3.1 Pro carries the tier’s only 943,718-token output ceiling at quality 9. Hy4 is the cheapest blended card in the tier at 0.80× its own base’s price. Ultra 1.0 is the fastest card on the shelf at 205.7 tokens a second.

But an unevidenced card has to be bought on a trial, and a trial has a cost that the price row does not show.

Why the missing rows are not a scandal

A benchmark of a refusal-removed model on an offensive-security task is expensive to run, needs a harness that will execute real tool calls, and produces results that are embarrassing in both directions — too high and it looks reckless, too low and it looks broken. Most vendors skip it. The four cards here that do publish are publishing the checkpoint’s numbers, which is cheaper and safer.

The consequence for a buyer is that absence is information. A card with no rows has no evidence, which is different from having evidence against it, and both are different from a card whose rows were measured on something else.

The ranking, stated plainly

If you need a security score you can cite, Zero 1.5 first and Zero 1.0 second, on the strength of rows that describe the product. If you need a card whose inherited rows are strongest and cheapest, MiMo V2.6 Flash’s CyberGym 95.1 sits at $0.80 in and 0.938% errors, with SEC-Bench Pro 47.5 as its own counterweight. If you need throughput, Ultra 1.0 or DeepSeek V4.1 Flash, neither of which gives you a security number you can put in a slide.

Everything else is a trial. That is a legitimate way to buy — three of these cards are cheap enough that a week of your own traffic answers more than any published row — but it should be a decision rather than an accident.

The four rows that would change this ranking

A benchmark is only useful if it is hard to pass and hard to game. Four of the twenty-eight rows on the shelf are worth arguing about, and each one would move a card if it moved.

ExploitGym 15.3 on DeepSeek V4.1 Flash. This is the lowest score anywhere in the tier and it sits on the card with the most traffic. ExploitGym asks a model to take a known vulnerability and produce a working exploit; 15.3% means it usually cannot. A card that is safe to run as a reviewer and weak as an attacker is a specific product, and the row is the only place that distinction is visible [VENDOR].

SEC-Bench Pro 47.5 on MiMo V2.6 Flash. The same card that leads the tier on CyberGym at 95.1 sits under half on SEC-Bench Pro, which asks for a patch rather than a detection. CyberGym rewards finding; SEC-Bench rewards fixing. A 47.6-point gap between two security rows on one card is the widest inside any single card here [VENDOR].

TAU-Bench Airline 68.0 on MiniMax M3.1 Flash. Against GPQA Diamond 88.6 and Vibe Code Bench 73.9, the tool-use row is the card’s weakest. If the plan is an agent that books, queries or files something, that row is the relevant one and it is the one nobody quotes [VENDOR].

Cybench 100 on Zero 1.5. A perfect score on any benchmark should be read as a statement about the harness. The row name says “unrestricted agent execution”, which is a specific and unusual condition — the model is allowed to act, not just to answer. Whether 100 is a ceiling or a saturation point depends on which tasks the harness contains, and the card does not say [VENDOR].

Add the cost of reproducing any of these and the practical reading is: quote the two Zero rows when you need a security claim, quote MiMo’s CyberGym when you need a detection claim, and flag ExploitGym 15.3 whenever someone proposes running the DeepSeek card as an offensive tool.

The short version

Four of eleven cards publish something under public benchmarks; two of those blocks describe the card, four describe its checkpoint, and five cards publish nothing at all.

Rank by what is evidenced, not by what is advertised. Zero 1.5 and Zero 1.0 are the only two whose rows are about themselves. The four abliterated blocks are ceilings inherited from a checkpoint and should be labelled that way every time they are quoted. The remaining five are trials.

Sourcing note: prices, context windows, output ceilings, modalities, quality scores, latency, throughput, error rates and traffic are read from the eleven cards’ public model data on 2026-11-15, with seven-day windows ending that day [OURS]. Benchmark rows are OrcaCyber’s own, unreproduced by any third party, and each is dated where quoted [VENDOR]. No independent score exists for any abliterated build of these checkpoints.

LEAVE A REPLY

Please enter your comment!
Please enter your name here

Popular Articles