What Do CyberGym and ExploitBench Actually Measure?

Related daily report

August 14, 2026: GLM-5.3 puts cyber capability and release controls in the same frame

Today’s complete report also covers SMIC wafer-price increases, private-credit financing for AI compute, Anthropic’s latest risk assessment and the still-pending DeepSeek tariff change.

Open the permanent August 14 report

The direct answer

A cyber benchmark score only makes sense after you ask: what step of the attack or defense process was tested?

An AI system can be good at reading code and reproducing a known vulnerability without being equally good at converting that vulnerability into a reliable exploit. It can also be good at building an exploit when it is handed a known bug, yet still be poor at discovering new bugs from scratch.

Those are different capabilities. They require different information, tools, time horizons and forms of reasoning.

The simplest rule

Do not translate “84% on a cyber benchmark” into “84% capable of hacking real systems.” First identify what the benchmark gives the model, what the model must produce, how success is graded and which protections are enabled.

Today’s trigger

What did Z.ai report for GLM-5.3?

Reuters reported on August 14 that Z.ai announced GLM-5.3 and supplied two very different cybersecurity results. The company said GLM-5.3 scored 84.5% on CyberGym, compared with 83.8% for Anthropic’s restricted-access Mythos 5. On ExploitBench, Z.ai reported 54.4% for GLM-5.3 versus 78.0% for Mythos 5.

Z.ai also reported a timed attack-development comparison: 105 tasks completed in two hours and 130 in six hours for GLM-5.3, versus 181 and 247 for Mythos 5.

CyberGym

84.5% vs 83.8%

Z.ai’s reported result suggests close performance on the tested vulnerability-analysis/reproduction task.

ExploitBench

54.4% vs 78.0%

The reported gap is much larger when the task moves toward constructing working exploitation.

Evidence status

Vendor-reported

Reuters says the results have not been independently verified. AIUpdateWatch is therefore not treating them as a permanent leaderboard result.

The release status matters too. GLM-5.3 was announced on August 14, but Z.ai said broad public release would follow in about two weeks after additional security assessment and safeguard work. Initial access is being limited, and the company says its most sensitive cybersecurity functions will use a verified-user access system.

So there are two separate questions: how capable is the model, and how is that capability being released?

CyberGym is mainly a vulnerability-reproduction benchmark

The official CyberGym benchmark contains 1,507 historical real-world vulnerabilities from 188 software projects. For its main vulnerability-reproduction setting, the agent receives a description of a known vulnerability and the corresponding unpatched codebase. It must then generate a proof-of-concept, or PoC, that triggers the vulnerability.

Success is checked by comparing the vulnerable and patched versions: the PoC should trigger the problem before the patch and stop triggering it after the patch.

Important nuance

This is not the same as dropping an agent into arbitrary software with no hint and asking it to discover every unknown vulnerability. CyberGym’s main task starts from a known historical vulnerability and asks the agent to reproduce it reliably.

That is still a difficult and useful test. The model may need to navigate a large repository, understand how the relevant code is reached, build inputs that exercise the vulnerable path, run tests and refine the PoC from execution feedback.

But a PoC that proves a bug exists is not automatically a weaponized exploit. A crash can demonstrate memory corruption without giving the attacker control of program execution. A file read can confirm a boundary failure without establishing arbitrary code execution. The next benchmark layer asks whether the model can cross that gap.

ExploitBench asks how far an agent can climb from a bug toward real exploitation

ExploitBench describes exploitation as a ladder rather than a single yes/no event. Its current V8 benchmark measures progress from reaching vulnerable code, through triggering the bug and constructing exploitation primitives, toward arbitrary code execution.

That design matters because two agents can both “fail to exploit” while being at very different capability levels. One may only cause a crash. Another may build a useful memory primitive but fail at the final step. A binary score can hide that difference.

The benchmark’s own methodology contrasts crash-class tests with deeper exploitation. It places vulnerability-triggering benchmarks near the lower part of the ladder and measures additional rungs above that floor.

A related project, ExploitGym, makes the same distinction from another direction. It gives the agent a known vulnerability and a proof-of-vulnerability input, then asks it to turn that starting point into concrete security impact such as unauthorized code execution. The public ExploitGym benchmark spans userspace programs, the V8 JavaScript engine and the Linux kernel.

The shared lesson is straightforward: finding or reproducing a bug is not the same task as engineering a reliable exploit around modern defenses.

A useful mental model

Think of AI cyber capability as a ladder, not one number

The exact benchmark taxonomies differ, but readers can use the following simplified ladder to interpret claims. This is an AIUpdateWatch explanatory model, not an official standard.

Understand the code

Locate relevant components, reason about program behavior and identify where a reported weakness may live.

Reproduce the vulnerability

Create a PoC or input that reliably triggers the known bug and confirms that the issue is real.

Build exploitation primitives

Move beyond a crash toward useful control such as a memory read, write or other primitive needed for exploitation.

Defeat relevant mitigations

Adapt the exploit around sandboxing, memory protections or other defenses present in the evaluation environment.

Achieve concrete impact

Reach a high-consequence result such as unauthorized code execution under the benchmark’s defined conditions.

There are other important cyber abilities outside this ladder: discovering genuinely unknown vulnerabilities, phishing, credential attacks, cloud misconfiguration analysis, malware development, persistence, lateral movement and end-to-end campaign planning. No single vulnerability benchmark measures all of them.

Why can two benchmark scores not be compared like exam grades?

Even when two tests both involve cybersecurity, the numbers may describe different tasks. A 70% result on vulnerability reproduction and a 70% result on exploit construction are not equivalent units.

And even results on the same benchmark can change materially with the evaluation setup.

Starting information

Does the agent receive a vulnerability description, a crashing input, a CVE reference, or only a codebase?

Tools and scaffolding

Can it execute code, use a debugger, search files, call external tools, retain state or receive automated hints?

Time and attempts

Is this one run, best-of-N, several hours of agent work, or a short fixed budget?

Security configuration

Are sandboxing, compiler protections, browser mitigations or kernel defenses enabled?

Success oracle

Is success a crash, a PoC, an LLM judge, a captured flag, a deterministic primitive, or arbitrary code execution?

Model access policy

Is the tested model the public product, a special research checkpoint, or a restricted version with safeguards changed?

This is why AIUpdateWatch does not take today’s GLM-5.3 percentages and insert them directly into a general model ranking. The public reporting gives useful evidence, but the exact model snapshot, full harness configuration and independent reproduction are still needed for a durable apples-to-apples comparison.

Does a high cyber score mean a model is dangerous?

It is evidence of capability under a defined test. It is not, by itself, a complete risk assessment.

Risk depends on at least four layers:

Capability: what can the model do under realistic conditions?
Access: who can use the model, and which high-risk functions are restricted?
Affordances: what tools, internet access, credentials and execution environments can the system reach?
Controls: what monitoring, rate limits, approval gates, identity checks and abuse detection surround it?

The same capability can support defenders and attackers. A model that finds a memory-safety flaw can help a maintainer patch it. The same discovery can help an attacker if paired with exploit construction and access to a target.

This dual-use character explains why the GLM-5.3 story is about release governance as well as model performance. Z.ai is arguing for broader defensive access while also delaying broad release and reserving sensitive functions for verified users. Whether those controls remain effective after wider distribution is a separate question that must be tested in practice.

How should businesses, security teams and readers judge the next cyber-benchmark claim?

Before accepting a headline such as “Model X matches Model Y in cybersecurity,” ask for a short evidence checklist.

Exact model: Which checkpoint or product version was evaluated?
Exact task: Vulnerability discovery, reproduction, patching, exploitation, attack planning or something else?
Starting point: What information was handed to the model before it began?
Environment: Which tools, permissions, defenses and time limits were enabled?
Scoring: What concrete event counts as success?
Trials: Is the number Pass@1, best-of-many attempts or another aggregation?
Independence: Was the result produced by the vendor, benchmark maintainer or an independent evaluator?
Reproducibility: Can another group run the same model under the same conditions?

If those details are missing, the number can still be worth noting, but it should remain a provisional capability signal rather than a permanent rank.

What today’s GLM-5.3 numbers actually support

Z.ai has reported evidence that GLM-5.3 is highly capable on a vulnerability-analysis/reproduction evaluation and meaningfully weaker than Mythos 5 on the reported exploit-development evaluation. That is more informative than saying the models are simply “close” or “far apart.” Independent reproduction is still needed.

Sources and related reading