Matching in "Finding," but a Large Difference in "Creating"—The Complexity of Capability Evaluation Revealed by a Chinese Open Model

On August 14th, Chinese AI company Z.ai (formerly Zhipu AI) released its new model, "GLM-5.3." The company announced that this model scored slightly higher than Anthropic's restricted model, "Mythos 5," in the cybersecurity benchmark "CyberGym." As a journalist with a background in AI research, I would like to carefully examine what this "slightly higher" result actually means and what it doesn't.

What the CyberGym Benchmark Measures

CyberGym is a benchmark consisting of 1,507 tasks that evaluate whether an AI model can inspect source code, find security flaws, and confirm that those flaws are genuine vulnerabilities. According to Z.ai's announcement, GLM-5.3 achieved a score of 84.5% in this benchmark, slightly surpassing Mythos 5's 83.8% and OpenAI's GPT-5.6 Sol's 83.6%.

Z.ai has also released details of the evaluation conditions. Using the Claude Code 2.1.207 harness (an evaluation execution environment), with maximum inference effort set, no web tools were used, and evaluations were conducted with one trial (Pass@1) per task under the conditions of temperature parameter 1.0, top P 1.0, and maximum output length of 128,000 tokens.

Correctly Interpreting a "Slight Difference"

First, it's important to consider how statistically significant this 84.5% vs. 83.8% difference is. This difference of less than one point is easily exacerbated by slight differences in benchmark design and evaluation conditions, and is not significant enough to definitively conclude that "GLM-5.3 clearly outperformed Mythos 5."

Furthermore, it's important to note that these figures were all published by Z.ai itself and have not undergone independent verification by a third party. When comparing scores from competitors' models, it's crucial to keep in mind that differences in evaluation conditions can influence the results.

The Decisive Difference Between "Finding" and "Building" Vulnerabilities

What's most noteworthy for researchers in this announcement isn't the closeness of the CyberGym scores, but the far greater difference in the subsequent "ExploitBench" benchmark. ExploitBench is a more advanced test that evaluates whether discovered vulnerabilities can be transformed into working attack code (exploits).

In this benchmark, GLM-5.3 scored only 54.4%, while Mythos 5 achieved 78.0% and GPT-5.6 Sol 76.5%, all significantly outperforming GLM-5.3. Even in a more practical evaluation metric—a time-limited attack development task (how many out of 105 tasks could be completed within two hours)—Mythos 5 reportedly completed the equivalent of 181 tasks, further highlighting the difference with GLM-5.3.

In short, while there is little difference between the two in the relatively passive ability to "find vulnerabilities and determine their authenticity," a significant gap still exists in the more active and advanced ability to "actually weaponize those vulnerabilities." This clearly demonstrates the danger of judging the overall picture based on a single benchmark score when evaluating the cybersecurity capabilities of AI models.

The Significance of Open Weights

Another point worth noting is that GLM-5.3 is provided as an open weight model (a model with publicly available trained parameters that anyone can download and use). However, the actual release of the model's weights is scheduled to be withheld until around August 28th for security review.

While Mythos 5 is a model with strict access restrictions by Anthropic, essentially a "gatekeeper" model, if a model with comparable capabilities becomes openly distributed, the nature of the "dual-use" problem in cybersecurity research will change significantly. In fact, it has been reported that GLM-5.3, in field tests conducted by a security team in China, discovered 2,436 vulnerabilities from 269 open-source projects, of which 1,097 were classified as medium to high severity.

Technical Evolution from the Previous Version

GLM-5.3 is based on the same 743 billion parameter mixed expert architecture as its predecessor, GLM-5.2. However, instead of training a new model from scratch, its capabilities have been enhanced through additional post-training. According to Z.ai, coding performance has improved by approximately 50% compared to the previous version.

This reflects the recent trend in efficiency in AI development, where, once the underlying model architecture is established, subsequent performance improvements can be continuously achieved through relatively low-cost additional training.

Points for Researchers to Consider

Z.ai's announcement is easily headlined as "Chinese AI company rapidly catching up to the US Frontier Laboratory in the field of cyber defense." However, a careful breakdown reveals a more precise picture: while they are indeed close in relatively basic tasks like "vulnerability discovery and verification," a clear gap still exists in the more advanced task of "actually weaponizing those vulnerabilities."

When evaluating the capabilities of an AI model, it's important to not only look at a single benchmark score that might easily grab headlines, but also to examine what that benchmark actually measures and how it compares to other benchmarks. We hope that once the GLM-5.3 weights are officially released, more detailed verification results from independent researchers will emerge.