We Cleared the XBOW Benchmark Suite — in Our Fastest Tier

XBEN is the open web-exploitation benchmark published by XBOW Engineering: 104 self-contained web applications, each hiding a flag. To solve one, an agent has to find and exploit a real vulnerability — an IDOR, an injection, an auth bypass, a template injection, an XXE — and pull out the flag. It has become the reference suite for autonomous pentest agents.

We pointed the same engine that runs our customers' scans at all 104, black-box: a running app and nothing else, no source, no hints. The notable part is not the 100% — XBOW themselves note the suite is now largely solved across the industry — it is which tier did it.

The result

All 104 solved, in the fast tier: our quickest, cheapest scan mode, backed by a smaller, faster model. Because the fastest tier already cleared the suite, we did not run the slower standard and deep tiers on it.

The 104 challenges span 26 distinct vulnerability classes: XSS (23), default credentials (18), IDOR (15), privilege escalation (14), server-side template injection (13), command injection (11), business logic (7), SQL injection (6), local file inclusion (6), insecure deserialization (6), arbitrary file upload (6), information disclosure (6), path traversal (5), XXE, SSRF, JWT, GraphQL, cryptographic flaws, blind SQLi, and one-offs like request smuggling, race conditions and NoSQL injection.

Average solve time was 9.7 minutes per challenge; the fastest was 1.5 minutes. The whole suite took about 16.8 hours of compute, fully unattended.

How a solve is scored

Every XBEN flag is deterministic — it is derived from the challenge name and baked into the app at build time. A run counts as solved only when the engine autonomously discovers and reports that exact flag. There is no partial credit and no human in the loop: the engine gets a URL and the instruction to capture the flag, and either it comes back with the right FLAG{...} or it does not.

Why the fast tier is the story

Autonomous pentest results are easy to inflate: give an agent source code (white-box), unlimited time, a top-end model, and a few retries, and a lot of suites fall over. We did the opposite. This was black-box, single-pass, unattended, on the cheapest model we ship. That a fast, inexpensive scan clears a suite built from real exploitation challenges is the part worth reporting — because that is the tier most of our customers' scans actually run in.

The honest part

A number like 100% deserves its asterisks, so here they are:

• XBEN is a validation set. It is XBOW's own public suite, designed to be solvable and now widely solved. A perfect score is a sanity check on our engine, not a claim that we have "solved web security."
• Grading is lenient by design. A challenge counts if the correct flag appears in the run — which is how the suite is meant to be scored, and generous compared to objective-based benchmarks.
• This ran on a paid API. Black-box and unattended. It is fully reproducible — the harness and every per-challenge log are published alongside these numbers.

For contrast, on CVE-Bench — a harder suite with strict, objective-based grading — the same engine solves a much lower fraction (and we found that some of its "failures" were actually bugs in the benchmark's grader). Different benchmarks measure different things; we publish both.

What it took to run it

Running a four-year-old benchmark end-to-end in 2026 is its own small adventure. Two environmental issues had to be fixed before a single challenge would even build:

• The newer Docker Compose plugin rejects the expose: "3306:3306" syntax the older challenges use; the database services needed their expose entries rewritten to a single port.
• Roughly forty challenges are built on archived Debian base images whose package repositories have moved to archive.debian.org, so apt-get returns 404s; their Dockerfiles needed their APT sources repointed at the archive.

A handful of individual challenges also needed version bumps — a Python image old enough that a dependency wouldn't build, a Composer version that now blocks an advisory-flagged package, a Node sidecar using syntax its old runtime couldn't parse. None of that is the engine; it is benchmark bit-rot, and worth noting for anyone who tries to reproduce the run.

See it, or run it

The full breakdown — by difficulty and vulnerability class, with the methodology — lives on our benchmark page. And the engine that cleared these 104 targets is the one that runs your scans: point it at your own application and see what it finds.