104 real pentests.
Black-box. Benchmarked.
Penetrify's engine run against the XBOW benchmark — 104 web-security challenges — with no source access, fully unattended. From ~13 minutes and $37.50 a scan.
Black-box · 104 challenges · run in every scan mode ·
What the benchmark is
XBEN is the open benchmark published by XBOW Engineering: 104 web-security capture-the-flag challenges, each a self-contained application with a hidden flag. Solving one means discovering and exploiting a real vulnerability — IDOR, XXE, injection, auth bypass, SSRF and the like — and extracting the flag. It has become the reference suite for autonomous penetration-testing agents.
We run it black-box: the engine gets a running app and nothing else — no source, no hints — the same starting point as a real attacker.
Results
Every scan mode was run over the full 104-challenge suite, black-box.
| Scan mode | Solved | Rate |
|---|---|---|
| Fast | 103 / 104 | 99% |
| Standard | 104 / 104 | 100% |
| Deep | 104 / 104 | 100% |
| Difficulty | Solved | Rate |
|---|---|---|
| Level 1 — Easy | 45 / 45 | 100% |
| Level 2 — Medium | 51 / 51 | 100% |
| Level 3 — Hard | 7 / 8 | 87.5% |
| Difficulty | Solved | Rate |
|---|---|---|
| Level 1 — Easy | 45 / 45 | 100% |
| Level 2 — Medium | 51 / 51 | 100% |
| Level 3 — Hard | 8 / 8 | 100% |
Fast missed a single hard challenge; standard and deep solved all 104, hard tier included.
Speed
Fully unattended — no human in the loop. Depth is a dial: spend more time for more thoroughness.
| Scan mode | Avg. solve | Fastest | Per scan |
|---|---|---|---|
| Fast | ~13 min | 1.8 min | from $37.50 |
| Standard | ~46 min | 8.3 min | from $75 |
| Deep | ~134 min | 24 min | from $150 |
What it costs
Each benchmark challenge is one Penetrify scan, so the cost is simply our pricing — a fast scan from $37.50, standard from $75, deep from $150, depending on plan.
A single traditional human web-app pentest — commonly low-to-mid five figures over one to three weeks — buys many full autonomous passes of the entire 104-target suite, each target done in minutes.
The honest part
A benchmark score alone means little today — XBOW itself notes XBEN is now largely solved across the industry. That is why we do not lead with the number. What still separates autonomous pentesters is how they solve: black-box, unattended, at production speed, and at a price you can reason about. Those are the numbers on this page, and they are our own engine's, run the same way your scans run.
XBEN is published by XBOW Engineering under the Apache-2.0 license.