104 real pentests, black-box, 100% solved
Penetrify's engine run against the full XBOW benchmark — 104 web-security challenges — with no source access, fully unattended, on Penetrify's fast tier. The harness and every per-challenge log ship alongside these numbers, so the run is reproducible end to end.
Result
All 104 challenges solved. Grading is flag-in-log against the deterministic FLAG{sha256(name)}, matching how XBOW scores the suite: a challenge counts as solved only when that exact flag appears in the agent’s transcript.
| Difficulty | Solved | Rate |
|---|---|---|
| Level 1 — Easy | 45 / 45 | 100% |
| Level 2 — Medium | 51 / 51 | 100% |
| Level 3 — Hard | 8 / 8 | 100% |
| Total | 104 / 104 | 100% |
Class labels are the suite’s own tags; a challenge often carries more than one, so the counts sum above 104. Because every challenge was solved, each class shown was also solved.
How the run was produced
Stated in full so the page and the attached logs agree with each other.
- Mode
- Black-box: running app only, no source, fully unattended.
- Model
- Penetrify fast-1.1 (fast tier).
- Grading
- Flag-in-log grep of the deterministic flag. No partial credit; exploit soundness is not manually re-verified.
- Cost
- ~$29 per pentest at Penetrify fast-tier list price — what a customer pays Penetrify for one scan, not an underlying LLM bill.
- Environment
- EC2 amd64. ~40 targets needed Debian bit-rot and Docker Compose fixes (both scripts shipped).
- Compute
- ~16.8 hours wall-clock across the 104 challenges.
- Run date
- 2026-09-02.
Reproducibility
The harness and all 104 per-challenge logs are published with these numbers. Anyone can rebuild each target and re-run.
| Artifact | Count | Note |
|---|---|---|
| Per-challenge logs | 104 | full agent transcript each |
| results.tsv | 104 | bench · solved · flag · secs |
| run harness | 1 | build + attack + grade |
| env repair scripts | 2 | Debian bit-rot + compose fixes |
Logs are sanitized: AWS account ID, ECR image URIs, and runner host paths removed. The cost shown in each transcript ($29) is Penetrify’s fast-tier list price per scan, not the underlying LLM bill.
Run it against your own application
The benchmark shows the engine on public targets. The real test is your app. Start a black-box pentest in minutes.
XBEN is published by XBOW Engineering under Apache-2.0. Figures computed from the 104 per-challenge logs of the fast-tier run, 2026-09-02.