Lab idea / AI-enabled offense
Does an evidence ledger make an assistant worth using?
Compare reviewer effort and unsupported findings, not the fluency of the final report.
Falsifiable hypothesis
A structured evidence ledger reduces independent review time without increasing unsupported findings.
Source-led motivation
LLM-assisted penetration-testing framework with an autonomous staged pipeline and a separate interactive legacy mode; the repository links the USENIX research paper.
Authorized lab scope
An owned deliberately vulnerable local application with fixed seeded findings; no public targets.
Prerequisites
- A manually verified answer key, withheld from the assistant.
- A pinned agent revision and backend; explicit egress policy.
- Two independent reviewers and equivalent assessment budgets.
Comparison & control
Compare analyst-only, assistant plus raw transcript, and assistant plus structured ledger. Rotate task order to reduce learning effects.
Observations to collect
- Time to independently verify each claim.
- Unsupported, duplicate, missed, and correctly verified findings.
- Reviewer corrections and incomplete tasks.
Disconfirming evidence
- Ledger review takes as long as transcript review, or unsupported findings increase.
Stop conditions
- Stop if any action reaches an unapproved target, real user data, or a non-lab service.
- Stop on unexpected egress, exposed credentials, or an exhausted predeclared budget; retain a sanitized failure record.
Source bibliography
- PentestGPT ↗
Gelei Deng and the PentestGPT authors / GreyDGL
Publication date: Date not supplied · Editorial check 2026-10-26 UTC
Project README / official documentation; source-led editorial check, no execution