AIPot Consistency Study

An autonomous LLM penetration testing orchestrator, and a 400-run study of how consistently four models attack a multi-service honeypot.

What this is

An autonomous penetration testing agent and the experimental harness used to measure how consistently it works. The orchestrator wraps a large language model in a command-execute-observe loop, points it at a target IP, and lets the model decide what to do. The consistency study runs that loop 100 times each against four models — Claude Sonnet 4, Gemini 2.5 Flash-Lite, GPT-4o-mini, and a local qwen2.5-coder:14b — and measures what changes from run to run when prompt, orchestrator, and target are held fixed.

The target is a deliberately vulnerable honeypot hosting OWASP Juice Shop, weak SSH, and anonymous FTP. This repository carries the orchestrator code, the analysis pipeline that produced the paper's statistics, the Terraform configuration for the honeypot, and eight sanitized sample run logs. The full 400-run dataset lives on Zenodo.

Findings at a glance

• Zero content refusals across 400 runs. No model refused under the orchestrator's authorization framing once past iteration 1.
• Full exploitation rates: Claude Sonnet 4 — 61 / 100; Gemini 2.5 Flash-Lite — 85 / 100; GPT-4o-mini — 56 / 100; qwen2.5-coder:14b — 25 / 100.
• Cross-service credential reuse appears only in models retaining full conversation history. qwen 57%, GPT-4o-mini 49%, Claude and Gemini both 0% on five-exchange windows.
• Cross-model differences are statistically significant (p < 0.001) with large effect sizes — qwen vs. Gemini on SQL injection differs at Cohen's h = 1.12.
• First exploit lands within 15–30 seconds of wall-clock time across models.
• An earlier draft mis-classified 91 HTTP 529 events as refusals. A full-log audit corrected this to upstream API failures, not model-level refusals; the audit is documented in methodology/.

Quick links

• Paper: arXiv:2605.30096
• Full 400-run dataset (canonical): Zenodo DOI 10.5281/zenodo.20421592
• Reproduce the study: REPRODUCING.md
• Ethics and responsible-use posture: ETHICS.md
• Vulnerability disclosure: SECURITY.md

Repository tour

• orchestrator/ — the autonomous agent loop and the 100-run wrapper. One run produces one JSON log; the wrapper runs N consecutive runs against one provider+model. See orchestrator/README.md.

• analysis/ — analyze_consistency.py reads run logs and writes consistency_report.json (the single source of truth for every statistic in the paper); generate_figures.py renders the data-driven figures. See analysis/README.md.
• terraform/ — Azure infrastructure-as-code for the honeypot used in the study. See terraform/README.md.
• paper/ — the published PDF and figures (paper/figures/, with diagrams-source/ holding the .drawio files for the hand-drawn diagrams).
• data/sample/ — eight sanitized sample run logs, two per model. The full 400-run dataset lives on Zenodo. See data/README.md.
• methodology/ — supplementary write-ups: temperature-defect note, refusal reclassification audit, and curation notes covering how this repository differs from the canonical Zenodo deposit.

Citation

@misc{erdem2026aipot,
  title         = {How Reliable Are AI Attackers Against a Fixed Vulnerable Target? A 400-Run Empirical Study of LLM Penetration Testing Consistency},
  author        = {Erdem, Galip Tolga},
  year          = {2026},
  eprint        = {2605.30096},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CR},
  doi           = {10.48550/arXiv.2605.30096}
}

Full citation metadata is in CITATION.cff.

License

Code is released under the MIT License; the paper, methodology documents, and figures under CC BY 4.0. See LICENSE.md for the partition by directory.

About

Galip Tolga Erdem — LinkedIn.