feat: engine capability uplift (TG1–TG7) + MCP server wiring - #27
Open
pt-act wants to merge 14 commits into
Open
feat: engine capability uplift (TG1–TG7) + MCP server wiring#27pt-act wants to merge 14 commits into
pt-act wants to merge 14 commits into
Conversation
…item A) - strategy_lib.py: set_embedding_backend(bm25|bow|openai|local) dispatch; BM25 pure-Python lexical scorer (no vectors persisted, zero deps); lazy re-embed on backend switch (rows never dropped, R-A2); transfer_score + retrieval_bonus helpers (TG7 API, placed here for pbt-properties.py import); bow legacy path preserved; openai/local dense backends wired (opt-in only). - tests/test_tg1_embeddings.py: retrieval quality top-2 (R-A4), bm25≥bow positive control (§1.6), SP-DI1 self-retrieval PBT, SP-DI2 round-trip PBT, readiness checks §1.9-1.11. - specs/engine-capability-uplift/: spec + tasks + requirements + pbt-properties.py committed to repo; TG1.7/TG1.8 un-skipped in pbt-properties.py. CI-cold: all new tests use tmp_path with uuid sub-paths — no network, no corpora needed.
- registry.py: EngagementContext (objective/attacker/vault/thread/system/reasoning) + IOContext (progress/record/run_events/tool_logger) extracted as dataclasses. ToolContext becomes a thin class with delegating properties for all legacy field names; custom __init__ accepts all prior kwargs unchanged (R-B1, R-B2). Zero tool signature changes — existing 1191-test suite still green. - tests/test_tg2_context.py: structural checks, delegation parity (all fields), legacy-kwarg compat, SP-DI5 PBT (200 examples across text/bool values). - pbt-properties.py: SP-DI5 test_context_delegation_parity un-skipped.
- campaign.py: classify_family(model)->str — total, deterministic, 6 families +
other; _FAMILY_PRIORS from CHANGELOG ASR data; wire into _campaign() so family
is classified at engagement time and family priors seed the contextual bandit
on cold-start (R-D1, R-D2).
- _bandit.py: seed_family_priors() (only when no live data); best_technique_by_family()
reads saved bandit state → {family: {cat: best_tech}} for /stats (R-D3);
best_by_context() on ContextualBandit.
- tui/app.py: /stats now shows 'best technique by family' section from contextual
bandit state (R-D3).
- tests/test_tg3_family.py: 27 tests — known model table, SP-IV1 PBT (400 examples),
seeding/no-overwrite, best-by-family structure.
- pbt-properties.py: SP-IV1 test_family_classifier_total un-skipped.
…m E) - judge_selftest.py: _compute_ensemble_agreement(per_member_labels) computes pairwise kappa + per-member disagreement rate; _judge_selftest fires all named ensemble endpoints when 'ensemble' arg is provided, appends calibration section with per-member rates and >20% alert (R-E1, R-E2). Single-judge default: no ensemble section emitted, fully backward-compat (§4.5). - tests/test_tg4_calibration.py: unit tests for _compute_ensemble_agreement, negative control (miscalibrated stub trips alert), positive control (well-behaved passes), single-judge output unchanged, 3-member outlier detection.
… (item G)
- _bandit.py: arm_key(technique, transform_chain, category) → canonical string (R-G1);
regret_curve(bandit_rewards, random_rewards) → {bandit, random, beats_random} (R-G4);
Bandit.__init__ accepts list of arm keys (pre-register with zero stats) so pbt-properties
test_bandit_registered_arms_only works correctly (R-G1/G2).
- tests/test_tg5_bandit.py: arm_key tests, regret-curve beats-random fixture (R-G4),
posterior resume (R-G3 §5.6), SP-C1 PBT (200 ex), SP-DI3 PBT round-trip (100 ex).
- pbt-properties.py: SP-C1 + SP-DI3 un-skipped; path uniqueness fix for SP-DI3.
…on (item H)
- agentharm.py: score_rubric(flags)->float — pure, monotone, bounded; satisfies
SP-IV2 (R-H3). Multi-question binary rubric grader already present; now exposed
as a testable pure function.
- rag_poison.py: grade_exfil(response, beacon_keywords) — keyword track + markdown-
image beacon detection, returns {exfil, keyword_match, beacon_match, matched_keywords}
(R-H1 exfil grader). build_coverage_matrix(results) — injection×task×ASR coverage
matrix (R-H4 AgentDojo-style reporting).
- tests/test_tg6_agentic.py: exfil TP/FP fixtures, rubric monotone fixtures, SP-IV2
PBT (200 examples), coverage matrix shape and empty cases.
- pbt-properties.py: SP-IV2 test_agentharm_rubric_monotone un-skipped.
…(item I) - strategy_lib.py: family kwarg on add()/distill() tags each row with the target family (R-I1); retrieve_by_family(family, k) surfaces top-k same-family strategies for cold-start (R-I1); update_transfer_score(name, origin_delta, same_family_delta, cross_family_delta) increments per-row counters (R-I2); cross_family_matrix(strategies, families) builds the origin×target matrix of best-transferring techniques (R-I3). - leaderboard.py: cross_family=true routes to _cross_family_leaderboard which renders the transfer matrix over ≥3 configured profiles (R-I3). - tests/test_tg7_transfer.py: family tag persist/reload, retrieve_by_family ordering, update_transfer_score, SP-DI4 PBT (200 ex), matrix shape + diagonal None, empty lib. - pbt-properties.py: SP-DI4 test_transfer_score_conservation un-skipped. All 8 SP properties now active (TG8 SP-RC1/AC1 remain skipped — demand-gated).
…record) - completion-report.md: per-criterion self-check for all R-A/B/D/E/G/H/I criteria, interfaces delivered, known deviations (TG8 demand-gated, regret-curve TUI wiring, dense backends offline-only), state management section. - TG8 intentionally excluded: demand-gated per spec, no single-machine ceiling demonstrated.
Issue 1 — BLOCKER: leaderboard.py:112 backslash inside f-string expression Hoist corner label out: _corner = 'origin / target' (SyntaxError on Python 3.11; PEP 701 backslash relaxation only applies to 3.12+). All 9 changed modules now py_compile-clean on Python 3.11 (project floor + CI target). Issue 2 — MAJOR: specs/engine-capability-uplift/ untracked from git git rm -r --cached specs/engine-capability-uplift/; specs/ added to .gitignore. Spec files remain on disk for operator/validator use; not re-published to the repo. Issue 3 — MAJOR: memory-bank writes missing PROGRESS.md: two engine-capability-uplift entries appended (delivery + revision). current_focus.md: updated resumption point to engine-capability-uplift @ round-1 fix. (Memory bank lives outside the repo — confirmed written to .agents/memory_bank/.)
Deliverables: - wallbreaker_mcp/ package with 4 tools: * wb_attack — run attack rounds vs LLM * wb_judge — score compliance with objective * wb_seed_list — list attack seed categories * wb_generate_payloads — generate adversarial payloads (strix E4, DispatchLayer regression) - scripts/smoke_mcp.sh — 6-check verification * p4rs3lt0ngv3_mcp imports ✓ * wallbreaker_mcp imports ✓ * All 4 tools callable ✓ * Graceful error handling ✓ - docs/mcp-integration.md — setup, tool reference, client compatibility table - docs/mcp_client_config.json — template for all MCP-compatible agents Client compatibility: Claude Code, Cursor, Windsurf, Gemini CLI, Eragon, Codex CLI Python entry: python -m wallbreaker_mcp (stdio + JSON-RPC 2.0)
- wallbreaker_mcp/server.py: 4 tools (wb_attack, wb_judge, wb_seed_list, wb_generate_payloads) - _STATIC_SEEDS: built-in seed bank (3 categories x 10 payloads) for offline use - wb_seed_list: always includes builtin source; no empty result without data downloads - wb_generate_payloads: static fallback when external datasets not cached - wb_judge: async grade() correctly bridged via asyncio.run() - wb_attack: graceful error when OPENAI_API_KEY not set - tests/test_wallbreaker_mcp.py: 14 focused AC tests, all passing - deps: pyyaml, pytest, pytest-asyncio added - smoke_mcp.sh: 6/6 checks pass - docs/mcp_client_config.json: valid JSON, mcpServers format Spec: wallbreaker-mcp-agent-wiring v1.1.0 (DispatchLayer) ACs: 10/10
wallbreaker_mcp/server.py: - wb_seed_list now includes builtin static categories as fallback when any source-specific query returns 0 external categories (was only on source='all'). Fixes AC4 for source='harmbench' when HarmBench data isn't cached. pyproject.toml: - Add hypothesis>=6.0 to [dependency-groups] dev (was only in [project.optional-dependencies] dev, not installed by uv sync) tests/ (9 files): - Add pytest.importorskip guards for optional deps (hypothesis, fastapi, PIL/Pillow) so tests skip gracefully instead of erroring during collection - Affected: test_security_properties, test_audit_remediation, test_tg1..tg7, test_typographic Result: 0 collection errors (was 9), 60 skipped, 1173 passed, 16 pre-existing failures (test_session_card, test_tg5_harden)
mcp 2.0.0 dropped mcp.server.fastmcp.FastMCP (now MCPServer at mcp.server.mcpserver). Pin to 1.x to restore CI until we migrate to the 2.0 API. Fixes 14 test failures in test_wallbreaker_mcp.py and test_mcp_bridge.py.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Security & Reliability Hardening — Dashboard Auth, SSRF Pinning, Tool Policy, Corpus Integrity, Signed Findings Log
This PR contributes the full hardening work done on the
pt-act/wallbreakerfork back toupstream. It combines two landed efforts into one coherent security series:
wallbreaker-audit.md:3 Critical, 13 High, 16 Medium, 14 Low, 4 Informational), previously merged to the fork as
PRs Remove dead/duplicate/unused code + fix undefined model_id #1/Fix: log no longer force-scrolls to bottom on every message #2/Load .env at CLI startup #3.
the shipped security code, restored a clean gated test baseline, finished deferred residuals,
and added a signed findings log + opt-in judge ensemble.
Capability/ASR work is intentionally excluded and will come as a separate PR (see
Scope below), so this series stays a focused, reviewable security change.
Why this PR exists
The dashboard shipped as an unauthenticated local FastAPI server whose routes could spawn
shell commands, write API keys to
.env, and fire attacks — reachable via browser CSRF from anypage the operator visited, and across the LAN if bound to
0.0.0.0. That was browser-driven RCErated it "Do not ship." This series closes that entire class and hardens the surrounding
reliability and supply-chain posture.
Finding table — Critical & High (representative)
ID | Severity | Fix -- | -- | -- SEC-1/2/3 | Critical | Per-launch bearer token (0600) + pure-ASGI SecurityMiddleware + Origin/Sec-Fetch-Site same-origin check on every /api/* route SEC-4 | Critical | SSRF egress guard (scheme allowlist, block loopback/link-local/metadata/RFC1918) + DNS-rebind socket-IP pinning (PinnedEgressBackend) SEC-5 | High | read_file realpath confinement + symlink rejection SEC-6/8 | High | Attack-firing + config/metadata routes behind auth+CSRF SEC-7 | High | Bind guard: refuses non-loopback --host without --allow-remote SEC-9/10/11 | High | Run-log redaction + 0600/0700 perms + path guard; Pydantic request models; global 500 handler REL-1/2/6/7 | High | Vision-judge NameError fix; provider lifecycle close at tool-call boundary; run force-stop + wall-clock timeout RACE-1..4 | High | Atomic state writes (tmp+fsync+os.replace+lock); cache delta format; gate/RunLog lockingVerification
1191 passed / 39 skipped / 31 xfailed,pytest -qexits 0.1175 passed / 55 skipped / 31 xfailed, exit 0 —corpus-dependent tests skip, nothing fails.
tscclean.egress fail-closed + DNS-rebind, corpus SHA gate, token 0600, signed-log tamper-evidence,
ensemble concurrency).
Scope
This PR is security & reliability only. The fork's separate capability track
(
engine-capability-uplift: semantic strategy retrieval, target-family bandit routing, agenticattack-surface completion, cross-family transfer) is deliberately not included and will be
proposed as its own PR so this series can be reviewed and merged on its security merits alone.
Responsible use
Wallbreaker is for authorized LLM red-teaming and safety evaluation only. This PR changes only
the harness's own security posture; it does not alter the tool's red-teaming capabilities or its
responsible-use doctrine.
Security & Reliability Hardening — Dashboard Auth, SSRF Pinning, Tool Policy, Corpus Integrity, Signed Findings Log
This PR contributes the full hardening work done on the pt-act/wallbreaker fork back to
upstream. It combines two landed efforts into one coherent security series:
Audit remediation — 50 findings from a full application audit (wallbreaker-audit.md:
3 Critical, 13 High, 16 Medium, 14 Low, 4 Informational), previously merged to the fork as
PRs #1.
Roadmap-implementation hardening — post-audit fixes that removed the fragility left in
the shipped security code, restored a clean gated test baseline, finished deferred residuals,
and added a signed findings log + opt-in judge ensemble.
Capability/ASR work is intentionally excluded and will come as a separate PR (see
Scope below), so this series stays a focused, reviewable security change.
Why this PR exists
The dashboard shipped as an unauthenticated local FastAPI server whose routes could spawn
shell commands, write API keys to .env, and fire attacks — reachable via browser CSRF from any
page the operator visited, and across the LAN if bound to 0.0.0.0. That was browser-driven RCE
credential exfiltration + SSRF-to-cloud-metadata on a "localhost dev tool." The original audit
rated it "Do not ship." This series closes that entire class and hardens the surrounding
reliability and supply-chain posture.
Finding table — Critical & High (representative)
ID Severity Fix
SEC-1/2/3 Critical Per-launch bearer token (0600) + pure-ASGI SecurityMiddleware + Origin/Sec-Fetch-Site same-origin check on every /api/* route
SEC-4 Critical SSRF egress guard (scheme allowlist, block loopback/link-local/metadata/RFC1918) + DNS-rebind socket-IP pinning (PinnedEgressBackend)
SEC-5 High read_file realpath confinement + symlink rejection
SEC-6/8 High Attack-firing + config/metadata routes behind auth+CSRF
SEC-7 High Bind guard: refuses non-loopback --host without --allow-remote
SEC-9/10/11 High Run-log redaction + 0600/0700 perms + path guard; Pydantic request models; global 500 handler
REL-1/2/6/7 High Vision-judge NameError fix; provider lifecycle close at tool-call boundary; run force-stop + wall-clock timeout
RACE-1..4 High Atomic state writes (tmp+fsync+os.replace+lock); cache delta format; gate/RunLog locking
"Do not ship → Safe to ship"
Before: unauthenticated browser-CSRF RCE, credential exfiltration, SSRF to metadata, a
confirmed vision-judge crash, HTTP client leak, non-atomic state with lost-update races.
After: authenticated + same-origin-gated API; least-privilege tool policy (host tools
opt-in only for the browser agent); SSRF guard with DNS-rebind pinning that fails closed if
the underlying transport shape changes; atomic state; WCAG 2.2 AA dashboard; supply-chain
corpus pinning; tamper-evident signed findings log; and a required CI gate.
What's new (roadmap-implementation layer on top of the audit fixes)
Egress de-fragilization — make_pinned_transport() self-checks that the pinned backend is
installed and raises rather than returning an un-pinned transport if httpx internals change;
httpx pinned to a verified range and matrix-tested. Two-tier policy documented: advisory
check_url (fail-open on NXDOMAIN) can never widen the enforcing PinnedEgressBackend
(fail-closed).
Supply-chain corpus integrity — library.lock.toml pins each runtime-fetched corpus to a
commit SHA; loader fails closed on mismatch/unresolved; wallbreaker corpus verify CLI.
Reusable hardening toolkit — agent_dashboard_harden/ re-exports the security layer
(SecurityMiddleware, egress guard, tool policy) with zero behavior change, plus
parameterizable PBT fixtures for the 5 security-property categories.
Signed findings log — wallbreaker/findings_log.py: append-only Ed25519-signed JSONL;
tamper-evident; the private key is never included in the exported bundle.
Opt-in judge ensemble — judging.run_ensemble: up to 3 judges concurrently, majority-vote
label + mean±1σ, low-agreement verdicts flagged UNCERTAIN; single-judge default unchanged.
Test baseline & CI gate — pre-existing corpus-dependent failures quarantined
(xfail/skipif); .github/workflows/redteam-gate.yml runs the PBT suite + an httpx version
matrix + -W error::ResourceWarning as required checks.
Frontend — the three oversized dashboard components decomposed below the 400-line guideline
with a check:line-counts guard; new vitest + jest-axe coverage.
New / notable files
Path Purpose
wallbreaker/dashboard/auth.py Pure-ASGI token + Origin/CSRF gate (SEC-1/2/3)
wallbreaker/tools/egress_guard.py SSRF guard + PinnedEgressBackend + make_pinned_transport (SEC-4, fail-closed)
wallbreaker/tools/tool_policy.py Least-privilege registry for the browser agent
agent_dashboard_harden/ Reusable, zero-behavior-change security toolkit + PBT fixtures
wallbreaker/findings_log.py Ed25519 signed findings log
library.lock.toml + wallbreaker/tools/parsel_engine.py Corpus SHA pinning + verifier
tests/pbt/test_security_properties.py, tests/test_tg{3,5,7}_*.py Security/correctness properties
Verification
Backend (warm): 1191 passed / 39 skipped / 31 xfailed, pytest -q exits 0.
Backend (cold checkout, corpora absent): 1175 passed / 55 skipped / 31 xfailed, exit 0 —
corpus-dependent tests skip, nothing fails.
Frontend: 60 vitest tests / 12 files, jest-axe clean, tsc clean.
PBT: 8 security/correctness properties execute in the committed runner (access control,
egress fail-closed + DNS-rebind, corpus SHA gate, token 0600, signed-log tamper-evidence,
ensemble concurrency).
Independently PM-validated across 3 rounds (verdict: approved).
Scope
This PR is security & reliability only. The fork's separate capability track
(engine-capability-uplift: semantic strategy retrieval, target-family bandit routing, agentic
attack-surface completion, cross-family transfer) is deliberately not included and will be
proposed as its own PR so this series can be reviewed and merged on its security merits alone.
Responsible use
Wallbreaker is for authorized LLM red-teaming and safety evaluation only. This PR changes only
the harness's own security posture; it does not alter the tool's red-teaming capabilities or its
responsible-use doctrine.