feat(verification): independent adversarial finding verification#822
feat(verification): independent adversarial finding verification#822kusonooyasumi wants to merge 3 commits into
Conversation
Add an opt-in verification pass (STRIX_VERIFY_FINDINGS + STRIX_VERIFICATION_MODEL) that sends every candidate finding to an independent model which tries to refute it before the finding is persisted. Fail-closed: rejected findings and verifier errors never reach customer-facing artifacts. The verifier can use its own model, reasoning effort, and API key, and its usage counts toward --max-budget-usd.
c7723d5 to
8211504
Compare
Greptile SummaryThis PR adds optional independent model verification before findings are saved. The main changes are:
Confidence Score: 4/5Verifier credential handling needs to be fixed before merging.
strix/interface/main.py and strix/report/verification.py
|
| Filename | Overview |
|---|---|
| strix/report/verification.py | Adds fail-closed verification, response parsing, usage recording, and budget checks, but does not reliably apply a separate same-provider key. |
| strix/interface/main.py | Adds verifier warm-up without applying the verifier-specific credential first. |
| strix/tools/reporting/tool.py | Gates both dynamic and dependency findings before persistence. |
| strix/report/state.py | Persists verification metadata and carries the scan budget into report state. |
| strix/report/writer.py | Renders verification status while remaining compatible with older reports. |
| strix/config/settings.py | Adds verifier configuration and validates that enabled verification has a model. |
| tests/test_finding_verification.py | Covers defaults, model validation, and rejected findings, but not verifier credential selection. |
Prompt To Fix All With AI
Fix the following 2 code review issues. Work through them one at a time, proposing concise fixes.
---
### Issue 1 of 2
strix/interface/main.py:358-371
**Verifier Key Missing During Warm-Up**
When the verification model uses a different provider and its credential exists only in `VERIFICATION_LLM_API_KEY`, this warm-up runs before that key is installed. The primary model can connect successfully while the verifier fails authentication, causing startup to exit even though the real verification path would configure the key.
### Issue 2 of 2
strix/report/verification.py:90-94
**Primary Key Overrides Verifier Key**
When the primary and verification models use the same provider with different credentials, `configure_sdk_model_defaults()` installs the primary key first and `_mirror_api_key_to_provider_env()` only fills missing environment variables. The configured verification key can therefore be ignored, so verification uses the primary account and can fail model access or charge the wrong quota.
Reviews (1): Last reviewed commit: "feat(verification): independent adversar..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| verifier = StrixProvider().get_model(verification_model) | ||
| await asyncio.wait_for( | ||
| verifier.get_response( | ||
| system_instructions="You are a helpful assistant.", | ||
| input="Reply with just 'OK'.", | ||
| model_settings=ModelSettings(), | ||
| tools=[], | ||
| output_schema=None, | ||
| handoffs=[], | ||
| tracing=ModelTracing.DISABLED, | ||
| previous_response_id=None, | ||
| conversation_id=None, | ||
| prompt=None, | ||
| ), |
There was a problem hiding this comment.
When the verification model uses a different provider and its credential exists only in VERIFICATION_LLM_API_KEY, this warm-up runs before that key is installed. The primary model can connect successfully while the verifier fails authentication, causing startup to exit even though the real verification path would configure the key.
Prompt To Fix With AI
This is a comment left during a code review.
Path: strix/interface/main.py
Line: 358-371
Comment:
**Verifier Key Missing During Warm-Up**
When the verification model uses a different provider and its credential exists only in `VERIFICATION_LLM_API_KEY`, this warm-up runs before that key is installed. The primary model can connect successfully while the verifier fails authentication, causing startup to exit even though the real verification path would configure the key.
How can I resolve this? If you propose a fix, please make it concise.| configure_sdk_model_defaults(settings) | ||
| if verification.api_key: | ||
| from strix.config.models import _mirror_api_key_to_provider_env | ||
|
|
||
| _mirror_api_key_to_provider_env(model_name, verification.api_key) |
There was a problem hiding this comment.
When the primary and verification models use the same provider with different credentials, configure_sdk_model_defaults() installs the primary key first and _mirror_api_key_to_provider_env() only fills missing environment variables. The configured verification key can therefore be ignored, so verification uses the primary account and can fail model access or charge the wrong quota.
Prompt To Fix With AI
This is a comment left during a code review.
Path: strix/report/verification.py
Line: 90-94
Comment:
**Primary Key Overrides Verifier Key**
When the primary and verification models use the same provider with different credentials, `configure_sdk_model_defaults()` installs the primary key first and `_mirror_api_key_to_provider_env()` only fills missing environment variables. The configured verification key can therefore be ignored, so verification uses the primary account and can fail model access or charge the wrong quota.
How can I resolve this? If you propose a fix, please make it concise.Provider env vars are global, so installing the verification key via the environment either clobbered or was clobbered by the primary key when both used the same provider (setdefault only fills missing vars). Pass the verification key as a per-call api_key in model_settings.extra_args instead, in both the runtime path and startup warm-up, so a separate-provider verifier authenticates correctly and warm-up no longer fails before the key exists.
Addressed review feedback — verifier credentialsTwo issues flagged:
Fix ( New tests assert the key rides on the request and is omitted when unset. |
Add VERIFICATION_LLM_API_BASE and send the verifier's api_base per call alongside its api_key. The process-wide OPENAI_BASE_URL / litellm api_base is set from the main model, so a verifier on a different endpoint (self-hosted gateway, separate provider) could not be reached. Passing api_base in model_settings.extra_args keeps the verifier endpoint independent of the main model, in both the runtime path and startup warm-up.
Addressed review feedback1. Missing API base URL. 2. Verification should be a real agent that independently tests, not a one-shot. Agreed — the current That follow-up is stacked on the routing PR (#824) because agentic verification needs per-agent models — otherwise the verifier would run on the main scan model instead of |
1 participant
Summary
Adds an opt-in independent verification pass for findings. When enabled, every candidate finding is sent to a separate model that tries to refute it before it is persisted — reducing false positives in customer-facing artifacts.
This is one of several split-out PRs for fine-grained model control. It is fully self-contained and targets
maindirectly (no dependency on the model-routing / budget-fallback PRs).Behavior
STRIX_VERIFY_FINDINGS=true.llm_usageledger and counts toward--max-budget-usd.Config
STRIX_VERIFY_FINDINGSfalse)STRIX_VERIFICATION_MODELVERIFICATION_LLM_API_KEYSTRIX_VERIFICATION_REASONING_EFFORThigh)Config validation raises immediately if
STRIX_VERIFY_FINDINGSis on but no model is set.Changes
strix/report/verification.py(new) — adversarial refutation call, fail-closed parsing.strix/tools/reporting/tool.py— gate both vuln and dependency findings on verification.strix/report/state.py/writer.py— persist + render verification status; carrymax_budget_usdin scan config.strix/config/settings.py—FindingVerificationSettings.Test plan
pytest tests/test_finding_verification.py— defaults-off, required-model validation, and rejected-finding-not-persisted (fail-closed).ruff+mypyclean.Note:
tests/test_runner_root_prompt.pycurrently fails onmain(its settings mock lacksruntimewhile the runner readssettings.runtime.max_context_images). That is a pre-existing upstream issue, unrelated to this PR, which does not touch the runner.