Skip to content

Latest commit

 

History

History
117 lines (93 loc) · 6.43 KB

security-model.md

File metadata and controls

117 lines (93 loc) · 6.43 KB

Security Model

This project treats every user message as hostile until proven otherwise. This document describes the threat model, the five defense layers, and the residual risks that remain.

Threat model

Assets to protect:

  1. The scope constraint: the assistant must only discuss the profile owner.
  2. The system prompt and its security tokens.
  3. The API key.
  4. The free-tier quota (an attacker should not be able to burn it cheaply).

Adversary capabilities: full control over the text of every question, including Unicode tricks, encodings, multi-turn setups, and social engineering of the model. The adversary cannot see the server side, the environment, or the source of randomness.

The five layers

Layer Module Defends against
1. Sanitizer security/sanitizer.py Invisible characters, full-width and compatibility Unicode disguises, control characters, oversized inputs
2. Injection scanner security/injection.py Known injection phrasing, role markers, jailbreak keywords, encoded payloads, delimiter mimicry
3. Scope pre-check engine.py Off-topic questions, using a second model as an independent judge
4. Hardened prompt prompting.py Instruction override, persona hijack, prompt extraction
5. Output guard security/output_guard.py Successful prompt leaks, canary exposure, empty or blocked responses

Layer 1: sanitization

NFKC normalization folds disguised characters into canonical forms before pattern matching, so an attack written in full-width characters looks like plain ASCII to the scanner. Zero-width characters, bidirectional control marks, and other invisible code points are stripped, closing the hidden-text channel. Input length is capped to bound both cost and attack surface.

Layer 2: injection scanning

A weighted signature set scores each message: instruction overrides, prompt extraction attempts, persona hijacks, chat role markers, jailbreak keywords, authority claims, base64 and hex blobs, and attempts to mimic the project's own delimiters. At or above the threshold (default 3) the message is blocked locally, which costs the attacker a try and costs you zero tokens. The signatures are intentionally conservative so normal questions do not trip them.

Layer 3: scope pre-check

An independent, cheap model receives only a preview of the profile and the question, and answers IN_SCOPE or OUT_OF_SCOPE. Because this classifier has its own instructions and no conversation history, an injection crafted for the main conversation rarely transfers to it. The check fails open by design, see architecture.md.

Layer 4: the hardened prompt

The system instruction pins the model to the profile context and defines an exact refusal sentence. The user message is spotlighted: wrapped in boundary markers built from secrets.token_hex, with an explicit rule that everything inside the markers is data, never instructions. The prompt also carries a canary token the model is told to never output. Low temperature (default 0.2) reduces creative drift away from the rules.

Layer 5: output review

The response is scanned for the canary, the rule sentinel, the boundary marker, and the context delimiters. Any hit means the model was manipulated into leaking its instructions, and the response is replaced with the standard refusal. Empty responses (typically safety-filter blocks) are also converted to refusals instead of confusing blank output.

Supporting controls

  • The API key lives in the environment only, is typed as SecretStr, and is masked in diagnostics output.
  • .gitignore excludes .env and personal profile files by default.
  • A client-side token bucket keeps request rates under free-tier ceilings.
  • Retries are bounded with jittered exponential backoff, so failures cannot turn into request storms.
  • Explicit Gemini safety settings block harmful content categories at medium-and-above thresholds.
  • CI runs bandit, pip-audit, and gitleaks on every push, and the bandit trojan-source check guards against bidirectional-character attacks on this project's own source code.
  • Blocked exchanges never enter conversation history.

Web deployment threats

The public web UI adds an internet-facing surface with its own controls:

Threat Control
Poisoned conversation replay Every replayed message passes the sanitizer and injection scanner server-side; failing pairs are dropped before they reach the model
Fabricated assistant turns in the replay Assistant messages in the replay are scanned with the same signatures as questions
Request flooding and quota burning Per-IP sliding window limiter, then the outbound token bucket, then Gemini's own quotas. Injection attempts are blocked before any token is spent
Arbitrary model invocation The model field is validated against the curated free-tier allowlist
Information leakage through errors Failures return a generic message; details go to server logs only. API docs endpoints are disabled
Clickjacking, script injection, sniffing Strict security headers on every page: a same-origin Content Security Policy, X-Frame-Options DENY, nosniff, HSTS, and a locked-down Permissions-Policy
Response caching of personal answers Every API response carries Cache-Control: no-store
Streaming leaks Responses never stream; the output guard reviews the complete text before the browser receives anything. The client then renders the reviewed Markdown
Rendered-answer XSS Answers are rendered with react-markdown and no raw-HTML plugin, so HTML in a model response is escaped, not executed. Links are forced to open with rel="noopener noreferrer nofollow"

Residual risks, stated honestly

  • A novel, well-crafted injection can still get an off-topic sentence out of the model. No known technique reduces this risk to zero. The layers here make the cheap attacks free to block and the expensive attacks detectable when they matter most (prompt leakage).
  • The heuristic scanner can produce false positives on unusual but legitimate questions. Tune CG_INJECTION_THRESHOLD if your use case trips it.
  • The profile context itself is sent to Google's API with every request. Only put information there that you are comfortable sharing with the API provider and with anyone who chats with the assistant.
  • Free-tier quotas are enforced per project by Google. This tool limits its own rate but cannot stop other consumers of the same key.