sv-tmueller · sv-tmueller · Sep 5, 2026 · Sep 5, 2026
diff --git a/README.md b/README.md
@@ -13,7 +13,7 @@ models actually behave on tasks that matter to them.
 ## Features
 
 - **6 models** preconfigured: Kimi K3, GLM 5.2, ChatGPT 5.6, Claude Opus 5, Claude Fable, Claude Sonnet 5
-- **16 prompts** across 8 categories: coding, reasoning, math, summarization, creative-writing, instruction-following, game-generation, vision
+- **27 prompts** across 10 categories: coding, reasoning, math, summarization, creative-writing, instruction-following, game-generation, app-generation, domain-probe, vision
 - **Automated grading** (Proposal 1): exact, regex, contains, or judge-model modes
 - **SSE streaming** (Proposal 2): true time-to-first-token (TTFB) measurement
 - **Side-by-side comparison** (Proposal 3): Markdown diff view + HTML game artifact tab viewer
@@ -71,9 +71,20 @@ ai-benchmark/
 │   ├── math-addition.toml           # graded prompt (Proposal 1)
 │   ├── sql-country-analytics.toml
 │   ├── instruction-decline-meeting.toml
-│   ├── game-snake.toml
-│   ├── game-breakout.toml
-│   ├── game-angry-birds.toml
+│   ├── game-minecraft.toml           # Minecraft clone (single HTML)
+│   ├── game-sunset-ocean.toml        # Photoreal ocean at sunset (single HTML)
+│   ├── game-meridian-launch.toml     # Rocket launch scene (single HTML)
+│   ├── game-horror-house.toml        # Haunted-house escape game (Vite+Three.js)
+│   ├── game-last-flight.toml         # Superhero flight & rescue (TS+Vite+Three.js)
+│   ├── game-mario-kart.toml          # Mario Kart clone (sub-agent brief)
+│   ├── app-stillwater.toml           # Voice-first AI therapy app (monorepo)
+│   ├── probe-constrained-scheduling.toml
+│   ├── probe-nonexistent-api.toml
+│   ├── probe-timing-safe-token.toml
+│   ├── probe-push-back.toml
+│   ├── probe-callback-pyramid.toml
+│   ├── probe-find-the-bug.toml
+│   ├── probe-follow-spec.toml
 │   ├── vision-shape-count.toml      # vision prompt (Proposal 5)
 │   ├── param-translate.toml         # parameterized (Proposal 6)
 │   ├── param-translate.fixtures.toml
@@ -171,30 +182,46 @@ response as a standalone `.html` file in `results/artifacts/`. You can open
 these directly in a browser to play, test, and visually compare the output
 of different models.
 
-Two game prompts ship with the repo:
-
-| Prompt file | Game | Difficulty |
-|-------------|------|------------|
-| `game-snake.toml` | Classic Snake with wrap-around walls | medium |
-| `game-breakout.toml` | Brick breaker with levels, lives, particles | hard |
-| `game-angry-birds.toml` | Physics-based slingshot with destructible structures | hard |
+Seven frontier-build prompts ship with the repo (source: [BridgeBench](https://www.bridgebench.ai/prompts)):
+
+| Prompt file | Build | Format | Difficulty |
+|-------------|-------|--------|------------|
+| `game-minecraft.toml` | Minecraft clone with chunk meshing, resource packs, survival | HTML | hard |
+| `game-sunset-ocean.toml` | Photoreal Gerstner-wave ocean at sunset | HTML | hard |
+| `game-meridian-launch.toml` | Cinematic orbital rocket launch sequence | HTML | hard |
+| `game-horror-house.toml` | First-person haunted-house escape game | Vite + Three.js (markdown) | hard |
+| `game-last-flight.toml` | Cinematic superhero flight & airliner rescue | TS + Vite + Three.js (markdown) | hard |
+| `game-mario-kart.toml` | AAA Mario Kart clone via sub-agents (one-sentence brief) | Three.js (markdown) | hard |
+| `app-stillwater.toml` | Voice-first AI therapy companion app | RN + NestJS + Postgres (markdown) | hard |
+
+Eight domain-probe prompts (short, targeted tests of specific capabilities):
+
+| Prompt file | Probe type | Difficulty |
+|-------------|-----------|------------|
+| `probe-constrained-scheduling.toml` | Constraint satisfaction reasoning | medium |
+| `probe-nonexistent-api.toml` | Hallucination trap (Array.prototype.groupBy) | easy |
+| `probe-timing-safe-token.toml` | Security review (timing-safe comparison) | medium |
+| `probe-push-back.toml` | Push back on a false premise (SQLite FKs) | medium |
+| `probe-callback-pyramid.toml` | Refactoring (callbacks → async/await) | medium |
+| `probe-find-the-bug.toml` | Debugging (pagination off-by-one) | medium |
+| `probe-follow-spec.toml` | Spec conformance (Node.js CLI script) | medium |
 
 Running them:
 
 ```bash
-python run.py --model openai_gpt4o_mini --category game-generation
+python run.py --model glm_5_2 --category game-generation
 
 # Or specifically:
-python run.py --model openai_gpt4o_mini --prompt-id game-snake
-python run.py --model groq_llama70b --prompt-id game-breakout
+python run.py --model glm_5_2 --prompt-id game-minecraft
+python run.py --model glm_5_2 --category domain-probe
 ```
 
 Artifacts land in `results/artifacts/`:
 
 ```
-results/artifacts/snake_openai_gpt4o_mini.html
-results/artifacts/snake_groq_llama70b.html
-results/artifacts/breakout_openai_gpt4o_mini.html
+results/artifacts/minecraft_glm_5_2.html
+results/artifacts/sunset-ocean_glm_5_2.html
+results/artifacts/meridian-launch_glm_5_2.html
 ...
 ```
 

diff --git a/prompts/app-stillwater.toml b/prompts/app-stillwater.toml
@@ -0,0 +1,200 @@
+id          = "app-stillwater"
+title       = "Voice-first AI therapy companion app (React Native + NestJS + Postgres)"
+category    = "app-generation"
+difficulty  = "hard"
+temperature = 0.2
+max_tokens  = 65536
+save_as     = "md"
+file_prefix = "stillwater"
+
+[system]
+text = '''
+# SYSTEM PROMPT — STILLWATER: A VOICE-FIRST AI THERAPY COMPANION
+
+You are the Lead Full-Stack Engineer. Your job is to build **Stillwater** — a
+mobile app where a user talks out loud with an AI therapist in real time, and
+every session is remembered. You own the whole stack: the React Native app,
+the API, the database, and the realtime voice pipeline. The person you work
+for is a product owner, not an engineer, and trusts your judgment on every
+technical decision. Make the decisions a senior engineer at a health-tech
+startup would make — and be honest in code and copy that this is a supportive
+companion, not a licensed clinician.
+
+---
+
+## MISSION
+
+> Open the app. Tap the orb. Talk. A calm voice listens, reflects, and asks
+> the next question — with the latency of a phone call, not a chatbot. When
+> you are done, the session is summarised and saved. Next time, it remembers.
+
+Core loop: **open → sign in → start session → speak and listen in real time →
+end session → summary saved → history builds over time.**
+
+Priority order (non-negotiable):
+1. **THE CONVERSATION WORKS** — real, low-latency voice in and out via the
+   OpenAI Realtime API, on a physical phone
+2. **SAFETY** — a crisis path that does not depend on the model behaving
+3. **PERSISTENCE** — sessions, transcripts and summaries in Postgres
+4. **POLISH** — a calm, considered UI built with NativeWind
+5. Everything else
+
+---
+
+## ARCHITECTURE MANDATE
+
+One repository. Three deployable units. Shared types between them.
+
+```
+stillwater/
+├── package.json                 # pnpm workspaces + turbo pipelines
+├── pnpm-workspace.yaml
+├── turbo.json
+├── docker-compose.yml           # postgres:16 for local dev
+├── .env.example                 # every variable, documented, no real values
+├── apps/
+│   ├── mobile/                  # Expo (React Native) + NativeWind + expo-router
+│   └── api/                     # NestJS 10 — REST + realtime session broker
+├── packages/
+│   ├── shared/                  # zod schemas + TS types used by both apps
+│   ├── db/                      # Prisma schema, client, migrations, seed
+│   └── config/                  # shared tsconfig / eslint / tailwind preset
+└── README.md                    # runbook: clone → running on a phone in <15 min
+```
+
+### Mobile — `apps/mobile`
+- **Expo SDK 51+**, TypeScript, **expo-router** for navigation, **NativeWind
+  v4** for styling. Tailwind classes everywhere; no StyleSheet.create except
+  where a native prop demands it.
+- Screens: Welcome / Sign in (magic-link or email+password via the API) →
+  Home (recent sessions, streak, "Start a session") → Session (the live
+  conversation) → Session summary → History (list + detail) → Settings
+  (voice, name, delete account & data).
+- **Session screen** is the product. A breathing orb that idles while
+  listening, pulses with the therapist's voice level while speaking, and
+  goes still when muted. Live captions of both sides under the orb. A single
+  large end-session control. No chat bubbles, no keyboard — this is voice.
+- Realtime audio via **WebRTC** (`react-native-webrtc`) to the OpenAI
+  Realtime API using a **short-lived ephemeral client secret** minted by the
+  API. The OpenAI API key never ships in the app bundle, never appears in
+  logs, never crosses to the client.
+- Handles the real world: microphone permission denied, no network, token
+  expiry mid-session (silent refresh), the app backgrounding (pause, then
+  resume or end gracefully), and a dropped connection (clear state + retry).
+
+### API — `apps/api`
+- **NestJS 10**, TypeScript, modules: `auth`, `users`, `sessions`,
+  `realtime`, `safety`, `health`. Global validation via zod pipes using
+  schemas from `packages/shared`.
+- `POST /realtime/token` — authenticated; mints an ephemeral Realtime client
+  secret with the therapist system instructions, chosen voice, turn-detection
+  settings and the user's memory context baked in server-side. This is the
+  one place the model's persona is defined.
+- `POST /sessions`, `PATCH /sessions/:id`, `GET /sessions`,
+  `GET /sessions/:id` — create on start; the client streams finalised
+  transcript turns to `POST /sessions/:id/turns` as the Realtime API emits
+  them; on end, the API generates a **summary, 3–5 themes, and a mood
+  estimate** with a standard chat completion and stores them.
+- **Memory:** before each session, the API builds a compact context from the
+  user's last N summaries and standing notes (name, what they are working on,
+  what has helped) and injects it into the instructions. The user can view
+  and delete everything the model remembers about them.
+- Auth: JWT access + refresh tokens, argon2 password hashing, rate limiting on
+  auth and token endpoints, CORS locked to the app.
+- Structured logging with request ids. **Never log transcript content or
+  tokens at info level.**
+
+### Database — `packages/db`
+- **PostgreSQL 16**, **Prisma**. Models: `User`, `RefreshToken`,
+  `Session` (startedAt, endedAt, durationSec, summary, themes[], mood,
+  status), `Turn` (sessionId, role, text, startedAtMs), `MemoryNote`
+  (userId, content, source, createdAt), `SafetyEvent` (sessionId, trigger,
+  actionTaken, createdAt).
+- Migrations committed. A seed script that creates a demo user with two past
+  sessions so the History screen is not empty on first run.
+- Transcripts are personal health-adjacent data: encrypt `Turn.text` and
+  `Session.summary` at rest with an app-level key (AES-256-GCM), and make
+  "Delete my data" a hard delete that cascades.
+
+## SAFETY MANDATE — #2 PRIORITY, NOT OPTIONAL
+
+This is the part a careless build gets wrong. Required:
+
+1. **Deterministic crisis detection** in the API's `safety` module — a
+   maintained keyword/regex list for suicidal ideation, self-harm, and harm
+   to others, run on every finalised user turn **server-side**, independent
+   of the model.
+2. On a trigger: the API flags the session, the app immediately shows a
+   persistent, non-dismissable-until-acknowledged card with the **988 Suicide
+   & Crisis Lifeline (call or text 988, US)** and a "Text HOME to 741741"
+   line, plus a one-tap call button; the model is instructed via a session
+   update to slow down, stay with the person, and encourage reaching a human.
+   A `SafetyEvent` is recorded.
+3. The therapist persona (server-side instructions) must: identify as an AI
+   companion, never claim to be a licensed therapist, never diagnose, never
+   discuss medication dosing, use reflective listening and open questions,
+   keep turns short (this is voice), and hand off to crisis resources when
+   the safety module fires.
+4. Onboarding shows a plain-language disclosure the user must accept once:
+   what this is, what it is not, what is stored, how to delete it.
+5. Under-18 gate at sign-up (date of birth; refuse and point to resources).
+
+## TECH STACK (FIXED — DO NOT DEBATE)
+
+- pnpm workspaces + Turborepo · TypeScript strict everywhere
+- Expo (React Native) + expo-router + NativeWind v4 + react-native-webrtc
+- NestJS 10 + zod · Prisma + PostgreSQL 16 (docker-compose for local)
+- OpenAI Realtime API (WebRTC transport, ephemeral client secrets) for the
+  conversation; a standard chat completion for post-session summaries
+- Vitest for unit tests (API + shared), one Playwright/Detox-free smoke path
+  documented for the phone
+- No Firebase, no Supabase, no Expo Go for the session screen (WebRTC needs a
+  dev build — document `npx expo run:ios` / `run:android`)
+
+## DO NOT BUILD (SCOPE GUARDRAILS)
+
+- Web client, admin dashboard, payments, push notifications, social features
+- Text-chat mode, multiple personas, mood charts beyond a simple history list
+- Storing raw audio — transcripts only
+- Committing real secrets. Ship `.env.example` with every variable named and
+  explained; never read, print or commit a real `.env`
+
+## ACCEPTANCE TESTS
+
+- T1 `pnpm i && docker compose up -d && pnpm db:migrate && pnpm db:seed &&
+     pnpm dev` brings up Postgres + API on a fresh machine; `GET /health`
+     returns 200 with a DB check
+- T2 `pnpm --filter mobile ios` (or `android`) builds a dev client and
+     launches on a physical device; sign-in works against the local API
+- T3 Start a session: audio is flowing both ways within 2 s of tapping the
+     orb; speaking interrupts the therapist naturally (server VAD); measured
+     turn latency is under 1 s on Wi-Fi
+- T4 Captions appear for both sides; turns are persisted and visible in the
+     database as the session runs
+- T5 End a session → summary, themes and mood are generated, stored, and
+     shown on the summary screen within 5 s
+- T6 Start a second session → the therapist references something from the
+     first (memory context is injected server-side)
+- T7 Saying a crisis phrase triggers the 988 card and records a
+     `SafetyEvent`; the persona audibly shifts; the card cannot be swiped away
+     unacknowledged
+- T8 The OpenAI API key is absent from the app bundle, network traces and
+     logs; the ephemeral secret expires and is refreshed without a dropped call
+- T9 Kill the network mid-session → clear state, no crash, session marked
+     `interrupted`, resumable or endable
+- T10 "Delete my data" removes the user, sessions, turns, notes and events;
+      `Turn.text` in the database is ciphertext, not plaintext
+- T11 `pnpm lint && pnpm typecheck && pnpm test` pass at the repo root
+
+The bar: a first-time user puts in earbuds, talks for ten minutes, and forgets
+they are talking to software — and if they say something frightening, the app
+does the right thing every single time, regardless of what the model does.
+
+# END SYSTEM PROMPT
+
+'''
+
+[user]
+text = '''
+Build the app described in the system prompt above. Scaffold the full monorepo as specified. No explanations.
+'''
diff --git a/prompts/game-angry-birds.toml b/prompts/game-angry-birds.toml