A6: connect the tone model to the mac server

docs/GRAPTION_CONTEXT.md

@@ -329,8 +329,9 @@ for caption [t0, t1]:
   replies to each schema-v1.1 `tone_request` with a matching `tone_result`, `tag: null`,
   zero probabilities (no prediction), and `model_version: "tone-stub-v1"` through
   `MockToneBackend`. `caption_log` is accepted and discarded until logging is implemented.
-  Audio stays in memory only. Real inference, SQLite logging, and deployment packaging
-  remain later work; the skeleton always uses the stub regardless of `TONE_BACKEND`.
+  Audio stays in memory only. `TONE_BACKEND=audeering` swaps in the real audEERING baseline
+  (A6, `server/app/audeering.py`); the default is still the stub (`mock`). SQLite logging and
+  deployment packaging remain later work.
   Invalid session messages return `session_error` and keep the connection open. Rejection
   warnings contain only a fixed rejection code, generated connection ID, and backend model version. The response
   echoes the received request to its originating client; it is never stored or logged.
@@ -351,12 +352,14 @@ for caption [t0, t1]:
   3. Learned softmax-weighted sum over the 13 layers, then mean-pool over time, giving a 768-dim vector.
   4. Head: `Linear(768,256) → ReLU → Dropout(0.3) → Linear(256,6) → softmax`.
   5. Tag mapper: `tag = argmax` if `max prob ≥ 0.6` and the argmax isn't `neutral`; otherwise `null`.
-- **Week-1 baseline mode** (`TONE_BACKEND=audeering`): the audEERING model outputs arousal, valence, and dominance in [0,1]. Map them with tunable thresholds:
-  - arousal > 0.65 and valence > 0.55 → `excited`
-  - arousal > 0.65 and valence < 0.4 → `upset`
-  - arousal < 0.3 → `calm`
-  - otherwise `null`
-- **Config (env vars):** `TONE_BACKEND` (`audeering` | `custom`), `TONE_HEAD_PATH`, `TAG_THRESHOLD`, `TONE_TOKEN`, `DEPLOY_MODE` (`local` | `tunnel` | `cloud`), `LOG_EXPORT_BUCKET` (cloud only), `TONE_LOG_DIR` (default repo-root `logs/`).
+- **Week-1 baseline mode** (`TONE_BACKEND=audeering`, implemented): the audEERING model outputs arousal, valence, and dominance in [0,1]. Map them with tunable thresholds (`AudeeringThresholds` in `server/app/tone.py`). These are the values tuned on team recordings in A4 (`ml/tone/DISCUSSION_LOG.md`), replacing the original 0.65/0.55/0.4/0.3 guesses:
+  - arousal > 0.64 and valence > 0.45 → `excited`
+  - arousal > 0.47 and valence < 0.4 → `upset`
+  - 0.38 < arousal < 0.48 and valence > 0.45 → `calm`
+  - otherwise `null`; segments under 0.5 s are always `null`
+  - `probs` are all zero in this mode: the schema's 6 emotions don't apply to arousal/valence.
+  - Weights load from the local Hugging Face cache (`uv run hf download audeering/wav2vec2-large-robust-12-ft-emotion-msp-dim`); torch, transformers, and scipy come from the uv workspace (`graption-ml`), not `graption-server`'s own dependencies.
+- **Config (env vars):** `TONE_BACKEND` (`mock` default | `audeering` | `custom` later), `TONE_HEAD_PATH`, `TAG_THRESHOLD`, `TONE_TOKEN`, `DEPLOY_MODE` (`local` | `tunnel` | `cloud`), `LOG_EXPORT_BUCKET` (cloud only), `TONE_LOG_DIR` (default repo-root `logs/`).
 - **CPU performance:** the audEERING baseline is the large wav2vec2 variant (~165M params) and may be slow on CPU. Benchmark per-sentence latency on Cloud Run before user testing. If it's over ~1s, use the custom WavLM-base head in cloud mode.
 - **SQLite tables:** `sessions`, `captions` (from `caption_log`), `tone_results` (probs, model_version, latency).
   - Cloud Run disks are ephemeral, so in cloud mode, export the SQLite file to `LOG_EXPORT_BUCKET` at session end.

scripts/README.md

@@ -32,3 +32,29 @@ Audio is read in chunks, without sending the WAV header or creating recordings.
 The file's sample rate is sent unchanged; 16 kHz is recommended. Other bit depths,
 stereo, and compressed WAV files must be converted first. Timestamps emulate a
 phone session starting at zero using sample offsets. Keep recordings out of git.
+
+# Tone accuracy evaluation
+
+`eval_tone.py` runs a folder of labeled clips through the server's audEERING backend
+(the same code as `TONE_BACKEND=audeering`) and prints accuracy, the wrong-tag rate,
+missed tones, per-tag precision/recall, a confusion table, and mean arousal/valence per
+label. Download the model first (see `server/README.md`).
+
+Labels come from filenames: A4's prefixes (`excited_*`, `upset_*`, `calm_*`, `none_*` or
+`neutral_*`) or CREMA-D names. Any soundfile format works (WAV, MP3, FLAC, OGG).
+
+```bash
+uv run python scripts/eval_tone.py path/to/clips --csv data/results.csv
+```
+
+For a public benchmark, download a CREMA-D sample (ODbL; acted speech) into the gitignored
+`data/` folder. 20 actors is about 1,300 clips and 100 MB:
+
+```bash
+uv run python scripts/fetch_crema_d.py --actors 20
+uv run python scripts/eval_tone.py data/crema-d --csv data/crema-d-results.csv
+```
+
+Only anger (upset), happy (excited), and neutral (none) are scored; sad, fear, and disgust
+are reported but not scored, and CREMA-D has no calm. It is acted, so team recordings
+remain the real test set. Keep clips and results out of git.

scripts/eval_tone.py

@@ -0,0 +1,149 @@
+"""Measure tone-tag accuracy of the server's audEERING backend on labeled clips.
+
+Labels come from filenames: CREMA-D names (1001_DFA_ANG_XX.wav) or A4's prefix convention
+(excited_*, upset_*, calm_*, none_* / neutral_*). Clips are read into memory only.
+"""
+
+import argparse
+import csv
+import re
+import sys
+import time
+from collections import Counter, defaultdict
+from pathlib import Path
+
+import numpy as np
+import soundfile as sf
+from scipy.signal import resample_poly
+
+from app.audeering import SAMPLE_RATE, AudeeringToneBackend
+from app.tone import map_tag
+
+TAGS = ("excited", "upset", "calm")
+SCORED = (*TAGS, "none")
+CREMA_D = re.compile(r"^\d{4}_[A-Z]{3}_(?P<emotion>[A-Z]{3})_[A-Z]{2}$")
+# Only clear matches are scored. Sad, fear, and disgust have no matching tag, so the
+# report shows what the model says for them without counting them as right or wrong.
+CREMA_D_LABELS = {
+    "ANG": "upset",
+    "HAP": "excited",
+    "NEU": "none",
+    "SAD": "sad",
+    "FEA": "fear",
+    "DIS": "disgust",
+}
+AUDIO_SUFFIXES = {".wav", ".mp3", ".flac", ".ogg"}
+
+
+def label_for(path: Path) -> str:
+    if match := CREMA_D.match(path.stem):
+        return CREMA_D_LABELS.get(match["emotion"], match["emotion"].lower())
+    prefix = path.stem.split("_")[0].lower()
+    return "none" if prefix == "neutral" else prefix
+
+
+def load_pcm16(path: Path) -> bytes:
+    """Read any soundfile format as 16 kHz mono PCM16, like the phone sends."""
+    audio, sample_rate = sf.read(path, dtype="float32", always_2d=True)
+    audio = audio.mean(axis=1)
+    if sample_rate != SAMPLE_RATE:
+        audio = resample_poly(audio, SAMPLE_RATE, sample_rate)
+    return (np.clip(audio, -1, 1) * 32767).astype("<i2").tobytes()
+
+
+def percent(part: int, whole: int) -> str:
+    return f"{100 * part / whole:5.1f}%" if whole else "    –"
+
+
+def report(rows: list[dict]) -> None:
+    scored = [row for row in rows if row["label"] in SCORED]
+    labels = sorted({row["label"] for row in rows}, key=lambda x: (x not in SCORED, x))
+    table = Counter((row["label"], row["tag"]) for row in rows)
+
+    print("\nRows: true label. Columns: tag the model returned.\n")
+    print(f"{'label':<10}" + "".join(f"{tag:>9}" for tag in SCORED) + f"{'clips':>8}")
+    for label in labels:
+        counts = [table[label, tag] for tag in SCORED]
+        marker = "" if label in SCORED else "  (not scored)"
+        print(f"{label:<10}" + "".join(f"{c:>9}" for c in counts) + f"{sum(counts):>8}{marker}")
+
+    if not scored:
+        print("\nNo clips with a scored label (excited, upset, calm, none).")
+        return
+
+    correct = sum(row["tag"] == row["label"] for row in scored)
+    wrong_tag = sum(row["tag"] != "none" and row["tag"] != row["label"] for row in scored)
+    tone_clips = [row for row in scored if row["label"] != "none"]
+    missed = sum(row["tag"] == "none" for row in tone_clips)
+    always_none = sum(row["label"] == "none" for row in scored)
+
+    print(f"\nScored clips: {len(scored)}")
+    print(f"  Accuracy              {percent(correct, len(scored))}  ({correct})")
+    print(f"  Wrong tag shown       {percent(wrong_tag, len(scored))}  ({wrong_tag})  <- worst")
+    print(f"  Tone missed (null)    {percent(missed, len(tone_clips))}  of clips with a tone")
+    print(f"  'Always null' gets    {percent(always_none, len(scored))}  (baseline to beat)")
+
+    print(f"\n{'tag':<10}{'precision':>11}{'recall':>9}   precision = when shown, how often right")
+    for tag in TAGS:
+        shown = sum(row["tag"] == tag for row in scored)
+        hits = sum(row["tag"] == tag == row["label"] for row in scored)
+        actual = sum(row["label"] == tag for row in scored)
+        print(f"{tag:<10}{percent(hits, shown):>11}{percent(hits, actual):>9}")
+
+    print(f"\n{'label':<10}{'arousal':>9}{'valence':>9}   mean scores, for threshold tuning")
+    by_label = defaultdict(list)
+    for row in rows:
+        if row["arousal"] is not None:
+            by_label[row["label"]].append((row["arousal"], row["valence"]))
+    for label in labels:
+        if values := by_label[label]:
+            arousal, valence = np.mean(values, axis=0)
+            print(f"{label:<10}{arousal:>9.3f}{valence:>9.3f}")
+
+
+def main(argv: list[str] | None = None) -> None:
+    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
+    parser.add_argument("folder", type=Path, help="Folder of labeled audio clips.")
+    parser.add_argument("--csv", type=Path, help="Also write per-clip scores and tags here.")
+    parser.add_argument("--limit", type=int, help="Only evaluate the first N clips.")
+    args = parser.parse_args(argv)
+
+    files = sorted(p for p in args.folder.iterdir() if p.suffix.lower() in AUDIO_SUFFIXES)
+    if args.limit:
+        files = files[: args.limit]
+    if not files:
+        parser.error(f"no audio files ({', '.join(sorted(AUDIO_SUFFIXES))}) in {args.folder}")
+
+    backend = AudeeringToneBackend()
+    print(f"Evaluating {len(files)} clips on {backend.device}...", file=sys.stderr)
+    started = time.perf_counter()
+    rows = []
+    for path in files:
+        arousal, valence = backend.scores(load_pcm16(path), SAMPLE_RATE)
+        tag = None if arousal is None else map_tag(arousal, valence, backend.thresholds)
+        rows.append(
+            {
+                "file": path.name,
+                "label": label_for(path),
+                "arousal": arousal,
+                "valence": valence,
+                "tag": tag or "none",
+            }
+        )
+    elapsed = time.perf_counter() - started
+    print(
+        f"{backend.model_version}: {len(rows)} clips in {elapsed:.1f} s "
+        f"({1000 * elapsed / len(rows):.0f} ms each)"
+    )
+    report(rows)
+
+    if args.csv:
+        with args.csv.open("w", newline="") as file:
+            writer = csv.DictWriter(file, fieldnames=list(rows[0]))
+            writer.writeheader()
+            writer.writerows(rows)
+        print(f"\nPer-clip results: {args.csv}")
+
+
+if __name__ == "__main__":
+    main()

scripts/fetch_crema_d.py

@@ -0,0 +1,72 @@
+"""Download a CREMA-D sample (ODbL, acted emotional speech) for tone evaluation.
+
+Clips go to the gitignored data/ folder; never commit them.
+"""
+
+import argparse
+import sys
+from concurrent.futures import ThreadPoolExecutor
+from pathlib import Path
+from urllib.error import HTTPError
+from urllib.request import urlopen
+
+BASE_URL = (
+    "https://media.githubusercontent.com/media/CheyneyComputerScience/CREMA-D/master/AudioWAV"
+)
+ACTORS = range(1001, 1092)
+# IEO is the only sentence recorded at several intensities; the rest are "XX".
+SENTENCES = ("DFA", "IOM", "ITH", "ITS", "IWL", "IWW", "MTI", "TAI", "TIE", "TSI", "WSI")
+EMOTIONS = ("ANG", "DIS", "FEA", "HAP", "NEU", "SAD")
+
+
+def fetch(name: str, out: Path, timeout: float) -> str:
+    path = out / name
+    if path.exists():
+        return "cached"
+    try:
+        with urlopen(f"{BASE_URL}/{name}", timeout=timeout) as response:
+            data = response.read()
+    except HTTPError as error:
+        if error.code == 404:
+            return "missing"  # A few actor/sentence/emotion combinations were never recorded.
+        raise
+    path.write_bytes(data)
+    return "downloaded"
+
+
+def main(argv: list[str] | None = None) -> None:
+    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
+    parser.add_argument("--out", type=Path, default=Path("data/crema-d"), help="Output folder.")
+    parser.add_argument(
+        "--actors", type=int, default=20, help="Actors to sample, spread across all 91."
+    )
+    parser.add_argument("--timeout", type=float, default=30, help="Per-file timeout (seconds).")
+    args = parser.parse_args(argv)
+    if not 1 <= args.actors <= len(ACTORS):
+        parser.error(f"--actors must be between 1 and {len(ACTORS)}")
+
+    # Spread the sample across actor IDs so it isn't one age/gender group.
+    step = len(ACTORS) / args.actors
+    actors = [ACTORS[int(i * step)] for i in range(args.actors)]
+    names = [
+        f"{actor}_{sentence}_{emotion}_XX.wav"
+        for actor in actors
+        for sentence in SENTENCES
+        for emotion in EMOTIONS
+    ]
+    args.out.mkdir(parents=True, exist_ok=True)
+
+    counts: dict[str, int] = {}
+    with ThreadPoolExecutor(max_workers=8) as pool:
+        for status in pool.map(lambda name: fetch(name, args.out, args.timeout), names):
+            counts[status] = counts.get(status, 0) + 1
+    print(
+        f"{len(actors)} actors, {len(names)} clips: "
+        + ", ".join(f"{count} {status}" for status, count in sorted(counts.items()))
+        + f" -> {args.out}",
+        file=sys.stderr,
+    )
+
+
+if __name__ == "__main__":
+    main()

server/README.md

@@ -34,9 +34,10 @@ and `schema_version: "1.1"`. Example sentence audio (PCM16 mono little-endian, b
 }
 ```
 
-Each request gets a `tone_result` with the same session and caption IDs, `tag: null`
-("no tone"), six zero probabilities, `model_version: "tone-stub-v1"`, and
-`latency_ms.recv_to_send`. The connection stays open for further requests.
+Each request gets a `tone_result` with the same session and caption IDs, a `tag`, six
+probabilities, `model_version`, and `latency_ms.recv_to_send`. With the default mock backend
+the tag is always `null` ("no tone"), the probabilities are zero, and the model version is
+`tone-stub-v1`; see "Real tone tags" below. The connection stays open for further requests.
 `caption_log` events are accepted and discarded; persistence is not implemented yet.
 Invalid messages receive a `session_error` reply and the connection stays open:
 
@@ -68,9 +69,33 @@ Query strings decode `+` as a space; encode a literal plus as `%2B`.
 Request contents, audio, authorization headers, and validation exception details are
 never logged or stored. Error replies echo payloads only to the client that sent them.
 
-`app/tone.py` defines `ToneBackend` and `MockToneBackend`. The server currently always
-uses the mock; Audio can plug in inference later. No audio is written to disk or logs.
-Torch and MediaPipe stay out of this package's dependencies.
+## Real tone tags (audEERING baseline)
+
+`TONE_BACKEND` picks the backend: `mock` (default) or `audeering`. One-time setup from the
+repo root downloads the model (~1.2 GB) into the Hugging Face cache, never into the repo:
+
+```bash
+uv run hf download audeering/wav2vec2-large-robust-12-ft-emotion-msp-dim
+```
+
+Then start the server with the real model:
+
+```bash
+TONE_BACKEND=audeering uv run --directory server python -m app
+```
+
+Tags are `excited`, `upset`, `calm`, or `null`, from the arousal/valence thresholds tuned in
+A4 (`AudeeringThresholds` in `app/tone.py`). Segments under 0.5 s always get `null`, and
+`probs` stay zero because the model has no 6-emotion output. `model_version` is
+`audeering-msp-dim-a4`. The model runs on `cuda`, `mps`, or `cpu`, whichever is available,
+one segment at a time, off the event loop. On an M-series Mac a 3 s segment takes ~40 ms
+after the first request (~0.5 s warm-up). Check it with the P3 replay client
+(`scripts/README.md`). If the model isn't downloaded, startup fails with the command above.
+
+`app/tone.py` defines `ToneBackend`, `MockToneBackend`, the tag rules, and `create_backend`.
+`app/audeering.py` holds the model and is imported only when selected. No audio is written
+to disk or logs. Torch, transformers, and scipy come from the uv workspace (`graption-ml`);
+they are not dependencies of this package, and MediaPipe stays out entirely.
 
 Export `TONE_TOKEN` to require `Authorization: Bearer <token>` on the handshake.
 Export `DEPLOY_MODE=tunnel` or `cloud` to require a configured token as well.
@@ -123,7 +148,8 @@ uv run ruff check .
 uv run pytest
 ```
 
-Tests cover health, repeated audio replies, the schema contract, caption logs,
+`RUN_MODEL_TESTS=1 uv run pytest server/tests/test_tone.py` also runs the downloaded model.
+Tests cover tag thresholds, backend selection, health, repeated audio replies, the schema contract, caption logs,
 recoverable errors with exact request echoes, and token checks. `app/events.py` is generated; never hand-edit it.
 Regenerate it from the repo root with:
 
@@ -136,5 +162,5 @@ uv run datamodel-codegen \
   --formatters ruff-check ruff-format
 ```
 
-Later: tag mapping, SQLite logging, and Docker packaging. Add server dependencies with
+Later: the custom WavLM backend, SQLite logging, and Docker packaging. Add server dependencies with
 `uv add --package graption-server <pkg>`; tests live in `server/tests/`.