Skip to content

Latest commit

 

History

History
171 lines (141 loc) · 11.3 KB

VERIFICATION.md

File metadata and controls

171 lines (141 loc) · 11.3 KB

Verification and known gaps

Verification update: 2026-09-27. Runtime code changed: the dashboard can delete what the persistent brain stored. With the daemon stopped, Delete what the brain stored lists every file through GET /daemon/erase on the control server and, after a confirmation, deletes exactly that list (POST, carrying a fingerprint of the file names). server.braind erase uses the same code, now in server/erase.py, so the always-on control server does not load torch (39 MB instead of about 0.5 GB). Logs still held open are emptied before they are unlinked. Linux container; real macOS sensors and permissions were not exercised.

Check (Linux, Python 3.11.15, PyTorch 2.14.0, pytest 9.1.1, Chromium 141) Result
Full Python suite, unfiltered 771 passed, 1 skipped
Observatory, live/capture/daemon UI contracts and bilingual tour 51 passed
3D structural checks PASS
Deleting from the dashboard in headless Chromium, mock-sensor daemon behind the control server While the daemon ran there was no list and no deletion. Stopped, the list showed the five stored files (1.1 MB); Delete stayed disabled until confirmed. After Delete 5 files the files were gone, a second look reported nothing stored, and the live view had dropped its retained window. The next start was fresh: nothing shared, consent revision 0, no words. A page on another local origin could neither read the list nor delete. No page errors
Deletion block at 320 px; tour steps 1–7 in English and German at 1360, 390 and 320 px No horizontal overflow, 44 px controls; tour targets visible, no request other than GET

Deliberately breaking the confirmation guard, the check against the reviewed list, the running check, the log emptying or the torch-free import makes one of the new tests fail.

Verification update: 2026-09-26. Runtime code changed: Observatory is now the one UI for both runtimes. Live session shows either the persistent daemon (new GET /api/telemetry on port 8000) or the session runner, and controls the daemon's five sources, its start and stop and its learned words. The control server serves the dashboard on 8900; the old training console is removed. Stop answers once the daemon has exited, a start keeps the sensor mode, and POST /api/feel refuses blank words and words over 64 characters, which the dashboard could not list. Also fixed: since f870ac4 every start without --mock-sensors crashed on a deleted permission check; a real-mode start is now tested. This run used a Linux container: real macOS sensors, permissions and a real microphone were not exercised.

Check (Linux, Python 3.11.15, PyTorch 2.14.0, snnTorch 1.0.0, MCP 2.2.0, pytest 9.1.1, Chromium 141) Result
Full Python suite, unfiltered 765 passed, 1 skipped
Observatory, live/capture/daemon UI contracts and bilingual tour 48 passed
3D structural checks PASS
Dashboard from the control server in headless Chromium, mock-sensor daemon First load: persistent brain chosen, daemon running, paused with nothing shared, no current named state, naming locked, runner controls hidden. Idle time, apps and microphone shared one at a time, each confirmed by the daemon; frames arrived as test inputs with one 3D canvas; keyboard and pointer showed as disabled, the app as a category, never a name. A taught word named the moment and was recognized. Session runner hid the daemon's controls and reported the runner as unavailable. Arrow keys moved the source choice only on its two buttons, with focus; the timeline slider kept its keys. Stop answered after about 1 s (0.92–1.01 s in three runs), once the daemon had exited; an immediate start came back in mock mode with the saved three sources and the word, without a lock conflict. The preview on 4178 showed the same state; a page on another local origin got no answer to reading or changing consent, stopping the daemon or reading telemetry, and afterwards the daemon ran on with the same pid and choice. No page errors; no traceback, lock refusal or respawn in the control log
Tour steps 1–7 without a daemon: English and German, 1360, 390 and 320 px wide Every target present and on screen, no horizontal overflow, no page errors, no request other than GET

Verification update: 2026-09-22. Runtime code changed: the daemon and control server refuse other web origins, the daemon has per-source consent, experience.db is pruned after 90 days and server.braind erase exists. This run used a Linux container, not macOS: real sensors, OS permissions, the launch agent and a real microphone were not exercised.

Check (Linux, Python 3.11.15, PyTorch 2.14.0, snnTorch 1.0.0, MCP 2.2.0, pytest 9.1.1) Result
Full Python suite, unfiltered 752 passed, 1 skipped
Observatory, live/capture UI contracts and bilingual tour 37 passed
3D structural checks PASS
Training console in headless Chromium against a mock-sensor daemon First start paused with 0 steps and live labeling locked; sharing idle time started steps; microphone switch on and off; a stop that could not be saved stayed off and was reported; a page on another local origin was blocked for POST, GET and WebSocket; a restart restored exactly the saved choice
server.braind erase against a mock-sensor daemon on port 8012 Refused while running although erase checks port 8000 by default; a second daemon on the same checkpoint did not start; dry run listed 6 files; --yes deleted them including the backup; the next start was fresh and shared nothing

The new consent, origin and missing-data tests fail against the previous code and pass now. An independent review of the change found edge cases that are now fixed and tested the same way: a keyboard listener without Input Monitoring counted as live, mock-mode consent carried over to real capture, a paused brain asked at once after resuming, erase missed a daemon on another port, and backup matching by prefix reached another checkpoint's backups. Real macOS behaviour of the listener check is inferred from pynput's source, not observed. The 2026-09-17 record below is unchanged.

Verification update: 2026-09-17. Runtime code was not changed; the control test now exercises child reaping without the unavailable macOS os.waitid API. These checks validate selected contracts, not product readiness or scientific claims.

Current results

Check Result
Python suite in clean PyTorch-2.13 environment 682 passed, 1 skipped, 3 native sensor tests deselected
Observatory, live/capture UI contracts and bilingual tour 37 passed
3D structural checks PASS: 1,000 units, shared Circuit palette, atlas placement and asset consistency
macOS control tests 11 passed after portable test repair
Fresh installation satisfying current dependency constraints Installed with PyTorch 2.13.0; uv pip check passed
Real sensor, wearable, cloud model, voice recording or personal checkpoint run Not performed
Longitudinal study or real speech/music accuracy evaluation Not performed

The earlier 2026-09-15 scoped run (85 passed, 1 skipped) used PyTorch 2.11.0, snnTorch 0.9.4, MCP 2.2.0 and pytest 9.0.3. That existing .venv does not meet the current torch>=2.13 requirement and was left untouched.

On 2026-09-17 a separate temporary environment was created with Python 3.11.13, PyTorch 2.13.0, snnTorch 1.0.0, MCP 2.2.0, pytest 9.1.1 and Anthropic SDK 1.6.0. All 90 installed distributions passed uv pip check. The install used the declared project dependencies with an explicit torch==2.13.0 constraint to exercise the minimum, not an in-place downgrade. Package source: official PyTorch distribution on PyPI. This verifies this macOS/Python combination, not every supported platform. The full selected Python run took 77.02 seconds and emitted one upstream Starlette/AnyIO BlockingPortal deprecation warning. There were no test failures. The 37 JavaScript tests were rerun in the existing environment; their Python telemetry subprocess still uses the repository's older .venv.

Reproduce the isolated installation and selected full suite:

brain_check_dir=$(mktemp -d)
uv venv --python 3.11 "$brain_check_dir/venv"
uv pip install --python "$brain_check_dir/venv/bin/python" -e ".[dev]" "torch==2.13.0"
uv pip check --python "$brain_check_dir/venv/bin/python"
"$brain_check_dir/venv/bin/python" -m pytest -q --override-ini addopts= \
  --deselect=tests/test_sensor_keymouse.py::test_keystroke_construction \
  --deselect=tests/test_sensor_keymouse.py::test_mouse_construction \
  --deselect=tests/test_sensor_mic.py::test_construction

Reproduce the scoped checks

From the repository root, in a prepared environment:

.venv/bin/python -m pytest -q --override-ini addopts= \
  tests/test_capture_controls.py tests/test_observation_telemetry.py \
  tests/test_audio_vad.py tests/test_brain_tools.py tests/test_tools_endpoint.py \
  tests/test_mcp_server.py tests/test_experience.py tests/test_episode_log.py \
  tests/test_daemon_telemetry.py tests/test_control.py tests/test_daemon_origin.py

node --test docs/dashboard-concepts/verify-onboarding.mjs \
  docs/dashboard-concepts/verify-observatory.mjs \
  docs/dashboard-concepts/verify-live.mjs \
  docs/dashboard-concepts/verify-capture.mjs \
  docs/dashboard-concepts/verify-daemon.mjs

node 3d/verify-brain.mjs
node scripts/check-docs.mjs
git diff --check

The optional real ONNX test is skipped without an explicitly supplied trusted model and fingerprint. It must not download or record audio implicitly. See the audio experiment.

The tour's separate 2026-09-15 browser check covered all 18 steps in both languages, reload persistence and narrow layouts. No new visual redesign was performed in this documentation pass. Historical visual checks are recorded in EINFUEHRUNG.md; they do not certify every browser, screen reader or hardware GPU.

Tests requiring care

Do not run the full unfiltered suite or scripts/run_tests.sh as a promise of sensor-free execution. In particular, these constructors can touch native APIs:

  • tests/test_sensor_keymouse.py::test_keystroke_construction
  • tests/test_sensor_keymouse.py::test_mouse_construction

Since 2026-09-22 MicSensor opens the microphone only when its source is shared, so tests/test_sensor_mic.py::test_construction no longer opens an audio stream. Deselecting it, as the 2026-09-17 command above does, is harmless.

tests/test_control.py::test_running_pid_reaps_a_crashed_child now lets the actual _running_pid() probe reap its own exited test child, with a bounded deadline and cleanup on failure. It still asserts that the child was reaped; the test is not skipped or replaced with a mock. Production supervision logic and running services were not changed.

Scientific evidence boundary

benchmark/run_all.py currently runs four synthetic diagnostic groups: stability, replay, discrimination and modulator reactivity. Its success text is not a scientific validation claim. The README no longer describes a fifth sleep-retention benchmark as part of that runner.

Do not present historical scores, target thresholds or a successful test harness as real-world focus/stress detection. Before reporting benefit, freeze an evaluation protocol with baselines, held-out sessions, provenance and failures.