Skip to content

10 V3 give each face an id guess the speaker - #77

Merged
ajaymuru26 merged 2 commits into
mainfrom
10-v3-give-each-face-an-id-guess-the-speaker
Oct 28, 2026
Merged

ajaymuru26 merged 2 commits into
mainfrom
10-v3-give-each-face-an-id-guess-the-speaker

Conversation

@AadarshJena

Copy link
Copy Markdown
Collaborator

Closes #10

WHAT CHANGED

  • Moved face tracking out of CameraView.swift into a new local package, apps/ios/Packages/GraptionVision, so it can be unit tested without a camera
  • Face IDs are now strings f1, f2, ... (matches FaceId in the event schema) and are never reused within a session
  • Matching follows section 6 of GRAPTION_CONTEXT.md: IoU of at least 0.3 first, then a center-distance fallback for fast movers (V2 used center distance only)
  • A face that disappears keeps its ID for 2 seconds (V2 dropped it after 2 frames), so turning your head away and back keeps the same ID. A missed face is still only drawn for 2 frames so boxes don't linger
  • Added a SpeakerScorer protocol and a LargestFaceScorer stub (biggest face = speaker). The Core ML speaker model will replace it behind the same protocol
  • Camera overlay: each box shows its ID. The guessed speaker reads "fX talking" in green, others are yellow
  • Rotating the phone or flipping the camera calls the new forgetTracks() (faces get new IDs, numbering keeps counting so an old ID can't land on someone else). Leaving the camera screen calls reset() (IDs restart at f1)
  • One Euro smoothing and coasting logic moved unchanged from V2
  • Linked the package in project.pbxproj. iOS CI now runs swift test on it
  • Updated GRAPTION_CONTEXT.md section 6 (FaceTracker: 2 second memory and why, speaker guess)
  • Works with P7 (match caption to speaker + tests #73) as is: captionSpeaker(from:during:) takes the f1 IDs and per-frame scores V3 produces

FUNCTIONALITY

  • two faces get f1/f2, biggest face labelled "talking"
  • labels stay with people walking past each other
  • turning away until the box disappears, then back within 2 seconds, keeps the ID
  • stepping closer moves "talking" to that person
  • eaving for longer than 2 seconds gives a new ID
  • rotating the phone works, IDs start fresh
  • leaving and reopening the camera restarts at f1

TESTING

  • swift test --package-path apps/ios/Packages/GraptionVision: 17 tests pass (moving faces, reordered detections, fast movers, 2 second dropout keeps ID, retirement after 2 seconds, no ID reuse, reset, forgetTracks, speaker guess)
  • The 2 second dropout test fails with the old 1 second value, so it covers the bug found on device
  • swift test on GraptionCore (P7): 7 tests pass
  • App builds for iOS Simulator with CI flags

Video: (drag the screen recording here, 10 MB max)

NOTES FOR REVIEWERS

  • Tradeoff: someone new stepping into the same spot within 2 seconds inherits the previous person's ID
  • Tracker tuning (IoU 0.3, 1.5x face size) is from the spec
  • swift test can't run on macOS for GraptionAudio and GraptionUI (iOS-only platform errors). This PR doesn't touch them
  • Unrelated leftovers on main for a follow-up: committed face_landmarker.task weights, "Python on PACE" in section 3, stray root Package.swift and Tests/ (breaks the VSCode Swift extension)

AadarshJena and others added 2 commits October 27, 2026 18:24
…t-face speaker guess

- Move tracking (OneEuroFilter, TrackedFace, matching) out of CameraView into a new
  GraptionVision package with unit tests.
- Match by IoU >= 0.3 first, then center distance; keep a track 1 s unseen, show it
  for 2 missed frames; IDs are f1, f2, ... per session (schema FaceId).
- Add SpeakerScorer protocol with LargestFaceScorer stub; camera view labels boxes
  with their ID and highlights the guessed speaker.
- Link the package in the Xcode project, run its tests in iOS CI, update §6.

Closes #10

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On-device test: turning the head away long enough to lose detection gave
the face a new ID on return, and the green-only speaker highlight was hard
to read.

- FaceTracker keeps an unseen face's ID for 2 s (was 1 s)
- Speaker box label reads "fX talking"
- Tests for a 2 s dropout keeping its ID and retirement after 2 s
- GRAPTION_CONTEXT.md §6 updated with the new value and tradeoff

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ajaymuru26
ajaymuru26 merged commit 6316d29 into main Oct 28, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

V3. Give each face an ID + guess the speaker

2 participants