AI tutor, macaque templates, reliability calibration + free open-source tutor default - #253
Open
ataha24 wants to merge 21 commits into
Open
AI tutor, macaque templates, reliability calibration + free open-source tutor default#253ataha24 wants to merge 21 commits into
ataha24 wants to merge 21 commits into
Conversation
Contributor
|
Going to quickly close and re-open the PR so that it can reflect the updated python versions in the CI tests. |
UI/Visual: - Complete CSS redesign with Space Mono font, scanline overlay, glow animations, and consistent dark/green design system - Responsive hero page with live template stats, how-it-works workflow, feature grid, and template browser - Validator page redesigned: drag-and-drop file zone, styled selects, animated submit button, dev-mode login notice - Larger AFIDs banner (clamp 360–620px) Validation features: - Regional radar chart (error profile across 8 neuroanatomical regions) - Directional error language per landmark (e.g. "2.1mm posterior") - Session summary dashboard: mean±SD, best/worst landmark, within-2mm count - Color-coded distance table (green/amber/red by threshold) Macaca support: - Add afids-templates/macaca/ with 6 mean consensus FCSV templates: D99, INIA19, MacaqueMNI, NMTv1.3, NMTv2.0asym, Yerkes19 - Templates sourced from afids/afids-macaca PHASE1_output_afid - Species dropdown now includes Macaca alongside Human
- New /learn page with in-browser MNI152NLin2009cAsym T1w MRI viewer (NiiVue WebGL) and real-time AI tutor powered by an OpenAI-compatible LLM (configurable via LLM_API_KEY, LLM_BASE_URL, LLM_MODEL env vars) - Flask Blueprint (learn.py) with routes for NIfTI serving, reference coords, placement checking, and streaming LLM intro/feedback/chat - Model-agnostic LLM wrapper (llm.py) with full AFIDs protocol grounded system prompt covering all 32 landmark definitions - Placement feedback includes placed coords, reference coords, directional offset, and viewer state (zoom, resolution, contrast); LLM recommends zoom/resolution adjustments when error is large - Celebration overlay on excellent placement (< 1 mm); auto-advances to next landmark - Next Landmark button auto-captures current crosshair if no fiducial placed; AbortController cancels in-progress intro when user places early - Crosshair turns red on placement, resets to green after feedback streams - Session export: downloads a formatted HTML report with placement summary table and full chat transcript - Home page: added AI Guided Learning callout section and feature card - Navbar: renamed "Learn" tab to "AI Tutor" (rebuilt React bundle) - openai dependency added to pyproject.toml
…alibration - Per-request LLM overrides (api_key/base_url/model) held client-side, with a graceful static fallback so the tutor never dead-ends; /learn/llm-status. - Retrieval-augmented landmark context (rag.py) + knowledge_chunks migration. - Rater-reliability prior from the AFIDs multi-rater release: per-landmark AFLE percentile/band injected into feedback and returned by /learn/check. - Tutor prompt constrained to real viewer controls, gentler tone, and plain-language glossing of anatomical terms.
…e education manuscript - analyze_afids_templates.py / make_figures.py regenerate the six paper figures and the MNI inter-template variability from the released templates. - compute_reliability.py output (rater_reliability.json) drives Figure 6. - AFIDs_Validator_Aperture_Education.md: Methods & Resource Paper draft for the Aperture Neuro education special issue.
…l abstract - Figure 1 rebuilt from a real /learn screenshot with annotated callouts. - Figure 3 rebuilt as a nilearn glass-brain render. - New graphical abstract (fig0) tying the tool, rater data, and calibration. - capture_screenshots.sh regenerates the screenshot assets via agent-browser. - Figure 1/3 and graphical-abstract legends updated to match.
Rework the quantitative story of the education manuscript and its figures so they serve the paper's two contributions instead of describing template statistics. Paper (AFIDs_Validator_Aperture_Education_AT_FINAL.md): - Restructure §5 into a landmark-difficulty benchmark (§5.1–5.2), a localize-vs-reproduce comparison (§5.3), and a worked quality-control catch (§5.4); demote inter-template variance and AC–PC to a correctness caveat. - Rewrite the abstract's quantitative sentence; merge all 21 templates into a single Table 1; renumber Methods subsections to 9.x and repair every section cross-reference; cite VanLehn (2011); fix the afids-utils reference. Figures (make_figures.py): - fig4_qc_catch: real MNI305-vs-default template-space mismatch (replaces the synthetic validation report). - fig5_difficulty: trained-rater difficulty spectrum, "same 1.2 mm two verdicts", and the mm→percentile calibration (replaces the variance figure). - fig6_two_difficulties: localize-vs-reproduce scatter plus demoted per-landmark variability. Tooling and cleanup: - make_paper_doc.py renders the .docx directly from the Markdown source and embeds the figure set, so the Word output never drifts. - Add the graphical-abstract generators (make_ga_assets.py, make_graphical_abstract.py) and their assets; remove superseded figures and the pre-merge draft; ignore the generated .docx.
- Apply objective proofreading fixes (subject-verb agreement, broken parallelism, awkward constructions) across the abstract, §1, §4, §5, and §8. - Reconcile Table 1 to the full 15 human templates by adding the MNI2009cAsym alias row (15 human + 6 macaque = 21). - Fix two internal inconsistencies: the 2 mm quality boundary now reads as the ~82nd percentile (matching the 82%-within-2mm statistic), and the inaccurate "runs client-side" claim is corrected to "runs in any modern browser". - Standardize the product name to "AFIDs-Validator" throughout. - Add an Ethics statement (secondary use of the Taha et al. 2023 release; Western HSREB REB# 109045 and R-17-156) and a CRediT author-contributions scaffold grounded in that paper and the repository history.
- Rebuild the affiliation block: remove the erroneous Stanford tags that had been applied to seven authors (only A. Taha is at Stanford), renumber to seven affiliations reconciled against the group's recent papers, and place D. Bansal and J. Thrower at Robarts. - Finalize CRediT author contributions: add D. Bansal, A. Thurairajah, and J. Thrower under Software (platform development and testing), Investigation, and Validation; drop the draft scaffolding note.
…ler cites
- Replace the Fig 3 glass-brain scatter with a visual field guide: 32 real
MNI152 T1w patches, one per landmark (crosshair on the point, region-coloured
border, median-AFLE chip, viewing-plane tag); update its legend accordingly.
- Remove the baked-in title from all six figures — the figure caption carries
it, per journal convention.
- Simplify the in-figure citations in the graphical abstract to author-year
(Taha et al., 2023; Lau et al., 2019).
- Fix a text overflow in Fig 2 (a design-principle label ran past its band) and
a wrong ordinal in Fig 5 ("42th" -> "42nd").
- The whole set regenerates deterministically from `python make_figures.py`.
Wire LLM_API_KEY/BASE_URL/MODEL into the production .env written by the deploy workflow, defaulting to Groq's free Llama 3.3 70B endpoint. The key is the only secret (PRODUCTION_LLM_API_KEY); URL and model fall back to Groq but stay overridable via optional secrets. LLM vars are only written when a key is present, so a keyless deploy keeps the tutor on its graceful static-reference fallback. Document Groq as the recommended zero-cost default in .env.template and warn against running ingest-knowledge against Groq (no embeddings API).
The knowledge_chunks migration branched from 7b4e00130929, but the live migration head is 56d89145adbb (the user-table change already on master). That left two Alembic heads, which makes 'flask db upgrade' ambiguous and fails the deploy's db-upgrade step. Re-point down_revision onto 56d89145adbb so the chain is linear and upgrades cleanly. The migration is purely additive (a new knowledge_chunks table), so re-parenting is safe.
pyproject declared openai ^1.0 but the lock was never regenerated, so
'poetry install' failed ("pyproject.toml changed significantly since
poetry.lock was last generated") — breaking CI and the deploy build.
Regenerate with poetry lock --no-update to add openai and its transitive
deps (httpx, pydantic, anyio, distro, h11, jiter, tqdm, ...) without
changing existing pins.
The CI matrix tested Python 3.8, which is EOL — modern virtualenv no longer ships a 3.8 seeder, so the job failed on every run regardless of the change under test. Move the matrix to 3.9/3.10/3.11 (all inside pyproject's >=3.8,<3.12 range). The deploy job had no Python setup and relied on the runner default, which is now 3.12 (outside the supported range, rejected by poetry). Add an explicit setup-python@v5 pinned to 3.11 so the release venv resolves.
Make Groq (llama-3.3-70b-versatile) the built-in default whenever an LLM key is present, so only LLM_API_KEY needs to be configured — the base URL and model now default to Groq automatically. Align the BYOK settings UI (default provider, placeholders, prefilled fields) and .env.template, fix the active-model badge that mislabeled Groq deployments, document the /learn tutor and deployment in the README, and add a deploy handoff runbook.
The percentile shown for a placement was the rank of the learner's error within the trained-rater error distribution, so a loose placement scored a high number and read as if 90th percentile meant a great placement. Invert it into a precision percentile (percent of trained raters the learner is at least as precise as), so higher is better, and update the badge, results table, and tutor feedback wording to match.
Learn mode previously only moved forward (Place / Show reference / Next), so a user could not return to an earlier landmark to review or re-place it. Add a Back control that revisits the previous landmark and lets the user place it again; it is disabled on the first landmark and on the completion screen. Re-placing now replaces the prior attempt for that landmark so the session report keeps one row per landmark.
Localization error is right-skewed and bounded at zero, so the mean overstates typical performance and the SD is inflated by outliers. Report the median and interquartile range alongside the existing mean/SD; the median is robust to the skew and is directly comparable to the percentile-based trained-rater reference distribution.
Dhananjhay
force-pushed
the
enh/ai-tutor-learning-mode
branch
from
July 23, 2026 17:10
74dbd98 to
dd1b18f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Brings the guided-learning AI tutor, macaque template support, rater-reliability calibration, and the accompanying reference-data analysis to
master, and makes the deployed tutor work out of the box on a free, open-source model with no paid API key.This is the accumulated work from
enh/ai-tutor-learning-mode(which also contains the visual-overhaul / macaque commits from #252).What's included
Guided learning tutor (
/learn)llm.py).{api_key, base_url, model}from an in-page Settings panel; the key stays in their browser (localStorage), sent per-request, never persisted or logged.Free open-source default (this is the deploy-facing change)
.envwritten bydeploy.ymlnow includesLLM_API_KEY/BASE_URL/MODEL, defaulting to Groq's free Llama 3.3 70B endpoint.PRODUCTION_LLM_API_KEY); URL/model default to Groq and stay overridable via optional secrets..env.templatedocuments Groq as the recommended zero-cost default, with a note that Groq has no embeddings endpoint (don't runingest-knowledgeagainst it).Rater-reliability calibration
rater_reliability.json) from the AFIDs multi-rater dataset;/learn/checkreports where a placement sits in the trained-rater distribution.Macaque templates
Reference-data analysis + manuscript
Deploy notes
PRODUCTION_LLM_API_KEYrepo secret (free Groq key) for the live tutor.workflow_dispatch).