Let admins roll up low-confidence predictions to a coarser rank#1361
Draft
mihow wants to merge 6 commits into
Draft
Let admins roll up low-confidence predictions to a coarser rank#1361mihow wants to merge 6 commits into
mihow wants to merge 6 commits into
Conversation
…dmin framework Rework of #999 onto the #1289 post-processing framework. The branch was cut from an old main with its own hand-rolled admin action; ClassMaskingTask and RankRollupTask now subclass BasePostProcessingTask with pydantic config schemas and are triggered through make_post_processing_action (collection scope on SourceImageCollection, single-occurrence scope on Occurrence for class masking). Correctness fixes from the review threads: - Class masking selects the top class from an -inf-masked softmax, so a class excluded by the taxa list can never win even when it had the highest logit; raises when the taxa list excludes every class in the category map. Stored logits stay raw (JSON-safe) and the mask is captured in scores (excluded -> 0). - The masked-output Algorithm is one per (source algorithm, taxa list) and its category map is persisted (previously set in memory only, so masked classifications referenced a null map). - applied_to is populated on new masked classifications (the provenance the API exposes was left blank). - Rank rollup preloads category-map labels in two queries and select_relates the per-row relations instead of dereferencing category_map per classification. Surfaces provenance in the API: applied_to is added to the Classification serializers, and applied_to__algorithm is prefetched in the occurrence list/detail prefetch and the classification viewset to avoid an N+1 on render. Tests: pydantic config validation, admin trigger for both scopes, the masking maths (including the excluded-class guarantee and the all-excluded error), ClassMaskingTask.run() end to end for both scopes, and rank rollup. 20 new tests; full post_processing suite and occurrence query-count tests pass. Co-Authored-By: Claude <noreply@anthropic.com>
The roll-up picks the global argmax over every taxon at each rank, not just ancestors of the source classification's taxon. On a diffuse, low-confidence distribution this can reparent a detection to a family outside its own lineage. Document the behavior at the candidate-pick loop and record the open design choice (lineage-constrained vs distribution roll-up) as a TODO; no behavior change. Co-Authored-By: Claude <noreply@anthropic.com>
…ection When an operator triggers class masking on an occurrence (or collection), the "Source classifier" dropdown now lists only the classification algorithms that actually produced classifications within the selected scope. Masking any other algorithm would be a no-op for those rows, so offering every classifier was misleading. The admin action hands the knob form the selected queryset (a small generic seam on the form base, ignored by forms that don't need it), and ClassMaskingActionForm uses it to filter the algorithm field by classifications__detection__occurrence / source_image__collections. Co-Authored-By: Claude <noreply@anthropic.com>
…llections The class-mask admin form narrows its "Source classifier" dropdown to the algorithms that actually produced classifications in the selected scope, so an operator cannot pick a classifier whose masking would be a no-op. On the occurrence path that lookup is cheap (a handful of classifications), but on the collection path it became an unbounded DISTINCT over every classification in the collection. On a large collection that join runs for tens of seconds or times out while the intermediate form is still rendering, before the operator can do anything. Scope the dropdown only for the occurrence path. A collection scope now keeps the full classifier list. Offering a classifier that produced nothing in the collection is harmless — masking it changes no rows — so the narrowing was a convenience, not a correctness guard, and is not worth a query that can hang the page. Co-Authored-By: Claude <noreply@anthropic.com>
Rank roll-up moves out of this PR so class masking can merge on its own. Class masking is validated end to end against a real classifier and is well covered by tests; rank roll-up still has an open design question (whether the per-rank pick should be constrained to the source taxon's ancestors or remain a global distribution roll-up) and thinner tests. Keeping them together would force reviewers to either block ready class-masking work or sign off on the less-settled roll-up. This removes the rank-roll-up task, its admin form and action, its registry entry, and its tests. The roll-up feature returns in its own PR, stacked on this one, where the lineage decision and fuller tests can land before it merges. Co-Authored-By: Claude <noreply@anthropic.com>
Add the rank roll-up post-processing task, split out of the class-masking PR so it can carry its own design discussion and tests. For each detection it walks the classification distribution from the finest rank upward and promotes the prediction to the first rank whose summed probability clears that rank's threshold, writing a new terminal classification at that rank and linking it back to the source via applied_to. An admin can trigger it on a capture set from the Django admin, choosing per-rank thresholds. Open design question, documented at the candidate-pick loop in rank_rollup.py: the per-rank winner is currently the global argmax across the whole distribution, not only ancestors of the source taxon, so a diffuse low- confidence distribution can reparent a detection to a family outside its own lineage. Whether to constrain candidates to the source taxon's ancestors (generalise only) or keep the global distribution roll-up (and revisit the applied_to provenance when the result is off-lineage) is left for review. Tests cover the config schema contract, the admin trigger, and a species->genus roll-up end to end. Co-Authored-By: Claude <noreply@anthropic.com>
✅ Deploy Preview for antenna-preview canceled.
|
Contributor
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This adds rank roll-up, a post-processing task an admin can run to handle detections the classifier could not confidently place at species level. For each detection it walks the classification distribution from the finest rank upward and promotes the prediction to the first rank whose summed probability clears that rank's threshold, writing a new terminal classification at that rank and linking it back to the source via
applied_to. An admin triggers it on a capture set from the Django admin and can set per-rank thresholds.It runs through the post-processing framework that merged in #1289, the same way as the small-size filter and class masking.
This was split out of the class-masking PR (#999) so each can carry its own review. It is stacked on #999 — the diff is against that branch and shows only the roll-up additions; once #999 merges this rebases onto
main. Opening as a draft until the design question below is settled.Open design question (needs a decision before merge)
The per-rank winner is currently the global argmax across the whole distribution, not constrained to ancestors of the source classification's taxon. On a confident classification the winner is its own lineage, but a diffuse, low-confidence distribution can spread enough mass across unrelated branches that the top taxon at a rank is not an ancestor of the source — so the roll-up reparents the detection to a family outside its own lineage. The choice, documented at the candidate-pick loop in
rank_rollup.py:applied_topointing at the single source classification is the right provenance when the result is off-lineage.A follow-up worth pairing with this: #1360 (offline eval of the post-processing filters) would quantify how each option moves final predictions against ground truth.
List of Changes
RankRollupTask(BasePostProcessingTask)+make_post_processing_actionwiring onSourceImageCollectionAdmin, with per-rank thresholds on the action formapplied_to; the source is demoted to non-terminalDetailed Description
applied_toalready exists onClassification, and the task runs under the existingpost_processingjob type, so the registry addition does not touch any field definition.select_related, rather than dereferencing the category map per classification.How to test
SourceImageCollection) → select one with species-level classifications → choose the Rank Rollup action → set per-rank thresholds → run.applied_to.Related