Conversation
…ement counting logic with unit tests
|
Thanks for the pull request, @tbain! This repository is currently maintained by Once you've gone through the following steps feel free to tag them in a comment and let them know that your changes are ready for engineering review. 🔘 Get product approvalIf you haven't already, check this list to see if your contribution needs to go through the product review process.
🔘 Provide contextTo help your reviewers and other members of the community understand the purpose and larger context of your changes, feel free to add as much of the following information to the PR description as you can:
🔘 Get a green buildIf one or more checks are failing, continue working on your changes until this is no longer the case and your build turns green. DetailsWhere can I find more information?If you'd like to get more details on all aspects of the review process for open source pull requests (OSPRs), check out the following resources: When can I expect my changes to be merged?Our goal is to get community contributions seen and reviewed as efficiently as possible. However, the amount of time that it takes to review and merge a PR can vary significantly based on factors such as:
💡 As a result it may take up to several weeks or months to complete a review and merge your PR. |
openedx-webhooks
added
the
open-source-contribution
jesperhodge
left a comment
jesperhodge
left a comment
There was a problem hiding this comment.
There seem to be changes missing. For example, src/taxonomy/data/api.ts.
Could you
- review this PR and make sure that all necessary changes are in this branch? Compare to the open Unicon PR.
- review discussions in the Unicon PR and either resolve them or copy them here to be addressed here.
- fix any pipeline errors
?
|
Since we're no longer using recursive SQL for this, is it possible to update the PR description for accuracy? |
|
…bain/253_add_tags_count_rebased # Conflicts: # src/openedx_tagging/models/base.py # tests/openedx_tagging/test_api.py
There was a problem hiding this comment.
Pull request overview
Adds rolled-up, de-duplicated tag usage counts (including ancestor rollups) to the tag listing query so the Taxonomies UI can display accurate “Usage Count” values per tag.
Changes:
- Replaced the prior per-tag direct usage counting subquery with a dynamic, depth-aware subquery that rolls counts up to ancestors with per-object de-duplication.
- Updated existing API/model tests to reflect rolled-up counts and added a broader set of usage-count test cases.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 9 comments.
| File | Description |
|---|---|
src/openedx_tagging/models/base.py |
Centralizes and updates include_counts behavior by annotating tag querysets with rolled-up, de-duplicated usage_count via a subquery. |
tests/openedx_tagging/test_models.py |
Updates expected usage counts and adds multiple new test scenarios validating ancestor rollup and sibling de-duplication. |
tests/openedx_tagging/test_api.py |
Updates autocomplete/search test expectations to reflect rolled-up usage counts returned by the API when include_counts=True. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
Feel free to ping me for review here once the AC are clarified and the comments from Copilot etc are addressed. |
@bradenmacdonald I think this is ready for re-review, I resolved all the Copilot issues and added the improvement you suggested for finding the depth via a query rather than depending on the constant |
I think I like option 2 better as well because I think it's clearer that it will help with the performance and likely take less implementation time. However, unfortunately, I think even if option 2 is only about a 2ish day effort to implement, the number of fast follows we've been promising are starting to stack up, so we're getting a bit more concerned about our timeline. I'd like to add a new Github Issue for our "Nice to Haves" to address the performance concerns here and proceed with merging as is, if possible @bradenmacdonald ? Some other related thoughts I have from a big picture use case perspective: We don't anticipate taxonomies much larger than the Lightcast sample. However, I do anticipate that for folks who create new course runs every term and have very short terms, the number of ObjectTag associations could get pretty large. I think that this is where it could potentially be valuable to add a filter to the tag count to only fetch ObjectTags where the Object corresponds to a course that is currently running or will be running in the future. I think the rest of the big picture for this is that for the folks who create new course runs every term and have very short terms, the usage count is probably meaningless for them and not very helpful anyways "Was it used 250 times or 275 times? How do I know how much usage I should be expecting compared to what I'm seeing?" I think there could also be an option to just hide the usage count column altogether if we detect that their usages are so high that this info is irrelevant for the instance. Or to hide the usage count for people who primarily use course runs each term instead of continuously running courses. |
|
We're trying to stabilize the APIs for Verawood, so if we think we're ultimately going to end up with a second API endpoint for getting the counts, then I'd prefer to split that off separately now, even if we just use the existing implementation exactly as it is in this PR. In that case, it would definitely take less than 2 days, because you don't have to change much (although you could simplify it if you get time). You can also mark that "counts" endpoint as unstable so we can freely evolve it in Willow while keeping the "get tags" endpoint stable.
That all makes a lot of sense, but will require a lot of discussion, because the current API is not aware of "courses" as a concept at all, and I'm a bit reluctant to make the tagging API aware of those things - right now tagging is a very low-level feature that other things build on. If you're even considering functionality like that, then I think it's another reason to move the tag usage counts to a separate endpoint, where it can support more elaborate options/filtering. @ormsbee Can I get your thoughts? |
|
@bradenmacdonald if I understand correctly:
Did I get that correctly? So can we consider this PR unblocked in this case? |
…bain/253_add_tags_count_rebased # Conflicts: # tests/openedx_tagging/test_api.py
|
@bradenmacdonald @tbain here is the new issue. I have not worked out an accurate title or description for it, so you can just edit the issue however you see fit. |
This implements openedx/modular-learning#253 , the task to add tag usage counts to the tags table under the taxonomies table. The corresponding backend part is openedx/openedx-core#506, which updates the count aggregations to ensure the correct count numbers are sent to the frontend. This frontend PR does not depend on the backend part.
What I meant was that we should change the PR to provide the desired tag count data via a separate endpoint. But using more or less the exact same code as you have now if you don't want to refactor it. So instead of But I guess that's going to require some major changes on the frontend side to combine those pieces of information, so maybe that's not going to work with your timeline. |
|
I guess before we consider merging this as is, I'd like to know if the slowness mostly scales with taxonomy size or object tag count or both? If the slowness is only a factor on large taxonomies and it's just ~1s, I think that's OK for now. But if it's slow as the # of object tags increases or it's O(n_tags * n_object_tags) or anything like that, then it'll seem fine now and slow to a crawl in prod once people start using thousands of these things and re-running tagged courses. |
I agree with this. If it's ~1s for an outlier taxonomy owing to the number of tags, it's acceptable for now, and we can figure out how to optimize later. If the time scales with the number of things tagged, this will rapidly become unusable.
I'd be cautious about assuming people don't care. I've been told that there's sometimes grant money riding on proving how much things get used. In any case, we'd definitely need product folks to weigh in on it. |
|
FWIW Claude analyzed the query and says it could be slow. I have not had time to validate this analysis, so take with a grain of salt. The generated SQL (at depth=3)SELECT ...,
COALESCE(
(SELECT COUNT(DISTINCT U0."object_id") AS "total_usage"
FROM "oel_tagging_objecttag" U0
INNER JOIN "oel_tagging_tag" U2 ON (U0."tag_id" = U2."id")
LEFT OUTER JOIN "oel_tagging_tag" U3 ON (U2."parent_id" = U3."id")
LEFT OUTER JOIN "oel_tagging_tag" U4 ON (U3."parent_id" = U4."id")
WHERE U0."taxonomy_id" = 1
AND (U0."tag_id" = outer."id"
OR U2."parent_id" = outer."id"
OR U3."parent_id" = outer."id"
OR U4."parent_id" = outer."id")
), 0) AS "usage_count"
FROM "oel_tagging_tag"
WHERE "oel_tagging_tag"."taxonomy_id" = 1The scaling problem: it will get meaningfully slowerThe old query filtered by The new query is a correlated subquery that, for each tag in the result set, does this:
Cost per tag: O(all_ObjectTags_in_taxonomy × D) So the total work is roughly T × O × D where:
It scales linearly with ObjectTag count, but since it's inside a correlated subquery that runs per-tag, the multiplier is the number of tags displayed. This will be painfully slow once a popular taxonomy gets applied to thousands of courses/modules/sections. Why the OR kills performanceThe condition |
|
Okay. So it sounds like the most straightforward thing is to do the up-front query for counts and stitch together the hierarchy counts in Python as @bradenmacdonald outlined in:
Does that sound right to everyone? |
|
That sounds good to me, and has the advantage of requiring no further changes to the frontend PR. |
|
@ormsbee @bradenmacdonald the only question I have is related to memory usage. |
|
@bradenmacdonald @ormsbee just to make sure we have considered all alternatives: AI is suggesting Recursive CTEs as the optimal solution. However, that requires MySQL >= 8. Do we need to support older MySQL versions? I haven't been able to evaluate the AI response in-depth so it may be incorrect AI suggestion:
N = Number of Tags in your main queryset |
…nstead of via expensive db query
|
Ultimately, via a conversation/clarification over Slack (dated 2026-04-02), we decided to address the performance concerns via in-memory python based code processing rather than trying to rely on django joins and sub-queries, or a recursive SQL/CTE implementation. Since we were seeing such an egregious performance hit, the implementation leans towards minimizing performance issues and bottlenecks where possible, potentially at the slight cost of straight-forwardness of what exactly the code is doing (e.g. performance wise it was very expensive to implement the 'annotation' of the |
|
Did some local testing with the large Lightcast taxonomy that Braden posted earlier; applied some tags from that taxonomy to an existing course on my local, and then watched the timings for the Various load times with Lightcast Taxonomy: However, I'm not quite sure this is the best representation of the times to reproduce the same circumstances as Braden saw above with the 10x increase in call time, since I don't have the same tags applied the same way to the same depth, the same course, etc. Also I have a brand new computer that is very fast, which is kind of throwing this off as well. I only have a handful of tags applied; if I could either get some direction from Braden on how he had applied his tickets, or have Braden perform a quick check with his same setup, that would be great. |
bradenmacdonald
left a comment
bradenmacdonald
left a comment
There was a problem hiding this comment.
Thanks! My performance concern is addressed now. I just caught a few more things but hopefully they're relatively straightforward to address.
… up/adding associated unit tests
…bain/253_add_tags_count_rebased
|
As per previous comments, moved the _add_counts logic out to the API level just before the return so that the logic to add the usage_counts doesn't interfere with the QuerySet logic until after all the filtering and paging logic is performed. This was necessitated by performance concerns with using something like a QuerySet annotation Case/When adding a multiplicative Big O complexity, so we had to work around that logic and manually add the value derived via Python logic to the list form of the return data rather than the QuerySet form which doesn't support such direct manipulation. Since we completely moved the counting logic out of the the DB layers up to the API level, this required moving all of the unit testing to the appropriate level and removing it from the lower level unit testing suites. I used AI help to build out as extensive a unit test suite as I could think of in addition to maintaining parity with the tests I had to remove from the lower levels. |
jesperhodge
left a comment
jesperhodge
left a comment
There was a problem hiding this comment.
Hey @tbain this is amazing work. I tested it for correctness and performance, and I love the very extensive tests - that's just great.
Please regard every comment from me as a nit, they are very minor, and I think should not block the PR from being merged at all. That said, it would be nice to have the improvements for the comments in here; but merging this very soon has priority due to the Verawood release.
Can you make sure to bump the version (just a patch I think, not a minor or major version bump)?
Probably squashing can be done while merging.
I'll approve but final approve should come from Braden or Dave
|
|
||
| class TestTaxonomyTagsUsageCount(TestTaxonomyViewMixin): | ||
| """ | ||
| Tests the usage_count rollup logic in the taxonomy tags view |
There was a problem hiding this comment.
I would like the tests to act as a sort of documentation of expected behavior, in general.
For anyone not deep into the Tag Usage Count logic, this will not be understandable.
Can we add the definition I wrote here that clarifies the behavior?
"
A tag can be directly applied to an object, which can be a course, library, module, or something else.
A tag can also be indirectly applied: when some of its children are applied to an object, it is considered automatically applied.
So, if tag "Chemistry" and tag "Physics" are applied once each to different objects, their parent tag "Natural Science" is considered indirectly applied to 2 objects.
Deduplication: A tag can only be applied to a single object once. So if two child tags are applied to the same object, the parent tag is only applied to it once, because no tag can be applied to the same object twice.
"
|
|
||
| def test_usage_count_rollup(self): | ||
| """ | ||
| Test that usage counts correctly roll up from children to parents, |
There was a problem hiding this comment.
I'd like the test description also to be very clear. You can orient yourself at the definition I wrote above.
I would only use the term "roll up" if it's coupled with an explanation on what rollup is ("When descendant tags are applied to an object, the parent tag is considered automatically applied to the object, which may result in an increase of the usage count value of the parent" or something like that)
There was a problem hiding this comment.
Also please specify what the expected behavior here actually is (when is it "correct"?)
|
|
||
| def test_usage_count_rollup_multi_level(self): | ||
| """ | ||
| Test that usage counts correctly roll up across more than two levels |
There was a problem hiding this comment.
How do we know what's "correct" vs "incorrect"? Please specify.
| results = {tag["value"]: tag for tag in response.data["results"]} | ||
|
|
||
| # --- Verification --- | ||
| # Arthropoda: applied to obj1, obj2 -> count: 2 |
There was a problem hiding this comment.
I love these comments that explain exactly what is expected.
| assert results["Cnidaria"]["usage_count"] == 1 | ||
|
|
||
| # Animalia: applied to obj1 (via Arthropoda, Chordata, Cnidaria) and obj2 (via Arthropoda). | ||
| # Should be 2, because it counts '1' per object regardless of how many children are applied. |
|
|
||
| def test_usage_count_one_level_root_and_child_rollup(self): | ||
| """ | ||
| Verify that usage counts roll up even when querying only a single level. |
|
|
||
| def test_usage_count_sibling_and_ancestor_deduplication(self): | ||
| """ | ||
| Test deduplication when multiple children of the same parent are applied to the same object. |
There was a problem hiding this comment.
This description sounds good to me, very understandable.
| if depth == 1: | ||
| # We're already returning just a single level. It will be paginated normally. | ||
| if include_counts: | ||
| return self._add_counts(results) |
There was a problem hiding this comment.
Very minor nit to be clear what this returns:
| return self._add_counts(results) | |
| results_with_counts = self._add_counts(results) | |
| return results_with_counts |
There was a problem hiding this comment.
This is a bit subjective, some might not like the suggestion. If @bradenmacdonald you like it please thumbs up this.
If Braden does not thumbs up this please do not implement it @tbain .
There was a problem hiding this comment.
I don't have a strong opinion here but I guess I do like your suggestion a bit more, as it's more clear.
| # We can load and display all the tags in this (sub)tree at once: | ||
| self.pagination_class = DisabledTagsPagination | ||
| if include_counts: | ||
| return self._add_counts(results) |
There was a problem hiding this comment.
| return self._add_counts(results) | |
| results_with_counts = self._add_counts(results) | |
| return results_with_counts |
There was a problem hiding this comment.
This is a bit subjective, some might not like the suggestion. If @bradenmacdonald you like it please thumbs up this.
If Braden does not thumbs up this please do not implement it @tbain .
| return results.filter(parent_value=parent_tag_value) | ||
| filtered_results = results.filter(parent_value=parent_tag_value) | ||
| if include_counts: | ||
| return self._add_counts(filtered_results) |
There was a problem hiding this comment.
| return self._add_counts(filtered_results) | |
| results_with_counts = self._add_counts(results) | |
| return results_with_counts |
There was a problem hiding this comment.
This is a bit subjective, some might not like the suggestion. If @bradenmacdonald you like it please thumbs up this.
If Braden does not thumbs up this please do not implement it @tbain .
bradenmacdonald
left a comment
bradenmacdonald
left a comment
There was a problem hiding this comment.
Looks great! I'll approve and merge as soon you've addressed the include_counts comment and as many of Jesper's suggestions as you'd like.
|
|
||
| return filtered_results | ||
|
|
||
| def _add_counts(self, tag_data: TagDataQuerySet) -> TagDataQuerySet: |
There was a problem hiding this comment.
Nit: this could be a top-level function in src/openedx_tagging/api.py.
Nit: Also, along the lines of @jesperhodge 's comments, it would be good to explain the de-duplication here or at lest say "refer to test case X for examples and details". (Or put clear details here and have the test cases refer back here).
| self, | ||
| search_term: str | None, | ||
| include_counts: bool, | ||
| include_counts: bool, # pylint: disable=unused-argument |
There was a problem hiding this comment.
Why are we keeping these include_counts parameters if they don't work? I think you can just remove them (from here and from the function that calls these ones).
There is no code referencing include_counts in openedx-platform as far as I can tell, so even though it's a breaking change, it's fine to include such changes in our 0.x version of openedx-core.
There was a problem hiding this comment.
Yeah, I wasn't sure about keeping these or not. I know it's probably a "you aren't going to need it" type thing, but for completions sake, I felt like it was worth continuing them down to this level since they are a param in the query. I'm perfectly fine removing them though, I was 50/50 either way.
Description
This implements openedx/modular-learning#253 , the task to add tag usage counts to the tags table under the taxonomies table. The frontend piece is where the results of this aggregation work is displayed is part of a separate pr to openedx/frontend-app-authoring. This change adds a subquery annotation onto the django query for retrieving tags. The original implementation of the counts for tags only counted raw usage of each tag. This feature/PR aggregatea sum of any tag and child tag usage with sibling de-duplication for the same usage (e.g. when two sibling nodes are used against the same course, module, etc. we still only need to count that as '1' for any parent/grandparent nodes) as specified in the AC for the issue above, so it was replaced with this more complicated bit of logic that sums across tag usage based on various courses, sections, modules, and libraries that might use a tag.
update:
The count logic is done in-memory, since we saw noticeable performance issues with trying to stay in the QuerySet/Django paradigm for calculating the counts. This makes the code a little less straightforward, since we break it out into a somewhat odd in-memory python application of the logic, but it still works as intended and resolves as many performance pain points as possible while still adhering to the counting requirements that end up necessitating such code.
update:
The count logic was moved out to the API level so that all the query data could complete and the logic to add the counts would not affect the ability to process the data as a QuerySet until the very end. This necessitated removing all the usage_count unit tests out of the _models and _api levels to the _view levels, so re-implemented that logic as appropriate (with a lot of AI help to speed it up)
AI Usage Disclosure: Claude was used via intelliJ IDE integration was used through the authoring process to work through complicated logic, craft the foundation of the unit tests, and also simplify it/make it more pythonic/alleviate performance concerns.
Supporting information
Github issue with AC: openedx/modular-learning#253
Testing instructions
Refer to the AC in the Github Issue. Steps to verify this is implemented and working via UX (Note, depends on the frontend part of this ticket):
Other information
Include anything else that will help reviewers and consumers understand the change.