Skip to content

feat: add outliers_percentage_target to reduce_outliers() for target-based outlier reduction [stacks on #2516]#2519

Draft
pidefrem wants to merge 4 commits into
MaartenGr:masterfrom
pidefrem:pr07-auto-threshold-reduce-outliers
Draft

feat: add outliers_percentage_target to reduce_outliers() for target-based outlier reduction [stacks on #2516]#2519
pidefrem wants to merge 4 commits into
MaartenGr:masterfrom
pidefrem:pr07-auto-threshold-reduce-outliers

Conversation

@pidefrem

@pidefrem pidefrem commented Jul 8, 2026

Copy link
Copy Markdown

What does this PR do?

feat: add outliers_percentage_target to reduce_outliers() for target-based outlier reduction

reduce_outliers() requires manually specifying a threshold parameter. Users must run the method repeatedly to find a threshold that achieves their desired outlier rate — a trial-and-error workflow evident in 10+ open issues about reduce_outliers behavior.

Changes:

Add an outliers_percentage_target parameter (float, 0–1): auto-searches for the threshold that achieves the target outlier percentage using binary search. Works with all 4 existing strategies (probabilities, distributions, c-tf-idf, embeddings) and requires no changes to the underlying reduction logic.

# Before: manual trial-and-error
new_topics = topic_model.reduce_outliers(docs, topics, strategy="probabilities", threshold=0.3)

# After: specify the target directly
new_topics = topic_model.reduce_outliers(docs, topics, strategy="probabilities",
                                          outliers_percentage_target=0.05)

Cannot set both threshold and outliers_percentage_target — raises ValueError. Backward compatible: defaults to None, existing behavior unchanged.

Fixes #2501

Before submitting

  • This PR fixes a typo or improves the docs (if yes, ignore all other checks!).
  • Did you read the contributor guideline?
  • Was this discussed/approved via a Github issue? Please add a link to it if that's the case.
  • Did you make sure to update the documentation with your changes (if applicable)?
  • Did you write any new necessary tests?

⚠️ Draft — stacks on #2516. This branch includes the recalculate_representative_docs() commit from #2516 (and its upstream prerequisites) as [prereq] commits, so the diff here shows that work too. Once #2516 merges, I'll rebase onto master and the prereq commits drop out, leaving only the outliers_percentage_target change. Marking ready for review after #2516 lands.

@pidefrem pidefrem changed the title feat: add outliers_percentage_target to reduce_outliers() for target-based outlier reduction feat: add outliers_percentage_target to reduce_outliers() for target-based outlier reduction [stacks on #2516] Jul 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

reduce_outliers: add target-based outlier reduction ("reduce to X% outliers")

2 participants