Skip to content

feat: add outliers_nb_target to approximate_distribution() for target-based min_similarity [stacks on #2519]#2520

Draft
pidefrem wants to merge 5 commits into
MaartenGr:masterfrom
pidefrem:pr08-auto-min-similarity
Draft

feat: add outliers_nb_target to approximate_distribution() for target-based min_similarity [stacks on #2519]#2520
pidefrem wants to merge 5 commits into
MaartenGr:masterfrom
pidefrem:pr08-auto-min-similarity

Conversation

@pidefrem

@pidefrem pidefrem commented Jul 8, 2026

Copy link
Copy Markdown

What does this PR do?

feat: add outliers_nb_target to approximate_distribution() for target-based min_similarity

approximate_distribution() requires manually specifying min_similarity. Users must iterate to find a value that produces an acceptable number of documents with zero topic distribution.

Changes:

Add an outliers_nb_target parameter (int): auto-searches for the min_similarity value that achieves the target number of zero-distribution documents using binary search.

# Before: manual trial-and-error
distributions = topic_model.approximate_distribution(docs, min_similarity=0.1)

# After: specify the target directly
distributions = topic_model.approximate_distribution(docs, outliers_nb_target=50)

Performance optimization: Refactors the internals so the expensive similarity matrix is computed once and reused across all binary search iterations:

  • compute_similarities_for_approximate_distribution() — compute once
  • compute_topic_distributions_for_approximate_distribution() — apply threshold, iterate

Cannot set both min_similarity and outliers_nb_target. Backward compatible: defaults to None.

Fixes #2502

Before submitting

  • This PR fixes a typo or improves the docs (if yes, ignore all other checks!).
  • Did you read the contributor guideline?
  • Was this discussed/approved via a Github issue? Please add a link to it if that's the case.
  • Did you make sure to update the documentation with your changes (if applicable)?
  • Did you write any new necessary tests?

⚠️ Draft — stacks on #2519. This branch includes the reduce_outliers auto-threshold commit from #2519 (and its upstream prerequisites) as [prereq] commits, so the diff here shows that work too. Once #2519 merges, I'll rebase onto master and the prereq commits drop out, leaving only the outliers_nb_target change for approximate_distribution(). Marking ready for review after #2519 lands.

@pidefrem pidefrem changed the title feat: add outliers_nb_target to approximate_distribution() for target-based min_similarity feat: add outliers_nb_target to approximate_distribution() for target-based min_similarity [stacks on #2519] Jul 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

approximate_distribution: add target-based min_similarity ("at most N unassigned documents")

2 participants