Generate API docs with sphinx-autodoc2 for Markdown support - #784
Draft
eb8680 wants to merge 1 commit into
Draft
Conversation
`sphinx.ext.autodoc` imports the package and introspects live objects; `sphinx-autodoc2` reads the source with `astroid` instead. Two things motivate the switch. The docstrings in `handlers.llm` are Markdown, and RST rendered them as literal `##` and broken fence anchors. They are model-facing -- the legibility framework and every `PromptInjectingInterpretation` inject them into the system prompt verbatim, and `serialization` parses their ATX headings to rebase heading levels -- so the renderer moves to the docstrings rather than the other way round. `autodoc2_docstring_parser_regexes` names the six that are Markdown; the rest of the package stays reStructuredText. The hand-written `.. autofunction::` overrides also go away. They existed because `automodule :members:` skips names whose `__module__` differs: the torch wrappers `functools.wraps` relabels as `torch.func`, and the jax operations re-exported from `._handlers`. Static analysis sees the `def` regardless, so all eleven torch functions render on their own; jax needs an `__all__` to resolve its re-exports, since analysis never follows imports. Two configuration choices are less obvious than they look. `autodoc2_render_plugin` stays `rst`: MyST pages would make MyST the default parser for every docstring, and CommonMark reads a `>>>` prompt as a blockquote, so `ops.syntax` alone turned 31 doctest blocks into 186 nested quotes. And `autodoc2_docstrings` is `direct`, not `all`, despite the latter looking like the analogue of `autodoc_inherit_docstrings`: the old `effectful.rst` never passed `:inherited-members:`, and `all` repeats a base class's docstring on every subclass -- 40 copies of `_DistributionTerm` across the numpyro wrappers, `Operation`'s doctests onto two subclasses in `ops.syntax`, and stdlib boilerplate onto classes whose base is a `dict` or an `ABC`. The API reference is now a page per module under `apidocs/`, so `effectful.rst` becomes a redirect and per-object anchors no longer resolve. `sphinx_autodoc_typehints` is dropped because it only hooks autodoc events, and `sphinxcontrib-bibtex` because nothing used it. The docs jobs keep their unpinned interpreter. `sphinx-autodoc2` pins `astroid<4`, so 3.14 looked like a risk, but building under 3.14.7 -- what `uv python install` resolves to -- gives an API tree byte-identical to 3.13, the same 30 warnings and the same 36 `_modules` pages. Static analysis does not see everything an import does. PEP 695 `type` aliases (including `Expr`), definitions inside module-level `try:` or `if TYPE_CHECKING:` blocks (`Encodable`), `@overload`-only members, and six of the seven torch functorch wrappers' signatures are absent, and per-module `:private-members:` is no longer expressible.
eb8680
force-pushed
the
worktree-autodoc2-migration
branch
from
September 5, 2026 01:30
084e7fa to
cf7fd7d
Compare
eb8680
marked this pull request as draft
September 5, 2026 02:03
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves #774
This seems to be the path of least resistance to resolving #774 according to the LLMs and the Myst and Sphinx documentation.
sphinx.ext.autodochas no built-in support for Markdown docstrings. Rewriting the offending harness docstrings as RST isn't ideal because RST in model-facing text doesn't play well with the renderer inhandlers.llm.harness.observability.rich. Migrating the entire rest of the repo away from Sphinx to Markdown and MkDocs/Zensical would be nice in theory but also touch almost every file and break all open PRs.