Rustdoc merging with RFC3662

[cormacrelf]

Fixes #1097. Basically you can now:

• Generate merged rustdoc for all your first-party crates, or any subset of them, via bxl
• With a search index of all of them
• With all srcs and cross-crate trait impls available
• With perfect caching of the single-crate doc builds
• With the option to link to docs.rs for 3rd party crates
• With all rustdoc_flags respected
• With forwards compat for the very recent change from rustdoc --emit=invocation-specific to --emit=html-non-static-files

Also deals somewhat with the problems in #876 by making -Zcrate-attr easier to use with a simple attr on rust_library(rustdoc_html_root_url= ...). If this is merged, we should update reindeer to generate this automatically.

Also makes rustdoc themes easier to use, by generating stub theme files so we don't force a rebuild of all crate docs when making tiny tweaks to themes. The only thing rustdoc actually looks at until the final merge step is the filename. This applies to regular [doc] subtargets, but they get the real theme files.

All this is exercised in the example project.

[meta-codesync]

@facebook-github-bot has imported this pull request. If you are a Meta employee, you can view this in D101353961. (Because this pull request was imported automatically, there will not be any future comments.)

[cormacrelf]

Maybe @dtolnay would like to review?

There are some comments about how rustdoc resolves external links to docs.rs for third party crates, to the effect of "rustdoc is pretty annoying here" that might be appropriate to PR against rustdoc itself. If its priority order for getting a URL while documenting a downstream crate were not so weird, we could use the --extern-html-root-url override (or however you spell it) and not have to use -Zcrate-attr=doc(...).

The hacky way of adding -Zcrate-attr works pretty well for now and does deal with incorrect-but-baked-into-source doc(html_root_url) metadata that appears on a fair few third party crates. I understand meta just uses a macro for such purposes but making an attribute on rust_library overall makes this feature way easier to use and allows us to exclude third party crates from doc builds at all in the BXL. If we fixed rustdoc and used flags on downstream deps instead, we would still need the upstream crate to know its URL to put it in a provider. And finally, if we get reindeer generating docs.rs urls from the version info, it will work out of the box.

Maybe the BXL could also do with some work, you are better at BXL than I am.

Only other thing I can think to add is a wrapper script for the BXL that hosts it on localhost or opens an appropriate index.html in the browser.