Fix docs canonical URLs and add missing redirects

[codingfrog27]

Google search console report

This PR was prompted by google search engine reported 112 pages not being indexed. Luckily, upon further inspection this seemed a lot worse than it is. Only 7 pages are real content Google declined to index. Given this and the monorepo merge being imminent, I'm now switching back to the CI cleanup. Since a lot of these issues will be easier to fix with it being in one repo. That said, I do still have a couple small fixes that should take out some entries and give small qol improvements.

The canonical URL bug

packages/astro-theme/index.ts sets build: { format: "file" }, so Astro emits overview.html and Astro.url.pathname carries the suffix throughout the build. Which in itself is not an issue since CF serves the URL extentionless and redirects the .html version to it.

But we built the canonical and og:url from the Astro pathname, resulting in every doc page declaring a canonical that points to a URL that points back to itself 😅.

This is the only place where we have links with .html on it (internal links and sitemap are all extensionless). This caused 24 "page with redirect" entries.

• The twitter url also used the .html version so I've changed that as well. 👌

Fix

Made canonical fallback and og:url take the normalisedpath which is extensionless. Note that this is a theme change that affects all doc sites.

Sitemap

The sitemap had /docs hard coded on high priority. But /docs doesn't serve the homepage, only a meta refresh tag. This is what causes the split second blank page. Since this redirect happens at html level google won't index it. So it doesn't make sense to submit it as our highest priority URL. I've removed the entry, and will add a redirect after the monorepo merge.

Post monorepo merge work

_redirects

There were also 8 doc paths that have been changed since being indexed, added them to the _redirects file at first, but due to the dual deployment setup and differing roots and deployment rules between testing and production I'm saving this for after the merge.

I've documented them as well but for visibility's sake the redirects will be:
/docs/tutorials/expressjs → /docs/getting-started/expressjs
/docs/tutorials → /docs/getting-started
/docs/licensing → /docs/more/licensing
/docs/guides/nginx → /docs/guides/hosting
/docs/guides/natives → /docs/guides/working-around-native-npm-dependencies
/docs/demos/PackagePod → /showcase/package-pod/
/docs/demos/SaySomething → /showcase/say-something/
/docs/demos → /showcase/
/docs/showcase → /showcase/
/docs/community → /showcase/
/docs/blog → https://labs.leaningtech.com/blog
/docs and /docs/ → /docs/overview # replaces the meta-refresh stub

Why add redirect rather than fixing our source?

Because there's no link to edit. The deprecated paths are the result of file renaming and moving. And the reference to the old URL's aren't in our repo but in googles index (plus potentially user bookmarks etc.).

[cloudflare-workers-and-pages]

Deploying labs-browserpod-previews with    Cloudflare Pages

Latest commit:	24b129f
Status:	✅  Deploy successful!
Preview URL:	https://27b0eee7.labs-browserpod-previews.pages.dev
Branch Preview URL:	https://fix-docs-canonical-and-redir.labs-browserpod-previews.pages.dev

View logs

[GabrielaReyna]

Thanks for working on this Merlin. I left a couple of comments, please address them and this should be ready to merge after that.