feat: migrate to nuxt-agent-discovery

docs/content/en/2.concepts/3.configuration.md

@@ -149,7 +149,7 @@ Your component receives `title`, `description`, and `headline` (docs only) as pr
 
 ### Sitemap
 
-Docus automatically generates a sitemap at `/sitemap.xml` containing all your documentation pages. This helps search engines discover and index your content.
+Docus generates a sitemap at `/sitemap.xml` with [`@nuxtjs/sitemap`](https://nuxtseo.com/docs/sitemap), listing every content page. On a multilingual site it becomes a sitemap index with one sitemap per locale. Every `sitemap` option of the module is available in your `nuxt.config.ts`.
 
 #### Excluding pages
 

docs/content/en/4.ai/2.mcp.md

@@ -33,6 +33,10 @@ For example, if your documentation is hosted at `https://docs.example.com`, your
 The MCP server is documented in your [`llms.txt`](/en/ai/llms) as an `MCP Server` section. An agent that discovers your site through `llms.txt` can connect without anyone pasting the URL into a client configuration.
 ::
 
+::note
+Your site also publishes an MCP server card at `/.well-known/mcp/server-card.json`. It lists the endpoint and the tools, resources and prompts the server exposes, so an agent can find the server without reading the docs.
+::
+
 ## Disable the MCP Server
 
 If you want to disable the MCP server, you can do so in your `nuxt.config.ts`:

docs/content/en/4.ai/3.skills.md

@@ -90,11 +90,11 @@ Keep your main `SKILL.md` under 500 lines. Move detailed reference material to s
 
 ## Configuration
 
-By default, Docus looks for skills in the `skills/` directory at the root of your project. You can change this with `docus.skills.dir` in your `nuxt.config.ts`:
+By default, Docus looks for skills in the `skills/` directory at the root of your project. You can change this with `agentDiscovery.skills.dir` in your `nuxt.config.ts`, or set `agentDiscovery.skills` to `false` to stop publishing them:
 
 ```ts [nuxt.config.ts]
 export default defineNuxtConfig({
-  docus: {
+  agentDiscovery: {
     skills: {
       dir: 'agent-skills'
     }

docs/content/en/4.ai/4.llms.md

@@ -77,49 +77,64 @@ curl -H "Accept: text/markdown" https://docus.dev/en/does-not-exist
 ```
 
 ```md
-# 404 — Page not found
+---
+title: "Page Not Found"
+status: 404
+---
+
+# 404 Page Not Found
 
-`/en/does-not-exist` was not found on this site.
+The page `/en/does-not-exist` does not exist on https://docus.dev.
 
 ## Where to look next
 
-- [/llms.txt](/llms.txt): index of Docus
-- [/llms-full.txt](/llms-full.txt): the full content of this site as a single markdown document
-- [/sitemap.xml](/sitemap.xml): every page, with its last modification date
-- [/.well-known/skills/index.json](/.well-known/skills/index.json): agent skills published by this site
-- [/](/): home page
+- [API catalog: every service document this site publishes](https://docus.dev/.well-known/api-catalog)
+- [Sitemap (Markdown): every page on the site](https://docus.dev/sitemap.md)
+- [MCP server card: MCP endpoint at https://docus.dev/mcp](https://docus.dev/.well-known/mcp/server-card.json)
+- [MCP endpoint (streamable HTTP)](https://docus.dev/mcp)
+- [llms.txt: index of the documentation for LLMs](https://docus.dev/llms.txt)
+- [llms-full.txt: the full documentation as a single file](https://docus.dev/llms-full.txt)
+- [Agent skills index: every skill published by this site](https://docus.dev/.well-known/skills/index.json)
+- [Agent skill: create-docs](https://docus.dev/.well-known/skills/create-docs/SKILL.md)
+- [Agent skill: review-docs](https://docus.dev/.well-known/skills/review-docs/SKILL.md)
+- [Sitemap (XML): every page, with its last modification date](https://docus.dev/sitemap.xml)
+
+## Fetching markdown
+
+Any documentation page is available as markdown: append `.md` to its URL
+or send `Accept: text/markdown`.
 ```
 
-The skills entry only appears when your site [publishes skills](/en/ai/skills), and `/llms-full.txt` only when it is enabled, so the document never points at a route that does not exist.
+The links come from the same registry as the `Link` header of your homepage and `/.well-known/api-catalog`, so the document never points at a route that does not exist: the skills entries only appear when your site [publishes skills](/en/ai/skills), and `/llms-full.txt` only when it is enabled.
 
-The response keeps the `404` status and is served as `text/markdown; charset=utf-8` with `Vary: Accept`, so CDNs never mix it up with the HTML variant.
+The response keeps the `404` status and is served as `text/markdown; charset=utf-8` with `Vary: Accept, User-Agent`, so CDNs never mix it up with the HTML variant.
 
 Only clients that clearly aren't rendering HTML get this document. Untouched:
 
 - **Browsers**: any request accepting `text/html` still renders the theme error page
-- **API clients**: requests accepting `application/json`, or targeting `/api/**` and `*.json`, keep the default JSON error body
+- **API clients**: requests accepting `application/json`, or under `/api/` and `/mcp`, keep the default JSON error body
 - **`fetch()` and `$fetch`**: browser-initiated requests keep the default JSON error body, so `error.data` stays parseable
-- **Assets**: a missing script, style, image or feed keeps the default error body, since markdown would be meaningless there
 
 To restore the default error body:
 
 ```ts [nuxt.config.ts]
 export default defineNuxtConfig({
-  docus: {
-    notFound: false,
+  agentDiscovery: {
+    errors: false,
   },
 })
 ```
 
 ## Raw Markdown Access
 
-When `nuxt-llms` is enabled, Docus also exposes a raw markdown endpoint so AI agents can fetch LLM-ready source files without going through the full rendering pipeline. This reduces token usage and improves response speed for AI-powered tools consuming your documentation.
+Docus exposes a raw markdown endpoint so AI agents can fetch LLM-ready source files without going through the full rendering pipeline. This reduces token usage and improves response speed for AI-powered tools consuming your documentation.
 
 ### How it works
 
-- **Endpoint**: `/raw/<content-path>.md` — use the same path as the page URL, drop trailing `/index`, and keep the `.md` extension
-- **Content-Type**: `text/markdown; charset=utf-8`
-- **Auto-enrichment**: if the requested document is missing a top-level heading or description, the route automatically prepends the title and description to the markdown body
+- **Endpoint**: `/raw/<content-path>.md`, the same path as the page URL with the `.md` extension
+- **Content-Type**: `text/markdown; charset=utf-8`, with a `Link` header pointing back at the HTML page
+- **Frontmatter**: every document opens with its `title`, `description` and `canonical_url`, and same-origin links are absolute, so a page still makes sense once it has been read away from your site
+- **Sections**: a path naming a section rather than a page (`/raw/en/getting-started.md`) redirects to the first document of that section
 - **LLMs.txt integration**: document links in `llms.txt` are automatically rewritten to the `/raw/...md` endpoint, so agents fetch compact markdown instead of full HTML
 
 ::note{to="/raw/en/ai/llms.md"}
@@ -128,66 +143,52 @@ Try accessing the raw Markdown version of this page.
 
 ### Configuration
 
-You can customize the raw markdown behavior from your `nuxt.config.ts`:
+Every page has a markdown twin by default. To keep a part of your site out of it, a page tree without markdown or a standalone `.md` document you serve yourself, extend `excludePrefixes`. An excluded path never negotiates, is listed nowhere and answers a 404 on the raw endpoint:
 
 ```ts [nuxt.config.ts]
 export default defineNuxtConfig({
-  llms: {
-    contentRawMarkdown: {
-      // Prevent specific page collections from being exposed
-      excludeCollections: ['landing', 'landing_en', 'landing_fr'],
-      // Keep llms.txt links pointing to rendered pages instead of raw markdown
-      rewriteLLMSTxt: false,
+  agentDiscovery: {
+    excludePrefixes: {
+      extend: ['/admin'],
     },
   },
 })
 ```
 
-To disable raw markdown access entirely:
-
-```ts [nuxt.config.ts]
-export default defineNuxtConfig({
-  llms: {
-    contentRawMarkdown: false,
-  },
-})
-```
+## Markdown Negotiation
 
-## Markdown Redirection
-
-::note
-This feature is only available when Docus is deployed on Vercel. We'll be able to make it agnostic once Nitro v3 supports global rewrites for multi vendors.
-::
-
-When deployed on Vercel, Docus automatically configures intelligent routing to serve markdown content to AI agents and CLI tools.
+Docus serves markdown to AI agents and CLI tools asking for it, on the same URL as the HTML page.
 
 ### Why?
 
-Agents like Claude Code use `Accept: text/markdown` headers by default, retuning raw Markdown is saving lots of data transfer and tokens in the process.
+Agents like Claude Code send `Accept: text/markdown` by default. Answering with raw Markdown saves a lot of data transfer and tokens in the process.
 
 ### How?
 
-Docus detects requests from AI agents and command-line tools using HTTP headers:
+A request gets markdown when:
 
-- **Accept header**: Requests with `Accept: text/markdown` are automatically redirected
-- **User-agent detection**: `curl` requests as agents are automatically redirected
+- its URL ends with `.md`: `/en/ai/llms.md` is the markdown twin of `/en/ai/llms`
+- its `Accept` header prefers `text/markdown` over `text/html`
+- its `User-Agent` is a known AI agent (`ClaudeBot`, `GPTBot`, `PerplexityBot` and the others from the [ai.robots.txt](https://github.com/ai-robots-txt/ai.robots.txt) list)
 
-### Redirect Rules
+Then:
 
-- **Root path**: `/` → `/llms.txt`
+- **Root path**: `/` → `/raw/index.md`, your landing page when it is a content document or a generated index when it is a Vue page, both ending with the list of everything this site publishes for agents
 - **Documentation pages**: `/{path}` → `/raw/{path}.md`
 
+Negotiation runs in development and on every host through a Nitro middleware. On Vercel it also runs at the edge, so prerendered pages answer in markdown before the CDN cache sees the request. On other hosts a prerendered page is served ahead of the middleware and stays HTML: agents reach its markdown through the `.md` URL.
+
 ### Example Usage
 
 ```bash
-# Get llms.txt from homepage
+# Get the agent index from the homepage
 curl -H "Accept: text/markdown" https://docus.dev/
 
-# Get llms.txt from locale homepage
-curl -H "Accept: text/markdown" https://docus.dev/en
-
 # Get raw markdown for a documentation page
 curl -H "Accept: text/markdown" https://docus.dev/en/ai/llms
+
+# Same page, through its markdown twin URL
+curl https://docus.dev/en/ai/llms.md
 ```
 
 All these commands will return markdown content instead of HTML.
@@ -201,6 +202,23 @@ curl -sI -H "Accept: text/markdown" https://docus.dev/en/ai/llms | grep -i vary
 # vary: Accept, User-Agent
 ```
 
+## Discovery
+
+An agent landing on your site can find everything it publishes without reading the docs:
+
+- `/.well-known/api-catalog`: every service document this site publishes ([RFC 9727](https://www.rfc-editor.org/rfc/rfc9727)), also advertised in the `Link` header of your homepage
+- `/openapi.json`: an OpenAPI document describing the markdown routes, the discovery documents and the MCP endpoint
+- `/sitemap.md`: a markdown index of every page, grouped by section, one per locale on a multilingual site
+- `/.well-known/mcp/server-card.json`: the [MCP server](/en/ai/mcp) endpoint and the tools it exposes
+- `/.well-known/skills/index.json`: the [agent skills](/en/ai/skills) you publish
+- `/robots.txt`: a group allowing each known AI agent, with a `Content-Signal` line
+
+All of this is provided by the [`nuxt-agent-discovery`](https://github.com/benjamincanac/nuxt-agent-discovery) module and configured under the `agentDiscovery` key of your `nuxt.config.ts`.
+
+::tip{to="https://github.com/benjamincanac/nuxt-agent-discovery"}
+Checkout the nuxt-agent-discovery documentation for every option.
+::
+
 ::tip{to="https://github.com/nuxt-content/nuxt-llms"}
 Checkout the nuxt-llms documentation for more information about the module.
 ::

docs/content/fr/2.concepts/3.configuration.md

@@ -163,7 +163,7 @@ Votre composant reçoit `title`, `description` et `headline` (pages de documenta
 
 ### Sitemap
 
-Docus génère automatiquement un sitemap à `/sitemap.xml` contenant toutes les pages de votre documentation. Cela aide les moteurs de recherche à découvrir et indexer votre contenu.
+Docus génère un sitemap à `/sitemap.xml` avec [`@nuxtjs/sitemap`](https://nuxtseo.com/docs/sitemap), qui liste toutes les pages de contenu. Sur un site multilingue, il devient un index de sitemaps avec un sitemap par locale. Toutes les options `sitemap` du module sont disponibles dans votre `nuxt.config.ts`.
 
 #### Exclure des pages
 

docs/content/fr/4.ai/2.mcp.md

@@ -33,6 +33,10 @@ Par exemple, si votre documentation est hébergée à `https://docs.example.com`
 Le serveur MCP est documenté dans votre [`llms.txt`](/fr/ai/llms) sous forme de section `MCP Server`. Un agent qui découvre votre site via `llms.txt` peut s'y connecter sans que personne ne colle l'URL dans la configuration d'un client.
 ::
 
+::note
+Votre site publie aussi une carte de serveur MCP à `/.well-known/mcp/server-card.json`. Elle liste l'endpoint ainsi que les outils, ressources et prompts exposés par le serveur, pour qu'un agent puisse trouver le serveur sans lire la documentation.
+::
+
 ## Désactiver le Serveur MCP
 
 Si vous souhaitez désactiver le serveur MCP, vous pouvez le faire dans votre `nuxt.config.ts` :

docs/content/fr/4.ai/3.skills.md

@@ -90,11 +90,11 @@ Gardez votre `SKILL.md` principal sous 500 lignes. Déplacez le matériel de ré
 
 ## Configuration
 
-Par défaut, Docus cherche les skills dans le dossier `skills/` à la racine de votre projet. Vous pouvez modifier cela avec `docus.skills.dir` dans votre `nuxt.config.ts` :
+Par défaut, Docus cherche les skills dans le dossier `skills/` à la racine de votre projet. Vous pouvez modifier cela avec `agentDiscovery.skills.dir` dans votre `nuxt.config.ts`, ou définir `agentDiscovery.skills` à `false` pour arrêter de les publier :
 
 ```ts [nuxt.config.ts]
 export default defineNuxtConfig({
-  docus: {
+  agentDiscovery: {
     skills: {
       dir: 'agent-skills'
     }