Skip to content

Feature: Add LLM-friendly content endpoints#104

Draft
neoskx wants to merge 7 commits into
masterfrom
feature/llms
Draft

Feature: Add LLM-friendly content endpoints#104
neoskx wants to merge 7 commits into
masterfrom
feature/llms

Conversation

@neoskx
Copy link
Copy Markdown

@neoskx neoskx commented May 8, 2026

Summary

  • Adds a build-time content generator that produces llms.txt index, llms-full.txt (all docs concatenated), and per-page raw markdown files at /{slug}/index.md
  • Every documentation page now has its source markdown accessible by appending index.md to the page URL (e.g., /getting-started/index.md, /types/basic-types/index.md)
  • Integrates into the deploy workflow as a GitHub Action that runs before the Astro build

Motivation

LLM agents and tools increasingly look for llms.txt to discover documentation in a machine-readable format. Serving raw markdown alongside rendered pages lets both LLMs and
developers access structured content without scraping HTML.

What's included

File Purpose
.github/actions/generate-llm-content/ Node.js action that parses all docs, generates llms.txt, llms-full.txt, and copies raw markdown to public/{slug}/index.md
.github/workflows/deploy.yaml Runs the generator before the Astro build
.gitignore Ignores generated files in public/
package.json Adds generate:llms script

Local development

cd .github/actions/generate-llm-content && npm install && npm run start:local
cd ../../.. && npm run dev
# Visit http://localhost:4321/llms.txt
# Visit http://localhost:4321/getting-started/index.md

Test plan

  • npm run start:local generates 22 files without errors
  • llms.txt links all use correct base URL
  • Every link in llms.txt resolves to an existing file in public/
  • Raw markdown at /{slug}/index.md matches source file exactly (including frontmatter)
  • Deploy workflow runs the action successfully

neo-roblox and others added 7 commits May 8, 2026 11:00
Generate llms.txt, llms-full.txt, and per-page raw markdown files
(/{slug}/index.md) so LLMs and users can access source markdown
alongside rendered pages. The generator runs as a GitHub Action during
deploy and locally via `npm run start:local`.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Both files use the BASE_URL env var so they point to localhost in
development and https://luau.org in production.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Consistent with the existing llms.txt and llms-full.txt templates.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Starlight already generates sitemap.xml at build time via @astrojs/sitemap.
Use a static robots.txt in public/ instead of generating it.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Astro/Starlight generates sitemap-index.xml, not sitemap.xml.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The action needs Node.js available for npm ci and tsx.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@bradsharp bradsharp requested a review from aatxe May 8, 2026 21:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants