Automate updating the manuals on docs.gap-system.org

README.md

@@ -102,8 +102,14 @@ advertised or used anywhere):
 - `.htaccess`: additional redirect rules; copy of `etc/docs.htaccess`
   in the `GapWWW` repository
 
-The directories `v4.X.Y` can be (re)generated from a GAP installation and the
-corresponding `.json` file in `_data/package-infos/`. For example:
+A new GAP release is picked up automatically: `etc/mirror-manuals.py`, run
+hourly by a systemd timer on that host, builds the manuals of any release it
+does not have yet and moves the `latest` symlink. See
+[`etc/README.gap-docs.md`](etc/README.gap-docs.md).
+
+The directories `v4.X.Y` can also be (re)generated by hand from a GAP
+installation and the corresponding `.json` file in `_data/package-infos/`. For
+example:
 
     etc/extract_manuals.py /usr/local/gap-4.27.3  _data/package-infos/4-27-3.json
 
@@ -116,5 +122,10 @@ symlink suitably.
 
 Various archives for GAP and packages are served from a separate
 subdomain, namely <https://files.gap-system.org>. The files served there
-are from the directory `/srv/www/www-gap-docs/files/http` on
+are from the directory `/srv/www/www-gap-files/data/http` on
 `www-admin13.rz.rptu.de`, username `www-gap-files`.
+
+Nothing there needs updating by hand: both the package archives and the
+archives of GAP releases are mirrored automatically. The scripts and systemd
+units that do this live in their own repository,
+<https://github.com/gap-system/gap-files>, which also documents that host.

etc/README.gap-docs.md

@@ -0,0 +1,120 @@
+# docs.gap-system.org
+
+The GAP and GAP package manuals, one directory per GAP release. See
+[`README.server.md`](README.server.md) for the conventions shared with the other
+two sites.
+
+    ssh gap-docs         # www-gap-docs@www-admin13.rz.rptu.de
+
+Note that this site is *not* the Jekyll website: nothing here is generated by
+Jekyll, and the webhook that rebuilds <https://www.gap-system.org> has no effect
+on it. The only thing the two have in common is that the scripts for both live
+in this repository.
+
+## What is where
+
+```
+/srv/www/www-gap-docs/data/          (== ~/data; ~/http is the document root)
+├── http/                    document root
+│   ├── v4.16.0/{doc,pkg}/   the manuals of one release, ~310 MB each
+│   ├── v4.15.1/...          one directory per release, back to v4.11.1
+│   ├── latest -> v4.16.0    always the newest installed release
+│   ├── doc -> latest/doc    stable URL for the GAP manual
+│   ├── pkg -> latest/pkg    stable URL for the package manuals
+│   ├── index.html           placeholder
+│   ├── manual.js
+│   └── .htaccess            hand-made copy of etc/docs.htaccess
+├── GapWWW/                  git clone of this repository
+└── .mirror-manuals/         scratch space, only while a build runs
+~/.config/systemd/user/gap-docs-manuals.{service,timer}
+```
+
+Nothing in the document root is version specific except the `vX.Y.Z` directories
+themselves and the `latest` symlink: `.htaccess` and `manual.js` contain no
+version numbers, and `doc` and `pkg` resolve through `latest`. Installing a
+release therefore only means creating one directory and moving one symlink.
+
+## How an update happens
+
+`gap-docs-manuals.timer` runs hourly. It first pulls this repository — both the
+job and `extract_manuals.py` live in it — and then runs
+[`mirror-manuals.py`](mirror-manuals.py), which:
+
+1. asks GitHub for the releases of `gap-system/gap`, ignoring pre-releases;
+2. skips every release for which `~/http/vX.Y.Z` already exists, so the usual
+   run does nothing and costs one API call;
+3. for anything left, downloads `gap-X.Y.Z.tar.gz` and `package-infos.json.gz`,
+   verifying both against the `.sha256` files published beside them, unpacks
+   them, and runs [`extract_manuals.py`](extract_manuals.py);
+4. moves the finished tree into `~/http/vX.Y.Z` in a single rename, so a
+   half-extracted set of manuals is never served;
+5. repoints `latest` by renaming a new symlink over the old one, so that
+   `latest` — and `doc` and `pkg` with it — is never even briefly missing;
+6. deletes its scratch directory.
+
+Because pre-releases are ignored, the manuals of a new release appear within an
+hour of it being switched from "pre-release" to "latest release" on GitHub, and
+not before. A build takes roughly ten minutes and needs about 2 GB of scratch
+space: the release tarball is ~570 MB and unpacks to ~1.3 GB.
+
+Releases older than 4.11.1 are ignored, because they predate the
+`package-infos.json` file that `extract_manuals.py` needs.
+
+This replaces the manual `./download_manuals.sh X.Y.Z` step that used to be part
+of the GAP release process; that script still exists in the home directory on
+this host and can be used if the automation is broken.
+
+## Troubleshooting
+
+    systemctl --user list-timers
+    systemctl --user status gap-docs-manuals.service
+    journalctl --user -u gap-docs-manuals.service -n 100
+
+To see what it would do, or to run it immediately rather than waiting:
+
+    ~/data/GapWWW/etc/mirror-manuals.py --dry-run
+    systemctl --user start gap-docs-manuals.service
+
+A failed run leaves the scratch directory behind; it is cleared at the start of
+the next run, and it is always safe to delete by hand. Since the check is
+"does `~/http/vX.Y.Z` exist", the way to force a rebuild of one release is to
+move that directory aside and start the service.
+
+Rebuilding the manuals of an arbitrary GAP installation by hand, without going
+through a release, is also possible:
+
+    etc/extract_manuals.py /usr/local/gap-4.27.3 _data/package-infos/4-27-3.json
+
+which produces a `Manuals` directory to be renamed to `~/http/v4.27.3`. Remember
+to update `latest` too.
+
+## `.htaccess`
+
+The document root's `.htaccess` is a hand-made copy of
+[`docs.htaccess`](docs.htaccess) in this directory; nothing deploys it
+automatically, so the two can drift apart, and have. If you change one, change
+the other. Most of it maps the old mixed-case package directory names to the
+lowercase ones used now.
+
+## Setting this up from scratch
+
+Beyond the shared requirements in [`README.server.md`](README.server.md), this
+site needs only Python 3 with `requests`. As `www-gap-docs`:
+
+    ln -s /srv/www/www-gap-docs/data ~/data
+    git clone https://github.com/gap-system/GapWWW ~/data/GapWWW
+
+    mkdir -p ~/.config/systemd/user
+    cp ~/data/GapWWW/etc/gap-docs-manuals.* ~/.config/systemd/user/
+    systemctl --user daemon-reload
+    systemctl --user enable --now gap-docs-manuals.timer
+
+Then copy `etc/docs.htaccess` to `~/http/.htaccess`, and create the `doc` and
+`pkg` symlinks:
+
+    ln -s latest/doc ~/http/doc
+    ln -s latest/pkg ~/http/pkg
+
+On an empty document root the first run will build the manuals of *every*
+release since 4.11.1, which takes hours and a lot of bandwidth. Use `--since` to
+limit that to the releases you actually want.

etc/README.gap-www.md

@@ -0,0 +1,119 @@
+# www.gap-system.org
+
+The GAP website itself: a Jekyll site built from this repository. See
+[`README.server.md`](README.server.md) for the conventions shared with the other
+two sites.
+
+    ssh gap-www          # www-gap-systems@www-admin13.rz.rptu.de
+
+## What is where
+
+```
+/srv/www/www-gap-systems/data/       (== ~/data; ~/http is the document root)
+├── http/                    document root; the Jekyll build output
+├── GapWWW/                  git clone of this repository, branch master
+├── gap-website.trigger      touched by webhook.php to request an update
+├── webhook.secret           the shared secret, see below; not in git
+├── ForumArchive/            symlinked into http/ by etc/update.sh
+└── ForumArchive2/
+~/.config/systemd/user/gap-website.{path,service}
+```
+
+The clone is owned by `www-gap-systems:www-gap-systems`.
+
+## How an update happens
+
+1. Something is pushed to `master` of this repository.
+2. GitHub calls the webhook at <https://www.gap-system.org/webhook.php>, which
+   is `webhook.php` from the top of this repository, served from the document
+   root.
+3. `webhook.php` checks the signature and, for a push event, `touch`es
+   `~/data/gap-website.trigger`. That is all it does — it deliberately runs no
+   code itself.
+4. `gap-website.path`, a systemd user unit watching that file, notices and
+   starts `gap-website.service`.
+5. That runs [`update.sh`](update.sh): it resets the clone to `origin/master`,
+   re-appends the webhook secret to `.htaccess`, runs `bundle install`, builds
+   the site with Jekyll into `~/http`, and restores the `ForumArchive` symlinks.
+
+Note step 5 resets hard and rebuilds unconditionally; the webhook payload is not
+inspected beyond checking that it is a push.
+
+## The webhook secret
+
+The secret lives only on the server, in `~/data/webhook.secret`, as a single
+line of Apache configuration:
+
+    SetEnv GITHUB_WEBHOOK_SECRET "MY_SECRET"
+
+`update.sh` appends this file to `.htaccess` after every `git reset --hard`,
+which is how the setting survives without the secret ever being committed. The
+side effect is that the working tree is permanently dirty in `.htaccess`, which
+is why the script resets rather than pulls.
+
+The same value must be set at
+<https://github.com/gap-system/GapWWW/settings/hooks>, where the webhook should
+be configured as:
+
+- Payload URL: <https://www.gap-system.org/webhook.php>
+- Content type: `application/x-www-form-urlencoded` (TODO: switch to JSON?)
+- Secret: matching `GITHUB_WEBHOOK_SECRET`
+- SSL verification enabled
+- Trigger: just the push event
+
+If `GITHUB_WEBHOOK_SECRET` is not set at all, `webhook.php` skips the signature
+check entirely rather than failing, so a lost secret does not break updates — it
+silently makes the endpoint unauthenticated. Worth remembering when debugging.
+
+## Troubleshooting
+
+    systemctl --user status gap-website.service gap-website.path
+    journalctl --user -f -u 'gap-website.*'
+
+If systemd reports that the units do not exist, reinstall them:
+
+    cp ~/data/GapWWW/etc/gap-website.* ~/.config/systemd/user/
+    systemctl --user daemon-reload
+    systemctl --user enable --now gap-website.service gap-website.path
+
+Broken file ownership — typically after poking at the clone as another user —
+stops git or Jekyll from writing. As root:
+
+    chown -R www-gap-systems:www-gap-systems ~/data/GapWWW ~/data/http
+    touch ~/data/gap-website.trigger
+    chown www-gap-systems:www-gap-systems ~/data/gap-website.trigger
+    chmod 0664 ~/data/gap-website.trigger
+
+The trigger file needs to be writable by the Apache/PHP user, which is a
+different uid from `www-gap-systems`, while being watched by the
+`www-gap-systems` systemd session — hence the group-writable mode. This is the
+fiddliest part of the whole setup.
+
+## Setting this up from scratch
+
+Requirements beyond the shared ones in [`README.server.md`](README.server.md):
+Ruby 2.7 or newer including development headers, and bundler
+(`apt install bundler`); PHP, needed only for the webhook.
+
+As root: create the account and virtual host with document root `~/http`, set
+`GITHUB_WEBHOOK_SECRET` in the vhost configuration, enable PHP, and
+`loginctl enable-linger www-gap-systems`.
+
+Then, as `www-gap-systems` (`sudo -u www-gap-systems -g www-gap-systems bash`):
+
+    ln -s /srv/www/www-gap-systems/data ~/data
+    git clone https://github.com/gap-system/GapWWW ~/data/GapWWW
+    # otherwise adjust the path in gap-website.service
+
+    touch ~/data/gap-website.trigger
+    chmod 0664 ~/data/gap-website.trigger
+
+    # create ~/data/webhook.secret with the SetEnv line described above
+
+    mkdir -p ~/.config/systemd/user/
+    cp ~/data/GapWWW/etc/gap-website.* ~/.config/systemd/user/
+    systemctl --user daemon-reload
+    systemctl --user enable --now gap-website.service gap-website.path
+
+Finally configure the webhook on GitHub as described above, and check that a
+push to `master` really does rebuild the site.