Myst yml metadata fallback

README.md

@@ -215,6 +215,39 @@ This should start multiple `gunicorn` workers, each one of them binding our flas
 
 > Reminder: Replace the **`<type>`** in the commands above either with `preprint` or `preview` depending on the server (e.g., `neurolibre-preview.service`) you are configuring. Note that this is not only a naming convention, but also defines a functional separation between the roles of the two servers.
 
+#### Isolate MyST build containers from instance metadata
+
+Build containers execute notebook code from submitted repositories. On the default Docker bridge they can reach the instance metadata service (`169.254.169.254` on OpenStack), which serves user-data and injected credentials.
+
+Create a dedicated network with a fixed bridge name, so the firewall rule has something stable to match:
+
+```
+docker network create --driver bridge \
+  --opt com.docker.network.bridge.name=br-mystbuild mystbuild
+docker pull busybox:latest
+```
+
+Install the service that blocks metadata for that bridge on every boot:
+
+```
+sudo cp ~/full-stack-server/systemd/neurolibre-mystbuild-firewall.service /etc/systemd/system/
+sudo systemctl daemon-reload
+sudo systemctl enable --now neurolibre-mystbuild-firewall.service
+```
+
+Verify — this is the only thing that proves the rule is working:
+
+```
+docker run --rm --network mystbuild curlimages/curl \
+  -s -m 3 http://169.254.169.254/openstack/ ; echo "exit=$?"
+```
+
+A non-zero exit (`7` rejected, `28` timeout) means blocked. `exit=0` with a listing of API versions means it is **not** blocked — check that `br-mystbuild` exists (`ip -o link show br-mystbuild`) and that metadata is not a local address (`ip addr | grep 169.254`, which should print nothing).
+
+Then pass `container_network = 'mystbuild'` to `JupyterHubLocalSpawner` in `api/neurolibre_celery_tasks.py` and restart the Celery worker. myst-libre re-checks this before every build session and refuses to spawn if metadata answers, so a rule lost after a reboot fails loudly instead of silently reopening.
+
+> Do not use `iptables-persistent` for this rule. It snapshots the entire ruleset including Docker's generated rules, and restoring those at boot before Docker starts causes duplicated and conflicting rules. The systemd unit is ordered after `docker.service` and re-adds only this rule.
+
 #### Configure Celery as a systemd service
 
 For Celery async task queue manager to work, there are two requirements:

about.md

@@ -87,6 +87,7 @@ celery -A neurolibre_celery_tasks worker --loglevel=info
 The application runs as systemd services:
 - `neurolibre-preview.service` - Preview server
 - `neurolibre-preprint.service` - Preprint server
+- `neurolibre-mystbuild-firewall.service` - Blocks instance metadata for build containers
 - Celery workers for async tasks
 
 ## Key Directories

api/common.py

@@ -541,13 +541,6 @@ def run_celery_subprocess(command, log_output=True):
         logging.error(f"Command: {' '.join(command)}")
         return -1, str(e)
 
-def get_active_ports(start=3001, end=3099):
-    active_ports = []
-    for conn in psutil.net_connections(kind='inet'):
-        if conn.status == psutil.CONN_LISTEN and start <= conn.laddr.port <= end:
-            active_ports.append(conn.laddr.port)
-    return active_ports
-
 def close_port_by_pid(target_pid):
     """Kill the entire process group rooted at target_pid.
 

api/github_client.py

@@ -1,9 +1,11 @@
 import os
 import re
-from common import get_time
+import logging
+from common import get_time, parse_front_matter
 import json
 import yaml
 import git
+from myst_frontmatter import merge_paper_metadata
 
 # Name of the GitHub organization where repositories 
 # will be forked into for production. Editorial bot 
@@ -330,6 +332,29 @@ def gh_get_paper_markdown(github_client,repo):
     file_content = gh_get_file_content(github_client,repo,"paper.md")
     return file_content
 
+def gh_get_paper_metadata(github_client, repo):
+    """Paper metadata for a submission, with myst.yml filling any gaps.
+
+    NeuroLibre requires myst.yml at the repository root beside paper.md, so a
+    submission need not repeat its title, authors, and affiliations in the
+    paper.md front matter. Returns None only when neither source names an
+    author.
+
+    This runs before the repository is cloned, so both files are fetched
+    through the GitHub API rather than read from disk.
+    """
+    paper = gh_get_file_content(github_client, repo, "paper.md")
+
+    front_matter = None
+    if paper:
+        try:
+            front_matter = parse_front_matter(paper)
+        except yaml.YAMLError as error:
+            logging.warning(f"Could not parse paper.md front matter: {error}")
+
+    myst = gh_get_file_content(github_client, repo, "myst.yml")
+    return merge_paper_metadata(front_matter, myst)
+
 def gh_read_from_issue_body(github_client,issue_repo,issue_id,tag):
     """
     Issue body of the reviews has markers around review entries

api/myst_frontmatter.py

@@ -0,0 +1,268 @@
+"""Translate a myst.yml `project` mapping into inara paper metadata.
+
+A NeuroLibre submission declares its title, authors, and affiliations in
+myst.yml for the living preprint. The publishing pipeline wants the same
+information in the shape the paper.md front matter uses: affiliations numbered
+by index, and each author's affiliations as a comma-joined string of those
+indices.
+
+This module is pure. Fetching files is the caller's job, which keeps the
+mapping testable without a GitHub client. It mirrors
+inara/data/filters/myst-frontmatter.lua; the two share the mapping documented
+in the design spec. It also merges a parsed paper.md front matter with a
+myst.yml, filling any gaps the front matter leaves.
+"""
+
+import logging
+
+import yaml
+
+# Parts of a myst.yml affiliation, joined into one name string. Department
+# precedes institution to match the convention in existing NeuroLibre front
+# matter.
+NAME_PARTS = (
+    "department",
+    "institution",
+    "address",
+    "city",
+    "region",
+    "postal_code",
+    "country",
+)
+
+# MyST accepts these aliases for two of the parts.
+ALIASES = {"institution": "name", "region": "state"}
+
+
+def _is_blank(value):
+    """Is a value absent, or present but carrying nothing?
+
+    A `paper.md` front matter of `title:` parses to `title: None`, not to a
+    missing key. `""`, `[]` and `{}` say the same thing. All of them must count
+    as absent or a key that was merely typed out defeats the myst.yml fallback.
+    Mirrors `is_blank` in inara's myst-frontmatter.lua.
+    """
+    if value is None:
+        return True
+    if isinstance(value, str):
+        return not value.strip()
+    if isinstance(value, (list, tuple, dict, set)):
+        return len(value) == 0
+    return False
+
+
+def _as_list(value):
+    """Normalise a myst.yml sequence to a list.
+
+    `affiliations: harvard` is legal MyST. Without this, iterating the string
+    would walk its characters. Mirrors `as_list` in myst-frontmatter.lua.
+    """
+    if value is None:
+        return []
+    if isinstance(value, (list, tuple)):
+        return list(value)
+    return [value]
+
+
+def _affiliation_name(affiliation):
+    """Join an affiliation's parts into a single display string."""
+    parts = []
+    for key in NAME_PARTS:
+        value = affiliation.get(key)
+        if value in (None, "") and key in ALIASES:
+            value = affiliation.get(ALIASES[key])
+        if value not in (None, ""):
+            parts.append(str(value).strip())
+    return ", ".join(parts)
+
+
+def _affiliation_tokens(value):
+    """Normalise an author's `affiliations` value to a list of tokens.
+
+    MyST accepts a list, a single id, or several ids in one ';'-separated
+    string.
+    """
+    if value in (None, ""):
+        return []
+    if isinstance(value, (list, tuple)):
+        return [str(entry).strip() for entry in value if str(entry).strip()]
+    return [token.strip() for token in str(value).split(";") if token.strip()]
+
+
+def myst_project_metadata(project):
+    """Return inara paper metadata derived from a myst.yml `project` mapping.
+
+    Returns only the keys the project actually defines, so the caller can treat
+    the result as a set of defaults to fill gaps with. Junk input yields an
+    empty dict: this fallback must never be why a deposit fails.
+    """
+    if not isinstance(project, dict):
+        return {}
+
+    metadata = {}
+
+    if not _is_blank(project.get("title")):
+        metadata["title"] = project["title"]
+    if not _is_blank(project.get("date")):
+        # `date: 2024-01-15` -- unquoted ISO, the MyST-canonical form -- is
+        # parsed by yaml.safe_load into a datetime.date. This value is carried
+        # into a Celery task payload, which is serialized as JSON, so a date
+        # object here is an HTTP 500 at enqueue time. Nothing downstream
+        # consumes `date` structurally, so the string form is the right shape.
+        date = project["date"]
+        metadata["date"] = date if isinstance(date, str) else str(date)
+    if not _is_blank(project.get("keywords")):
+        metadata["tags"] = project["keywords"]
+    if not _is_blank(project.get("bibliography")):
+        metadata["bibliography"] = project["bibliography"]
+
+    affiliations = []
+    index_of = {}
+    for source in _as_list(project.get("affiliations")):
+        index = len(affiliations) + 1
+        if not isinstance(source, dict):
+            # MyST's validator accepts a bare string where an affiliation
+            # mapping is expected. It becomes an affiliation named after that
+            # string, with no id, and it still consumes its index position --
+            # the Lua filter applies the same rule, so both sides agree on
+            # every author's index.
+            affiliations.append({"index": index, "name": str(source).strip()})
+            continue
+        affiliations.append({"index": index, "name": _affiliation_name(source)})
+        if source.get("id") is not None:
+            index_of[str(source["id"])] = index
+
+    authors = []
+    for source in _as_list(project.get("authors")):
+        if not isinstance(source, dict):
+            # Same MyST rule for authors: `authors: [Ada Lovelace]` is valid.
+            # A bare string becomes a named author with no affiliations, still
+            # holding its position in the list.
+            authors.append({"name": str(source).strip()})
+            continue
+        author = {"name": source.get("name")}
+        for target, key in (
+            ("email", "email"),
+            ("orcid", "orcid"),
+            ("corresponding", "corresponding"),
+            ("equal-contrib", "equal_contributor"),
+        ):
+            if source.get(key) is not None:
+                author[target] = source[key]
+
+        indices = []
+        tokens = _affiliation_tokens(
+            source.get("affiliations") or source.get("affiliation")
+        )
+        for token in tokens:
+            index = index_of.get(token)
+            if index is None:
+                # MyST permits ad-hoc affiliations. Inventing an entry beats
+                # dropping the author's affiliation.
+                index = len(affiliations) + 1
+                affiliations.append({"index": index, "name": token})
+                index_of[token] = index
+            indices.append(str(index))
+        if indices:
+            author["affiliation"] = ",".join(indices)
+
+        authors.append(author)
+
+    if authors:
+        metadata["authors"] = authors
+    if affiliations:
+        metadata["affiliations"] = affiliations
+
+    return metadata
+
+
+def merge_paper_metadata(front_matter, myst_text):
+    """Paper metadata from paper.md, with myst.yml filling any gaps.
+
+    `front_matter` is the already-parsed paper.md front matter, or None for a
+    paper that has none. `myst_text` is the raw contents of myst.yml, or None.
+    Parsing myst.yml happens here rather than in the caller so that a malformed
+    file is tolerated in one place.
+
+    Returns None when neither source names any author — the same signal the
+    deposit path already treats as "cannot extract metadata".
+    """
+    metadata = dict(front_matter) if isinstance(front_matter, dict) else {}
+
+    if myst_text:
+        try:
+            project = (yaml.safe_load(myst_text) or {}).get("project")
+        except yaml.YAMLError as error:
+            logging.warning(f"Could not parse myst.yml: {error}")
+            project = None
+        except AttributeError:
+            # yaml.safe_load returned something that is not a mapping.
+            project = None
+        fallback = myst_project_metadata(project)
+
+        # Authors and affiliations are filled as a pair. An affiliation index
+        # only means something relative to the list that defines it, so mixing
+        # the two sources would silently attach authors to the wrong
+        # institutions.
+        #
+        # A key that is present but empty counts as absent -- see `_is_blank`.
+        if _is_blank(metadata.get("authors")) or _is_blank(metadata.get("affiliations")):
+            if fallback.get("authors"):
+                metadata["authors"] = fallback["authors"]
+                metadata["affiliations"] = fallback.get("affiliations", [])
+        for key in ("title", "date", "tags", "bibliography"):
+            if _is_blank(metadata.get(key)) and key in fallback:
+                metadata[key] = fallback[key]
+
+    if not metadata.get("authors"):
+        return None
+    return metadata
+
+
+def first_affiliations(authors, affiliations):
+    """Resolve each author's first affiliation to a display name.
+
+    `authors` is a list of author dicts as produced by `merge_paper_metadata`
+    (or a hand-written paper.md front matter); each may carry an `affiliation`
+    value that is an int, a comma-separated string of indices, an empty
+    string, or absent entirely. `affiliations` is the corresponding list of
+    `{"index": ..., "name": ...}` mappings.
+
+    Returns a list the same length as `authors`. An element is `None` when the
+    author has no affiliation, or names an index the affiliation list does not
+    define -- both are legitimate, not errors: myst.yml permits an author with
+    no affiliation (see `test_author_without_affiliations_gets_no_affiliation_key`),
+    and a caller should not have the deposit fail just because one author
+    lacks one.
+
+    An empty `affiliations` list is legitimate too -- a myst.yml project may name
+    authors and no institutions at all -- and resolves every author to `None`.
+    """
+    mapping = {
+        str(affiliation["index"]): affiliation["name"]
+        for affiliation in affiliations or []
+    }
+
+    resolved = []
+    for author in authors:
+        affiliation = author.get("affiliation")
+        if not affiliation:
+            resolved.append(None)
+            continue
+        if isinstance(affiliation, int):
+            affiliation_index = affiliation
+        else:
+            affiliation_indices = [affiliation_index for affiliation_index in str(affiliation).split(",")]
+            affiliation_index = affiliation_indices[0]
+        name = mapping.get(str(affiliation_index))
+        if name is None:
+            # A typo'd index used to crash loudly; now it silently records a
+            # creator with no institution. Say so, so it is diagnosable.
+            logging.warning(
+                f"Affiliation index {affiliation_index!r} for author "
+                f"{author.get('name')!r} is not defined by the affiliation "
+                f"list; recording no affiliation for this author."
+            )
+        resolved.append(name)
+
+    return resolved