Skip to content

Commit 8bcfdca

Browse files
authored
feat(api): resolve a reference by work and locator without its UUID (#84) (#88)
Publish the reference lookup as data. Each work gets a locator index at /reg/work/{key}/aliases.json that maps a locator to its reference UUID, grouped by citation system key, so a client resolves a passage in two fetches and needs no UUIDv5 implementation of its own. Publish the complete alias table at /dump/aliases.json, and register it in datapackage.json with a byte count and a sha256 hash. data/README.md and data/.gitignore both named that artifact already; no code wrote it. Split writeDump() into pure halves so a plain JSON resource fits beside the four JSONL ones. Sort the alias keys, so the body and its hash depend on the registry content alone. Add the new file to the release workflow, which sets fail_on_unmatched_files. Correct the four contract statements that the new artifacts falsify: the Collections tag, the Bulk tag, the datapackage resource count, and the claim that every .json body is JSON-LD by content. Largest emitted index is tanakh at 1.12 MB, under the agreed 2 MB cap, so no pagination. All 67,959 references are reachable from their work index, and the four JSONL bodies are byte-identical to staging.
1 parent a4a3b85 commit 8bcfdca

13 files changed

Lines changed: 658 additions & 84 deletions

File tree

‎.github/workflows/release.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,7 @@ jobs:
3434
with:
3535
files: |
3636
dist/dump/*.jsonl
37+
dist/dump/aliases.json
3738
dist/dump/datapackage.json
3839
fail_on_unmatched_files: true
3940
generate_release_notes: true

‎AGENTS.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@ Husky installs on `npm install`: Prettier runs on staged files, `commitlint` enf
2424
- `src/content/docs/{association,standard,api,community,get-started}/` — site sections (autogenerated sidebar)
2525
- `src/content/docs/de/` — German mirror (see Localization)
2626
- `src/pages/id/{work,system,ref,mapping}/` — canonical record pages (`/id/.../` HTML and `.json` JSON-LD siblings)
27-
- `src/pages/reg/` — registry browser (index + paginated reference lists); does not host individual records
27+
- `src/pages/reg/` — registry browser (index + paginated reference lists), the registry-wide JSON-LD collections, and the per-work locator index (`/reg/work/{key}/aliases.json`); does not host individual records
2828
- `src/pages/cite/` — short-alias redirects (`/cite/{work}/{system}/{locator}` always; bare `/cite/{work}/{locator}` for a work's preferred citation system, and retargetable if that preference changes) → canonical `/id/` URL
2929
- `src/styles/brand.css` — brand tokens
3030
- `public/{logo.svg,favicon.svg,fonts/}` — brand assets

‎README.md‎

Lines changed: 13 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -71,19 +71,19 @@ Prerequisites: Node 24 and npm.
7171

7272
Configuration lives in `.env`; use [`.env.example`](./.env.example) as the starting point. `SITE_DOMAIN` controls Astro's canonical `site` URL and defaults to `textrefs.org` when unset.
7373

74-
| Command | Action |
75-
| :---------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
76-
| `npm install` | Install dependencies; wires git hooks (husky + lint-staged) |
77-
| `npm run dev` | Start local dev server at `localhost:4321` |
78-
| `npm run build` | Build the production site to `./dist/` |
79-
| `npm run build:fast` | Build the site against a tiny fixture registry, without compiling full data |
80-
| `npm run preview` | Preview the build locally |
81-
| `npm run compile:data` | Read hand-authored YAML under `data/works/` and `data/systems/`, expand the in-memory registry, and emit JSONL resources plus `datapackage.json` under `dist/dump/` |
82-
| `npm run validate:data` | Validate every compiled record against the canonical Zod schemas |
83-
| `npm run build:data` | `compile:data` then `validate:data` — the contributor data pipeline |
84-
| `npm run verify:fast` | Fast local check using fixture registry data |
85-
| `npm run verify` | Prettier + `astro check` + tests + production build — the CI gate |
86-
| `npm run changelog` | Regenerate `CHANGELOG.md` from git history (git-cliff) |
74+
| Command | Action |
75+
| :---------------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
76+
| `npm install` | Install dependencies; wires git hooks (husky + lint-staged) |
77+
| `npm run dev` | Start local dev server at `localhost:4321` |
78+
| `npm run build` | Build the production site to `./dist/` |
79+
| `npm run build:fast` | Build the site against a tiny fixture registry, without compiling full data |
80+
| `npm run preview` | Preview the build locally |
81+
| `npm run compile:data` | Read hand-authored YAML under `data/works/` and `data/systems/`, expand the in-memory registry, and emit JSONL resources, the alias table, and `datapackage.json` under `dist/dump/` |
82+
| `npm run validate:data` | Validate every compiled record against the canonical Zod schemas |
83+
| `npm run build:data` | `compile:data` then `validate:data` — the contributor data pipeline |
84+
| `npm run verify:fast` | Fast local check using fixture registry data |
85+
| `npm run verify` | Prettier + `astro check` + tests + production build — the CI gate |
86+
| `npm run changelog` | Regenerate `CHANGELOG.md` from git history (git-cliff) |
8787

8888
Contributors edit the YAML under [`data/works/`](https://github.com/textrefs/registry/tree/main/works) and [`data/systems/`](https://github.com/textrefs/registry/tree/main/systems); the directory is a git submodule pointing at [`textrefs/registry`](https://github.com/textrefs/registry). Run `git submodule update --init --recursive` after cloning. The compiler expands the pinned submodule into the flat registry dump (works, systems, refs, mappings) under `dist/dump/`. See [`docs/get-started/authoring`](https://textrefs.org/get-started/authoring/) for the format. For documentation, styling, and route work, use `npm run verify:fast` locally; run the full `npm run verify` before PRs that touch registry data, release output, production build behaviour, or CI behaviour.
8989

‎api/openapi.yaml‎

Lines changed: 99 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -23,16 +23,20 @@ info:
2323
yet live.
2424
2525
A client that does not yet know a key starts at `/reg/works.json` or
26-
`/reg/systems.json`. `/dump/` carries the whole registry as
27-
newline-delimited JSON for bulk consumers.
26+
`/reg/systems.json`. A client that knows a work key, a citation system
27+
key, and a locator reads the reference UUID from
28+
`/reg/work/{key}/aliases.json`. It needs no UUIDv5 implementation of its
29+
own. `/dump/` carries the whole registry as newline-delimited JSON, plus
30+
the whole alias table as one JSON object, for bulk consumers.
2831
2932
The site is static, and the host derives every `Content-Type` from the
3033
file extension. This contract therefore declares the media type that
3134
the host actually sends, not the media type that best describes the
32-
payload. A `.json` body is JSON-LD by content and arrives as
33-
`application/json`. A `.jsonl` body is newline-delimited JSON by
34-
content and arrives as `application/octet-stream`. Parse by
35-
documented shape, not by response header.
35+
payload. A record or collection `.json` body is JSON-LD by content and
36+
arrives as `application/json`. The two alias artifacts are plain JSON
37+
and arrive the same way. A `.jsonl` body is newline-delimited JSON by
38+
content and arrives as `application/octet-stream`. Parse by documented
39+
shape, not by response header.
3640
license:
3741
name: AGPL-3.0-or-later
3842
url: https://www.gnu.org/licenses/agpl-3.0.html
@@ -47,13 +51,15 @@ tags:
4751
description: Canonical record URLs under `/id/`.
4852
- name: Collections
4953
description:
50-
Registry-wide JSON-LD collections under `/reg/`. Entry points for a
51-
client that does not yet know a key.
54+
Registry-wide JSON-LD collections and per-work locator indexes under
55+
`/reg/`. Entry points for a client that knows no key, or that knows a
56+
locator but no reference UUID.
5257
- name: Bulk
5358
description:
5459
Whole-registry artifacts under `/dump/`, rebuilt on every deploy. The
55-
host serves them as `application/octet-stream`; the bodies are
56-
newline-delimited JSON.
60+
`.jsonl` bodies are newline-delimited JSON, and the host serves them
61+
as `application/octet-stream`. The `.json` bodies are single JSON
62+
objects, and the host serves them as `application/json`.
5763
paths:
5864
/id/work/{key}/:
5965
get:
@@ -200,6 +206,28 @@ paths:
200206
content:
201207
application/json:
202208
schema: { $ref: '#/components/schemas/SystemCollection' }
209+
/reg/work/{key}/aliases.json:
210+
get:
211+
tags: [Collections]
212+
summary: Work locator index — JSON
213+
operationId: getWorkAliasIndex
214+
description: Maps every locator of one work to the UUID of its
215+
CanonicalReference, grouped by citation system key. A client that
216+
knows a passage as work, citation system, and locator reads the UUID
217+
here, then fetches `/id/ref/{uuid}.json`. It needs no UUIDv5
218+
implementation of its own. The body is plain JSON, not JSON-LD, and
219+
carries no `@context`.
220+
parameters:
221+
- $ref: '#/components/parameters/FlatKey'
222+
responses:
223+
'200':
224+
description:
225+
Every reference of this work, in every status, keyed by citation
226+
system key and then by locator.
227+
content:
228+
application/json:
229+
schema: { $ref: '#/components/schemas/WorkAliasIndex' }
230+
'404': { $ref: '#/components/responses/NotFound' }
203231
/dump/works.jsonl:
204232
get:
205233
tags: [Bulk]
@@ -252,6 +280,23 @@ paths:
252280
content:
253281
application/octet-stream:
254282
schema: { type: string }
283+
/dump/aliases.json:
284+
get:
285+
tags: [Bulk]
286+
summary: Alias table — JSON
287+
operationId: getAliasesDump
288+
description:
289+
The complete alias table as one JSON object, with sorted keys and no
290+
indentation. For a single lookup, use
291+
`/reg/work/{key}/aliases.json` instead. This file serves a consumer
292+
that mirrors the whole table.
293+
responses:
294+
'200':
295+
description: One JSON object. Each key is an alias. Each value is the
296+
canonical IRI that the alias resolves to.
297+
content:
298+
application/json:
299+
schema: { $ref: '#/components/schemas/AliasTable' }
255300
/dump/datapackage.json:
256301
get:
257302
tags: [Bulk]
@@ -260,9 +305,10 @@ paths:
260305
responses:
261306
'200':
262307
description:
263-
The Frictionless data-package descriptor that lists the four
264-
JSONL resources with, for each, its byte count and its
265-
`sha256:` hash, plus the registry version.
308+
The Frictionless data-package descriptor that lists the five
309+
resources — the four JSONL files and the JSON alias table —
310+
with, for each, its byte count and its `sha256:` hash, plus the
311+
registry version.
266312
content:
267313
application/json:
268314
schema: { type: object }
@@ -481,6 +527,46 @@ components:
481527
'@graph':
482528
type: array
483529
items: { $ref: '#/components/schemas/CitationSystem' }
530+
WorkAliasIndex:
531+
description:
532+
The `/reg/work/{key}/aliases.json` locator index of one work. Plain
533+
JSON, not JSON-LD; it carries no `@context`. The index states
534+
identity alone. Read the CanonicalReference for the status and the
535+
resolver targets.
536+
type: object
537+
required: [work_key, preferred_citation_system_key, refs]
538+
properties:
539+
work_key: { $ref: '#/components/schemas/FlatKey' }
540+
preferred_citation_system_key:
541+
allOf:
542+
- $ref: '#/components/schemas/FlatKey'
543+
description:
544+
The system that the bare `/cite/{work_key}/{locator}` alias uses
545+
(ADR-0005). A default for presentation, never an identity claim.
546+
The other keys of `refs` have equal standing.
547+
refs:
548+
type: object
549+
description: One entry per citation system key that this work carries
550+
references under. An empty object means the work has no
551+
references yet. The same locator under two systems denotes a
552+
different passage, and a different UUID, each time.
553+
additionalProperties:
554+
type: object
555+
description: Locator to CanonicalReference UUID.
556+
additionalProperties:
557+
type: string
558+
pattern: '^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$'
559+
AliasTable:
560+
description:
561+
The `/dump/aliases.json` table, with one entry per alias. A key is
562+
either a `/cite/` alias path — `{work_key}/{locator}` or
563+
`{work_key}/{citation_system_key}/{locator}` — or an external
564+
identifier IRI carried by a MappingAssertion. A value is the
565+
canonical IRI that the key resolves to — an `/id/ref/` IRI for a
566+
`/cite/` alias, an `/id/work/` IRI for an external identifier. A
567+
`://` in the key marks the second kind.
568+
type: object
569+
additionalProperties: { $ref: '#/components/schemas/Iri' }
484570
RegistryObject:
485571
oneOf:
486572
- { $ref: '#/components/schemas/Work' }

‎scripts/compile.test.ts‎

Lines changed: 104 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,13 @@ import { mkdtempSync, mkdirSync, writeFileSync, rmSync } from 'node:fs';
1010
import { tmpdir } from 'node:os';
1111
import { join } from 'node:path';
1212
import { v5 as uuidv5 } from 'uuid';
13-
import { compileRegistry, type CompiledRegistry } from './compile.js';
13+
import { createHash } from 'node:crypto';
14+
import {
15+
compileRegistry,
16+
dumpResources,
17+
describeResource,
18+
type CompiledRegistry,
19+
} from './compile.js';
1420
import { LanguageTag } from '../standard/schema/common.js';
1521

1622
type RegistryFiles = {
@@ -671,3 +677,100 @@ test('regular grandfathered tags parse and irregular ones do not', () => {
671677
[],
672678
);
673679
});
680+
681+
// --- Dump resources (#84) -------------------------------------------------
682+
683+
// `workWithMappings` gives an alias table with both kinds of entry: `/cite/`
684+
// aliases targeting reference IRIs, and external identifiers targeting the work
685+
// IRI.
686+
const dumped = () => dumpResources(compileFixture(workWithMappings()));
687+
688+
const aliasSpec = (specs: ReturnType<typeof dumpResources>) => {
689+
const spec = specs.find((s) => s.name === 'aliases');
690+
assert.ok(spec, 'the dump carries an alias resource');
691+
return spec;
692+
};
693+
694+
test('the dump carries the alias table beside the four JSONL resources', () => {
695+
const specs = dumped();
696+
assert.deepEqual(
697+
specs.map((s) => s.name),
698+
['works', 'citation-systems', 'references', 'mappings', 'aliases'],
699+
);
700+
const spec = aliasSpec(specs);
701+
assert.equal(spec.filename, 'aliases.json');
702+
assert.equal(spec.format, 'json');
703+
assert.equal(spec.mediatype, 'application/json');
704+
});
705+
706+
test('the alias resource body is the whole alias table', () => {
707+
const reg = compileFixture(workWithMappings());
708+
const spec = aliasSpec(dumpResources(reg));
709+
const table = JSON.parse(spec.body) as Record<string, string>;
710+
assert.deepEqual(table, reg.aliases);
711+
// A `/cite/` alias resolves to a reference; an external identifier resolves
712+
// to the work. A `://` in the key marks the second kind.
713+
assert.match(
714+
table['test.work/primary-section/5'],
715+
/^https:\/\/textrefs\.org\/id\/ref\//,
716+
);
717+
assert.equal(
718+
table['https://www.wikidata.org/entity/Q1'],
719+
'https://textrefs.org/id/work/test.work',
720+
);
721+
});
722+
723+
test('alias keys are sorted, so the body does not depend on visit order', () => {
724+
const keys = Object.keys(JSON.parse(aliasSpec(dumped()).body));
725+
assert.ok(keys.length > 1);
726+
assert.deepEqual(keys, [...keys].sort());
727+
});
728+
729+
test('the alias body is minified and newline-terminated', () => {
730+
const body = aliasSpec(dumped()).body;
731+
assert.equal(body.split('\n').length, 2);
732+
assert.ok(body.startsWith('{"'));
733+
assert.ok(body.endsWith('}\n'));
734+
});
735+
736+
test('every descriptor states the bytes and sha256 of its own body', () => {
737+
for (const spec of dumped()) {
738+
const d = describeResource(spec);
739+
// Recomputed the way a downstream consumer verifies a download.
740+
assert.equal(d.bytes, Buffer.byteLength(spec.body, 'utf8'));
741+
assert.equal(
742+
d.hash,
743+
`sha256:${createHash('sha256').update(spec.body).digest('hex')}`,
744+
);
745+
}
746+
});
747+
748+
test('every descriptor carries the Frictionless fields', () => {
749+
for (const spec of dumped()) {
750+
const d = describeResource(spec);
751+
assert.equal(d.profile, 'data-resource');
752+
assert.equal(d.encoding, 'utf-8');
753+
assert.equal(d.path, spec.filename);
754+
assert.equal(d.name, spec.name);
755+
}
756+
});
757+
758+
test('the JSONL bodies are unchanged by the alias-resource refactor', () => {
759+
const reg = compileFixture(workWithMappings());
760+
const specs = dumpResources(reg);
761+
const bodyOf = (name: string) => specs.find((s) => s.name === name)?.body;
762+
assert.equal(
763+
bodyOf('works'),
764+
reg.works.map((r) => JSON.stringify(r)).join('\n') + '\n',
765+
);
766+
assert.equal(
767+
bodyOf('references'),
768+
reg.references.map((r) => JSON.stringify(r)).join('\n') + '\n',
769+
);
770+
// An empty record array still yields an empty body, not a bare newline.
771+
assert.equal(
772+
dumpResources({ ...reg, mappings: [] }).find((s) => s.name === 'mappings')
773+
?.body,
774+
'',
775+
);
776+
});

0 commit comments

Comments
 (0)