[igounton]

The bug

Every NARA search fails. The adapter calls /api/v2/search with rows/offset and reads results[] — a shape the catalog no longer serves. That path now returns the catalog's HTML single-page app, so r.json() raises on the first request and the source contributes nothing to any corpus.

$ curl -s 'https://catalog.archives.gov/api/v2/search?q=apollo&rows=2'
<!doctype html><html lang="en"><head><script>var hostname=window.location.hostname…

Two things hid it. search() re-raises transport failures (correctly — that's the #511 contract), so the failure surfaces as a per-source error rather than a wrong result. And is_available() returned True unconditionally, so NARA counted toward the "configured sources" total on every install while being incapable of returning a clip.

The key was documented as optional — "NARA works without an API key. Set NARA_API_KEY in .env for higher rate limits." It isn't: without the x-api-key header the API serves HTML instead of JSON. The adapter now gates on NARA_API_KEY like the other key-backed sources, and the install text says why.

What changed

Corrected against the live API:

	before	after
endpoint	/api/v2/search	/api/v2/records/search
paging	rows + offset	limit + 1-based page
envelope	results[]	body.hits.hits[]._source.record
files	objects[].url	digitalObjects[].objectUrl
kind	mimeType	extension, objectType label as fallback
kind filter	type=moving-image	typeOfMaterials (the catalog's own facet)

offset, from and start are all silently ignored by v2 — only page moves the window, which is why the old call returned page 1 forever even when it did parse.

The kind filter now runs server-side using the catalog's own typeOfMaterials facet value, so we stop over-fetching and discarding: apollo goes from 47,075 hits to 3,970 for video. availableOnline=true is added too — a record with no digitised file can never yield a candidate.

Two honest gaps rather than invented data:

• v2 exposes no dimensions or duration, so width/height/duration stay 0, per the Candidate docstring ("fields that don't apply default to zero/empty"). Duration filters can only be applied downstream once a file is probed.
• scopeAndContentNote no longer exists. The description is spread across generalNotes (list), shotList (str) and subjects (list of dicts); _describe flattens them into the text channel used for ranking.

One quirk documented at the call site: result ordering is not stable across identical requests — relevance ties reshuffle — so consecutive pages can overlap or skip a record. De-dup on source_id if you walk several pages.

Verification

Against the live catalog with a real key:

is_available: True
video: 2 candidates
   video 87110_183896824    https://catalog.archives.gov/medialz/mopix/428/NPC/428-npc-42113.mp4
   video 102046893_400240282 https://catalog.archives.gov/medialz/mopix/111/DD/111-dd-219-69.mp4
image: 7 candidates
   image 6922343_16220695   https://catalog.archives.gov/medialive/43/9223/6922343/content/…

page=1/2/3 at limit=3 tile a single limit=9 call exactly. A returned .mp4 serves 206 video/mp4.

Five tests pin the endpoint, the paging params (including that offset is not sent), the server-side kind filter, the envelope walk and the tag flattening. The existing suite still passes — 40 passed, 3 xfailed in tests/tools/test_stock_source_adapters.py, with the pre-existing xfails untouched. The adapter keeps re-raising transport errors, so it stays out of _STILL_SWALLOWS_TRANSPORT_ERRORS.

🤖 Generated with Claude Code

https://claude.ai/code/session_01QsqAJj445U3Uj9gmDTfXXV

[calesthio]

Thank you for updating NARA's endpoint and record parsing. Reviewed the complete diff against main 63d1087a and checked the official NARA getting-started/API documentation: /api/v2/records/search, x-api-key, availableOnline and material filters agree with the published interface (https://www.archives.gov/research/catalog/help/api-getting-started ; https://www.archives.gov/research/catalog/help/api). Unknown dimensions/duration remain explicit rather than invented. Validation: adapter suite 40 passed, 3 expected failures for unrelated adapters. No credentials or live media downloads were used; mocked response-envelope coverage is not a live-service guarantee.