build(deps): bump nomenklatura from 4.10.0 to 4.14.0

[dependabot]

Bumps nomenklatura from 4.10.0 to 4.14.0.

Release notes

Sourced from nomenklatura's releases.

Nomenklatura 4.14.0

This release covers everything since 4.13.1, including the unreleased 4.13.2 tag. The headline is a new DuckDB-backed batch store built for read-heavy export workloads, the removal of the Redis store backends, and consistent handling of external (enrichment-candidate) statements across the SQL store, xref and the dedupe TUI.

DuckDB batch store

• New DuckDBBatchStore (#348), exported from nomenklatura.store: a materialized, read-heavy statement store on top of a caller-provided DuckDB relation (e.g. a view over parquet statement artifacts). Creating a view bakes the linker's resolution, the view's dataset scope and its external flag into a statement table sorted and indexed on canonical_id, plus a pre-resolved edge table — so the read loop pays index probes with no per-query dataset/external filters or join work. Compared to reading through a join view, single-entity reads drop 9× (11.1ms → 1.2ms) and batched reads reach raw-table speed (~0.07ms/entity), for a materialization cost of ~1s per few million statements. update() re-keys a cluster in every open view via SQL UPDATEs on the immutable source entity ids, making xref/dedupe decisions visible without a rebuild.
• View.entities() gains a prefetch_nested flag: the DuckDB batch view buffers 1,000 assembled entities per batch and bulk-loads their adjacency (inverted owners, forward values, and both directions of edge-schema neighbors) in four queries, replacing the per-entity point lookups that dominate export runtime. Ids with more than 1,000 owners fall back to the complete lazy path, so results are never truncated. Other store backends accept the flag as a no-op.
• The edge table is indexed on value_canonical_id, so lazy get_inverted() calls — hub fallbacks and non-scan callers like enrichers and xref — probe by equality instead of scanning the table on every call.
• New nomenklatura.duck module: the DuckDB counterpart to nomenklatura.db, carrying the typed statement-relation contract between producers (statement import jobs) and consumers (stores), a DESCRIBE-based validation helper, and a connection factory with the shared memory/thread policy. The blocker index migrates onto the factory, replacing its inline configuration block.

External statements

• Fixed SQLView ignoring the external flag (#360): the view accepted the flag but never applied it, so SQL-backed views always behaved as external=True, leaking enrichment-candidate statements into published-only reads. All four query paths now bind the predicate, the contract is stated on the base View, and the shared store test asserts the round-trip for every backend.
• xref skips pairs where both candidates are backed only by external statements (#358). An external-only entity is an enrichment suggestion that has not been through verification; comparing two of them cannot anchor anything into the graph, so the pair is worth neither a scoring run nor a slot in the review queue. The check lives in the pair loop rather than as a recorded judgement, because externality is transient: the same pair becomes worth scoring as soon as either side is verified.
• The dedupe TUI marks external candidates in the comparison header with a distinct schema-label colour and a trailing asterisk, so reviewers can tell published data from unconfirmed enrichment suggestions before judging.

Stores

• Removed the Redis and versioned-Redis store backends (#359). Neither had a production consumer; the KV niche is covered by LevelStore and batch workloads by the SQL and DuckDB stores. This drops the stores, the shared Redis connection helper (nomenklatura.kv), the NOMENKLATURA_REDIS_URL setting, and the redis/fakeredis dependencies.

Compatibility

• RedisStore and VersionedRedisStore are gone, along with the redis package extra and the NOMENKLATURA_REDIS_URL environment variable.
• The followthemoney floor moves up to 4.10.2, which introduces the entity-level external aggregate that the xref and TUI changes rely on.
• Consumers reading from a SQLStore view with external=False (the default) will no longer see external statements. That was always the documented contract; if you relied on the leak, pass external=True when creating the view.

Full changes: opensanctions/nomenklatura@4.13.1...4.14.0

Relevant pull requests: #348, #358, #359, #360.

Nomenklatura 4.13.1

This release covers everything since 4.12.0, including the unreleased 4.12.1–4.13.0 tags. The bulk of it is a rework of blocker candidate ranking so that rare tokens drive the ordering, a lossless CSV round-trip for resolver state, and substantially faster statement loading.

Blocking

• Blocker candidates are ranked by token rarity instead of raw term frequency (#349). Token weights are now presence-based (boost × IDF, smoothed so ubiquitous tokens keep their boost), name-part and symbol tokens are dampened by the square root of the entity's name count, and per pair and field the best shared token counts in full while additional ones earn only logarithmic credit. Previously a shared common forename scored the same as a shared distinctive company name, and every additional alias diluted the evidence of the name that actually matched: on the sanctions scope 3.5% of exact-name duplicate pairs never surfaced within a 1M-pair budget, the worst ranked at ~1M of 11.2M candidates. On the same scope every exact-name duplicate pair now ranks within the first ~21k candidates. pairs() and match_entities() share the weighting, and both orderings carry deterministic tie-breaks.
• Candidate truncation moved into the DuckDB query: each subject's list is cut to max_candidates and floored at min_score_ratio (default 0.1) of its best candidate's score, so rows discarded in Python are no longer sorted and shipped. Subjects present in the indexed view no longer receive themselves as candidates, and mid-chunk subjects no longer received max_candidates + 1 candidates. The relative floor replaces zavod's max_bin walk, whose rounded-absolute-score binning the rarity-based scoring broke.
• Removed match_batch chunking. Measured against a 273k-entity index with 50k subjects it produced identical candidate lists slightly slower, and DuckDB's own spilling handles constrained memory down to a 512MB limit. Tune NOMENKLATURA_DUCKDB_THREADS instead.
• Tokenizer fixes: matchable email values are now indexed (they fell through every type branch despite the index defining a 10.0 boost for the field), the 3–30 character gate on name parts checks the string actually emitted (part.comparable, not part.form), and address-derived tokens accumulate term frequency instead of being deduplicated per entity, so a word appearing in several of an entity's addresses outweighs an incidental one.

Resolver

• Fixed Resolver.dump() silently resurrecting reverted judgements: soft-deleted edges were written out, but the jsonl line format has no deleted_at field, so a dump/load roundtrip reinstated every judgement that had ever been reverted. Dumps are now also actually sorted by created_at.
• dump-resolver and load-resolver gain -f/--format {jsonl,csv}, defaulting to the existing jsonl. CSV rows carry a deleted_at column, so dump-resolver -f csv --include-deleted preserves tombstones through a roundtrip — this is now the lossless way to move resolver state between databases. The flag is rejected for jsonl, which cannot represent deleted edges.
• New nk dump-mapping command writes a two-column entity_id,canonical_id CSV for consumers that need a join table rather than the full edge history. Every known identifier gets a row, including canonicals themselves and stale intermediate canonicals absorbed by later merges, so any past reference resolves via a plain join.
• Serialization moved onto two format-agnostic primitives: Resolver.all_edges() yields edges in chronological order (with opt-in flags for soft-deleted edges and NO_JUDGEMENT suggestions) and Resolver.load_edges() registers a stream of edges. Edge.from_dict() now tolerates csv.DictReader rows, where every value is a string and missing values arrive as "".
• Indexed resolver.created_at and resolver.deleted_at to lighten the refresh-from-database path that runs on every decide().

Cross-referencing

... (truncated)

Commits

• 5624353 Bump version: 4.13.2 → 4.14.0
• 1ff5b70 Run CI once per PR: limit push triggers to main and tags
• 537cc78 Merge pull request #360 from opensanctions/pudo/sql-view-external
• d911295 Merge pull request #359 from opensanctions/pudo/remove-redis-stores
• c103023 Filter external statements in SQLView reads
• ef3e3f8 Remove the Redis and versioned-Redis store backends
• b330a67 Merge pull request #348 from opensanctions/pudo/duckdb-store
• 6b45f71 Index the edge table on value_canonical_id
• aa1ef7a Add chunked adjacency prefetch to the DuckDB batch store
• 7c6146b Replace the draft DuckDB store with a materialized batch store
• Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

---

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

• @dependabot rebase will rebase this PR
• @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
• @dependabot show <dependency name> ignore conditions will show all of the ignore conditions of the specified dependency
• @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	pypi/​nomenklatura@​4.10.0 ⏵ 4.14.0

View full report