Add qlever check-sync-with-wikidata - #308
Open
hannahbast wants to merge 8 commits into
Open
Conversation
New command that checks whether entities on the endpoint are exactly in sync with wikidata.org, by comparing each entity against the canonical data from `Special:EntityData` (which renders the live revision on demand; deliberately NOT against the WDQS SPARQL endpoint, which is a replica with its own lag and no longer contains the scholarly subgraph). See the command's help text and the comments for the details.
There was a problem hiding this comment.
Pull request overview
Adds a new qlever check-sync-with-wikidata CLI command to verify that entities served by a QLever Wikidata endpoint exactly match the canonical data from Special:EntityData, with support for retrying on version mismatches and optional munging via a local service-*/munge.sh.
Changes:
- Introduces entity-level sync checking by fetching canonical TTL, querying the endpoint for the full entity “document”, and comparing normalized triples for exact equality.
- Adds configurable sampling (
--num-entities,--recent-fraction,--seed) and manual selection (--entities) of entities to check. - Adds optional canonical-data munging (
--munge) and optional artifact retention (--keep-files) for inspection/debugging.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+76
to
+79
| subparser.add_argument( | ||
| "--sparql-endpoint", | ||
| help="URL of the QLever server, default is {host_name}:{port}", | ||
| ) |
Comment on lines
+517
to
+528
| keep_dir = Path.cwd() if args.keep_files else None | ||
| if args.entities: | ||
| entities = args.entities.split(",") | ||
| else: | ||
| log.info( | ||
| f"Sampling {args.num_entities} entities" | ||
| f" ({args.recent_fraction:.0%} recently edited," | ||
| f" seed {args.seed}) ..." | ||
| ) | ||
| entities = self.sample_entities( | ||
| args.num_entities, args.recent_fraction, args.seed | ||
| ) |
Comment on lines
+484
to
+487
| for line in sorted(missing)[:5]: | ||
| log.error(f" missing: {line}") | ||
| for line in sorted(extra)[:5]: | ||
| log.error(f" extra: {line}") |
Comment on lines
+493
to
+499
| def execute(self, args) -> bool: | ||
| endpoint = ( | ||
| args.sparql_endpoint | ||
| if args.sparql_endpoint | ||
| else f"http://{args.host_name}:{args.port}" | ||
| ) | ||
| munge_scripts = sorted(glob.glob("service-*/munge.sh")) |
Move all imports to the top of the module (as in the other commands), deduplicate the version extraction into `entity_version`, correct the help text of `--sparql-endpoint`, validate `--entities` and `--recent-fraction`, add a timeout for `munge.sh`, and add unit tests for the deterministic parts (normalization, numeric and geographic canonicalization, version extraction).
A 100-entity random-sample run found a value where the export of the index differs from the canonical lexical form in the 12th significant digit (168.73846826 vs 168.738468261): the encoding used by the index rounds slightly differently than IEEE string parsing. Round to 10 significant digits, which is safely below such differences and still far more precision than any real divergence would survive.
Each run of `munge.sh` starts a JVM, which took 3-4 of the ~6 seconds per checked entity. Munge a whole batch (default: 50 entities) in one run instead: the canonical documents are downloaded and the endpoint is queried pairwise as before (the snapshot semantics of the version gate do not change), the concatenation of the downloaded documents is munged in one go, and `extract_document` then reconstructs the entity boundaries from the munged output, mirroring exactly the queries used on the endpoint side. This brings the cost down to ~2 seconds per entity, dominated by the polite pacing of the downloads.
The 1000-entity run found a coordinate where the export of the index differs from the canonical form by 1e-6 degrees (5.483421 came back as 5.48342), so rounding to 6 decimal places is not enough.
The second 1000-entity run found a coordinate at a rounding boundary (50.81588 exported as 50.81587), where rounding to any fixed number of decimal places compares unequal. Exclude the geographic values from the exact set comparison (their triples still take part via a placeholder, so a missing or extra triple is still detected exactly) and compare the values separately with a tolerance of 2e-5.
Two findings from the third 1000-entity run: 1. QLever's export omits the timezone designator for dates with years outside [-9999, 9999] (`-11700-01-01T00:00:00` instead of `-11700-01-01T00:00:00Z`), so dates are now compared without it (all times in Wikidata are UTC, so it carries no information here). 2. A recently merged entity is served by `Special:EntityData` as a redirect stub without an HTTP redirect. Detect this at download time and report the entity as a redirect. The stub must in particular not go into the munged batch, because its dangling document node makes `munge.sh` graft its version onto the NEXT entity of the batch, which then reports a spurious version mismatch. As a second line of defense, an `owl:sameAs` on either side is also detected at comparison time.
A 1000-entity run hit a single transient HTTP 500 from wikidata.org on one download, which was counted as an error for that entity. Retry HTTP 5xx once (after a short pause); client errors are not retried.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
New command that checks whether the entities on a QLever wikidata endpoint are exactly in sync with wikidata.org. It is the natural companion of
qlever update-wikidata: one command keeps the endpoint in sync, the other verifies that it actually is.For each checked entity, the command downloads the canonical data from
Special:EntityData(which renders the live revision on demand), immediately queries the endpoint for the full entity document (the triples with the entity as subject, the statement nodes with their references and values, the sitelink article blocks, and the wiki metadata), and then compares the two for exact equality. The comparison is deliberately NOT against the WDQS SPARQL endpoint, which is a replica with its own lag and no longer contains the scholarly subgraph.Two key elements make an exact comparison possible:
Version gate: the canonical data contains the revision ID (
schema:version). If it matches the entity's revision on the endpoint, both sides render the same revision and every difference is a real divergence (zero tolerance). If it does not match, the entity was edited after the endpoint's position in the update stream; the check retries once and otherwise reports the entity as undecidable (not as a failure).Munging: for an index built from the munged dump, the canonical data is passed through the
munge.shof theservice-*directory in the working directory, i.e., the exact munge version the index was built with (option--munge, default: munge when such a directory exists).A small number of documented normalizations remain: numeric literals are compared by value (the index stores values with limited precision, not lexical forms; integers are kept exact), geographic coordinates are rounded to the precision of QLever's fixed-precision encoding, and the
rdf:type wikibase:Referenceandwikibase:quantityNormalizedtriples are excluded (the munged dump does not contain them, but the update stream does, so an updated index has them for some references and not for others).Munging is done in batches (
--batch-size, default: 50 entities per run ofmunge.sh), which amortizes the JVM startup and brings the cost down to about 2 seconds per entity; the entity boundaries are reconstructed from the munged batch by the same reachability used for the queries against the endpoint.By default, the command checks 10 randomly sampled entities, half of them drawn from the recently edited entities (which exercise the update path), half uniformly at random (which exercise the index-build path); see
--num-entities,--recent-fraction,--seed, and--entitiesfor checking specific entities. The exit code is non-zero if and only if at least one entity is divergent, so the command can be used in automated checks.Example run against https://qlever.dev/api/wikidata (10 random entities, all exact matches, about 2 seconds per entity):