Skip to content

Commit 7989827

Browse files
wietzesuijkerwietzesuijker
committed
conformance test datasets: 16 Zarr v3 stores for v1 conventions
12 valid + 4 invalid datasets covering proj:, spatial:, and multiscales conventions. Includes generation script, JSON schema validation, and GDAL cross-implementation tests.
1 parent 1618b5d commit 7989827

70 files changed

Lines changed: 4530 additions & 0 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎test-datasets/README.md‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
# GeoZarr Conformance Test Datasets
2+
3+
16 Zarr v3 stores (12 valid, 4 invalid) for testing against the v1 convention schemas:
4+
[proj:](https://github.com/zarr-conventions/geo-proj/blob/v1/README.md) |
5+
[spatial:](https://github.com/zarr-conventions/spatial/blob/v1/README.md) |
6+
[multiscales](https://github.com/zarr-conventions/multiscales/blob/v1/README.md)
7+
8+
> **Note**: `proj:` CMO URLs use `zarr-experimental` to match upstream [geo-proj schema.json](https://github.com/zarr-conventions/geo-proj/blob/v1/schema.json) `const` values. Will update when [#126](https://github.com/zarr-developers/geozarr-spec/issues/126) resolves.
9+
10+
```bash
11+
python scripts/generate_datasets.py --verify # generate + self-check
12+
python scripts/validate_schema.py # validate against upstream JSON schemas
13+
python scripts/validate_gdal.py # cross-check with GDAL >= 3.13
14+
```
15+
16+
Requires: `zarr>=3.0`, `numpy`. Schema validation requires `jsonschema`. GDAL validation requires GDAL >= 3.13.
17+
18+
## Datasets
19+
20+
| Dataset | What to check |
21+
|---|---|
22+
| `crs-epsg-4326` | Resolve `proj:code` EPSG code to CRS |
23+
| `crs-wkt2` | Parse `proj:wkt2` string (UTM 10N) |
24+
| `crs-projjson` | Parse `proj:projjson` object (Web Mercator) |
25+
| `crs-override` | Group=4326, band2=32632 - array-level overrides group |
26+
| `spatial-north-up` | Affine `[1.25, 0, 0, 0, -1.25, 10]` with bbox |
27+
| `spatial-rotated` | 30-degree rotation in UTM, omits `transform_type` and `bbox` (both optional) |
28+
| `spatial-pixel-is-point` | `registration: "node"` - 9x9 for 8-unit extent (N+1 pixels) |
29+
| `spatial-registration-default` | Field omitted - readers must default to `"pixel"` |
30+
| `spatial-multiband` | Shape (3,8,8), `spatial:shape`=[8,8] identifies spatial axes |
31+
| `multiscales-2-levels` | Level 1 = 2x block-average of level 0 (verifiable) |
32+
| `multiscales-sentinel2` | All conventions composed - 10m/20m/60m Sentinel-2 layout |
33+
| `nodata-float32-nan` | `fill_value: NaN` with actual NaN in border rows |
34+
| **`invalid-bad-crs`** | **`epsg:4326` fails `^[A-Z]+:[0-9]+$` - reject** |
35+
| **`invalid-missing-required`** | **`spatial:dimensions` missing - reject** |
36+
| **`invalid-multiscales-no-transform`** | **`derived_from` without `transform` - reject** |
37+
| **`invalid-multiple-crs`** | **Both `proj:code` and `proj:wkt2` (oneOf) - reject** |
38+
39+
Transform order is rasterio/Affine `[a, b, c, d, e, f]`, **not** GDAL GeoTransform.
40+
41+
## For implementers
42+
43+
**Start here**: `crs-epsg-4326` + `spatial-north-up` - the required fields for the most common case.
44+
45+
**Full conformance**: 12 valid pass without error, 4 invalid rejected with diagnostics.
46+
47+
[`manifest.json`](manifest.json) has expected values per dataset for CI. All scripts use exit codes: 0 = pass, 1 = fail, 2 = skip (missing dependency).
48+
49+
Binary Zarr chunks are committed so `git clone` gives working datasets without running Python. To regenerate: `python scripts/generate_datasets.py --verify`.

‎test-datasets/invalid/invalid-bad-crs.zarr/ar/c/0/0‎

25 Bytes
Binary file not shown.

‎test-datasets/invalid/invalid-bad-crs.zarr/ar/zarr.json‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
{
2+
"shape": [
3+
4,
4+
4
5+
],
6+
"data_type": "uint8",
7+
"chunk_grid": {
8+
"name": "regular",
9+
"configuration": {
10+
"chunk_shape": [
11+
4,
12+
4
13+
]
14+
}
15+
},
16+
"chunk_key_encoding": {
17+
"name": "default",
18+
"configuration": {
19+
"separator": "/"
20+
}
21+
},
22+
"fill_value": 0,
23+
"codecs": [
24+
{
25+
"name": "bytes"
26+
},
27+
{
28+
"name": "zstd",
29+
"configuration": {
30+
"level": 0,
31+
"checksum": false
32+
}
33+
}
34+
],
35+
"attributes": {
36+
"zarr_conventions": [
37+
{
38+
"schema_url": "https://raw.githubusercontent.com/zarr-experimental/geo-proj/refs/tags/v1/schema.json",
39+
"spec_url": "https://github.com/zarr-experimental/geo-proj/blob/v1/README.md",
40+
"uuid": "f17cb550-5864-4468-aeb7-f3180cfb622f",
41+
"name": "proj:",
42+
"description": "Coordinate reference system information for geospatial data"
43+
}
44+
],
45+
"proj:code": "epsg:4326"
46+
},
47+
"dimension_names": [
48+
"y",
49+
"x"
50+
],
51+
"zarr_format": 3,
52+
"node_type": "array",
53+
"storage_transformers": []
54+
}

‎test-datasets/invalid/invalid-bad-crs.zarr/zarr.json‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
{
2+
"attributes": {},
3+
"zarr_format": 3,
4+
"node_type": "group",
5+
"consolidated_metadata": {
6+
"kind": "inline",
7+
"must_understand": false,
8+
"metadata": {
9+
"ar": {
10+
"shape": [
11+
4,
12+
4
13+
],
14+
"data_type": "uint8",
15+
"chunk_grid": {
16+
"name": "regular",
17+
"configuration": {
18+
"chunk_shape": [
19+
4,
20+
4
21+
]
22+
}
23+
},
24+
"chunk_key_encoding": {
25+
"name": "default",
26+
"configuration": {
27+
"separator": "/"
28+
}
29+
},
30+
"fill_value": 0,
31+
"codecs": [
32+
{
33+
"name": "bytes"
34+
},
35+
{
36+
"name": "zstd",
37+
"configuration": {
38+
"level": 0,
39+
"checksum": false
40+
}
41+
}
42+
],
43+
"attributes": {
44+
"zarr_conventions": [
45+
{
46+
"schema_url": "https://raw.githubusercontent.com/zarr-experimental/geo-proj/refs/tags/v1/schema.json",
47+
"spec_url": "https://github.com/zarr-experimental/geo-proj/blob/v1/README.md",
48+
"uuid": "f17cb550-5864-4468-aeb7-f3180cfb622f",
49+
"name": "proj:",
50+
"description": "Coordinate reference system information for geospatial data"
51+
}
52+
],
53+
"proj:code": "epsg:4326"
54+
},
55+
"dimension_names": [
56+
"y",
57+
"x"
58+
],
59+
"zarr_format": 3,
60+
"node_type": "array",
61+
"storage_transformers": []
62+
}
63+
}
64+
}
65+
}

‎test-datasets/invalid/invalid-missing-required.zarr/ar/c/0/0‎

25 Bytes
Binary file not shown.

‎test-datasets/invalid/invalid-missing-required.zarr/ar/zarr.json‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
{
2+
"shape": [
3+
4,
4+
4
5+
],
6+
"data_type": "uint8",
7+
"chunk_grid": {
8+
"name": "regular",
9+
"configuration": {
10+
"chunk_shape": [
11+
4,
12+
4
13+
]
14+
}
15+
},
16+
"chunk_key_encoding": {
17+
"name": "default",
18+
"configuration": {
19+
"separator": "/"
20+
}
21+
},
22+
"fill_value": 0,
23+
"codecs": [
24+
{
25+
"name": "bytes"
26+
},
27+
{
28+
"name": "zstd",
29+
"configuration": {
30+
"level": 0,
31+
"checksum": false
32+
}
33+
}
34+
],
35+
"attributes": {
36+
"zarr_conventions": [
37+
{
38+
"schema_url": "https://raw.githubusercontent.com/zarr-conventions/spatial/refs/tags/v1/schema.json",
39+
"spec_url": "https://github.com/zarr-conventions/spatial/blob/v1/README.md",
40+
"uuid": "689b58e2-cf7b-45e0-9fff-9cfc0883d6b4",
41+
"name": "spatial:",
42+
"description": "Spatial coordinate information"
43+
}
44+
],
45+
"spatial:transform": [
46+
1.0,
47+
0.0,
48+
0.0,
49+
0.0,
50+
-1.0,
51+
4.0
52+
]
53+
},
54+
"dimension_names": [
55+
"y",
56+
"x"
57+
],
58+
"zarr_format": 3,
59+
"node_type": "array",
60+
"storage_transformers": []
61+
}

‎test-datasets/invalid/invalid-missing-required.zarr/zarr.json‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
{
2+
"attributes": {},
3+
"zarr_format": 3,
4+
"node_type": "group",
5+
"consolidated_metadata": {
6+
"kind": "inline",
7+
"must_understand": false,
8+
"metadata": {
9+
"ar": {
10+
"shape": [
11+
4,
12+
4
13+
],
14+
"data_type": "uint8",
15+
"chunk_grid": {
16+
"name": "regular",
17+
"configuration": {
18+
"chunk_shape": [
19+
4,
20+
4
21+
]
22+
}
23+
},
24+
"chunk_key_encoding": {
25+
"name": "default",
26+
"configuration": {
27+
"separator": "/"
28+
}
29+
},
30+
"fill_value": 0,
31+
"codecs": [
32+
{
33+
"name": "bytes"
34+
},
35+
{
36+
"name": "zstd",
37+
"configuration": {
38+
"level": 0,
39+
"checksum": false
40+
}
41+
}
42+
],
43+
"attributes": {
44+
"zarr_conventions": [
45+
{
46+
"schema_url": "https://raw.githubusercontent.com/zarr-conventions/spatial/refs/tags/v1/schema.json",
47+
"spec_url": "https://github.com/zarr-conventions/spatial/blob/v1/README.md",
48+
"uuid": "689b58e2-cf7b-45e0-9fff-9cfc0883d6b4",
49+
"name": "spatial:",
50+
"description": "Spatial coordinate information"
51+
}
52+
],
53+
"spatial:transform": [
54+
1.0,
55+
0.0,
56+
0.0,
57+
0.0,
58+
-1.0,
59+
4.0
60+
]
61+
},
62+
"dimension_names": [
63+
"y",
64+
"x"
65+
],
66+
"zarr_format": 3,
67+
"node_type": "array",
68+
"storage_transformers": []
69+
}
70+
}
71+
}
72+
}

‎test-datasets/invalid/invalid-multiple-crs.zarr/ar/c/0/0‎

25 Bytes
Binary file not shown.

‎test-datasets/invalid/invalid-multiple-crs.zarr/ar/zarr.json‎

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
{
2+
"shape": [
3+
4,
4+
4
5+
],
6+
"data_type": "uint8",
7+
"chunk_grid": {
8+
"name": "regular",
9+
"configuration": {
10+
"chunk_shape": [
11+
4,
12+
4
13+
]
14+
}
15+
},
16+
"chunk_key_encoding": {
17+
"name": "default",
18+
"configuration": {
19+
"separator": "/"
20+
}
21+
},
22+
"fill_value": 0,
23+
"codecs": [
24+
{
25+
"name": "bytes"
26+
},
27+
{
28+
"name": "zstd",
29+
"configuration": {
30+
"level": 0,
31+
"checksum": false
32+
}
33+
}
34+
],
35+
"attributes": {
36+
"zarr_conventions": [
37+
{
38+
"schema_url": "https://raw.githubusercontent.com/zarr-experimental/geo-proj/refs/tags/v1/schema.json",
39+
"spec_url": "https://github.com/zarr-experimental/geo-proj/blob/v1/README.md",
40+
"uuid": "f17cb550-5864-4468-aeb7-f3180cfb622f",
41+
"name": "proj:",
42+
"description": "Coordinate reference system information for geospatial data"
43+
}
44+
],
45+
"proj:code": "EPSG:32610",
46+
"proj:wkt2": "PROJCRS[\"WGS 84 / UTM zone 10N\",BASEGEOGCRS[\"WGS 84\",DATUM[\"World Geodetic System 1984\",ELLIPSOID[\"WGS 84\",6378137,298.257223563]],ID[\"EPSG\",4326]],CONVERSION[\"UTM zone 10N\",METHOD[\"Transverse Mercator\"],PARAMETER[\"Latitude of natural origin\",0],PARAMETER[\"Longitude of natural origin\",-123],PARAMETER[\"Scale factor at natural origin\",0.9996],PARAMETER[\"False easting\",500000],PARAMETER[\"False northing\",0]],CS[Cartesian,2],AXIS[\"easting\",east],AXIS[\"northing\",north],UNIT[\"metre\",1],ID[\"EPSG\",32610]]"
47+
},
48+
"dimension_names": [
49+
"y",
50+
"x"
51+
],
52+
"zarr_format": 3,
53+
"node_type": "array",
54+
"storage_transformers": []
55+
}

0 commit comments

Comments
 (0)