Correct metadata emission for generation > 1 and appending buffers - #30
Merged
Conversation
…ng buffers Ports the substantive fixes from vEcoli PR #414 commit f62a183a, reconciled against viva-emitters' diverged writer/transducer. Two bugs, both invisible on a single-buffer generation-1 run: 1. generation > 1 arrays got Zarr's default chunking. Coordinate-data emission and *encoding* computation were coupled under one flag (`include_static`, true only for the first buffer of generation 1), so every generation > 1 partition's freshly created arrays were written with no encoding. Decouple into two predicates on AsyncBufferWriter: `is_1st_buf_in_lineage` gates the lineage-shared coordinate DATA, while `is_1st_buf_in_generation` gates ENCODING computation — recomputed on the first buffer of every generation, so gen>1 arrays get their intended `b * buf_size` chunks. 2. child-variable unit attributes were mis-keyed and erased. The unit was stored under the per-generation `gen=N` key instead of the variable's own name (VariableSpec.attr_name); and appending buffers reassembled the child node from data variables only (no attrs), so dump_to_store erased the units the first buffer wrote. render() now re-attaches child_coords[p].attrs on every appending buffer. Mechanics: split XarrayBuffer.render (returns the DataTree) from a new XarrayBuffer.encodings; VariableSpec.encoding takes include_coo and puts explicit chunks on the right of the codec union so they always win. XarrayTransducer.flush derives both predicates from the writer instead of taking include_static. close() keeps its `num_writes > 0` guard to avoid a behavior change against viva's mark_success() num_writes=-1 sentinel. Skipped from f62a183a: the explicit coo/var default-codec classmethods (Fix 6, depends on zarr-internal symbols) and vEcoli-specific test changes. Fixes already present in viva (single-lookup merge_attributes; alloc() invariant via check_layout) are left as-is. New test tests/test_xarray_multigen_encoding.py drives two colony generations with a unit-bearing view and buffers_per_chunk=2; verified to FAIL on the pre-fix code (time chunk 3 != 6; unit absent) and pass after. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This was referenced Aug 21, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ports the substantive fixes from vEcoli PR #414 commit
f62a183a("Fixes to Xarray/Zarr metadata emission"), reconciled against viva-emitters' diverged writer/transducer.Two bugs (both invisible on a single-buffer, generation-1 run)
1.
generation > 1arrays got Zarr's default chunking. Coordinate-data emission and encoding computation were coupled under one flag (include_static, true only for the first buffer of generation 1). So everygeneration > 1partition's freshly created arrays were written with no encoding and fell back to default chunks.Fix: decouple into two predicates on
AsyncBufferWriter—is_1st_buf_in_lineagegates the lineage-shared coordinate data (written once),is_1st_buf_in_generationgates encoding computation (recomputed on the first buffer of every generation), so gen>1 arrays get their intendedb * buf_sizechunks.2. Child-variable unit attributes were mis-keyed and erased. The unit was stored under the per-generation
gen=Nkey instead of the variable's own name; and appending buffers reassembled the child node from data variables only (no attrs), sodump_to_storeerased the units. Fix:VariableSpec.attr_namekeys the unit by variable name, andrender()re-attacheschild_coords[p].attrson every appending buffer.Mechanics
XarrayBuffer.render(returns theDataTree) from newXarrayBuffer.encodings.VariableSpec.encodingtakesinclude_coo; explicitchunkson the right of the codec union so they always win.XarrayTransducer.flushderives both predicates from the writer.close()keeps itsnum_writes > 0guard to avoid a behavior change against viva'smark_success()num_writes = -1sentinel.Skipped from
f62a183a: the explicit coo/var default-codec classmethods (Fix 6 — depends on zarr-internal symbols; revisit with the zarr 3.3 bump) and vEcoli-specific test changes. Fixes viva already has (single-lookupmerge_attributes;alloc()invariant viacheck_layout) are left as-is.Verification
tests/test_xarray_multigen_encoding.pydrives two colony generations with a unit-bearing view andbuffers_per_chunk=2. Verified to FAIL on the pre-fix code (time chunk 3 != 6; unit absent) and pass after. Full suite: 109 passed, 4 skipped.Part of the PR #414 reconciliation (follows the buffer-sizing PR #29).
🤖 Generated with Claude Code