Add an optional compact JSON export (format_version 2), alongside the existing one - #1492
Open
masiarek wants to merge 1 commit into
Open
Add an optional compact JSON export (format_version 2), alongside the existing one#1492masiarek wants to merge 1 commit into
masiarek wants to merge 1 commit into
Conversation
The "Download JSON" button emits JSON.stringify({Election, Ballots,
Results}) of the raw in-memory objects, which puts the tabulator's
internal shape into a file people archive and parse: every candidate's
O(n^2) votesPreferredOver / winsAgainst maps keyed by UUID (self-pairs
included) repeated across elected/tied/other/summaryData/roundResults,
Results in camelCase beside snake_case Election and Ballots, an ISO
create_date beside an epoch-ms-string update_date, and no version field
to tell consumers which shape they have. Issue Equal-Vote#1420.
This adds a second menu item rather than changing the first. The
existing download is untouched and still byte-identical, so nothing
consuming it breaks; "Download JSON (compact)" writes the new shape to
a .v2.json filename. Promoting it to the default later is a one-line
change in downloadJson.
New packages/shared/src/utils/exportFormat.ts (buildElectionExport):
candidates listed once, a deduped pairwise matrix keyed by name with
self-pairs dropped, {id,name} refs, snake_case throughout, ISO-8601
timestamps, null/empty fields omitted, format_version: 2. Ballot scores
are preserved verbatim including score: null, so a blank stays distinct
from an explicit 0 and from a NOTA vote.
Transform-only: no change to the tabulation engine, the results
endpoint, or what the UI renders.
✅ Deploy Preview for bettervoting ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
Contributor
📝 WalkthroughSummary by CodeRabbit
WalkthroughThe PR adds a versioned compact election export format. It normalizes keys and timestamps, removes empty fields, preserves null ballot scores, and transforms results into compact snake_case data. The frontend adds a compact JSON download with a Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/shared/src/utils/exportFormat.ts`:
- Around line 73-77: Update cleanElection to normalize create_date and
update_date before passing the election data through omitEmpty, preserving Date
values as ISO-8601 timestamps rather than allowing recursive cleanup to convert
them to empty objects. Ensure the normalized timestamp fields still undergo the
existing empty-value cleanup.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 3094c6d4-cea9-4f89-9a4e-a1366ef6b56a
📒 Files selected for processing (3)
packages/backend/src/test/exportFormat.test.tspackages/frontend/src/components/Election/Results/BallotDataExport.tsxpackages/shared/src/utils/exportFormat.ts
Comment on lines
+73
to
+77
| const cleanElection = (e: Election): any => { | ||
| const cleaned: any = omitEmpty({ ...e }); | ||
| if (cleaned.create_date) cleaned.create_date = normalizeTimestamp(cleaned.create_date); | ||
| if (cleaned.update_date) cleaned.update_date = normalizeTimestamp(cleaned.update_date); | ||
| return cleaned; |
Contributor
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Normalize timestamps before recursive empty-value cleanup.
Line 74 converts a Date value into {} because omitEmpty processes it as a generic object. normalizeTimestamp then exports "[object Object]" instead of an ISO-8601 timestamp. The Election contract permits Date values.
Proposed fix
const cleanElection = (e: Election): any => {
- const cleaned: any = omitEmpty({ ...e });
+ const cleaned: any = { ...e };
if (cleaned.create_date) cleaned.create_date = normalizeTimestamp(cleaned.create_date);
if (cleaned.update_date) cleaned.update_date = normalizeTimestamp(cleaned.update_date);
- return cleaned;
+ return omitEmpty(cleaned);
};📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| const cleanElection = (e: Election): any => { | |
| const cleaned: any = omitEmpty({ ...e }); | |
| if (cleaned.create_date) cleaned.create_date = normalizeTimestamp(cleaned.create_date); | |
| if (cleaned.update_date) cleaned.update_date = normalizeTimestamp(cleaned.update_date); | |
| return cleaned; | |
| const cleanElection = (e: Election): any => { | |
| const cleaned: any = { ...e }; | |
| if (cleaned.create_date) cleaned.create_date = normalizeTimestamp(cleaned.create_date); | |
| if (cleaned.update_date) cleaned.update_date = normalizeTimestamp(cleaned.update_date); | |
| return omitEmpty(cleaned); |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/shared/src/utils/exportFormat.ts` around lines 73 - 77, Update
cleanElection to normalize create_date and update_date before passing the
election data through omitEmpty, preserving Date values as ISO-8601 timestamps
rather than allowing recursive cleanup to convert them to empty objects. Ensure
the normalized timestamp fields still undergo the existing empty-value cleanup.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds a second JSON download rather than changing the existing one.
#1419bundled this format change with CSV bug fixes; you split the bug fixes out into #1428 (thank you — they're merged) and set the format aside as needing contemplation. That was the right call on a bundled PR, because the two halves carry very different risk. This is the format half on its own, and reshaped so it needs much less contemplation: it changes nothing that exists.JSON.stringify({ Election, Ballots, Results }), byte-for-byte. Any script, archive, or downstream tool reading it keeps working..v2.jsonfilename so the two don't collide in a Downloads folder.So the decision here isn't "should we migrate the export format," which is the hard question. It's "may people opt into a cleaner one." If you later decide the compact shape should be the default, that's a one-line change in
downloadJson, and you'd make it knowing the format has been in the wild and used.What the compact format fixes (#1420)
The current export is the tabulator's in-memory objects handed straight to
JSON.stringify, so it publishes internal shape:votesPreferredOver/winsAgainstmaps, keyed by UUID, self-pairs included, repeated acrosselected/tied/other/summaryDataand everyroundResultsentryResultsin camelCase beside snake_caseElectionandBallotscreate_datebeside an epoch-ms-stringupdate_datebuildElectionExportin the newpackages/shared/src/utils/exportFormat.tsemits: candidates listed once, a dedupedpairwisematrix keyed by name with self-pairs dropped,{id, name}refs everywhere a candidate is referenced, snake_case throughout, ISO-8601 timestamps, null/empty fields omitted, andformat/format_version: 2.Ballot scores are preserved verbatim, including
score: null— a blank (voter didn't score this candidate) stays distinct from an explicit0and from a NOTA vote. That's the JSON half of #1160's raw-audit ask; no CSV changes here.Transform-only. No change to the tabulation engine, the results endpoint, or anything the UI renders.
Verification
Unit tests —
packages/backend/src/test/exportFormat.test.ts, 8 tests: versioning, pairwise dedup, snake_case,{id,name}refs, compact ballots, timestamp normalization, and a regression test thatnull≠0≠ dropped. Full backend suite passes (24 suites / 172 tests).tsc --noEmitand eslint clean on the frontend.Real data — ran
buildElectionExportover 197 distinct real BetterVoting exports (every frozen export in masiarek/star-voting-library, spanning STAR, STAR_PR, Approval, Plurality, IRV, STV and Ranked Robin; 2–61 candidates), asserting on each one that the O(n²) maps are gone, the ballot-score multiset is identical including nulls, the elected set is unchanged, ballot and race counts match, and timestamps normalize. 0 failures.Size, honestly: it's a real reduction but a modest one on typical elections, because ballots dominate most files and ballots are kept nearly verbatim.
The savings track candidate count, as you'd expect from an O(n²) map: the 9-candidate dead-heat election drops to 43% and the 51-candidate board election to 74%, while 2-candidate elections sit around 76% and a 61-candidate election with a large ballot set only reaches 95%.
On the one review comment from last time
CodeRabbit flagged the ballot-score assertion in the test as expecting a "legacy
candidate_id" field. That was a false positive — the format emitscandidate_idon ballot scores deliberately (matchingElection.races[].candidates[].candidate_id), and the test is correct. It passes as written.Screenshots / Videos (frontend only)
No visible page change beyond one added menu item; the outputs are the downloaded files. The Download menu now offers: CSV, JSON, JSON (compact).
Related Issues
logs+tieBreakType) would build onbuildElectionExport. Out of scope here.