Combine scores updates to simplify code.#1118
Open
davidbaines wants to merge 3 commits into
Open
Conversation
- Skip empty scores CSVs instead of crashing on rows[0] (IndexError) - Strip header whitespace at read time so column lookups stay consistent with is_current_style and don't silently drop score columns in merge - Guard old-style rows against ragged/blank lines before DataFrame build - Use a fixed-depth glob (*/*/scores-*.csv) so series/experiment are read from the right path parts instead of being mislabeled at other depths - Raise a clear error in check_for_lock_file for unsupported file types instead of an UnboundLocalError Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Read each scores file directly with pd.read_csv(skipinitialspace=True) and concat/reindex to the canonical column set, replacing the manual csv.reader + list-of-lists + defaultdict layer. - Fix: clean_dataframe now strips string columns before filtering, so space-padded " ALL" rows with no chrF3++ score are dropped as intended (previously they survived because the filter compared against raw, un-stripped values on comma-space separated files). - Remove dead current-style machinery (is_current_style, transform_current_style_rows, CURRENT_STYLE_OUTPUT_COLUMNS): the detection could never return True because CURRENT_STYLE_COLUMNS lists the Series/Experiment/Steps columns this script itself synthesizes. - skipinitialspace removes the need for manual header/value stripping and eliminates the whitespace hazard at its root; on_bad_lines="warn" skips ragged rows instead of crashing, and EmptyDataError skips empty files. - Add type hints and derive numeric/string column sets from named lists. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Scores files live one level below the passed folder
(folder/<experiment>/scores-*.csv), but the fixed-depth glob
"*/*/scores-*.csv" required two levels and matched nothing. Restore the
original rglob("*/scores-*.csv") so per-experiment subfolders (and any
deeper nesting) are found; parts[-2]/parts[-3] remain the experiment and
series directories.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
TaperChipmunk32
approved these changes
Jul 14, 2026
TaperChipmunk32
left a comment
TaperChipmunk32
left a comment
Collaborator
There was a problem hiding this comment.
@TaperChipmunk32 made 1 comment.
Reviewable status: 0 of 1 files reviewed, all discussions resolved.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Remove the is_current_style function that isn't needed with the pandas refactor.
Ensure that the "ALL" results are skipped.
This change is