Skip to content

fix: keep generated datasets readable for simulation and bootstrap - #147

Open
nptaylor-irx wants to merge 1 commit into
mainfrom
fix/dataset-header-binding-sep-16
Open

nptaylor-irx wants to merge 1 commit into
mainfrom
fix/dataset-header-binding-sep-16

Conversation

@nptaylor-irx

Copy link
Copy Markdown

Fixes CSV header parsing failures that blocked simulation and bootstrap during the Sep 16 Astra bug bash. Generated datasets now remain readable when loaded and written into a run folder.

What failed and why

  • Michael’s bootstrap treated the quoted header "TAI" as a number. Passing a CSV filename to create_model_from_file() bypassed the unquoted rewrite used for data frames; IGNORE=@ does not skip a row starting with a quote.
  • Jordan’s simulations also failed on unquoted headers such as ID and EVID: those models had no character IGNORE rule, so unquoting alone could not fix them.

The fix

A shared writer binds the working CSV and its model’s $DATA record together. It writes unquoted data fields and makes the header match the existing character IGNORE rule, adding IGNORE=# when none exists. Run-folder preparation uses the same writer so it cannot undo the repair.

Existing character and conditional filters, $INPUT DROP flags, column order, and NONMEM null behavior are preserved. Source files are untouched. Blanket IGNORE=@ would broaden some existing filters and silently discard valid rows, so the fix preserves each model’s rule. Package version: 0.0.0.9141.

Validation

Regression tests reproduce both failures and cover CSV/data-frame inputs, filters, DROP columns, source preservation, nulls, and reloading prepared models. Focused tests passed on Pharmpy 2.0 (8 skips: 7 runtime-dependent and 1 explicitly skipped); loader tests passed on Pharmpy 1.8 with no skips. Full NONMEM simulation/bootstrap execution was not run.

After merge, rebuild the pharma_ai_r image with this package version and run simulation/bootstrap smoke tests.

Related: Sep 16 bug-bash feedback.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant