refactor(int32): per-model label dtype instead of sticky process-global#828
Merged
Conversation
Replace the int32 overflow ValueError in add_variables/add_constraints with a monotonic, sticky widen of options["label_dtype"] to int64. Once a model crosses ~2.1 billion labels the process uses int64 labels everywhere (surviving options.reset()), so no cast site can silently narrow an int64 label back to int32. Mixed int32/int64 arrays upcast on concat, so previously-allocated int32 batches stay valid.
read_netcdf reconstructs label arrays directly and restores the counters without going through the add_variables/add_constraints widen path, so a model that had auto-widened to int64 would load with int64 label arrays while the process option stayed int32 -- the silent-downcast corner, reachable via read_netcdf and solve(remote=...). Widen the option on load whenever a restored counter exceeds the int32 maximum.
…odels Address review: keep auto-widening a convenience but emit a UserWarning at the int32->int64 transition, advising users to set label_dtype=int64 upfront (avoids the mid-build upcast, faster and lighter).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ss-global Store the label dtype on the Model (snapshot of options['label_dtype'] at creation, exposed as Model.label_dtype) and widen only that model to int64 on overflow. Every cast site reads the dtype from its object's model reference; save_join takes it as an explicit fill_dtype parameter. This removes the sticky global state: the option and other models are unaffected by one model's widening, reset() keeps its usual semantics, and the widened dtype travels with the model through netCDF and pickle round-trips instead of relying on process-global stickiness. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Model(label_dtype=np.int64) is more direct than mutating the global option before construction; None falls back to options['label_dtype'] so embedders that construct the Model internally keep a process-wide default. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Extract Model._allocate_labels (shared by variable and constraint label assignment) and config.validate_label_dtype (shared by the option setter and Model.__init__). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… argument
The option predated the constructor argument; with per-model storage it
only duplicated the knob. Embedders forward constructor kwargs (PyPSA:
n.optimize(model_kwargs={'label_dtype': np.int64})), so no process-wide
setting is needed. config.py returns to its pre-int32 state.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
FBumann
marked this pull request as ready for review
July 14, 2026 14:20
FabianHofmann
requested changes
Jul 15, 2026
FabianHofmann
left a comment
Collaborator
There was a problem hiding this comment.
generalize label_dtype accessor to dtypes which is property pointing to a frozen dict with fields "labels" at first stage. can be expanded later.
|
|
||
| def _allocate_labels(self, start: int, end: int) -> np.ndarray: | ||
| """Return the label range ``[start, end)``, widening the dtype on overflow.""" | ||
| if end > np.iinfo(self._label_dtype).max: |
Collaborator
There was a problem hiding this comment.
should only be true if dtype!=int64
Address review: generalize the per-model label-dtype accessor into a
`Model.dtypes` property returning a read-only mapping (`MappingProxyType`)
with a single `"labels"` field for now, extensible to more fields later.
The constructor takes `dtypes={"labels": ...}` in place of `label_dtype=`;
unknown keys and unsupported dtypes are rejected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Merging this PR will not alter performance
Comparing Footnotes
|
FBumann
marked this pull request as draft
July 15, 2026 13:08
Drop the redundant _label_dtype field: the model now holds one `_dtypes: dict[DtypeKey, type]` as the single source of truth, and `Model.dtypes` returns a read-only view over it. Keys are typed with a `DtypeKey = Literal["labels"]` alias; the constructor validates against it via `get_args`. All internal cast sites read `model._dtypes["labels"]`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Move the `dtypes` argument validation out of `__init__` into a small `Model._resolve_dtypes` staticmethod that merges the argument onto the defaults in one loop, rejecting unknown keys and non-int32/64 dtypes. `__init__` now just assigns its result. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The alias name is self-explanatory; a Literal alias cannot carry a real docstring anyway. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
FBumann
marked this pull request as ready for review
July 15, 2026 13:49
Member
|
This implemented auto-widening the dtypes now? |
Collaborator
Author
Yes! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This implements int dtype control per Model instead of through a global config option (global kept).
This resolves all potential issues with the global for little complexity.
I think this is the right approach.
Note
The following content was generated by AI.
Builds on the auto-widening behavior from #822 (now closed; this PR rebased onto
master), but moves the label dtype from the sticky process-global option onto theModelitself.What changed
Model()argument:Model(dtypes={"labels": np.int64}), defaultnp.int32. It is exposed read-only asModel.dtypes— atypes.MappingProxyTypemapping (a frozen dict) with a single"labels"field for now, extensible to more fields later (per review feedback). Unknown keys and non-int32/int64dtypes are rejected. Thelinopy.options["label_dtype"]setting is removed entirely — embedders forward constructor kwargs instead (PyPSA:n.optimize(model_kwargs={"dtypes": {"labels": np.int64}})), soconfig.pyreturns to its pre-perf: default integer arrays to int32 for ~25% memory reduction #566 state.int64when its labels would exceed the int32 maximum; theUserWarningrecommends passingdtypes={"labels": np.int64}upfront.common.save_jointakes it as an explicitfill_dtypeparameter instead of reading the global.read_netcdfrestores the widened dtype per model (from the counters, as before). Pickle round-trips preserve it automatically since it is an instance attribute.OptionSettings.widen_label_dtype()and its sticky-reset()semantics are removed, along with the test-suite fixture that guarded against global-state leakage.Why per-model
#822 chose the process-global because "the four float→int cast sites … have no reliable model reference (expressions can be detached)". That premise doesn't hold in the current codebase:
BaseExpression.__init__raises unlessmodelis aModelinstance, and everylabel_dtypecast site is a method on an object with a validated model reference (BaseExpression,Variable,Variables,Constraints,CSRConstraint). The only model-less site,save_join, is called withinteger_dtype=Truefrom exactly three container properties that all haveself.modelin scope.With a reliable model reference, a per-model monotonic widen gives the same no-narrowing guarantee at the same zero hot-path cost (an attribute read), and removes the global design's edge cases:
options["label_dtype"] = np.int32) while a widened model is alive lets the cast sites silently narrow>2^31labels —int64 → int32.astype()wraps to negative values without any error. Per-model, there is nothing to revert.Cast-site → model-reference verification
expressions.pyvars cast in initBaseExpression.__init__modelparam, enforcedisinstance(model, Model)(check moved above the cast)expressions.pysanitizeBaseExpressionself.modelexpressions.pysimplify (2×)LinearExpressionself.modelvariables.pyffill/bfill/sanitizeVariableself.model(init enforcesisinstance(model, Model))variables.pyflatremapVariablesself.modelconstraints.py_to_dataset(2×)CSRConstraintself._modelconstraints.pyflatremapConstraintsself.modelcommon.pysave_joinVariables.labels,Constraints.labels,Constraints.vars)Tests
test/test_dtypes.py: the widen tests assert other models stayint32; the init-arg test drives thedtypes={"labels": ...}mapping (+ invalid-dtype and unknown-key rejection), plustest_dtypes_is_read_only_mapping(mutatingm.dtypesraisesTypeError),test_widen_applies_to_expressions, andtest_auto_widen_survives_pickle; the netcdf test asserts the loaded model is widened. Therestore_label_dtypefixture and the option tests are gone. Full suite passes.🤖 Generated with Claude Code