perf: scatter groupby-sum terms directly instead of unstacking

[FBumann]

Note

The following content was generated by AI.

What this does

The fast path of LinearExpression.groupby(...).sum() previously did
ds.unstack(group_dim, fill_value=...) followed by a stack. That
materializes 2–3 intermediate copies of the padded result
(n_groups × max_group_size × nterm) and routes through pandas
MultiIndex machinery sized by the number of elements.

This change factorizes the groups and scatters coeffs/vars directly
into preallocated padded result arrays; constants are group-summed with
np.add.at. Peak memory drops to input + result (the minimum for the
padded layout), and the grouping itself gets considerably faster. The
result is unchanged: same dims, coords, term ordering and padding.

The unstack-based implementation is kept as _sum_by_unstack and is
still used for chunked (dask-backed) data, which cannot be scattered into
numpy arrays. NaN group labels now raise an informative ValueError
instead of failing inside unstack.

Notes

• Self-contained: touches only linopy/expressions.py and adds tests in
  test/test_linear_expression.py (124 new lines).
• _sum_by_unstack retains the current master names_to_drop logic
  (drop every coordinate aligned to group_dim), so the slow path keeps
  the existing behavior.

Verification

• pytest test/test_linear_expression.py → 309 passed
• groupby/sum subset (-k "group or sum or scatter or unstack") → 73 passed
• ruff check, ruff format --check, mypy linopy/expressions.py → clean

[codspeed-hq]

Merging this PR will improve performance by ×2.1

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 10 improved benchmarks
✅ 128 untouched benchmarks
⏩ 138 skipped benchmarks¹

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
⚡	Memory	test_to_lp[nodal_balance-severity=100]	17.9 MB	6 MB	×3
⚡	Memory	test_to_lp[nodal_balance-severity=50]	9.2 MB	3.1 MB	×3
⚡	Memory	test_to_lp[nodal_balance-severity=0]	385.3 KB	135.3 KB	×2.8
⚡	Memory	test_build[nodal_balance-severity=100]	32 MB	12.8 MB	×2.5
⚡	Memory	test_build[nodal_balance-severity=50]	16.8 MB	7 MB	×2.4
⚡	Memory	test_to_solver[highs-nodal_balance-severity=100]	24.9 MB	13.3 MB	+87.47%
⚡	Memory	test_to_solver[gurobi-nodal_balance-severity=100]	25.1 MB	13.5 MB	+86.1%
⚡	Memory	test_to_solver[highs-nodal_balance-severity=50]	12.9 MB	7.1 MB	+81.68%
⚡	Memory	test_to_solver[gurobi-nodal_balance-severity=50]	13.1 MB	7.3 MB	+79.32%
⚡	Memory	test_build[nodal_balance-severity=0]	1.4 MB	1.2 MB	+19.65%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing fluxopt:perf/groupby-sum-scatter-upstream (81d7c23) with master (fe798b1)}

Footnotes

1. 138 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[FabianHofmann]

@FBumann this is super strong. I added release notes and cleaned up some comments. feel free to merge.

[coroa]

Cool, need to wrap my head around it

[coroa]

i don't understand the second condition. What means no coordinates tied to the grouped dimension besides its own index?

[coroa]

ie. can we please improve the docstring

[coroa]

My thinking is, that if the second condition does basically not happen, then we could wrap _can_sum_by_scatter into an xr.apply_ufunc, similar to https://github.com/PyPSA/atlite/blob/966380825089e5f913eb3dd45141abbff4fc075d/atlite/gis.py#L1077-L1089 , and get rid of the groupby path entirely.

[coroa]

Alternatively, we might think about deprecating the dask layer.

[FBumann]

As im working on the v1 convention and the multiindex stuff, this might even become obsolete indirection. I need to think about it