[FEATURE] Add configurable timestep selection for individual eDisGo grids in eGo

[MoritzSchloesser]

Add configurable timestep selection for individual eDisGo grids in eGo

Summary

Today, the timeindex analysed for each eDisGo grid in an eGo run is effectively
fixed: it is set inside the oedb_ts task via the timeindex kwarg in the
uc4_example.yaml preset (full year / a fixed range). All older eGo mechanisms
for timestep selection are dead code (see "Background" below).

We want a clean, configurable way to determine the analysed timeindex per grid,
with the selection logic living in eDisGo and the configuration exposed in
eGo. Two selection modes:

• manual — an explicitly defined timeindex (as currently done in
  uc4_example.yaml), given either as an ISO-8601 list of timesteps or as
  start + (periods | end) + freq.
• auto — call eDisGo's built-in get_most_critical_time_steps
  (edisgo/tools/temporal_complexity_reduction.py:837) to pick the most critical
  steps from the full-year series, then reduce all time series to that index.

The eGo configuration must allow:

• setting one timeindex/selection config for all analysed grids, and
• setting an individual selection config per grid.

Background (current state)

• eDisGo built-in reducers available:
  • get_most_critical_time_steps — flat DatetimeIndex of most critical
    individual steps (temporal_complexity_reduction.py:837).
  • get_most_critical_time_intervals — worst contiguous 168 h weeks
    (temporal_complexity_reduction.py:639). Deferred — not needed in v1.
• In eGo, get_most_critical_time_intervals is imported and defined in
  _run_edisgo_task_temporal_complexity_reduction
  (eGo/ego/tools/edisgo_integration.py:1487-1684) but is DEAD — its only
  caller is the commented-out legacy run_edisgo block.
• All other eGo timestep mechanisms are dead too: set_timeindex(...) in the
  legacy setup task, timeindex rewrite from etrago_obj.snapshots,
  reduce_timeseries_data_to_given_timeindex per interval, eTraGo
  start_snapshot/end_snapshot, the stale
  tasks: ["3_temporal_complexity_reduction", ...] config key, etc.
• The active path is appl.py → EDisGoNetworks → _run_one_grid_via_runner
  → edisgo.run.run_edisgo(cfg, ...). The cfg from _build_run_edisgo_config
  (edisgo_integration.py:718-759) has zero timestep fields; everything is
  delegated to the eDisGo runner + the uc4_example preset.

So: timestep selection must be configured on the eDisGo side (preset/runner),
and eGo must learn to inject that config.

Proposed design

eDisGo side — new task select_timesteps

Add a single task in edisgo/run/tasks/timeseries.py (alongside the existing
timeindex/oedb tasks), registered once, that branches on mode and respects an
enabled flag:

@register_task("select_timesteps")
def task_select_timesteps(edisgo, ctx, *, enabled=False, mode="manual",
        timesteps=None, start=None, periods=None, end=None, freq="h",
        percentage=0.1, num_steps_loading=None, num_steps_voltage=None,
        weight_by_costs=True, use_troubleshooting_mode=True,
        run_initial_analyze=True, persist=True, reduce_kwargs=None):
    if not enabled:
        return edisgo

    if mode == "manual":
        # build ti from timesteps OR start/periods/freq OR start/end/freq
        if timesteps is not None:
            ti = pd.DatetimeIndex(pd.to_datetime(timesteps))
        elif start is not None and end is not None:
            ti = pd.date_range(start=start, end=end, freq=freq)
        elif start is not None and periods is not None:
            ti = pd.date_range(start=start, periods=periods, freq=freq)
        else:
            raise ValueError("manual: need 'timesteps' or 'start'+('periods'|'end').")

        if edisgo.timeseries.timeindex.empty:
            # pre-oedb_ts: just set the index; oedb downloads only these steps
            edisgo.set_timeindex(ti)
        else:
            # post-loading: reduce all TS to the chosen subset
            reduce_timeseries_data_to_given_timeindex(edisgo, ti, **(reduce_kwargs or {}))

    elif mode == "auto":
        if edisgo.timeseries.timeindex.empty:
            raise ValueError("auto: TS must be loaded earlier in the pipeline.")
        # set reactive internally (do NOT flip the reactive_power_set flag)
        edisgo.set_time_series_reactive_power_control()
        ti = get_most_critical_time_steps(
            edisgo, percentage=percentage,
            num_steps_loading=num_steps_loading,
            num_steps_voltage=num_steps_voltage,
            weight_by_costs=weight_by_costs,
            use_troubleshooting_mode=use_troubleshooting_mode,
            run_initial_analyze=run_initial_analyze,
        )
        if len(ti) == 0:
            ctx.logger.warning("auto: no critical steps; skipping reduction.")
            return edisgo
        reduce_timeseries_data_to_given_timeindex(edisgo, ti, **(reduce_kwargs or {}))

    else:
        raise ValueError(f"unknown mode={mode!r}")

    if persist and ctx.results_dir is not None:
        ctx.results_dir.mkdir(parents=True, exist_ok=True)
        pd.DataFrame({"timestep": edisgo.timeseries.timeindex}).to_csv(
            ctx.results_dir / "timestep_selection.csv", index=False)

    return edisgo

Notes:

• Reduction uses reduce_timeseries_data_to_given_timeindex
  (edisgo/tools/tools.py:1073), not plain set_timeindex, because it slices
  EV / HP / DSM / overlying-grid series consistently (it covers
  overlying_grid=True). Plain set_timeindex does not slice the DataFrames.
• A selection CSV is persisted to <results_dir>/timestep_selection.csv for
  reproducibility.

eDisGo side — preset places the task TWICE (one active per run)

The two modes need different positions in the pipeline:

• manual must run before oedb_ts so OEDB downloads only the requested
  timestamps (cheap). It only sets the index when the timeindex is still empty.
• auto must run after import_overlying_grid_data because it needs all
  active-power TS (including overlying-grid-distributed generator TS) plus
  reactive power to run a meaningful power flow inside
  get_most_critical_time_steps.

The runner walks the pipeline as an ordered list (name, params) and looks up
the same registered function each time — there is no dedup on task names, and
each step carries its own independent params dict. So select_timesteps can
appear twice, each defaulting to enabled: false (net effect: full year, like
today). Only the position matching the chosen mode is enabled at run time.

Proposed uc4_example.yaml pipeline:

pipeline:
  - setup_grid
  - import_generators
  - import_home_batteries
  - import_heat_pumps
  - import_dsm
  - import_electromobility: {charging_strategy: null, flexibility_bands_ucs: [home, work, public, hpc]}
  - select_timesteps: {enabled: false, mode: manual}        # manual position (pre-oedb_ts)
  - oedb_ts: {dispatchable: {other: 0.7}, timeindex: {start: "2035-01-01", periods: 8760, freq: h}}
  - apply_charging_strategy: {strategy: dumb}
  - apply_heat_pump_strategy: {strategy: uncontrolled}
  - import_overlying_grid_data
  - select_timesteps: {enabled: false, mode: auto}          # auto position (post-overlying-grid)
  - reactive_power
  - optimize: {flexible: [heat_pumps, storage, charging_points, dsm], method: soc, opf_version: 3}
  - reinforce
  - save: {archive: true, save_opf_results: true}

Required supporting change in oedb_ts (timeseries.py:177): only call
set_timeindex when the index is still empty, so a manual pre-step can set a
smaller index and oedb_ts respects it instead of overwriting with the full
year:

if timeindex is not None and edisgo.timeseries.timeindex.empty:
    ...
    edisgo.set_timeindex(ti_df)

The oedb_ts.timeindex kwarg is then the full-year default used when both
selectors are disabled or when auto mode is active. (Note: today the preset has
periods: 24 — change to periods: 8760 so auto mode has a full year to reduce
from.)

eDisGo side — validator

In edisgo/run/validator.py:

• Allow select_timesteps anywhere after a TS task (auto) or before it (manual).
• Do not add it to _TS_TASKS — it is a reducer, not a TS setter, so it must
  not be treated as satisfying "timeseries set".
• Add a rule: at most one select_timesteps step may be enabled: true per run
  (else risk of double reduction).

eGo side — scenario configuration

New optional eDisGo.timestep_selection block. Polymorphic shape: a flat dict =
global config for all grids; a {default, per_grid} dict = per-grid config.

Global:

"timestep_selection": {
  "mode": "auto", "percentage": 0.1,
  "weight_by_costs": true, "run_initial_analyze": true
}

Per-grid (presence of per_grid key is the discriminator):

"timestep_selection": {
  "default": { "mode": "auto", "percentage": 0.1 },
  "per_grid": {
    "32377": { "mode": "manual", "timesteps": ["2035-06-21T08:00:00", "2035-06-21T18:00:00"] },
    "30885": { "mode": "auto", "percentage": 0.2, "num_steps_loading": 48 },
    "1056": null
  }
}

Resolver rules:

• Per-grid keys are stringified mv_grid_id (JSON requires string keys).
• Resolution precedence: per_grid[str(id)] > per_grid[int(id)] > default > None.
• Per-grid config replaces default wholesale (no deep merge — avoids
  ambiguity when modes differ).
• null for a grid = explicit opt-out (full year).
• A grid missing from per_grid falls back to default, or to full year if no
  default.
• default may itself be mode: manual with explicit timesteps.

eGo side — plumbing (eGo/ego/tools/edisgo_integration.py)

• Cache self._timestep_selection = self._edisgo_args.get("timestep_selection")
  in _set_scenario_settings (~line 508).
• Add module-level helpers _resolve_timestep_selection(block, mv_grid_id) and
  _inject_select_timesteps_step(pipeline, ts_block).
• In _build_run_edisgo_config, before return cfg: resolve the block for the
  grid; if non-None, load_config(cfg), find the matching select_timesteps
  step (manual position if mode==manual, auto position if mode==auto),
  set its params and force enabled: true, write back as cfg["pipeline"], and
  cfg.pop("extends") to prevent a double preset merge.

Resolved decisions

• Selection logic stays in eDisGo; eGo only configures it.
• get_most_critical_time_steps is left as-is (no changes to the function).
• Reactive power must be the last TS step. Auto mode therefore sets reactive
  internally (set_time_series_reactive_power_control) without flipping the
  reactive_power_set flag, just to run a meaningful PF for step selection; the
  final reactive_power task then re-runs on the reduced active-power series.
  (Verified that analyze does not auto-set reactive power; with no reactive
  TS, PyPSA fills q_set with 0, which is why PF "worked" but gave wrong voltage
  results — unsuitable for critical-step selection.)
• manual accepts either timesteps or start+(periods|end)+freq.
• A manual default is supported.
• Single task body branched on mode, placed at two positions in the preset,
  each enabled: false by default; eGo enables exactly one based on mode.
• auto_intervals mode (get_most_critical_time_intervals) is not needed in
  v1.

Open questions

1. run_initial_analyze=True for auto mode adds one full-year power flow per
   grid (≈ hours for ~100 grids). Keep it as the default, or default to False
   plus an explicit analyze step in the preset?
2. Auto-mode reactive handling — confirm Option A (task sets reactive
   internally without flipping the flag, final reactive_power re-runs on the
   reduced index) vs. Option B (relax the validator and place reactive_power
   before select_timesteps).
3. oedb_ts default periods — confirm changing the preset from 24 to
   8760 (full year) so auto mode has a full year to reduce from.
4. CSV-import propagation — _set_scenario_settings
   (edisgo_integration.py:476-486) can overwrite _json_file["eDisGo"] from a
   prior run's edisgo_args.json. Decide how timestep_selection propagates
   across re-runs.
5. Duplicate EDisGoNetworks in io.py:569-625 — verify there is no second
   code path that needs the same plumbing.

Implementation order

1. Add task_select_timesteps in eDisGo.
2. oedb_ts "only set timeindex if empty" fix.
3. Validator rule (placement + single-enabled + not a TS-setter).
4. Preset YAML: two select_timesteps entries (enabled: false), oedb_ts
   periods → 8760.
5. eDisGo unit tests: manual list / manual range / auto / empty-result warning /
   CSV persistence.
6. Validator tests.
7. eGo _timestep_selection cache.
8. eGo _resolve_timestep_selection + _inject_select_timesteps_step.
9. eGo _build_run_edisgo_config splice + extends handling.
10. Scenario JSON example (no block / global / per-grid with opt-out).
11. E2E smoke test.
12. CSV-import propagation handling.
13. Check duplicate EDisGoNetworks in io.py.

Repos affected

• openego/eDisGo — task, preset, validator, tests.
• openego/eGo — scenario config, plumbing in edisgo_integration.py.