diff --git a/documentation/api/change_log.rst b/documentation/api/change_log.rst index d3ac7297c8..edc816ba37 100644 --- a/documentation/api/change_log.rst +++ b/documentation/api/change_log.rst @@ -5,7 +5,24 @@ API change log .. note:: The FlexMeasures API follows its own versioning scheme. This is also reflected in the URL (e.g. `/api/v3_0`), allowing developers to upgrade at their own pace. -v3.0-31 | 2026-04-28 +v3.0-32 | 2026-05-xx +"""""""""""""""""""" + +- Introduced explicit ``inflexible-loads`` and ``inflexible-generators`` fields in the ``flex-context``, replacing the ambiguous ``inflexible-device-sensors`` field: + + - ``inflexible-loads``: list of sensor IDs for inflexible consumers; sensor values are interpreted using the consumption-is-positive sign convention. + - ``inflexible-generators``: list of sensor IDs for inflexible generators; sensor values are interpreted using the production-is-positive sign convention (the FlexMeasures default). + - ``inflexible-device-sensors`` is deprecated and will be removed in a future version. + Use ``inflexible-loads`` for sensors with the consumption-is-positive convention and ``inflexible-generators`` for sensors with the production-is-positive convention. + +- Introduced explicit ``consumption`` and ``production`` fields in the per-sensor entry of the ``flex-model`` list (for asset-level scheduling), replacing the ambiguous ``sensor`` field: + + - ``consumption``: sensor ID for the flexible consumer being scheduled; the scheduler applies the consumption-is-positive sign convention. + - ``production``: sensor ID for the flexible producer being scheduled; the scheduler applies the production-is-positive sign convention. + - ``sensor`` is deprecated for asset-level scheduling. Use ``consumption`` or ``production`` instead for unambiguous sign conventions. + The ``sensor`` fallback still works and falls back on the sensor's ``consumption_is_positive`` attribute to determine the sign convention. + + """""""""""""""""""" - Added a unified job status endpoint ``GET /api/v3_0/jobs/`` that returns the current execution status and a human-readable result message for any background job (scheduling, forecasting, etc.) identified by its UUID. diff --git a/documentation/changelog.rst b/documentation/changelog.rst index 41842b1a9d..a1ca7e857e 100644 --- a/documentation/changelog.rst +++ b/documentation/changelog.rst @@ -8,6 +8,7 @@ v0.33.0 | May XX, 2026 New features ------------- +* Make flex-config more explicit about production vs consumption: replace ``inflexible-device-sensors`` in the flex-context with ``inflexible-loads`` and ``inflexible-generators``, and replace ``sensor`` in the asset-level flex-model with ``consumption`` or ``production`` [see `PR #XXXX `_] * Added API and UI support for copying assets and their subtrees [see `PR #2017 `_ and `PR #2120 `_] * Improve UX after deleting a child asset through the UI [see `PR #2119 `_] * Improve source filtering in the sensor data GET endpoint by exposing the documented query parameters in Swagger and allowing filtering by the account linked to data sources [see `PR #2083 `_] diff --git a/documentation/concepts/data-model.rst b/documentation/concepts/data-model.rst index 6c2267768a..29b747786b 100644 --- a/documentation/concepts/data-model.rst +++ b/documentation/concepts/data-model.rst @@ -170,10 +170,29 @@ We assume that this is what users send in. Note that, if forecasts are created, they will have the same sign as original data. -For schedules, the sign of resulting power data (beliefs) is being switched when data is stored (assuming consumption , and you can prevent that by setting ``sensor.attributes["consumption_is_positive"] = True``. +For schedules, the sign convention of resulting power data (beliefs) depends on how the sensor is configured: +- By default, FlexMeasures stores schedule results with **production as positive** and consumption as negative (the FlexMeasures default). +- If you set ``sensor.attributes["consumption_is_positive"] = True`` on the power sensor, consumption is stored as positive. -.. note:: We will soon document better what the scheduler does in detail, and how the attribute works. +When it comes to inflexible devices in the :ref:`flex_context `, the convention should be made explicit using the new ``inflexible-loads`` and ``inflexible-generators`` fields: + +- Use ``inflexible-generators`` for sensors that store **production as positive** values (e.g. rooftop solar where positive readings = generation). This is also the FlexMeasures default. +- Use ``inflexible-loads`` for sensors that store **consumption as positive** values (e.g. a building load sensor where positive readings = consumption). + +For example, if your solar PV sensor uses the default FlexMeasures convention (positive = production), include it as:: + + "flex-context": {"inflexible-generators": []} + +If your building consumption sensor stores positive values for load, include it as:: + + "flex-context": {"inflexible-loads": []} + +When triggering asset-level schedules, use ``consumption`` or ``production`` in the flex-model to make the sign convention explicit:: + + "flex-model": [{"production": }, {"consumption": , "soc-at-start": "50 kWh"}] + +This replaces the ambiguous ``sensor`` key and avoids relying on the ``consumption_is_positive`` sensor attribute for scheduling purposes. Accounts & Users diff --git a/documentation/features/scheduling.rst b/documentation/features/scheduling.rst index 3801f2bda0..0ec846139f 100644 --- a/documentation/features/scheduling.rst +++ b/documentation/features/scheduling.rst @@ -58,7 +58,13 @@ And if the asset belongs to a larger system (a hierarchy of assets), the schedul * - Field - Example value - Description - * - ``inflexible-device-sensors`` + * - ``inflexible-loads`` + - |INFLEXIBLE_LOADS.example| + - .. include:: ../_autodoc/INFLEXIBLE_LOADS.rst + * - ``inflexible-generators`` + - |INFLEXIBLE_GENERATORS.example| + - .. include:: ../_autodoc/INFLEXIBLE_GENERATORS.rst + * - ``inflexible-device-sensors`` *(deprecated)* - |INFLEXIBLE_DEVICE_SENSORS.example| - .. include:: ../_autodoc/INFLEXIBLE_DEVICE_SENSORS.rst * - ``aggregate-power`` @@ -120,6 +126,9 @@ And if the asset belongs to a larger system (a hierarchy of assets), the schedul .. [#old_production_price_field] This field replaced the ``production-price-sensor`` field, which only accepted an integer (sensor ID). +.. [#old_inflexible_device_sensors_field] These fields replace the ``inflexible-device-sensors`` field, which did not distinguish between loads and generators. + Use ``inflexible-loads`` for sensors recording consumption (consumption-is-positive convention) and ``inflexible-generators`` for sensors recording production (production-is-positive convention, which is the FlexMeasures default). + .. [#asymmetric] ``site-consumption-capacity`` and ``site-production-capacity`` allow defining asymmetric contracted transport capacities for each direction (i.e. production and consumption). .. [#minimum_capacity_overlap] In case this capacity field defines partially overlapping time periods, the minimum value is selected. See :ref:`variable_quantities`. diff --git a/documentation/tut/scripts/run-tutorial2-in-docker.sh b/documentation/tut/scripts/run-tutorial2-in-docker.sh index b0c8587be1..9492a4aea4 100755 --- a/documentation/tut/scripts/run-tutorial2-in-docker.sh +++ b/documentation/tut/scripts/run-tutorial2-in-docker.sh @@ -45,7 +45,7 @@ docker exec -it flexmeasures-server-1 flexmeasures show beliefs --sensor 3 --sta echo "[TUTORIAL-RUNNER] update schedule taking solar into account ..." docker exec -it flexmeasures-server-1 flexmeasures add schedule --sensor 2 \ --start ${TOMORROW}T07:00+01:00 --duration PT12H --soc-at-start 50% \ - --flex-context '{"inflexible-device-sensors": [3]}' \ + --flex-context '{"inflexible-generators": [3]}' \ --flex-model '{"soc-min": "50 kWh"}' echo "[TUTORIAL-RUNNER] showing schedule ..." diff --git a/documentation/tut/scripts/run-tutorial3-in-docker.sh b/documentation/tut/scripts/run-tutorial3-in-docker.sh index 3cc3b87ac3..f5913e0356 100755 --- a/documentation/tut/scripts/run-tutorial3-in-docker.sh +++ b/documentation/tut/scripts/run-tutorial3-in-docker.sh @@ -35,7 +35,7 @@ docker exec -it flexmeasures-server-1 flexmeasures add beliefs --sensor 3 --sour echo "[TUTORIAL-RUNNER] Now running both battery and PV together, still using block price profiles ..." docker exec -it flexmeasures-server-1 flexmeasures add schedule --asset 2 \ --start ${TOMORROW}T07:00+01:00 --duration PT12H \ - --flex-model '[{"sensor": 3, "consumption-capacity": "0 kW", "production-capacity": {"sensor": 3, "source": 4}}, {"sensor": 2, "soc-at-start": "225 kWh", "soc-min": "50 kWh"}]'\ + --flex-model '[{"production": 3, "consumption-capacity": "0 kW", "production-capacity": {"sensor": 3, "source": 4}}, {"consumption": 2, "soc-at-start": "225 kWh", "soc-min": "50 kWh"}]'\ --flex-context tutorial3-priceprofile-flex-context.json echo "[TUTORIAL-RUNNER] showing PV and battery schedule ..." @@ -47,7 +47,7 @@ docker exec -it flexmeasures-server-1 flexmeasures add beliefs --sensor 3 --sour echo "[TUTORIAL-RUNNER] Now running both battery and PV together, with realistic DA prices and larger battery ..." docker exec -it flexmeasures-server-1 flexmeasures add schedule --asset 2 \ --start ${TOMORROW}T07:00+01:00 --duration PT12H \ - --flex-model '[{"sensor": 3, "consumption-capacity": "0 kW", "production-capacity": {"sensor": 3, "source": 4}}, {"sensor": 2, "soc-at-start": "225 kWh", "soc-min": "50 kWh", "soc-max": "900kWh"}]' + --flex-model '[{"production": 3, "consumption-capacity": "0 kW", "production-capacity": {"sensor": 3, "source": 4}}, {"consumption": 2, "soc-at-start": "225 kWh", "soc-min": "50 kWh", "soc-max": "900kWh"}]' echo "[TUTORIAL-RUNNER] showing PV and battery schedule ..." docker exec -it flexmeasures-server-1 flexmeasures show beliefs --sensor 3 --sensor 2 --start ${TOMORROW}T07:00:00+01:00 --duration PT12H diff --git a/documentation/tut/toy-example-expanded.rst b/documentation/tut/toy-example-expanded.rst index 3922ad9a3e..641207aa96 100644 --- a/documentation/tut/toy-example-expanded.rst +++ b/documentation/tut/toy-example-expanded.rst @@ -15,7 +15,7 @@ When solar production is high, less battery output can be send to the grid, as t How does it work? -- We will tell FlexMeasures to take the solar production into account (using the ``inflexible-device-sensors`` flex-context field). +- We will tell FlexMeasures to take the solar production into account (using the ``inflexible-generators`` flex-context field). - The battery's power capacity is not the limiting factor, but the `site-power-capacity` of the building (already a flex-context field, see :ref:`tut_toy_schedule`). - The flows of the building's child assets are summed up on building level, and that constraint now will play a role. @@ -96,7 +96,7 @@ This will have an effect on the available headroom for the battery, given the `` --start ${TOMORROW}T07:00+01:00 \ --duration PT12H \ --soc-at-start 50% \ - --flex-context '{"inflexible-device-sensors": [3]}' + --flex-context '{"inflexible-generators": [3]}' --flex-model '{"soc-min": "50 kWh"}' \ New schedule is stored. @@ -115,7 +115,7 @@ This will have an effect on the available headroom for the battery, given the `` "soc-min": "50 kWh" }, "flex-context": { - "inflexible-device-sensors": [3] + "inflexible-generators": [3] } } @@ -149,7 +149,7 @@ This will have an effect on the available headroom for the battery, given the `` "soc-min": "50 kWh", }, flex_context={ - "inflexible-device-sensors": [3], # solar production (sensor ID) + "inflexible-generators": [3], # solar production (sensor ID) }, ) print(schedule) diff --git a/documentation/tut/toy-example-multiasset-curtailment.rst b/documentation/tut/toy-example-multiasset-curtailment.rst index b9b0e3ff99..68c183735f 100644 --- a/documentation/tut/toy-example-multiasset-curtailment.rst +++ b/documentation/tut/toy-example-multiasset-curtailment.rst @@ -178,7 +178,7 @@ Note that we are still passing in the flex-context with block price profiles her --asset 2 \ --start ${TOMORROW}T07:00+01:00 \ --duration PT12H \ - --flex-model '[{"sensor": 3, "consumption-capacity": "0 kW", "production-capacity": {"sensor": 3, "source": 4}}, {"sensor": 2, "soc-at-start": "225 kWh", "soc-min": "50 kWh"}]'\ + --flex-model '[{"production": 3, "consumption-capacity": "0 kW", "production-capacity": {"sensor": 3, "source": 4}}, {"consumption": 2, "soc-at-start": "225 kWh", "soc-min": "50 kWh"}]'\ --flex-context tutorial3-priceprofile-flex-context.json New schedule is stored. @@ -193,12 +193,12 @@ Note that we are still passing in the flex-context with block price profiles her "duration": "PT12H", "flex-model": [ { - "sensor": 3, + "production": 3, "consumption-capacity": "0 kW", "production-capacity": {"sensor": 3, "source": 4}, } { - "sensor": 2, + "consumption": 2, "soc-at-start": "225 kWh", "soc-min": "50 kWh" }, @@ -232,12 +232,12 @@ Note that we are still passing in the flex-context with block price profiles her duration="PT12H", flex_model=[ { - "sensor": 3, # solar production (sensor ID) + "production": 3, # solar production (sensor ID) "consumption-capacity": "0 kW", "production-capacity": {"sensor": 3, "source": 4}, }, { - "sensor": 2, # battery power (sensor ID) + "consumption": 2, # battery power (sensor ID) "soc-at-start": "225 kWh", "soc-min": "50 kWh", }, diff --git a/flexmeasures/data/migrations/versions/9ed0e39b0447_migrate_inflexible_device_sensors_and_sensor_flex_model.py b/flexmeasures/data/migrations/versions/9ed0e39b0447_migrate_inflexible_device_sensors_and_sensor_flex_model.py new file mode 100644 index 0000000000..5c7182651d --- /dev/null +++ b/flexmeasures/data/migrations/versions/9ed0e39b0447_migrate_inflexible_device_sensors_and_sensor_flex_model.py @@ -0,0 +1,189 @@ +"""Migrate inflexible-device-sensors to inflexible-loads/generators, and sensor to consumption/production in flex-model + +Revision ID: 9ed0e39b0447 +Revises: f0ee99278f6f +Create Date: 2025-05-30 00:00:00.000000 + +""" + +from __future__ import annotations + +from alembic import op +import sqlalchemy as sa +from sqlalchemy.dialects.postgresql import JSONB + +# revision identifiers, used by Alembic. +revision = "9ed0e39b0447" +down_revision = "f0ee99278f6f" +branch_labels = None +depends_on = None + + +def _get_consumption_is_positive(connection, sensor_id: int) -> bool: + """Look up a sensor's consumption_is_positive attribute (defaults to False).""" + row = connection.execute( + sa.text("SELECT attributes FROM sensor WHERE id = :sensor_id"), + {"sensor_id": sensor_id}, + ).fetchone() + if row is not None: + attributes = row[0] or {} + return attributes.get("consumption_is_positive", False) + # Sensor not found: default to production-positive (FlexMeasures default) + return False + + +def _migrate_flex_context(connection, asset_id: int, flex_context: dict) -> None: + """Convert inflexible-device-sensors → inflexible-loads / inflexible-generators.""" + old_sensor_ids = flex_context.get("inflexible-device-sensors", []) + if not old_sensor_ids: + return + + loads = list(flex_context.get("inflexible-loads", [])) + generators = list(flex_context.get("inflexible-generators", [])) + + for sensor_id in old_sensor_ids: + if _get_consumption_is_positive(connection, sensor_id): + loads.append(sensor_id) + else: + generators.append(sensor_id) + + new_flex_context = dict(flex_context) + del new_flex_context["inflexible-device-sensors"] + if loads: + new_flex_context["inflexible-loads"] = loads + if generators: + new_flex_context["inflexible-generators"] = generators + + connection.execute( + sa.text( + "UPDATE generic_asset SET flex_context = :flex_context WHERE id = :asset_id" + ), + {"flex_context": sa.cast(new_flex_context, JSONB), "asset_id": asset_id}, + ) + + +def _migrate_flex_model(connection, asset_id: int, flex_model: list) -> None: + """Convert sensor key → consumption or production in flex-model entries.""" + changed = False + new_flex_model = [] + for entry in flex_model: + if not isinstance(entry, dict) or "sensor" not in entry: + new_flex_model.append(entry) + continue + sensor_id = entry["sensor"] + new_entry = dict(entry) + del new_entry["sensor"] + if _get_consumption_is_positive(connection, sensor_id): + new_entry["consumption"] = sensor_id + else: + new_entry["production"] = sensor_id + new_flex_model.append(new_entry) + changed = True + + if changed: + connection.execute( + sa.text( + "UPDATE generic_asset SET flex_model = :flex_model WHERE id = :asset_id" + ), + {"flex_model": sa.cast(new_flex_model, JSONB), "asset_id": asset_id}, + ) + + +def upgrade(): + """ + Migrate flex-context and flex-model on generic_asset: + + 1. flex-context: Convert `inflexible-device-sensors` list to `inflexible-loads` and + `inflexible-generators` based on each sensor's `consumption_is_positive` attribute. + + 2. flex-model: Convert entries with `sensor` key to either `consumption` or `production` + based on the sensor's `consumption_is_positive` attribute. + """ + connection = op.get_bind() + + for asset_id, flex_context in connection.execute( + sa.text( + "SELECT id, flex_context FROM generic_asset " + "WHERE flex_context ? 'inflexible-device-sensors'" + ) + ).fetchall(): + if flex_context: + _migrate_flex_context(connection, asset_id, flex_context) + + for asset_id, flex_model in connection.execute( + sa.text( + "SELECT id, flex_model FROM generic_asset " + "WHERE flex_model IS NOT NULL AND jsonb_typeof(flex_model) = 'array'" + ) + ).fetchall(): + if flex_model: + _migrate_flex_model(connection, asset_id, flex_model) + + +def _downgrade_flex_context(connection, asset_id: int, flex_context: dict) -> None: + """Combine inflexible-loads + inflexible-generators back to inflexible-device-sensors.""" + sensor_ids = list(flex_context.get("inflexible-loads", [])) + list( + flex_context.get("inflexible-generators", []) + ) + new_flex_context = dict(flex_context) + new_flex_context.pop("inflexible-loads", None) + new_flex_context.pop("inflexible-generators", None) + if sensor_ids: + new_flex_context["inflexible-device-sensors"] = sensor_ids + connection.execute( + sa.text( + "UPDATE generic_asset SET flex_context = :flex_context WHERE id = :asset_id" + ), + {"flex_context": sa.cast(new_flex_context, JSONB), "asset_id": asset_id}, + ) + + +def _downgrade_flex_model(connection, asset_id: int, flex_model: list) -> None: + """Convert consumption/production back to sensor in flex-model entries.""" + changed = False + new_flex_model = [] + for entry in flex_model: + if not isinstance(entry, dict): + new_flex_model.append(entry) + continue + new_entry = dict(entry) + if "consumption" in new_entry: + new_entry["sensor"] = new_entry.pop("consumption") + changed = True + elif "production" in new_entry: + new_entry["sensor"] = new_entry.pop("production") + changed = True + new_flex_model.append(new_entry) + if changed: + connection.execute( + sa.text( + "UPDATE generic_asset SET flex_model = :flex_model WHERE id = :asset_id" + ), + {"flex_model": sa.cast(new_flex_model, JSONB), "asset_id": asset_id}, + ) + + +def downgrade(): + """ + Reverse migration: convert inflexible-loads/generators back to inflexible-device-sensors, + and consumption/production back to sensor in flex-model. + """ + connection = op.get_bind() + + for asset_id, flex_context in connection.execute( + sa.text( + "SELECT id, flex_context FROM generic_asset " + "WHERE flex_context ? 'inflexible-loads' OR flex_context ? 'inflexible-generators'" + ) + ).fetchall(): + if flex_context: + _downgrade_flex_context(connection, asset_id, flex_context) + + for asset_id, flex_model in connection.execute( + sa.text( + "SELECT id, flex_model FROM generic_asset " + "WHERE flex_model IS NOT NULL AND jsonb_typeof(flex_model) = 'array'" + ) + ).fetchall(): + if flex_model: + _downgrade_flex_model(connection, asset_id, flex_model) diff --git a/flexmeasures/data/models/planning/storage.py b/flexmeasures/data/models/planning/storage.py index c23e9e09d2..9920bad8eb 100644 --- a/flexmeasures/data/models/planning/storage.py +++ b/flexmeasures/data/models/planning/storage.py @@ -172,6 +172,8 @@ def _prepare(self, skip_validation: bool = False) -> tuple: # noqa: C901 inflexible_device_sensors = self.flex_context.get( "inflexible_device_sensors", [] ) + inflexible_loads = self.flex_context.get("inflexible_loads", []) + inflexible_generators = self.flex_context.get("inflexible_generators", []) # Fetch the device's power capacity (required Sensor attribute) power_capacity_in_mw = self._get_device_power_capacity(flex_model, assets) @@ -468,18 +470,32 @@ def _prepare(self, skip_validation: bool = False) -> tuple: # noqa: C901 ) commitments.append(commitment) + # Build the combined list of inflexible sensors with their sign convention. + # Each entry is (sensor, consumption_is_positive): + # - inflexible_loads: positive stored values = consumption → consumption_is_positive=True + # - inflexible_generators: positive stored values = production → consumption_is_positive=False + # - inflexible_device_sensors (deprecated): use the sensor's own attribute + all_inflexible = ( + [(s, True) for s in inflexible_loads] + + [(s, False) for s in inflexible_generators] + + [(s, None) for s in inflexible_device_sensors] + ) + # Set up device constraints: scheduled flexible devices for this EMS (from index 0 to D-1), plus the forecasted inflexible devices (at indices D to n). device_constraints = [ initialize_df(StorageScheduler.COLUMNS, start, end, resolution) - for i in range(num_flexible_devices + len(inflexible_device_sensors)) + for i in range(num_flexible_devices + len(all_inflexible)) ] - for i, inflexible_sensor in enumerate(inflexible_device_sensors): + for i, (inflexible_sensor, consumption_is_positive) in enumerate( + all_inflexible + ): device_constraints[i + num_flexible_devices]["derivative equals"] = ( get_power_values( query_window=(start, end), resolution=resolution, beliefs_before=belief_time, sensor=inflexible_sensor, + consumption_is_positive=consumption_is_positive, ) ) @@ -1054,6 +1070,9 @@ def deserialize_flex_config(self): ).load(sensor_flex_model["sensor_flex_model"]) self.flex_model[d]["sensor"] = sensor_flex_model.get("sensor") self.flex_model[d]["asset"] = sensor_flex_model.get("asset") + self.flex_model[d]["is_consumption_sensor"] = sensor_flex_model.get( + "is_consumption_sensor" + ) # Extend schedule period in case a target exceeds its end self.possibly_extend_end( @@ -1433,6 +1452,22 @@ def _ensure_variable_quantity( ) return q + @staticmethod + def _build_is_consumption_sensor_map( + flex_model: list[dict], + ) -> dict: + """Build a mapping from sensor object to its is_consumption_sensor flag. + + Returns a dict where values are True (consumption-positive), False + (production-positive), or None (fall back to sensor attribute). + """ + mapping: dict = {} + for fm_item in flex_model: + s = fm_item.get("sensor") + if s is not None and s not in mapping: + mapping[s] = fm_item.get("is_consumption_sensor") + return mapping + class StorageFallbackScheduler(MetaStorageScheduler): __version__ = "3" @@ -1490,11 +1525,19 @@ def compute(self, skip_validation: bool = False) -> SchedulerOutputType: } if self.return_multiple: + # Build a mapping from sensor to is_consumption_sensor (from flex_model if available) + is_consumption_sensor_map = {} + flex_model = self.flex_model + if isinstance(flex_model, list): + is_consumption_sensor_map = self._build_is_consumption_sensor_map( + flex_model + ) return [ { "name": "storage_schedule", "sensor": sensor, "data": storage_schedule[sensor], + "is_consumption_sensor": is_consumption_sensor_map.get(sensor), } for sensor in sensors if sensor is not None @@ -1663,12 +1706,17 @@ def compute(self, skip_validation: bool = False) -> SchedulerOutputType: } if self.return_multiple: + # Build a mapping from sensor to is_consumption_sensor (from flex_model if available) + is_consumption_sensor_map = self._build_is_consumption_sensor_map( + flex_model + ) storage_schedules = [ { "name": "storage_schedule", "sensor": sensor, "data": storage_schedule[sensor], "unit": sensor.unit, + "is_consumption_sensor": is_consumption_sensor_map.get(sensor), } for sensor in storage_schedule.keys() if sensor is not None diff --git a/flexmeasures/data/models/planning/tests/test_utils_fresh_db.py b/flexmeasures/data/models/planning/tests/test_utils_fresh_db.py new file mode 100644 index 0000000000..4776df559c --- /dev/null +++ b/flexmeasures/data/models/planning/tests/test_utils_fresh_db.py @@ -0,0 +1,134 @@ +"""Tests for get_power_values sign-convention behavior. + +These tests write beliefs into the database and mutate sensor attributes, so they require +the function-scoped ``fresh_db`` fixture to prevent state leaking between parametrized runs. +""" + +import numpy as np +import pandas as pd +from datetime import timedelta + +import pytest + +import timely_beliefs as tb + +from flexmeasures.data.models.generic_assets import GenericAsset, GenericAssetType +from flexmeasures.data.models.time_series import Sensor, TimedBelief +from flexmeasures.data.models.data_sources import DataSource +from flexmeasures.data.models.planning.utils import get_power_values + + +@pytest.fixture() +def inflexible_power_sensor(fresh_db, app): + """Create a fresh sensor with known power values for each test. + + The sensor stores a positive value (100 kW = 0.1 MW). In FlexMeasures' default + convention ``consumption_is_positive`` is *False*, so positive values represent + production and must be negated when the scheduler needs consumption-positive MW values. + """ + source = DataSource(name="inflexible-test-source", type="test") + fresh_db.session.add(source) + + asset_type = GenericAssetType(name="inflexible-asset-type") + fresh_db.session.add(asset_type) + + asset = GenericAsset(name="inflexible-asset", generic_asset_type=asset_type) + fresh_db.session.add(asset) + + sensor = Sensor( + name="inflexible-power", + generic_asset=asset, + event_resolution=timedelta(hours=1), + unit="kW", + ) + fresh_db.session.add(sensor) + fresh_db.session.flush() # ensure sensor.id is populated + + query_window = ( + pd.Timestamp("2025-06-01 00:00:00+00:00"), + pd.Timestamp("2025-06-01 01:00:00+00:00"), + ) + + bdf = tb.BeliefsDataFrame( + pd.DataFrame( + { + "event_start": pd.date_range( + start=query_window[0], freq="1h", periods=1 + ), + "event_value": [100.0], # 100 kW → 0.1 MW after unit conversion + } + ), + belief_horizon=pd.Timedelta(0), + sensor=sensor, + source=source, + event_resolution=sensor.event_resolution, + ) + TimedBelief.add(bdf) + fresh_db.session.commit() + + return sensor, query_window + + +@pytest.mark.parametrize( + "consumption_is_positive, expected_mw", + [ + (True, 0.1), # consumption-positive: return value unchanged + (False, -0.1), # production-positive: negate the stored value + ], +) +def test_get_power_values_sign_convention( + app, inflexible_power_sensor, consumption_is_positive, expected_mw +): + """get_power_values respects an explicit ``consumption_is_positive`` override. + + The stored value is 100 kW (0.1 MW). + + * ``consumption_is_positive=True`` → return as-is (+0.1 MW, consumption) + * ``consumption_is_positive=False`` → negate (-0.1 MW, production) + """ + sensor, query_window = inflexible_power_sensor + with app.app_context(): + result = get_power_values( + query_window=query_window, + resolution=timedelta(hours=1), + beliefs_before=None, + sensor=sensor, + consumption_is_positive=consumption_is_positive, + ) + assert isinstance(result, np.ndarray) + assert len(result) == 1 + assert result[0] == pytest.approx(expected_mw) + + +def test_get_power_values_falls_back_to_sensor_attribute(app, inflexible_power_sensor): + """get_power_values falls back to the sensor's ``consumption_is_positive`` attribute. + + When the parameter is ``None`` the sensor attribute is used: + + * Default (no attribute set) → ``False`` → values are negated → -0.1 MW + * After setting attribute to ``True`` → values returned unchanged → +0.1 MW + """ + sensor, query_window = inflexible_power_sensor + + # No attribute set: default is False (production-positive), so value is negated. + with app.app_context(): + result_default = get_power_values( + query_window=query_window, + resolution=timedelta(hours=1), + beliefs_before=None, + sensor=sensor, + consumption_is_positive=None, + ) + assert result_default[0] == pytest.approx(-0.1) + + # Explicitly set attribute to True (consumption-positive): value is returned as-is. + sensor.attributes["consumption_is_positive"] = True + with app.app_context(): + result_attr_true = get_power_values( + query_window=query_window, + resolution=timedelta(hours=1), + beliefs_before=None, + sensor=sensor, + consumption_is_positive=None, + ) + assert result_attr_true[0] == pytest.approx(0.1) diff --git a/flexmeasures/data/models/planning/utils.py b/flexmeasures/data/models/planning/utils.py index 111174b0b2..0f5ed0b9d3 100644 --- a/flexmeasures/data/models/planning/utils.py +++ b/flexmeasures/data/models/planning/utils.py @@ -173,6 +173,7 @@ def get_power_values( resolution: timedelta, beliefs_before: datetime | None, sensor: Sensor, + consumption_is_positive: bool | None = None, ) -> np.ndarray: """Get measurements or forecasts of an inflexible device represented by a power or energy sensor as an array of power values in MW. @@ -180,11 +181,14 @@ def get_power_values( If the requested schedule lies in the past, the returned data will consist of (the most recent) measurements (if any exist). The latter amounts to answering "What if we could have scheduled under perfect foresight?". - :param query_window: datetime window within which events occur (equal to the scheduling window) - :param resolution: timedelta used to resample the forecasts to the resolution of the schedule - :param beliefs_before: datetime used to indicate we are interested in the state of knowledge at that time - :param sensor: power sensor representing an energy flow out of the device - :returns: power measurements or forecasts (consumption is positive, production is negative) + :param query_window: datetime window within which events occur (equal to the scheduling window) + :param resolution: timedelta used to resample the forecasts to the resolution of the schedule + :param beliefs_before: datetime used to indicate we are interested in the state of knowledge at that time + :param sensor: power sensor representing an energy flow out of the device + :param consumption_is_positive: if True, positive sensor values represent consumption (no negation needed); + if False, positive sensor values represent production (negation applied); + if None (default), the sensor's ``consumption_is_positive`` attribute is used. + :returns: power measurements or forecasts (consumption is positive, production is negative) """ bdf: tb.BeliefsDataFrame = TimedBelief.search( sensor, @@ -211,9 +215,14 @@ def get_power_values( event_resolution=sensor.event_resolution, ) - if sensor.get_attribute( - "consumption_is_positive", False - ): # FlexMeasures default is to store consumption as negative power values + # Determine the sign convention: if consumption_is_positive is explicitly provided, + # use it; otherwise fall back to the sensor's own attribute. + if consumption_is_positive is None: + consumption_is_positive = sensor.get_attribute( + "consumption_is_positive", False + ) # FlexMeasures default is to store consumption as negative power values + + if consumption_is_positive: return series return -series diff --git a/flexmeasures/data/schemas/scheduling/__init__.py b/flexmeasures/data/schemas/scheduling/__init__.py index 2050830857..3ca2d2a14c 100644 --- a/flexmeasures/data/schemas/scheduling/__init__.py +++ b/flexmeasures/data/schemas/scheduling/__init__.py @@ -1,4 +1,5 @@ from __future__ import annotations +import warnings from datetime import timedelta from typing import Any, Callable, Dict @@ -297,6 +298,16 @@ class FlexContextSchema(Schema): data_key="inflexible-device-sensors", metadata=metadata.INFLEXIBLE_DEVICE_SENSORS.to_dict(), ) + inflexible_loads = fields.List( + SensorIdField(), + data_key="inflexible-loads", + metadata=metadata.INFLEXIBLE_LOADS.to_dict(), + ) + inflexible_generators = fields.List( + SensorIdField(), + data_key="inflexible-generators", + metadata=metadata.INFLEXIBLE_GENERATORS.to_dict(), + ) aggregate_power = VariableQuantityField( to_unit="MW", data_key="aggregate-power", @@ -304,6 +315,20 @@ class FlexContextSchema(Schema): metadata=metadata.AGGREGATE_POWER.to_dict(), ) + @post_load + def warn_deprecated_inflexible_device_sensors(self, data: dict, **kwargs): + """Emit a deprecation warning if the deprecated `inflexible-device-sensors` field is present in the input.""" + if "inflexible_device_sensors" in data: + warnings.warn( + "The `inflexible-device-sensors` field is deprecated. " + "Use `inflexible-loads` for sensors with consumption-positive convention " + "and `inflexible-generators` for sensors with production-positive convention " + "(the FlexMeasures default).", + UserWarning, + stacklevel=2, + ) + return data + def set_default_breach_prices( self, data: dict, fields: list[str], price: ur.Quantity ): @@ -569,6 +594,16 @@ def _to_currency_per_mwh(price_unit: str) -> str: "description": rst_to_openapi(metadata.INFLEXIBLE_DEVICE_SENSORS.description), "example-units": EXAMPLE_UNIT_TYPES["power"], }, + "inflexible-loads": { + "default": [], + "description": rst_to_openapi(metadata.INFLEXIBLE_LOADS.description), + "example-units": EXAMPLE_UNIT_TYPES["power"], + }, + "inflexible-generators": { + "default": [], + "description": rst_to_openapi(metadata.INFLEXIBLE_GENERATORS.description), + "example-units": EXAMPLE_UNIT_TYPES["power"], + }, "commitments": { "default": None, "description": rst_to_openapi(metadata.COMMITMENTS.description), @@ -816,14 +851,21 @@ def _validate_field(self, data: dict, field_type: str, field: str, unit_validato ) def _validate_inflexible_device_sensors(self, data: dict): - """Validate inflexible device sensors.""" - if "inflexible_device_sensors" in data: - for sensor in data["inflexible_device_sensors"]: - if not is_power_unit(sensor.unit) and not is_energy_unit(sensor.unit): - raise ValidationError( - f"Inflexible device sensor '{sensor.id}' must have a power or energy unit.", - field_name="inflexible-device-sensors", - ) + """Validate inflexible device sensors (deprecated) and new inflexible-loads/generators fields.""" + for field_name, data_key in ( + ("inflexible_device_sensors", "inflexible-device-sensors"), + ("inflexible_loads", "inflexible-loads"), + ("inflexible_generators", "inflexible-generators"), + ): + if field_name in data: + for sensor in data[field_name]: + if not is_power_unit(sensor.unit) and not is_energy_unit( + sensor.unit + ): + raise ValidationError( + f"Inflexible device sensor '{sensor.id}' must have a power or energy unit.", + field_name=data_key, + ) def _forbid_fixed_prices(self, data: dict, **kwargs): """Do not allow fixed consumption price or fixed production price in the flex-context fields saved in the db. @@ -864,6 +906,24 @@ class MultiSensorFlexModelSchema(Schema): { "sensor": , + "is_consumption_sensor": None, + "sensor_flex_model": { + "soc-at-start": "10 kWh" + } + } + + And: + + { + "consumption": 1, + "soc-at-start": "10 kWh" + } + + becomes: + + { + "sensor": , + "is_consumption_sensor": True, "sensor_flex_model": { "soc-at-start": "10 kWh" } @@ -872,6 +932,9 @@ class MultiSensorFlexModelSchema(Schema): sensor = SensorIdField(required=False) asset = GenericAssetIdField(required=False) + consumption = SensorIdField(required=False) + production = SensorIdField(required=False) + is_consumption_sensor = fields.Bool(required=False, load_default=None) # it's up to the Scheduler to deserialize the underlying flex-model sensor_flex_model = fields.Dict(data_key="sensor-flex-model") @@ -884,11 +947,33 @@ def ensure_sensor_or_asset(self, data, **kwargs): ): raise ValidationError("Sensor does not belong to asset.") if "sensor" not in data and "asset" not in data: - raise ValidationError("Specify either a sensor or an asset.") + raise ValidationError( + "Specify either a sensor (or consumption/production) or an asset." + ) + + @post_load + def warn_deprecated_sensor_field(self, data: dict, **kwargs): + """Emit a deprecation warning if the old `sensor` field is used without an explicit sign convention. + + The `sensor` field leaves `is_consumption_sensor` as None (no explicit sign convention), + unlike `consumption` (sets True) or `production` (sets False). + Note: `is_consumption_sensor` is set internally by the `@pre_load` hook when + `consumption` or `production` fields are used; it is not a user-facing input. + """ + if "sensor" in data and data.get("is_consumption_sensor") is None: + warnings.warn( + "The `sensor` field in flex-model entries is deprecated. " + "Use `consumption` to explicitly indicate a consumption sensor " + "or `production` to indicate a production sensor, " + "so FlexMeasures does not have to guess the sign convention from the sensor's attributes.", + UserWarning, + stacklevel=2, + ) + return data @pre_load def unwrap_envelope(self, data, **kwargs): - """Any field other than 'sensor' and 'asset' becomes part of the sensor's flex-model.""" + """Any field other than 'sensor', 'asset', 'consumption', 'production', 'is_consumption_sensor' becomes part of the sensor's flex-model.""" extra = {} rest = {} for k, v in data.items(): @@ -896,6 +981,20 @@ def unwrap_envelope(self, data, **kwargs): extra[k] = v else: rest[k] = v + # Validate mutual exclusion of consumption and production before remapping + if "consumption" in rest and "production" in rest: + raise ValidationError( + "Specify either 'consumption' or 'production', not both." + ) + # Map consumption/production to sensor and set is_consumption_sensor explicitly. + # The deprecated 'sensor' field leaves is_consumption_sensor unset (None), + # meaning the sensor's own `consumption_is_positive` attribute will be used as a fallback. + if "consumption" in rest: + rest["sensor"] = rest.pop("consumption") + rest["is_consumption_sensor"] = True + elif "production" in rest: + rest["sensor"] = rest.pop("production") + rest["is_consumption_sensor"] = False return {"sensor-flex-model": extra, **rest} @post_dump diff --git a/flexmeasures/data/schemas/scheduling/metadata.py b/flexmeasures/data/schemas/scheduling/metadata.py index 5852d6d286..4ae8d5fd39 100644 --- a/flexmeasures/data/schemas/scheduling/metadata.py +++ b/flexmeasures/data/schemas/scheduling/metadata.py @@ -28,13 +28,28 @@ def to_dict(self): INFLEXIBLE_DEVICE_SENSORS = MetaData( - description="""Power sensors representing devices that are relevant, but not flexible in the timing of their demand/supply. -For example, a sensor recording rooftop solar power that is connected behind the main meter, and whose production falls under the same contract as the flexible device(s) being scheduled. -Their power demand cannot be adjusted but still matters for finding the best schedule for other devices. + description="""[Deprecated] Power sensors representing devices that are relevant, but not flexible in the timing of their demand/supply. +Use ``inflexible-loads`` and ``inflexible-generators`` instead for an unambiguous sign convention. Must be a list of integers. """, example=[3, 4], ) +INFLEXIBLE_LOADS = MetaData( + description="""Power sensors representing inflexible loads (consumers) whose demand is relevant but cannot be adjusted. +For example, a sensor recording power consumption of a building's HVAC system. +Positive sensor values are interpreted as consumption (consumption-is-positive convention). +Must be a list of integers. +""", + example=[3], +) +INFLEXIBLE_GENERATORS = MetaData( + description="""Power sensors representing inflexible generators (producers) whose supply is relevant but cannot be adjusted. +For example, a sensor recording rooftop solar power that is connected behind the main meter. +Positive sensor values are interpreted as production (production-is-positive convention, which is the FlexMeasures default). +Must be a list of integers. +""", + example=[4], +) AGGREGATE_POWER = MetaData( description="""Sensor used to record the aggregate power schedule of all flexible and inflexible devices involved when scheduling this asset.""", example={"sensor": 9}, diff --git a/flexmeasures/data/schemas/tests/test_scheduling.py b/flexmeasures/data/schemas/tests/test_scheduling.py index 797edae364..1c34c76437 100644 --- a/flexmeasures/data/schemas/tests/test_scheduling.py +++ b/flexmeasures/data/schemas/tests/test_scheduling.py @@ -5,7 +5,11 @@ from marshmallow.validate import ValidationError import pandas as pd -from flexmeasures.data.schemas.scheduling import FlexContextSchema, DBFlexContextSchema +from flexmeasures.data.schemas.scheduling import ( + FlexContextSchema, + DBFlexContextSchema, + MultiSensorFlexModelSchema, +) from flexmeasures.data.schemas.scheduling.process import ( ProcessSchedulerFlexModelSchema, ProcessType, @@ -824,3 +828,102 @@ def test_db_flex_model_schema(db, app, setup_dummy_sensors, flex_model, fails): ) else: schema.load(flex_model) + + +# --- Tests for FlexContextSchema: new inflexible-loads / inflexible-generators fields --- + + +def test_flex_context_inflexible_loads(db, app, setup_dummy_sensors): + """inflexible-loads deserializes correctly to a list of sensors with power/energy units.""" + # sensor1 has MWh unit (energy) and sensor4 has MW unit (power) + sensor_energy, _, _, sensor_power = setup_dummy_sensors + schema = FlexContextSchema() + result = schema.load({"inflexible-loads": [sensor_energy.id, sensor_power.id]}) + assert "inflexible_loads" in result + assert len(result["inflexible_loads"]) == 2 + assert result["inflexible_loads"][0].id == sensor_energy.id + assert result["inflexible_loads"][1].id == sensor_power.id + + +def test_flex_context_inflexible_generators(db, app, setup_dummy_sensors): + """inflexible-generators deserializes correctly to a list of sensors with power/energy units.""" + sensor_energy, _, _, sensor_power = setup_dummy_sensors + schema = FlexContextSchema() + result = schema.load({"inflexible-generators": [sensor_energy.id, sensor_power.id]}) + assert "inflexible_generators" in result + assert len(result["inflexible_generators"]) == 2 + assert result["inflexible_generators"][0].id == sensor_energy.id + assert result["inflexible_generators"][1].id == sensor_power.id + + +def test_flex_context_inflexible_device_sensors_emits_warning( + db, app, setup_dummy_sensors +): + """inflexible-device-sensors still deserializes correctly but emits a UserWarning.""" + sensor_energy, _, _, sensor_power = setup_dummy_sensors + schema = FlexContextSchema() + with pytest.warns(UserWarning, match="inflexible-device-sensors.*deprecated"): + result = schema.load( + {"inflexible-device-sensors": [sensor_energy.id, sensor_power.id]} + ) + assert "inflexible_device_sensors" in result + assert len(result["inflexible_device_sensors"]) == 2 + + +def test_flex_context_inflexible_device_sensors_invalid_unit( + db, app, setup_dummy_sensors +): + """inflexible-device-sensors raises ValidationError for non-power/energy sensor units.""" + # sensor3 has unit "EUR" — not a power or energy unit + _, _, sensor_price, _ = setup_dummy_sensors + schema = DBFlexContextSchema() + with pytest.raises(ValidationError) as exc_info: + schema.load({"inflexible-device-sensors": [sensor_price.id]}) + assert "inflexible-device-sensors" in exc_info.value.messages + + +# --- Tests for MultiSensorFlexModelSchema --- + + +def test_multi_sensor_flex_model_consumption(db, app, setup_dummy_sensors): + """consumption key maps to sensor with is_consumption_sensor=True and wraps extra fields.""" + sensor, _, _, _ = setup_dummy_sensors + schema = MultiSensorFlexModelSchema() + result = schema.load({"consumption": sensor.id, "soc-at-start": "10 kWh"}) + assert result["sensor"].id == sensor.id + assert result["is_consumption_sensor"] is True + assert result["sensor_flex_model"] == {"soc-at-start": "10 kWh"} + + +def test_multi_sensor_flex_model_production(db, app, setup_dummy_sensors): + """production key maps to sensor with is_consumption_sensor=False.""" + sensor, _, _, _ = setup_dummy_sensors + schema = MultiSensorFlexModelSchema() + result = schema.load({"production": sensor.id}) + assert result["sensor"].id == sensor.id + assert result["is_consumption_sensor"] is False + assert result["sensor_flex_model"] == {} + + +def test_multi_sensor_flex_model_consumption_and_production_raises( + db, app, setup_dummy_sensors +): + """Specifying both consumption and production raises ValidationError.""" + sensor, _, _, _ = setup_dummy_sensors + schema = MultiSensorFlexModelSchema() + with pytest.raises(ValidationError) as exc_info: + schema.load({"consumption": sensor.id, "production": sensor.id}) + # The pre_load hook raises ValidationError with a plain message + assert "consumption" in str(exc_info.value) or "production" in str(exc_info.value) + + +def test_multi_sensor_flex_model_deprecated_sensor_field_emits_warning( + db, app, setup_dummy_sensors +): + """Old sensor key still works but emits a UserWarning about the deprecated field.""" + sensor, _, _, _ = setup_dummy_sensors + schema = MultiSensorFlexModelSchema() + with pytest.warns(UserWarning, match="`sensor`.*deprecated"): + result = schema.load({"sensor": sensor.id}) + assert result["sensor"].id == sensor.id + assert result["is_consumption_sensor"] is None diff --git a/flexmeasures/data/services/scheduling.py b/flexmeasures/data/services/scheduling.py index 8c32c01c72..409dd4f9f3 100644 --- a/flexmeasures/data/services/scheduling.py +++ b/flexmeasures/data/services/scheduling.py @@ -381,6 +381,9 @@ def create_sequential_scheduling_job( previous_job = depends_on for child_flex_model in flex_model: sensor = child_flex_model.pop("sensor") + # Extract the explicit sign convention if set via 'consumption' or 'production' fields. + # None means "fall back to the sensor's own `consumption_is_positive` attribute". + is_consumption_sensor = child_flex_model.pop("is_consumption_sensor", None) current_scheduler_kwargs = deepcopy(scheduler_kwargs) @@ -393,6 +396,8 @@ def create_sequential_scheduling_job( if "resolution" not in current_scheduler_kwargs: current_scheduler_kwargs["resolution"] = sensor.event_resolution current_scheduler_kwargs["asset_or_sensor"] = sensor + if is_consumption_sensor is not None: + current_scheduler_kwargs["is_consumption_sensor"] = is_consumption_sensor job = create_scheduling_job( **current_scheduler_kwargs, @@ -534,6 +539,7 @@ def make_schedule( # noqa: C901 flex_config_has_been_deserialized: bool = False, scheduler_specs: dict | None = None, dry_run: bool = False, + is_consumption_sensor: bool | None = None, **scheduler_kwargs: dict, ) -> bool: """ @@ -612,6 +618,7 @@ def make_schedule( # noqa: C901 "name": "consumption_schedule", "data": consumption_schedule, "sensor": asset_or_sensor, + "is_consumption_sensor": is_consumption_sensor, } ] @@ -639,10 +646,16 @@ def make_schedule( # noqa: C901 sign = 1 - if result["sensor"].measures_power and not result["sensor"].get_attribute( - "consumption_is_positive", False - ): - sign = -1 + if result["sensor"].measures_power: + # Use the explicit sign convention if set (via 'consumption' or 'production' field), + # otherwise fall back to the sensor's own `consumption_is_positive` attribute. + consumption_is_positive = result.get("is_consumption_sensor") + if consumption_is_positive is None: + consumption_is_positive = result["sensor"].get_attribute( + "consumption_is_positive", False + ) + if not consumption_is_positive: + sign = -1 ts_value_schedule = [ TimedBelief(