feat: Floor datetimes on ingestion

[BelhsanHmida]

Description

This PR addresses Round datetimes on ingestion #1767 by implementing the FlexMeasures-side flooring behavior for off-clock datetimes.

The main goal of this branch is to make ingestion and scheduling more robust when datetimes are slightly off the expected sensor clock, such as 10:00:40 instead of 10:00:00.

• Floor the start datetime for direct POST sensor data on non-instantaneous sensors
• Floor datetimes in uploaded sensor data before saving and before any further handling
• Floor flex-model datetimes such as soc-minima, soc-maxima, and soc-targets before scheduling
• Keep the normal POST endpoint behavior otherwise aligned with the previous compatibility rules
• Added changelog item in documentation/changelog.rst

Look & Feel

N/A

Examples of the behavior in this branch:

• direct POST: 2025-11-21T10:00:40+01:00 -> 2025-11-21T10:00:00+01:00
• upload datetimes: 2025-11-21T10:15:40+01:00 -> 2025-11-21T10:15:00+01:00
• flex-model datetimes: soc-minima.datetime=2015-01-02T23:00:40+01:00 -> 2015-01-02T23:00:00+01:00

How to test

Run the relevant test modules:

• uv run pytest flexmeasures/api/common/schemas/tests/test_sensor_data_schema.py -vv
• uv run pytest flexmeasures/api/v3_0/tests/test_sensor_data.py -vv
• uv run pytest flexmeasures/api/v3_0/tests/test_sensor_data_fresh_db.py -vv
• uv run pytest flexmeasures/api/v3_0/tests/test_sensor_schedules.py -vv
• uv run pytest flexmeasures/api/v3_0/tests/test_sensors_api_freshdb.py -vv

Further Improvements

• Add a changelog entry once the PR scope is finalized
• Revisit broader mismatched-period resampling separately if that is still desired for upload flows or future follow-up work

Related Items

• Addresses Round datetimes on ingestion #1767
• Builds on the earlier spike in Floor event start #1828
• Keeps the current PR scoped to FlexMeasures-side datetime flooring, without requiring the Timely Beliefs spike branch

Closes #1767

---

Sign-off

• I agree to contribute to the project under Apache 2 License.
• To the best of my knowledge, the proposed patch is not based on code under GPL or other license that is incompatible with FlexMeasures

[Copilot]

Pull request overview

This PR implements datetime flooring at multiple ingestion points (direct POST, file upload, and scheduling flex-model inputs) to make ingestion/scheduling robust against slightly off-clock timestamps, as discussed in issue #1767.

Changes:

• Add support for flooring timed-event datetime/start/end fields based on a configurable event resolution in schema validation.
• Floor event starts in uploaded sensor data before further handling/resampling, gated by round_datetimes_on_ingestion.
• Floor flex-model timed-event datetimes (soc-minima/maxima/targets) before enqueuing scheduling jobs, plus add/adjust tests for all three flows.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
flexmeasures/data/schemas/sensors.py	Adds timed-event flooring support and floors uploaded-data event starts before conversions/resampling.
flexmeasures/data/schemas/scheduling/storage.py	Passes a rounding resolution into flex-model schemas so timed-event datetimes can be floored consistently.
flexmeasures/api/v3_0/sensors.py	Floors flex-model timed-event datetimes prior to job creation in trigger_schedule.
flexmeasures/api/common/schemas/sensor_data.py	Floors direct-POST start for non-instantaneous sensors and adds validation for the zero-duration single-value edge case.
flexmeasures/api/v3_0/tests/test_sensors_api_freshdb.py	Adds coverage for flooring behavior during upload ingestion.
flexmeasures/api/v3_0/tests/test_sensor_schedules.py	Adds coverage for flooring flex-model datetimes prior to scheduling.
flexmeasures/api/v3_0/tests/test_sensor_data.py	Adds coverage for flooring start in direct POSTs for non-instantaneous sensors.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[BelhsanHmida]

@copilot can you also add add a regression test to check that an invalid flex-model datetime still returns a 422 validation error instead of causing a 500.

[Copilot]

@copilot can you also add add a regression test to check that an invalid flex-model datetime still returns a 422 validation error instead of causing a 500.

Implemented in 3a56e0c: I added a regression test (test_trigger_schedule_with_invalid_flex_model_datetime) that sends an invalid flex-model datetime and verifies the API returns 422, and I fixed the trigger path to convert the flooring validation error into unprocessable_entity(...) instead of returning a 500.

[Flix6x]

• An addition to the documentation seems in order, perhaps at the end of this section.
• We are introducing an undocumented sensor attribute.
• Discussion point: is flooring always the best choice? I see no examples like an soc-target at 2026-05-12T08:29:58+02, which would now be floored to 08:15.
• Discussion point: for the flex-config, the conversion could also happen after the job is picked up rather than before the job is created. Otherwise, we are not storing the user's original intention in the job description. I wonder which choice would lead to fewer questions.

[Flix6x]

What is the historical purpose of this attribute? How does that relate to the new functionality?

[BelhsanHmida]

I looked into this a bit more. The historical purpose of frequency seems to be to align incoming measurements to a configured Pandas frequency by rounding. That is useful when a sensor stores instantaneous data, where event_resolution is zero, but we still want incoming point measurements to land on a predictable time grid.

This PR adds a different behavior for non-instantaneous sensors. For those, the sensor already has an interval grid through event_resolution, so off-clock starts are floored to that resolution before saving. So they are related in that both normalize incoming datetimes, but frequency is a custom rounding grid, while the new behavior is flooring to the sensor’s own event resolution.

I renamed the new flag to floor_datetimes_to_resolution to avoid mixing up those two concepts, and documented the distinction.

[BelhsanHmida]

• An addition to the documentation seems in order, perhaps at the end of this section.
• We are introducing an undocumented sensor attribute.
• Discussion point: is flooring always the best choice? I see no examples like an soc-target at 2026-05-12T08:29:58+02, which would now be floored to 08:15.
• Discussion point: for the flex-config, the conversion could also happen after the job is picked up rather than before the job is created. Otherwise, we are not storing the user's original intention in the job description. I wonder which choice would lead to fewer questions.

I addressed first two points concerning the documentation and attribute parts in the latest commits: floor_datetimes_to_resolution is now documented in the API notation docs, sensor schema/OpenAPI examples, and changelog.

I see the point. For flex-model constraints, flooring before job creation can change both the user’s submitted request and what we store in the job kwargs.

Your soc-target example shows the semantic risk: flooring 08:29:58 to 08:15 makes the target earlier and potentially stricter than intended.

My guess is that preserving the submitted flex-model in the job kwargs would lead to fewer questions. The job would show exactly what the user sent, which makes debugging easier, and the scheduler could still normalize internally when interpreting the constraints.

So my preferred direction would be to move flex-model datetime normalization out of the API trigger path and into scheduler deserialization/execution. Another option is to keep the current behavior but document clearly that timed flex-model fields are floored before the job is stored. I lean toward the first option.

[Flix6x]

I agree with keeping the submitted flex-model in the job kwargs. For the normalization of soc-minima, soc-maxima and soc-targets I feel we can stay a little closer to the user's intent. I propose to study #10 (comment) and come up with a more elegant proposal than blindly flooring.

I also suggest to always turn on relax-soc-constraints if those fields are not originally on the tick, just in case more complex combinations would lead to infeasibilities.