[Bug] v1.17.0 - Altitude spikes in fixed wing mode

[afwilkin]

Describe the bug

When flying in autonomous flight modes, the vehicle's altitude has various spikes that are not getting filtered out. This caused the fixed-wing drone to take a few nose dives and almost crash while flying.

I will note that the vibrations on the vehicle were fairly high as this was a very windy day.

In order to recreate:

1. Take off a fixed-wing UAV.
2. Fly it around.
3. Switch it into mission mode.
4. Watch altitude

Flight Log / Additional Information

https://review.px4.io/plot_app?log=d8852605-ac7e-46fa-bbff-8fd977b24f0e

Version: 1.17 RC 2

Explanation: https://www.loom.com/share/d07207cdb2f6448facec4f6f61e0e680

[mrpollo]

@ryanjAA Can you take a look at the logs here? you have a keen eye for these things

[ryanjAA]

Buc-ees hat on :) Standby

[ryanjAA]

Multi Ekf issue. Specifically an estimator selector problem.

EKF0 declares a baro fault at t=200s (cs_baro_fault goes true). After that its baro height innovations go completely off the rails, mean innovation of -571m, test ratios over 3000. Its z estimate diverges from EKF1 by up to 105m...

Te prob here is the selector keeps EKF0 as healthy (healthy[0] = 1.0 the entire flight) despite the baro fault. EKF1 gets intermittently marked unhealthy even though its baro innov are totally normal (test ratio max 0.55). To me it looks like combined_test_ratio[1] isn't even being populated it's NaN globally so the selector has nothing to actually compare against and just falls back to instance 0. Boooo

The result is 16 instance switches during flight, each one causing an instantaneous altitude step in vehicle_local_position and your jumps.. The worst one is the the 128m jump at t=255s. TECS sees that as altitude error and pitches down hard, which is where the nose dives come from. Also booo

FYI - vibs are high (accel_vibration_metric peaks ~28.5) which probably or at least possibly caused the baro fault on EKF0 in the first place, but that's not the real issue. The real issue is the selector keeps going back to an instance with a known height source fault and wildly divergent state. EKF1 was tracking fine the whole time.

Im inclined to call this a 1.17 estimator selector regression (or bug)... where baro faults on one EKF instance don't properly disqualify it from selection.

At @mrpollo this is a blocker.

Buc-ee's hat off

Edit: Markdown added for dramatic effect

[mrpollo]

Thanks for the overview @ryanjAA. I'm going to try to reproduce in SITL, on the release and main branches

[dakejahl]

Took an independent look at the log with Claude. Ryan's overall conclusion is correct — this is a selector bug and a blocker — but the mechanism is different from what he described.

What the log actually shows:

• EKF0 (BMI055) declares cs_baro_fault at t=199.5s, stops fusing baro, switches to GPS-only height. During this transition its Z-state diverges up to 155.6m from EKF1.
• EKF1 (ICM42688P) has severe Z-axis accel clipping (up to 33,565 clipping events/window) due to vibrations, which triggers intermittent filter_fault_flags — bit 11 (bad_acc_clipping) and bit 10 (bad_acc_vertical) across 6 brief fault periods.
• Every single 1→0 switch (7 of them) is triggered by EKF1 going unhealthy from these clipping faults. The selector falls back to EKF0 — the only "healthy" instance — despite its Z-state being off by up to 128m at the time of the switch.
• 15 instance switches total, 14 altitude jumps >5m, worst is 128.5m at t=255s.

Correction on Ryan's analysis: combined_test_ratio[1] is NOT NaN globally — only 1/4365 samples is NaN. The selector can and does compare instances, and relative_test_ratio correctly identifies EKF1 as better (mean -0.856). The problem isn't that the selector can't compare; it's that EKF1's intermittent clipping faults force the selector back to a diverged EKF0.

The bug (in EKF2Selector.cpp:298):

const bool healthy = (status.filter_fault_flags == 0) && (combined_test_ratio > 0.f);

Two issues:

1. cs_baro_fault is a control status flag, not a filter_fault_flag — the selector is completely blind to sensor source faults. EKF0 stays "healthy" with 155m of Z divergence.
2. This is a regression from b7efd4f947 (Dec 2021) which removed (combined_test_ratio < 1.f) from the health check (previously added in fc155bc57e). Under the original rule, EKF0 would also have been unhealthy (its combined_test_ratio is pegged at 2.0), preventing the oscillation back to a diverged instance.

The selector's health model treats brief IMU clipping faults as more severe than a persistent baro fault with massive state divergence. That's backwards for this failure mode.

---

Analysis performed with Claude Code (claude-opus-4-6)

[ryanjAA]

This Claude thing is so cool. We need to just enable it on all things. Thanks for my correction. Next time will leave hat on longer and not rush

[mrpollo]

Cross-checked against ulog-convert and the onboard logger messages. Confirms Jake. Reframing first, then three things worth adding. Verified against release/1.17 (flight firmware sha 0b6e4687defb is the current tip); code is identical on main, so any fix applies to both.

The upstream cause of this chain is high accel vibration, not the EKF. accel_vibe_max = 34.37 m/s². IMU1 (ICM42688P) sensor_accel.z is railing at [-127.1, +109.6] m/s² (roughly ±13g, getting close to the ±16g rail), with a peak of 33,565 clipping events in one 1s window on the Z axis. All 17 "Accel 1 clipping, not safe to fly!" logger warnings are on IMU1. Whether this is airframe (isolators, prop balance, motor mount) or wind-induced is worth checking before blaming hardware. I pulled historical weather for the location and time of the flight (41.90°N, 87.70°W, 2026-04-08 16:21 UTC, near Chicago): wind around 4.2 m/s with gusts near 9.8 m/s (roughly 22 mph), pressure falling 1 hPa/hr, temp around 9°C. Andrew, you described it as "very windy" but the reanalysis data shows moderate not exceptional conditions. Gusts of that magnitude can drive some vibration on a fixed-wing but 33k clipping events per second and a sustained 34 m/s² peak looks larger than turbulence alone would produce, so the airframe almost certainly contributes as well. How's the prop balance and isolator condition on this airframe? Has it flown clean before, or is this vibration profile typical?

With that said, the selector bug is the blocker, because PX4 is supposed to gracefully handle a single IMU going bad. Here's what the firmware adds on top:

The switch messages are in the log itself. 7 explicit [ekf2] primary EKF changed 1 (filter fault) -> 0 entries, each preceded within ~1s by an [vehicle_imu] Accel 1 clipping warning. Returns 0→1 aren't logged because they go through the lower_error_available path, not the fault path, which is why the explicit count (7) is less than instance_changed_count (15).

Small corrections on the bit indices. bad_acc_vertical and bad_acc_clipping are bits 16 and 17 in filter_fault_flags, not 10 and 11 (per EstimatorStatus.msg:71-72 on release/1.17). The bits visible in the logged stream (1024 and 2048) are bits 10/11 = bad_vel_east / bad_vel_down. The acc clipping bit toggles at EKF rate (100 Hz) and is almost always missed by the 5 Hz estimator_status logger, but estimator_status_flags.fs_bad_acc_clipping captures it at 100 Hz and lines up exactly with every 1→0 selector transition.

There's a second reason the selector is stuck, beyond the cs_baro_fault blindness. The 2021 commit b7efd4f947 that removed the ctr < 1 clause also added an escape hatch: if the primary has been in warning for >1s and an alternate has relative_test_ratio <= -EKF2_SEL_ERR_RED, force-switch. It should have fired. combined_test_ratio[0] mean over the bad window is 1.42 (max 2.0, pinned) and relative_test_ratio[1] mean is -0.98. But the path is gated on hrt_elapsed_time(&_instance[i].time_last_selected) > 10_s at EKF2Selector.cpp:765 (release/1.17). Every time the whipsaw briefly lands on EKF1, that timer resets. The one path specifically designed to rescue this situation is defeated by the whipsaw it's trying to prevent. Fix idea: use time_last_healthy there, or bypass the 10s guard when the current primary's combined_test_ratio has been saturated for more than some minimum duration.

Third fix worth making, closer to the source: bad_acc_clipping has no debounce. bad_acc_vertical already uses isRecent(_time_bad_vert_accel, BADACC_PROBATION) in height_control.cpp:173 (release/1.17) to latch for a probation period. Applying the same pattern to bad_acc_clipping at height_control.cpp:155 would kill the 100 Hz toggling at the source and remove the selector-level triggers entirely, independent of the cs_baro_fault fix.

Max |EKF0.z minus EKF1.z| = 155.63 m at t=260.45s. vehicle_local_position.delta_z range [-179.5, +129.1] m.

Config note: EKF2_MULTI_IMU=3 (multi-EKF enabled for redundancy) and SENS_BAR_AUTOCAL=0. Baro autocal being off may be part of why EKF0 declared cs_baro_fault in the first place, but that's secondary.

Board: PX4_FMU_V6C, v1.17.0-rc (0b6e4687defb). IMU0 = BMI055, IMU1 = ICM42688P (devtype decode from the logged device IDs).

[afwilkin]

Cross-checked against ulog-convert and the onboard logger messages. Confirms Jake. Reframing first, then three things worth adding. Verified against release/1.17 (flight firmware sha 0b6e4687defb is the current tip); code is identical on main, so any fix applies to both.

The upstream cause of this chain is high accel vibration, not the EKF. accel_vibe_max = 34.37 m/s². IMU1 (ICM42688P) sensor_accel.z is railing at [-127.1, +109.6] m/s² (roughly ±13g, getting close to the ±16g rail), with a peak of 33,565 clipping events in one 1s window on the Z axis. All 17 "Accel 1 clipping, not safe to fly!" logger warnings are on IMU1. Whether this is airframe (isolators, prop balance, motor mount) or wind-induced is worth checking before blaming hardware. I pulled historical weather for the location and time of the flight (41.90°N, 87.70°W, 2026-04-08 16:21 UTC, near Chicago): wind around 4.2 m/s with gusts near 9.8 m/s (roughly 22 mph), pressure falling 1 hPa/hr, temp around 9°C. Andrew, you described it as "very windy" but the reanalysis data shows moderate not exceptional conditions. Gusts of that magnitude can drive some vibration on a fixed-wing but 33k clipping events per second and a sustained 34 m/s² peak looks larger than turbulence alone would produce, so the airframe almost certainly contributes as well. How's the prop balance and isolator condition on this airframe? Has it flown clean before, or is this vibration profile typical?

With that said, the selector bug is the blocker, because PX4 is supposed to gracefully handle a single IMU going bad. Here's what the firmware adds on top:

The switch messages are in the log itself. 7 explicit [ekf2] primary EKF changed 1 (filter fault) -> 0 entries, each preceded within ~1s by an [vehicle_imu] Accel 1 clipping warning. Returns 0→1 aren't logged because they go through the lower_error_available path, not the fault path, which is why the explicit count (7) is less than instance_changed_count (15).

Small corrections on the bit indices. bad_acc_vertical and bad_acc_clipping are bits 16 and 17 in filter_fault_flags, not 10 and 11 (per EstimatorStatus.msg:71-72 on release/1.17). The bits visible in the logged stream (1024 and 2048) are bits 10/11 = bad_vel_east / bad_vel_down. The acc clipping bit toggles at EKF rate (100 Hz) and is almost always missed by the 5 Hz estimator_status logger, but estimator_status_flags.fs_bad_acc_clipping captures it at 100 Hz and lines up exactly with every 1→0 selector transition.

There's a second reason the selector is stuck, beyond the cs_baro_fault blindness. The 2021 commit b7efd4f947 that removed the ctr < 1 clause also added an escape hatch: if the primary has been in warning for >1s and an alternate has relative_test_ratio <= -EKF2_SEL_ERR_RED, force-switch. It should have fired. combined_test_ratio[0] mean over the bad window is 1.42 (max 2.0, pinned) and relative_test_ratio[1] mean is -0.98. But the path is gated on hrt_elapsed_time(&_instance[i].time_last_selected) > 10_s at EKF2Selector.cpp:765 (release/1.17). Every time the whipsaw briefly lands on EKF1, that timer resets. The one path specifically designed to rescue this situation is defeated by the whipsaw it's trying to prevent. Fix idea: use time_last_healthy there, or bypass the 10s guard when the current primary's combined_test_ratio has been saturated for more than some minimum duration.

Third fix worth making, closer to the source: bad_acc_clipping has no debounce. bad_acc_vertical already uses isRecent(_time_bad_vert_accel, BADACC_PROBATION) in height_control.cpp:173 (release/1.17) to latch for a probation period. Applying the same pattern to bad_acc_clipping at height_control.cpp:155 would kill the 100 Hz toggling at the source and remove the selector-level triggers entirely, independent of the cs_baro_fault fix.

Max |EKF0.z minus EKF1.z| = 155.63 m at t=260.45s. vehicle_local_position.delta_z range [-179.5, +129.1] m.

Config note: EKF2_MULTI_IMU=3 (multi-EKF enabled for redundancy) and SENS_BAR_AUTOCAL=0. Baro autocal being off may be part of why EKF0 declared cs_baro_fault in the first place, but that's secondary.

Board: PX4_FMU_V6C, v1.17.0-rc (0b6e4687defb). IMU0 = BMI055, IMU1 = ICM42688P (devtype decode from the logged device IDs).

THis airframe is by no means perfect. Under stable conditions the vibrations still increase, but not as aggressively as they did today. the tuning could also use some adjustments and probably make moderate vibrations worse. But, obviously this revealed something in the ekf2