Repository navigation
AC_AutoTune: don't run level-timeout while yaw target is slewing to desired - #33628
peterbarker wants to merge 3 commits into
Conversation
| @@ -332,6 +332,15 @@ bool AC_AutoTune::currently_level() | |||
| // reset if the target yaw rate is above half the slew rate | |||
| level_start_time_ms = now_ms; | |||
| } | |||
| if (fabsf(wrap_PI(attitude_control->get_att_target_euler_rad().z - desired_yaw_rad)) > cd_to_rad(AUTOTUNE_LEVEL_ANGLE_CD)) { | |||
| // the attitude controller is still slewing its yaw target towards | |||
There was a problem hiding this comment.
This is all correct but it does basically what the check above does but with a smaller window in some cases.
| @@ -332,6 +332,15 @@ bool AC_AutoTune::currently_level() | |||
| // reset if the target yaw rate is above half the slew rate | |||
| level_start_time_ms = now_ms; | |||
| } | |||
| if (fabsf(wrap_PI(attitude_control->get_att_target_euler_rad().z - desired_yaw_rad)) > cd_to_rad(AUTOTUNE_LEVEL_ANGLE_CD)) { | |||
| // the attitude controller is still slewing its yaw target towards | |||
There was a problem hiding this comment.
This is all correct but it does basically what the check above does but with a smaller window in some cases.
There was a problem hiding this comment.
Maybe the fix could simply be:
if (fabsf(attitude_control->get_rate_ef_target_rads().z) > cd_to_rad(AUTOTUNE_LEVEL_RATE_Y_CD)) {
That would be consistent with the checks below but on the target.
There was a problem hiding this comment.
I let the robot loose on this as I can get this to relatively-reliably fail here:
● All 117 commits rebased clean onto the QAUTOTUNE branch. Full status:
PR #33628 is updated — pushed with: fresh-master rebase, the angle-guard restored, and the commit message now carrying the decisive new evidence: "a rate-target-based deferral
(AUTOTUNE_LEVEL_RATE_Y_CD) was tried and is not sufficient — a shaped re-point's spin-up/spin-down tails run below the rate threshold while the target is still far from desired; it failed
the deterministic repro 2 runs out of 4" — plus a code comment at the guard saying the same. Validation on the pushed form: repro 4/4, standard QAUTOTUNE pass, all on current master.
Suggested reply to Leonard (yours to post): "Tried your suggestion — it's cleaner and I wanted it to work, but the low-yaw-acceleration repro fails 2/4 with it: the shaped traverse's
spin-up and spin-down tails sit below LEVEL_RATE_Y while the target is still tens of degrees from desired, and with Q_A_ACC_Y_MAX=5 those tails alone eat the 6s budget. The angle-based
form freezes through the tails and passes 8/8, so I've kept it (and the existing half-slew check), with a comment recording why a rate-target check isn't sufficient."
Working branch now carries the QAUTOTUNE fix underneath everything, so every future sweep validates it continuously — exactly your intent. Threads deployment is deliberately held: the
zero_turn hunt is mid-loop there and a reset-plus-rebuild would disturb it; it deploys the moment the hunt captures or completes.
One process note worth saying plainly: Leonard's suggestion looked right to me too — the validation harness on that branch is what caught the difference. Good PR hygiene paid for itself
twice today.
110a3a7 to
5e263ab
Compare
5e263ab to
ce9c8ff
Compare
ce9c8ff to
c043bf3
Compare
Previous review (2026-09-16)Automated review note — AI-generated (Claude), validated against the live diff. Please sanity-check before acting. Reviewed at head Full report, including everything that was checked and found clean: https://uav.tridgell.net/DevCallReviews/2026_09_17_AIReview/devcall_pr_reviews.html#pr33628 Verdict: APPROVE The obvious risk — that suppressing the level timeout during a yaw slew turns a bounded wait into a hang — does not exist: the new Lower-priority notes (3)
What was checked and found cleanNo hang is possible; the timeout is deferred, never disabled. Every write to the two clocks was traced. All four Units and angle wrap are correct, and the new guard is not a no-op. The new test is a genuine regression test and is not flaky. Proven by mutation: reverting only |
|
@peterbarker can you rebase? The QAUTOTUNE test is failing quite a lot on master |
208c29d to
e25311b
Compare
Done |
Previous review (2026-10-05)Automated review note — AI-generated (Claude+Codex), validated against the live diff. Please sanity-check before acting. Reviewed at head Re-checked after the rebase. The PR was at c043bf3 when last reviewed and is now at e25311b. The rebase changed no code; AC_AutoTune is byte-identical to what was reviewed before. At the new head the level wait is still bounded at about 26 s, since all four WAITING_FOR_LEVEL entries reset the new clock (https://github.com/ArduPilot/ardupilot/pull/33628/files#diff-fbca931a7b0f35b820a756508a4224940a91fa351fa84f259de890179f3e83d0R338). On Plane SITL, QAUTOTUNEYawRepoint passed 6/6 and QAUTOTUNE passed 6/6. With AC_AutoTune reverted, QAUTOTUNEYawRepoint failed 3/3 and QAUTOTUNE failed 1/6, matching the flakiness seen on master. Copter AutoTune and AutoTuneYawD pass. One optional hardening: the new test reads self.re_match (https://github.com/ArduPilot/ardupilot/pull/33628/files#diff-f8a09249c37bc1b530e461d3f60f0c230f9edf735acbf501eb32752bf0e44675R750), which is not set when the text was already collected. That cannot happen in this test, and the existing QAUTOTUNE test does the same, so no change is requested. The older out-of-scope notes still stand for a follow-up: the missing fabsf() on the gyro level checks at libraries/AC_AutoTune/AC_AutoTune.cpp:376-382, and the 2*/3* comment mismatch at :332. Not re-tested here: the minimum-slew/very-low-acceleration corner, lthall's rate-only variant, and Heli. CI at this head was still pending. |
Global Size Summary (Total Flash delta in bytes)Built from fb47bdf rebased onto a42f84a
|
e25311b to
fb5b05b
Compare
Previous review (2026-10-17)Automated review note — AI-generated (Claude+Codex), validated against the live diff. Please sanity-check before acting. Reviewed at head Re-checked after another rebase. The last review was at e25311b; the PR is now at fb5b05b on afeb9c4, and a range-diff shows both commits unchanged. The level wait is still bounded at about 26 s, since all four WAITING_FOR_LEVEL entries reset the new clock and the gate is its only read (https://github.com/ArduPilot/ardupilot/pull/33628/files#diff-fbca931a7b0f35b820a756508a4224940a91fa351fa84f259de890179f3e83d0R338). On Plane SITL at this head, QAUTOTUNEYawRepoint passed 4/4 and QAUTOTUNE passed 4/4. With AC_AutoTune reverted, QAUTOTUNEYawRepoint failed 3/3 with 'Failed to level'. Copter and Heli build. Optional, no change requested: the test reads the shared self.re_match (https://github.com/ArduPilot/ardupilot/pull/33628/files#diff-f8a09249c37bc1b530e461d3f60f0c230f9edf735acbf501eb32752bf0e44675R772). It can be stale or None if a terminal text arrives as autotune mode is entered, but that needs at least 6 s in the mode, and the existing QAUTOTUNE test does the same. The earlier out-of-scope notes still stand: missing fabsf() on the gyro level checks at libraries/AC_AutoTune/AC_AutoTune.cpp:376-382, and the 2*/3* comment mismatch at :332. Not tested here: Copter AutoTune SITL, the minimum-slew/very-low-acceleration corner, lthall's rate-only variant, and Heli. CI at this head was still pending. |
Previous review (2026-10-17)Automated review note — AI-generated (Claude+Codex), validated against the live diff. Please sanity-check before acting. Reviewed at head The head moved from fb5b05b to b6d1616, which adds the settle-window commit. The first two commits are unchanged, and this review covers b6d1616. The level wait is still bounded at about 26 s. The new settle block is gated by the same may_defer_timeout check (https://github.com/ArduPilot/ardupilot/pull/33628/files#diff-fbca931a7b0f35b820a756508a4224940a91fa351fa84f259de890179f3e83d0R370), and all four WAITING_FOR_LEVEL entries reset its clock. On SITL at this head, QAUTOTUNEYawRepoint passed 40/40 and QAUTOTUNE 6/6, and Copter AutoTune, AutoTuneYawD and Helicopter AutoTune pass. With AC_AutoTune reverted, QAUTOTUNEYawRepoint fails 3/3 with 'Failed to level'. Note, no change requested: during the settle window (https://github.com/ArduPilot/ardupilot/pull/33628/files#diff-fbca931a7b0f35b820a756508a4224940a91fa351fa84f259de890179f3e83d0R374) the level tolerance is zero, so levelling waits RATE_WPY_MAX/ACC_Y_MAX after the last slew, as the existing deferrals already do. With Q_A_ACC_Y_MAX=5 the test's tune took 315-509 s against 302-381 s before this commit. Still optional: the shared self.re_match read (https://github.com/ArduPilot/ardupilot/pull/33628/files#diff-f8a09249c37bc1b530e461d3f60f0c230f9edf735acbf501eb32752bf0e44675R772), the missing fabsf() on the gyro level checks at libraries/AC_AutoTune/AC_AutoTune.cpp:396-404, and the 2*/3* comment mismatch at :332. Not tested: the minimum-slew/very-low-acceleration corner, lthall's rate-only variant, the intermittent pre-fix failure commit 3 targets, and hardware. CI at this head was pending. |
b6d1616 to
f967d44
Compare
Previous review (2026-10-20)Automated review note — AI-generated (Claude+Codex), validated against the live diff. Please sanity-check before acting. Reviewed at head The head moved from b6d1616 to f967d44. This is a pure rebase onto fa78fe1: all three commits are unchanged and the diff is byte-identical. This review covers f967d44. The level wait is still bounded at about 26 s (https://github.com/ArduPilot/ardupilot/pull/33628/files#diff-fbca931a7b0f35b820a756508a4224940a91fa351fa84f259de890179f3e83d0R338). On SITL at this head, QAUTOTUNEYawRepoint passed 6/6 and QAUTOTUNE 2/2. With AC_AutoTune reverted to the merge base, QAUTOTUNEYawRepoint fails 3/3 with 'Failed to level'. Note, no change requested: during the settle window (https://github.com/ArduPilot/ardupilot/pull/33628/files#diff-fbca931a7b0f35b820a756508a4224940a91fa351fa84f259de890179f3e83d0R374), levelling waits RATE_WPY_MAX/ACC_Y_MAX after the last slew. The test's tune took 316-473 s. Still optional and unchanged: the shared self.re_match read (https://github.com/ArduPilot/ardupilot/pull/33628/files#diff-f8a09249c37bc1b530e461d3f60f0c230f9edf735acbf501eb32752bf0e44675R772), the missing fabsf() on the gyro level checks at libraries/AC_AutoTune/AC_AutoTune.cpp:396-404 (it predates this PR), and the 2*/3* comment mismatch at :332. Not re-run at this head: Copter and Heli AutoTune (the code is identical to the previous head, where they passed). Not tested: the low-acceleration corner, lthall's variant and hardware. CI passed 110/110 at b6d1616; at f967d44 it was still pending. |
f967d44 to
64317d4
Compare
Previous review (2026-11-01)Automated review note — AI-generated (Claude+Codex), validated against the live diff. Please sanity-check before acting. Reviewed at head The head moved from f967d44 to 64317d4 in another pure rebase: all three commits and the AC_AutoTune code are unchanged. This review covers 64317d4, and nothing needed fixing since the last review. The level wait is still bounded at about 26 s (https://github.com/ArduPilot/ardupilot/pull/33628/files#diff-fbca931a7b0f35b820a756508a4224940a91fa351fa84f259de890179f3e83d0R338). On SITL at this head, QAUTOTUNEYawRepoint passed 6/6 and QAUTOTUNE 2/2, and with AC_AutoTune reverted to the merge base the new test fails 3/3. Copter AutoTune, AutoTuneYawD and Heli AutoTune pass too. Optional
Minor (1)
Not tested: the low-acceleration/minimum-slew corner, lthall's variant and hardware. CI passed 110/110 at f967d44; at 64317d4 it was still running. |
…esired When position hold commands a large yaw re-point (get_poshold_attitude points along the drift direction, +90deg when tuning pitch), the attitude controller slews its yaw target towards the new desired yaw over several seconds. currently_level() measures yaw against the final desired yaw, so it reports not-level for the whole traverse and the 3 * AUTOTUNE_LEVEL_TIMEOUT_MS abort clock races the traverse: a ~125deg re-point at the ~25deg/s shaped slew rate takes ~5.5s against the 6s abort, and on the losing side of the race the tune dies with "AutoTune: Failed to level, please tune manually". The existing guard (eb3c110) resets the clock when the target yaw rate exceeds half the slew limit, but a long smooth re-point runs well below that: in a failing QuadPlane CI log the shaped rate peaked around 28deg/s against a 30deg/s threshold (MIN(Q_A_RATE_Y_MAX=75, Q_A_RATE_WPY_MAX=60)/2), the clock was never reset and the abort fired exactly 6.000s after the level-wait began -- 0.2s before yaw would have arrived; roll and pitch were within 0.1deg throughout. Also reset the clock while the shaped yaw target is more than the level threshold away from desired_yaw_rad: the vehicle cannot possibly be at the desired yaw before its commanded target arrives there, so the timeout should only run once success is geometrically possible. Neither deferral may postpone the abort indefinitely, though: position hold recomputes desired_yaw_rad from the drift bearing on every run(), so a bearing which keeps moving could keep the target from ever settling and autotune waiting for level forever. Both deferrals now stop AUTOTUNE_LEVEL_DEFER_MAX_MS (20s) after the wait for level began, after which the clock runs normally and a vehicle which still cannot level aborts within a further 3 * AUTOTUNE_LEVEL_TIMEOUT_MS. That bound also applies to the existing rate-based deferral, which had none. A rate-target-based deferral (resetting while the target yaw rate exceeds AUTOTUNE_LEVEL_RATE_Y_CD) was tried instead and is not sufficient: a shaped re-point's spin-up and spin-down tails run below the rate threshold while the target is still far from desired, and with low yaw acceleration those tails alone exceed the timeout - it failed the deterministic repro below in 2 runs out of 4. This was the cause of intermittent Plane QAUTOTUNE autotest failures ("Pitch Angle P Up 0%" repeated, then failure): repeated twitches drift the vehicle >5m, triggering the re-point. Validation: - natural conditions: stock failed 1/25 (unloaded) and 3/25 (host CPU contention); fixed passed 100/100 (50 unloaded + 50 under contention) - deterministic repro (Q_A_ACC_Y_MAX=5 slows the traverse past the 6s timeout while staying under the rate-guard threshold): stock fails 8/8 with "Failed to level", fixed succeeds 8/8 - the 20s bound: with Q_A_ACC_Y_MAX=0.5, whose 180deg re-point takes far longer, the unbounded deferral waited it out and succeeded after 853 simulated seconds; bounded, it reports "Failed to level" (2/2). Q_A_ACC_Y_MAX=5 still succeeds (5/5) and QAUTOTUNE passes (3/3)
QAutoTune's position hold commands a yaw re-point along the drift direction (+90deg while tuning pitch) once the vehicle has drifted more than 5m from the tune start position. Run the tune with a low yaw target acceleration so that re-point's traverse takes longer than autotune's failed-to-level timeout while the shaped target yaw rate stays below the rate threshold that defers the timeout, checking that the timeout is not run while the yaw target is still slewing towards the desired yaw. The test also records how far the vehicle drifts and yaws from where the tune started, and fails unless it drifted beyond the 5m re-point threshold and yawed by at least 90deg, so it cannot pass without exercising the re-point (seen: 11.2m and 140deg every run). Fails before the previous commit with "AutoTune: Failed to level, please tune manually" (3/3, having yawed only 42deg of the re-point when it gave up), passes after it (5/5). Also add a hover() helper matching the Copter test suite's.
Deferring the level timeout while the shaped yaw target travels to
desired_yaw_rad is not enough on its own: the airframe lags the target it
is given, so the re-point leaves the vehicle still turning at the moment
the target arrives, and it swings through desired and back before it is
level. A parallel autotest sweep caught QAUTOTUNEYawRepoint failing that
way twice in 39 runs, the vehicle overshooting by 12 degrees and taking
longer than the 6s timeout to come back:
t=296.9 yaw 306.5 target 310.6 target within 5deg -> timeout starts
t=298.6 yaw 297.4 target 305.5 target settled, vehicle 8deg behind
t=301.0 yaw 293.6 target 305.4 worst case, 11.9deg behind
t=302.7 yaw 296.8 target 305.5 6.0s elapsed, aborts while converging
Allow the time it takes to arrest a full-rate yaw before the timeout
starts counting. On a normally-configured vehicle that is a fraction of
a second; it is only long where the yaw acceleration limit is low, which
is exactly where the overshoot is slow to decay. Like the deferrals
above it this is bounded by AUTOTUNE_LEVEL_DEFER_MAX_MS, so a vehicle
which cannot level still aborts.
64317d4 to
fb47bdf
Compare
|
Automated review note — AI-generated (Claude+Codex), validated against the live diff. Please sanity-check before acting. Reviewed at head Still good to merge at fb47bdf: the level wait remains bounded, the new test passes on SITL and fails without the fix, and CI is green (112/112). Since the last review (64317d4) the branch was rebased onto master's AC_AutoTune radians conversion. Only the units were renamed, the PR logic is unchanged, and master's 418b4d7 has fixed the missing fabsf() from the earlier minor note. Optional
Minor (1)
Not tested: the minimum-slew plus very-low-acceleration corner, lthall's rate-only variant, Helicopter AutoTune at this head, and hardware. |
Summary
Stops intermittent failure of QAUTOTUNE test (and probably problem on real aircraft, depending on their control)
Classification & Testing (check all that apply and add your own)
Contains an autotest which fails reliably before and reliably passes after. In case the offered solution sucks.
Description
When position hold commands a large yaw re-point (get_poshold_attitude points along the drift direction, +90deg when tuning pitch), the attitude controller slews its yaw target towards the new desired yaw over several seconds. currently_level() measures yaw against the final desired yaw, so it reports not-level for the whole traverse and the 3 * AUTOTUNE_LEVEL_TIMEOUT_MS abort clock races the traverse: a ~125deg re-point at the ~25deg/s shaped slew rate takes ~5.5s against the 6s abort, and on the losing side of the race the tune dies with "AutoTune: Failed to level, please tune manually".
The existing guard (eb3c110ad9) resets the clock when the target yaw rate exceeds half the slew limit, but a long smooth re-point runs well below that: in a failing QuadPlane CI log the shaped rate peaked around 28deg/s against a 30deg/s threshold (MIN(Q_A_RATE_Y_MAX=75, Q_A_RATE_WPY_MAX=60)/2), the clock was never reset and the abort fired exactly 6.000s after the level-wait began -- 0.2s before yaw would have arrived; roll and pitch were within 0.1deg throughout.
Also reset the clock while the shaped yaw target is more than the level threshold away from desired_yaw_rad: the vehicle cannot possibly be at the desired yaw before its commanded target arrives there, so the timeout should only run once success is geometrically possible. A genuinely-unable-to-level vehicle still aborts: target shaping converges regardless of the vehicle response, after which the clock runs normally.
This was the cause of intermittent Plane QAUTOTUNE autotest failures ("Pitch Angle P Up 0%" repeated, then failure): repeated twitches drift the vehicle >5m, triggering the re-point. Validation: