[cloud-init] Add support for shutdown with power_state - #5079
Conversation
Detect when cloud-init uses power_state mode poweroff/halt and treat as successful launch instead of timeout. Created expects_shutdown_from_cloud_init() helper function. Implemented in all backend constructors and detect_aborted_start. Handles intentional shutdown in wait functions. Fixes VirtualBox current_state to detect stopped state immediately. Fixes #4456 [daemon] Move parsing to backend constructors [backends] Add shutdown detection to all Created expects_shutdown_from_cloud_init() helper function. Implemented in all backend constructors and detect_aborted_start. Handles intentional shutdown in wait functions. [review] Address review feedback - Restore missing 'starting' state check in VirtualBox current_state - Fix VirtualBox current_state to detect stopped state immediately - Remove duplicate TODO comment in wait_for_cloud_init - Remove extra newline in wait_for_cloud_init - Add if constexpr check to only treat IntentionalShutdownException as success for LaunchRequest, not StartRequest [review] Fix whitespace and test failure - Remove trailing whitespace (per GIT9) - Add expected_shutdown check to detect_aborted_start - Fixes BaseVM.waitForCloudInitVMDownReconnects test
224c25c to
38a2f7a
Compare
There was a problem hiding this comment.
Pull request overview
This PR aims to allow multipass launch to complete successfully when cloud-init intentionally powers off/halts the VM at the end of initialization (via power_state), avoiding launch failures/timeouts when the instance ends up stopped.
Changes:
- Add a helper to detect whether the provided cloud-init config expects a shutdown (
power_state.mode). - Propagate an “expected/intentional shutdown” flag via
StartExceptionso launch can suppress the error. - Update daemon launch flow to swallow intentional
StartExceptions forLaunchReply.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| src/utils/utils.cpp | Adds expects_shutdown_from_cloud_init() YAML inspection helper. |
| include/multipass/utils.h | Declares the new cloud-init shutdown expectation helper. |
| include/multipass/exceptions/start_exception.h | Extends StartException with an “expected” flag and accessor. |
| src/platform/backends/shared/base_virtual_machine.cpp | Marks aborted-start shutdown exceptions as intentional based on cloud-init config. |
| src/daemon/daemon.cpp | Suppresses intentional launch start errors when shutdown is expected. |
| src/platform/backends/hyperv/hyperv_virtual_machine.cpp | Updates/normalizes StartException construction and formatting changes. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #5079 +/- ##
==========================================
- Coverage 73.12% 73.10% -0.02%
==========================================
Files 331 331
Lines 17712 17732 +20
==========================================
+ Hits 12950 12961 +11
- Misses 4762 4771 +9 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
| bool mp::utils::expects_shutdown_from_cloud_init(const YAML::Node& user_data_config) | ||
| { | ||
| if (user_data_config["power_state"]) | ||
| { | ||
| auto ps = user_data_config["power_state"]; | ||
| if (ps["mode"]) | ||
| { | ||
| std::string mode = ps["mode"].as<std::string>(); | ||
| return (mode == "poweroff" || mode == "halt"); |
| catch (const StartException& e) | ||
| { | ||
| if (!std::is_same_v<Reply, LaunchReply> || !e.was_intentional()) | ||
| { | ||
| fmt::format_to(std::back_inserter(errors), "{}", e.what()); |
|
This PR is blocked by #5080 |
Description
power_state, but not running scripts or commands that shut down the VM.cloud-initoffers.Related Issue(s)
Closes #4456
Testing
Unit tests
Manual testing steps:
multipass lsshould report that the instance is stoppedmultipass start instanceshould workChecklist
Additional Notes
Apparently we may be blocking on the main thread if there is a detected aborted start because of QemuVirtualMachine::on_shutdown.
There also may be a race condition hidden in the startup process, when an aborted start happens. I have experienced somewhat infrequently that multipass will not return and instead time out. Possible causes are:
1. Attempting to authenticate with infinity timeout in the PlainSSHSession. This is indicated by the fact that the timeout seems to happen if the aborted start occurs before an ssh session is fully established ((I still reproduced after giving auth a 5s timeout)cloud-initfinishes before the shutdown).2. Thessh_disconnectcall fromPlainSSHSessionblocks until the message has been ack by the other side. The race condition happens when the guest tears down the entire ssh before we can get the disconnect message through.3. From gdb traces, the culpable seems to be the
state_mutbeing locked in exec_process in BaseVM. While this is locked, the ssh session hangs because the VM is down, and there is no timeout. As a consequence, the mutex stays locked and later deadlocks the main thread uponon_shutdownbeing called.Posted issue #5080
Info:
on_shutdownhas run.