managing-devices.md

Managing Devices

Enrolling Devices

The first time the Flight Control Agent runs, it generates a cryptographic key pair that serves as the device's unique cryptographic identity. The pair's private key never leaves the device, so that the device cannot be duplicated or impersonated. The cryptographic identity is registered with the Flight Control Service during enrollment and gets wiped during device decommissioning.

When the device is not yet enrolled, the agent performs service discovery to find its Flight Control Service instance. It then establishes a secure, mTLS-protected network connection to the Service using the X.509 enrollment certificate it has been provided with during image building or device provisioning. Next, it submits an Enrollment Request to the service that includes a description of the device's hardware and operating system as well as an X.509 Certificate Signing Request (CSR) including its cryptographic identity to obtain its initial management certificate. At this point, the device is not yet considered trusted and therefore remains quarantined in a "device lobby" until its Enrollment Request has been approved or denied by an authorized user (e.g. a administrator, an installer persona, or an auto-approver process).

Note

The device management certificate is automatically renewed by the flightctl-agent before it expires, once it reaches approximately 75% of its lifetime. Renewal happens proactively to avoid service disruption and does not require user action.

Enrolling using the Web UI

Enrolling using the CLI

You can list all devices currently waiting to be approved by running the following command:

flightctl get enrollmentrequests

The output should look similar to this:

NAME                                                  APPROVAL  APPROVER  APPROVED LABELS
54shovu028bvj6stkovjcvovjgo0r48618khdd5huhdjfn6raskg  Pending   <none>    <none>    

The unique device name is generated by the agent and cannot be changed. By default, the agent chooses the "device fingerprint", a base32-encoded hash of the agent's public key, as device name.

You can approve an Enrollment Request using the flightctl approve command and the name of the Enrollment Request to be approved. You can optionally also add labels to the device (see Organizing Devices) using the --label or -l flag. For example:

flightctl approve -l region=eu-west-1 -l site=factory-berlin enrollmentrequest/54shovu028bvj6stkovjcvovjgo0r48618khdd5huhdjfn6raskg

By default, labels specified during approval are merged with any labels the agent provided during enrollment (such as default-labels or labels from label-from-systeminfo). Approval labels take precedence when keys overlap.

If you want to set the exact set of labels on the device, ignoring agent-provided labels, use the --replace-labels flag:

# Only set these labels, ignore all agent-provided labels
flightctl approve -l env=production --replace-labels enrollmentrequest/54shovu028bvj6stkovjcvovjgo0r48618khdd5huhdjfn6raskg

# Remove all labels (empty device)
flightctl approve --replace-labels enrollmentrequest/54shovu028bvj6stkovjcvovjgo0r48618khdd5huhdjfn6raskg

When listing devices waiting to be approved once more using the flightctl get enrollmentrequests command, the output should look similar to this:

NAME                                                  APPROVAL  APPROVER  APPROVED LABELS
54shovu028bvj6stkovjcvovjgo0r48618khdd5huhdjfn6raskg  Approved  demouser  region=eu-west-1,site=factory-berlin

Once approved, the device will get issued its initial management certificate and get registered to the device inventory and is now ready to be managed.

Viewing the Device Inventory and Device Details

Flight Control automatically gathers system information from each device to help identify its hardware, OS, and environment. This data is shown in the status.systemInfo field. Fields can optionally be promoted to labels during the enrollment process, this must be done manually or through external automation. Promoting fields to labels enables powerful grouping and querying capabilities, such as filtering devices by region or OS version. You can also define your own fields in status.systemInfo.customInfo, allowing the agent to collect user-defined metadata through custom commands.

Considerations for System Information

Here are key considerations when using this feature:

• What's Collected: By default, the agent collects basic system information such as hostname, kernel version, OS distribution, product identifiers, and default network interface details. Additional fields such as BIOS data, GPU info, memory, and CPU details can be enabled through configuration. See Installing the Agent for a full list of supported fields and how to customize collection.

• Custom Fields: You can configure the agent to collect additional custom attributes specific to your environment. These are displayed under systemInfo.customInfo and can be used for labeling or grouping devices. See Installing the Agent for example usage.

• Collection Timing: System info is collected during process bootstrap and then cached. It refreshes only if the agent restarts or receives a reload signal (SIGHUP). This avoids unnecessary overhead during regular status updates.

• Reboot Awareness: The agent tracks boot time and boot ID, allowing Flight Control to detect whether the device has rebooted. This is useful for update coordination and lifecycle monitoring.

• Partial Data: Not all fields may be available on every device or on every process start. Collection is best-effort missing values errors or timeouts will result in empty values.

Viewing using the Web UI

Viewing using the CLI

You can see the devices in the device inventory by running the following command:

flightctl get devices

The output will be a table similar to this:

NAME                                                  ALIAS    OWNER   SYSTEM  UPDATED     APPLICATIONS
54shovu028bvj6stkovjcvovjgo0r48618khdd5huhdjfn6raskg  <none>   <none>  Online  Up-to-date  <none>

You can see one or more specific devices in the inventory using any of these formats:

# Single device using slash format
flightctl get device/54shovu028bvj6stkovjcvovjgo0r48618khdd5huhdjfn6raskg

# Single device using space format
flightctl get device 54shovu028bvj6stkovjcvovjgo0r48618khdd5huhdjfn6raskg

# Multiple devices by name
flightctl get devices device1 device2 device3

To see the details of a single device or list of devices in YAML or JSON formats, you can specify the -o yaml or -o json flags, respectively, e.g.

flightctl get device/54shovu028bvj6stkovjcvovjgo0r48618khdd5huhdjfn6raskg -o yaml

The output will look similar to this:

apiVersion: flightctl.io/v1beta1
kind: Device
metadata:
  name: 54shovu028bvj6stkovjcvovjgo0r48618khdd5huhdjfn6raskg
  labels:                                    # <-- user-defined labels assigned to this device
    region: eu-west-1
    site: factory-berlin
spec:
  os:
    image: quay.io/flightctl/rhel:9.5        # <-- the device's target OS image version
  config:
  - name: my-os-configuration                # <-- the device's target OS configuration (here: read from a git repo)
    configType: GitConfigProviderSpec
    gitRef:
      path: /configuration
      repository: my-configuration-repo
      targetRevision: production
status:
  os:
    image: quay.io/flightctl/rhel:9.5        # <-- the device's current OS image version
  config:
    renderedVersion: "1"                     # <-- the device's current OS configuration version
  applications:
    data: {}                                 # <-- the device's current list of deployed applications
    summary:
      status: Unknown                        # <-- health status of applications on the device
  resources:                                 # <-- whether sufficient CPU/disk/memory resources are available
    cpu: Healthy
    disk: Healthy
    memory: Healthy
  systemInfo:                                # <-- basic information about the system
    architecture: amd64
    bootID: 037750f7-f293-4c5b-b06e-481eef4e883f
    operatingSystem: linux
  summary:
    info: ""
    status: Online                           # <-- online status of the device
  updated:
    status: UpToDate                         # <-- update status of the device
  lastSeen: "2024-08-28T11:45:34.812851905Z" # <-- when the device last checked-in
[...]

Organizing Devices

You can organize your devices by assigning them labels, for example to record their location ( ("region=emea", "site=factory-berlin"), hardware type ("hw-model=jetson", "hw-generation=orin"), or purpose ("device-type=autonomous-forklift"). This then allows you select devices by these labels when viewing the device inventory or applying operations to them.

As good practice, labels should take the form of key=value pairs, whereby the key is the criterion you want to group by. However, labels that only consist of keys are also allowed.

Labels must follow certain rules to be valid (in fact, these are the same as for Kubernetes):

• Keys and value must each be 63 characters or less. Value may be omitted.
• Keys and values may consist of alphanumeric characters (a-z, A-Z, 0-9). They may also contain dashes (-), underscores (_), dots (.), but not as the first or last character.

Once devices are labeled, you can select a subset of devices by writing a "label selector". Label selectors can select based on equality, inequality, or set operators:

Example label selector	Devices it selects
site=factory-berlin	All devices with a label key site and a label value factory-berlin.
site!=factory-berlin	All devices with a label key site and not a label value factory-berlin.
site in (factory-berlin,factory-madrid)	All devices with a label key site and a label value of either factory-berlin or factory-madrid

You can specify multiple label selectors in a comma-separated list, for example site=factory-berlin,device-type=autonomous-forklift, to have a device selected only if all selectors in the list match.

There are multiple ways when and how to apply labels to devices:

• You can define a set of default labels during image building that get automatically applied to all devices deploying that image.
• You can assign initial labels during enrollment (see Enrolling Devices).
• You can edit labels post-enrollment, which is described in the following sections.

Using Labels on the Web UI

Using Labels on the CLI

You can view devices in your inventory including their labels by using the -o wide option:

flightctl get devices -o wide

NAME                                                  ALIAS    OWNER   SYSTEM  UPDATED     APPLICATIONS  LABELS
54shovu028bvj6stkovjcvovjgo0r48618khdd5huhdjfn6raskg  <none>   <none>  Online  Up-to-date  <none>        region=eu-west-1,site=factory-berlin
hnsu33339f8m5pjqrbh5ak704jjp92r95a83sd5ja8cjnsl7qnrg  <none>   <none>  Online  Up-to-date  <none>        region=eu-west-1,site=factory-madrid

You can view devices in your inventory with a specific label or set of labels by using the -l key=value option one or more times:

flightctl get devices -l site=factory-berlin -o wide

NAME                                                  ALIAS    OWNER   SYSTEM  UPDATED     APPLICATIONS  LABELS
54shovu028bvj6stkovjcvovjgo0r48618khdd5huhdjfn6raskg  <none>   <none>  Online  Up-to-date  <none>        region=eu-west-1,site=factory-berlin

You can update the labels of a given device using one of two methods:

Method 1: Using the edit command (Recommended)

The flightctl edit command provides a streamlined way to edit resources directly in your preferred text editor, similar to kubectl edit. This command automatically fetches the current resource definition, opens it in an editor, and applies your changes when you save and exit:

flightctl edit device/54shovu028bvj6stkovjcvovjgo0r48618khdd5huhdjfn6raskg

This will open the device definition in your default editor (defined by FLIGHTCTL_EDITOR, EDITOR environment variables, or defaults to vi). You can also specify a different editor:

flightctl edit device/54shovu028bvj6stkovjcvovjgo0r48618khdd5huhdjfn6raskg --editor=nano

The command supports both TYPE/NAME and TYPE NAME formats:

# Both of these are equivalent:
flightctl edit device/my-device
flightctl edit device my-device

When you save and exit the editor, the command will automatically apply your changes to the server. If there are any errors (such as validation failures or conflicts), your changes will be saved to a temporary file for recovery.

Method 2: Export, edit, and apply manually

Alternatively, you can update labels by exporting the device's current definition into a file, editing the specification to update the labels, and then applying the updated definition. To export the device's current definition into a file called my_device.yaml, run the flightctl get device command with the device's name and the -o yaml output flag:

flightctl get device/54shovu028bvj6stkovjcvovjgo0r48618khdd5huhdjfn6raskg -o yaml > my_device.yaml

Next, use your preferred editor to edit my_device.yaml, for example:

apiVersion: flightctl.io/v1beta1
kind: Device
metadata:
  labels:
    some_key: some_value
    some_other_key: some_other_value
  name: 54shovu028bvj6stkovjcvovjgo0r48618khdd5huhdjfn6raskg
spec:
[...]

Save the edit, then apply the updated device definition using:

flightctl apply -f my_device.yaml

When you now view the device's labels using flightctl get devices -o wide once more, you should see your changes applied:

NAME                                                  ALIAS    OWNER   SYSTEM  UPDATED     APPLICATIONS  LABELS
54shovu028bvj6stkovjcvovjgo0r48618khdd5huhdjfn6raskg  <none>   <none>  Online  Up-to-date  <none>        some_key=some_value,some_other_key=some_other_value
hnsu33339f8m5pjqrbh5ak704jjp92r95a83sd5ja8cjnsl7qnrg  <none>   <none>  Online  Up-to-date  <none>        region=eu-west-1,site=factory-madrid

Updating the OS

You can update a device's OS by updating the target OS image name or version in the device's specification. The next time the agent checks in, it learns of the requested update and automatically starts downloading and verifying the new OS version in the background. It then schedules the actual system update to be performed according to the update policy. When the time has come to update, it installs the new version in parallel and performs a reboot into the new version.

Flight Control currently supports the following image types and image references formats:

Image Type	Image Reference
bootc	An OCI image reference to a container registry. Example: quay.io/flightctl-demos/rhel:9.5

During the process, the agent sends status updates to the service. You can monitor the update progress by viewing the device status.

Updating the OS on the Web UI

Updating the OS on the CLI

To update a device using the CLI, get the device's current resource manifest, edit it to specify the new OS name and version target, then apply the updated resource.

apiVersion: flightctl.io/v1beta1
kind: Device
metadata:
  name: some_device_name
spec:
[...]
  os:
    image: quay.io/flightctl/rhel:9.5
[...]

Using Image Pull Secrets

If your device relies on containers from a private repository, authentication credentials (pull secrets) must be placed in the appropriate system paths.

• OS Image: Uses /etc/ostree/auth.json
• Container Images: Uses the system default for Podman, ~/.config/containers/auth.json. You will need an auth.json file for each user that needs to pull from a private repo. When deploying quadlet or container apps, the runAs configuration determines which user needs the credentials.

Auth File Format

The authentication file should follow this format:

{
  "auths": {
    "registry.example.com": {
      "auth": "base64-encoded-credentials"
    }
  }
}

Note

Authentication must exist on the device before it can be consumed.

Managing OS Configuration

With image-based Linux OSes, it is best practice to include OS-level / host configuration into the OS image for maximum consistency and repeatability. To update configuration, a new OS image should be created and devices updated to the new image.

However, there are scenarios where this is impractical, for example, when configuration is missing in the image, needs to be specific to a device, or needs to be update-able at runtime without updating the OS image and rebooting. For these cases, Flight Control allows users to declare a set of configuration files that shall be present on the device's file system.

Conceptually, this set of configuration files can be thought of as an additional, dynamic layer on top of the OS image's layers. The Flight Control Agent applies updates to this layer transactionally, ensuring that either all files have been successfully updated in the file system or have been returned to their pre-update state. Further, if the user updates both a devices OS and configuration set at the same time, the Flight Control Agent will first update the OS, then apply the specified configuration set on top.

Important

After the Flight Control Agent has updated the configuration on disk, this configuration still needs to be activated. That means, running services need to reload the new configuration into memory for it to become effective. If the update involves a reboot, services will be restarted by systemd in the right order with the new configuration automatically. If the update does not involve a reboot, many services can detect changes to their configuration files and automatically reload them. When a service does not support this, you use Device Lifecycle Hooks to specify rules like "if configuration file X has changed, run command Y". Also refer to this section for the set of default rules that the Flight Control Agent applies.

Users can specify a list of configurations sets, in which case the Flight Control Agent applies the sets in sequence and on top of each other, such that in case of conflict the "last one wins".

Configuration can come from multiple sources, called "configuration providers" in Flight Control. Flight Control currently supports the following configuration providers:

• Git Config Provider: Fetches device configuration files from a Git repository.
• Kubernetes Secret Provider: Fetches a Secret from a Kubernetes cluster and writes its content to the device's file system.
• HTTP Config Provider: Fetches device configuration files from an HTTP(S) endpoint.
• Inline Config Provider: Allows specifying device configuration files inline in the device manifest without querying external systems.

These providers are described in the following.

Getting Configuration from a Git Repository

You can store device configuration in a Git repository such as GitHub or GitLab and let Flight Control synchronize it to the device's file system by adding a Git Config Provider.

The Git Config Provider takes the following parameters:

Parameter	Description
Repository	The name of a Repository resource defined in Flight Control.
TargetRevision	The branch, tag, or commit of the repository to checkout.
Path	The subdirectory of the repository that contains the configuration.

The Repository resource definition tells Flight Control the Git repository to connect to and which protocol and access credentials to use. It needs to be set up once (see Setting Up Repositories) and can then be used to configure multiple devices or fleets.

The subdirectory of the repository (pointed to by Path) will be mounted to the root of the device. When specifying a Path, make sure the directories in it already exist with writable access on the system (the application cannot write to the root).
Using non-existent or read-only directories may result in errors due to insufficient permissions or read-only file systems.

Example

A assume a Git repository github.com/flightctl/flightctl-demos that in its branch production stores device network and time server configuration organized by deployment site as follows:

.
├── factory-a
│   └── etc
│       ├── chrony.conf
│       └── NetworkManager
│           └── system-connections
│               └── wifi-access.nmconnection
└── factory-b
    └── etc
        ├── chrony.conf
        └── NetworkManager
            └── system-connections
                └── wifi-access.nmconnection

First, create a new Repository resource. Give it a name that you can later reference in your configurations, for example "site-settings", and select the Repository type "git". Next, enter the URL of your git repository, in this case https://github.com/flightctl/flightctl-demos.git.

You can now reference this Repository when you configure devices. For example, to apply the configuration files under /factory-a to a device, add a Git Config Provider to the device's specification using the following parameters:

Parameter	Value
Repository	site-settings
TargetRevision	production
Path	/factory-a

Authenticating to Private Repositories

Flight Control supports authentication for private Git repositories and HTTP endpoints using SSH keys, HTTPS credentials, or client certificates.

SSH Authentication

For SSH-based Git repositories, configure SSH known hosts and provide your private key (base64-encoded).

Step 1: Configure SSH Known Hosts

Create a known_hosts file containing the SSH host keys for your Git server:

ssh-keyscan github.com >> known_hosts
ssh-keyscan gitlab.com >> known_hosts
ssh-keyscan private-git-server.com >> known_hosts

Example known_hosts file content:

github.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOMqqnkVzrm0SdG6UOoqKLsabgH5C9okWi0dh2l9GKJl

Once you have created the known_hosts file, deploy it to Flight Control based on your deployment method:

For Helm deployments: Use --set-file to include the known_hosts file in your deployment:

helm upgrade --install --version=<version-to-install> \
    --namespace flightctl --create-namespace \
    flightctl oci://quay.io/flightctl/charts/flightctl \
    --set-file global.sshKnownHosts.data=known_hosts

For Quadlet deployments: Place the file on the host at /etc/flightctl/ssh/known_hosts:

sudo install -m 0644 known_hosts /etc/flightctl/ssh/known_hosts

Note

If the services are already running and you're updating the configuration, restart them to apply the changes:

sudo systemctl restart flightctl-worker.service flightctl-periodic.service

Step 2: Configure Repository with SSH Private Key

Base64 encode your private key: cat ~/.ssh/id_rsa | base64 -w 0

Create a Repository resource with SSH authentication:

apiVersion: flightctl.io/v1beta1
kind: Repository
metadata:
  name: private-ssh-repo
spec:
  type: git
  url: git@github.com:myorg/private-repo.git
  sshConfig:
    sshPrivateKey: <base64-encoded-private-key>
    # Optional: privateKeyPassphrase: your-passphrase
    # Optional: skipServerVerification: true

HTTPS Authentication

For HTTPS repositories, use basic authentication, bearer tokens, or client certificates:

Basic authentication (username/password):

spec:
  type: git
  url: https://github.com/myorg/private-repo.git
  httpConfig:
    username: myusername
    password: ghp_xxxxxxxxxxxxxxxxxxxx  # Use Personal Access Token

Bearer token authentication:

spec:
  type: git
  url: https://git-server.example.com/myorg/private-repo.git
  httpConfig:
    token: your-bearer-token

Mutual TLS (client certificates):

spec:
  type: git
  url: https://secure-git.example.com/myorg/private-repo.git
  httpConfig:
    tls.crt: <base64-encoded-certificate>
    tls.key: <base64-encoded-key>
    # Optional: ca.crt: <base64-encoded-ca-cert>
    # Optional: skipServerVerification: true

Base64 encode certificates: cat client.crt | base64 -w 0

HTTP Endpoints

For HTTP-based configuration repositories (non-Git):

spec:
  type: http
  url: https://config-server.example.com
  httpConfig:
    username: myusername
    password: mypassword
    # Or use: token: your-api-token

Verifying Access

Check repository accessibility after creation:

flightctl get repository/private-ssh-repo
# Output shows ACCESSIBLE: True/False

Getting Secrets from a Kubernetes Cluster

You can let Flight Control query the Kubernetes cluster it is running on for a Kubernetes Secret. The content of that Secret can then be written to a path on the device file system.

The Kubernetes Secret Provider takes the following parameters:

Parameter	Description
Name	The name of the Secret.
NameSpace	The namespace of the Secret.
MountPath	The directory in the device's file system to write the secret's content to.

Note that Flight Control needs to have the permissions access Secrets in that namespace, for example by creating a ClusterRole and ClusterRoleBinding allowing the flightctl-worker service account "get" and "list" Secrets in that namespace.

Getting Configuration from an HTTP Server

You can let Flight Control query an HTTP server for configuration. This HTTP server can then serve static or dynamically generated configuration for a device.

The HTTP Config Provider takes the following parameters:

Parameter	Description
Repository	The name of a Repository resource defined in Flight Control.
Suffix	The suffix to append to the base URL defined in the Repository resource. It can include path and query parameters such as /path/to/endpoint?query=param
FilePath	The path to the file on the device's file system in which to store the returned value of the HTTP Server.

The Repository resource definition tells Flight Control the HTTP server to connect to and which protocol and access credentials to use. It needs to be set up once (see Setting Up Repositories) and can then be used to configure multiple devices or fleets.

Specifying Configuration Inline in the Device Spec

You specify configuration inline in a device's specification, so Flight Control does not need to connect to external systems to fetch configuration.

The Inline Config Provider takes a list of file specifications, whereby each file specification takes the following parameters:

Parameter	Description
Path	The absolute path to the file on the device. Note that any existing file will be overwritten.
Content	The plain text (UTF-8) or base64-encoded content of the file.
ContentEncoding	How the contents are encoded. Must be either "plain" or "base64". Defaults to "plain".
Mode	(Optional) The file’s permission mode. You may specify the more familiar octal with a leading zero (e.g., 0644) or as a decimal without a leading zero (e.g., 420). Setuid/setgid/sticky bits are supported. If not specified the permission mode for files defaults to 0644.
User	(Optional) The file's owner, specified either as a name or numeric ID. Defaults to "root".
Group	(Optional) The file's group, specified either as a name or numeric ID.

Managing Configuration on the Web UI

Managing Configuration on the CLI

To implement the example from Getting Configuration from a Git Repository, first, create a file site-settings-repo.yaml that contains the following definition for a Repository resource named site-settings:

apiVersion: flightctl.io/v1beta1
kind: Repository
metadata:
  name: site-settings
spec:
  type: git
  url: https://github.com/flightctl/flightctl-demos.git

Create the Repository resource by applying the file:

flightctl apply -f site-settings-repo.yaml`

Verify the resource has been correctly created and is accessible by Fight Control by running:

flightctl get repository/site-settings

The output should look like this:

NAME           TYPE  REPOSITORY URL                                ACCESSIBLE
site-settings  git   https://github.com/flightctl/flightctl-demos  True

To apply the configuration for factory-a to a device, you would update the device's specification as follows:

apiVersion: flightctl.io/v1beta1
kind: Device
metadata:
  name: some_device_name
spec:
[...]
  config:
  - name: factory-a-settings
    configType: GitConfigProviderSpec
    gitRef:
      repository: site-settings
      targetRevision: production
      path: /factory-a
[...]

Managing Applications

You can deploy, update, or undeploy applications on a device by updating the list of applications in the device's specification. The next time the agent checks in, it learns of the change in the specification, downloads any new or updated application packages and images from an OCI-compatible registry, and deploys them to the appropriate application runtime or removes them from that runtime.

Container Image Versioning and Floating Tags

Flight Control uses a declarative API model that expects container images to be immutable for each rendered device version. Floating tags like latest are not recommended as they can change unexpectedly and cause version skew across your fleet. Changes to floating tags are not automatically reconciled by the service or agent, updates must be explicitly declared.

Best practices:

• Use explicit version tags v1.2.3 or image digests @sha256:abc123...
• When updating applications, explicitly declare new tags or digests in the device specification
• This ensures consistent, predictable deployments and prevents unintended drift

The following table shows the application runtimes and formats supported by Flight Control:

Runtime: Podman

Specification	Format	Source / Delivery
Compose specification (via podman-compose)	OCI Image	OCI registry
Compose specification (via podman-compose)	Unpackaged Inline	Inline in device specification
Quadlet specification (via Podman Quadlet)	OCI Image	OCI registry
Quadlet specification (via Podman Quadlet)	Unpackaged Inline	Inline in device specification
Simplified container specification	OCI Image	OCI registry

Runtime: Kubernetes (Helm)

Specification	Format	Source / Delivery
Helm chart (via Helm)	OCI Image	OCI registry

Runtime: Virtual machine (KVM)

Specification	Format	Source / Delivery
Virtual machine definition	Unpackaged inline	Inline in device specification

Note

Compose applications require podman-compose to be installed on the device.

Note

quadlet, container, and vm applications require podman version 5.0 or above.

Note

vm applications require the /dev/kvm device node to be present on the device. This requires CPU virtualization extensions (Intel VT-x or AMD-V) and the kvm kernel module to be loaded.

Note

Image downloads adhere to the pull-timeout configuration.

Tip

Short image names (e.g., nginx) are not supported. Use fully qualified references like docker.io/nginx to avoid ambiguity.

To deploy an application to a device, create a new entry in the "applications" section of the device's specification, specifying the following parameters:

Parameter	Description
Name	A user-defined name for the application. This will be used when the web UI and CLI list applications.
Image	A reference to an application package in an OCI registry.
AppType	The application format type. Currently supported types: compose, quadlet, container, helm, vm.
EnvVars	(Optional) A list of key/value-pairs that will be passed to the deployment tool as environment variables or command line flags.

For each application in the "applications" section of the device's specification, there exist a corresponding device status information that contains the following information:

Status Field	Description
Preparing	Application deployed; containers initialized but not yet running.
Starting	Application started; at least one container running, awaiting results.
Running	All containers are running.
Error	All containers failed.
Unknown	Application started, no containers observed.
Completed	All containers have completed successfully.
Stopping	The application was stopped and is shutting down its containers.
Stopped	The application was stopped and has no running containers.

Managing Applications on the Web UI

Managing Applications on the CLI

To deploy an application package from an OCI registry, specify it in the device's spec.applications[] as follows:

apiVersion: flightctl.io/v1beta1
kind: Device
metadata:
  name: some_device_name
spec:
[...]
  applications:
  - name: wordpress
    image: quay.io/flightctl-demos/wordpress-app:v1.2.3
    appType: container
    runAs: flightctl
    envVars:
      WORDPRESS_DB_HOST: "mysql"
      WORDPRESS_DB_USER: "user"
      WORDPRESS_DB_PASSWORD: "password"
[...]

Managing Application Lifecycle

In addition to deploying, updating, or undeploying applications through the device specification, you can start, stop, or restart a deployed application on demand, without changing the specification. This is useful for temporarily taking an application out of service, or for triggering a restart to recover from an issue.

Use the flightctl app stop, flightctl app start, and flightctl app restart commands to control an application without changing the device or fleet specification. See the flightctl app start, flightctl app stop, and flightctl app restart CLI reference pages for command syntax, flags, and examples.

Note

Restarting an application is only supported on individual devices, not on fleets.

Stopping or starting an application on a fleet sets a fleet-wide default that applies to every device currently in the fleet, and to any device that joins the fleet later. If you separately stop or start the same application directly on one of those devices, that device-level action takes precedence over the fleet-wide default for that device, until you issue another fleet-level action.

While an application is stopped, its status is reported as Stopped. Starting it again returns it to the normal Preparing/Starting/Running progression.

VM applications

VM applications deploy virtual machines on the device. You define the VM using an inline manifest file; Flight Control handles the conversion and deployment automatically.

Flight Control manages the lifecycle of a VM application — deploying, starting, stopping, and restarting the virtual machine — but does not manage anything running inside the VM. Managing the VM's content and workloads is left to you, using whatever tooling you would otherwise use for that, for example Ansible Automation Platform (AAP), or by connecting to the VM directly.

Device requirements

The device must have hardware virtualization support:

Requirement	Description
/dev/kvm	Hardware virtualization device node. Requires CPU virtualization extensions (Intel VT-x or AMD-V) and the kvm kernel module to be loaded.

The VM runtime is included in the device OS image. You do not need to install it separately.

Note

When an OS update is pending, the agent defers validation. This allows you to deploy VM applications alongside an OS image that includes the VM runtime. The agent applies the OS update and reboots first, then validates and deploys applications.

VM application specification

To deploy a VM application, add an entry to the applications section of the device specification with appType: vm:

Parameter	Description
name	Required. Application name. Must match metadata.name inside the vm.yaml file.
appType	Must be vm for VM applications.
inline	Required. Exactly one file named vm.yaml. The file must be a KubeVirt VirtualMachine manifest with apiVersion: kubevirt.io/v1, kind: VirtualMachine, and metadata.name matching the application name.
publishPorts	Optional. List of host-to-guest port mappings. Each entry must use the format "hostPort:guestPort" or "hostPort:guestPort/protocol" (for example, "8080:80" or "8080:80/tcp").

Note

The control plane converts the vm.yaml manifest into Quadlet units before the agent deploys it. The agent does not run the KubeVirt manifest directly. You can configure the virt-launcher image and passt workarounds used for that conversion; see Configuring VM application rendering.

Note

The spec.running field in vm.yaml is ignored. Use application lifecycle commands (flightctl app start / stop / restart) to control whether the VM is running.

Examples

Fedora

This example deploys a Fedora guest from a public container disk image and publishes SSH on host port 2222:

apiVersion: flightctl.io/v1beta1
kind: Device
metadata:
  name: edge-device-001
spec:
  applications:
    - name: fedora-vm
      appType: vm
      publishPorts:
        - "2222:22"
      inline:
        - path: vm.yaml
          content: |
            apiVersion: kubevirt.io/v1
            kind: VirtualMachine
            metadata:
              name: fedora-vm
            spec:
              running: true
              template:
                spec:
                  domain:
                    devices:
                      disks:
                        - name: containerdisk
                          disk:
                            bus: virtio
                    resources:
                      requests:
                        memory: 1Gi
                  volumes:
                    - name: containerdisk
                      containerDisk:
                        image: quay.io/containerdisks/fedora:40

Windows

This example installs a Windows guest from ISOs staged on the device (hostDisk), with a blank root disk, VirtIO drivers media, UEFI firmware, Hyper-V enlightenments, and a TPM. The Hyper-V settings in the manifest are guest enlightenments exposed for Windows compatibility and performance; the VM still runs on KubeVirt/KVM on the device, not on Microsoft Hyper-V, and they do not enable nested virtualization.

Because each hostDisk path must already exist on every target device, this hostDisk layout is mainly for manual or illustrative setups. For production or larger-scale deployments, pull the ISOs from an OCI registry (for example as a containerDisk) instead. RDP is published on host port 3389:

apiVersion: flightctl.io/v1beta1
kind: Device
metadata:
  name: edge-device-001
spec:
  applications:
    - name: windows-vm
      appType: vm
      publishPorts:
        - "3389:3389"
      inline:
        - path: vm.yaml
          content: |
            apiVersion: kubevirt.io/v1
            kind: VirtualMachine
            metadata:
              name: windows-vm
            spec:
              running: true
              dataVolumeTemplates:
              - metadata:
                  name: rootdisk
                spec:
                  source:
                    blank: {}
                  storage:
                    resources:
                      requests:
                        storage: 60Gi
              template:
                spec:
                  domain:
                    firmware:
                      bootloader:
                        efi:
                          secureBoot: false
                    cpu:
                      cores: 2
                      sockets: 1
                      threads: 1
                    features:
                      acpi:
                        enabled: true
                      apic:
                        enabled: true
                      hyperv:
                        relaxed:    { enabled: true }
                        vapic:      { enabled: true }
                        spinlocks:  { enabled: true, spinlocks: 8191 }
                        vpindex:    { enabled: true }
                        runtime:    { enabled: true }
                        synic:      { enabled: true }
                        synictimer: { enabled: true, direct: { enabled: true } }
                        reset:      { enabled: true }
                        frequencies: { enabled: true }
                        tlbflush:   { enabled: true }
                        ipi:        { enabled: true }
                        stimer:     { enabled: true }
                    clock:
                      utc: {}
                      timer:
                        hpet: { present: false }
                        pit: { tickPolicy: delay }
                        rtc: { tickPolicy: catchup }
                        hyperv: { present: true }
                    devices:
                      autoattachMemBalloon: false
                      disks:
                      - disk:
                          bus: virtio
                        cache: writeback
                        dedicatedIOThread: true
                        name: rootdisk
                        bootOrder: 2
                      - cdrom:
                          bus: sata
                          readonly: true
                        name: windows-iso
                        bootOrder: 1
                      - cdrom:
                          bus: sata
                          readonly: true
                        name: virtio-drivers
                      interfaces:
                      - masquerade: {}
                        name: default
                        model: virtio
                        ports:
                        - port: 3389
                      inputs:
                      - type: tablet
                        bus: usb
                        name: tablet
                      tpm: {}
                    memory:
                      guest: 4Gi
                    resources: {}
                  networks:
                  - name: default
                    pod: {}
                  volumes:
                  - name: rootdisk
                    dataVolume:
                      name: rootdisk
                  - name: windows-iso
                    hostDisk:
                      path: /path/to/Win11.iso
                      type: DiskOrFail
                  - name: virtio-drivers
                    hostDisk:
                      path: /path/to/virtio-win.iso
                      type: DiskOrFail

Note

In this example the Windows installation ISO and VirtIO drivers ISO are referenced via hostDisk paths on the device. Replace /path/to/... with the locations where you have staged those files, or change the volumes to pull the ISOs from an OCI registry instead.

VM application status

The agent reports the following status values for a VM application:

Status Field	Description
Preparing	The agent is pulling OCI images required by the VM before starting it.
Starting	The VM has been submitted to the system and is initializing.
Running	The virtual machine is running and accepting workloads.
Error	The VM failed to start or terminated with an error. Check the device events for details.
Unknown	The VM was submitted but no running state has been observed yet.
Completed	The VM exited cleanly.
Stopping	The VM was stopped with flightctl app stop and is shutting down.
Stopped	The VM was stopped with flightctl app stop and is no longer running.

VM restart behavior

The spec.running field in the KubeVirt VirtualMachine manifest is ignored. Flight Control controls whether the VM is running through application lifecycle actions instead.

By default, a deployed VM application is kept running and is automatically restarted if it crashes or is stopped unexpectedly. To stop, start, or restart a VM application on demand, use flightctl app stop, flightctl app start, and flightctl app restart. See Managing Application Lifecycle.

Accessing a VM Application Console

A VM application is not otherwise reachable over the network, so a user with the appropriate authorization (get permission on the devices/applications/console resource) can connect to its serial or VNC console through the agent. This works even if the device is behind a NAT or has a dynamic IP address.

Use the flightctl app console command to open a serial or VNC console to a VM application. See the flightctl app console CLI reference page for command syntax, flags, and examples.

Only one console session — serial or VNC — is allowed per application at a time. If you attempt to connect while another session is already active, the command fails unless you pass --force, which takes over the existing session and disconnects it.

Note

A VNC console session additionally supports only one connected viewer. The command exits once that viewer disconnects; run it again to start a new session.

Helm Applications

Helm applications allow you to deploy Kubernetes workloads to edge devices running a local Kubernetes distribution such as MicroShift. The Flight Control agent uses Helm to install, upgrade, and uninstall charts on the device's local cluster.

Device Requirements

The following dependencies must be installed on the device:

Dependency	Required Version	Description
helm	>= 3.8	Helm CLI for chart management. Version 3.8+ is required for OCI registry support.
kubectl or oc	Any	Kubernetes CLI for cluster operations. Either kubectl or oc (OpenShift CLI) is supported.
crictl	Any	CRI compatible CLI for pulling container images referenced by Helm charts.

Note

The agent automatically detects whether kubectl or oc is available (with preference for kubectl) and uses the first one found.

Note

When an OS update is pending, the agent defers dependency validation, allowing you to deploy new application types alongside an OS image that includes the required dependencies. The agent will first apply the OS update and reboot, then validate and deploy applications after booting into the new OS.

Kubeconfig Configuration

The agent automatically discovers the kubeconfig file to connect to the local Kubernetes cluster. The following locations are checked in order:

1. KUBECONFIG environment variable: If set, the agent checks each path in the colon-separated list and uses the first existing file.
2. MicroShift default path: /var/lib/microshift/resources/kubeadmin/kubeconfig
3. Standard default path: $HOME/.kube/config

If you are using MicroShift, the kubeconfig is automatically available after the cluster starts. For other Kubernetes distributions, ensure the kubeconfig is placed in one of the above locations.

Tip

For MicroShift deployments, you can use a lifecycle hook to wait for the kubeconfig to become available before proceeding with updates:

- run: /usr/bin/bash -c "until [ -f $KUBECONFIG ]; do sleep 1; done"
  timeout: 5m
  envVars:
    KUBECONFIG: "/var/lib/microshift/resources/kubeadmin/kubeconfig"

Authentication Configuration

Helm applications may require authentication for OCI registries (to pull charts) and container registries (to pull images referenced by the chart). Configure authentication files on the device before deploying Helm applications.

Configuration File	Purpose
/root/.config/containers/auth.json	Container registry authentication for pulling images. This is also used as a fallback for Helm OCI registry authentication.
/root/.config/helm/registry/config.json	Helm OCI registry authentication (Docker config format). If not present, the container auth file is used.
/root/.config/helm/repositories.yaml	Helm repository configuration for non-OCI chart repositories.
/etc/crictl.yaml	CRI runtime configuration for crictl to pull images. Required if your cluster uses a non-default CRI socket path.

Auth File Format:

The registry authentication files follow the Docker config format:

{
  "auths": {
    "quay.io": {
      "auth": "base64-encoded-credentials"
    },
    "registry.example.com": {
      "auth": "base64-encoded-credentials"
    }
  }
}

Note

Authentication files must exist on the device before deploying Helm applications. You can provision these files using the Inline Config Provider or by including them in your OS image.

CRI Configuration

The crictl tool requires configuration to communicate with the container runtime. Create /etc/crictl.yaml with the appropriate runtime endpoint:

For MicroShift (CRI-O):

runtime-endpoint: unix:///var/run/crio/crio.sock
image-endpoint: unix:///var/run/crio/crio.sock

For containerd:

runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock

Helm Application Specification

To deploy a Helm application, add an entry to the applications section of the device specification with appType: helm:

Parameter	Description
name	(Optional) A user-defined name for the application. If not specified, the chart reference is used as the name.
appType	Must be helm for Helm applications.
image	OCI reference to the Helm chart. Must include registry, repository, and version tag or digest. Example: quay.io/myorg/mychart:v1.0.0
namespace	(Optional) The Kubernetes namespace for deployment. If not specified, defaults to flightctl-<app-name>.
values	(Optional) Inline values to pass to the Helm chart. Supports arbitrarily nested structures.
valuesFiles	(Optional) List of values files to apply. Paths are relative to the chart root and applied in order before inline values.

Example: Basic Helm Application

apiVersion: flightctl.io/v1beta1
kind: Device
metadata:
  name: edge-device-001
spec:
  applications:
    - name: my-app
      appType: helm
      image: quay.io/myorg/my-app-chart:v1.2.0
      namespace: my-app

Example: Helm Application with Values

apiVersion: flightctl.io/v1beta1
kind: Device
metadata:
  name: edge-device-001
spec:
  applications:
    - name: monitoring
      appType: helm
      image: quay.io/myorg/monitoring-chart:v2.0.0
      namespace: monitoring
      values:
        replicaCount: 1
        service:
          type: NodePort
          port: 8080
        resources:
          limits:
            cpu: "500m"
            memory: "256Mi"

Example: Helm Application with Values Files

Charts may supply values files (e.g. values-prod.yaml). Those optional files can be referenced from a helm application's definition by supplying a relative path to the values file within the chart. The agent validates that these files exist in the chart before deployment.

apiVersion: flightctl.io/v1beta1
kind: Device
metadata:
  name: edge-device-001
spec:
  applications:
    - name: my-app
      appType: helm
      image: quay.io/myorg/my-app-chart:v1.2.0
      namespace: my-app
      valuesFiles:
        # These values must exist within 'my-app-chart:v1.2.0'
        - values-production.yaml
        - values-edge.yaml
        - values/hardening.yaml
      values:
        # Inline values are applied last and override values files
        nodeSelector:
          kubernetes.io/arch: arm64

Image Prefetching

The Flight Control agent attempts to prefetch all container images referenced by Helm charts before deployment. This allows applications to start without network access after the initial sync.

The agent performs a Helm dry-run to extract all container images from the rendered manifests, then uses crictl to pull these images to the local CRI cache.

Important

While the agent prefetches images to enable networkless operation, charts that specify imagePullPolicy: Always in their manifests will still attempt to pull images at runtime. For fully offline deployments, ensure your charts use imagePullPolicy: IfNotPresent or Never.

Image and Artifact Pruning

The Flight Control agent can automatically remove unused container images and OCI artifacts from devices to free up disk space. This feature helps prevent storage exhaustion on edge devices with limited capacity.

How Pruning Works

Image and artifact pruning operates on a "lost reference" model:

1. Reference Tracking: Before each device update, the agent records all images and artifacts referenced in the current and desired device specifications to a file (/var/lib/flightctl/image-artifact-references.json).

2. Pruning Execution: After a successful spec update, the agent compares the previously recorded references with the current device specifications. Any images or artifacts that were previously referenced but are no longer needed are eligible for removal.

3. Safe Pruning: The agent only removes items that:

   • Were previously recorded in the references file
   • Are no longer referenced in the current or desired device specifications
   • Exist locally on the device

4. Preservation: Images and artifacts required for current operations or potential rollback (desired state) are always preserved, even if they appear in the references file.

Note

Pruning runs automatically after successful device spec updates. The references file is maintained even when pruning is disabled, ensuring accurate pruning behavior when the feature is later enabled.

What Gets Pruned

The pruning process considers the following types of OCI targets:

• Application Images: Container images referenced by Compose, Quadlet, Container, or Helm applications
• OS Images: Operating system images specified in the device's OS configuration
• Helm Charts: Charts referenced by Helm applications

Configuring Pruning

Image and artifact pruning is disabled by default. To enable it, configure the agent's pruning settings.

Base Configuration

Add the pruning configuration to the main agent configuration file at /etc/flightctl/config.yaml:

image-pruning:
  enabled: true

Drop in Configuration

You can also configure pruning using drop in files in /etc/flightctl/conf.d/. This allows you to enable or disable pruning for specific devices or fleets without modifying the base configuration.

Create a drop in file (e.g., /etc/flightctl/conf.d/enable-pruning.yaml):

image-pruning:
  enabled: true

Drop in files are processed in lexical order and override the base configuration. For example, to disable pruning via drop in:

image-pruning:
  enabled: false

Note

Configuration changes take effect after the agent reloads its configuration. The agent automatically reloads configuration when it receives a SIGHUP signal. When files in /etc/flightctl/conf.d/ are created, updated, or removed, the agent automatically sends itself a SIGHUP signal to reload configuration. You can also manually restart the agent service to apply changes.

Tip

Consider disabling pruning when experimenting with many images that were previously pruned because their references were lost and are now present again. This prevents the agent from immediately pruning images you're actively testing, allowing you to iterate on configurations without repeatedly re-downloading images.

Pruning Behavior

• Automatic Execution: Pruning runs automatically after successful device spec updates. No manual intervention is required.

• Non-Blocking: Pruning errors do not block device reconciliation. If pruning fails, the agent logs a warning and continues normal operation.

• Reference Accumulation: The references file accumulates references over time, even when pruning is disabled. This ensures that if pruning is later enabled, it has accurate historical data about what was previously referenced.

• Incremental Updates: The references file is updated before each upgrade to include all currently referenced images and artifacts, ensuring the tracking remains current.

Example Scenarios

Scenario 1: Application Update**

• Device initially runs application quay.io/myorg/app:v1.0
• Application is updated to quay.io/myorg/app:v2.0
• After successful update, v1.0 image is pruned (if it was previously recorded)

Scenario 2: Application Removal**

• Device runs multiple applications
• One application is removed from the device specification
• After successful update, images and artifacts used only by the removed application are pruned

Scenario 3: OS Image Update**

• Device OS is updated from quay.io/flightctl/rhel:9.4 to quay.io/flightctl/rhel:9.5
• After successful update, the old OS image (9.4) is pruned (if it was previously recorded)

Monitoring Pruning

Pruning operations are logged by the agent. You can monitor pruning activity through:

• Agent Logs: Check the agent's systemd journal for pruning messages:

  journalctl -u flightctl-agent -f | grep -i pruning

• Log Messages: The agent logs messages such as:

  • "Starting pruning of X eligible images and Y eligible artifacts"
  • "Pruning complete: removed X of Y eligible images, Z of W eligible artifacts"

Creating Applications

Creating OCI Registry Application Package

You can package both Compose and Quadlet applications as OCI images for distribution via container registries.

Compose OCI Image Package

Define the application's functionality with the Compose specification and embed the compose file in a scratch container. Then build and push the container to your OCI registry.

FROM scratch

COPY podman-compose.yaml /podman-compose.yaml

Quadlet OCI Image Package

For Quadlet applications, package your quadlet unit files (.container, .volume, .network, etc.) in a scratch container:

FROM scratch

COPY web.container /web.container
COPY app-net.network /app-net.network

OCI Artifact Packages

OCI artifacts provide a simpler alternative to OCI images for packaging Compose and Quadlet applications.

Creating OCI Artifacts: OCI artifacts can be created using tools like ORAS or Podman.

Using Podman:

# Create a Compose artifact
podman artifact add quay.io/my-org/my-compose-app:v1.0 podman-compose.yaml

# Create a Quadlet artifact with multiple files
podman artifact add quay.io/my-org/my-quadlet-app:v1.0 app.container app.volume

# Create a Quadlet artifact with a single tar.gz bundle
tar czf app-files.tar.gz web.container app-net.network config/
podman artifact add quay.io/my-org/my-quadlet-app:v1.0 app-files.tar.gz

File Format Support:

The Flight Control agent automatically processes artifact contents:

• Raw files: Copied directly to the application directory
• Tar archives: Files with .tar, .tar.gz, or .tgz extensions are automatically unpacked
• Directories: Recursively copied to the application directory

The agent will automatically extract app-files.tar.gz when deploying the application.

Note

Podman version 5.5 or above is required for artifact use.

After building and pushing either package type, reference the image in spec.applications[]:

apiVersion: flightctl.io/v1beta1
kind: Device
metadata:
  name: some_device_name
spec:
  applications:
  - name: my-app
    image: quay.io/my-org/my-app:v1.2.3
    appType: container  # or quadlet
    runAs: flightctl

Specifying Applications Inline in the Device Spec

Both Compose and Quadlet application manifests can be specified inline in a device's specification, so building an OCI Registry Application Package is not required.

The Inline Application Provider accepts a list of application content with the following parameters:

Parameter	Description
Path	The relative path to the file on the device. Note that any existing file will be overwritten.
Content (Optional)	The plain text (UTF-8) or base64-encoded content of the file.
ContentEncoding	How the contents are encoded. Must be either "plain" or "base64". Defaults to "plain".

Compose Inline Example:

apiVersion: flightctl.io/v1beta1
kind: Device
metadata:
  name: some_device_name
spec:
[...]
  applications:
    - name: my-app
      appType: compose
      inline:
        - content: |
            version: "3.8"
            services:
              service1:
                image:  quay.io/flightctl-tests/alpine:v1
                command: ["sleep", "infinity"]
          path: podman-compose.yaml
[...]

Note

Inline compose applications can have at most two paths. The first should be named podman-compose.yaml, and the second (override) must be named podman-compose.override.yaml.

Creating Quadlet Applications

Quadlet applications use Podman Quadlet to manage containers as native systemd services. This allows full integration with systemd's dependency management, restart policies, resource limits, and logging.

We recommend running regular workloads under a rootless podman instance under the provided flightctl user's systemd instance on the agent machine. This user is automatically created when installing the flightctl-agent package. You do this by specifying the runAs: flightctl option in the application yaml config. If runAs is not specified, the application will run under the root podman and systemd instance.

You may also provision your own user to run rootless applications under, but it is necessary to run loginctl enable-linger <username> to ensure the systemd instance for that user runs in the background without the user being logged in. Also, the user needs to have a home directory set and writable by that user.

Reasons for running an application under the root podman include:

• Exposing ports < 1024 from the host
• Accessing volumes where root permissions are needed
• Other limitations to be aware of

But many applications should not need access to the root podman instance.

Note

If you want to set cpu limits on your rootless quadlet application, you might need to add configuration to systemd to allow delegation of cpu cgroups to non-root users. See this troubleshooting guide for more details.

Supported Quadlet File Types

Flight Control supports the following Quadlet file types:

File Extension	Description
.container	Defines a single Podman container
.volume	Defines and manages a Podman volume (can be populated from OCI images)
.network	Defines a Podman network
.image	Manages a container image (ensures it's pulled and available)
.pod	Defines a group of containers sharing resources

Important

At least one workload file (.container) must be included in every quadlet application. Supporting files like .volume, .network, .image, and .pod can be used alongside container files but cannot be deployed alone.

Note: The following Quadlet types are not supported:

• .build - Building images from Containerfile (use prebuilt images instead)
• .artifact - Experimental OCI artifact support
• .kube - Kubernetes YAML support

File Naming Requirements

All Quadlet file names must be unique across all systemd service files on a Device. The Flight Control agent automatically namespaces quadlet files to prevent service name collisions between applications.

Important

Quadlet files can reference other quadlet files by name (e.g., Volume=my-data.volume). However, only quadlet files defined within the same application are guaranteed to preserve their references. Cross-application references are not supported due to namespacing.

Example - Valid references (same application):

applications:
  - name: web-app
    appType: quadlet
    runAs: flightctl
    inline:
      - path: app-net.network
        content: |
          [Network]
          NetworkName=app-network
      - path: backend.container
        content: |
          [Container]
          Image=quay.io/myorg/backend:v1
          Network=app-net.network  # ✓ Valid - same application

Example - Invalid references (cross-application):

applications:
  - name: network-config
    appType: quadlet
    runAs: flightctl
    inline:
      - path: shared-net.network
        content: |
          [Network]
          NetworkName=shared-network
  - name: web-service
    appType: quadlet
    runAs: flightctl
    inline:
      - path: web.container
        content: |
          [Container]
          Image=quay.io/myorg/web:v1
          Network=shared-net.network  # ✗ Invalid - different application

Inline Quadlet Examples

Simple Container Example:

apiVersion: flightctl.io/v1beta1
kind: Device
metadata:
  name: some_device_name
spec:
  applications:
    - name: nginx-server
      appType: quadlet
      runAs: flightctl
      inline:
        - path: nginx.container
          content: |
            [Unit]
            Description=Nginx web server

            [Container]
            Image=quay.io/library/nginx:latest
            PublishPort=8080:80

            [Service]
            Restart=always

            [Install]
            WantedBy=default.target

Container with Volume:

spec:
  applications:
    - name: postgres-db
      appType: quadlet
      runAs: flightctl
      inline:
        - path: db.volume
          content: |
            [Volume]
            VolumeName=postgres-data

        - path: postgres.container
          content: |
            [Container]
            Image=quay.io/library/postgres:16
            Volume=db.volume:/var/lib/postgresql/data:Z
            Environment=POSTGRES_PASSWORD=secret

            [Service]
            Restart=always

            [Install]
            WantedBy=default.target

Container with Host Mount Volume:

spec:
  applications:
    - name: nginx
      appType: quadlet
      runAs: flightctl
      inline:
        - path: nginx.container
          content: |
            [Container]
            Image=quay.io/library/nginx:v1.25
            Volume=/etc/nginx/html:/usr/share/nginx/html:ro,Z

            [Service]
            Restart=on-failure

            [Install]
            WantedBy=default.target

Multi-Container with Network:

spec:
  applications:
    - name: web-app
      appType: quadlet
      runAs: flightctl
      inline:
        - path: app-net.network
          content: |
            [Network]
            NetworkName=app-network

        - path: backend.container
          content: |
            [Container]
            Image=quay.io/myorg/backend:v1
            Network=app-net.network

        - path: frontend.container
          content: |
            [Container]
            Image=quay.io/myorg/frontend:v1
            Network=app-net.network
            PublishPort=8080:80

Environment Variables:

Environment variables are injected using the envVars field. The agent generates a .env file and systemd drop in configuration to apply them:

spec:
  applications:
    - name: api-server
      appType: quadlet
      runAs: flightctl
      envVars:
        DATABASE_HOST: "db.internal"
        API_PORT: "8080"
        LOG_LEVEL: "info"
      inline:
        - path: api-server.container
          content: |
            [Container]
            Image=quay.io/myorg/api-server:v2.1
            PublishPort=8080:8080

Volume from OCI Image:

Volumes can be populated from OCI images, useful for delivering configuration files or data:

spec:
  applications:
    - name: config-app
      appType: quadlet
      runAs: flightctl
      inline:
        - path: config.volume
          content: |
            [Volume]
            Driver=image
            Image=quay.io/myorg/app-config:v1

        - path: app.container
          content: |
            [Container]
            Image=quay.io/myorg/myapp:v1
            Volume=config.volume:/config:ro

Runnable Container Applications

To reduce the friction in running simple applications, the container application type is provided. For more complicated applications, quadlet definitions are the recommended approach.

Supported Properties:

• Image - Required - Reference to OCI runnable image
• RunAs - Optional - This determines which container runtime the application will run under. By default it runs under the root podman/systemd instance. If set to a non-root user, it runs under a rootless podman instance for that user.
• Environment Variables - Optional - Variables to be injected into the running container
• Port Mappings - Optional - Must be in the format hostPort:containerPort, with each port limited in the range of 1-65535
• CPU Limits - Optional - Positive decimal number (e.g., "1.5", "2", "0.5")
• Memory Limits - Optional - Number followed by optional unit - b (bytes), k (kibibytes), m (mebibytes), or g (gibibytes). Examples: "512m", "2g", "1024k"
• Named Volume Mounts - Optional - Persistent volumes mounted within the container
• Artifact Backed Mounts - Optional - Volumes mounted within the container that are populated via OCI artifacts

Note

The volume mount path format supports standard Podman mount syntax: destination[:options]

Note

Artifact backed mounts must adhere to the requirements defined in OCI artifact requirements.

Runnable Container Example

spec:
  applications:
    - name: production-api
      appType: container
      runAs: flightctl
      image: quay.io/myorg/production-api:v2.3.1 # the runnable image
      envVars:
        DATABASE_URL: "postgresql://db:5432/app"
        LOG_LEVEL: "info"
        CACHE_SIZE: "1000"
      ports:
        - "8080:80"
        - "9090:9090"  
      resources:
        limits:
          cpu: "4.0"
          memory: "2g"
      volumes:
        - name: config # named volume mounted as readonly
          mount:
            path: "/app/config:ro"
        - name: logs # named volume mounted with no configuration
          reclaimPolicy: Retain
          mount:
            path: "/app/logs"
        - name: ml-models # OCI artifact backed volume
          mount:
            path: "/mnt/models"
          image:
            reference: quay.io/myorg/models:latest
            pullPolicy: Always

Adding Application Volumes

Note

This feature requires the Flight Control Agent to run with Podman version 5.5 or higher.

Compose and Quadlet applications can declare persistent data volumes that are populated from OCI artifacts. This allows delivering large datasets (such as ML models or static assets) as part of the application deployment.

Volumes are declared under each application in the volumes field. These volumes are mounted into the application containers via the Compose volumes section.

Note

Quadlet applications support native volume definitions (.volume files) for image based volumes.

Note

Container applications support volume definitions, but require a mount location.

Specifying Volumes

Each volume definition includes:

Field	Description
name	Logical volume name. Must match the volume name referenced in the Compose file.
image.reference	Fully qualified OCI artifact reference containing the volume contents.
image.pullPolicy	(Optional) Defines pull behavior: Always, IfNotPresent, or Never. Defaults to IfNotPresent if not specified.
reclaimPolicy	(Optional) Defines what happens to the managed volume when the application is removed. Only Retain is currently supported (meaning the volume is preserved). Defaults to Retain if not specified.

Important

We recommend using image- or artifact-backed volumes only for static content such as configuration files, models, or other seed data. They get refreshed on every update, so any changes inside them are overwritten. If you need data to survive updates (databases, uploads, logs), use a mount-based volume instead.

Note

Volumes created directly inside Compose or Quadlet manifests without a matching entry in the API are treated as user-managed and are always retained.

Important

In the Compose file, volumes must be declared as external: true to allow the agent to handle preparation and mounting.

Example Inline Application with Volume

apiVersion: flightctl.io/v1beta1
kind: Device
metadata:
  name: some_device_name
[...]
spec:
  applications:
    - name: my-inline
      appType: compose
      inline:
        - path: docker-compose.yaml
          content: |
            version: "3.8"
            services:
              service1:
                image: quay.io/flightctl-tests/alpine:v1
                command: ["sleep", "infinity"]
                volumes:
                  - my-data:/data
            volumes:
              my-data:
                external: true
      volumes:
        - name: my-data
          image:
            reference: quay.io/flightctl-tests/models/gpt2
            pullPolicy: IfNotPresent

Example Quadlet Inline Application with Volume

apiVersion: flightctl.io/v1beta1
kind: Device
metadata:
  name: some_device_name
[...]
spec:
  applications:
    - name: config-app
      appType: quadlet
      runAs: flightctl
      inline:
        - path: config.volume
          content: |
            [Volume]
            Driver=image
            Image=quay.io/myorg/app-config:v1

        - path: app.container
          content: |
            [Container]
            Image=quay.io/myorg/myapp:v1
            Volume=config.volume:/config:ro # reference to image backed volume
            Volume=my-data:/mnt/models/gpt2 # reference to artifact backed volume
      volumes:
        - name: my-data
          image:
            reference: quay.io/flightctl-tests/models/gpt2
            pullPolicy: IfNotPresent

OCI Artifact Requirements

Volume images must follow the OCI artifact specification:

• Published as OCI images (media type: application/vnd.oci.image.manifest.v1+json).
• Contain one or more layers representing the volume file contents.
• Hosted on any OCI-compatible registry accessible by the device.

Tip

If the artifact contains more than one layer, the mount path should be an already existing directory, into which the layers will each be copied as a separate file using the name specified in the layer's org.opencontainers.image.title field. For single layer archives if the mount path does not exist as a directory the single layer will be extracted as a file at that path, otherwise it will be placed into the existing directory using the file name in the name field for the layer.

Note

Artifact downloads adhere to the pull-timeout configuration.

Device Requirements

The following are required on the device to support application volumes:

• Podman 5.5 or newer installed.
• OCI registry authentication (if needed) must be configured prior to deployment.

Compose Applications:

• podman-compose installed.

Using Device Lifecycle Hooks

You can use device lifecycle hooks to make the agent run user-defined commands at specific points in the device's lifecycle. For example, you can add a shell script to your OS images that backs up your application data and then specify that this script shall be run and complete successfully before the agent can start updating the system.

The following device lifecycle hooks are supported:

Lifecycle Hook	Description
beforeUpdating	This hook is called after the agent completed preparing for the update and before actually making changes to the system. If an action in this hook returns with failure, the agent aborts the update.
afterUpdating	This hook is called after the agent has written the update to disk. If an action in this hook returns with failure,the agent will abort and roll back the update.
beforeRebooting	This hook is called before the agent reboots the device. The agent will block the reboot until running the action has completed or timed out. If any action in this hook returns with failure, the agent will abort and roll back the update.
afterRebooting	This hook is called when the agent first starts after a reboot. If any action in this hook returns with failure, the agent will report this but continue starting up.

Refer to the Device API status reference a state diagram defining when each device lifecycle hook is called by the agent.

Device lifecycle hooks can be defined by adding rule files to one of two locations in the device's filesystem, whereby ${lifecyclehook} is the all-lower-case name of the hook to be defined:

• Rules in the /usr/lib/flightctl/hooks.d/${lifecyclehook}/ drop in directory are read-only and thus have to be added to the OS image during image building.
• Rules in the /etc/flightctl/hooks.d/${lifecyclehook}/ drop in directory are read-writable and can thus be updated at runtime using the methods described in Managing OS Configuration.

If rules are defined in both locations they will be merged, whereby files under /etc take precedence over files of the same name under /usr. If multiple rule files are added to a hook's directory, they are processed in lexical order of their file names.

A rule file is written in YAML format and contains a list of one or more actions. An action can be to run an external command ("run action"). When multiple actions are specified for a hook, these actions are performed in sequence, finishing one action before starting the next. If an action returns with failure, later actions will not be executed.

A run action takes the following parameters:

Parameter	Description
Run	The absolute path to the command to run, followed by any flags or arguments. Example: /usr/bin/nmcli connection reload. Note that the command is not executed in a shell, so you cannot use shell variables like $FOO_PATH or chain commands (| or ;). However, it is possible to start a shell yourself if necessary by specifying the shell as command to run. Example: /usr/bin/bash -c 'echo foo'
EnvVars	(Optional) A list of key/value-pairs to set as environment variables for the command.. Note to use these variables MY_VAR: value, execute the command through a shell like /bin/sh -c 'command "$MY_VAR"'.
WorkDir	(Optional) The directory the command will be run from.
Timeout	(Optional) The maximum duration allowed for the action to complete. The duration must be be specified as a single positive integer followed by a time unit. Supported time units are s for seconds, m for minutes, and h for hours. Default: 10s
If	(Optional) A list of conditions that must be true for the action to be run (see below). If not provided, actions will run unconditionally.

Note

When using a shell with run, the executed environment does not inherit the system environment, any required environment variables must be provided explicitly via the envVars field in the API.

- run: /usr/bin/bash -c "until [ -f $KUBECONFIG ]; do sleep 1; done"
  timeout: 5m
  envVars:
    KUBECONFIG: "/var/lib/microshift/resources/kubeadmin/kubeconfig"

By default, actions are performed every time the hook is triggered. However, for the afterUpdating hook you can use the If parameter to add conditions that must be true for an action to be performed, otherwise the action will be skipped.

In particular, to only run an action if a given file or directory has changed during the update, you can define a "path condition" that takes the following parameters:

Parameter	Description
Path	An absolute path to a file or directory that must have changed during the update as condition for the action to be performed. Paths must be specified using forward slashes (/) and if the path is to a directory it must terminate with a forward slash /. If you specify a path to a file, the file must have changed to satisfy the condition. If you specify a path to a directory, a file in that directory or any of its subdirectories must have changed to satisfy the condition.
Op	A list of file operations (created, updated, removed) to further limit the kind of changes to the specified path as condition for the action to be performed.

If you have specified a "path condition" for an action in the afterUpdating hook, you have the following variables that you can include in arguments to your command and that will be replaced with the absolute path(s) to the changed files:

Variable	Description
${ Path }	The absolute path to the file or directory specified in the path condition.
${ Files }	A space-separated list of absolute paths of the files that were changed (created, updated, or removed) during the update and are covered by the path condition.
${ CreatedFiles }	A space-separated list of absolute paths of the files that were changed (created, updated, or removed) during the update and are covered by the path condition.
${ UpdatedFiles }	A space-separated list of absolute paths of the files that were updated during the update and are covered by the path condition.
${ RemovedFiles }	A space-separated list of absolute paths of the files that were removed during the update and are covered by the path condition.

The Flight Control Agent comes with a built-in set of rules defined in /usr/lib/flightctl/hooks.d/afterupdating/00-default.yaml:

If files changed below	then the agent runs	Description
/etc/systemd/system/	systemctl daemon-reload	Changes to systemd units will be activated by signaling the systemd daemon to reload the systemd manager configuration. This will rerun all generators, reload all unit files, and recreate the entire dependency tree.
/etc/NetworkManager/system-connections/	nmcli conn reload	Changes to Network Manager system connections will be activated by signaling Network Manager to reload all connections.
/etc/firewalld/	firewall-cmd --reload	Changes to firewalld's permanent configuration will be activated by signaling firewalld to reload firewall rules as new runtime configuration.
/etc/flightctl/conf.d/	kill -HUP ${FLIGHTCTL_AGENT_PID}	Changes to configuration drop in files will automatically trigger the agent to reload its configuration by sending itself a SIGHUP signal. This ensures that configuration changes in /etc/flightctl/conf.d/ take effect immediately after being applied.

Monitoring Device Resources

You can set up monitors for device resources and define alerts when the utilization of these resources crosses a defined threshold. When the agent alerts the Flight Control service, the service sets the device status to "degraded" or "error" (depending on the severity level) and may suspend the rollout of updates and alarm the user as a result.

Resource monitors take the following parameters:

Parameter	Description
MonitorType	The resource to monitor. Currently supported resources are "CPU", "Memory", and "Disk". [TODO: Check whether the "Custom" resource type is implemented.]
SamplingInterval	The interval in which the monitor samples utilization, specified as positive integer followed by a time unit ('s' for seconds, 'm' for minutes, 'h' for hours).
AlertRules	A list of alert rules.
Path	(Disk monitor only) The absolute path to the directory to monitor. Utilization reflects the filesystem containing the path, similar to df, even if it’s not a mount point.

Alert rules take the following parameters:

Parameter	Description
Severity	The alert rule's severity level out of "Info", "Warning", or "Critical". Only one alert rule is allowed per severity level and monitor.
Duration	The duration that resource utilization is measured and averaged over when sampling, specified as positive integer followed by a time unit ('s' for seconds, 'm' for minutes, 'h' for hours). Must be smaller than the sampling interval.
Percentage	The utilization threshold that triggers the alert, as percentage value (range 0 to 100 without the "%" sign).
Description	A human-readable description of the alert. This is useful for adding details about the alert that might help with debugging. By default it populates the alert as <severity>: <type> load is above <percentage>>% for more than <duration>

Monitoring Device Resources on the Web UI

Monitoring Device Resources on the CLI

To monitor resource utilization, add resource monitors in the resources: section of the device's specification.

For example, to monitor disk utilization on the filesystem associated with the path /applications, which can trigger a warning alert if the average utilization exceeds 75% for more than 30 minutes and a critical alert if it exceeds 90% for over 10 minutes with a sampling interval of 5 seconds.

apiVersion: flightctl.io/v1beta1
kind: Device
metadata:
  name: some_device_name
spec:
[...]
  resources:
  - monitorType: Disk
    samplingInterval: 5s
    path: /application_data
    alertRules:
    - severity: Warning
      duration: 30m
      percentage: 75
      description: Disk space for application data is >75% full for over 30m.
    - severity: Critical
      duration: 10m
      percentage: 90
      description: Disk space for application data is >90% full over 10m.
[...]

Note

When a critical disk alert is active, device upgrades that require downloading OCI images will automatically fail to prevent upgrade failures due to insufficient disk space. The upgrade will fail with an error message prompting you to clear storage before attempting the upgrade again.

Accessing Devices Remotely

For troubleshooting an edge device, a user with the appropriate authorization (get permission on the devices/console resource) can remotely connect to the device's console through the agent. This does not require an SSH connection and so works even if that device is on a private network (behind a NAT), has a dynamic IP address, or has its SSH service disabled.

Accessing Devices on the Web UI

Accessing Devices on the CLI

To connect, use the flightctl console command specifying the device's name, and the agent will establish the console connection the next time it calls home:

flightctl console device/<some_device_name>

To disconnect, enter "exit" on the console. To force-disconnect, press newline followed by ~. (tilde period).

The console can be used to run commands on the device, for example to check the status of the flightctl-agent service:

flightctl console device/<some_device_name> -- systemctl status flightctl-agent

The console can be used to download a file from the device to your local machine, for example to download the systemd journal log:

flightctl console device/<some_device_name> -- journalctl -o short-precise --no-pager > journal.log

Decommissioning Devices

Decommissioning a device is the proper way to unenroll it and permanently remove it from Flight Control management. When a user requests the decommissioning of a device, the Flight Control service signals to the Flight Control agent to run a decommissioning process. This process includes erasing the agent's management certificate and key and with it the device's Flight Control identity. This is an action that cannot be undone. Decommissioning should be performed before deleting a device.

To decommission a device:

flightctl decommission devices/<some_device_name>

You can see that the decommissioning request was properly received by the Flight Control service when it includes a decommissioning target in its device specification and its lifecycle status (status.lifecycle.status) moves to Decommissioning:

$ flightctl get devices/<some_device_name> -o yaml

...
spec:
  decommissioning:
    target: Unenroll
...
status:
  ...
  lifecycle:
    status: Decommissioning
...

Once the device receives the decommissioning request from the server, it will acknowledge the request with a decommissioning Condition that can also be seen in the device info:

$ flightctl get devices/<some_device_name> -o yaml

...
status:
...
  conditions:
  ...
    - lastTransitionTime: "2025-03-05T20:40:48.443917332Z"
    message: The device has completed decommissioning and will wipe its management
      certificate
    reason: Completed
    status: "True"
    type: DeviceDecommissioning

When the device has completed its decommissioning steps, the status.lifecycle.status field will show the value Decommissioned. At this point, it is safe to delete the device with:

flightctl delete devices/<some_device_name>

Understanding Device Error Messages

Flight Control provides structured error messages within device status conditions to identify update or operation failures. These messages identify the time, phase, component, resource, and reason for a failure. Error categorization uses gRPC status codes to provide consistent error classification across all operations.

Error Message Syntax

Error messages follow this pattern:

[timestamp] While <Phase>, <Component> failed [for "<Element>"]: <Category> issue - <Status Message>

Field	Description	Allowable Values / Examples
Phase	Update stage	Preparing (Preparation), ApplyingUpdate (Sync), Rebooting (Activation)
Component	System area	os, config, applications, resources, update policy, lifecycle, systemd
Element	Specific resource	File paths (/etc/app.conf), Service names, Volume names, Image refs
Category	Functional area	See Error Categories table below.
Status	Human-readable detail	Description of the specific error, derived from gRPC status codes.

Error Categories

Category	Status Messages	Common Causes
Network	"Service unavailable", "Request timed out"	Connectivity issues, DNS failure, registry unreachable
Configuration	"Invalid configuration", "Precondition not met"	Invalid YAML/JSON, missing fields, policy waiting
Filesystem	"Required resource not found", "Resource already exists"	Missing files, directory conflicts
Security	"Permission denied", "Authentication failed"	Insufficient permissions, auth errors
Storage	"Unrecoverable data loss detected"	Data corruption, storage hardware failure
Resource	"Insufficient resources"	Disk full, Out of memory
System	"Operation cancelled", "Internal error", "Not supported"	Internal faults, feature gaps, manual aborts

Viewing Errors

To view error conditions, retrieve the device status in YAML format:

flightctl get device/<device-name> -o yaml

Output Example:

status:
  conditions:
  - type: DeviceUpdating
    status: "False"
    reason: Error
    # Breakdown: [Timestamp] While [Phase], [Component] failed for [Element]: [Category] - [Status Code]
    message: "[2025-01-15 10:30:00 UTC] While Preparing, applications failed for \"myapp\": Network issue - Service unavailable"

Scheduling Updates and Downloads

The Flight Control agent supports time-based scheduling for update and download operations using cron style expressions. This allows you to restrict system modifications to defined maintenance windows or operational periods.

Each device can define two independent schedules in the updatePolicy section of the DeviceSpec:

• downloadSchedule: Defines when the device is allowed to download update artifacts such as OS image layers.
• updateSchedule: Defines when the device is allowed to apply updates.

Each schedule supports:

Parameter	Description
at	(Required) A cron expression specifying valid run times.
timeZone	(Optional) The time zone used to evaluate the schedule. Defaults to the device's local system time zone. Must be a valid IANA time zone.
startGraceDuration	(Required) A duration string that extends the allowed start time window after a schedule trigger. Follows the Go duration format, such as "1h" or "45m".

The Flight Control agent evaluates these schedules during its control loop to determine whether each policy is currently allowed to proceed. While the device waits for the update window the device status will read OutOfDate. For more details please see Device API Statuses.

Tip

Use crontab guru to create and test cron expressions interactively.

Examples

updatePolicy:
  downloadSchedule:
    at: "0 2 * * *"               # every day at 2:00 AM
    timeZone: "America/New_York"
    startGraceDuration: "30m"     # allow downloads until 2:30 AM
  updateSchedule:
    at: "0 4 * * 1"               # every Monday at 4:00 AM
    timeZone: "America/New_York"
    startGraceDuration: "1h"      # allow update until 5:00 AM

updatePolicy:
  updateSchedule:
    at: "0 5 14 5 *"              # May 15th at 5:00 AM
    timeZone: "America/New_York"
    startGraceDuration: "2h"      # allow update until 7:00 AM

Note

The startGraceDuration field is required and allows for potential delays in agent execution. Without a sufficient grace period, the update window may be missed. Once an update begins within the allowed window, there is no enforced timeout the update may continue running beyond the grace period.