Skip to content

feat(backup): Add remote S3-compatible backup destinations #168

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
nfebe merged 5 commits into from
Jul 9, 2026
Merged

feat(backup): Add remote S3-compatible backup destinations #168

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
66298ed
feat(backup): Add remote S3-compatible backup destinations
nfebe Jul 7, 2026
03c8192
feat(templates): Add MinIO template and object store spec
nfebe Jul 8, 2026
5e7d5c9
refactor(deps): Promote aws-sdk s3 to a direct dependency
nfebe Jul 8, 2026
57c0600
feat(backup): Record object store kind (external vs managed)
nfebe Jul 8, 2026
0e51f1a
test(api): Cover object store credential and destination endpoints
nfebe Jul 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions config.example.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -126,3 +126,19 @@ ai:
api_key: your-gemini-api-key-here
model: gemini-2.5-flash
timeout: 1m0s
backup:
# Remote destinations that backups are mirrored to after the local copy is
# written. The local copy under <deployments_path>/.flatrun/backups always
# remains the primary. Secrets live in the credential manager, never here:
# credential_id references an s3 credential managed via the API.
destinations: []
# - name: s3-prod
# type: s3
# kind: external # external (connect only) or managed (FlatRun runs it)
# endpoint: https://s3.us-east-1.amazonaws.com
# region: us-east-1
# bucket: flatrun-backups
# prefix: agent-01
# credential_id: <s3-credential-id>
# use_path_style: false
# enabled: true
57 changes: 57 additions & 0 deletions docs/BACKUPS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
# Backups

## What is backed up

A backup captures a deployment as a self-contained archive. Each component is
best-effort: a component that cannot be read is logged and skipped, and the rest
of the backup still completes.

| Component | Source | Included |
|-----------|--------|----------|
| Compose file | `docker-compose.yml` or `compose.yml` | Always |
| Env files | `.env`, `.env.flatrun` | Always |
| Deployment metadata | `.flatrun.yml` | Always |
| Mounted data | `data/`, `uploads/`, `storage/`, `config/`, `logs/` under the deployment | Always, when present |
| Container paths | Paths declared in the deployment's backup spec | When configured |
| Databases | MySQL (`mysqldump`) / PostgreSQL (`pg_dump`) dumps of declared services | When configured |

Containers keep running during a backup. Optional pre and post hooks run inside
a service container around the capture.

## How it is stored

Components are staged, described by a `backup.json` manifest, and written as a
single gzip-compressed tar archive named `<deployment>_<timestamp>.tar.gz`.

## Where it is stored

The primary copy is always local:

```
<deployments_path>/.flatrun/backups/<deployment>/<deployment>_<timestamp>.tar.gz
```

If one or more remote destinations are configured, the archive is also uploaded
to each after the local write. A remote upload failure is logged and never fails
the backup, whose local copy already succeeded. A backup's `locations` field
reports where it currently exists (`local` and/or destination names). Local
retention prunes only the on-disk copies; remote copies are governed by the
bucket's own lifecycle policy.

## Remote (S3-compatible) destinations

Any S3-compatible service works (AWS S3, Cloudflare R2, Backblaze B2, MinIO).

1. Store the access key and secret as an S3 credential
(`POST /api/v1/storage-credentials`, `kind: s3`). Secrets are held by the
credential manager, written to a `0600` file, and never returned by the API
or stored in the agent config.
2. Add a destination under `backup.destinations` referencing that credential by
id (via the config API or `config.yml`). See `config.example.yml` for the
shape.
3. Validate reachability with `POST /api/v1/backup-destinations/test` before
relying on it; it performs a probe write and delete.

Restore and download read from the local copy when present, otherwise stream the
archive from the first remote that holds it, so a backup remains usable after
its local copy has been pruned.
96 changes: 96 additions & 0 deletions docs/OBJECT_STORES.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
# Object Stores

## What this is

An **object store** is a named, S3-compatible endpoint FlatRun knows about. One
abstraction covers every store, whoever runs it, so backups, deployments, and the
UI treat them all the same: each resolves to `(endpoint, region, bucket, credential)`
and is reached through one S3 client.

This supersedes the earlier standalone "backup destination": a backup destination
is now just an object store used as a backup target.

## Two kinds

A store's `kind` records **who runs the store**, nothing else:

- **external**: a store running somewhere else that FlatRun only connects to. You
provide an endpoint and credentials. FlatRun does not manage its lifecycle.
Examples: AWS S3, Cloudflare R2, Backblaze B2, an existing MinIO.
- **managed**: a store FlatRun runs itself, deployed from a template as a container
on the host (MinIO, SeaweedFS, Garage). Because it is a normal FlatRun deployment,
FlatRun knows its endpoint, issues its credentials, and can start, stop, and back
it up like any other deployment. Its data lives in a flat-file bind mount.

Everything downstream (browsing objects, using a store as a backup target, mounting
it into a deployment) is identical across kinds.

## Data model

```
ObjectStore
id string
name string
kind "external" | "managed"
endpoint string # empty means AWS default
region string
bucket string
prefix string # optional key prefix
credential_id string # references an s3 credential in the credential manager
use_path_style bool
deployment string # managed only: the deployment that runs the container
enabled *bool # nil means enabled
backup_target bool # whether backups mirror to this store
```

Secrets never live on the store record. `credential_id` points at an S3 credential
held by the credential manager (written 0600, masked in API responses). For a managed
store, the credential is the one issued to its container at deploy time.

## How backups use it

The backup manager mirrors each new archive to every enabled store with
`backup_target = true`, on top of the always-local copy. List, download, and restore
fall back to a store when the local archive has been pruned. This is the existing
mirror mechanism; only the source of the target list changes from "destinations" to
"object stores marked as backup targets".

## Managed stores via templates

A managed store is created through the normal deploy flow using an object-store
template. A template is a directory under `agent/templates/<name>/` with
`metadata.yml` and `docker-compose.yml`, auto-discovered from the embedded set.

Deploying one:
1. User picks an object-store template (MinIO, SeaweedFS) and deploys it. The
template generates a root credential per deployment.
2. FlatRun registers a `managed` object store linked to that deployment, deriving
the endpoint from the container and storing the issued credential.
3. The store then appears alongside external stores and can be used as a backup
target or, later, mounted into other deployments.

Data stays in the deployment's flat-file `./data` mount, consistent with the
no-hidden-volumes philosophy.

## Slices

1. **Abstraction + managed template** (this cycle): the store model with `kind`,
a MinIO template (SeaweedFS and Garage to follow), and folding backup
destinations into stores. The Object Stores UI lists external and managed stores
and can add an external store or deploy a managed one.
2. **Object browser**: list and manage buckets and objects in a store from the UI
(upload, download, delete), the "visualization" half of the feature.
3. **Deployment consumption**: inject a store's endpoint and issued credentials into
another deployment's environment so apps can use it directly.
4. **Replication**: sync one store to another (managed to external for offsite, or
external to managed for local cache), on a schedule.

## API sketch

- `GET /object-stores` list stores (external and managed), non-secret.
- `POST /object-stores` register an external store.
- `PUT /object-stores/:id`, `DELETE /object-stores/:id`.
- `POST /object-stores/:id/test` connectivity probe (write and delete).
- `POST /object-stores/managed` deploy a managed store from a template, returning the
new store plus the deploy job.
- S3 credentials continue through `/storage-credentials`.
17 changes: 11 additions & 6 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,11 @@ module github.com/flatrun/agent
go 1.25.0

require (
github.com/aws/aws-sdk-go-v2 v1.42.1
github.com/aws/aws-sdk-go-v2/config v1.32.6
github.com/aws/aws-sdk-go-v2/credentials v1.19.6
github.com/aws/aws-sdk-go-v2/service/route53 v1.62.0
github.com/aws/aws-sdk-go-v2/service/s3 v1.105.0
github.com/cloudflare/cloudflare-go v0.116.0
github.com/compose-spec/compose-go/v2 v2.10.1
github.com/creack/pty v1.1.24
Expand Down Expand Up @@ -39,18 +41,21 @@ require (
github.com/DefangLabs/secret-detector v0.0.0-20250403165618-22662109213e // indirect
github.com/Microsoft/go-winio v0.6.2 // indirect
github.com/acarl005/stripansi v0.0.0-20180116102854-5a71ef0e047d // indirect
github.com/aws/aws-sdk-go-v2 v1.41.0 // indirect
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.14 // indirect
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.16 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.16 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.16 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.30 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.30 // indirect
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.4 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.4 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.16 // indirect
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.31 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.13 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.23 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.30 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.31 // indirect
github.com/aws/aws-sdk-go-v2/service/signin v1.0.4 // indirect
github.com/aws/aws-sdk-go-v2/service/sso v1.30.8 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.12 // indirect
github.com/aws/aws-sdk-go-v2/service/sts v1.41.5 // indirect
github.com/aws/smithy-go v1.24.0 // indirect
github.com/aws/smithy-go v1.27.3 // indirect
github.com/buger/goterm v1.0.4 // indirect
github.com/bytedance/sonic v1.13.3 // indirect
github.com/bytedance/sonic/loader v0.2.4 // indirect
Expand Down
34 changes: 22 additions & 12 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,26 +22,36 @@ github.com/anchore/go-struct-converter v0.1.0 h1:2rDRssAl6mgKBSLNiVCMADgZRhoqtw9
github.com/anchore/go-struct-converter v0.1.0/go.mod h1:rYqSE9HbjzpHTI74vwPvae4ZVYZd1lue2ta6xHPdblA=
github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 h1:DklsrG3dyBCFEj5IhUbnKptjxatkF07cF2ak3yi77so=
github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2/go.mod h1:WaHUgvxTVq04UNunO+XhnAqY/wQc+bxr74GqbsZ/Jqw=
github.com/aws/aws-sdk-go-v2 v1.41.0 h1:tNvqh1s+v0vFYdA1xq0aOJH+Y5cRyZ5upu6roPgPKd4=
github.com/aws/aws-sdk-go-v2 v1.41.0/go.mod h1:MayyLB8y+buD9hZqkCW3kX1AKq07Y5pXxtgB+rRFhz0=
github.com/aws/aws-sdk-go-v2 v1.42.1 h1:9eOTgu1z/dVtYpNZ3/8/XbbaX0x/BqE3HUzAzs6K0ek=
github.com/aws/aws-sdk-go-v2 v1.42.1/go.mod h1:5pKeft2eJj+gElQ38Jqg4ibCqh+/AK33/0X3hip7IjM=
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.14 h1:3IZY0XAJquT3aHzbkHfPzy4ACPcEjVG0x87KOwtpqGY=
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.14/go.mod h1:zwM6veDkhGgQFqkBy+uT28AAYpLu+uFMlPl+rCg/73E=
github.com/aws/aws-sdk-go-v2/config v1.32.6 h1:hFLBGUKjmLAekvi1evLi5hVvFQtSo3GYwi+Bx4lpJf8=
github.com/aws/aws-sdk-go-v2/config v1.32.6/go.mod h1:lcUL/gcd8WyjCrMnxez5OXkO3/rwcNmvfno62tnXNcI=
github.com/aws/aws-sdk-go-v2/credentials v1.19.6 h1:F9vWao2TwjV2MyiyVS+duza0NIRtAslgLUM0vTA1ZaE=
github.com/aws/aws-sdk-go-v2/credentials v1.19.6/go.mod h1:SgHzKjEVsdQr6Opor0ihgWtkWdfRAIwxYzSJ8O85VHY=
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.16 h1:80+uETIWS1BqjnN9uJ0dBUaETh+P1XwFy5vwHwK5r9k=
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.16/go.mod h1:wOOsYuxYuB/7FlnVtzeBYRcjSRtQpAW0hCP7tIULMwo=
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.16 h1:rgGwPzb82iBYSvHMHXc8h9mRoOUBZIGFgKb9qniaZZc=
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.16/go.mod h1:L/UxsGeKpGoIj6DxfhOWHWQ/kGKcd4I1VncE4++IyKA=
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.16 h1:1jtGzuV7c82xnqOVfx2F0xmJcOw5374L7N6juGW6x6U=
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.16/go.mod h1:M2E5OQf+XLe+SZGmmpaI2yy+J326aFf6/+54PoxSANc=
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.30 h1:xM/Is9cKMHa8Jj8zkvWhvrFkZsXJV9E+BB4g0HW0duQ=
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.30/go.mod h1:WueJeNDZvK1fMYEWJIkcivBfEzUkTpBhzlrUKKY8EuA=
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.30 h1:jn46zC9LdsVR/ZpMIJqMqb8hHv31BlLx3ulVqNspUOk=
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.30/go.mod h1:1hTMsAgbdS/AtUi4bw8+gUuh1pceo+eXRLfpSuSQj3M=
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.4 h1:WKuaxf++XKWlHWu9ECbMlha8WOEGm0OUEZqm4K/Gcfk=
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.4/go.mod h1:ZWy7j6v1vWGmPReu0iSGvRiise4YI5SkR3OHKTZ6Wuc=
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.4 h1:0ryTNEdJbzUCEWkVXEXoqlXV72J5keC1GvILMOuD00E=
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.4/go.mod h1:HQ4qwNZh32C3CBeO6iJLQlgtMzqeG17ziAA/3KDJFow=
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.16 h1:oHjJHeUy0ImIV0bsrX0X91GkV5nJAyv1l1CC9lnO0TI=
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.16/go.mod h1:iRSNGgOYmiYwSCXxXaKb9HfOEj40+oTKn8pTxMlYkRM=
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.31 h1:3GUprIsfmGcC5SACIyB0e7E0BM1O1b3Erl5CePYIAeQ=
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.31/go.mod h1:7PuV1yl5e2xnUbm+RqvVg5i2iBM8EyijZNoI9wsOoOc=
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.13 h1:mbRIur/BiHK6SKPjoBIXSE/hJ6g6JGRLuxQy1jGjlN4=
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.13/go.mod h1:ITg9em2KbJx1s0y4aqRX5OYWG6HBZ5TVR//OdpEZ2CQ=
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.23 h1:9Fjh6fi/U5JEStVZijmaMpUwE/gvBJj7x2B/PjbO9To=
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.23/go.mod h1:iMoT2f1tClxrWAAnKCXjZQ6LOmfLrMG14wmnWpM+F14=
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.30 h1:/Z5jmNrKsSD7EmDjzAPsm/3L9IuOkzaynklJZ1qX7S4=
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.30/go.mod h1:lEzEZnOosE7zi8Z6royW1cFJTD9fpab4Ul1SBrllewk=
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.31 h1:uao4A3QZ5UmB326V6KF+qRpv9Tjz7IlnlnTbbANntlU=
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.31/go.mod h1:I/1+z0VwL1GhQyLgkoHDlygpUZ+iTAwOQ/NsftiUL2I=
github.com/aws/aws-sdk-go-v2/service/route53 v1.62.0 h1:80pDB3Tpmb2RCSZORrK9/3iQxsd+w6vSzVqpT1FGiwE=
github.com/aws/aws-sdk-go-v2/service/route53 v1.62.0/go.mod h1:6EZUGGNLPLh5Unt30uEoA+KQcByERfXIkax9qrc80nA=
github.com/aws/aws-sdk-go-v2/service/s3 v1.105.0 h1:XptwLL+UHXgafYMIHTy59IRovLbhz3znkxY2uS/pbXU=
github.com/aws/aws-sdk-go-v2/service/s3 v1.105.0/go.mod h1:zdmCoFO/dSI7GlrwsPqFJI+WlFnSU4Tc8TJnlXrM1Do=
github.com/aws/aws-sdk-go-v2/service/signin v1.0.4 h1:HpI7aMmJ+mm1wkSHIA2t5EaFFv5EFYXePW30p1EIrbQ=
github.com/aws/aws-sdk-go-v2/service/signin v1.0.4/go.mod h1:C5RdGMYGlfM0gYq/tifqgn4EbyX99V15P2V3R+VHbQU=
github.com/aws/aws-sdk-go-v2/service/sso v1.30.8 h1:aM/Q24rIlS3bRAhTyFurowU8A0SMyGDtEOY/l/s/1Uw=
Expand All @@ -50,8 +60,8 @@ github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.12 h1:AHDr0DaHIAo8c9t1emrzAlV
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.12/go.mod h1:GQ73XawFFiWxyWXMHWfhiomvP3tXtdNar/fi8z18sx0=
github.com/aws/aws-sdk-go-v2/service/sts v1.41.5 h1:SciGFVNZ4mHdm7gpD1dgZYnCuVdX1s+lFTg4+4DOy70=
github.com/aws/aws-sdk-go-v2/service/sts v1.41.5/go.mod h1:iW40X4QBmUxdP+fZNOpfmkdMZqsovezbAeO+Ubiv2pk=
github.com/aws/smithy-go v1.24.0 h1:LpilSUItNPFr1eY85RYgTIg5eIEPtvFbskaFcmmIUnk=
github.com/aws/smithy-go v1.24.0/go.mod h1:LEj2LM3rBRQJxPZTB4KuzZkaZYnZPnvgIhb4pu07mx0=
github.com/aws/smithy-go v1.27.3 h1:F3Zb497UhhskkfpJmfkXswyo+t0sh9OTBnIHjogWbVY=
github.com/aws/smithy-go v1.27.3/go.mod h1:YE2RhdIuDbA5E5bTdciG9KrW3+TiEONeUWCqxX9i1Fc=
github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
github.com/blang/semver v3.5.1+incompatible h1:cQNTCjp13qL8KC3Nbxr/y2Bqb63oX6wdnnjpJbkM4JQ=
Expand Down
Loading
Loading