group(enhancements): Albacore

[nfebe]

Albacore release. Theme: full deployment lifecycle and self-healing operations.
Umbrella for the next release; each item below ships this cycle.

Features

• Observability & self-recovery (feat: Add AI-native observability and self-recovery module #148): shipped in feat(observ): Add observability engine, plugin tool bridge, notifications #165 and feat(observ): Add observability UI, plugin slots, notifications settings ui#78. OTel-semconv metrics with a native time-series UI, health self-heal scoped to FlatRun-managed deployments only (capped retries with cooldown, so an external container can't trigger a restart loop), a core notification system wite email/webhook targets and test-send, and a plugin framework that lets a plugin inject UI sections, contribute settings, and expose tools the AI assistant can call. Follow-ups tracked under Improvements; none blocked the release.
• S3-compatible backups (feat(backup): Add remote S3-compatible backup destinations #168, ui: Object stores, a base component library, and dark-mode fixes ui#79): backups now mirror to remote S3-compatible object storage (AWS, R2, B2, MinIO) on top of the always-local copy, best-effort so a remote outage never fails a backup. What/how/where is documented in docs/BACKUPS.md; object-storage secrets live in the credential manager, never the flat-file config. This also seeds the object store abstraction (external vs managed stores, a MinIO template so FlatRun can run its own S3 endpoint), specced in docs/OBJECT_STORES.md. Follow-up slices, still open: one-click managed deploy with auto-register, an object browser, deployment consumption, and store-to-store replication.
• Dashboard & API performance (performance: Loading deployments and the dashboard is generally too slow #154): progressive / lazy loading for the deployments
  list and dashboard instead of loading everything at once (parked: the backend is no
  longer the bottleneck, so whether the UI still needs this is unmeasured); profile and
  optimize the backend paths behind those views. This item is about how fast FlatRun's own
  UI and API respond, not how fast the deployed apps serve their traffic (tracked
  separately below).
  Backend profiling (this cycle): deployment status was recomputed on every request by
  shelling out to the docker compose CLI with nothing cached, so latency scaled with
  the number of deployments and was dominated by process startup. Listing N deployments
  spawned 2N to 4N docker processes; a single detail view spawned 4 to 8. The backend
  status path is addressed in perf(deployments): Read deployment status from the Docker engine #171: status now comes from one Engine API
  query covering every deployment at once, so listing is flat with the deployment count
  instead of scaling with it. The query is deliberately restricted to live containers,
  which is what keeps reported status identical to what the compose CLI reported.
  Progressive / lazy loading on the UI side is still open. Concrete work:
  • Memoize compose-command detection so docker compose version runs once per process,
    not once per runCompose.
  • Serve list/detail status from one Engine API ContainerList filtered by
    com.docker.compose.project labels, reusing the existing APIClient, instead of
    per-deployment docker compose ps.
  • Collapse the detail path that runs the compose sequence twice into one status read.
  • Skip the status fan-out for endpoints that only need on-disk metadata (certificates,
    proxy sync, traffic), reading via FindDeployments rather than ListDeployments.
    Cluster is excluded: it marshals the full deployment list to clients with status
    included, so it does need the status read.
  • Add a short-TTL cache for container status so request bursts don't each recompute
    it. Parked: it was specced when a list cost ~486ms and spawned 2N to 4N processes.
    A list is now ~22ms, of which ~20ms is the single Engine API call, so a cache would
    save ~20ms in exchange for stale status right after a start/stop and the invalidation
    needed to avoid it. The call scales with live container count, so revisit if a busy
    host shows it in real numbers.
  • Record before/after list and detail timings at 1, 10, and 50 deployments: list 128ms
    to 25ms at 1, 104ms to 22ms at 10, and 486ms to 26ms at 50; detail ~150ms to ~22ms.
• Deployment serving performance: how fast a deployed app answers real end-user
  requests through FlatRun's reverse proxy, distinct from dashboard/API speed above. The
  proxy is the shared path every deployment's traffic crosses, so its defaults set the
  floor for everyone. Establish a baseline and remove the obvious drags:
  • Measure request latency and throughput through the proxy to a deployment (p50/p95,
    requests/sec) against hitting the container directly, so proxy overhead is a number.
  • Audit the generated nginx/openresty vhost defaults: keepalive to upstreams, gzip,
    HTTP/2, sensible proxy buffer sizes, and static-asset caching headers.
  • Confirm no per-request work scales badly (excess logging, TLS session cache off,
    DNS re-resolution of upstreams).
  • Surface per-deployment request latency in the observability UI so a slow app is
    visible next to its CPU and memory, closing the loop with feat: Add AI-native observability and self-recovery module #148.
  • Document knobs an operator can turn per deployment (worker limits, cache toggles).
• Integrations: deploy from a GitHub or git URL and from uploaded code, not only
  from compose content or a template. Today a deployment must already have a compose
  file; this adds source-based deployment with auth.
• Builders: build code and then deploy it. Today only an existing compose build:
  section runs. This adds build-from-source (Dockerfile detection, buildpacks / nixpacks)
  with build config (args, cache, secrets).
• More AI tools (feat(ai): Add more tools for quick actions, file editing, summarization etc #147): expand the AI assistant with quick actions, file editing,
  summarization, and related tools (UI side: feat(ai): Add AI workflow to file viewers/editors accross the application ui#71).
• Persist AI chat sessions: store conversation history so chats survive reloads and
  can be resumed, instead of being lost per session.
• Marketplace-sourced templates: the deployment flow's templates should come from the
  marketplace (api.flatrun.dev) and be kept current, with flatrun/marketplace on GitHub
  as the fallback source for public publications when the API is unreachable. Today the
  deploy flow uses the templates embedded in the agent binary.
• Firewall enforcement: the built-in firewall app persists, validates, and previews a
  host-wide inbound/outbound policy, but does not enforce it yet. Translate rules to
  nftables/iptables with a safeguard that preserves the active SSH session before a
  default-deny inbound policy takes effect.

Improvements

• nginx vhost generation (nginx vhost generation: WebSocket timeouts, unconditional upgrade header, ssl_stapling noise, missing target validation #156): configurable / long WebSocket proxy timeouts;
  conditional Connection: upgrade via a map instead of unconditional; skip
  ssl_stapling when the cert has no OCSP responder; validate the target service/port
  actually listens before saving a vhost.

• Streamed deployment actions (feat: Stream start/stop progress and run deployment actions as jobs #150): run start/stop/restart as background jobs that
  return a job id; job status and buffered output survive a page reload; stream compose
  output over the WebSocket keyed by job id with a poll fallback; serialize concurrent
  actions per deployment. Shipped in feat(agent): Run deployment and service actions as streamed jobs #163 and feat(ui): Stream deployment and service action progress ui#76. (Jobs are
  in-memory with ~15-minute retention: reload survives, an agent restart does not.)

• Security trace & false-block fix (fix: Security module blocking after 1 404 #153): stop blocking on a single 404; give every
  blocked IP a trace of the events (counts, paths) that led to the block.

• Effective rebuild / pull / env (bug: Rebuild, pulls etc not effective with picking new env vars #149): expose force-recreate, no-cache rebuild,
  and non-cached pull so updated env vars and images actually take effect, matching what
  currently requires a manual docker compose from the terminal.

• UI design refresh: improve the UI design and adopt Iconify for iconography. First pass shipped in feat(ui): Design refresh with dark mode, assistant, and global search ui#74 (dark mode, design tokens, Iconify, the assistant rework, global search). Per-screen polish and modal consolidation still to follow.

• Seed bind mounts from image content (Seed bind mounts from image content when host path is empty #139): opt-in per-mount seeding so an empty or
  missing host path is populated from the image (mirroring named-volume copy-on-first-use)
  before compose up, only when the target is empty. Fixes single-file config mounts
  becoming directories and empty mounts for images that don't self-populate.

• nginx additional-domain upstream collision (fix(nginx): Multi-domain upstream collides across deployments; server_names_hash not configurable #155): the extra-domain path proxies by
  bare service name (e.g. app), which is not unique across deployments, so a custom
  domain on a shared host intermittently serves another deployment's container. Route by
  the unique deployment name like the primary-domain path.

• nginx security base image (bug: Use flatrun/openresty for security base #106, check: Ensure base nginx image is configured to flatrun/openresty #39): use flatrun/openresty as the nginx base
  instead of upstream openresty/openresty, which is incompatible with the FlatRun
  security configs.

• User-set primary service (enh: Smarter primary service detection in multi-service compose files #111): the smarter fallback shipped (detection now prefers
  app/web before the first service with ports, instead of random map order). Letting the
  user explicitly set a deployment's primary service is still open.

• Propagate metadata-save errors (enh: Propagate SaveMetadata errors during compose updates #110): surface SaveMetadata failures during compose
  updates instead of discarding them, so compose and service.yml can't silently diverge.

• Suppress ACME challenge log noise (enh: Suppress logging for ACME challenge requests in nginx configs #112): set access_log off / log_not_found off
  on the well-known ACME challenge locations so cleaned-up challenge 404s stop burying real
  errors. Complements the fix: Security module blocking after 1 404 #153 security trace.

• --help lists subcommands (enh: --help command flag omits subcommands (update, setup, version) #130): top-level help should list update, setup, and
  version, and help should print usage instead of falling through to start the server.

• Compose validation working directory (fix(compose): Resolve relative env_file against the deployment directory #161): validation hardcodes the working
  directory to ., so a relative env_file (e.g. ./.env) fails on the image-set /
  update path even though the file exists in the deployment directory. Resolve relative
  paths against the deployment directory, as the up path already does.

• Certificate marked for auto renewal not auto-renewed (fix: Auto-renew certificates by default, with honest renew reporting and proxy tuning #167): auto-renewal
  was off by default and, even when on, the global flag could veto a certificate the user had
  marked for renewal. Auto-renewal now defaults on, the renewer runs whenever certbot is
  enabled, and each certificate's own setting wins over the global default. Single Renew also
  stops reporting success when nothing was reissued, with force as an explicit option. Serving
  performance was tuned in the same PR (HTTP/1.1 to upstreams on all routes, buffered access
  logging, wider gzip, larger proxy buffers).

• Routing-only hostnames for externally-fronted proxies (feat(nginx): Support routing-only hostnames with externally-terminated TLS #166): a hostname fronted by
  an external proxy (e.g. dashboard.app.com on Cloudflare proxying to a deployment on
  api.app.com) arrives with an unmatched Host and is rejected instead of routed, and it
  cannot be added as an alias because aliases force certificate issuance that fails when the
  DNS points at the external proxy. Needs an opt-in route-only hostname whose TLS is
  terminated upstream. Root cause and checklist in feat(nginx): Support routing-only hostnames with externally-terminated TLS #166.

• Observability follow-ups (from feat: Add AI-native observability and self-recovery module #148 review, non-blocking): (a) the health watcher
  holds its lock across the docker restart call, briefly blocking health reads during a
  restart; (b) notification target URLs, including SMTP passwords, are stored in cleartext
  in the runtime config, consistent with the existing flat-file secret handling; (c) the
  notification HTTP handlers (including the plugin-token gate on emit) lack boundary tests;
  (d) set_auto_restart is a global toggle but requires a deployment-scoped session, so it
  fails in an unscoped assistant session.

Housekeeping

• Document the release naming system: keep the convention below current and reuse
  it when opening the next umbrella issue.

---

Release naming: umbrella releases are codenamed after sea creatures, advancing one
letter per release. Albacore (A) -> Barnacle (B) -> Cuttlefish (C) -> Dolphin (D) ...
Pick the next unused letter when opening each new group(enhancements): issue.