Skip to content

CONFIG_AND_USER_MIGRATION.md

Latest commit

 

History

History
225 lines (146 loc) · 10.3 KB

CONFIG_AND_USER_MIGRATION.md

File metadata and controls

225 lines (146 loc) · 10.3 KB
Raw

Config and User Migration

Covers (1) local development: keeping your local users.json and config from being overwritten, and (2) deployment: migrating config/users between instances and consolidating users across Blue/Green.

Table of Contents

Local Development: Keeping Users and Config (Laptop Only)

Applies to: Local development on a laptop only. Docker, AWS EC2 Blue/Green, and other deployments use their own USERS_PATH (e.g. Docker volume, EC2 /opt/oscal/data/users.json) and do not use .env for this.

If config/app/users.json gets reset to a default (e.g. after pulling code or running setup), you can keep your laptop’s user list intact by storing users outside the repo and pointing the app at that file.

Why it can happen

  • setup.sh only creates config/app/users.json from users.json.example when the file is missing. It never overwrites an existing file.
  • Git does not touch config/app/users.json (it’s in .gitignore).
  • If the file is ever deleted or reverted (e.g. by another script or by mistake), the next run may recreate it from the example.

Recommended: use a users file outside the repo

  1. Create a directory and copy your users

    mkdir -p ~/Documents/OSCAL_Reports_data
cp config/app/users.json.backup ~/Documents/OSCAL_Reports_data/users.json

    (If you don’t have a backup, use your current config/app/users.json once you’ve restored it.)

  2. Point the app at that file with .env (from repo root):

    cp .env.example .env

    Edit .env and set an absolute path:

    USERS_PATH=/Users/yourusername/Documents/OSCAL_Reports_data/users.json

    Replace yourusername with your macOS username (use an absolute path; .env does not expand $HOME).

  3. Run the app as usual: npm run dev. The backend loads .env from the repo root and uses USERS_PATH. All user load/save use your external file; config/app/users.json is not used when USERS_PATH is set.

Optional: config.json outside the repo

You can do the same for config.json so it isn’t overwritten:

CONFIG_PATH=/Users/yourusername/Documents/OSCAL_Reports_data/config.json

Copy your current config/app/config.json to that path and use it as the single source of truth.

Safeguards (local)

  • setup.sh: Only creates users.json from users.json.example when config/app/users.json does not exist. If the file exists, setup leaves it unchanged.
  • Backend: Uses USERS_PATH from the environment. On the laptop only (when NODE_ENV is not production), the backend also loads .env from the repo root. Docker and EC2 do not load .env for this; they use USERS_PATH set by the container entrypoint or systemd.

Summary (local)

Goal Action
Keep users across code changes (laptop only) Set USERS_PATH in .env to a path outside the repo and copy your users there once.
Keep config outside repo Set CONFIG_PATH in .env to a path outside the repo.
Never overwrite by mistake Use USERS_PATH so the app doesn’t rely on config/app/users.json.

Deployments: Docker and EC2 Blue/Green are unchanged. They set USERS_PATH (and optionally CONFIG_PATH) via their own mechanisms (Docker entrypoint, systemd, or shell) and do not use .env for users or config.

Instance → Port → Directory Mapping

AWS EC2 (Green / Blue)

Green and Blue are separate EC2 instances; both use the same app port (3020, oscal_app_port in Terraform). Role is distinguished by DEPLOYMENT_ROLE, ALB target group, and S3 log prefix—not by TCP port. Config and users are shared via s3://<bucket>/config/active/.

Role App port Data on instance
Green 3020 /opt/oscal/data (config.json, users.json)
Blue 3020 /opt/oscal/data (same files; synced from S3)

TrueNAS / Docker (legacy, same host)

On TrueNAS (e.g. truenas.keekar.au) or any host running multiple OSCAL containers on one machine, the deploy scripts historically used different host ports so two containers could bind simultaneously:

Port Instance Container name Data directory / volume
3019 Green oscal-report-generator-green data-green (or data-green/ in deploy root)
3020 Blue oscal-report-generator-blue data-blue (or data-blue/ in deploy root)
3021 (not in scripts) — —
  • 3019 → Green → config and users live in data-green (files: config.json, users.json under that directory or its mounted path).
  • 3020 → Blue → config and users live in data-blue (same file names).

Port 3021: The codebase only defines Blue (3020) and Green (3019). If you have a third instance on 3021, it was likely created separately (e.g. another TrueNAS Custom App or chart release). Its data lives in whatever storage path that app was given (e.g. a third host path or PVC for that release). To see which directory a 3021 instance uses, check that app’s Storage or Volumes in the TrueNAS Apps UI, or the deploy path for that clone.

TrueNAS volume paths (examples): If you use scripts/install_from_dockerhub.sh or custom compose paths, Blue might be /mnt/pool/oscal-data-blue and Green /mnt/pool/oscal-data-green. Mounts and data directories follow your docker-compose.yml / install script layout under each clone (see DEPLOYMENT_AND_OPERATIONS.md).

Config Migration

Files to Migrate

  • config/app/config.json – Application settings (AI, messaging, SSO, API gateways, SOA URL).
  • config/app/users.json – User accounts, roles, credentials (PBKDF2 hashed).

Both contain sensitive data; handle with care and restrict access.

Methods

Docker volume: Mount config/app/ so config persists across container restarts. Copy config.json and users.json from source to target host, then start the new container with the same volume paths.

TrueNAS Blue-Green: Locate Blue deployment volumes (e.g. under ix-applications/releases/), backup config/app/config.json and config/app/users.json, copy to Green deployment volume, set permissions (e.g. chmod 644), then restart the Green app.

Cloud (Azure/AWS/GCP): Use the platform’s config/secret store or mounted volumes; copy the same two files from source to target and restart the app.

Verification

  • Log in with an existing user.
  • Check AI, messaging, and SSO settings.
  • Confirm report URLs and integrations.

User Consolidation

Consolidate users between Blue and Green so the same credentials work on both.

Prerequisites

  • Backend route fix deployed: /api/users/export and /api/users/import must be defined before /api/users/:userId (see commit that fixed route ordering).
  • Both instances running and reachable.

Method 1: Docker (immediate)

Script: scripts/consolidate-users.sh --docker

Run on the host where both containers run:

cd /path/to/OSCAL_Reports
sudo ./scripts/consolidate-users.sh --docker

The script reads users.json from both containers via docker exec, merges users (deduplicates), writes back, and restarts containers. Backups are created under ~/oscal-user-consolidation-TIMESTAMP/. Use --docker when the API-based flow is not available (e.g. older backend without export/import endpoints).

Method 2: API-based (after backend fix)

Script: scripts/consolidate-users.sh

# Interactive (prompts for password)
./scripts/consolidate-users.sh

# Automatic bi-directional
BLUE_PASSWORD='...' GREEN_PASSWORD='...' ./scripts/consolidate-users.sh --auto

# Custom URLs
./scripts/consolidate-users.sh --auto \
  --blue-url http://blue.example.com \
  --green-url http://green.example.com \
  --blue-password '...' --green-password '...'

# One-way: Blue → Green
./scripts/consolidate-users.sh --blue-to-green

# One-way: Green → Blue
./scripts/consolidate-users.sh --green-to-blue

# Docker mode (direct docker exec, run ON server with containers)
./scripts/consolidate-users.sh --docker

Sync script: Use scripts/sync-consolidation-script.sh to copy consolidate-users.sh to Blue/Green script directories if they are separate clones.

Verification

  • Call GET /api/users/export (with auth) on both Blue and Green; confirm user lists match expectations.
  • Log in on both instances with the same user.

Recovery After Accidental Rollback

If Green (or Blue) was rolled back and config and users were wiped, restore /opt/oscal/data from the golden config/default/ S3 snapshot. On the affected instance (via ./scripts/ssh-ec2.sh green|blue):

sudo bash /opt/oscal/scripts/debug/restore-config-from-s3-default.sh
sudo systemctl restart oscal-reporter.service

Then verify /health/ready and SSO login. See Golden config.default and the recovery runbook in DEPLOYMENT_AND_OPERATIONS.md for publishing a fresh snapshot and rebuilding config from local backups (repair-ec2-config-from-backups.mjs).

Verification

Security

  • Backup before any migration or consolidation.
  • Restrict access to config.json and users.json (they contain secrets and hashes).
  • Use secure copy (SCP/rsync over SSH) when moving files between hosts.
  • Rotate or change credentials after migration if policy requires.

Related Documentation

Consolidates former KEEPING_LOCAL_USERS.md (local dev) and deployment migration/consolidation. Last updated: February 2026