Skip to content

About

Minimal headless download-queue daemon + CLI inspired by JDownloader, designed for Docker and terminal use.

Resources

Stars

0 stars

Watchers

1 watching

Forks

Repository files navigation

DLQ (Download Queue)

CI Go License: MIT

Minimal headless download-queue daemon + CLI inspired by JDownloader, designed for Docker and terminal use.

Table of Contents

Features

  • Persistent SQLite-backed job queue with retries, pause/resume, and soft delete
  • Aria2-powered downloads with progress, speed, and ETA reporting
  • Optional automatic archive decrypt/extract after download completion
  • Optional Telegram notifications for finished and failed jobs
  • Pluggable URL resolvers (Webshare anonymous mode, HTTP/HTTPS passthrough)
  • CLI for scripting and automation (dlq add, dlq status --watch, ...)
  • Optional SvelteKit web UI with batch add, folder browser, and live dashboard
  • Docker-first: runs as two containers (API + UI), supports PUID/PGID
  • Unraid-friendly with deploy script and DATA_* volume presets

Architecture

┌─────────────────────────────────────────────────────────────┐
│  Docker host                                                │
│                                                             │
│  ┌───────────────────────────────────────────────────────┐  │
│  │  dlq container                                        │  │
│  │                                                       │  │
│  │   ┌─────────┐    ┌──────────────────────────────┐     │  │
│  │   │  dlq    │───▶│  dlqd  (HTTP API :8099)      │     │  │
│  │   │  (CLI)  │    │                              │     │  │
│  │   └─────────┘    │  ┌──────────┐  ┌──────────┐  │     │  │
│  │                  │  │ resolver │  │  queue   │  │     │  │
│  │                  │  │ webshare │  │ service  │  │     │  │
│  │                  │  │ http(s)  │  └────┬─────┘  │     │  │
│  │                  │  │ mega     │       │        │     │  │
│  │                  │  └──────────┘       ▼        │     │  │
│  │                  │             ┌──────────────┐ │     │  │
│  │                  │             │  SQLite DB   │ │     │  │
│  │                  │             │ /state/dlq.db│ │     │  │
│  │                  │             └──────────────┘ │     │  │
│  │                  │                    │         │     │  │
│  │                  │                    ▼         │     │  │
│  │                  │             ┌──────────────┐ │     │  │
│  │                  │             │  downloader  │ │     │  │
│  │                  │             │  (aria2 RPC) │ │     │  │
│  │                  │             └──────┬───────┘ │     │  │
│  │                  └────────────────────┼─────────┘     │  │
│  │                                       ▼               │  │
│  │                                ┌─────────────┐        │  │
│  │                                │   aria2c    │        │  │
│  │                                │  (JSON-RPC) │        │  │
│  │                                └──────┬──────┘        │  │
│  │                                       ▼               │  │
│  │                                   /data/*             │  │
│  └───────────────────────────────────────────────────────┘  │
│                                                             │
│  ┌───────────────────────────────────┐                      │
│  │  dlq-webui container (optional)   │                      │
│  │                                   │                      │
│  │   SvelteKit app :8098 ───────────────▶ dlqd API :8099    │
│  │   (SSR, proxies API calls)        │                      │
│  └───────────────────────────────────┘                      │
│                                                             │
│  Browser ───▶ :8098 (UI)                                    │
│  Terminal ──▶ docker exec dlq dlq ...                       │
└─────────────────────────────────────────────────────────────┘

Quick start (Docker)

docker build -t dlq:local .

docker run -d --name dlq \
  -p 8099:8099 \
  -v /path/to/downloads:/data \
  -v /path/to/state:/state \
  dlq:local

Add a job:

docker exec -it dlq dlq add <url> --out /data/downloads

Check status:

docker exec -it dlq dlq status

If you change the port, set DLQ_HTTP_PORT to match.

CLI

CLI status output

  • dlq add <url> [<url2> ...] --out /data/downloads [--name optional] [--site mega|webshare|http|https] [--archive-password batch-pass]
  • dlq add --file urls.txt --out /data/downloads
  • dlq add --stdin --out /data/downloads
  • dlq status (summary + table)
  • dlq status --watch [--interval 1] [--status queued|resolving|downloading|paused|decrypting|completed|failed|decrypt_failed|deleted]
  • dlq files (shows all jobs in DB, including soft-deleted)
  • dlq logs <job_id> [--tail 50]
  • dlq retry <job_id>
  • dlq pause <job_id>
  • dlq resume <job_id>
  • dlq remove <job_id> (soft delete)
  • dlq clear (soft delete completed jobs)
  • dlq purge (hard delete all jobs and events; IDs are not reused)
  • dlq settings (show current settings)
  • dlq settings --concurrency <1-10> --auto-decrypt <true|false> (update settings)
  • dlq help

UI (SvelteKit)

Web UI – dashboard Web UI – add job

The UI is optional and lives under ui/. It proxies the DLQ HTTP API server-side. The Unraid deploy script runs the UI as a separate container (dlq-webui) on DLQ_WEBUI_PORT (default 8098).

cd ui
npm install
DLQ_API=http://127.0.0.1:8099 npm run dev

Presets for out_dir are served from GET /meta and derived from DATA_* volume mappings (container paths).

GET /healthz (UI: GET /api/healthz) reads the jobs table and returns 503 when the database fails, so probe it rather than /meta.

How it works

  • dlqd daemon stores jobs in SQLite, resolves URLs, and starts downloads in aria2 via JSON-RPC.
  • Resolvers default to anonymous mode and surface blocking reasons as error codes: login_required, quota_exceeded, captcha_needed, temporarily_unavailable.
  • --site forces a resolver; unknown values return unknown_site.
  • Queue is persistent across restarts (/state/dlq.db).

Telegram notifications

Configured in the UI settings dialog and stored in settings.json (written with mode 0600). The bot token is never returned by the API; leave the field empty to keep the stored one. Disabling notifications keeps the token and chat ID.

  • Events, each toggleable: completed (after extraction), failed (retries exhausted), retrying (an attempt failed, another follows), extract failed.
  • Events within 30 seconds of the first one are sent as one message, one line per job. A multipart archive is reported once, after all parts complete; part failures in the same window collapse into one line.
  • Send test sends the completed template with sample values using the form's current token and chat ID.
  • Delivery is fire-and-forget; send errors go to the daemon log with the token masked.
  • {time} uses the container time zone (TZ).

Templates are plain text. Unknown placeholders are left as-is.

Placeholder Value
{name} job name, file name, or archive name for multipart sets
{filename} resolved file name
{size}, {size_bytes} human-readable / raw size (summed for multipart)
{time}, {duration}, {speed} finish time, start-to-finish duration, average speed
{site}, {url}, {dir} resolver site, source URL, output directory
{id}, {parts} job ID(s), number of archive parts
{status}, {event} job status, event label
{error}, {error_code}, {attempts}, {max_attempts} failure details
{series}, {episode}, {episode_title} series name, S01E02, episode name (series jobs only)

Defaults: completed ✅ {name} ({size}) finished in {duration}, failure ❌ {name}: {event} - {error}.

Environment variables

DLQ

Variable Default Description
DLQ_STATE_DIR /state Directory for state files
DLQ_DB /state/dlq.db Path to the SQLite database
DLQ_HTTP_PORT 8099 API server port
DLQ_HTTP_HOST 0.0.0.0 API server bind address
DLQ_HTTP_ADDR — Explicit host:port override (takes precedence over host/port)
DLQ_API — Client base URL for CLI/UI (e.g. http://127.0.0.1:8099)
DLQ_WEBUI_PORT 8098 Web UI container port
DLQ_WEBSHARE_WST — Webshare session token used by series search and downloads
PUID / PGID — Run dlqd + aria2 as this user/group
UMASK — (002 in deploy-unraid.sh) File mode mask for downloads and created directories; extracted archive entries get the group/other write bits it allows
DATA_* — Volume mappings (e.g. DATA_TVSHOWS=/mnt/user/tvshows:/data/tvshows)

Aria2

Variable Default Description
ARIA2_RPC http://127.0.0.1:6800/jsonrpc Aria2 JSON-RPC endpoint
ARIA2_RPC_LISTEN_PORT parsed from ARIA2_RPC, fallback 6800 Aria2 listen port
ARIA2_SECRET — RPC secret token (recommended)
ARIA2_DISABLE — Set to 1 to disable the built-in aria2c process
ARIA2_DIR first DATA_* path, fallback /data Default download directory for aria2
ARIA2_EXTRA_OPTS — Extra aria2c command-line flags
ARIA2_MAX_CONNECTION_PER_SERVER 4 Max connections per server
ARIA2_SUMMARY_INTERVAL 0 Summary log interval in seconds (0 = disabled)
ARIA2_CONSOLE_LOG_LEVEL warn Aria2 console log level
ARIA2_SHOW_CONSOLE_READOUT false Show aria2 console readout

Note: concurrency, max_attempts, auto_decrypt, and telegram are stored in settings.json under DLQ_STATE_DIR and can be updated via dlq settings or the UI (telegram via the UI only). The file is created with defaults on first start.

UI out_dir presets are derived from DATA_* env values (container paths); make sure they are passed into the container. All job out_dir values must live under one of the DATA_* container paths.

Security

DLQ is designed for trusted networks (home LAN, Docker internal networking). The HTTP API has no authentication.

  • Do not expose DLQ ports directly to the public internet. Use a reverse proxy with authentication (e.g., Caddy, Traefik, nginx) if remote access is needed.
  • Expose only the web UI port through that proxy; the UI forwards every API call. Browser job notifications need https (or localhost), so they work through the proxy but not over plain http on a LAN IP.
  • Set ARIA2_SECRET even in Docker to prevent unauthorized RPC access to aria2.
  • All out_dir values are validated against DATA_* container paths to prevent path traversal.
  • Supply the Webshare session token through DLQ_WEBSHARE_WST (preferred), WEBSHARE_WST, or WS_WST. It is sent only in Webshare request bodies/headers and is never persisted in jobs or logged.

Notes

  • Webshare resolver uses the public API in anonymous mode when possible and forces single-connection downloads for reliability. Series search generally requires a valid WST supplied through the environment.
  • MEGA resolver supports public file links (mega.nz/file/...) by resolving temporary download URLs and decrypting MEGA file payloads after download.
  • Auto decrypt/extract runs after successful download for archive extensions (.zip, .rar, .7z, .tar*, .gz, .bz2, .xz) when auto_decrypt=true in settings.
  • Pass --archive-password in dlq add (or archive_password in API/UI) for password-protected archives in that add batch.
  • DLQ extracts archives with 7z. The Docker image includes Debian's 7zip-rar plugin from non-free so RAR/RAR5 archives are handled by the same extractor.
  • Tar-compressed archives (.tar.gz, .tgz, .tar.bz2, .tbz2, .tar.xz, .txz) are unpacked through a two-pass 7z stream so the inner tar contents are extracted directly.
  • One add batch uses one password for all links; for different passwords, add links in separate batches.
  • If decrypt/extract fails (missing/wrong password, tool error), the job moves to decrypt_failed and the failure is logged to job events.
  • If aria2 restarts, dlq resume <id> will re-queue the job and re-resolve the URL.
  • If you set PUID/PGID, ensure /data and /state are writable by that user on the host.
  • If you see attempt to write a readonly database, fix permissions on the host (e.g., chown -R 99:100 /path/to/state).

Docker Compose

# Copy the example compose file and edit paths/ports
cp docker-compose.example.yml docker-compose.yml

# Start the service
docker-compose up -d

Note: keep the API port consistent across DLQ_HTTP_PORT, the ports mapping, and the web UI DLQ_API value.

Deploy to Unraid

# Copy .env.example to .env and edit it
cp .env.example .env

# Build, transfer, and deploy containers
scripts/deploy-unraid.sh all

# Deploy only the CLI/API container
scripts/deploy-unraid.sh cli

# Deploy only the web UI container
scripts/deploy-unraid.sh webui

The script reads its configuration from .env. It can also take the same variables from the environment, so the config can live elsewhere, e.g. an encrypted file: sops exec-env deploy.sops.env 'scripts/deploy-unraid.sh all'. A local .env is sourced on top and wins.

Variables:

  • REMOTE_HOST - SSH alias for your Unraid server
  • DATA_* - Volume mappings (e.g., DATA_TVSHOWS=/mnt/user/tvshows:/data/tvshows)
  • STATE_MOUNT - State/config volume mapping. Use a pool path such as /mnt/cache/appdata/dlq, not /mnt/user/...: the SQLite database runs in WAL mode, and Unraid's shfs FUSE layer does not share its locks or -shm memory with anything opening the file through /mnt/cache (e.g. a backup job), which can delete the live WAL and break the database
  • DLQ_HTTP_PORT - Port for the DLQ API container
  • DLQ_WEBUI_PORT - Port for the web UI container
  • PUID/PGID/UMASK/TZ - Runtime user, file mode mask (default 002) and timezone settings

The deploy script automatically discovers all DATA_* variables and passes them through so presets can be derived in the app.

Version

dlq --version

Testing

Use this single flow for local verification:

# 1) Ensure Docker is running
docker info

# 2) Prepare local dev env file (needed by script integration test)
cp -n .env.example .env.dev

# 3) Run all Go tests (unit + integration), verbose and without cache
go test -v -count=1 ./...

# 4) Verify web UI build
cd ui
npm install
npm run build

Development

cp .env.example .env.dev
scripts/run-dev.sh

License

MIT. See LICENSE. Third-party notices in THIRD_PARTY_NOTICES.md.

About

Minimal headless download-queue daemon + CLI inspired by JDownloader, designed for Docker and terminal use.

Resources

Stars

0 stars

Watchers

1 watching

Forks

Releases

Packages

Contributors

Languages