Skip to content

CONTRIBUTING.md

Latest commit

 

History

History
114 lines (96 loc) · 7.48 KB

CONTRIBUTING.md

File metadata and controls

114 lines (96 loc) · 7.48 KB
Raw

Contributing

Homelab network + hardware topology generator: scanners and collectors gather a machine's or a network's real topology, a pipeline normalizes + enriches it, and the dashboard draws it live. Reporting agents push each host's map + telemetry to the dashboard server.

Domain vocabulary — Topology, Collector, Card, Store, Widget — is defined in CONTEXT.md. Use those names. This file is operational: how to work in the repo.

Pipeline

collectors/*.py + scanners/*.py gather raw dicts → core/normalize.py (dedup by MAC, reconcile by IP/hostname, VLAN by subnet) → core/enrich.py (vendor via core/oui.csv, kind, aging) → renderers/card.py view-model → renderers/html/ (the dashboard).

Two dashboard subsystems live beside it, both server-side only (not in the agent bundle): Glances (this host's live metrics — core/glances.py; remote hosts push via the agent) and the Widget Store (widgets/ + renderers/html/widget_store.py, /api/widget-* routes — see CONTEXT.md → Widget, docs/prd/widget-store.md, docs/widgets-adding.md).

Commands

Install: pip install -r requirements.txt

  • Dashboard server: python renderers/html/topo_server.py [--port 8770]
  • Map this machine: python scanners/make_pc_topo.py (Windows) · python scanners/make_linux_topo.py (Linux)
  • Map the network: python scanners/make_network_topo.py
  • Report to a dashboard: agent/report.sh http://<dashboard-ip>:8770
  • Run it as a container: docker compose up -d (see docker-compose.yml)

Tests

Every check is offline and runs in seconds. This list is exactly what CI runs (.github/workflows/ci.yml) — keep the two in sync.

python -m compileall -q .
python tests/test_pipeline.py                    # collectors -> normalize -> enrich, on fixtures
python tests/test_cards.py                       # card contract + network-cards adapter
python tests/test_routes.py                      # boots topo_server on a free port, hits its routes
python -m collectors.unifi                       # unifi transforms + the single GET/POST request path
python -m core.identity                          # node id rule + link endpoint resolution
python -m collectors.arpscan                     # arp-scan + nmap output parsers
python -m collectors.unifi_snmp                  # snmpwalk parser + FDB OID -> MAC decode
python -m collectors.docker                      # docker ps line parser
python -m collectors.tailscale                   # peer shaping + overlay mesh
python -m collectors.dns                         # hosts file + pi-hole custom dns rows
python -m collectors.pingsweep                   # arp -a table parser (win + posix)
python -m collectors.opnsense                    # vlan zones + lease/arp merge
python renderers/html/_guard.py                  # shared path-injection barrier
python renderers/html/pushcache.py               # push-freshness cache (live/stale/miss)
python renderers/html/store.py                   # store slug policy + save/load
python renderers/html/widget_store.py            # widget store CRUD round-trip
python renderers/html/icons.py                   # service-icon slug candidates
python renderers/html/report.py                  # diagnostic HTML report escaping
python renderers/html/pve_tiles.py               # proxmox guests -> dashboard tiles
python renderers/html/agent_bundle.py            # agent tar.gz/zip builder
python collectors/transport.py                   # shared collector HTTP (ssl ctx + get_json)
python -m widgets.net                            # widget SSRF guard
python -m widgets.engine                         # widget engine (auth/mapping)
python -m widgets.fetchers                       # widget stat parsers + collector-backed SSRF guard
python -m widgets.policy                         # widget config/secret policy (mask/whitelist/inherit)
python -m widgets.registry                       # catalog integrity + full_catalog merge
python scanners/make_linux_topo.py --selftest
python scanners/make_pc_topo.py --selftest       # perf-counter iface + cap vocabulary
bash tests/test_uninstall_unraid.sh              # uninstaller only removes its own boot 'go' lines

(widgets/* use relative imports — run via python -m widgets.<mod>, like collectors/.) Also in CI: bash -n *.sh tests/*.sh; .ps1 files must be ASCII; dashboard JS via node --check on the extracted <script> block.

Non-trivial logic leaves one runnable check behind — an assert-based __main__ self-check in the module itself, or a small tests/test_*.py. No frameworks, no fixtures beyond tests/fixtures/.

Gotchas that will bite you

These are not style preferences. Each one has broken something real.

  1. Never commit config.yaml — it is gitignored and holds secrets (API keys, tokens). Keep real IPs / users / hostnames out of the tracked config.example.yaml; use placeholders.
  2. scanners/make_linux_topo.py imports nothing from this repo. The dashboard's remote scan pipes it alone over SSH, so it must stay single-file self-contained (stdlib only). Verify: grep -nE '^(from|import) (core|renderers|collectors|scanners|make_)' scanners/make_linux_topo.py returns nothing. (Background: CONTEXT.md → Card.)
  3. Collectors are stdlib-only (urllib, not requests), return [] on any source failure (never raise), and must be registered in scanners/make_network_topo.py. (CONTEXT.md → Collector.)
  4. .ps1 files must be ASCII — Windows PowerShell 5.1 reads BOM-less files as ANSI, so a stray non-ASCII byte corrupts them (CI enforces this). Bind PS named params with a hashtable splat, not an array (an array splat binds positionally).
  5. Dashboard index.html is served from disk per request — HTML/CSS/JS edits need only a browser hard-refresh, no restart. Only Python changes need the server restarted.
  6. The dashboard's SVG builder uses textContent, never innerHTML, for scanned or host-supplied strings — device labels are untrusted. Keep it that way.
  7. Restarting the server depends on how it was started. systemctl restart topo-server only applies on a systemd Linux box. The production dashboard runs on Unraid (Slackware, no systemd), as a container: docker restart topographer. There is no systemctl on an Unraid host — never tell a user to run it there. The older native Unraid install (flash launcher + a boot go hook) is retired: install.sh refuses on Unraid and points at the container, while uninstall.sh still removes it so existing installs can migrate. Agents on Unraid are unaffected — bootstrap.sh still uses the go script for topo-agent.
  8. Reporting clients are NOT git checkouts. A reporting host (e.g. a Pi agent) runs the agent bundle the dashboard serves — bootstrap.sh fetches /agent.tar.gz and extracts it (no .git), so git pull fails there. To ship agent-side code to a client: update + restart the dashboard server first (it builds the bundle from its own repo — AGENT_PATHS in renderers/html/agent_bundle.py), then re-run the bootstrap on the client to re-fetch.

Style

  • Python: stdlib-first, type hints, from __future__ import annotations. requirements.txt is intentionally tiny — don't add a dependency for what a few lines of stdlib can do.
  • Match the terseness and comment density of the file you are editing.
  • Pushes go straight to main; there is no feature-branch / PR flow on this repo.