feat: KNX device programming, knxproj support, and GUI improvements

.gitignore

@@ -34,6 +34,7 @@ htmlcov/
 .DS_Store
 NodeEditor.json
 XKNX_Toolkit.ini
+.references
 
 scan_repeats.py
 

README.md

@@ -1,7 +1,7 @@
 # XKNX Toolkit
 
 > [!WARNING]
-> **Alpha, experimental software.** XKNX Toolkit is not intended for end users and comes with no stability or safety guarantees — expect breaking changes, rough edges, and bugs. It's mostly useful for developers experimenting with the [xknx](https://github.com/XKNX/xknx) library. It cannot program devices, and no support is offered to end users. Contributions are welcome. Large parts of this project were built using LLMs.
+> **Alpha, experimental software.** XKNX Toolkit is not intended for end users and comes with no stability or safety guarantees — expect breaking changes, rough edges, and bugs. It's mostly useful for developers experimenting with the [xknx](https://github.com/XKNX/xknx) library. It can program real KNX devices (see below), but that path is experimental — always review the change set with a preflight first and keep a way to restore the device. No support is offered to end users. Contributions are welcome. Large parts of this project were built using LLMs.
 
 XKNX Toolkit is a desktop application and set of Python libraries for working with [KNX](https://www.knx.org/) home and building automation installations — browsing product catalogs, editing installation projects, and talking to real or simulated KNX devices.
 
@@ -15,16 +15,20 @@ Devices and group addresses are laid out as nodes on a canvas. Communication obj
 
 ### Project management
 
-Devices are organized by area/line/segment, matching standard KNX topology. Each device's parameters, com object flags, load procedures, and raw memory layout can be inspected and edited directly.
+Devices are organized by area/line/segment, matching standard KNX topology. Each device's parameters, com object flags, load procedures, and raw memory layout can be inspected and edited directly. Projects import from and export to ETS `.knxproj` archives: the export bundles the referenced manufacturer data (verbatim from the original `.knxprod` archives) and a merged `knx_master.xml`, so applications resolve from the archive alone. The project can be browsed like ETS through the device topology, the group address tree with assignments, and the building/space tree with functions.
 
 ### Product catalog
 
-Import `.knxprod` archives to build up a searchable catalog of manufacturers, hardware, and application programs, independent of any single project. Catalog entries can be dragged into a project as new devices.
+Import `.knxprod` archives to build up a searchable catalog of manufacturers, hardware, and application programs, independent of any single project. Catalog entries can be dragged into a project as new devices. The GUI also connects to the **KNX online catalog** (the same anonymous service ETS uses — no account, no license check on the service side) and can browse and refresh the manufacturer list, cached locally so it stays available offline; downloading the actual products from the online catalog is not implemented yet.
 
 ### Real KNX connections
 
 Connect to a real KNX interface over tunneling (TCP/UDP) or routing (multicast), with automatic gateway discovery or manual IP entry.
 
+### Programming real devices
+
+The toolkit can commission a physical KNX device end-to-end — the job ETS does on "Download" — without ETS. It assembles the device's memory image from the product database and your parameter and group-address choices, then executes the application's Load Procedure over a live bus: driving each loadable part's Load State Machine, writing memory and properties, laying down the group communication tables, and restarting the device. It also programs a virgin device's individual address. A read-only **preflight** reads back every location a write would touch and shows the exact diff before anything is written. **Test Before Programming** runs that preflight and reports per memory segment and property whether the generated image matches what is already programmed on the device (with the current and planned bytes exportable as a report), guarding against writing a wrongly generated image. Programming requires a live connection; every connection-dependent action (test, program, sending frames) is refused with a visible "no KNX connection" notice when the bus is not linked. This is a vendor-independent implementation derived from the KNX Standard v3.0.0, verified on real hardware; it lives in the `xknx-download` package and can be used without the GUI.
+
 ### Virtual devices and proxy — test without hardware
 
 No KNX interface on hand? XKNX Toolkit can stand in for one:
@@ -45,9 +49,13 @@ Only running from source is supported for now — this is developer-focused soft
 
 ```bash
 uv sync
-uv run python -m knx_gui.main
+uv run --package knx-gui python -m knx_gui.main
 ```
 
+`knx-gui` is a workspace member that the root project does not depend on, so a plain `uv sync` does
+not install it (or its `imgui-bundle` dependency). The `--package knx-gui` flag runs it in — and
+installs it — from the workspace.
+
 ## Packages
 
 The application is built on a set of standalone, typed Python libraries (the `xknxmono` namespace) that can also be used independently of the GUI:
@@ -59,9 +67,10 @@ The application is built on a set of standalone, typed Python libraries (the `xk
 | `xknx-catalog` | `xknxmono.catalog` | Product catalog built from imported `.knxprod` archives |
 | `xknx-project` | `xknxmono.project` | Project state management for KNX installations |
 | `xknx-keyring` | `xknxmono.keyring` | Parses and serializes KNX keyring XML (KNX IP Secure keys) |
+| `xknx-download` | `xknxmono.download` | Programs applications and individual addresses into real KNX devices |
 
 ```bash
-pip install xknx-models xknx-product xknx-catalog xknx-project xknx-keyring
+pip install xknx-models xknx-product xknx-catalog xknx-project xknx-keyring xknx-download
 ```
 
 ## Development

apps/knx-gui/.gitignore

@@ -3,5 +3,6 @@
 *.xknx
 knxprod/
 NodeEditor.json
+online_catalog_manufacturers.json
 .venv/
 internal/

apps/knx-gui/pyproject.toml

@@ -10,6 +10,7 @@ dependencies = [
     "structlog>=24.0.0",
     "xknx>=3.18.0",
     "xknx-catalog",
+    "xknx-download",
     "xknx-models",
     "xknx-product",
     "xknx-project",
@@ -31,6 +32,7 @@ packages = ["src/knx_gui"]
 
 [tool.uv.sources]
 xknx-catalog = { workspace = true }
+xknx-download = { workspace = true }
 xknx-models = { workspace = true }
 xknx-product = { workspace = true }
 xknx-project = { workspace = true }

apps/knx-gui/src/knx_gui/concurrency.py

@@ -0,0 +1,38 @@
+"""A tiny guard for service reads during a background import.
+
+A long ``.knxproj`` import runs on a worker thread that mutates the catalog and project databases.
+The GUI panels read those same services every frame on the UI thread. To keep the UI responsive
+without racing the writer, the importer holds a shared *re-entrant* lock for the whole import, and
+every per-frame service read is wrapped with :func:`io_guarded`: it acquires the lock without
+blocking and, if the importer holds it, returns an empty placeholder instead of touching the
+database. Because the lock is re-entrant, the importing thread itself (which already holds it) still
+reads real data while building the project view.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Callable
+from functools import wraps
+from typing import Any
+
+
+def io_guarded[T](
+    default_factory: Callable[[], T],
+) -> Callable[[Callable[..., T]], Callable[..., T]]:
+    """Return the wrapped read's result, or ``default_factory()`` if a background import is running.
+
+    The instance must expose a re-entrant ``self._io_lock`` (``threading.RLock``)."""
+
+    def decorator(fn: Callable[..., T]) -> Callable[..., T]:
+        @wraps(fn)
+        def wrapper(self: Any, *args: Any, **kwargs: Any) -> T:
+            if not self._io_lock.acquire(blocking=False):
+                return default_factory()
+            try:
+                return fn(self, *args, **kwargs)
+            finally:
+                self._io_lock.release()
+
+        return wrapper
+
+    return decorator

apps/knx-gui/src/knx_gui/device.py

@@ -306,3 +306,8 @@ def find_com_object(self, co_id: str) -> ComObject | None:
             if co.id == co_id:
                 return co
         return None
+
+    @property
+    def dynamic_ui(self) -> DynamicUI | None:
+        """The live evaluator holding this device's current parameter state."""
+        return self._dynamic_ui

apps/knx-gui/src/knx_gui/dpt.py

@@ -43,6 +43,25 @@ def lookup_or_make_dpt(code: str | None) -> DPT:
     return DPT(major, minor, f"DPT {major}.{minor:03d}", code)
 
 
+@lru_cache(maxsize=512)
+def transcoder_for(dpt_str: str | None) -> type[DPTBase] | None:
+    """Resolve an xknx transcoder from a project DPT string (``"DPST-1-1"`` / ``"DPT-5"``).
+
+    The project stores ETS-token DPTs, but xknx's ``parse_transcoder`` wants dotted ``"1.1"`` (or
+    ``"DPT-1"`` for main-only). We convert and fall back to the main-only transcoder."""
+    if not dpt_str:
+        return None
+    parts = dpt_str.split("-")
+    if parts[0] == "DPST" and len(parts) >= 3:
+        transcoder = DPTBase.parse_transcoder(f"{parts[1]}.{parts[2]}")
+        if transcoder is not None:
+            return transcoder
+        return DPTBase.parse_transcoder(f"DPT-{parts[1]}")
+    if parts[0] == "DPT" and len(parts) >= 2:
+        return DPTBase.parse_transcoder(f"DPT-{parts[1]}")
+    return DPTBase.parse_transcoder(dpt_str)
+
+
 DPT_MAJOR_COLORS: dict[int, imgui.ImVec4] = {
     1: imgui.ImVec4(0.9, 0.3, 0.3, 1.0),
     2: imgui.ImVec4(0.9, 0.5, 0.5, 1.0),

apps/knx-gui/src/knx_gui/locales/de/LC_MESSAGES/knx_gui.po

@@ -66,3 +66,6 @@ msgstr "XKNX Projekt speichern"
 
 msgid "XKNX project (*.xknx)"
 msgstr "XKNX Projekt (*.xknx)"
+
+msgid "No KNX connection"
+msgstr "Keine KNX-Verbindung"