feat(workers): bus-backed Workers panel (registry, control, installed flag)

Ships the backend half of Config → Workers:

* Worker registry aggregates `system.*.health` + `system.bus.health`
  heartbeats into a last-seen dict; OK / STALE / UNKNOWN tiers drop
  out of a 90s window (3× the 30s heartbeat interval).
* `GET /api/v1/workers` returns the snapshot plus `bus_connected`
  (so the UI can explain "all UNKNOWN" when the bus socket is down)
  and a per-row `installed` flag populated from
  `systemctl list-unit-files decnet-*.service` (cached 30s).
* `POST /api/v1/workers/{name}/stop` publishes a stop intent on
  `system.<name>.control`; workers listen via the shared control
  listener in `bus/publish.py`.
* Heartbeat + control listener wired into collector / profiler /
  sniffer / prober / mutator worker loops. API self-heartbeats too
  so the panel always has one ground-truth row.
* Topic helper `system_control(name)` + tests covering builder
  validation, control listener shutdown path, and the API surface
  (auth gating, bus-connected field, unknown-name 404).

Adds `StartFailure` / `StartAllResponse` models in anticipation of
the upcoming start endpoints (DEBT-034).
This commit is contained in:
2026-04-22 14:10:39 -04:00
parent fcaac648a4
commit 0fbb07c2ec
18 changed files with 863 additions and 10 deletions

View File

@@ -20,7 +20,11 @@ from typing import Any, Callable, Optional
from decnet.bus import topics as _topics
from decnet.bus.factory import get_bus
from decnet.bus.publish import make_thread_safe_publisher
from decnet.bus.publish import (
make_thread_safe_publisher,
run_control_listener_signal,
run_health_heartbeat,
)
from decnet.logging import get_logger
from decnet.telemetry import traced as _traced, get_tracer as _get_tracer, inject_context as _inject_ctx
@@ -416,6 +420,14 @@ async def log_collector_worker(log_file: str) -> None:
_publish_log = _make_system_log_publisher(bus, loop)
# Workers panel health heartbeat + bus-driven stop control. The
# heartbeat beacons on system.collector.health every 30s; the
# control listener translates a bus stop intent into a SIGTERM to
# this process (collector's main loop is a blocking thread pool, so
# self-signalling is cleaner than threading a shutdown event).
heartbeat_task = asyncio.create_task(run_health_heartbeat(bus, "collector"))
control_task = asyncio.create_task(run_control_listener_signal(bus, "collector"))
# Dedicated thread pool so long-running container log streams don't
# saturate the default asyncio executor and starve short-lived
# to_thread() calls elsewhere (e.g. load_state in the web API).
@@ -465,6 +477,10 @@ async def log_collector_worker(log_file: str) -> None:
logger.error("collector error: %s", exc)
finally:
collector_pool.shutdown(wait=False)
for t in (heartbeat_task, control_task):
t.cancel()
with contextlib.suppress(Exception, asyncio.CancelledError):
await t
if bus is not None:
with contextlib.suppress(Exception):
await bus.close()