Worker consolidation release. batch + cpu supervisor groups (verified live, -737MB / 2.57GB->1.83GB). Prefork (process-model change) deferred to 1.2.0.
2.7 KiB
2.7 KiB
Changelog
All notable changes to DECNET are documented here.
The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.
1.1.0 - 2026-06-18
Worker consolidation: cut the long-running worker fleet's resident memory by hosting co-resident workers in shared supervisor processes instead of one OS process per worker. Behaviour-preserving — workers run the same code; only where they are hosted changes, and any worker remains extractable back to its own unit.
Added
decnet supervise <group>— hosts a co-resident worker group in one process, paying the Python import floor and the DB connection pool once instead of once per worker. Groups:batchandcpu.decnet.supervisor— in-process supervision primitive: each worker runs in its own restart loop with exponential backoff (in-processRestart=on-failure), run concurrently so one worker crashing never cancels its siblings. Deliberately notasyncio.TaskGroup, whose all-or-nothing cancellation would break worker isolation.decnet.offload— shared-pool CPU-kernel offload. Thecpugroup runs its two O(n²) connected-components kernels (cluster_observations,cluster_identities) in one sharedProcessPoolExecutor(forkserver) so they run in parallel instead of serialising under the GIL. Inline when no pool is installed, so standalone workers and tests are unchanged.- systemd units
decnet-supervise-batch.serviceanddecnet-supervise-cpu.service(auto-rendered bydecnet init); eachConflicts=the individual units it replaces, preventing accidental double-run.
Changed
decnet.topologyno longer eagerly imports the topology generator (and the SQLModel ORM behind it) at package import.generateis now a lazy PEP 562 re-export; the public API is unchanged.
Performance
- batch group (
reconcile+enrich+orchestrate+mutate): 509 MB across 4 processes → 129 MB in one. −380 MB (75%), verified live. - cpu group (
clusterer+campaign-clusterer+attribution+reuse-correlate): 502 MB → ~146 MB (incl. forkserver). −357 MB (71%), verified live. - Fleet total: 2.57 GB → ~1.83 GB (−737 MB).
Notes
webhook(external-HTTP egress; needs hard timeouts) andcanary(manages its own repo) intentionally remain standalone for now.bus,api/web,profiler, andttpremain separate by design (broker / multiprocess servers / heavy resident state + sustained CPU).
1.0.0 - 2026
Initial 1.0 release. See tag v1.0.0.