From 87eb986467703c8faeb4e127c32789bc11760322 Mon Sep 17 00:00:00 2001 From: anti Date: Wed, 17 Jun 2026 16:36:04 -0400 Subject: [PATCH] =?UTF-8?q?docs(1.1):=20correct=20strategy=20=E2=80=94=20c?= =?UTF-8?q?onsolidation=20is=20primary=20lever=20(C3)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Measurement after C2: all 25 CLI modules transitively pull the SQLModel ORM, and the table chain is a hub (one table loads the whole registry). Most workers genuinely use the DB, so lazy imports only help the 2-3 DB-less workers. Consolidation (pay the 86MB floor once for the herd) is the reliable ~600MB win — promoted from fallback to primary. --- development/RELEASE-1.1.md | 38 ++++++++++++++++++++++++++++---------- 1 file changed, 28 insertions(+), 10 deletions(-) diff --git a/development/RELEASE-1.1.md b/development/RELEASE-1.1.md index 53a8539c..d0103ebe 100644 --- a/development/RELEASE-1.1.md +++ b/development/RELEASE-1.1.md @@ -1,10 +1,23 @@ # DECNET 1.1 — RAM / Process-Footprint Release -Predecessor: `v1.0.0`. Theme: cut the fleet resident set from **2.57 GB → target ~1.3 GB** -with near-zero risk, then optionally further via worker consolidation. +Predecessor: `v1.0.0`. Theme: cut the fleet resident set from **2.57 GB → target ~1.0 GB** +by paying the import floor once per process-group instead of once per worker. Analysis & measurements: see [improvements.md](improvements.md). +## ⚠️ Strategy correction (after C2 measurement) + +Initial assumption: lazy imports would drop idle workers below the 86 MB floor → ~1.3 GB. +**Measurement disproved this.** All 25 CLI command modules transitively pull the SQLModel +ORM, and the 38 MB table chain is a *hub*: importing any one table loads the whole registry. +Most workers legitimately touch the DB, so they can't shed the ORM. Lazy registration only +helps the 2-3 genuinely **DB-less** workers (`forwarder`, `listener`, maybe `bus`). + +**Therefore consolidation is the PRIMARY lever, not the fallback.** Paying the 86 MB floor +once for the idle herd (instead of ~8×) is the reliable ~600 MB win. C2/C3 stay as cheap +hygiene and to let the DB-less workers actually skip the ORM, but they are no longer the +headline. + ## Why 18 long-running workers, ~2.57 GB resident. ~1.5 GB of that is the **same 86 MB import @@ -27,12 +40,15 @@ the ORM into every worker. No production code imports `generate` from the packag ## Commit plan (incremental, one concern each) - [x] **C1 — docs.** `improvements.md` (analysis) + this release plan. -- [ ] **C2 — lazy topology re-export.** PEP 562 `__getattr__` in `topology/__init__.py` +- [x] **C2 — lazy topology re-export.** PEP 562 `__getattr__` in `topology/__init__.py` so `generate` loads on access, not on package import. Public API unchanged. - Test: `import decnet.cli` must NOT pull `decnet.web.db.models`. Re-measure floor. -- [ ] **C3 — sweep remaining eager model pulls.** After C2, re-trace `import decnet.cli`; - defer any other command module that drags the ORM in for registration only. - Test: assert idle-worker floor stays under target. + Guard test added. *Finding: correct, but ORM pull is pervasive (see correction above) + — C2 alone does not move the floor.* +- [ ] **C3 — DB-less worker lazy registration (reduced scope).** Confirm `forwarder` / + `listener` (and audit `bus`) never touch models at runtime; make ONLY their command + modules import-clean so those specific processes skip the 38 MB ORM. Skip the + DB-touching majority — they can't shed it. Test: those modules don't pull + `decnet.web.db.models`. - [ ] **C4 — extract idle-herd coroutines.** Hoist the inline `_run()` closures (`webhook`, `canary`, `listener`, `forwarder`, `mutate`, `enrich`) into reusable `async def run(bus, cfg)` in their packages, so they're hostable by a supervisor. @@ -59,7 +75,9 @@ the ORM into every worker. No production code imports `generate` from the packag restart + `MemoryMax`. **Medium.** Verify on the live fleet before adopting; keep the individual units as the fallback. Do C2–C4 first; C5 only if RAM still bites. -## Projected +## Projected (revised) -- C2–C3 (import floor): 2.57 GB → **~1.3 GB**. Nearly free. -- C4–C6 (consolidation): → **~0.9 GB**. Costs process isolation. +- C2–C3 (import floor): only the 2-3 DB-less workers shed the ORM. **~100 MB.** Cheap hygiene. +- **C4–C5 (consolidation): the main event.** Idle herd ~8 × 86 MB → 1 × 86 MB ≈ **−600 MB**. +- C6 (scapy merge): 3 × 76 MB → 1 × 76 MB ≈ **−150 MB**. +- Total: 2.57 GB → **~1.0 GB**. The bulk comes from consolidation, which costs isolation.