The batch/cpu supervisor groups + heavy fleet replace 10 per-worker units
(reconciler/enrich/orchestrator/mutator/clusterer/campaign-clusterer/
attribution/reuse-correlator/profiler/ttp). Removed their deploy/*.service.j2
templates and rewired decnet.target to the 3 consolidated units. Dropped
test_orchestrator_unit.py (tested a removed unit). CLI commands (decnet ttp,
mutate, …) stay for manual runs; new units' Conflicts= still name the old
units defensively for hosts mid-migration.
Wires the prefork primitive into a CLI command. 'decnet fleet heavy' imports
the shared base floor once in the master, then forks profiler + ttp as
CoW-sharing child processes (own process/GIL, full isolation, shared ~71MB
floor). DB-only tier => systemd unit carries no extra privilege (prefork's
privilege-union cost is nil for this fleet). Unit Conflicts= the profiler/ttp
units it replaces. Heavy per-worker state (ATT&CK/ML) still loads per-child;
warming it in the master to share is deferred until a live RSS measurement
shows the big object graph CoW-shares rather than refcount-dirties.
Bundle pointer moved from repo root to decnet/data/ (with LICENSE.txt),
gitignored + fetched on demand (51MB, MITRE-licensed). Version pin bumped
19.0->19.1 with the new sha256; license unchanged. All _REPO_BUNDLE test
constants repointed. Fixes test-web failures after the repo-root bundle
was deleted.
CoW measurement on CPython 3.14: forked idle child keeps ~71MB shared,
dirties ~1MB private; working child ~26MB. PEP 683 immortal objects keep
code/module pages clean so gc.freeze() is unnecessary (freeze==nofreeze).
prefork.run_fleet: master imports the base floor once, forks one child
per worker (own process/GIL, CoW-shared floor), reaps + restarts with
backoff, graceful SIGTERM->SIGKILL shutdown. Not yet wired to a command
(that lands when 1.2 picks the target worker set).
confidence_max is a ceiling (min(base, ceiling)), not a multiplier — the
ASVS pass fixed this (BUG-8: min(base, base*ceiling) -> min(base, ceiling)),
but 4 lifter clip tests still encoded the old base*ceiling math (0.45/0.4/
0.35) and were masked by the make test-web bundle error fail-fast. All four
now assert the 0.5 ceiling. Separately, test_topics_matches_documented_set
lacked attacker.fingerprinted, which worker.py legitimately subscribes to
(JARM/HASSH/tcpfp/ipv6_leak -> TTP tagging). Located via turbovec + git pickaxe.
(cherry picked from commit f83b467c35649a06fa36f4b350e6666379cd71cb)
Hosts clusterer/campaign-clusterer/attribution/reuse-correlate in one
process. The two O(n^2) connected-components kernels (cluster_observations,
cluster_identities) offload to ONE shared forkserver pool via decnet.offload
.run_kernel, so they run in parallel instead of serialising under the GIL.
- offload.run_kernel: pool when installed + offload_if holds, else inline.
Standalone workers and all tests run inline => behaviour unchanged
(424 clustering/correlation tests green).
- offload_if gates on input size (>=256) to skip pickle cost on small passes.
- forkserver (not fork): supervisor is multithreaded via bus clients.
- attribution/reuse co-located but not offloaded yet (lighter; same run_kernel
path extends to them if profiling shows contention).
- systemd unit Conflicts= the 4 units it replaces; no docker/raw-socket priv.
Controlled unit swap on mothership: single PID hosts all 4 workers,
shared repo + 30s reconcile pass confirmed working, no crashes. Live
delta beat the floor estimate. Batch group complete.
enrich is a batch-group member; its individual unit must also be mutually
exclusive with the supervisor. Unit auto-renders via init.py glob of
deploy/decnet-*.service.j2 — no installer list change needed.
4 live batch workers = 509MB; consolidated startup floor = 118.5MB
(imports 102.5 + repo pool 15.7 once vs 4x + bus 0.3). Side-effect-free
measurement (no mutation tick). Floor is a lower bound; live adds modest
per-worker working set. Exact live number pending controlled unit swap.
Hosts reconcile/enrich/orchestrate/mutate in one process via the
supervision primitive: one import floor, one shared repo/DB pool instead
of 4. Static group registry (membership is architectural, not a knob);
factories lazy-import only the hosted workers. systemd unit Conflicts=
the individual units it replaces and documents the union-of-privileges
cost. Worker code unchanged — any member is extractable by editing _build_specs.
forwarder/listener are role-split swarm singletons, not co-resident with
the herd — drop from grouping. Real master-resident batch group shares
the repo singleton (one DB pool when consolidated). Stage 1 = batch:
reconcile/enrich/orchestrate/mutate. webhook/canary deferred standalone.
HOW to consolidate: supervision-loop primitive (not TaskGroup, whose
all-or-nothing cancel breaks isolation); group by failure domain +
resource profile keeping per-group cgroup limits; every worker remains
config-extractable. Recommend process-groups now (~18->~9 units),
evaluate prefork+gc.freeze CoW on 3.14 as the higher-ceiling follow-on.
Measurement after C2: all 25 CLI modules transitively pull the SQLModel
ORM, and the table chain is a hub (one table loads the whole registry).
Most workers genuinely use the DB, so lazy imports only help the 2-3
DB-less workers. Consolidation (pay the 86MB floor once for the herd)
is the reliable ~600MB win — promoted from fallback to primary.
topology/__init__ eagerly imported generator -> allocator -> repository ->
the full SQLModel ORM. Defer via PEP 562 __getattr__ so importing the
package doesn't drag the DB layer into DB-less workers. Public API
(from decnet.topology import generate) unchanged. Guard test locks it in.
Fleet resident set ~2.57GB across 18 workers; ~1.5GB is the 86MB import
floor paid 18x. Pinned root cause: topology/__init__ eager re-export of
generate drags the full SQLModel ORM (26 tables, ~38MB) into every worker.
The core CLI scans decnet/pro/cli/ and calls each module's register(app),
registered before the master-only gate so pro commands are mode-filtered like
the rest. Lets the Professional tier add commands and standalone daemon entry
points (decnet pro-<cmd> serve, supervised by a systemd unit). No-op in the
Community build (no decnet.pro). Test asserts the shipped pro group registers
when mounted; skips otherwise.
Move the pro mount decnet/services/pro/ -> decnet/pro/ so the Professional tier
can contribute to more than honeypots. The core wires each surface only when
decnet/pro/ is present (absence stays the entitlement gate):
* services — registry scans decnet/pro/services/ (was decnet/services/pro/)
* API routes — decnet/pro/routes.py exposes ROUTERS, mounted under /api/v1
* web pages — Vite aliases @pro to the pro frontend (community -> empty stub),
App.tsx maps proRoutes into <Route>s, Layout renders a
PROFESSIONAL nav group; both tree-shake out of the community build
Frontend gate mirrors the existing VITE_DECNET_DEVELOPER tree-shake pattern.
Tests: registry + router seams (backend), empty-stub contract (frontend).
Newer pytest raises 'Marks cannot be applied to fixtures' instead of ignoring
it. The async test methods already carry @pytest.mark.anyio, which is what
selects the backend; the fixture must not.
Pro-tier honeypots load from an optional decnet/services/pro/ subpackage that
the registry auto-discovers when present; the Community build omits it, so the
directory's absence IS the entitlement gate (no runtime licence check). Recurse
subclasses so a pro service may extend a community one. Exclude pro from the
community wheel and git-ignore the path (it lives in the private
decnet-professional repo).
Add LICENSING.md documenting the dual-license: AGPL-3.0-or-later core plus a
commercial EULA for the Professional tier.
Campaign clusterer gains a keystroke edge: when two identities'
kd_digraph_simhash centroids are within KD_HAMMING_MAX bits, a graded
weight (1.0 at identical, fading to 0 at the cutoff) feeds the campaign
graph. Supporting tier (0.6) — a typing match plus temporal overlap
reaches threshold, but typing alone never merges (FP guard against
coarse, noisy terminal timing).
Projects the column through IdentityFeatures + from_identity_row.
The identity clusterer folds an identity's per-session
motor.digraph_simhash observations into one 8-byte bitwise-majority
centroid (denoises per-session jitter) and writes it to
AttackerIdentity.kd_digraph_simhash via update_identity_fingerprints —
the orphaned column is now populated. list_identities_for_clustering
projects it so the campaign clusterer can read it.
Extends the repo abstract + DummyRepo stub/coverage.
Per-session 64-bit SimHash of inter-keystroke digraph flight times:
walk single-char input events, accumulate flight time per (c1,c2),
bucket the median, Charikar-SimHash the bucketed pairs. Locality-
sensitive so the same typist is Hamming-close across sessions; pastes
and think-pauses break the chain; silent below the sample-size floor.
New shared decnet/util/simhash.py (simhash64/hamming64/bytes helpers).
Registered as a conditional Tier-A primitive (count 37->38); requires
behave-shell>=0.1.2.
initialize() now delegates to _apply_schema(): real boots run
'alembic upgrade head' (schema owned by the migration history); tests
(DECNET_TESTING=1) keep create_all, which is faster and needs no upgrade
path. MySQL wraps the upgrade in the existing GET_LOCK advisory lock so
concurrent uvicorn workers don't race on DDL.
Deletes the three _migrate_* crimes (attackers-table legacy drop +
GeoIP backfill, TEXT->MEDIUMTEXT widening) — all now handled by the
baseline migration and the _BIG_TEXT model variants. Drops the test
file that only exercised the deleted helpers; adds tests pinning the
alembic-vs-create_all gate and guarding that every model table is in
the migration head.
Introduce Alembic at v1. Migrations live inside the package
(decnet/web/db/migrations) so they ship with installs; alembic.ini at the
repo root drives the CLI. env.py is async and dual-backend, selecting the
engine from DECNET_DB_TYPE (mirroring db/factory.py) and reusing the app's
own connection when run programmatically.
The baseline captures all 39 tables. _BIG_TEXT round-trips as
Text().with_variant(MEDIUMTEXT, 'mysql'), so both backends get the right
column type from the migration. kd_digraph_simhash gains a sqlite BLOB
variant: BINARY(8) reflects as NUMERIC on SQLite and would otherwise trip
'alembic check' forever.
Live topology edits fired one mutation per canvas action. That coupled
each edit to an immediate enqueue+apply, which (post-serialization)
raced the SSE refetch and duplicated optimistic placeholders, and gave
the user no chance to assemble a coherent changeset (add a net AND
bridge it) before any of it landed.
Live edits now STAGE: each editor primitive records its op and returns
immediately; the optimistic placeholders callers already draw are the
staged preview. The action button reads UPDATE (n) when live (DEPLOY
when pending) and flushes the batch through the slice-1 submit queue —
sequential, version-cursored, each awaited to a terminal state, stopping
loudly on the first failure with the unapplied remainder kept for retry.
REFRESH becomes DISCARD (n) to drop the batch. SSE refetch is paused
during a commit so per-mutation applied events don't wipe still-staged
placeholders mid-batch; one refetch reconciles at the end.
Also fix _dropArchetype, which bailed without an optimistic node on the
staged path, leaving a decky added to an uncommitted LAN invisible until
UPDATE.
Live MazeNET edits fired their mutations fire-and-forget: each canvas
action enqueued immediately and never awaited the result. Two failures
followed from that:
- expected_version is bumped at ENQUEUE (not at apply), so two ops fired
back-to-back raced — the second carried a stale version and 409'd.
Edits only worked when hand-paced (an SSE refetch landed between them).
- A failed mutation degrades the topology, but the only signal was a 4s
toast, so the user saw DEGRADED with no cause.
useTopologyEditor now routes every live op through a serialized submit
queue: one enqueue in flight at a time (submission order preserved), an
optimistic expected_version cursor advanced per enqueue so back-to-back
ops (e.g. reparent's detach+attach) don't need a refetch between them,
and each mutation awaited to a terminal state. A 'failed' row throws
MutationFailedError, which the page pins as a persistent UPDATE FAILED
banner instead of a vanishing toast.
Slice 1 of the live-edit rework; stage+UPDATE-button batching and louder
backend materialisation reporting to follow.
The Fleet UI only showed TEARDOWN for swarm-pinned deckies (POST
/swarm/hosts/{uuid}/teardown). Local deckies had no delete control though
the API now exposes DELETE /deckies/{name}.
teardown() branches on swarm vs local; the card's two-step arm/CONFIRM
button renders for any admin, keyed td:${host_uuid ?? 'local'}:${name}.
The Fleet module had no delete — neither UI nor API — though the engine
capability existed (engine.teardown(decky_id=...), exposed only via
`decnet teardown --id`). Wire it to HTTP.
DELETE /deckies/{name} (admin-gated, 204). Synchronous: a single decky's
compose stop/rm is quick, so it's awaited off-thread rather than the
202+lifecycle path deploy/mutate use for slow builds. The single-decky
teardown never touches the host macvlan interface, so it needs no extra
CAP_NET_ADMIN.
State consistency: engine.teardown removes the containers and the
fleet_deckies row but leaves the decky in decnet-state.json. Left as is, the
reconciler would see "present in JSON, absent from DB" and re-INSERT the row,
resurrecting the decky. So the handler prunes it from both decnet-state.json
and the DB deployment key after teardown; deleting the last decky clears
state entirely (DecnetConfig.deckies has min_length=1).
Route ordering: the dynamic DELETE /deckies/{decky_name} is registered AFTER
the fixed /deckies/* routes (Starlette matches in registration order), so it
no longer shadows DELETE /deckies/files (file-drop).
Tests cover 401/403/404/422, single-delete pruning, and last-decky clear.
These had been red since the changes they cover landed — invisible because
the pre-commit gate runs mypy/ruff/bandit/pip-audit but NOT pytest, so failing
tests don't block commits and quietly accumulate.
- SSE stream/events auth migrated from ?token=<jwt> to a single-use ?ticket=
(commit efb4e49d). Three tests still passed a raw JWT as ?token= and got
401. Updated to mint a ticket via POST /auth/sse-ticket and pass ?ticket=
(attacker events, topology events, /stream).
- The user-creation password policy is min_length=12; the RBAC admin-access
test still used a 10-char password and was rejected. Bumped to a valid one.
pip-audit flagged fixable advisories in the web stack:
- cryptography -> >=48.0.1 (GHSA-537c-gmf6-5ccf)
- python-multipart -> >=0.0.31 (CVE-2026-53538/53539/53540)
- starlette (transitive via fastapi) -> add direct floor >=1.3.1
(CVE-2026-48817/48818/54282/54283)
Venv synced to cryptography 49.0.0, python-multipart 0.0.32, starlette
1.3.1; full tests/api/ suite green against the bump. Also drops the stray
browser-use[core] dev dep (the browser-use skill uses a global CLI; the
package is imported nowhere in DECNET).
Two defects exposed after the deploy-success loop fix (verified live):
1. Duplicated / skipped transcript lines. The placeholder-log interval did
`setLog(prev => [...prev, msgs[i]])` then `i++`. React 18 auto-batches
setInterval updaters, so the updater ran after i had advanced and read the
wrong index — skipping some lines ([NET], [SENSE]) and duplicating others
([TLS]). Fixed by capturing `const line = msgs[i]` before scheduling the
update. A placeholderStartedRef also gates the effect to one run per deploy
(reset in startDeploy) as defense-in-depth against re-render churn.
2. Wizard never closed on success. The completedRef guard combined with the
auto-close effect's cleanup was self-defeating: a re-render inside the
700ms window (e.g. the [OK] terminal-log append) ran the cleanup, clearing
the pending close, and the guard then blocked rescheduling — so onComplete
never fired. The timer now lives in a ref cleared only on unmount, so a
scheduled close always fires exactly once regardless of re-renders.
Adds a regression test that a re-render during the close countdown does not
cancel the close. Verified end-to-end against the live instance: all 8 log
lines render once in order, the wizard auto-closes, and /deckies does not
storm.
After a successful deploy the wizard hammered GET /deckies and stacked
"DEPLOYED" toasts unbounded (~1.4/s, forever). Two compounding causes:
1. The auto-close effect's dep array includes onComplete, which the parent
passes as an inline arrow (new reference every render). onComplete calls
refresh(), re-rendering the parent, producing a new onComplete ref,
re-running the effect, which reschedules onComplete — a feedback loop.
2. DeployWizard stayed permanently mounted (open only toggled a child
overlay), so its hooks kept running with lifecycleDone===true after close.
Fix both: a completedRef guard makes the auto-close fire exactly once per
deploy regardless of effect re-runs (reset in startDeploy), and the parent
now mounts the wizard only while open so closing tears down its hooks and
clears the latent "reopen re-completes stale rows" path.
Lifecycle polling itself was never the runaway — it stops cleanly at
terminal status; the bounded ~25-poll build window is expected.
Adds a regression test asserting onComplete fires once across a simulated
re-render storm.
