_topology_compose_path returned a CWD-relative Path, so every
deploy/mutate/dry-run wrote decnet-topology-<id8>-compose.yml into the
process CWD (the install dir). Teardown computed the same relative path
against its own CWD, so when it differed the unlink() missed the orphan
and files accumulated forever.
Anchor to $DECNET_RUN_DIR (default /var/lib/decnet/topologies, tempdir
fallback) so write and teardown always agree regardless of CWD.
Adds the @pro ScanImport contract (ProScanImportProps/ProScanImport) and
a null community stub, and slots a third SCAN-BASED card into
CreateTopologyWizard, gated on the pro panel being present so it
tree-shakes out of the community build. The scan->topology importer
itself ships in decnet/pro v1.2.0. CHANGELOG updated under [1.2.0].
Prefork supervisor (decnet.prefork) + 'decnet fleet heavy' (profiler+ttp,
CoW-shared, ~412MB Pss vs 661MB). ATT&CK bundle -> decnet/data/ (19.1).
Removed 10 per-worker unit templates superseded by the supervisor groups
and the heavy fleet.
Prefork worker consolidation (decnet.prefork + decnet fleet heavy),
ATT&CK 19.1 relocation to decnet/data/, and removal of the 10 per-worker
unit templates superseded by the supervisor groups + heavy fleet.
The batch/cpu supervisor groups + heavy fleet replace 10 per-worker units
(reconciler/enrich/orchestrator/mutator/clusterer/campaign-clusterer/
attribution/reuse-correlator/profiler/ttp). Removed their deploy/*.service.j2
templates and rewired decnet.target to the 3 consolidated units. Dropped
test_orchestrator_unit.py (tested a removed unit). CLI commands (decnet ttp,
mutate, …) stay for manual runs; new units' Conflicts= still name the old
units defensively for hosts mid-migration.
Wires the prefork primitive into a CLI command. 'decnet fleet heavy' imports
the shared base floor once in the master, then forks profiler + ttp as
CoW-sharing child processes (own process/GIL, full isolation, shared ~71MB
floor). DB-only tier => systemd unit carries no extra privilege (prefork's
privilege-union cost is nil for this fleet). Unit Conflicts= the profiler/ttp
units it replaces. Heavy per-worker state (ATT&CK/ML) still loads per-child;
warming it in the master to share is deferred until a live RSS measurement
shows the big object graph CoW-shares rather than refcount-dirties.
Bundle pointer moved from repo root to decnet/data/ (with LICENSE.txt),
gitignored + fetched on demand (51MB, MITRE-licensed). Version pin bumped
19.0->19.1 with the new sha256; license unchanged. All _REPO_BUNDLE test
constants repointed. Fixes test-web failures after the repo-root bundle
was deleted.
CoW measurement on CPython 3.14: forked idle child keeps ~71MB shared,
dirties ~1MB private; working child ~26MB. PEP 683 immortal objects keep
code/module pages clean so gc.freeze() is unnecessary (freeze==nofreeze).
prefork.run_fleet: master imports the base floor once, forks one child
per worker (own process/GIL, CoW-shared floor), reaps + restarts with
backoff, graceful SIGTERM->SIGKILL shutdown. Not yet wired to a command
(that lands when 1.2 picks the target worker set).
confidence_max is a ceiling (min(base, ceiling)), not a multiplier — the
ASVS pass fixed this (BUG-8: min(base, base*ceiling) -> min(base, ceiling)),
but 4 lifter clip tests still encoded the old base*ceiling math (0.45/0.4/
0.35) and were masked by the make test-web bundle error fail-fast. All four
now assert the 0.5 ceiling. Separately, test_topics_matches_documented_set
lacked attacker.fingerprinted, which worker.py legitimately subscribes to
(JARM/HASSH/tcpfp/ipv6_leak -> TTP tagging). Located via turbovec + git pickaxe.
(cherry picked from commit f83b467c35649a06fa36f4b350e6666379cd71cb)
Hosts clusterer/campaign-clusterer/attribution/reuse-correlate in one
process. The two O(n^2) connected-components kernels (cluster_observations,
cluster_identities) offload to ONE shared forkserver pool via decnet.offload
.run_kernel, so they run in parallel instead of serialising under the GIL.
- offload.run_kernel: pool when installed + offload_if holds, else inline.
Standalone workers and all tests run inline => behaviour unchanged
(424 clustering/correlation tests green).
- offload_if gates on input size (>=256) to skip pickle cost on small passes.
- forkserver (not fork): supervisor is multithreaded via bus clients.
- attribution/reuse co-located but not offloaded yet (lighter; same run_kernel
path extends to them if profiling shows contention).
- systemd unit Conflicts= the 4 units it replaces; no docker/raw-socket priv.
Controlled unit swap on mothership: single PID hosts all 4 workers,
shared repo + 30s reconcile pass confirmed working, no crashes. Live
delta beat the floor estimate. Batch group complete.
enrich is a batch-group member; its individual unit must also be mutually
exclusive with the supervisor. Unit auto-renders via init.py glob of
deploy/decnet-*.service.j2 — no installer list change needed.
4 live batch workers = 509MB; consolidated startup floor = 118.5MB
(imports 102.5 + repo pool 15.7 once vs 4x + bus 0.3). Side-effect-free
measurement (no mutation tick). Floor is a lower bound; live adds modest
per-worker working set. Exact live number pending controlled unit swap.
Hosts reconcile/enrich/orchestrate/mutate in one process via the
supervision primitive: one import floor, one shared repo/DB pool instead
of 4. Static group registry (membership is architectural, not a knob);
factories lazy-import only the hosted workers. systemd unit Conflicts=
the individual units it replaces and documents the union-of-privileges
cost. Worker code unchanged — any member is extractable by editing _build_specs.
forwarder/listener are role-split swarm singletons, not co-resident with
the herd — drop from grouping. Real master-resident batch group shares
the repo singleton (one DB pool when consolidated). Stage 1 = batch:
reconcile/enrich/orchestrate/mutate. webhook/canary deferred standalone.
HOW to consolidate: supervision-loop primitive (not TaskGroup, whose
all-or-nothing cancel breaks isolation); group by failure domain +
resource profile keeping per-group cgroup limits; every worker remains
config-extractable. Recommend process-groups now (~18->~9 units),
evaluate prefork+gc.freeze CoW on 3.14 as the higher-ceiling follow-on.
Measurement after C2: all 25 CLI modules transitively pull the SQLModel
ORM, and the table chain is a hub (one table loads the whole registry).
Most workers genuinely use the DB, so lazy imports only help the 2-3
DB-less workers. Consolidation (pay the 86MB floor once for the herd)
is the reliable ~600MB win — promoted from fallback to primary.
topology/__init__ eagerly imported generator -> allocator -> repository ->
the full SQLModel ORM. Defer via PEP 562 __getattr__ so importing the
package doesn't drag the DB layer into DB-less workers. Public API
(from decnet.topology import generate) unchanged. Guard test locks it in.
Fleet resident set ~2.57GB across 18 workers; ~1.5GB is the 86MB import
floor paid 18x. Pinned root cause: topology/__init__ eager re-export of
generate drags the full SQLModel ORM (26 tables, ~38MB) into every worker.
The core CLI scans decnet/pro/cli/ and calls each module's register(app),
registered before the master-only gate so pro commands are mode-filtered like
the rest. Lets the Professional tier add commands and standalone daemon entry
points (decnet pro-<cmd> serve, supervised by a systemd unit). No-op in the
Community build (no decnet.pro). Test asserts the shipped pro group registers
when mounted; skips otherwise.
Move the pro mount decnet/services/pro/ -> decnet/pro/ so the Professional tier
can contribute to more than honeypots. The core wires each surface only when
decnet/pro/ is present (absence stays the entitlement gate):
* services — registry scans decnet/pro/services/ (was decnet/services/pro/)
* API routes — decnet/pro/routes.py exposes ROUTERS, mounted under /api/v1
* web pages — Vite aliases @pro to the pro frontend (community -> empty stub),
App.tsx maps proRoutes into <Route>s, Layout renders a
PROFESSIONAL nav group; both tree-shake out of the community build
Frontend gate mirrors the existing VITE_DECNET_DEVELOPER tree-shake pattern.
Tests: registry + router seams (backend), empty-stub contract (frontend).
Newer pytest raises 'Marks cannot be applied to fixtures' instead of ignoring
it. The async test methods already carry @pytest.mark.anyio, which is what
selects the backend; the fixture must not.
Pro-tier honeypots load from an optional decnet/services/pro/ subpackage that
the registry auto-discovers when present; the Community build omits it, so the
directory's absence IS the entitlement gate (no runtime licence check). Recurse
subclasses so a pro service may extend a community one. Exclude pro from the
community wheel and git-ignore the path (it lives in the private
decnet-professional repo).
Add LICENSING.md documenting the dual-license: AGPL-3.0-or-later core plus a
commercial EULA for the Professional tier.
Campaign clusterer gains a keystroke edge: when two identities'
kd_digraph_simhash centroids are within KD_HAMMING_MAX bits, a graded
weight (1.0 at identical, fading to 0 at the cutoff) feeds the campaign
graph. Supporting tier (0.6) — a typing match plus temporal overlap
reaches threshold, but typing alone never merges (FP guard against
coarse, noisy terminal timing).
Projects the column through IdentityFeatures + from_identity_row.
The identity clusterer folds an identity's per-session
motor.digraph_simhash observations into one 8-byte bitwise-majority
centroid (denoises per-session jitter) and writes it to
AttackerIdentity.kd_digraph_simhash via update_identity_fingerprints —
the orphaned column is now populated. list_identities_for_clustering
projects it so the campaign clusterer can read it.
Extends the repo abstract + DummyRepo stub/coverage.
Per-session 64-bit SimHash of inter-keystroke digraph flight times:
walk single-char input events, accumulate flight time per (c1,c2),
bucket the median, Charikar-SimHash the bucketed pairs. Locality-
sensitive so the same typist is Hamming-close across sessions; pastes
and think-pauses break the chain; silent below the sample-size floor.
New shared decnet/util/simhash.py (simhash64/hamming64/bytes helpers).
Registered as a conditional Tier-A primitive (count 37->38); requires
behave-shell>=0.1.2.
initialize() now delegates to _apply_schema(): real boots run
'alembic upgrade head' (schema owned by the migration history); tests
(DECNET_TESTING=1) keep create_all, which is faster and needs no upgrade
path. MySQL wraps the upgrade in the existing GET_LOCK advisory lock so
concurrent uvicorn workers don't race on DDL.
Deletes the three _migrate_* crimes (attackers-table legacy drop +
GeoIP backfill, TEXT->MEDIUMTEXT widening) — all now handled by the
baseline migration and the _BIG_TEXT model variants. Drops the test
file that only exercised the deleted helpers; adds tests pinning the
alembic-vs-create_all gate and guarding that every model table is in
the migration head.
Introduce Alembic at v1. Migrations live inside the package
(decnet/web/db/migrations) so they ship with installs; alembic.ini at the
repo root drives the CLI. env.py is async and dual-backend, selecting the
engine from DECNET_DB_TYPE (mirroring db/factory.py) and reusing the app's
own connection when run programmatically.
The baseline captures all 39 tables. _BIG_TEXT round-trips as
Text().with_variant(MEDIUMTEXT, 'mysql'), so both backends get the right
column type from the migration. kd_digraph_simhash gains a sqlite BLOB
variant: BINARY(8) reflects as NUMERIC on SQLite and would otherwise trip
'alembic check' forever.
Live topology edits fired one mutation per canvas action. That coupled
each edit to an immediate enqueue+apply, which (post-serialization)
raced the SSE refetch and duplicated optimistic placeholders, and gave
the user no chance to assemble a coherent changeset (add a net AND
bridge it) before any of it landed.
Live edits now STAGE: each editor primitive records its op and returns
immediately; the optimistic placeholders callers already draw are the
staged preview. The action button reads UPDATE (n) when live (DEPLOY
when pending) and flushes the batch through the slice-1 submit queue —
sequential, version-cursored, each awaited to a terminal state, stopping
loudly on the first failure with the unapplied remainder kept for retry.
REFRESH becomes DISCARD (n) to drop the batch. SSE refetch is paused
during a commit so per-mutation applied events don't wipe still-staged
placeholders mid-batch; one refetch reconciles at the end.
Also fix _dropArchetype, which bailed without an optimistic node on the
staged path, leaving a decky added to an uncommitted LAN invisible until
UPDATE.
Live MazeNET edits fired their mutations fire-and-forget: each canvas
action enqueued immediately and never awaited the result. Two failures
followed from that:
- expected_version is bumped at ENQUEUE (not at apply), so two ops fired
back-to-back raced — the second carried a stale version and 409'd.
Edits only worked when hand-paced (an SSE refetch landed between them).
- A failed mutation degrades the topology, but the only signal was a 4s
toast, so the user saw DEGRADED with no cause.
useTopologyEditor now routes every live op through a serialized submit
queue: one enqueue in flight at a time (submission order preserved), an
optimistic expected_version cursor advanced per enqueue so back-to-back
ops (e.g. reparent's detach+attach) don't need a refetch between them,
and each mutation awaited to a terminal state. A 'failed' row throws
MutationFailedError, which the page pins as a persistent UPDATE FAILED
banner instead of a vanishing toast.
Slice 1 of the live-edit rework; stage+UPDATE-button batching and louder
backend materialisation reporting to follow.
The Fleet UI only showed TEARDOWN for swarm-pinned deckies (POST
/swarm/hosts/{uuid}/teardown). Local deckies had no delete control though
the API now exposes DELETE /deckies/{name}.
teardown() branches on swarm vs local; the card's two-step arm/CONFIRM
button renders for any admin, keyed td:${host_uuid ?? 'local'}:${name}.
