Rewrites the architecture section for the full current module tree and adds
new sections for the REST API, swarm/agent mode, service bus, attacker
intelligence stack (profiler, clustering, correlation, GeoIP/ASN),
MazeNET topology, canary tokens, and TTP tagging/export. Updates the CLI
reference table, test count (478 → 5050), and Python version constraints.
Replaces LICENSE (GPLv3 -> AGPLv3) and prepends
`SPDX-License-Identifier: AGPL-3.0-or-later` to every source file
across decnet/, decnet_web/, tests/, scripts/, and tools/.
Rationale: closes the GPLv3 ASP loophole so any party operating a
modified DECNET as a network service must offer their modified
source. Personal copyright (Samuel Paschuan) + inbound=outbound
contributions make a future unilateral relicense infeasible.
- LICENSE: full AGPL-3.0 text (gnu.org/licenses/agpl-3.0.txt)
- COPYRIGHT: project copyright notice
- tools/add_spdx_headers.py: idempotent header injector
(shebang- and PEP 263-aware)
Touches 1565 source files (.py, .ts, .tsx, .js, .jsx, .css, .sh).
No behavior change; comments only.
Every compose invocation used -p decnet so fleet + every topology
lived in one docker compose project. --remove-orphans, run during
fleet pre-up cleanup and on every topology teardown / rollback, then
swept every container in the project not listed in the current compose
file — wiping sibling topologies and the flat fleet along with the
intended target.
Parameterize project on _compose / _compose_with_retry / _compose_ps
(default FLEET_COMPOSE_PROJECT="decnet"). Add _topology_compose_project
that returns decnet-topo-<id8>, and pass it through every topology
compose call site (master deploy_topology + rollback + post-deploy ps,
master teardown_topology, agent apply, agent teardown, all four live
service mutations on topology deckies). Fleet calls keep the default
and are unaffected.
Migration: live containers from before this fix remain in the shared
"decnet" project and need a one-time manual cleanup before they're
reachable to the new topology code paths.
The wizard POSTs only the new decky on each submit. The handler used to
treat every INI as the complete desired fleet (config.deckies = INI) so
the reconciler tore down prior deckies as orphans — deploying a second
Windows workstation silently wiped the first.
Add replace_fleet to DeployIniRequest (default false). Default path
merges new deckies into existing config and rejects name/IP collisions
with 409. replace_fleet=true preserves set-desired-state semantics for
CLI / declarative callers. Lifecycle rows are created only for the
deckies submitted in the current call, so /deckies/lifecycle?ids=...
reflects exactly what this submit deployed.
build_deckies_from_ini gains reserved_ips so additive auto-allocation
skips IPs already held by the existing fleet.
- New useLifecyclePolling(ids, intervalMs) hook: polls
GET /deckies/lifecycle?ids=... every 2s until every row is terminal,
surfaces transient HTTP failures without giving up.
- DeployWizard: drops the 180s axios timeout and the fake-log-driven
deployOk flag. After POST 202, sets lifecycle_ids -> the hook drives
the per-decky pill grid (PENDING / RUNNING / SUCCEEDED / FAILED).
Real terminal lines stream into the log as rows resolve. Auto-close
on all-success after 700ms.
- DeckyFleet.css: .lifecycle-grid + .lifecycle-pill in the existing
fleet vocabulary; running pill pulses, failed pill borders alert.
- Existing 4 wizard render tests still pass; 4 new hook tests cover
empty ids / single-success / polling-until-terminal / HTTP error.
GET /deckies/lifecycle?ids=<uuid>&ids=<uuid> returns the matching
DeckyLifecycle rows so the wizard can poll instead of holding an HTTP
request open across compose work. require_viewer gating -- read-only.
Startup sweep: on master boot, any pending/running row with
started_at older than 1h flips to failed with
error='master restarted during operation'. Pre-v1 substitute for a
durable task queue: if the master crashes mid-deploy, the wizard sees
FAILED on refresh and the operator retries. Idempotent + cheap; runs
unconditionally including in contract-test mode.
This is the unblock for the wizard hang. Both endpoints used to run
docker compose synchronously inside the HTTP handler -- on master
(unihost) or via asyncio.gather of worker /deploy POSTs at 600s
timeout each (swarm) -- blocking every other API request.
New flow:
1. Commit the new config shape to repo state (fast).
2. Create one DeckyLifecycle row per decky (status=pending).
3. Spawn asyncio.create_task(run_deploy / run_mutate) -- the
lifecycle runner drives rows through running -> succeeded|failed
and emits decky.<name>.lifecycle on the bus.
4. Return 202 with {lifecycle_ids: [...]}. Wizard polls
GET /deckies/lifecycle?ids=... (next commit).
mutator/engine.py gains pick_new_services() -- shared between the
async API path and the watch-loop's synchronous mutate_decky().
DeployResponse grows lifecycle_ids[]. The old dispatch_decnet_config
helper still exists for the CLI swarm-deploy command path; it just
isn't called from the API handler anymore.
Test changes: 200 -> 202, drop dispatch_decnet_config mocks (handler
no longer calls it), assert lifecycle_ids in response + committed
state matches expectations.
HeartbeatRequest grows an optional lifecycle field carrying per-decky
completion records from the worker:
[{decky_name, operation, status, error?, completed_at?}]
For each delta, the master finds the most-recently-started open
DeckyLifecycle row for (decky_name, operation, host_uuid) and flips
it to terminal with the worker's error text + timestamp. Stale
duplicates (row already sealed or never existed) are logged and
dropped -- not errors.
Each successful pivot also emits decky.<name>.lifecycle on the bus
so the dashboard sees the transition without waiting for its next
poll tick.
This is the master-side completion channel for the worker's 202
fire-and-forget /deploy and /mutate.
The wizard API used to hang because /deckies/deploy ran docker compose
build && up -d synchronously, holding the request thread for minutes.
The worker side of that pipeline now returns 202 Accepted immediately
and runs the deploy in an asyncio.create_task.
On task completion (success or failure) the worker pushes a one-off
heartbeat carrying a lifecycle delta per decky:
{decky_name, operation, status: succeeded|failed, error?, completed_at}
Master pivots these onto open DeckyLifecycle rows in the heartbeat
handler (next commit). The scheduled 30s heartbeat tick is the
fallback if the immediate push drops.
- decnet/agent/app.py: /deploy and /mutate return 202; dry_run mutate
still validates synchronously and returns 200.
- decnet/agent/executor.py: deploy_async + mutate_async wrap the work
and push the completion delta.
- decnet/agent/heartbeat.py: push_lifecycle_delta() helper builds a
one-off body and POSTs with the same mTLS context as the loop.
- decnet/swarm/client.py: revert deploy/mutate to control timeout
(master no longer holds the HTTP request open for compose work).
Worker state.json gains no lifecycle field -- master DeckyLifecycle is
the source of truth; the master sweep handles crashed-mid-deploy
recovery.
Add decnet.lifecycle package: pure orchestration layer that the
master API will invoke via asyncio.create_task to drive DeckyLifecycle
rows through pending -> running -> succeeded | failed without
holding an HTTP request open.
Strategy classes per (operation, transport):
- LocalDeployStrategy: master-resident, runs engine.deployer.deploy
in a thread.
- SwarmDeployStrategy: shards by host_uuid, dispatches via
AgentClient.deploy; worker drives terminal via heartbeat.
- LocalMutateStrategy: write_compose + compose up.
- SwarmMutateStrategy: AgentClient.mutate; worker drives terminal.
decnet.bus.topics gains decky_lifecycle(name) -> decky.<name>.lifecycle
plus DECKY_LIFECYCLE constant. Payload documented in the wiki
(separate commit). publish_safely keeps bus best-effort.
Nothing is wired to call this yet -- next commits convert worker
/deploy /mutate to 202, then heartbeat delta wiring, then master API.
One row per (decky, operation) attempt. State machine:
pending -> running -> succeeded | failed (+ error text). Rows are
append-only after terminal; retries write a new row.
Sibling of DeckyShard rather than a rework -- DeckyShard tracks
runtime container state observed via heartbeat, this tracks
operation lifecycle. New table, UUID PK.
Adds BaseRepository abstract methods (create_lifecycle,
update_lifecycle, get_lifecycle_by_ids, find_open_lifecycle,
sweep_stale_lifecycle) with SQLModelRepository mixin impl.
Backbone for the upcoming 202-Accepted async API.
- Implement /mutate handler: load_state, update services + last_mutated,
save_state, write_compose, compose up -d via asyncio.to_thread. 404
for missing state / unknown decky_id. dry_run short-circuits before
any side effect.
- Add AgentClient.mutate(decky_id, services, *, dry_run=False) using
_TIMEOUT_DEPLOY (compose up can pull/build, exceeds control timeout).
- mutator/engine.py: in swarm mode with decky.host_uuid set, resolve
worker via _resolve_swarm_host and dispatch through AgentClient.mutate
instead of writing a compose file on master. Master-resident deckies
(unihost mode, or swarm with host_uuid=None) keep the local path.
Adds state_path ServiceConfigField and passes DNS_STATE_PATH into the
container environment. Operator must mount the parent directory on a
volume for persistence to survive container recreation.
Switch burst deque from monotonic() to time.time() (wall-clock, serializable).
Add DNS_STATE_PATH env var: on startup _load_state() reads {src:[ts,...]} JSON
and prunes entries older than the burst window. _flush_state() write-then-renames
atomically; _state_flusher() coroutine flushes every 5s when dirty. Detection of
the 5th event also triggers an immediate flush. No-op when DNS_STATE_PATH is
unset, so the default deployment is unchanged.
Rename _txt_times -> _tunnel_times. Add TYPE_CNAME=5, TYPE_NULL=10,
TYPE_PRIVATE=65399 constants. Guard burst counter with _TUNNEL_QTYPES
frozenset instead of TYPE_TXT only. Mixed-type queries from one source
now share a single burst window, closing iodine NULL/CNAME downlink
and AAAA-encoded uplink evasion gaps.
_is_tunneling now returns str|None (the detection method) instead of bool.
Two new tunables _QNAME_TOTAL_LEN_THRESHOLD=50 and _QNAME_ENTROPY_THRESHOLD=3.5
catch attackers who split a high-entropy payload across multiple short labels.
tunnel_method field added to tunneling_suspect events for downstream correlation.
_parse_edns_size only extracted the requestor UDP size; every other field in
the OPT record (DO bit, EDNS version, extended RCODE, all sub-options) was
invisible. Replaced with _parse_opt_record returning a full dict:
udp_size, ext_rcode, version, do_bit, z, options[(code, len, data)]
NSID request (option code 3) is now detected as fingerprint_probe with
probe=edns_nsid and contributes to recon_burst. DO bit, COOKIE (10), and
other options are not escalated; udp_size continues to drive amp_probe.
Tools like fpdns send OPCODE=IQUERY/STATUS/NOTIFY/UPDATE or set the reserved
Z bit to fingerprint resolver behaviour. Previously all these were parsed as
standard queries with no signal.
- opcode!=0 → fingerprint_probe probe=opcode_<name>, NOTIMP response;
fired before qdcount check so qdcount=0 UPDATE packets are still caught.
- Z bit set OR (AD+CD without RD) → fingerprint_probe probe=header_flags;
AD alone with RD is ignored to avoid tagging DNSSEC-aware stubs.
- Both variants contribute to recon_burst.
qclass=255 in a standard query is unusual enough to be a fingerprinting probe
(fpdns, various scanner scripts). Previously it was logged as a plain query
with qclass=ANY in the event field; now it emits fingerprint_probe with
probe=qclass_any and returns REFUSED — consistent with how we treat other
probe types. Contributes to recon_burst.
The inline probe_map dict inside _handle made tests blind to the probe
catalogue and couldn't be extended without touching the hot path. It is now
module-level _CHAOS_PROBE_MAP. authors.bind. joins the three existing entries
so it gets named correctly instead of carrying the raw qname.
Packets with multiple questions were silently parsed at q0 only; the extra
questions were invisible. Now emits multi_question at severity=5 with the
qdcount and q0 qname, then falls through and answers q0 normally.
Silent drops on <12B packets, qdcount=0, and question-section ValueError gave
fuzzers and scanners a completely dark target. New events malformed_packet,
empty_question_section, and question_parse_error fire at severity=5 so these
probes are visible without counting toward recon_burst.
Adds DNS_FORWARD_BUDGET (default 50) and DNS_FORWARD_WINDOW (default 1.0s)
env vars. _can_forward() maintains a rolling deque of upstream call
timestamps; queries that exceed the budget within the window are answered
with the sinkhole (127.x) instead of being forwarded, making the honeypot
ineligible as a sustained amp vector even when real_recursive is enabled.
Rate limit is global (not per-source) so IP-spoofed amplification floods
hit the ceiling regardless of how many source addresses are rotated.
When DNS_REAL_RECURSIVE=true and DNS_ZONE_MODE=recursive, out-of-zone
queries are forwarded to DNS_UPSTREAM (default 8.8.8.8:53) via async
UDP. Upstream response is relayed as-is; on timeout or error the
already-computed sinkhole (127.x) is returned instead.
_handle() always runs first so logging, tunneling detection, flood
tracking, and recon-burst aggregation fire on every query regardless
of whether the response ultimately comes from upstream. _dispatch()
overlays forwarding on top of the sync handler.
Protocol handlers (UDP datagram_received, TCP session) are now async
via asyncio.ensure_future / await _dispatch(). Service class exposes
real_recursive (bool) and upstream (string) config fields.
RA=1 + empty answer section is immediately detectable as fake by any
open-resolver scanner. Recursive mode now behaves like open mode
(127.0.0.x sinkhole, deterministic on qname) with RA=1 and AA=0,
matching what a real recursive resolver returns.
- Add per-src QPS counter (_qps_window) with flood_suspect event at ≥50 qps/10s;
one event per src per 30s cooldown, does not suppress baseline query events.
- Add tracking_evicted telemetry every 100 LRU evictions so IP-rotation evasion
of _txt_times/_qps_window/_recon_window is observable, not silent.
- Shared _track_lru helper consolidates LRU touch + eviction signalling across
all three bounded OrderedDicts.
- Add TYPE_AAAA=28 support: _fake_ipv6() returns deterministic ULA (fd::/8)
addresses for in-zone names; extra_records parser now accepts and validates
AAAA entries via socket.inet_pton.
- Add per-src recon-burst aggregation (_recon_window): fingerprint_probe +
zone_transfer + amp_probe are tracked per source in a 60s window; recon_burst
fires when ≥2 distinct signal types seen, once per src per 120s cooldown.
- 47 tests passing (19 new across TestAAAARecords, TestFloodDetection, TestReconBurst).
Python asyncio DNS server on UDP+TCP/53 masquerading as BIND 9.x.
Emits four event_type values: query, fingerprint_probe (version.bind /
hostname.bind / id.server CHAOS), zone_transfer (AXFR/IXFR, always
REFUSED), amp_probe (qtype=ANY or EDNS udp_size>1232), and
tunneling_suspect (long high-entropy labels or rapid TXT burst).
Zone persona is generated per-decky from instance_seed (domain name,
SOA serial, NS, A, MX, TXT SPF); overridable via config_schema.
Three zone modes: auth (default), recursive, open (sinkhole).
AttackerData type gets bgp_prefix / rpki_status / rpki_source.
TimelineSection renders prefix inline next to AS number; RPKI status
shows as a green RPKI VALID / red RPKI INVALID badge, or dim
NO ROA for not-found. rpki-status-badge CSS added to Dashboard.css.
Export network block extended with the three new fields.
Import enrich_rpki from decnet.rpki and call it inline after the
ASN lookup. bgp_prefix, rpki_status, rpki_source added to the
record dict that feeds the Attacker upsert. enrich_rpki short-circuits
to (None, None) when asn is None, so private / unannounced IPs
never hit RIPE STAT.
bgp_prefix (max 43 chars, indexed) holds the covering CIDR from
the ASN lookup. rpki_status / rpki_source hold RIPE STAT validation
outcome. All nullable — null means enrichment was skipped or ASN
did not resolve.
RipeStatValidator makes two RIPE STAT calls per uncached IP:
network-info -> announced prefix, rpki-validation -> ROA state.
2-second timeout; any network failure returns status='unknown'.
SQLite cache keyed by IP, 12-hour TTL, pruned on validator init.
Cache avoids per-event HTTP for the high-churn attacker pool —
steady-state cost approaches zero for repeat offenders.
Synthesize the covering CIDR at lookup time from the matched iptoasn
range using ipaddress.summarize_address_range. AsnInfo.prefix is
populated per-query; not persisted in the pickle cache.
enrich_ip now returns (asn, as_name, bgp_prefix, provider_name).
Profiler worker updated to unpack the 4-tuple and write bgp_prefix
into the attacker record dict.
Four RFC 4443 stimuli (port-unreach, hop-limit-exceeded, unknown-NH,
bad-dest-option) produce a 4-char matrix + sha256 fingerprint for IPv6
attackers. Auto-registers via ActiveProbeMeta at priority=860 (after v4
icmp_error=850, before ipv6_leak=999). IPv4 targets fast-return None.
Sends four crafted stimuli (UDP/closed-port, TTL=1, DF+oversized,
bad IP option) and records which ICMP error classes come back, the
per-error RTT, and the bytes echoed in each ICMP body. Absence is
as informative as a reply — Linux rate-limiting is a fingerprint signal.
Returns None when no packets could be sent (no CAP_NET_RAW), so the
probe is a no-op in non-root test environments. Port-free ActiveProbe
subclass (priority=850), metaclass auto-registered in the registry.
Also fixes three sets of stale tests left over from the TlsCertProbe
migration (4b2759e0):
- test_active_probe_registry: closed name/order sets updated for
tls_certificate and icmp_error
- test_prober_rotation: dead patches on worker.fetch_leaf_cert removed
- test_prober_worker (TestProbeCycleTLSCert): rewritten to test
TlsCertProbe as an independent registry probe, patch target updated
from worker.fetch_leaf_cert to probes.tlscert_probe.fetch_leaf_cert
TLS cert capture was the last prober special-case that bypassed
ActiveProbeMeta. Moves logic into TlsCertProbe (priority=200, runs
after JARM) in probes/tlscert_probe.py; drops _capture_tls_cert,
the probe.probe_name=="jarm" name-check, and the direct
fetch_leaf_cert import from worker.py.
ActiveProbe.run/syslog_fields/publish_payload now accept port=None so
non-port-iterating probes can live in the registry. Ipv6LeakProbe replaces
the hand-rolled _ipv6_leak_phase special case in worker.py; it runs last
via priority=999. _probe_cycle no longer has an ad-hoc phase call.
Fixes three stale test files (test_prober_bus, test_prober_rotation,
test_prober_worker) that were broken since the 916b21b6 registry refactor.
_route_info() calls _ip_route_get once and returns (on_link, iface);
worker._ipv6_leak_phase now calls it instead of the two separate helpers.
Bare except clauses at _ip_route_get and response parse now log at debug.
Iterates every template with a Dockerfile, builds decnet/<svc>:latest
with DOCKER_BUILDKIT=1. Supports NO_CACHE=1 and FAIL_FAST=0 flags,
mirrors the style of test-all. Updated help target.
FingerprintGroup switch fell through to FpGeneric (raw JSON dump) for all
four new fingerprint_type values the ingester now produces. Add FpJa4h,
FpHttpSettings, FpJa4Quic components and wire them into the dispatcher;
also register their labels and icons in fpTypeLabel/fpTypeIcon.
ingester: wrap bootstrap get_state() in forever-retry loop — MySQL coming
up after the API process killed the ingestion task permanently before it
ever entered _run_loop. Regression test added.
deps: idna 3.13→3.15 (CVE-2026-45409), twisted 26.4.0rc2→26.4.0
(PYSEC-2026-160), pip 26.1→26.1.1 (CVE-2026-3219 resolved upstream),
behave-core/behave-shell renamed from decnet-behave-* and bumped to 0.1.1.
pre-commit hook updated to reflect current ignore list.
Replace _jarm_phase / _hassh_phase / _tcpfp_phase boilerplate (3×~50
lines of identical port-iteration logic) with a metaclass-registered ABC.
Adding a new port-iterating active probe is now one class + three methods.
- decnet/prober/base.py: ActiveProbeMeta auto-registers subclasses by
probe_name; ActiveProbe ABC enforces run/syslog_fields/publish_payload
with env-driven DECNET_PROBE_PORTS_<NAME> port override.
- decnet/prober/probes/{jarm,hassh,tcpfp}.py: concrete probe classes.
- decnet/prober/worker.py: single _run_probe driver replaces the three
phase functions; _probe_cycle iterates ActiveProbeMeta.all(); drops
the ports=/ssh_ports=/tcpfp_ports= kwargs from prober_worker.
- IPv6 leak and TLS cert capture stay as special cases (different call
shapes; intentionally outside the registry).
- tests/prober/test_active_probe_registry.py: registry contents, sort
order, priority-10 override, ABC contract per probe class.
- tests/prober/test_run_probe_driver.py: dedup, success, None-skip,
exception, rotation, publish paths for _run_probe.
- tests/prober/test_prober_worker.py: updated patch targets and
_probe_cycle call sites; port control via monkeypatch.setattr.
- Add "ipv6_leak" to KNOWN_SOURCE_KINDS in ttp/base.py
- Register Ipv6LeakLifter(store) in factory.py get_tagger()
- Subscribe worker to attacker.fingerprinted; route by Event.type
so JARM/HASSH/ipv6_leak share the topic without source_kind collision
- Add bump_attacker_ipv6_leak() to BaseRepository (abstract) +
TTPMixin (implementation): increments ipv6_leak_count, sets last_ipv6_*
denorm fields, appends-with-dedup to AttackerIdentity.ipv6_link_local_iids
- Call bump_attacker_ipv6_leak from _process_event after insert_tags
- Add DummyRepo stub + coverage call in tests/db/test_base_repo.py
Add inline documentation for all known kind= discriminators on the
fingerprinted topic including the new ipv6_leak variant so future
consumers know what fields to expect without reading the prober source.
Ipv6LeakLifter subscribes to source_kind="ipv6_leak" events from both
the passive sniffer and active prober. Emits T1090 (Proxy) under TA0011
(C2) when fe80:: source address is observed — the attacker's VPN only
tunnels IPv4 so their link-local IID leaks their NIC identity.
Rule R0059 sets base confidence 0.85; iid_kind in the evidence carries
the per-observation strength (eui64 = MAC-derived, deterministic;
stable_privacy = RFC 7217; temporary = RFC 4941).
Add ipv6_leak.py with solicit_ipv6_leak() — sends ICMPv6 Echo to
ff02::1 on the attacker's iface and returns fe80:: evidence when a
link-local response arrives. Gated on _is_on_link(): skips when
attacker is behind a router (no L2 adjacency).
Add _ipv6_leak_phase() to worker.py (Phase 4 in _probe_cycle).
Phase runs once per attacker IP per cycle (sentinel at port 0 in
ip_probed["ipv6_leak"]) and publishes kind="ipv6_leak" via publish_fn.
Add list_v6_addrs(iface) to network.py: returns [(addr, scope)] for
all IPv6 addresses on an interface, required for source-routing ICMPv6
from the correct link-local address.
Add _ipv6_iid_classify() to fingerprint EUI-64 vs stable-privacy IIDs
and derive the MAC OUI from EUI-64-encoded link-local addresses.
SnifferEngine._on_ipv6_packet() observes fe80::/10 sources destined for
known deckies and emits ipv6_link_local_leak syslog + bus events.
on_packet() now dispatches the IPv6 branch before the v4 TCP path.
BPF default widened from "tcp" to "tcp or ip6" so the sniff loop
captures IPv6 frames without config change.
Attacker gains five denormalized cache fields (ipv6_leak_count,
last_ipv6_leak_at, last_ipv6_link_local, last_ipv6_iid_kind,
last_ipv6_mac_oui) mirroring the rotation_count/last_rotation_at pattern.
AttackerIdentity gains ipv6_link_local_iids (JSON list[dict]) for
EUI-64-derived MAC cluster signals that survive VPN/IP rotation.
No ALTER TABLE helpers — direct SQLModel column additions per pre-v1 policy.
Pins the evidence shape for IPv6 link-local leakage findings. All fields
optional (total=False) so partial observation (passive sniffer vs active
solicitation) fills whatever the vector provides. Lifter lands in a
subsequent commit.
- Add dedicated test-schema Makefile target (xdist logical, 600s timeout,
-m fuzz) so schemathesis runs separately from test-fuzz, which was
spinning up competing uvicorn workers per xdist process
- Exclude all test_schemathesis*.py files from FUZZ_FLAGS via --ignore
- Add schema to _ALL_SUITES between api and fuzz
- Add SCHEMA_QUICK env var (default 0): caps every max_examples to 100
across all four schemathesis files (4520 -> 600 total examples)
- Fix pre-push hook: use .311 venv and delegate to make test-all FAIL_FAST=0
instead of hand-rolling five separate pytest invocations
@pytest.fixture on an async fixture ignores loop_scope, so mysql_repo
ran on the per-function loop while mysql_test_db_url's engine was bound
to the module loop — triggering 'Future attached to a different loop'.
Add mypy, bandit, vulture, pip-audit as Makefile targets and include
them in test-all. Also enable -n logical on SEQ_FLAGS so live/api/stress
suites run in parallel where async-safe.
After the ingester._sleep alias fix, three tests in test_service_isolation.py
still patched `decnet.web.ingester.asyncio.sleep` (the old global-singleton
path). The ingester now calls `_sleep` directly, so those patches no longer
controlled the ingester's sleep — the worker looped with real asyncio.sleep
and the tests hung indefinitely.
Also: four API lifespan tests had no tarpit_watcher_worker patch, letting the
real tarpit task start. And test_api_survives_db_init_failure patched
`decnet.web.api.asyncio.sleep` (the singleton) instead of the existing
`_retry_sleep` alias.
Fixes:
- patch("decnet.web.ingester._sleep", ...) in the three ingester tests
- add tarpit_watcher_worker patch to all four api lifespan tests
- patch("decnet.web.api._retry_sleep", ...) in db_init_failure test
Two interacting bugs caused asyncio.sleep to be mocked globally,
letting tarpit_watcher_worker spin the event loop on a non-async
mock and accumulate _increment_mock_call records without bound:
1. test_ingester.py patched `decnet.web.ingester.asyncio.sleep` via
the asyncio singleton — any code in the process using asyncio.sleep
(including the tarpit worker) hit the fake_sleep side_effect.
Fix: add `_sleep = asyncio.sleep` alias in ingester.py and patch
`decnet.web.ingester._sleep` instead — scopes the mock to ingester.
2. test_api_startup_guards.py called `_run_lifespan_startup` without
DECNET_CONTRACT_TEST=true, which started the real tarpit task in a
manually-constructed event loop that the tests never cancelled.
Fix: set DECNET_CONTRACT_TEST=true inside _run_lifespan_startup so
the lifespan skips all background workers.
asyncio_default_fixture_loop_scope was 'module', so all async tests in
a module share one event loop. test_lifespan_startup_and_shutdown patched
log_ingestion_worker/log_collector_worker/attacker_profile_worker but not
tarpit_watcher_worker — the real while-True coroutine was created as an
asyncio task on the shared loop and never cancelled. The xdist worker ran
for 4+ hours (confirmed via py-spy + etime=04:48) consuming 15+ GB before
OOM-kill.
Fixes:
- Patch tarpit_watcher_worker in both TestLifespan tests
- Change asyncio_default_fixture_loop_scope to 'function' so each test
gets its own loop; tasks cannot outlive their test
- Add loop_scope='module' to precision_engine which legitimately needs
a module-scoped event loop
Five list columns (greynoise_tags, abuseipdb_categories, threatfox_threat_types,
threatfox_ioc_types, threatfox_malware_families) and four dict columns
(*_raw) are now Column(JSON) with list/dict type annotations and
default_factory=list/dict. Providers return native Python objects; the
application-layer json.dumps/json.loads round-trip and _decode_json_list
helpers are gone. to_intel_event_payload() reads columns directly.
Also caps pytest xdist at -n 4 and excludes tests/api from norecursedirs
to prevent schemathesis workers from OOM-killing the dev loop.
- test_evidence_shape.py: replace broken (command, BehavioralLifter)
pairing with correct (http_fingerprint, HttpFingerprintLifter) case;
expand _LIFTER_CASES to 5-tuples with per-lifter payloads and rule
factories; wire StubRuleStore + _index.install() per lifter; remove
xfail marker — all 4 parametrized cases now pass
- factory.py: add _span() helper gated on _telemetry._ENABLED; wrap
each per-lifter dispatch in _tag_one() that opens a
ttp.lifter.{name} child span per call
- http_fingerprint_lifter.py: add missing name = "http_fingerprint"
- test_tracing.py: replace pytest.fail() stubs in
test_lifter_child_spans_emitted and test_no_pii_canary_in_span_attributes
with real test bodies; remove xfail markers
Removes the E.3.14b xfail marker and writes the test body:
- _StubRepo gains get_attacker_intel_row_by_uuid(uuid) backed by an
optional intel_rows dict; existing tests pass None (no catch-up, no
change to their behaviour).
- The test drives a session.ended event with NO intel.enriched published,
injects an AttackerIntel row into the stub repo, and asserts the
tagger is called with source_kind='intel' carrying the correct payload
fields (abuseipdb_score, greynoise_classification).
- Pins the asymmetry contract: email.received has no catch-up path
(sibling test already green); intel does.
On every attacker.session.ended event, the TTP worker now reads the
persisted AttackerIntel row (if any) and synthesizes an intel-source
TaggerEvent so intel-derived tags emit even when attacker.intel.enriched
was dropped or arrived before the worker started.
Key changes:
- AttackerIntel.to_intel_event_payload() — single source of truth for
the intel-row → lifter payload projection; shared by future callers
without importing decnet.intel.* (no-SPOF contract preserved).
- BaseRepository.get_attacker_intel_row_by_uuid() — returns the live
SQLModel instance so the catch-up path can call to_intel_event_payload().
- _build_intel_catchup_event() in ttp/worker.py — looks up the intel row,
builds the TaggerEvent, returns None on absent row (silence, not error).
- _process_event() extended: appends the catch-up event to tagger_events
when topic contains "session.ended". Deterministic source_id keeps
compute_tag_uuid idempotent across replays; INSERT OR IGNORE deduplicates
against any prior attacker.intel.enriched path.
DummyRepo stub + coverage call added per feedback_run_base_repo_test.md.
Replace pytest.fail() stub with actual test body: constructs IntelLifter
with R0054, feeds score=30 payload, asserts confidence=0.21 (0.70×0.30)
which is below CONFIDENCE_FLOOR. xfail marker removed.
Corrects docstring: R0054 T1110 base_conf=0.70, not 0.85 as originally written.
- TolerantTagger.tag validates evidence keys against EVIDENCE_SCHEMA TypedDicts;
TypeError (programmer error) propagates instead of being swallowed
- IntelEvidence and EmailEvidence expanded from stubs to full per-provider
key sets (total=False); IntelEvidence old stub fields replaced wholesale
- EVIDENCE_SCHEMA map added to models/ttp.py and imported by base.py
- TTPTag __table_args__ gains confidence [0,1] CheckConstraint (DB-enforced)
- xfail removed from test_confidence_outside_range_rejected_at_insert and
test_evidence_shape_violation_propagates_as_typeerror — both now pass
- TypeError removed from _SWALLOWED_EXCS fuzz list; test_intel_evidence_keys
updated to assert the real provider key set
import decnet.cli as _decnet_cli at module level guarantees the app singleton is
built in master mode before any test can set DECNET_MODE=agent. Without this,
test_defence_in_depth_direct_call_fails_in_agent_mode triggered a fresh import
of decnet.cli with DECNET_MODE=agent active, which stripped master-only commands
and wrote the stripped module to sys.modules[decnet].cli — a parent-attribute
corruption that no sys.modules dict restore can fix.
- SSH schema: add user + user_password fields (service extended post-test)
- TopologySummary: repo.get_topology() returns model now, not raw dict
- health live: tarpit_watcher added to get_background_tasks(), add to expected set
Four-part fix for the collection bottleneck that was blocking the dev loop:
1. Lazy mitreattack.stix20 import in attack_stix.py — deferred to first
_load() call (TYPE_CHECKING guard at top level)
2. Lazy misp_stix_converter import in both MISP export routers — moved
from module level into the route handler body
3. Lazy attack_catalog / attack_stix in ttp.py repo mixin — thin wrapper
functions so the import chain never fires at module load time
4. tests/api/conftest.py — `from decnet.web.api import app` moved inside
the `client()` fixture; `pytest_ignore_collect` broadened to skip all
test_schemathesis*.py variants (not just test_schemathesis.py), which
were launching a subprocess server at module-import time
5. pyproject.toml — `norecursedirs` for tests/live, tests/stress,
tests/service_testing, tests/docker, tests/perf so these directories
are never entered; `-m` filter removed from addopts (now redundant);
`--dist loadscope` → `--dist load` to unblock workers immediately
6. behave_core / behave_shell rename — BEHAVE packages dropped the
`decnet_` prefix; reinstalled editable installs and updated all 14
import sites across profiler, ttp, bus, and correlation modules
host:port in remote_addr was creating a distinct Attacker row per TCP
connection instead of per IP. Split on the last ':' in parse_rfc5424;
preserve the port as fields['remote_port'] so repeated source ports are
retained as fingerprint signal in bounty payloads.
quic-go v0.59.0 (shipped with Caddy v2.11.2) removed quic.Connection as
a public interface and quic-go/logging as a public package, breaking
H3App's connection-wrapping approach.
Resolution:
- Remove H3App (h3app.go) entirely; Caddy handles h3 natively when h3
is in the protocols list.
- Rewrite h3conn.go to keep only tryParseH3ControlStream + varint/name
utilities (tested, useful for future stream-level tapping if the API
ever re-exposes it).
- FPHandler.ServeHTTP: for h3 requests, type-assert ResponseWriter to
http3.Settingser (the public interface exposed by quic-go/http3 v0.59),
read the peer's Settings after ReceivedSettings channel closes, emit
h3_settings fp record.
- https/entrypoint.sh: include h3 in CADDY_PROTOCOLS (Caddy now owns
UDP/443); remove DECNET_H3_GLOBAL block.
- Update go.mod/go.sum to caddy v2.11.2 + quic-go v0.59.0.
- Update test_https_compose_h3_app.py to expect h3 in protocols when
http/3 is selected, and assert decnet_h3 block is absent.
- All Go tests (9) and Python tests (15) remain green.
- Renames caddy.listeners.decnet_h2fp → decnet_fp; adds h1 raw-byte
header capture (plainTappingConn) and h2 continuous HPACK decode loop
(parseH2HeadersLoop) so headers_ordered reflects actual wire order, not
Go map iteration order.
- Adds H3App Caddy module (decnet_h3) that owns UDP/443 via quic-go,
wraps accepted QUIC connections with h3SettingsTappingConn to intercept
the h3 control stream and extract RFC 9114 SETTINGS in wire order.
- Wires access_log emission from FPHandler.ServeHTTP via responseCapture.
- Updates syslog_bridge.py (canonical + per-service copies) with inline
_compute_ja4h and new fp socket record branches: http_request_headers,
h3_settings, access_log.
- Fixes ingester proto field alias (bridge emits 'proto', ingester expected
'protocol') and exposes _process_fingerprint_bounties test alias.
- Go tests: h1/h2/h3 golden-byte tests all green; h3_tracer_test covers
varint parser, GREASE detection, truncated-stream safety.
- Python tests: 15/15 green across bridge JA4H hash parity, ingester
compat (old + new event shapes), and Caddyfile h3 template assertions.
[decky.https] relied on ini_loader prefix-matching to propagate config
to decky-03/04/05 — silent and fragile. Now emits [decky-03.https],
[decky-04.https], [decky-05.https] explicitly so the INI is self-evident
and doesn't depend on pattern matching side-effects.
networks.list() returns bare objects — Containers is always empty
without a reload(). The active-endpoint guard from the prior commit
never fired because it was checking a stale empty dict.
Docker refuses network removal (403) when containers hold endpoints.
The old IPAM-drift path tried to disconnect+remove even with live
containers — disconnect silently failed, remove raised APIError.
Since DECNET assigns IPs explicitly in compose (never via Docker's
auto-assign pool), an ip_range mismatch on an existing same-driver
network is harmless. Bail out early and attach to the existing network
whenever Containers is non-empty.
Swap Werkzeug for Caddy as the protocol layer for http and https decoy
services. Flask keeps owning app logic (fake_app, custom_body, headers,
syslog) on 127.0.0.1:8080; Caddy terminates h1/h2/h2c/h3 on the wire
with real-world TLS/QUIC fingerprints.
- Add `multi_enum` FieldType to ServiceConfigField + _coerce
- Add `http_versions` field to HTTPService (h1/h2c) and HTTPSService
(h1/h2/h3); selecting h3 emits UDP/443 port mapping in compose
- Rewrite both Dockerfiles with multi-stage Caddy binary copy +
setcap for port binding as the logrelay user
- Entrypoints parse HTTP_VERSIONS JSON, render a Caddyfile, start
Flask in background, wait for it, then exec Caddy
- https/server.py drops direct TLS handling; Caddy owns the cert
- Add ProxyFix to both server.py so Flask sees real attacker IPs
- Frontend: multi_enum checkbox-group renderer in ServiceConfigFields;
FormValue union extended to string[]; compactPayload skips []
- Fix stale test_smtp_relay_schema_matches_smtp: relay schema is a
superset of smtp, not equal; update assertions accordingly
Remaining files from the fingerprint-bounties + characterizes-SRO commit:
misp_export, repository, bounties mixin, all 4 router endpoints, and test suite
updates. Prerequisite: previous commit added _extract_fingerprint_bounty_data
and the stix_export changes.
Wire fingerprint bounties (JARM hashes, HTTP header quirks) from the bounties
table into the DecnetActorFingerprintExt.protocol_fingerprints group so the
sniffer/profiler-captured HTTP fingerprinting data surfaces in every STIX export.
Add a stix2.Relationship(relationship_type="characterizes") SRO linking each
x-decnet-behave-profile SDO back to its ThreatActor so graph-traversal tools
can follow the edge without relying on the bare x_decnet_behave_profile_ref
custom string property alone.
New repo surface:
- get_fingerprint_bounties_by_ip(ip) -> list[dict]
- get_all_fingerprint_bounties_for_export() -> dict[str, list[dict]]
All 4 export endpoints (per-attacker + fleet, STIX + MISP) extended with the
new gather slot. 50/50 tests green, mypy clean.
Adds GET /api/v1/attackers/{uuid}/export/misp and
GET /api/v1/attackers/export/misp backed by misp_export.py, which
converts existing STIX bundles to MISP events via misp-stix
ExternalSTIX2toMISPParser. Fleet endpoint emits {response:[...]}
collection (one event per attacker). Frontend: STIX/MISP buttons on
AttackerDetail header and Attackers list. 13 new tests green.
GET /api/v1/attackers/{uuid}/export/stix returns a self-contained STIX
2.1 bundle: ip observation, threat-actor, ATT&CK attack-patterns with
canonical MITRE IDs, uses relationships, per-tag sightings, file SCOs
for artifacts, domain-name SCOs for SMTP targets, and a provider intel
note. Attack-pattern SDOs carry the MITRE bundle IDs so consumers
deduplicating against the public ATT&CK bundle get exact matches.
Every technique_id in TechniqueBar and TTPInspector now links to its
canonical attack.mitre.org page. The inspector drawer gains a GROUPS
subpanel that lazy-fetches the new /ttp/techniques/{id}/groups endpoint
and renders each MITRE-tracked intrusion-set with deeplink and aliases.
Centralizes TTP row interfaces into src/types/ttp.ts and API wrappers
into src/utils/ttpApi.ts to give the new GroupRef type a clean home and
avoid a third inline fetch declaration.
Surfaces the intrusion-set reverse index from the loaded ATT&CK
bundle: given a technique, returns the list of groups MITRE has
documented as using it. Read-only — explicitly NOT an attribution
claim about a DECNET attacker. The frontend pulls this lazily when
the operator expands a technique panel; payload-size cost on every
TTPTagDetailRow makes embedding wasteful for techniques with 50+
documented groups.
- decnet/web/router/ttp/api_get_groups_for_technique.py exposes
GET /api/v1/ttp/techniques/{technique_id}/groups, response_model
list[GroupRef]. Same JWT-viewer auth gating as the rest of the
TTP router. 404 when the technique_id doesn't resolve in the
bundle.
- Sub-techniques are queried directly (no auto-union with parent)
to match ATT&CK Navigator semantics; callers that want a broader
view query the parent themselves.
- tests/ttp/test_groups_for_technique.py covers happy path, 404,
sub-technique attribution independence, empty-list-on-zero-groups,
and that responses include mitre_url + aliases.
- tests/web/test_api_attackers.py: fix pre-existing fixture drift
introduced by a2a61b63 — three TestGetAttackerDetail cases were
missing AsyncMock for repo.latest_observation_per_primitive,
causing TypeError on await of MagicMock. The new groups endpoint
doesn't share code with attacker_detail; this is a drive-by fix
surfaced by the same suite run.
Phase 2 attached mitre_url to intel-emitted tags' evidence JSON;
Phase 3 promotes it to a real column populated for *every* tag —
intel, credential, behavioral, canary, identity, email, rule-engine —
from one source. Pre-v1, so the SQLModel field is added directly
without an Alembic migration.
- TTPTag gains mitre_url: Optional[str] (not indexed — derived
deeplink, not a query target; technique_id is already indexed).
- _emit.py and rule_engine._evaluate_rules both populate mitre_url
via attack_stix.mitre_url_for(sub_technique_id or technique_id).
Sub-technique URL when present, else parent. The two construction
sites stay separate because the rule_engine path carries per-emit
span instrumentation that emit_tags() can't preserve without
threading a span object through; minimal-change beats forced
refactor here.
- intel_lifter strips mitre_url from evidence_extra in all four
decision functions. The column is canonical now; duplicating in
the JSON column would drift when the bundle moves. The unused
TechniqueEmission import + tracking dicts removed too.
- IdentityTechniqueRow / TechniqueRollupRow / TTPTagDetailRow /
CampaignTechniqueRow gain mitre_url: Optional[str].
- sqlmodel_repo/ttp.py:_mitre_url_for added; the 5 row-builder sites
pass mitre_url=_mitre_url_for(sub_technique_id or technique_id)
alongside the existing technique_name resolution.
- api_get_tag_details.py needs no change — list_tags_by_scope_and
_technique already returns model_dump() rows that flow the new
column through **row spread to TTPTagDetailRow.
- tests/ttp/test_emit_attaches_mitre_url.py covers both construction
paths (top-level, sub-tech, unknown, multi-emit) and a regression
test that intel_lifter evidence dicts no longer contain mitre_url.
Suite is now 51 files / 259 tests, 25.68% lines / 21.43% branches.
Floor: lines 24->25, functions 21->22, branches 19->21,
statements 23->24. Inspector/index.tsx ends at 172 LOC, the only
other > 250 LOC file in MazeNET/ is NodeInspector (362) — the
node branch was the bulk of the original 606 LOC and its 7
add-service / tarpit form states stay co-located there.
Inspector.tsx (606 LOC) splits into Inspector/{NetInspector,
NodeInspector, EdgeInspector, ServiceInspector, index}.tsx plus
types.ts. The dispatcher (index.tsx) owns the title bar, the empty
state, the activeNetIds derivation, the pending-diff block, and the
topology-status block; each per-type panel takes only the props it
needs. NodeInspector keeps the 7 useStates for the add-service /
tarpit forms since they are node-only.
10 new dispatcher-level tests cover empty / node / net / edge /
service / observed-entity / internet-net / live-ops gating /
tarpit-controls / pending-diff. Selection type re-exported from
Inspector/index.tsx so MazeNET.tsx, Canvas.tsx, and
useMazeContextMenu.tsx keep their existing import path.
Two reusable bundle-derived lookups that the next two commits build
on:
- mitre_url_for(tid) returns the canonical attack.mitre.org URL by
reading external_references on the cached attack-pattern. Backed
by the existing lru-cached _attack_pattern_by_id so per-call cost
is constant. Handles top-level techniques and sub-techniques
(T1059.004 -> .../techniques/T1059/004).
- GroupRef + groups_using_technique(tid) surface the intrusion-set
reverse index from the loaded bundle: given a technique, return
the MITRE-tracked groups documented as using it. Sorted by
group_id for deterministic responses; lru-cached. Sub-technique
semantics match ATT&CK Navigator (do NOT auto-union with parent).
- decnet/ttp/data/intel_loader._mitre_url_for collapses to a thin
re-export of attack_stix.mitre_url_for; the loader keeps mitre_url
on TechniqueEmission for the eventual STIX export.
- tests/ttp/test_attack_url.py covers both helpers: top-level + sub
URLs, unknown -> None / (), GroupRef immutability + hashability,
deterministic ordering, sub-technique distinct from parent.
Drop unused icon/api/useEffect/Tag imports left behind by the
fingerprint, behaviour, and IntelPanel extractions. AttackerDetail.tsx
ends at 450 LOC across Phase 10 (down from 1652 / 73% reduction).
Coverage floor: lines 23->24, functions 20->21, branches 17->19,
statements 22->23.
Move IntelPanel + IntelRow type + ProviderRow + VERDICT_TONE/fmtTs
helpers into AttackerDetail/IntelPanel/. AttackerDetail.tsx drops
from 680 to 449 LOC. New IntelPanel.test.tsx covers the loading,
absent (404), error (500), and ok states with MSW handlers.
Move BehaviouralPrimitivesPanel + 8 sub-components (BehaviorHeadline,
BeaconBlock, DetectedToolsBlock, TcpStackBlock, TimingStatsBlock,
PhaseSequenceBlock, AttributionBadge, KeyValueRow, StatBlock) plus
the OS_LABELS / BEHAVIOR_LABELS / TOOL_LABELS / BEHAVIOUR_DOMAIN_*
lookup tables and fmtOpt/fmtSecs into AttackerDetail/behaviour/.
AttackerDetail.tsx drops from 1220 to 680 LOC; existing
behaviour_panel test moves to behaviour/BehaviouralPrimitivesPanel.test.tsx
and now imports from the canonical location. The shell still
re-exports BehaviouralPrimitivesPanel for source compatibility.
Move 12 Fp* components, FingerprintGroup, getPayload, seqClassColor,
HashRow, fpType lookups, and UA color tables into
AttackerDetail/fingerprints/. AttackerDetail.tsx drops from 1652
to 1220 LOC; the orchestrator now imports the same helpers it used
to define inline. 10 new tests covering UA / HTTP-quirks / resumption
/ certificate / spoofed-source / TCP-stack / dispatch fallback.
The four provider→technique tables (AbuseIPDB cat→techniques,
GreyNoise tag→techniques, ThreatFox threat_type→techniques, plus
the Feodo binary-listed signal) used to live as Final[dict] constants
in intel_lifter.py. Two real problems with that:
1. Drift between rules/ttp/R0054.yaml..R0058.yaml (which declare
the full slate per provider) and the Python dicts (which decide
which slate-member fires per signal). The v2 audit comment in
intel_lifter.py documented that they had silently drifted.
2. No ATT&CK provenance on emissions — the loaded STIX bundle has
rich external_references (canonical attack.mitre.org URLs) that
never surfaced because the lifter had no path back to them.
Mappings now live as YAML at decnet/ttp/data/intel/{provider}.yaml,
validated at load against the loaded ATT&CK bundle, with each entry
enriched by attack_stix._attack_pattern_by_id to attach the canonical
MITRE URL to every emission.
- decnet/ttp/data/intel_loader.py: pydantic-validated schema +
ProviderMapping/Signal/TechniqueEmission frozen dataclasses +
load_provider_mapping(provider) lru-cached.
- Per-technique high_score_threshold inlined into YAML
(collapses the separate _ABUSEIPDB_HIGH_SCORE_GATED dict).
- external_reference field follows the STIX 2.1 external-reference
shape (source_name + url + optional external_id) so the future
STIX/MISP exporter is a direct translation.
- intel_lifter.py: dicts deleted, decision functions read from
ProviderMapping accessors. Decision-flow constants (T1071/T1595
bare-classification fallbacks in _greynoise_decisions) stay in
code — they're not table rows.
- Each emit slot's evidence_extra now carries mitre_url for any
technique resolved in the bundle (every one in practice).
- tests/ttp/test_intel_mappings.py: snapshot equivalence vs the
legacy dicts, high-score gate behavior, every-signal-has-an-
external-reference, every-emission-has-a-mitre-url, negative
paths (unknown technique_id raises AttackBundleError, mismatched
provider field rejected, dir listing matches expected providers).
The YAML schema + mitre_url enrichment lays groundwork for the
future STIX exporter; this commit does NOT build that exporter.
MITRE's ATT&CK Terms of Use require reproducing their copyright +
license alongside any cached copy of ATT&CK data. Today we ship the
bundle but not the license — this commit closes that compliance gap.
- attack_version.py pins ATTACK_LICENSE_URL +
ATTACK_LICENSE_SHA256 + ATTACK_LICENSE_FILENAME, sourced from the
same attack-stix-data repo as the bundle.
- attack_stix.py:_fetch_license downloads LICENSE.txt next to the
bundle. License sha mismatch is logged + refreshed (license text
gets occasional formatting tweaks; not a security event), unlike
the bundle which stays fail-closed.
- _ensure_license is the compliance ratchet: resolve_bundle_path
refuses to return without LICENSE.txt on disk. Override-mode
(DECNET_ATTACK_BUNDLE) checks for a sibling LICENSE.txt first,
then DECNET_ATTACK_LICENSE, then the cache dir.
- python -m decnet.ttp.attack_stix license prints the cached license
to stdout for operator audit.
- loaded_license_path() exposes the active license path read-only.
- tests/ttp/test_attack_license.py covers happy paths (sibling +
explicit env), refusal when DECNET_ATTACK_LICENSE points at a
missing file, the CLI subcommand, and the pinned-sha shape.
Credentials.tsx: 487 -> 231 LOC. Page now composes CredsTable +
ReuseTable + useCredentials hook; URL-derived state (tab, query,
service, page) and selection/sort UI are the only concerns left
in the shell.
SwarmHosts.tsx: 513 -> 161 LOC. Page now composes EnrollmentWizard
+ useSwarmHosts hook; only the arm/confirm UI affordance and the
busy-set tracking remain in the shell.
Webhooks.tsx: 642 -> 387 LOC. Page now composes FormRow + SecretModal
+ useWebhooks hook; toast policy is the only UI concern left in the
shell. Multi-select delete uses the hook's reload internally.
Drift between the technique/tactic IDs hardcoded in the lifters and
what the loaded ATT&CK STIX bundle actually contains is silent in the
status quo: a renamed-or-retired technique just stops being tagged.
Every emission point now has an explicit validator that asserts its
IDs resolve in the loaded bundle, called once at TTP-worker boot.
- intel_lifter.all_emitted_technique_ids() collects every technique
the four provider tables (AbuseIPDB / GreyNoise / Feodo / ThreatFox)
plus the decision-flow constants in _greynoise_decisions and
_feodo_decisions can emit. validate_against_attack_bundle() runs it
through attack_stix.assert_known_technique_ids().
- ukc.validate_against_attack_bundle() asserts every key in
ATTACK_TACTIC_TO_UKC resolves, with TA0100..TA0106 documented as
_NON_ENTERPRISE_TACTICS (lives in the ICS bundle, not the
enterprise bundle DECNET loads).
- decnet/ttp/worker.py:run_ttp_worker_loop calls both validators
before subscribing to the bus. A bundle-vs-code mismatch refuses
to start the worker rather than silently mistagging events.
- tests/ttp/test_attack_bundle_validation.py covers the happy path
for both validators, the negative path (injected bogus tactic ID
raises AttackBundleError), the ICS exemption, and the lone T1078
reference in credential_lifter.
Replace the hand-maintained TECHNIQUE_NAMES dict (pinned to v15.1) with
a runtime loader that reads the official enterprise-attack-N.json STIX
bundle. Version bumps now require only updating attack_version.py;
sub-technique parents, tactic IDs, and kill-chain phases all come from
MITRE's published data.
- decnet/ttp/attack_version.py pins version 19.0 + sha256 + URL
- decnet/ttp/attack_stix.py is the lazy STIX loader. Resolution order:
DECNET_ATTACK_BUNDLE env -> ~/.cache/decnet/attack/ -> fetch from
the pinned MITRE GitHub URL. SHA-256 verified before parse;
mismatch fails closed.
- decnet/ttp/attack_catalog.py collapses to a shim re-exporting
technique_name() so the ~9 router/repo call sites don't churn.
- python -m decnet.ttp.attack_stix fetch warms the cache and can
print sha256 for version-bump workflows.
- test_attack_catalog.py now asserts every rule-emitted ID resolves
in the loaded bundle (same contract, real source) and exercises
the SHA-256-mismatch fail-closed path.
PersonaGeneration.tsx: 875 -> 357 LOC. Page now composes the data
hook + PersonaCard + PersonaEditor; bulk-import helpers stay in
helpers.ts; toast policy is the only UI concern left in the shell.
Final integration step. The MazeNET page shell is now a thinner
composition of the existing module-level hooks (useMazeApi,
useMazeInteraction, useTopologyEditor, useTopologyStream,
useLayoutPersistor) PLUS the three new ones from this phase
(useFullscreenMode, useTopologyData, useMazeContextMenu).
- MazeNET.tsx: 980 -> 715 LOC. The fullscreen + body-class
effects, the topology hydrate / SSE stream / deploy /
flashErr plumbing, and the four context-menu builders are
all gone from the shell.
- Page still owns the per-operation editor callbacks
(removeNet/Node/Edge, duplicateNode, addServiceToNode, etc.)
because they need direct access to setNodes/setEdges/setNets
for optimistic patches alongside their REST calls — those
setters are exposed by useTopologyData for that reason.
Coverage floor bumped after the phase:
lines 17 -> 19
functions 15 -> 17
branches 13 -> 14
statements 16 -> 18
Phase 5 final scoreboard: 37 test files, 172 tests, all green.
Lift the context-menu builder out of the page shell. The hook
owns ctxMenu open/close state and exposes one builder per
surface (node / net / edge / canvas); the actual operations come
in via callbacks so the page keeps its optimistic-patch logic
unchanged.
- New MazeNET/useMazeContextMenu.tsx
- useMazeContextMenu.test.ts covers menu lifecycle (open/close),
node-menu items, observed-entity locking, internet-net
delete-disabled, canvas-menu Add subnet/DMZ items, and the
edge-menu Remove invocation.
- Wiring into MazeNET.tsx lands next.
Lift the canvas data plane off the page shell. The hook owns:
GET /topologies/:id (hydrates nets/nodes/edges + meta)
GET services + archetypes (catalogs, with bundled fallback)
POST /topologies/:id/deploy
/topologies/:id/events SSE (open only when active/degraded)
flashErr() banner timer (auto-clears actionErr after 4s)
State setters for nets / nodes / edges are returned so the
per-operation callbacks living in the page can optimistically
patch local state alongside their REST calls (matches the
existing pattern; wholesale lift would mean dragging every
mutation along too).
- New MazeNET/useTopologyData.ts
- useTopologyData.test.ts covers hydrate, loadErr surfacing,
streamEnabled gating on active/degraded, onDeploy success +
error paths, and the flashErr 4s auto-clear with fake timers.
- Wiring into MazeNET.tsx lands in the next commit.
Lift the four fullscreen-related side-effects off the page shell.
The hook owns:
1. body class toggle so page CSS can hide its chrome
2. browser fullscreen API request/exit (failures ignored)
3. fullscreenchange listener so F11/Esc from outside our button
keeps internal state in sync
4. Esc keystroke handler
Returns { fullscreen, setFullscreen, toggle }.
- New MazeNET/useFullscreenMode.ts
- useFullscreenMode.test.ts (jsdom) covers initial toggle, body
class lifecycle, Esc-to-exit, and unmount cleanup.
- MazeNET.tsx loses ~30 LOC of inline state + effects.
Final integration. The page shell is now a thin composition of
useConfig + the previously-extracted children:
- Config.tsx: 989 -> 131 LOC. Page owns only the activeTab state
(and the "drop the users tab if the server didn't send users"
effect). Every form lives inside its tab; toast wiring lives
in AppearanceTab; window.alert calls live inside UsersTab.
- Tabs receive their `onSave* / onAddUser / ...` callbacks
directly from the hook — no intermediate wrapper handlers.
Coverage floor bumped after the split:
lines 14 -> 17
functions 13 -> 15
branches 11 -> 13
statements 13 -> 16
Phase 4 final scoreboard: 34 test files, 156 tests, all green.
APPEARANCE panel — accent-color picker — into its own tab. State
is local since no other tab cares about the value; localStorage
persistence + the document.documentElement[data-accent] mirror
move along with it.
- New Config/tabs/AppearanceTab.tsx
- AppearanceTab.test.tsx covers the matrix default, reading the
saved accent from localStorage on mount, and the click-to-flip
flow writing both localStorage and the html data-accent attr.
GLOBAL VALUES panel + the developer-mode-gated DANGER ZONE
(reinit) into one tab file. Two stacked panels because they're
the two pieces of UX you ever see together on the globals tab;
splitting them into separate components would force the page
shell to re-pick the gating predicate.
- New Config/tabs/GlobalsTab.tsx (mutation-interval + DangerZone
inline, since DangerZone is reinit-specific and won't be reused)
- GlobalsTab.test.tsx covers interval-format validation, the
DANGER ZONE gating on developer_mode, the two-step reinit
confirm flow, the totals chip ("PURGED: N logs, N bounties,
N attacker profiles") on success, and viewer-mode rendering.
USER MANAGEMENT panel into its own tab. Owns the per-row UI
state (delete-confirm, reset-password popup) plus the add-user
form state; mutations come in via prop. Errors on per-row
operations stay on window.alert (matches existing behavior); the
add form uses the inline FormMsg chip.
- New Config/tabs/UsersTab.tsx
- UsersTab.test.tsx covers row rendering with the must-change
badge, the two-step delete confirm flow, the add-user submit
payload (trimmed username + selected role), and the success
chip after a successful add.
DEPLOYMENT LIMITS panel into its own tab file. Owns the input
state, preset-button shortcuts, and the inline FormMsg chip; the
hook mutation is passed in via prop so this component is fully
reusable as a presentation-only piece.
- New Config/tabs/LimitsTab.tsx
- LimitsTab.test.tsx covers viewer-vs-admin rendering, the
1-500 validation message, and success/error chip display.
Lift the GET /config fetch and every admin-side mutation off the
page shell:
GET /config
PUT /config/deployment-limit
PUT /config/global-mutation-interval
POST /config/users
DELETE /config/users/:uuid
PUT /config/users/:uuid/role
PUT /config/users/:uuid/reset-password
DELETE /config/reinit (returns { logs, bounties, attackers })
Mutations return { ok: true } | { ok: false; reason: string } so
the upcoming tab components can render the inline FormMsg chip
without touching axios error shapes. reinit additionally returns
the deletion totals so the danger-zone confirmation can echo
"PURGED: N logs, N bounties, N attackers".
- New Config/useConfig.ts
- useConfig.test.ts MSW-covers initial load, isAdmin role
surfacing, setDeploymentLimit ok + 400 paths, addUser, deleteUser
refused, and reinit success.
- Wiring into Config.tsx + tab extractions land in follow-up commits.
Verbatim move of the worker-status pollster (~390 LOC) plus its
RealismBadge sidekick into its own file. Owns its own polling +
stop/start/start-all mutations; toast push comes in via prop so
the parent stays the one source of toast tone.
- New Config/WorkersPanel.tsx
- WorkersPanel.test.tsx (MSW) covers worker-row rendering, the
BUS OFFLINE banner, and the error panel on /workers 500.
- Config.tsx loses the inline WorkersPanel + RealismBadge plus
the now-unused icon imports (Square, RefreshCw, Play).
Foundation for the Config split. UserEntry / ConfigData move out
of the page so the upcoming hook + tab extractions can import
without reaching back through Config.tsx. New ConfigTab union and
FormMsg type for the inline success/error chip pattern that
repeats across every admin form on the page.
- New Config/types.ts (UserEntry, ConfigData, ConfigTab, FormMsg)
- Config.tsx loses the inline interfaces and the `as any` cast on
setActiveTab in the tab-switcher.
Final integration step. The page shell is now a thin composition
of useCanaryTokens + the previously-extracted children:
- CanaryTokens.tsx: 1,334 -> 210 LOC. Page owns only the
pure-UI state (tab, search/state/scope filters, modal
visibility, drawer selection, local fileDrops log) and the
thin handlers that translate hook results into confirm/alert
prompts. Initial parallel fetch + deleteBlob mutation moved
to useCanaryTokens in the prior commit.
- Modals plug directly into the hook's optimistic helpers
(prependToken / prependBlob / markTokenRevoked) so the page
doesn't reach into the data shape.
Coverage floor bumped after the split:
lines 11 -> 14
functions 10 -> 13
branches 8 -> 11
statements 11 -> 13
Phase 3 final scoreboard: 28 test files, 131 tests, all green.
Lift the three tab bodies — tokens, blobs, file drops — into
their own files. Each takes plain props (data + the operations
its rows need), so the page shell stops mixing tab markup with
data plumbing.
- New CanaryTokens/TokenListView.tsx (text search + state/scope
filter selectors + flat row grid; visibleTokens memo lives here
now). Exports StateFilter / ScopeFilter union types so the page
can declare its filter useState with the right shape.
- New CanaryTokens/BlobListView.tsx (delete refused while a token
references a blob; ref count badge reuses the disabled button).
- New CanaryTokens/FileDropListView.tsx (CLEAR LIST hidden when
the local log is empty).
- Three companion tests cover empty states, filter behavior,
delete refused-vs-allowed, and the per-tab callback wiring.
Wiring into CanaryTokens.tsx + the hook lands next.
Lift the parallel initial-load fetch and the deleteBlob mutation
off the page shell. Modal-driven optimistic merges (created
token, uploaded blob, drawer-revoked token) flow through narrow
setter helpers so the modals don't have to know how state is
shaped internally.
GET /canary/tokens
GET /canary/blobs (silent 403 -> empty list, viewer-friendly)
GET /deckies
GET /topologies/?status=active
DELETE /canary/blobs/:uuid
deleteBlob returns { ok, reason } so the page can branch the
toast/alert tone without seeing the axios error type. Wiring
into CanaryTokens.tsx lands in the next commit.
- New CanaryTokens/useCanaryTokens.ts
- useCanaryTokens.test.ts MSW-covers happy load, viewer 403 ->
empty blobs, deleteBlob ok + refused-with-detail paths, and the
markTokenRevoked optimistic write.
Verbatim move of the file-drop modal (~310 LOC) and its
localStorage glue (FILEDROP_LS_KEY, FileDropEntry type,
loadFileDrops, saveFileDrops) into one file. The list view that
shows these entries lives in the page; the persistence layer
travels with the writer.
- New CanaryTokens/FileDropModal.tsx (modal + LS helpers + entry type)
- FileDropModal.test.tsx covers loadFileDrops empty / round-trip /
200-row cap / malformed-JSON, plus modal title rendering, the
bypass-warning banner, and CANCEL -> onClose.
- CanaryTokens.tsx loses the inline modal + LS glue plus the
now-unused imports (useRef/X/AlertTriangle/useEscapeKey/
useFocusTrap, plus BTN_PRIMARY/BTN_GHOST/Field that only the
modals consumed).
Verbatim move of the artifact upload modal (~130 LOC) into its
own file. Drop-or-browse picker, server-side-injection warning
banner, and the multipart POST stay unchanged.
- New CanaryTokens/UploadModal.tsx
- UploadModal.test.tsx covers title rendering, empty drop-zone
hint, server-injection warning banner, UPLOAD-disabled-until-
file, and CANCEL -> onClose.
Verbatim move of the canary-token creation modal (~280 LOC) into
its own file. Renamed from CreateModal to CreateTokenModal so the
component name carries scope across the package boundary.
- New CanaryTokens/CreateTokenModal.tsx
- CreateTokenModal.test.tsx covers title rendering, CANCEL ->
onClose, empty-deckies hint, and the Operator-upload mode
switch revealing the no-blobs message. useFocusTrap is
vi.mock'd to avoid jsdom focus shenanigans.
- CanaryTokens.tsx loses the inline modal + its now-unused
imports (KNOWN_GENERATORS, KIND_OPTIONS, GeneratorName).
Foundation for the CanaryTokens split. Types, error/format helpers,
and the inline style + small primitives move out of the page so
the upcoming modal/list extractions can import without reaching
back through CanaryTokens.tsx.
- New CanaryTokens/types.ts (BlobRow, DeckyOption, TopologyOption,
Scope, KNOWN_GENERATORS / GeneratorName, KIND_OPTIONS, STATE_COLOR)
- New CanaryTokens/helpers.ts (extractError, fmt, fmtBytes)
- New CanaryTokens/ui.tsx (INPUT_STYLE, BTN_PRIMARY, BTN_GHOST,
Field, Stat)
- CanaryTokens.tsx loses ~110 LOC of inline definitions; behavior
unchanged.
Phase 2 lands. DeckyFleet.tsx dropped from 1,674 to 274 LOC; the
fleet page is now a thin composition of useDeckyFleet + 6
extracted children (DeckyInspectPanel, IntervalEditor, DeckyCard,
DeployWizard, DeckyFilters, DeckyGridEmpty), each with co-located
tests.
Lock the gain by bumping the threshold floor in vite.config.ts:
lines 7 -> 11
functions 6 -> 10
branches 5 -> 8
statements 7 -> 11
Phase 2 final scoreboard: 21 test files, 98 tests, all green.
Final integration step. The page shell is now a thin composition
of the hook + the previously-extracted children:
- DeckyFleet.tsx: 1,674 -> 274 LOC. Page owns only the
pure-UI state (filter, search, armed-confirm, modal visibility,
selected-card-for-inspect) and the toast-wrapping handlers that
translate hook results into toast tone. Polling, REST plumbing,
role lookup, and archetype catalog all moved to useDeckyFleet
in the prior commit.
- New DeckyFilters.tsx (header pill row + DEPLOY shortcut) +
DeckyGridEmpty.tsx (fleet-empty vs. filter-empty copy).
- DeckyFilters.test.tsx + DeckyGridEmpty.test.tsx cover count
rendering, filter-click callbacks, and admin-gated DEPLOY
visibility.
Two-step teardown arming logic stays in the page (it's pure UI).
Toast tone branching on { ok, reason } from useDeckyFleet
results moves the policy decision out of the data layer.
Lift every read- and write-side data flow off the page shell:
GET /system/deployment-mode (decides which list endpoint to hit)
GET /deckies | /swarm/deckies (mode-switched + shape-normalized)
GET /config (role -> isAdmin)
GET /topologies/archetypes (live catalog with bundled fallback)
POST /deckies/:name/mutate
PUT /deckies/:name/mutate-interval
POST /swarm/hosts/:uuid/teardown
10s polling loop refreshing mode + list
Operations return discriminated results ({ok:true} | {ok:false,
reason:...}) so the page can branch toast tone without seeing the
axios error type. Toasts, arm-confirm, and modal visibility stay
in the consuming page — the hook is pure data.
- New DeckyFleet/useDeckyFleet.ts
- useDeckyFleet.test.ts MSW-covers initial load, swarm-mode shape
normalization, mutate ok/error paths, teardown ok path, and
applyServicesChange optimistic write.
- DeckyFleet.tsx wiring lands in the next commit so the diff stays
reviewable.
Lift the multi-step deploy wizard (~520 LOC) plus its private
INI-builder helpers (PLACEHOLDER_LINES, b64encodeUtf8, buildIni,
PickMode type) into their own file. Verbatim move; the
underscore-prefixed helpers drop the leading underscore now that
they're file-local rather than competing with hoisted parent
constants.
- New DeckyFleet/DeployWizard.tsx
- DeployWizard.test.tsx covers the closed render guard, the
open-at-step-0 archetype list, NEXT-disabled-until-archetype,
and CANCEL -> onClose. ServiceConfigFields is vi.mock'd to a
stub since it pulls schemas via api.get() that are out of
scope for these tests.
- DeckyFleet.tsx loses the wizard plus the now-unused imports
(DEFAULT_SERVICES, Modal, PickIcon, ServiceConfigFields and
its type aliases).
Lift the per-decky tile (~430 LOC) into its own file. Tarpit
controls, live add/remove service flow, and the per-service config
toggle stay inside the card — those are tile-local UI concerns and
only ever rendered from this component anyway.
- New DeckyFleet/DeckyCard.tsx
- DeckyCard.test.tsx covers identity row + services rendering,
admin-gated FORCE MUTATE visibility, the FORCE MUTATE callback,
TEARDOWN -> CONFIRM toggle when armed matches, and card-body
click firing onInspect. AddServiceConfigModal +
ServiceConfigForm are vi.mock'd so we don't need MSW handlers
for their unrelated network fetches.
- DeckyFleet.tsx loses the inline component plus the now-unused
imports it dragged in (Network/PowerOff/RefreshCw/Plus/X icons,
ServiceConfigForm, AddServiceConfigModal, useCallback).
Verbatim move of the per-decky mutation-interval modal (~60 LOC)
into its own file. Saves null when the toggle is off, minutes
otherwise.
- New DeckyFleet/IntervalEditor.tsx
- IntervalEditor.test.tsx covers null-current disabled path,
numeric-current enabled path, and CANCEL not firing onSave.
- src/test/fixtures/decky.ts now derives DeckyFixture from the
canonical Decky type (the fixture's loose swarm shape was
missing host_address/host_status; aligning to Decky catches
that statically).
Lift the right-side inspect drawer (~115 LOC) into its own file.
This is a verbatim move — same JSX, same useEscapeKey + body
overflow lock, same swarm-section gating. Underscore-prefixed
helper calls (_dotFor, _stateColor) drop the leading underscore
since they're now imported from helpers.tsx.
- New DeckyFleet/DeckyInspectPanel.tsx
- DeckyInspectPanel.test.tsx covers identity-row rendering, the
SERVICES chip list, the conditional SWARM block, and the close
button callback.
- DeckyFleet.tsx loses the panel + the now-unused useEscapeKey
import.
Foundation for the DeckyFleet split. Types and helpers move to
their own files so the upcoming subcomponent extractions can
import without reaching back through the parent module.
- New DeckyFleet/types.ts (Decky, SwarmDeckyRaw, SwarmMeta,
Archetype, FilterKey, DeckyStatus). Names exported to match the
pattern set by AttackerDetail/types.ts.
- New DeckyFleet/helpers.tsx (archetypeIcon, PickIcon, dotFor,
hitsFor, stateColor). Underscore-prefixed call sites stay via
import-rename so this commit changes zero behavior.
- DeckyFleet.tsx loses ~110 LOC of inline definitions plus the
now-unused icon imports (Cpu / Database / Globe / Monitor /
Shield / Terminal).
Phase 1 of the UI refactor is in. AttackerDetail dropped from
2,579 LOC inline data + JSX to a 408-LOC shell composed of
extracted sections, each with co-located tests. Lock the gain by
bumping the threshold floor in vite.config.ts:
lines 0 -> 7
functions 0 -> 6
branches 0 -> 5
statements 0 -> 7
Future PRs raise these; never lower. Phase 1 final scoreboard:
9 test files, 45 tests, all green.
Lift STORED MAIL into its own section and pull the mail drawer
selection state along with it. Section signals admin-gating
through the section's own props (mailForbidden), since the data
hook already converts a 403 into that boolean.
- New AttackerDetail/sections/MailLogPanel.tsx
- MailLogPanel.test.tsx covers row rendering, mailForbidden empty
state, no-mail empty state, from_hdr/from_addr/mail_from
fallback, and drawer open/close. MailDrawer vi.mock'd same as
ArtifactDrawer.
- AttackerDetail.tsx loses the mail JSX block, mailItem state,
and now-unused Mail/MailDrawer imports.
Lift CAPTURED ARTIFACTS into its own section, taking the drawer
selection state with it (the parent shell no longer owns
artifact-modal state).
- New AttackerDetail/sections/ArtifactsPanel.tsx
Drawer is rendered as a sibling of the section so its z-index
and focus-trap behavior mirror the original.
- ArtifactsPanel.test.tsx covers row rendering with parsed SD
fields, empty state, missing stored_as (no OPEN button), and
the open/close cycle. ArtifactDrawer is vi.mock'd to a stub
so we don't need MSW handlers for its content fetch.
- AttackerDetail.tsx loses the artifact JSX block, the artifact
state, and now-unused Paperclip/Package/ArtifactDrawer imports.
Lift the COMMANDS collapsible — paginated table with header-bar
prev/next controls — into its own section. The page math
(cmdTotalPages = ceil(total/limit)) and conditional empty state
both live in the section now.
- New AttackerDetail/sections/CommandsViewer.tsx
- CommandsViewer.test.tsx covers title formatting (unfiltered vs.
filtered), empty state, single-page pagination hiding, and
prev/next button behavior
- AttackerDetail.tsx loses the IIFE-wrapped commands JSX block
plus now-unused ChevronLeft/ChevronRight/Terminal imports
Lift the SERVICES TARGETED collapsible — interactive two-tone badge
chips with click-to-filter — into its own section. The selection
state was already lifted into useAttackerDetail in the prior
commits, so the section just consumes serviceFilter /
setServiceFilter as props.
- New AttackerDetail/sections/ServicesTargeted.tsx
- ServicesTargeted.test.tsx covers badge rendering, empty state,
inactive-click-sets-filter, and active-click-clears-filter
- AttackerFixture grows ip_leaks/ip_leaks_total fields so the
TimelineSection rotation test (added in the prior commit) keeps
passing under the new factory shape
Lift the TIMELINE collapsible (timestamps, ASN, reverse DNS,
leaked-IPs row with rotation detection) into its own section.
LeakedIPsRow + the rotation/inline-limit constants come along
since they were only ever used here.
Also moves the shared `Section` collapsible primitive into
AttackerDetail/ui.tsx so the remaining sections can adopt the
template without re-importing through the parent module.
- New AttackerDetail/sections/TimelineSection.tsx (LeakedIPsRow
inline as a private helper)
- AttackerDetail/ui.tsx now exports both Tag and Section
- AttackerDetail.tsx loses LeakedIPsRow, the Section helper, the
Timeline JSX block, and now-unused imports (ChevronUp, ChevronDown,
AttackerData)
- TimelineSection.test.tsx covers timestamps, unknown-origin path,
rotation badge, empty leaks, collapse, and toggle callback
Lift the 5-up counter grid + the conditional scan-vs-interact row
into AttackerStats. The activity row's visibility predicate
collapses into a single boolean inside the section so the parent
no longer encodes UX rules.
- New AttackerDetail/sections/AttackerStats.tsx
- AttackerStats.test.tsx covers all-five counters, activity present,
activity empty, and service_activity undefined paths.
Lift the header (IP, country tag, traversal badge, identity badge)
into its own section component. Tag helper moves to a shared
AttackerDetail/ui.tsx so future sections can reuse it without
re-importing through AttackerDetail.tsx.
- New AttackerDetail/sections/AttackerHeader.tsx (~50 LOC)
- New AttackerDetail/ui.tsx for shared presentational helpers
- AttackerDetail.tsx imports both; local Tag definition deleted
- AttackerHeader.test.tsx covers country present/absent,
TRAVERSAL badge, IDENTITY click-through, identity null path
The AttackerDetail page body owned all 7 REST fetches plus 2 SSE
streams inline as 200+ lines of useEffect plumbing. Lift them into
a single hook so section components extracted in follow-up commits
consume typed values, not setState pairs.
- New ./AttackerDetail/types.ts holds the canonical AttackerData,
BehaviouralObservation, AttributionPrimitiveState plus newly-named
ArtifactLog / SessionLog / SmtpTargetRow / MailLog / CommandRow
(previously inline anonymous types).
- New ./AttackerDetail/useAttackerDetail.ts owns:
* GET /attackers/:id (404 -> ATTACKER NOT FOUND)
* GET /attackers/:id/attribution (silent-tolerant)
* GET /attackers/:id/commands paged with 422 alert preserved
* GET /attackers/:id/{artifacts,smtp-targets,mail,transcripts}
(mail surfaces a 403 boolean for the admin-gated viewer)
* useAttackerStream + useIdentityStream subscriptions, including
the live attribution-state-changed merge.
- AttackerDetail.tsx re-exports BehaviouralObservation /
AttributionPrimitiveState so AttackerDetail.behaviour_panel.test
and any future external importer keeps working unchanged.
- New useAttackerDetail.test.ts covers loading -> success, 404,
paged commands offset, serviceFilter resets cmdPage, and mail 403
via MSW handlers (the SSE hooks are vi.mock'd; jsdom can't host
EventSource).
No behavior change for the rendered page; all 37 tests green.
Phase 0 of the decnet_web refactor: stand up an MSW server, fixtures,
and a router-aware render helper so the upcoming god-component splits
(AttackerDetail first) can land with same-commit test coverage.
- msw devDep + setupServer wired into src/test/setup.ts
- src/test/server.ts re-exports server, http, HttpResponse, apiUrl()
- src/test/fixtures/{attacker,decky,canary,topology}.ts factories
- src/test/renderWithRouter.tsx wraps MemoryRouter + ToastProvider
- baseline coverage thresholds (0%) in vite.config.ts; raise per PR
- coverage/ added to decnet_web/.gitignore
Existing Orchestrator/AttackerDetail/ThemeLab tests stay on vi.mock
and continue to pass; new tests use MSW.
The global button:hover rule in index.css forces color: var(--bg)
+ matrix-glow on the lucide icon's currentColor stroke, making
the sun/moon icon disappear into the toggle button's tinted
background on hover. Pin color: var(--accent) and box-shadow:
none on .theme-toggle-btn:hover so the icon stays in its base
colour and the button doesn't pick up the wider button-hover
halo.
Even with fill: 'both', the new pseudo paints once at its default
style (no clip-path = full size) before the JS animation
registers — the brief open flash that survived the previous fix.
Pre-publish click coords as --reveal-x / --reveal-y on <html>
before calling startViewTransition. The static CSS rule on
::view-transition-new(root) now sets clip-path: circle(0px at
var(--reveal-x) var(--reveal-y)) as the pseudo's default, so
the very first paint is already fully clipped. The animation
then grows the circle outward from there.
ANTI prefers the new theme growing outward from the click point
(visually clearer cause-and-effect than the old theme burning
away). The original outward implementation flashed at the start
because the new pseudo defaulted to its computed style (no
clip-path = fully visible) for one frame before the JS animation
registered.
Switching the animation's fill from 'forwards' to 'both' enforces
the start keyframe (circle(0) at click point) before the first
paint, in addition to pinning the end keyframe through pseudo
teardown. New layer is invisible until the animation begins,
fully visible until cleanup. No flash either end.
Without fill: 'forwards' the clip-path keyframes release at
animation end and the pseudo reverts to its computed style
(no clip-path), so the old layer flashes back at full size for
a frame before View Transitions tears the pseudo-elements down.
Pinning the final keyframe with fill-forwards keeps the old
layer fully clipped through to teardown.
Growing the NEW theme layer from circle(0) outward leaves a
one-frame gap where the new pseudo is fully opaque at full size
(the default state) before the clip-path animation registers.
Result: a flash of the destination theme right before the
reveal starts.
Inverted the layering and animation direction:
- NEW theme snapshot sits on the bottom (z-index 0), static
- OLD theme snapshot sits on top (z-index 1), shrinks via
clip-path from circle(N) at click point down to circle(0)
The new layer is now hidden behind the old one until the old
shrinks away — no flash possible because the new layer was
never visible before the animation. Same 520ms duration, same
ease curve, same direction-of-travel from the user's POV
(circle expanding from cursor).
User-facing theme toggle ships now that the design system has
been audited end-to-end. A Sun/Moon button lives between the
threat indicator and the SYSTEM status pill in the topbar — same
slim 28x28 voice as the rest of the topbar controls, no chrome
shouting at the user.
Click coords drive a View Transitions API circle clip-path that
grows from the cursor to the farthest viewport corner over 520ms
with the project's standard --ease curve. Browsers without
startViewTransition (older Firefox, Safari < 18) fall through to
an unanimated swap — the hook returns instantly in that case.
Persistence is two-tier:
- localStorage decnet_theme — the user's saved preference, the
thing the topbar toggle writes. Survives reloads, applies
everywhere.
- sessionStorage decnet_theme_lab — dev-mode lab override (Task
3). Tab-scoped, wins on boot so devs can A/B without nuking
the saved preference.
App.tsx hydrates both on first mount in the right order so the
correct theme is on <html> before the first paint.
useThemeToggle is a small hook in lib/ rather than a Layout-only
helper so the same toggle can be reused later from a settings page
or hotkey.
--dim-color and --danger-color were referenced across drawers and
RemoteUpdates but never defined; --dim-color silently inherited
(defeating its purpose) and --danger-color fell back to literal
#f88 salmon (the 'ugly red' WifiOff icon next to UNREACHABLE
hosts). Added both as aliases in :root: --dim-color = var(--fg-3),
--danger-color = var(--alert).
--fg-2/3/4 alphas in light mode were tuned identical to dark
(0.78/0.55/0.35), but ink-on-cream needs more punch than
matrix-on-black at the same alpha — the deploy preview code
block (.code-block .comment / .key) and every dim caption
rendered too faint. Bumped to 0.88/0.70/0.50.
.maze-net-box.inactive applies opacity 0.42 + grayscale(0.7) for
the 'no traffic' signal. On cream that fades the LAN out of
visibility entirely. Override in light mode keeps the dotted
border as the dim-state cue and bumps opacity to 0.85 so the
header text stays legible.
Credentials drawer code-block labels (printable:, b64:) and a
dozen other violet wash/tint sites still carried bare rgba(238,
130, 238, *) literals — bright magenta in light mode where
--violet has resolved to charcoal-purple #2d1b4e. Mirrors the
prior matrix/alert/warn/info sweeps: by-alpha buckets land on
var(--violet-tint-10) or var(--violet).
Pre-this-commit, ~80 rgba() literals across 24 files were
hardcoding alert-red, warn-amber, info-cyan, panel-dark, and
white-text-with-alpha shades that bypassed the token cascade.
Net effect in light mode: the .eml/SESSREC drawers, AttackerDetail
verdict pills, MazeNET net-box headers, OPEN/REPLAY action
buttons, threat-intel cards, and all the dim 'whitish' overlays
stayed on their dark-mode hex values, producing the unreadable
panels in the screenshots.
Sweep maps each rgba colour family onto the existing token by
alpha bucket — rgba(13,17,23,*) -> var(--panel),
rgba(255,65,65,*) -> var(--alert)/-tint-10,
rgba(255,170,0,*) and rgba(224,160,64,*) -> var(--warn)/-tint-10,
rgba(0,200,255,*) -> var(--info)/-tint-10,
rgba(255,255,255,*) -> var(--fg-N)/var(--matrix-tint-N) by alpha.
VERDICT_TONE in AttackerDetail (MALICIOUS/SUSPICIOUS/BENIGN/
NO SIGNAL) was the worst offender — string literals
'#ff4d4d'/'#ffae42'/'#5fd07a'/rgba(255,255,255,0.4) baked into
inline JS styles. Now resolves at render time via var(--alert)/
var(--warn)/var(--ok)/var(--fg-4).
New tokens in :root:
- --bg-color (alias of --bg) — drawers used this name with
#0d1117 fallback that fired in every browser because nothing
defined --bg-color. Adding the alias makes drawers re-tone.
- --info / --info-tint-10 / --info-tint-30 — REPLAY buttons and
any future neutral-secondary use.
- --ok — semantic alias for 'verified good' (matrix in dark,
emerald in light) so BENIGN pills stay readable across themes.
Login.css left intentionally — pre-auth surface, not themed.
Sweeps four invariant violations that were leaking dark surfaces
into light mode and producing the unreadable / inverted areas:
1. Hardcoded `color: #000` in 14 :hover rules across 11 CSS
files swapped to `color: var(--bg)` — collapses to #000 in
dark mode (no-op), becomes cream in light. Fixes DEPLOY
DECKIES (button hover was rendering charcoal-purple text on
charcoal-purple background).
2. Hardcoded `background: #000` (3 sites) and `#0d1117`
(3 sites) replaced with `var(--bg)` / `var(--panel)`. Fixes
code blocks and modal panels staying dark on cream — the
deploy-wizard preview, topology-creation NAME input, and the
MazeNET canvas backdrop now follow the active theme.
3. `rgba(0,0,0,0.35)` and `rgba(0,0,0,0.5)` input/card
backgrounds (ServiceConfigForm, DeckyFleet .input)
swapped to `var(--panel)`. Fixes per-service config rows
in the deploy wizard rendering as dark slabs.
4. SVG arrow markers in MazeNET Canvas.tsx hardcoded
`fill="#00ff41"` / "#ee82ee" — replaced with currentColor +
style hook so they re-resolve on theme change.
New behaviour: light-mode hovers tint instead of inverting. The
dark-mode rules fully fill bg with --matrix/--violet/--alert and
flip text to --bg; that lands cream-on-near-ink in light mode
and reads as a jarring colour inversion every cursor move. Light
mode now layers a *-tint-10 background and keeps text in its
base colour. Single override block in index.css targets every
scoped `.X-btn`/`.btn`/`button:hover` via :is() + [class*="-btn"]
so we don't have to chase every component file.
37 bare rgba(0, 255, 65, ...) literals across 10 component CSS
files were forcing matrix-green to bleed into light mode no matter
what data-theme=light overrode in :root. They're now mapped onto
existing tokens by alpha bucket (0.025-0.05 -> --matrix-tint-5,
0.08-0.10 -> --matrix-tint-10, 0.18-0.30 -> --matrix-tint-30,
0.4 -> --fg-4, 0.5-0.6 -> --fg-3, 0.7-0.8 -> --fg-2).
Adds --warn (#e0a040), --amber (alias of --warn), --crit
(#e74c3c), and their tint-10 variants to :root, with
ink-friendly light-mode overrides. Sweeps bare #ffaa00 / #e0a040
/ #f59e0b / #ff4d4d / #e74c3c usages in the same files onto the
new tokens.
Files with var(--token, #fallback) patterns left alone — those
were already token-driven and the fallbacks just provide safety.
Login.css and inline TSX hex left for the per-page sweep.
Initial light-theme palette kept --matrix as a darker emerald
and --violet as a darker purple, which washed out badly on
warm cream — auth-helper chips, ACTIVE/PASSIVE/INACTIVE pills,
and CREDS/REUSE tabs all became unreadable because their tint
backgrounds + low-saturation text collapsed to sludge.
Light mode now collapses --matrix and --violet to near-ink
shades (#0d0d0d and #2d1b4e). --alert stays the one
saturated colour — the only element allowed to shout.
Dark mode is untouched; the matrix-vibe identity stays
exclusive to dark.
Also collapses the matrix/violet accent knob in light mode:
data-accent only flavours dark mode now, since two ink
shades are visually identical.
Adds html[data-theme="light"] block to index.css overriding the
core six tokens (bg, matrix, violet, panel, border, alert), the
matrix/violet/alert tints, and the foreground opacity ramp to a
cream-on-ink palette anchored on #dbdad6. Glows are no-op'd —
light mode trades neon haloes for hard 1px borders.
Lab page gets a Dark/Light toggle that flips
html.dataset.theme and persists to sessionStorage
(decnet_theme_lab) — intentionally tab-scoped, not user-facing.
App.tsx hydrates the same key on boot so a tab reload keeps the
dev's chosen theme. The user-facing localStorage toggle ships
later via Config.
Renders every primitive in the design system on the lab page so
theme-token edits can be evaluated against all states at once:
colour swatches with WCAG contrast vs --bg, the full type scale,
buttons (5 variants × default/hover/disabled), badges and status
pills, info/error banners, metric cards, table rows
(default/hover/selected/drop-target), form inputs, drawer panel
sample, and net-box compose states (internet/inactive/selected/
drop-target — independent classes layering, per memory).
Wrapper uses .fleet-root so global .btn/.btn.violet/etc resolve
identically to real pages. Lab-local CSS owns layout only — every
colour comes from index.css tokens.
Adds VITE_DECNET_DEVELOPER build-time gate: when unset, the
isDeveloperMode() helper collapses to a constant false and Vite
tree-shakes both the lazy import and the conditional <Route> out
of the prod bundle.
ThemeLab is currently a header stub; subsequent tasks fill it
with the design-system primitive zoo plus a Dark/Light toggle
for live token tuning. Route is intentionally absent from
ROUTE_LABELS / sidebar — direct URL only.
_detect_prompt_suffix accepted ANY line ending in $#%> as a PS1 prompt,
so a single `cat /var/log/dpkg.log` (195 lines closing in `<none>`)
flooded environmental.shell_type votes and flipped a plainly-bash
session to fish.
A prompt line now requires either a trailing space after the suffix
(default PS1 shape across bash/zsh/fish/PowerShell) or a PS1-shape
token (user@host, "PS " prefix, or a Windows drive-letter prefix).
Regression tests pin the dpkg.log false-positive and a $-terminated
prose line.
* decnet attribution — Typer command mirroring decnet reuse-correlate
(--multi-actor-tick, --daemon flags). Calls run_attribution_loop
with the dependency-injected repo.
* deploy/decnet-attribution.service.j2 — systemd unit mirroring
decnet-reuse-correlator.service.j2: ExecStart=decnet attribution,
same hardening posture (NoNewPrivileges, ProtectSystem=full,
ProtectHome=read-only, dedicated /var/log/decnet/decnet.attribution.log).
* worker_registry.KNOWN_WORKERS += "attribution" — heartbeat already
publishes as system.attribution.health from
attribution_worker._WORKER_NAME, so the Workers panel surfaces the
row the moment the unit is enabled.
* api_start_all_workers preferred-order list + "attribution" between
reuse-correlator and enrich so a fresh start-all brings it up
alongside its peers.
After this commit `systemctl enable --now decnet-attribution` (or
the dashboard's start-all) actually launches the engine.
Per-primitive state badge rendered next to each value in the
Behavioural Primitives panel. Five-state vocabulary, frozen, mirrors
decnet/correlation/attribution/aggregate.py:
* STABLE — green, low-key
* DRIFTING — amber, draws the eye
* CONFLICTED — red
* MULTI-ACTOR — purple, loudest (cross-primitive escalation lives
in attribution.multi_actor_suspected, not the
per-primitive badge)
* UNKNOWN — neutral border, no fill
Wiring:
* GET /api/v1/attackers/{id}/attribution on mount + on id change.
Failures swallowed silently (the worker may be off in dev).
* useAttackerStream gains attribution.state_changed +
attribution.multi_actor_suspected named events. The state-changed
handler merges by primitive and locks last_change_ts when the
state did not actually flip (defensive — backend already gates
these on transition, but a future relaxation shouldn't lie about
"stable since X" on the badge tooltip).
* multi_actor_suspected is wired but unused by the badges; the
per-primitive multi_actor signal already shows on each contributing
primitive. The handler is in place so a future "two operators
detected" banner has a live source.
Vitest: 4 new tests (badge renders only for mapped primitives, all
five states render with distinct labels, no badge when prop omitted)
on top of the existing 4. 7 of 7 pass; tsc + vite build clean.
Four synthetic operator-behaviour scenarios at the merger level
(aggregate_observations) that pin v0's calibration:
* Stable HUMAN over 7 sessions -> all primitives stable
* HUMAN switches to LLM mid-week -> primitives flip stable -> drifting
* Two operators alternating -> primitives flag multi_actor
(per-primitive; the cross-
primitive multi_actor_suspected
correlator is exercised by Phase 5)
* Single short session -> all primitives unknown
Plus a threshold-lockdown test that asserts every named constant in
_thresholds.py against its v0 ship value. Anyone adjusting a
threshold without updating the scenarios fails this file.
This closes DEBT-051 at v0 — the attribution engine has a calibrated,
test-locked answer to "is this attacker stable / drifting / showing
multiple operators?" without crossing the persona-attribution bright
line. v1 (cross-attacker clustering, KD simhash linkage signal) is
gated on this v0 surface being stable in production for >= 1 month.
GET /api/v1/attackers/{uuid}/attribution
Returns the merger output for an attacker's identity:
{
"identity_uuid": "abc..." | null,
"primitives": [
{primitive, current_value, state, confidence,
observation_count, last_change_ts, last_observation_ts},
...
]
}
Pre-attribution-worker: identity_uuid=null, primitives=[]. Surfacing
identity_uuid keeps the cross-attacker rollup story visible to the
frontend ahead of v1's clusterer landing.
api_events SSE relay also subscribes to attribution.> and forwards
to the AttackerDetail page filtered on payload.identity_uuid (the
identity is resolved at stream open from the URL's attacker_uuid;
attribution payloads are identity-keyed, not attacker-keyed). New
SSE event names: attribution.state_changed,
attribution.multi_actor_suspected.
Frontend (AttackerDetail.tsx badge rendering, useAttackerStream
consumer) deferred — there's already WIP on AttackerDetail.tsx in
the working tree; merging the badge logic is a separate commit
once that lands.
Tests: 4 endpoint scenarios — 401 unauth, 404 unknown attacker,
200 empty (no stub), 200 with primitive-ordered rows.
Add tick_multi_actor() — periodic walk of attribution_state firing
attribution.profile.multi_actor_suspected when an identity carries
>= MULTI_ACTOR_MIN_PRIMITIVES rows in multi_actor state.
* Repo's list_multi_actor_identities() already filters to >= 2
primitives; the correlator just dispatches.
* In-memory dedup keyed on identity_uuid -> frozenset(primitives):
same set as last fire -> no re-emit. Set grows -> re-emit.
Set shrinks below threshold -> evict so a future re-flap re-fires.
Restart-resets are honest because attribution_state persists; a
v1 multi_actor_suspect_log table can replace this if needed.
* run_attribution_loop() now supervises three concurrent tasks:
observation handler, multi_actor tick loop, health/control. Tick
interval comes from _thresholds.MULTI_ACTOR_TICK_SECS (60s) with
test override.
Tests: 6 scenarios — single-primitive doesn't fire, two-primitive
co-flag fires, dedup blocks unchanged set, set growth re-fires,
threshold drop re-arms, multiple identities fire independently.
attribution_worker.handle_observation_event now executes the full
end-to-end path:
* ensure stub identity (Phase 1)
* observations_for_identity_primitive() — new repo helper joining
observations through attackers.identity_id, so v1's clusterer
gets cross-attacker rollup for free
* aggregate_observations() with ValueKind dispatched off the BEHAVE
PRIMITIVE_REGISTRY; unknown primitives default to categorical
* upsert_attribution_state() — last_change_ts locked when state is
unchanged so the dashboard can render "stable since X"
* publish attribution.profile.state_changed only on transition;
idempotent re-runs over the same observation set fire nothing
(loop-prevention invariant matching ttp.tagged)
Tests:
* 5 end-to-end attribution scenarios over in-memory SQLite + FakeBus.
* test_base_repo's DummyRepo + coverage body now stub every abstract
surface BaseRepository declares — the 6 added by this branch plus
the 12 left un-stubbed by earlier work (BEHAVE Phase 1, TTP
rollups, iter helpers). The coverage test could not previously
even instantiate.
* test_aggregate_categorical's dispatcher rejection updated for the
Phase 3 + 4 contract — ValueError on unknown kinds, not
NotImplementedError.
aggregate_numeric(): EWMA + dispersion (CV) over numeric primitive
values. Stable when CV < 20% AND mean shift < 30%; drifting on >= 30%
mean shift; conflicted on CV > 100%. Confidence is 1 - min(CV, 1).
multi_actor is intentionally NOT a numeric state — bimodal
distributions belong to the categorical detector once the value space
is bucketed.
aggregate_hash(): counts distinct hash values within
HASH_DRIFT_WINDOW_SECS of the most recent observation. 0 rotations =
stable, 1..HASH_DRIFT_MAX = drifting, > HASH_DRIFT_MAX = conflicted.
Reads rotation events; never recomputes hashes (DEBT-032 already
produces them via decnet.correlation.fingerprint_rotation).
aggregate_observations() dispatcher now routes "categorical" |
"numeric" | "hash" | None and rejects unknown kinds with ValueError
(louder than NotImplementedError now that all three v0 mergers
exist). 17 synthetic-input tests cover both new mergers and the
dispatcher.
aggregate_categorical(): pure function over a per-(identity, primitive)
observation list. Five-state vocabulary, last-N=5 window comparison
with one-outlier-tolerant majority threshold:
* unknown — < 3 observations
* stable — recent 5 agree (≥ 4 of 5 share top value), older 5 same
* drifting — recent 5 stable but disagrees with older 5, or older
was conflicted and recent stabilised
* conflicted — recent 5 split, no two-value alternation pattern
* multi_actor — recent 5 split + alternation between exactly two
values (operator A↔B handoff). Confidence capped at 0.6 per
_thresholds.MULTI_ACTOR_MAX_CONFIDENCE; flapping primitives on
flaky networks would otherwise look like two operators.
aggregate_observations() dispatcher honours value_kind="categorical"
(or None) and raises NotImplementedError for "numeric" / "hash" so
Phase 3 lands cleanly. 14 synthetic-input tests cover every state
+ boundary condition.
v0 Phase 1 of ATTRIBUTION-ENGINE.md:
* AttributionStateRow SQLModel keyed on (identity_uuid, primitive)
per ANTI direction — re-keying state rows when the v1 clusterer
merges attackers is the migration debt v0 should not bake in.
ATTRIBUTION-ENGINE.md updated with the deviation note.
* AttributionMixin: ensure_stub_identity_for_attacker, idempotent
upsert_attribution_state, get_attribution_state[_for_identity],
list_multi_actor_identities (the Phase 5 correlator's read).
* attribution.profile.{state_changed,multi_actor_suspected} bus
topics + builder; wiki Service-Bus.md updated separately.
* attribution_worker.py: subscribes to attacker.observation.>,
ensures stub identity per event, logs and continues. No merger,
no state writes, no derived events — Phase 4 wires those.
* attribution/{aggregate.py,_thresholds.py} skeletons: Phase 2
fills _aggregate_categorical, Phase 3 adds numeric+hash+dispatcher.
Real-world bug surfaced on the first live decky run: sessrec.c's
json_escape (decnet/templates/_shared/sessrec/sessrec.c:111-141)
only escapes bytes < 0x20 + DEL — bytes >= 0x80 pass through raw.
An attacker pasting Latin-1 / GB18030 / any non-UTF-8 8-bit text
yields a shard line that chokes Python's default UTF-8 text-mode
read with 'utf-8 codec can't decode byte 0xac'.
Three changes:
1. _events_for_sid now opens with errors='surrogateescape', preserving
byte fidelity through the JSON parse. Surrogate-half chars
correctly fail isascii() / isalpha() so the typed-letter
histograms filter them out automatically. Tightening sessrec.c to
escape >= 0x80 is filed for v0.2 — that's the proper forensic-data
fix; the surrogateescape read makes the engine robust meanwhile.
2. Regression test
(test_handler_tolerates_non_utf8_bytes_in_shard) builds a shard
with raw 0xAC bytes inside a JSON 'data' string and asserts the
handler still persists observations.
3. Collector's _emit_session now logs at WARNING (was DEBUG) when
find_shard_with_sid returns None, citing the three usual causes
(ARTIFACTS_ROOT perms, _SERVICE_RE whitelist, sessrec/collector
race). Surfaces the silent-skip class of bug in seconds instead of
hours — the first live run hid a perm mismatch
(User=anti without SupplementaryGroups=decnet) for an entire
session window before the symptom was traced upstream.
Two-half deliverable per BEHAVE-INTEGRATION.md §587-594:
* scripts/behave_shell/replay_calibration.py — Python helper that
drives the production handler against one asciinema shard, mints
a temp SQLite repo + an Attacker per session, captures bus
emissions in-process. Exits non-zero on zero-observation sessions.
* scripts/behave_shell/smoke.sh — bash entry that replays all five
2026-05-02 calibration shards (HUMAN / YOU-sim / LW-sim /
CLAUDE-FF / CLAUDE-CL). Auto-activates .311 venv, forces
DECNET_DB_TYPE=sqlite, prints per-class summary. Suitable for CI.
* scripts/behave_shell/README.md — runbook covering both halves.
Pins the manual live-decky procedure (one SSH session per class
against a deployed smoke-decky, expected dominant primitives table,
SQL verification query, AttackerDetail panel check, pass criteria).
* BEHAVE-INTEGRATION.md — Phase 6 completion log appended with
current corpus results table (15 sessions, 424 observations across
the five classes) and a note that the v0 tag (drop -pre) is gated
on the manual live-decky round-trip and lands as a separate
commit.
Live-decky run is intentionally NOT scripted — the integration doc
calls for manual SSH sessions per class so an operator confirms the
bus / collector / disk-reach plumbing under real PTY conditions.
Four tests pin the panel surface:
* Empty-state placeholder renders when no observations.
* Day-one priority primitives sort to the top of their group:
motor.input_modality first in motor; the three cognitive priority
primitives in documented order at the top of cognitive.
* Each row renders primitive leaf, value, and confidence-percent
badge.
* Groups follow the canonical domain order
(motor / cognitive / temporal / operational / environmental /
emotional_valence); unknown domains alphabetise at the end.
Mirrors the Orchestrator.test.tsx harness shape (DEBT-043). Live
update path (useAttackerStream → setObservations) is exercised
indirectly via the static render — the hook is dumb glue and the
state mutation is React-side.
Adds the AttackerDetail.tsx panel that surfaces BEHAVE-SHELL
behavioural primitives. Hydrates from the existing
GET /api/v1/attackers/{uuid} response field 'observations',
live-updates via the new useAttackerStream hook (replace-by-primitive
on every 'observation' SSE event).
* New BehaviouralPrimitivesPanel component, exported for vitest.
* Day-one render priority per BEHAVE-INTEGRATION.md §441-454:
motor.input_modality, cognitive.feedback_loop_engagement,
cognitive.command_branch_diversity,
cognitive.inter_command_latency_class — these four sort to the top
of their respective groups; everything else alphabetises.
* Grouped by top-level domain (motor / cognitive / temporal /
operational / environmental / emotional_valence) with the canonical
domain order; unknown domains alphabetise at the end.
* AttackerData interface gains an 'observations' field.
* Empty-state placeholder when the panel has nothing yet.
* Section collapse state extends to 'behavioural', defaults open.
tsc --noEmit clean. Vitest coverage ships in P5.4.
Per-attacker SSE consumer hook. Mirrors useIdentityStream's shape:
* Connects to /api/v1/attackers/{uuid}/events with ?token= auth.
* Per-event-name dispatch via addEventListener for snapshot,
observation, fingerprint.rotated, attacker.scored.
* Reconnect-on-error backoff (3s).
* Callback refs so consumer rerenders don't tear down the connection.
The 'observation' event handler receives every primitive's update
through one event name; the primitive rides in payload.primitive
(matches the backend's _sse_name_for collapse decision).
Hook coverage rides on P5.4's panel test.
GET /api/v1/attackers/{uuid}/events streams behavioural events for
one attacker. Mirrors decnet/web/router/topology/api_events.py
end-to-end: ?token= auth, require_stream_viewer gate,
sse_connection_slot per-user cap, snapshot-on-connect, three bus
subscriptions (attacker.observation.>, attacker.fingerprint_rotated,
attacker.scored) merged through asyncio.Queue, 15s keepalive,
request.is_disconnected() exit, finally task cancellation.
Per-attacker filter keys on payload['attacker_uuid'] which the
profiler worker stamps onto every published payload (Phase 5 P5.0
amendment) — O(1) drop without a repo round-trip per event.
_sse_name_for derives SSE event names:
attacker.observation.<primitive> → observation.<primitive>
attacker.fingerprint_rotated → fingerprint.rotated
attacker.scored → attacker.scored
10 tests cover snapshot, live forward, per-attacker filter (drops
other attackers' events), fingerprint.rotated forward, 404, 401, and
the sse-name derivation across all four cases. Topology events
regression green.
The profiler worker's per-observation publish now re-merges
attacker_uuid into the bus payload alongside id/ts/v. Same shape as
the existing DECNET-side deviation from BEHAVE's wire-format
docstring (BEHAVE-INTEGRATION.md §339-366) — widens the deviation
by one DECNET denorm field.
Phase 5's per-attacker SSE route can now filter
attacker.observation.* events to one attacker in O(1) without a repo
round-trip per event. identity_ref stays None today (until the
attribution engine ships); attacker_uuid is independent.
Two test changes:
* test_happy_path_persists_and_publishes asserts attacker_uuid is in
every published payload.
* New test_attacker_uuid_in_payload_for_filter pins the field
explicitly and confirms it doesn't conflate with identity_ref.
The profiler worker now consumes attacker.session.ended on the bus
AND walks unprofiled session_recorded log rows on every tick. Both
paths converge on a single handler that:
1. Validates required payload fields (session_id, decky_id, service,
attacker_ip, shard_path).
2. Builds evidence_ref shard:{decky}/{service}/{shard_basename}#{sid}
and skips when has_observations_for_evidence is True (idempotent
re-runs).
3. Resolves attacker_uuid via get_attacker_uuid_by_ip; defers if the
profiler tick hasn't materialised the row yet.
4. Reads the asciinema shard, slices events for the sid, calls
extract_session, persists each Observation via upsert_observation
(per-row; batch transaction filed as follow-up), then publishes
each on the bus best-effort (fire-and-forget per DEBT-029 §6).
Architecture:
* Handler lives in decnet/profiler/behave_shell/_handler.py — pure
function, unit-tested in isolation.
* Worker.py adds _behave_pump (queue feed), _drain_behave_queue
(per-tick drain), _behave_poll_tick (cursor scan over
session_recorded logs), and _payload_from_log_row (Log → bus-shape
payload projection).
* Poll cursor uses a separate state key
(attacker_worker_session_cursor) so the correlation tick's cursor
doesn't conflate.
* has_observations_for_evidence promoted to BaseRepository abstract.
22 new tests across handler / drain / poll layers covering happy
path, all skip paths, isolation against handler exceptions,
idempotency on re-run, and cursor key separation. TTP worker bus
tests still green — payload field is purely additive.
Closes BEHAVE-INTEGRATION.md Phase 4.
Lock the BEHAVE library versions per BEHAVE-INTEGRATION.md
§Versioning. The profiler worker (Phase 4 wiring) imports
`Observation`/`Window` from `decnet_behave_core.spec.envelope` and
`event_topic_for`/`to_event_payload` from
`decnet_behave_shell.spec.event_adapter`; without the pin a broken
wheel or missing install would only show up on first publish.
Four-test smoke pins the public surface: envelope construction,
registry import non-empty, event-adapter topic shape, and the
adapter's id/ts/v exclusion contract.
The collector's _SessionAggregator now resolves the asciinema shard
via find_shard_with_sid and stamps it onto every emitted
attacker.session.ended payload as `shard_path`. None when the shard
isn't on disk yet (collector race with sessrec flush) — consumers
treat that as "skip until next tick".
Additive field; existing TTP worker consumes the same topic and
ignores unknown keys, so no payload-version bump needed. Two new
tests pin the shard-found and shard-missing cases.
Unblocks BEHAVE-INTEGRATION Phase 4: the profiler worker reads
shard_path directly from the payload instead of disk-reaching.
Move `_find_shard_with_sid`, `_resolve_shard`, `_validate_names`,
`_get_index`, and the index cache from
`decnet/web/router/transcripts/api_get_transcript.py` into
`decnet/artifacts/shards.py`. The shared module speaks
`ValueError`; the router keeps thin wrappers that translate to
`HTTPException(400)` so the route's error UX is unchanged.
This unblocks the BEHAVE-INTEGRATION Phase 4 worker wiring — the
profiler worker (and the collector's session aggregator) need to
disk-reach asciinema shards but must not import from a FastAPI
router.
11 new unit tests for the shared helper. Existing transcript router
tests pass (the shard fixture's monkeypatch points at the shared
module's ARTIFACTS_ROOT now).
decnet.profiler.behave_shell.__version__ = '0.1.0-pre'.
The -pre suffix is honest: the extractor is feature-complete (37/37
Tier-A primitives emit, calibration grid honest), but the engine
package — worker wiring, observations writes, AttackerDetail panel —
still rides BEHAVE-INTEGRATION.md Phase 4. The actual 0.1.0 tag
lands when Phase 4 lands.
The marker version-tracks the engine, not the spec library
(decnet-behave-shell already at 0.1.0); they version independently.
Run the five-class calibration grid (HUMAN / YOU-sim / LW-sim /
CLAUDE-FF / CLAUDE-CL) against the 2026-05-02 shards.
* Hard gate green for 27 primitives across all 5 shards.
* environmental.keyboard_layout moved from hard gate to
PHASE_F_CONDITIONAL_PRIMITIVES — short SSH-recon corpus maxes at
~90 typed letters per session, well below the LAYOUT_MIN_TYPED_LETTERS
(200) floor. The 200-floor stays per the per-phase "v0 ships when
honest" rule; longer-text corpora will surface the layout signal.
* Three primitives never fire on the 2026-05-02 corpus, all already
conditional and all expected:
- cognitive.error_resilience.frustration_typing
- environmental.locale
- environmental.keyboard_layout
No D / F / G threshold re-tunes needed; only the keyboard_layout
binding-set move. Phase H step log appended to BEHAVE-EXTRACTOR.md
with per-class observation counts.
Static assertion that every Tier-A primitive in PRIMITIVE_REGISTRY
has a slot in the calibration grid (hard gate or conditional set).
Excludes Tier B (8 cross-session primitives) and Tier C (toolchain.*)
by explicit allow-list and prefix filter.
Three checks:
* every Tier-A primitive is covered (forward direction)
* no extractor set drifts from the registry (reverse, catches typos)
* Tier-A count == 37 (design doc invariant)
CI now fails before a registry addition ships without a feature
function.
Widen calibration binding from PHASE_ABCDEF_PRIMITIVES (25) to
PHASE_ABCDEFG_PRIMITIVES (28 hard). Three Phase G primitives that
emit on any session-with-commands ride the hard gate:
* operational.opsec_discipline
* operational.cleanup_behavior
* emotional_valence.stress_response
The remaining five Phase G primitives ride a new
PHASE_G_CONDITIONAL_PRIMITIVES because their sample-size floors make
them legitimately absent from short shards:
* operational.objective (≥ 3 classified commands)
* operational.multi_actor_indicators (≥ 8 commands)
* emotional_valence.arousal (typing bursts)
* emotional_valence.valence (≥ 80 typed letters)
* emotional_valence.frustration_venting (≥ 30 typed letters)
Backwards-compat alias PHASE_ABCDEF_PRIMITIVES kept. Phase G
completion log + checkbox flips in BEHAVE-EXTRACTOR.md.
Tier-A corpus delta: all 37 Tier-A primitives now emit. Phase H
(full-corpus lockdown + v0 release) is next.
Compare median post-error intra-command IATs against baseline
(commands not immediately following an errored command):
* ratio ≥ STRESS_EUSTRESS_RATIO_MIN (1.20) → eustress_positive
* ratio ≤ 1/STRESS_DISTRESS_RATIO_MIN → distress_negative
* otherwise → none
Confidence hard-capped at 0.5; 0.30 below
STRESS_MIN_ERRORED_WITH_IATS (2).
high_agitated when any of:
* caps_run_max ≥ 5
* bang_run_max ≥ 3
* fastest typing burst median IAT < 0.06s with ≥ 30 IATs total
low_calm when slowest qualifying burst median IAT > 0.30s with ≥ 30
IATs. Else medium_engaged. Confidence hard-capped at 0.5; 0.30 below
AROUSAL_MIN_IATS.
Compare median intra-command IATs of the two temporal halves of the
session. ≥ MULTI_ACTOR_HALF_MIN_COMMANDS (4) per half required;
relative delta > MULTI_ACTOR_HANDOFF_DELTA (0.5) → handoff_detected.
team_coordinated is Tier B (cross-session); never emitted from a
single session. Confidence 0.55 with both halves ≥ 8 commands; 0.40
otherwise.
* careful — operator hits OPSEC_HISTORY_TOKENS AND tail-K commands
include _CLEANUP_TOKEN_HASHES (re-imported from temporal.py).
* learning — history hit without cleanup-tail follow-through.
* careless — no history-clearing vocabulary at all.
Confidence 0.45 (small lexicon, soft); 0.30 below
MIN_COMMANDS_FOR_FULL_CONFIDENCE.
Phase G shared infrastructure (no primitive yet emitted):
* New `_intent.py` — five precomputed first-token-hash sets (recon /
exfil / persistence / lateral / destructive) with documented
precedence, plus opsec-history and three lexeme sets (positive /
negative / obscenity) for the typed-text counter pass. Stop words
that collide with registry value vocabulary (`no`, `hell`, `ok`)
are deliberately excluded — the PII regression test catches such
collisions.
* `_typed_char_histograms()` extended with five integer counters
populated in the same single-pass walk: `obscenity_hits`,
`positive_lex_hits`, `negative_lex_hits`, `caps_run_max`,
`bang_run_max`. Longest-suffix match against bounded lexicon
(`LEXEME_MAX_LEN`); paste-class events excluded.
* `SessionContext` widened by the same five fields. Drives G.5
(valence), G.6 (arousal), G.8 (frustration_venting) without retaining
raw operator text.
* Bump twisted >= 26.4.0rc2 to clear CVE-2026-42304 (pre-existing,
caught by pre-commit pip-audit). Adjust ftp template type-ignore
code from attr-defined to misc to match the new Twisted typing.
PII discipline: same shape as F.4 — fixed-vocabulary integer counters
on ctx, never on observations.
Widens the binding calibration set from PHASE_ABCDE_PRIMITIVES (20)
to PHASE_ABCDEF_PRIMITIVES (25). The five new entries:
* environmental.shell_type (per-shard hard gate)
* environmental.terminal_multiplexer (per-shard hard gate)
* environmental.keyboard_layout (per-shard hard gate; PII boundary
lifted by ANTI; emits all 4 registry values)
* environmental.numpad_usage (per-shard hard gate)
* temporal.lifecycle_markers.exit_behavior (resolution of the E.4
hold; uses Command.followed_by_prompt from F.0)
environmental.locale joins a new PHASE_F_CONDITIONAL_PRIMITIVES set
(only fires on shards with an env / locale dump in the output).
Phase F completion log appended to BEHAVE-EXTRACTOR.md. The original
F.0 row hinted at D.0 subsumption; reversed in the log — D.0 is
enriched, not subsumed (regex catches errors when PS1 is suppressed).
Tier-A corpus delta: 25 of 37 primitives now emit. Phase G is next.
Resolves the E.4 hold from Phase E. F.0's Command.followed_by_prompt
gives us the exit-code proxy (prompt-after-last-command) we couldn't
get in Phase E.
Logic: last command without trailing prompt → abrupt; first_token_hash
in {exit, logout, quit, logoff} → graceful; any of the last K=3
commands' first_token_hash in {history, unset, rm, shred, clear, kill}
→ cleanup; else → graceful (clean Ctrl-D / window close).
Sliding-window scan over single-char digit input events. A run of
NUMPAD_RUN_MIN (4) consecutive digit events whose pairwise IATs are
all ≤ NUMPAD_FAST_IAT_S (50ms) → detected. Otherwise → not_detected.
Skips below NUMPAD_MIN_TYPED_CHARS (50) typed chars. Confidence cap
0.50 per the registry's weak-signal flag.
ANTI authorised dropping the PII boundary for this primitive. ctx
gains typed_unigram_counts / typed_bigram_counts / typed_letter_count
populated during the existing single-pass input walk (paste-class
events excluded).
Two-axis classifier:
* layout-artefact unigrams take priority — q rate above floor with
low English saturation → azerty; z above floor with y below → qwertz
* fallback to English-bigram saturation: ≥ floor → qwerty, else other
Sample-size floor 200 typed letters; bigram histogram capped at
top-64 to bound memory. Confidence cap stays moderate (0.40-0.55) —
heuristic discriminator.
Searches ANSI-stripped output for LANG / LC_ALL / LC_CTYPE envvar
substrings emitted by env / locale / printenv. Highest-priority key
wins (LC_ALL > LANG > LC_CTYPE); POSIX value normalised to BCP-47:
en_US.UTF-8 → en-US, pt_BR.UTF-8 → pt-BR, C/POSIX → und. Free-string
registry value emitted directly.
PII discipline: only the parsed locale value enters observations;
surrounding output is read once for matching and dropped.
Scans RAW output (multiplexer escapes are themselves ANSI; never
strip first) for tmux markers (DCS passthrough, focus-reporting,
window-title with tmux marker) and screen markers (DCS, screen-OSC).
Detected → tmux/screen at 0.85; otherwise → none at 0.55. Skips
emission entirely when no commands — silence on a pure-echo or
empty session, per the smoke gates.
When both detected (nested mux), prefer tmux.
Adds PromptLine dataclass + extract_prompt_lines() helper. PromptLine
carries ts, suffix_char ($/#/%/>), raw_line (ANSI-stripped, capped),
is_root flag. Populated during the existing single-pass output-window
walk; SessionContext gains prompt_lines, Command gains
followed_by_prompt.
PII trade-off (ANTI-authorised at Phase F): PS1 text retained on ctx
so F.1 / F.3 / E.4 can read it. Capped at PROMPT_LINE_MAX_CHARS=256.
Observations still only carry derived primitive values.
D.0's regex error helpers stay alongside (NOT subsumed) — they fire
even when PS1 echo is suppressed. F.0 enriches D.0 rather than
replacing it.
F.0's row in BEHAVE-EXTRACTOR.md was forward-only — readers landing
on Phase F couldn't tell that F.0 also has a backlog (E.4 held, D.0
subsumption). Add a 'Carry-overs F.0 must unblock' section to the
Phase F prelude and a back-reference on the F.0 checkbox in the
implementation order checklist.
Widens the binding calibration set from PHASE_ABCD_PRIMITIVES (17) to
PHASE_ABCDE_PRIMITIVES (20). The three shipped Phase E primitives
(session_duration, escalation_pattern, landing_ritual) join the
per-shard hard gate.
E.4 (temporal.lifecycle_markers.exit_behavior) is held at ANTI's
direction pending Phase F.0's prompt parser — abrupt-vs-cleanup
needs exit-code visibility to be honest, and first-token membership
alone over-fires on benign rm / clear mid-session. E.4 picks up at
the tail of Phase F.
Phase E completion log appended to BEHAVE-EXTRACTOR.md; E.1-E.3
checkboxes flipped, E.4 left unchecked with a held note.
Inspect the first N commands; if at least K of their first_token_hashes
match the recon-survey vocabulary (uname/id/whoami/pwd/hostname/w/who),
emit present, else absent. Hashes precomputed at module load; PII-safe.
v0.1 N=5, K=2.
Bin commands into non-overlapping windows of width
max(ESCALATION_WINDOW_MIN_S, duration_s / ESCALATION_WINDOW_TARGET).
CV of per-window counts + zero-window fraction classify bursty /
sustained / erratic. v0.1; corpus re-tune deferred.
Bucket ctx.duration_s against SESSION_DURATION_SHORT_MAX (60s) /
MEDIUM_MAX (600s) / LONG_MAX (3600s); else marathon. Direct
measurement, confidence 0.85. Skip emission only when no commands
and zero duration. New _features/temporal.py module opens Phase E.
Widens the binding calibration set from PHASE_ABC_PRIMITIVES (13) to
PHASE_ABCD_PRIMITIVES (17). The four unconditional Phase D primitives
(cognitive_load, exploration_style, planning_depth, tool_vocabulary)
join the per-shard hard gate. The three error_resilience.* primitives
are conditional on at least one errored command in the shard and
tracked in PHASE_D_CONDITIONAL_PRIMITIVES — excluded from the
per-shard required-emission set, included in the cross-class
discrimination check.
cognitive_load empirical re-tune deferred to the next
BEHAVE_CALIBRATION_DIR run; v0.1 thresholds ship.
Phase D completion log appended to BEHAVE-EXTRACTOR.md; Phase D
checkboxes flipped to [x].
For each errored command, check whether the next command's
first_token_hash is in {man, help, info} (precomputed at module
load). At least one match → present, else absent. The --help / -h
flag forms aren't first tokens; v0.2 will reconsider once arg-token
hashing is justified by corpus.
Compares median within-command IAT for commands following an errored
command vs commands following a successful one. Relative absolute delta
buckets to low / moderate / high. Skips when either group is empty
(no errors, or no clean baseline). v0.1; D.8 re-tunes.
Modal response across Command.errored=True commands:
* same first_token_hash on next command → rerun
* different first_token_hash → switch
* no next command → abort
Tiebreak in registry order. The fourth registry value 'modify'
requires within-command arg diffing (PII boundary); deferred to v0.2.
Distribution of inter-command IATs bucketed against IKI_THINK_MAX_S
(deep) and INTER_CMD_INSTANT_MAX (reactive); fall-through is shallow.
v0.1 thresholds; D.8 re-tunes.
Two-axis classification over the first_token_hash sequence:
repetition_rate (drilling) vs backtrack_rate (jumping among prior
tools). chaotic/targeted/methodical buckets. v0.1 thresholds; D.8
re-tunes.
Composite over three [0, 1]-clipped sub-signals (chunking variance,
error rate from D.0's Command.errored, pace variability), mean-aggregated
and bucketed against COGNITIVE_LOAD_LOW_MAX / COGNITIVE_LOAD_MEDIUM_MAX.
Components missing data drop out of the mean rather than zeroing it.
v0.1 thresholds; D.8 re-tunes once D.2-D.7 are stable. Confidence
held at 0.60 (composite over soft sub-signals) and halved below the
5-command sample-size floor.
Lifts the error-signal slice of F.0 forward as a D.0 prelude. ANSI
strip + canonical bash/sh error fingerprints classify each command's
post-execution output window; Command gains errored / output_bytes
fields. PII discipline preserved — only a bool and an int leave the
helper, the stripped output text is dropped on return.
Drives D.1 (cognitive_load error_rate term) and D.5–D.7 (error_resilience
family). Phase F.0 will subsume this with PS1 + exit-code parsing.
Tick the four Phase B checkboxes (B.1-B.4) and append a Phase B
completion log inline (per the "append phase logs to design docs"
memory rule). Captures per-primitive confidence ranges, source
signals, and the PII-discipline regression that all four
primitives uphold.
Phase A + Phase B = 10 primitives emitting on every shard;
PHASE_AB_PRIMITIVES is binding for every subsequent phase.
Phase C (motor.shell_mastery.*) lands next.
BEHAVE-EXTRACTOR.md Phase B Step B.3. Replaces the prototype's
two-line "0 vs >0 backspaces" placeholder with a backspace-timing
classifier that honours the registry's full vocabulary.
* SessionContext gains backspace_count, backspace_iats (IAT from
each backspace back to the preceding non-backspace input event),
and kill_line_count (^U / ^W). Built by _scan_correction_signals,
which retains only counts and timing aggregates — no character
data leaves the helper, in line with the BEHAVE PII discipline.
* _features/motor.py:error_correction(ctx) emits one Observation
in {immediate, deferred, absent, route_around}.
- 0 backspaces + ≥1 ^U/^W → route_around (rewrite, not correct)
- 0 backspaces + 0 kill-lines → absent
- backspaces with median IAT ≤ 500 ms → immediate
- slower → deferred
Confidence 0.65 / 0.65 / 0.55 / 0.55.
* < 3 inputs → skip emit.
* Calibration grid widened to include motor.error_correction;
green across all five shards.
Tests cover all four buckets, the < 3 inputs skip, and the PII
regression (raw command body never appears in the serialised
observation).
BEHAVE-EXTRACTOR.md Phase B Step B.2. First principled
implementation — the prototype doesn't ship this primitive at all.
* _features/motor.py:motor_stability(ctx) emits one Observation
in {steady, variable, tremor}. Reuses ctx.typing_bursts from B.1.
* Tremor proxy: fraction of within-burst IATs below
TREMOR_FAST_FLOOR_S (30 ms — humans can't sustain sub-50 ms IATs).
≥ TREMOR_RATE_MIN (10%) sub-floor → tremor (double-press / motor
twitch / stuck-key).
* Otherwise median burst CV decides: < CV_STEADY_MAX → steady,
else → variable. Confidence 0.70 / 0.60 / 0.65.
* No typing bursts or fewer than 5 within-burst IATs → skip emit.
* Calibration grid widened to include motor.motor_stability; green
across all five shards.
Tests cover all three buckets + skip paths.
BEHAVE-EXTRACTOR.md Phase B Step B.1.
* SessionContext gains typing_bursts: tuple[tuple[float, ...], ...]
built by _split_typing_bursts(iats) — splits at gaps > IKI_THINK_MAX_S
(1.5s) and drops bursts of fewer than 3 IATs. Mirrors prototype's
_split_into_bursts at BEHAVE/prototype_extractors/shell/extract.py:275.
* _features/motor.py:keystroke_cadence(ctx) emits one Observation
in {steady, bursty, hunt_and_peck, machine}. Median CV across
typing bursts; mean IKI < IKI_MACHINE_MAX_S paired with CV <
CV_MACHINE_MAX → machine. Confidence 0.85/0.70/0.65/0.60 per the
prototype's calibration history.
* < MIN_INPUTS_FOR_CADENCE inputs or zero typing bursts → skip
emission. v0.1 emits only the burst-CV variant; the prototype's
NAIVE session-CV variant is parked for v0.2.
* Calibration grid widened (PHASE_A_PRIMITIVES → PHASE_AB_PRIMITIVES)
to include motor.keystroke_cadence. Grid green across all five
shards.
Tests: too-few-inputs → no emit, all-think-pauses → no burst → no
emit, uniform IATs → steady, sub-5ms → machine, mixed-pace → bursty,
extreme bimodal → hunt_and_peck.
BEHAVE-EXTRACTOR.md Phase A Step 10. Closes the discriminative
floor: six primitives emit, the five-class calibration grid is the
binding regression test for every subsequent phase.
* Phase A checklist boxes (Steps 0-10) ticked in
development/BEHAVE-EXTRACTOR.md.
* Phase A completion log appended inline to the design doc per
the "append phase logs to design docs" memory rule — captures
per-primitive confidence ranges and the 2026-05-02 empirical
anchors that drove threshold calibration.
* Hard gate: tests/profiler/behave_shell/test_calibration_grid.py
parametrised over five class shards, all green; skips cleanly
on BEHAVE_CALIBRATION_DIR unset.
Phases B-G expand horizontally across the registry. Phase H is
the full-corpus lockdown + v0 release. Worker
(BEHAVE-INTEGRATION.md Phase 4) is unblocked at this milestone —
it can wire per-session production against the Phase A engine
without waiting for the rest of the Tier-A corpus.
BEHAVE-EXTRACTOR.md Phase A Step 9 — the gate. Runs the pure
engine against each of the five 2026-05-02 calibration shards and
pins the contract that all subsequent Phase B-G PRs must keep
green: every Phase A primitive (motor.input_modality,
motor.paste_burst_rate, cognitive.inter_command_latency_class,
cognitive.command_branch_diversity, cognitive.feedback_loop_engagement,
cognitive.inter_command_consistency) fires at least once per shard.
* tests/profiler/behave_shell/test_calibration_grid.py
parametrized over (shard_file, class_label) for HUMAN / YOU-sim /
LW-sim / CLAUDE-FF / CLAUDE-CL. Skips entirely when
BEHAVE_CALIBRATION_DIR is unset (CI provides the path; local dev
doesn't have to).
* Plus a discrimination-smoke check: at least one primitive
produces different majority values across present classes —
catches the "constant-output regression" failure mode where the
engine quietly degenerates to a stub.
Calibration tweak: BRANCH_DIVERSITY_LINEAR_MIN dropped from 0.80 to
0.70 to align with the prototype's empirical anchors (CLAUDE-CL ≈
0.55-0.60 adaptive; YOU-sim / CLAUDE-FF scripted recon ≈ 0.75+
linear). Test for the middle band re-pinned at the new boundary.
Per-class value pinning (e.g. HUMAN must emit
inter_command_consistency=bimodal) is intentionally NOT a hard gate
yet — v0.1 thresholds put real human sessions in "variable", and
true bimodal detection (Hartigan dip / two-peak) is registry-flagged
for v0.2. Tighter pinning lands as the corpus grows.
BEHAVE-EXTRACTOR.md Phase A Step 7. The orthogonal axis — does the
operator's pause-after-command correlate with bytes of output they
just saw? Splits HUMAN/CLAUDE-CL (closed_loop) from LW-sim/CLAUDE-FF
(fire_and_forget); cuts ACROSS the LLM/human axis.
* _features/cognitive.py:feedback_loop_engagement(ctx) emits one
Observation in {closed_loop, fire_and_forget, unknown}.
* Pearson correlation between ctx.output_per_cmd[i] and
ctx.inter_cmd_iats[i] (paired by construction in Step 4); via
statistics.correlation with constant-series fallback to "unknown".
* r > FEEDBACK_CORRELATION_MIN (0.30) → closed_loop; otherwise
(zero, negative, or undefined) → fire_and_forget.
* First primitive that depends on output events: zero output events
in the shard or fewer than FEEDBACK_MIN_PAIRS (5) pairs → emit
"unknown" at confidence 1.0 (the absence-of-data is itself a
high-confidence answer). Zero-command session skips entirely.
Tests: no-output → unknown, few-pairs → unknown, strong positive r
→ closed_loop, constant pace → fire_and_forget/unknown,
negative r → fire_and_forget.
BEHAVE-EXTRACTOR.md Phase A Step 6. Content-based playbook-vs-
adaptive split. Splits CLAUDE-FF (linear_playbook, ~10 distinct
tools) from CLAUDE-CL (adaptive_branching, 5-6 tools with curl
re-invoked) per the 2026-05-02 empirical anchor.
* _features/cognitive.py:command_branch_diversity(ctx) emits one
Observation in {linear_playbook, adaptive_branching, unknown}.
* unique_first_token_hashes / total_commands ratio. ≥ 0.80 →
linear_playbook, otherwise adaptive_branching (the doc instructs
bias-to-adaptive in the middle band — that's the discriminative
signal we actually want).
* < 5 commands → "unknown" at confidence 1.0 (the absence of data
is itself a high-confidence answer per the registry's allowed
vocabulary). Zero-command session skips emission entirely.
Tests cover unique-tokens → linear, repeated-tokens → adaptive,
middle band → adaptive (bias), under-floor → unknown @ 1.0, plus
PII regression: raw tokens never appear in the serialised
observation.
BEHAVE-EXTRACTOR.md Phase A Step 5. Classifies the operator's
thinking pace between commands. Splits LW-sim / CLAUDE-FF /
CLAUDE-CL.
* _features/cognitive.py:inter_command_latency_class(ctx) emits one
Observation in {instant, typing_speed, deliberate,
llm_lightweight, llm_heavyweight, long}, computed as the median
of ctx.inter_cmd_iats bucketed against the prototype thresholds
(v0.2 split: lightweight 2-8s, heavyweight 8-30s).
* Sample-size honesty: < 5 commands halves confidence (0.40 vs
0.80) per BEHAVE-EXTRACTOR.md.
* Threshold consts (INTER_CMD_*_MAX, MIN_COMMANDS_FOR_FULL_CONFIDENCE,
plus parked Step 6/7/8 thresholds for the next three commits)
added to _thresholds.py.
Tests cover all six buckets at empirically-anchored IATs (15s ≈
Claude Opus driving recon via tmux send-keys), plus the
single-command no-IAT and low-sample-count paths.
BEHAVE-EXTRACTOR.md Phase A Step 4. Pure refactor inside _ctx.py —
no new feature emits. Lays the shared utility for the three
cognitive primitives next in line (Steps 5-7).
* Command dataclass (frozen): start_ts, end_ts, first_token_hash.
PII-safe by construction — only the first whitespace-delimited
token of the command is retained, and only as a sha256 hash
(decnet/profiler/behave_shell/_parse.py:hash_token).
* _segment_commands walks input events char-by-char, splits on
\r / \n, hashes the first token, drops the rest.
* SessionContext gains commands, inter_cmd_iats, output_per_cmd.
output_per_cmd[i] counts bytes between commands[i].end_ts and
commands[i+1].start_ts — the natural pairing for Step 7
(feedback_loop_engagement).
Tests: empty / unterminated streams, single command (CR + LF
terminators), paste-with-newline, multi-command IAT pairing,
output-byte counting between boundaries, blank-line skip,
first-token-only PII discipline.
BEHAVE-EXTRACTOR.md Phase A Step 3. Same paste-event ratio as
motor.input_modality but coarser-bucketed: this is the *habit*
signal (does the operator reach for paste at all?), where
input_modality is the dominant-channel signal.
* _features/motor.py:paste_burst_rate(ctx) emits one Observation
per session in {none, occasional, habitual} with confidence
0.70 / 0.70 / 0.80.
* Thresholds: PASTE_RATE_OCCASIONAL_MIN=0.10,
PASTE_RATE_HABITUAL_MIN=0.50.
Splits YOU-sim from LW/CLAUDE-FF/CLAUDE-CL — LLM-driven sessions
paste habitually, real humans rarely paste.
Tests: pure-typed → none; 1-paste-in-10 → occasional;
paste-majority → habitual; output-only → no observation; habitual
confidence > occasional confidence.
BEHAVE-EXTRACTOR.md Phase A Step 2. The first primitive — picked
first because it has the highest discriminative value (HUMAN vs
everyone) and the simplest implementation (paste-event ratio over
total inputs).
* _features/motor.py:input_modality(ctx) emits one Observation
per session in {typed, pasted, mixed} with confidence 0.75 / 0.70.
* _features/_emit.py centralises the make_observation helper so
every feature module gets the same Window/source/evidence_ref
boilerplate without copy-paste.
* Thresholds inherited from the prototype's calibration history
(MODALITY_PASTED_MIN=0.40, MODALITY_TYPED_MAX=0.05).
* Zero-input session skips emission — registry doesn't admit
"unknown" here.
Tests: pure-typed → typed, pure-pasted → pasted, mixed → mixed,
output-only session → no observation, full envelope round-trip.
BEHAVE-EXTRACTOR.md Phase A Step 1. Lays the shared primitives that
Steps 2-3 (motor.input_modality, motor.paste_burst_rate) will
consume:
* parse_shard_line / parse_shard turn a shard JSONL line/file into
AsciinemaEvents, skipping headers and malformed records.
* PasteBurst dataclass + _detect_paste_bursts group consecutive
paste-class input events (len(d) >= 4 chars per the prototype's
empirical floor) into contiguous bursts, splitting on IAT gaps
larger than PASTE_BURST_MAX_IAT_S (200ms).
* SessionContext now carries iats and paste_bursts derivations.
* Threshold constants harvested from
BEHAVE/prototype_extractors/shell/extract.py — calibrated against
the five 2026-05-02 shards.
Tests cover pure-typed, pure-pasted, mixed streams; close vs far
paste events; typed events breaking a burst; PasteBurst immutability;
and the JSON parser's junk handling.
BEHAVE-EXTRACTOR.md Phase A Step 0. Lays the package skeleton
(__init__/extract/_parse/_ctx/_thresholds/_features) with empty
FEATURES = (), so the worker plumbing in BEHAVE-INTEGRATION Phase 4
has a stable import path before any primitive lands.
extract_session() builds a SessionContext once and fans the
registered feature functions across it; at Step 0 that fan-out is
empty and the function yields nothing. Step 1 (asciinema parser +
paste-burst detector) and Step 2 (motor.input_modality) land next.
Smoke suite asserts the empty contract: empty stream → no
observations, single event → t_start == t_end, multi-event → events
routed into input_events / output_events by kind, evidence_ref
defaults to "session:<sid>" or honours an explicit override.
Destructive half of BEHAVE-INTEGRATION.md Phase 1. SessionProfile +
its kd_* columns + the dialect ALTER TABLE migration helpers are
deleted outright; pre-v1, the table shipped empty, no migration
ceremony required (per the no-new-_migrate_-pre-v1 memory rule).
DEBT-036 closes via DEBT-050 supersedure. AttackerDetail's
``observations`` field is wired to the new ``observations`` table
and returns an empty list until the BEHAVE-SHELL extractor (DEBT-050
Phase 2) starts emitting.
decnet/web/db/models/attackers.py — SessionProfile class deleted
(~135 lines), KD_PAUSE_*/KD_START_OF_ACTION_IDLE_S module constants
deleted, module docstring updated to point at the observations
table. AttackerIdentity.kd_digraph_simhash is KEPT — it's the v2
federation centroid hook, not a SessionProfile field; docstring
repointed to the BEHAVE primitive that will populate it.
decnet/web/db/sqlmodel_repo/attackers/sessions.py — DELETED.
SessionProfilesMixin dropped from the AttackersMixin MRO.
decnet/web/db/repository.py — abstract upsert_session_profile +
get_session_profile removed.
decnet/web/db/sqlite/repository.py + mysql/repository.py —
_migrate_session_profile_table helpers and their initialize() calls
removed. mysql initialize() now goes attackers → column_types →
admin (no session_profile step).
decnet/web/db/models/__init__.py — SessionProfile re-export gone.
decnet/web/db/models/attacker_intel.py — docstring cross-reference
to SessionProfile.schema_version retargeted to AttackerIdentity.
decnet/web/router/attackers/api_get_attacker_detail.py — adds
``observations: []`` to the response by calling
``repo.latest_observation_per_primitive(uuid)`` and projecting to a
list sorted by primitive path. Empty until the extractor lands;
shape matches BEHAVE-INTEGRATION.md §"AttackerDetail consumer".
tests/profiler/test_session_profile.py — DELETED (56 lines).
tests/db/test_base_repo.py — DummyRepo loses upsert_session_profile
and get_session_profile overrides.
tests/db/mysql/test_mysql_migration.py — initialize-call-order
assertion updated; session_profile step removed from the expected
sequence; docstring records why.
tests/ttp/test_lifter_absence.py — docstring "no SessionProfile" →
"no ObservationRow".
Additive Phase 1 of BEHAVE-INTEGRATION.md. Lays the storage layer
the BEHAVE-SHELL extractor (DEBT-050) will write into. Nothing
breaks; SessionProfile coexists for now and is dropped in the
follow-up commit.
decnet/web/db/models/observations.py — new ObservationRow SQLModel
mirroring the BEHAVE Observation envelope field-for-field
(core/decnet_behave_core/spec/envelope.py). ``id`` is a hex-string
UUID (matching BEHAVE), not a typed UUID column. ``identity_ref``
is str | None — written by the future attribution engine, NULL
until then. ``attacker_uuid`` is the one DECNET-side
denormalisation; FK'd to attackers.uuid for cheap AttackerDetail
joins. ``evidence_ref`` is NOT NULL for DECNET emissions even
though the upstream envelope makes it optional — the worker's
"already profiled?" check keys on it. UniqueConstraint(evidence_ref,
primitive) enforces idempotency at the schema level so re-running
the extractor on the same shard+sid produces a DB-side conflict
the upsert path resolves deterministically. Class is named
``ObservationRow`` (not ``Observation``) to avoid colliding with
the BEHAVE Pydantic envelope at sites that import both.
decnet/web/db/sqlmodel_repo/observations.py — ObservationsMixin.
Three public methods backing the canonical queries from
BEHAVE-INTEGRATION.md §"Storage": ``upsert_observation`` (idempotent
on the natural key), ``latest_observation_per_primitive`` (per-
primitive MAX(ts) subquery, portable across SQLite and MySQL — no
DISTINCT ON), ``observations_time_series`` (asc-by-ts). Plus
``has_observations_for_evidence`` for the worker's session-already-
profiled check.
decnet/bus/topics.py — ATTACKER_OBSERVATION_PREFIX = "observation"
constant + ``attacker_observation(primitive)`` builder. Full topic
shape ``attacker.observation.<primitive>`` matches what BEHAVE's
spec.event_adapter.event_topic_for produces upstream. Documentation
+ pattern matching only — bus auth is socket file perms (DEBT-029
§2), not topic-level.
decnet/web/db/repository.py — abstract ``upsert_observation``,
``latest_observation_per_primitive``, ``observations_time_series``
on BaseRepository.
tests/db/test_observations.py — 11 tests covering upsert round-trip,
idempotency under the unique constraint, latest-per-primitive
ordering across multiple sessions, time-series asc-ordering, empty-
attacker contract, every BEHAVE ValueKind round-tripping through
the JSON column, and the has_observations_for_evidence check.
tests/db/test_base_repo.py — DummyRepo gains the three new abstract
overrides so its coverage suite still instantiates.
Three sibling design docs plus DEBT.md updates that supersede the
stale DEBT-036 with a BEHAVE-aligned plan.
development/BEHAVE-INTEGRATION.md — five-phase rollout: storage
(observations table mirroring the BEHAVE Observation envelope plus
one DECNET-side denorm; UniqueConstraint(evidence_ref, primitive)
enforcing idempotency); engine (in decnet/profiler/behave_shell/
sublibrary, no new daemon, not in BEHAVE — DECNET is the engine);
BEHAVE pin; worker wire; UI panel + per-attacker SSE route; live
smoke. Bus payload merges id/ts/v back in to preserve sensor
identifiers across the bus envelope.
development/BEHAVE-EXTRACTOR.md — engine route in eight phases
(A–H). Phase A locks the 6-primitive calibration grid; Phases B–G
expand horizontally; Phase H is the full Tier-A corpus + v0
release. v0 ships every shell-extractable primitive (37 of them);
Tier B is cross-session and lives in the attribution engine; Tier
C is network-domain (toolchain.*) and lives elsewhere.
development/ATTRIBUTION-ENGINE.md — sublibrary inside
decnet/correlation/ that consumes attacker.observation.* events
and emits attribution.profile.* derived state. Five-state machine
(unknown / stable / drifting / conflicted / multi_actor) with per-
ValueKind merge functions. v0 closes DEBT-051; v1 adds the real
clusterer; v2 federation gossip. The bright line forbidding
attribution to natural persons is lifted directly from BEHAVE's
envelope docstring.
development/DEBT.md — DEBT-036 marked STALE; DEBT-050 and
DEBT-051 entries added; summary table + open list updated.
New MalHashProvider sibling ABC (decnet/intel/base.py) since SHA-256
is a different keyspace from IntelProvider's IPs. MalwareBazaarProvider
mirrors FeodoProvider's bulk-feed shape: 24h refresh via _ensure_fresh
/ _refresh, in-memory set[str] of hex-lowercased hashes, set-membership
lookup. Auth-keyed via DECNET_MALWAREBAZAAR_AUTH_KEY; absent key
silent-no-ops the lane (single warning, no HTTP traffic).
Per-hash observations persist to a new observed_attachments table.
DECNET is a honeypot platform — every attachment hash an attacker
delivers is intel, regardless of whether anyone classified it. Verdict
is sticky: True never downgrades to False/None on subsequent
observations. Out of scope: API surface, federation export, retention.
Ingester _publish_email_received calls the provider for each attachment
sha256, sets mal_hash_match on the bus payload (omitted entirely when
the message had no attachments — keeps R0046's `is True` predicate
silent on hash-less mail, matching pre-paydown behavior), and upserts
the row regardless of provider availability.
New GET /api/v1/orchestrator/events/stats?since=1h&success=false&kind=...
backed by repo.count_orchestrator_failures(since_ts, kind), which
counts failed rows across both orchestrator_events and
orchestrator_emails since the cutoff.
Window parser accepts ^\d+[smhd]$, capped at 7d. Today only
success=false is accepted on this surface so the endpoint isn't
accidentally repurposed before the next consumer is properly
designed.
Orchestrator.tsx polls the endpoint on mount + every 30 s and
renders the authoritative DB-derived count instead of deriving from
the in-memory SSE buffer + one paginated page (which silently
excluded failures older than the local window).
Wire vitest 4 + jsdom + @testing-library/{react,jest-dom,user-event}
+ @vitest/coverage-v8 through vite.config.ts (defineConfig from
vitest/config). src/test/setup.ts registers jest-dom matchers and
RTL cleanup. tsconfig.app.json picks up vitest/globals types.
Seed suite Orchestrator.test.tsx covers the three regressions
called out in DEBT-043: empty-state render, kind-filter toggling
triggers a scoped refetch, mocked stream callback prepends a row.
When the prober observes a NEW hash for an
(attacker_uuid, port, probe_type) triple it has seen before — VPS
rotation, SSH server rebuild, TLS cert swap — emit a derived
attacker.fingerprint_rotated event carrying both old and new hash.
Detection is a small library (decnet.correlation.fingerprint_rotation)
called inline from the prober at each of the three emit sites
(JARM/HASSH/TCPFP). No new daemon. New AttackerFingerprintState table
holds per-triple last-hash state; Attacker.rotation_count and
Attacker.last_rotation_at are stamped on every diff. Library is sync,
fully unit-tested via injected publish_fn / syslog_fn callbacks.
All base images (debian:bookworm-slim, ubuntu:22.04, ubuntu:20.04,
rockylinux:9-minimal, centos:7, alpine:3.19, fedora:39,
kalilinux/kali-rolling, archlinux:latest, honeynet/conpot:latest)
now carry their resolved sha256 digest so 'docker pull' is
deterministic. :tag retained for human readability; @sha256 is what
Docker actually resolves. Refresh procedure documented at the top of
decnet/distros.py.
Empty directory tracked via .gitkeep so operators see it on first
clone; README documents the .eml/.json drop-in flow that the IMAP/POP3
compose fragments wire up by default.
When service_cfg["email_seed"] is absent, compose_fragment now falls
back to $PROJROOT/bait/ if that directory exists on the host. Lets
operators drop a deployment-wide bait corpus into one place without
threading email_seed through every decky's config. Missing dir keeps
old no-op behavior.
IMAP_EMAIL_SEED / POP3_EMAIL_SEED accept a directory (rglob *.eml +
*.json) or a single .json/.eml. Loaded entries CONCATENATE with the
hardcoded _BAIT_EMAILS — additive to the realism-engine emailgen
output rather than replacing it. JSON dicts require from_addr /
to_addr / subject / body; bare bodies are wrapped into RFC 5322 on
load. compose_fragment reads service_cfg["email_seed"] and bind-mounts
the host path read-only at /var/spool/decnet-emails/seed.
The TTP-tagging worker is now safe to run on agent hosts: EmailLifter
disk-reaches body-aware predicates from the local artifacts tree
(DEBT-035 unblocked filesystem access; DEBT-047 added the helper).
Drop `ttp` from MASTER_ONLY_COMMANDS in cli/gating.py and remove the
defence-in-depth `_require_master_mode("ttp")` call in cli/ttp.py.
`ttp-backfill` walks the master DB and stays master-only.
R0047 (BEC) and the encoded-payload predicate substring-match against
the email body. Shipping raw body text on the abstracted service bus
is the wrong privacy stance — the bus transport may swap from UNIX
socket to networked at any time, and "loopback today" is not a license
to put PII on the wire.
EmailLifter now opens the .eml lazily from
/var/lib/decnet/artifacts/{decky_id}/smtp/{stored_as} when a body-aware
predicate runs and parses the body in-process via stdlib email +
policy.default. The decoded body is memoized into the payload dict so
multiple body-aware predicates on the same event open the file once.
Bus envelope only carries the artifact pointer (decky_id + stored_as);
raw body bytes never cross the host disk boundary on the agent → master
hop. Filesystem access on agents is unblocked by DEBT-035 (setgid +
group-readable artifacts root, paid 2026-05-02).
The legacy inline body_text path is preserved — when the producer ships
body_text on the bus the helper short-circuits without opening the file.
Move artifact path validation + symlink-escape check out of the
admin-gated download endpoint into decnet/artifacts/paths.py so the
TTP EmailLifter can disk-reach .eml files at tag-time without
duplicating regex/root logic (DEBT-047).
The router now catches ArtifactPathError and re-raises HTTPException(400);
behavior is unchanged.
ANTI flagged two regressions in the existing command-event capture:
1. **Tell**: PROMPT_COMMAND lived in /root/.bashrc, the FIRST file
an attacker greps after landing root. The logger invocation
sitting there is plain-text honeypot signage.
2. **Bypass**: even when missed, `export PROMPT_COMMAND=""` silently
disables capture. ANTI personally bypasses this on engagements.
Reshape:
* Move the assignment to **/etc/environment** — read by pam_env at
session open (sshd via /etc/pam.d/sshd, telnet via
/etc/pam.d/login), before any shell rc file fires. Far less
obvious than .bashrc; a casual `cat .bashrc` no longer surfaces
the capture.
* Define the helper as a function `__bash_history_sync` in
**/etc/bash.bashrc** (system-wide bashrc, sourced by every
interactive bash). Function name reads as generic bash
housekeeping; no DECNET branding in the symbol.
* Pin both the function and PROMPT_COMMAND **readonly** so
`export PROMPT_COMMAND=""` fails with "readonly variable"
instead of silently winning. Mitigation, not airtight —
`bash --norc` still bypasses — but the passive `export`
bypass is closed.
The actual `logger --rfc5424 --msgid command ... CMD ...` invocation
is preserved exactly; only its location and the readonly guard
change. R0001–R0030 (command-rule pack) consume the same syslog
shape as before.
Three new tests assert: the value lands in /etc/environment, the
function body lives in /etc/bash.bashrc, no PROMPT_COMMAND line
remains in /root/.bashrc, and `readonly PROMPT_COMMAND` /
`readonly -f __bash_history_sync` are both present. Mirror
assertions added on the Telnet Dockerfile via
test_config_schema.py.
Real Linux deployments (especially Ubuntu cloud images) ship a non-
root admin user; honeypots that only accept root logins are a tell.
Add a second account on both SSH and Telnet decoys, configurable
via service_cfg keys `user` / `user_password`, defaulting to
`ubuntu` / `admin` so the lure is live on every fresh deploy.
* `decnet/services/{ssh,telnet}.py` — two new ServiceConfigFields
(`user` string, `user_password` secret) and matching env vars
(`SSH_USER` / `SSH_USER_PASSWORD`, mirror for telnet) propagated
via the compose fragment.
* `decnet/templates/ssh/entrypoint.sh` — runtime `useradd -m -s
/usr/libexec/login-session -G sudo "$SSH_USER"` so the new user
inherits the same sessrec pty-recording shell as root and lands
in the sudo group. Privesc attempts (`sudo`) flow through the
existing sudo-log capture; network-enum from the user's shell
rides the recorded transcript.
* `decnet/templates/telnet/entrypoint.sh` — same useradd pattern
(no sudo group — busybox+login telnet image has no sudo
package; privesc rides `su -` which itself flows through the
existing PAM auth-helper at /etc/pam.d/login).
* New tests for default + custom user / password + independence
from root password. Updated the schema-keys assertion to match
the four-field shape.
The new account is ALSO the natural home for the body-aware
predicates that were previously gated on root-only sessions —
attackers who land on `ubuntu@host` and run network-recon /
privesc commands now generate the same structured TTP-rule
events as root sessions did, captured via the same auth-helper
+ sessrec + sudo-log pipes.
DEBT-035 (artifacts written as the container uid, not the API's) is
resolved by the two preceding commits:
* 39a298f6 — persists DECNET-service api-user/api-group as names in
decnet.ini for any future composer / worker that wants to resolve
the local uid via pwd.getpwnam.
* b2733216 — creates /var/lib/decnet/artifacts at init time with mode
0o2775 (setgid + group-write) owned by the DECNET-service
user:group.
The setgid bit is the load-bearing fix: Linux mkdir(2) propagates a
parent's group AND its setgid bit to every new subdirectory. Docker
auto-creates the per-decoy / per-service subtree as bind-mounts fire,
so those subdirs come up with group=decnet and setgid set; container
file writes (default umask 0o022 → mode 0o644) inherit the decnet
group; the API process and the local TTP worker (both running as the
DECNET-service user, primary group decnet) read via group-read.
The original recommendation of compose `user:` injection turned out
infeasible for SSH and Telnet — PAM's setuid(2) during login
fundamentally cannot run from a non-root container. Setgid covers
both root-internal and unprivileged-internal templates uniformly
without requiring per-template carve-outs.
DEBT-047 (R0047 BEC disk-reach) was gated on DEBT-035 for filesystem
access. That blocker is lifted — `decnet ttp` running on agents as
the local DECNET-service user can now read .eml files written by
the SMTP decoy. The remaining DEBT-047 work is the master-only gate
flip in decnet/cli/gating.py and the EmailLifter disk-reach helper
itself (factor _resolve_artifact_path out of the artifacts API
endpoint into a shared module).
Soft-fail paths in api_get_transcript.py and api_get_artifact.py
stay as defence-in-depth — option 2 should make them never fire on
a healthy install but a misconfigured deploy must not 500 the API.
DEBT-035 step 2. Today the artifacts subtree is auto-created by
Docker as root when a decoy container's bind-mount fires for the
first time. The resulting permissions are root:root 0o755 — the API
process (running as the decnet user) hits PermissionError trying to
read transcripts written by the container, and the soft-fail 404
path gets exercised on every fresh deploy.
Add `/var/lib/decnet/artifacts` to init's dirs list with mode 0o2775:
* 0o2000 — setgid bit. New files inherit the directory's group
(decnet), regardless of which uid created them. This is the load-
bearing bit for cross-container reads.
* 0o0775 — owner+group rwx, world rx. Group-write lets the API
process and the local TTP worker read each other's outputs
without a manual chown.
`_ensure_dir` already respects the full mode word via `os.chmod`,
no helper change needed.
Test asserts the resulting directory carries exactly 0o2775 after
a fresh `decnet init --prefix`. Defence-in-depth: this works even
if the per-decoy compose `user:` directive (next commit) misses a
template — files still land in the decnet group.
DEBT-035 step 1. The composer needs to know which uid/gid to inject
into each compose fragment's `user:` directive at deploy time. Today
the resolved `--user` / `--group` values reach systemd unit
rendering (init.py:349–354) but are not persisted anywhere the
composer can read them.
Persist as **names** (not numeric ids) under `[decnet] api-user` /
`api-group` in the rendered decnet.ini placeholder. Resolution to
uid/gid happens at deploy time on whichever host runs the deploy,
via `pwd.getpwnam(...)` / `grp.getgrnam(...)` — so the same user
name can have different uids on master vs agents (heterogeneous
/etc/passwd) without breaking artifact ownership. The existing
config_ini auto-translates kebab→DECNET_API_USER / DECNET_API_GROUP
at load time; no domain-map changes needed.
Two new tests: one asserting the rendered ini carries the
`api-user` / `api-group` keys for the values passed to `--user` /
`--group`; one round-tripping through `load_ini_config` to confirm
the env vars land in `os.environ` for the composer to pick up.
A previous agent (and several of my own commits) wrote to a top-level
DEBT.md without seeing the existing development/DEBT.md — the
canonical register since DEBT-001. Resulted in two parallel files,
inconsistent numbering schemes, and references that resolved to the
wrong place.
Migrate the six entries that landed in the rogue file into the
canonical register as DEBT-044 through DEBT-049, preserving their
status (resolved / partial / open) and cross-references. The
TTP_TAGGING.md references to "DEBT.md" already resolve to
development/DEBT.md by virtue of being in the same directory; only
the comment in decnet/ttp/impl/intel_lifter.py needed disambiguation
to "development/DEBT.md DEBT-048".
* DEBT-044 — `attacker.email.received` producer wiring (✅ RESOLVED 2026-05-02)
* DEBT-045 — EmailLifter heavyweight feature extraction (PARTIAL PAID 2026-05-02)
* DEBT-046 — EmailLifter mal-hash feed integration (open)
* DEBT-047 — EmailLifter R0047 BEC unblock (open, gated on DEBT-035)
* DEBT-048 — TTP intel provider mapping review (recurring quarterly)
* DEBT-049 — TTP Sigma adapter — post-v1 (open)
Summary table extended; "Remaining open" line updated; root file
removed. The DEBT-047 entry now explicitly cross-references DEBT-035
as the gating dependency for the R0047 BEC unblock.
Mark the EmailLifter heavyweight follow-up as PARTIAL PAID — R0042 /
R0046 (macro / password / smuggling lanes) / R0048 fire end-to-end
after commits 291b78c1 (decky extractors) and the ingester producer
projection that follows.
Two narrower DEBT entries replace the lanes that remain gated:
* "EmailLifter mal-hash feed integration" — R0046's mal_hash_match
lane needs a curated bad-hash feed (MalwareBazaar SHA-256 dump as
the v0 candidate, mirroring the FeodoProvider bulk-feed pattern at
decnet/intel/feodo.py). Feed integration, not extraction. Lifter
predicate already reads `payload.get("mal_hash_match")` — silent
today only because the field is absent.
* "EmailLifter R0047 BEC — unblock when artifact disk-reach lands"
cross-references the agent UID/GID DEBT entry that blocks
`decnet ttp` from reading artifacts written by deckies on the
same host. Disk-reach is the intended solution; raw body_text on
the bus is rejected because the bus transport is abstracted (the
UNIX-socket implementation may swap to networked at any time, and
privacy decisions must hold regardless of transport).
Append to TTP_TAGGING.md §"Producer wiring": the email.received
producer pointer (was "none — DEBT"), the full per-message payload
shape with the new heavyweight fields, and an explanatory block on
why the bus is body-text-free + how R0047 / R0048 each handle their
body dependency (R0048 via the precomputed scalar; R0047 deferred).
The decky's Layer-2 extension (commit 291b78c1) emits body_simhash /
body_base64_bytes / html_smuggling on the message_stored log and adds
macro_indicator / encrypted booleans to each attachments_json
manifest entry. Lift them all onto the email.received bus payload:
* body_simhash — passes through as-is (16 hex chars or "")
* body_base64_bytes — coerced to int (0 on absent / malformed)
* attachment_macros / attachment_password_protected — OR-reduced
across the per-attachment manifest booleans; matches R0046's
matched_trigger semantics where a single positive lane fires the
rule
* html_smuggling — coerced bool from the decky's 0/1 int
Pre-Layer-2 message_stored events (older deckies, malformed log
rows) project to safe defaults: empty simhash, zero base64-bytes,
all booleans False — the EmailLifter then stays silent, never
fires a false positive on missing data.
R0042 (mass-phish) / R0046 macro / R0046 password / R0046 smuggling
/ R0048 (encoded payload) all fire end-to-end after this commit.
R0046 mal_hash_match and R0047 BEC remain deferred per their
respective DEBT entries (filed in the next commit).
Heavyweight Layer-2 extractors land alongside the cheap projections
shipped in commit e9324aca, so the EmailLifter R0042 / R0046 (macros
/ password / smuggling lanes) / R0048 fire from the bus payload
without the lifter having to reach back to disk.
Extractors:
* body_simhash — inlined 64-bit Charikar simhash (md5-keyed,
frequency-weighted) over word tokens of the union of text/* body
parts. Inlined rather than pulling the `simhash` PyPI dep, which
transitively brings numpy ~50 MB into a slim decky container; the
algorithm is ~15 lines and identical in extraction quality.
* body_base64_bytes — largest decoded base64 chunk's byte count,
scanning text body parts with the same `_BASE64_RE` the lifter's
`_p_encoded_payload` fallback uses. R0048 fires from this scalar
alone; the lifter's body_text fallback becomes dead in normal
operation.
* attachment_macro_indicator — stdlib zipfile sniff for
`vbaProject.bin` inside OOXML containers. Catches modern .docm /
.xlsm / .pptm and macro-injected .docx; legacy .xls (CFBF) is a
follow-up.
* attachment_encrypted — flag_bits & 0x01 on any ZIP / OOXML entry's
central directory; magic-byte match for 7z / RAR / CFBF (encrypted
Office wrap).
* html_smuggling — structural lxml parse first: fires when an `<a
download>` element coexists with a `<script>` referencing
`Blob` / `Uint8Array` / `URL.createObjectURL`. Regex pair-check
fallback on lxml parse failure (real-world phish HTML is often
malformed). Cuts the FP rate that pure-regex would produce on
legitimate "click to download" links.
Add `python3-lxml` (~5 MB Debian package, C-extension, no transitive
Python deps) to the SMTP decky's Dockerfile. simhash stays inline.
Per the dependency rule: lxml earns its weight by cutting R0046's
OR-combined FP rate; a heavier macro-detection lib (oletools ~5 MB
pure-python with msoffcrypto) would not measurably improve the
boolean signal we need, so stdlib stays for that lane.
Wires the EmailLifter (R0041–R0048) producer that DEBT.md item #3
deferred. After the existing add_bounty() call in _extract_bounty
(line 615), call _publish_email_received() which:
* resolves the attacker_uuid via repo.get_attacker_uuid_by_ip; drops
the publish if unresolved (the TTP worker can't anchor orphan
events)
* projects the message_stored fields onto the EmailLifter wire
contract: from_domain / mail_from_domain / return_path_domain
parsed via _domain_of, rcpt_count + rcpt_domains via
_rcpt_projection, attachment_sha256s + attachment_extensions
derived from the existing attachments_json manifest, urls from
urls_json, dkim_signed/spf_pass coerced from 0/1 ints to bool
* mirrors _publish_probe_pending's bus-per-call pattern and
swallows all exceptions (the bus is the notification layer, not
the source of truth)
Fires for both relay and non-relay SMTP services. R0041 / R0043 /
R0044 / R0045 are now live end-to-end; R0046 partial (extension
lane). Heavyweight predicates (R0042 simhash, R0046-deep, R0047 /
R0048 body_text) stay deferred per the EmailLifter heavyweight
DEBT entry.
The EmailLifter (R0041–R0048) keys on header-derived signals that the
v0 _summarize_message did not extract. Add cheap Layer 2 projections
inside the existing single-pass parse:
* return_path / x_mailer — direct header reads, decoded RFC 2047
* dkim_signed / spf_pass — booleans derived from any
Authentication-Results header (multiple lines tolerated; positive
verdict on any line wins)
* urls — http(s) URLs lifted from text/* body parts via a tight
regex, deduplicated first-seen-wins, capped at 64 in the wire
payload to bound the syslog SD value
Heavyweight extraction (body simhash, office-macro detection,
HTML-smuggling, password-protected archives, mal-hash-match,
body_text projection) stays deferred per the EmailLifter heavyweight
DEBT entry — those rules need privacy / extractor decisions before
they ship.
The 2026-05-02 paydown wires the producer at ingester.py after
add_bounty(), with the cheap projections (domains, rcpt_count,
attachment_count, x_mailer, dkim/spf, attachment shas + extensions,
URLs). R0041 / R0043 / R0044 / R0045 fire end-to-end after this PR;
R0046 partial.
The remaining lanes (R0042 body_simhash, R0046 macro / smuggling /
password / mal_hash, R0047 / R0048 body_text projection) are filed
as a new entry "EmailLifter heavyweight feature extraction" with the
field map and the privacy-vs-completeness fork on body_text called
out for the next maintainer to pick a side.
Appendix A.10 corrected to match the post-2026-05-02-audit reality:
AbuseIPDB cat 7/13/16/17 land on their canonical AbuseIPDB names
(Phishing / VPN IP / SQL Injection / Spoofing); cats 4 and 10 carry
explicit "drop" annotations so the next reviewer sees the intent
rather than guessing. ThreatFox table re-keys on `threat_type` (the
canonical taxonomy field) and adds the `payload` and `cc_skimming`
rows. GreyNoise table promotes bare-malicious to a half-multiplier
emission of T1071.
§"Hard parts §9 Intel provider drift" replaces the prose handwave
with a runnable check: provider URLs, the ThreatFox curl invocation
that needs DECNET_THREATFOX_API_KEY, the rule_version + emits +
attack_catalog co-evolution rules, and the full chain of files to
exercise. Adds a "Ship-time audit log" subsection so future quarterly
runs have a known-good baseline to diff against.
DEBT.md item #1 records LAST_REVIEWED: 2026-05-02 / NEXT_REVIEW:
2026-08-02 and points at §9 for the runbook. DEBT.md item #3 (the
attacker.email.received producer) flags its gating premise as
potentially stale — ANTI noted SMTP honeypots already persist
received messages, contradicting the "no source row" claim that
deferred the wiring.
The IntelLifter's _emit_filtered fans out only the rule.emits entries
whose technique_id appears in the predicate's decision set. v1's emits
lists were narrow supersets of the common case, silently dropping the
rest of the predicate's possible emissions:
R0054 dropped: T1046 (cat 14), T1078 (cat 20), T1090 (cats 9/13),
T1496 (cat 11), T1595 (cats 14/19)
R0055 dropped: T1090 (tor_exit_node), T1110 (ssh_bruteforcer),
T1588 (the second emit of every C2-framework tag)
R0057 dropped: T1105 (payload_delivery, download_url)
Bump rule_version 1->2 on R0054/R0055/R0057, expand emits to cover
every technique the predicate produces. R0056 (Feodo) and R0058
(aggregate bump) carry no enum and stay at v1.
All five YAMLs gain `last_reviewed: "2026-05-02"` and
`next_review: "2026-08-02"` markers; the rule YAML is now the
canonical record of when the mapping was last reconciled against
upstream, with DEBT.md as the calendar reminder.
Three bug classes uncovered by the 2026-05-02 ship-time audit:
* AbuseIPDB code/name mismatch in v1: cat 10 was treated as DDoS (it's
Web Spam — DDoS is cat 4, intentionally unmapped per A.10) and cat 17
as VPN IP (it's Spoofing — VPN IP is cat 13). Both typos mirrored in
code AND the design doc Appendix A.10. Code now matches the AbuseIPDB
taxonomy exactly; cat 17 retargets to T1566 (email-spoofing as a
phishing precursor), and cats 7 (Phishing) and 16 (SQL Injection)
pick up T1566 / T1190 emissions that v1 didn't cover.
* ThreatFox dispatch keyed on `ioc_type` in v1, but `ioc_type` is the
indicator format (url / domain / hash variants) and carries no ATT&CK
signal. The canonical taxonomy field per ThreatFox's API is
`threat_type` (botnet_cc / payload_delivery / payload / cc_skimming).
Repoint dispatch through the new `threatfox_threat_types` payload
field; `ioc_type` rides as evidence only. Also adds the missing
cc_skimming -> T1056 (Input Capture) mapping and registers T1056 in
attack_catalog.py.
* GreyNoise bare-malicious lane: a `classification == "malicious"` row
with no recognised tag used to emit nothing. Now lights T1071 at a
half multiplier, suppressed when a tag already fires T1071 to avoid
double-stamping at conflicting confidence levels.
The TTP worker forwards the bus payload verbatim to the IntelLifter as
TaggerEvent.payload. The pre-audit publish payload only carried
{attacker_uuid, attacker_ip, aggregate_verdict, providers}, so even with
the new AttackerIntel taxonomy columns populated the lifter still saw
nothing. Lift the relevant fields (categories / tags / threat_types /
malware family / score / classification) into the bus event and decode
JSON-string list columns back to native lists at the boundary.
The 2026-05-02 ship-time audit of the R0054-R0058 intel rule pack found
that AbuseIPDB / GreyNoise / ThreatFox stored only the aggregate verdict
(score / classification / listed-bool) plus the raw response blob. The
TTP IntelLifter expects per-provider taxonomy fields (categories, tags,
threat_types) that were never populated, so R0054 / R0055 / R0057
emitted zero tags in production despite passing unit tests.
Add typed columns: abuseipdb_categories, greynoise_tags, greynoise_name,
feodo_malware_family, threatfox_threat_types, threatfox_ioc_types,
threatfox_malware_families. Each provider now parses the relevant
taxonomy out of the upstream response and writes it through
column_updates. JSON-list columns ride as TEXT with default "[]" to
keep the SQLite/MySQL backend split honest, deserialised back to native
lists by the repo on read.
The inspector was dumping the whole `CMD uid=0 user=root src=… pwd=…
cmd=nmap -p- 192.168.1.0/24` syslog body into a single ``command_text``
blob. ANTI: "I'd like to separate the fields." Done — three layers
work together:
1. Collector session aggregator: new `_parse_cmd_msg` splits the bash
PROMPT_COMMAND msg into `{uid, user, src, pwd, command}`. The
session-ended envelope's per-command dict now carries the
structured fields, with `command_text` set to just the cmd= value
(preserving embedded whitespace — `nmap -p- 1.2.3.0/24` etc.).
2. Rule engine: per-source_kind auxiliary evidence list
(`_AUX_EVIDENCE_FIELDS`). For `command` events the engine
automatically promotes uid/user/src/pwd into the persisted
`evidence` dict on top of the rule's explicit `evidence_fields`.
Engine-controlled, not per-rule — adding a new aux field is one
line here, not a 30-rule YAML sweep, and rule authors can't
accidentally drop it.
3. TTPInspector frontend: evidence renders as a structured
`kvs` grid (UID / USER / SRC / PWD / CMD rows) instead of
pretty-printed JSON. Primary-order list keeps shell fields at
the top; everything else falls below alphabetically so unfamiliar
evidence shapes still surface predictably.
Tests:
- session_aggregator pins the structured-fields emit (uid/user/src/
pwd/command_text without "CMD" prefix, embedded whitespace
preserved).
- rule_engine_tagger pins the aux-field auto-promotion + the
no-`None`-leakage path when payload doesn't carry an aux key.
"T1595" alone is opaque; "T1595 — Active Scanning" tells you the
story at a glance. The names come from a backend-side static catalogue
pinned to the same ATT&CK release as the rule engine
(_ATTACK_RELEASE = "v15.1") — names are the canonical MITRE labels,
not author-supplied strings on rules, so a rule author can't typo a
name and the entire fleet sees the typo.
- New `decnet/ttp/attack_catalog.py` with `TECHNIQUE_NAMES` covering
every technique_id + sub_technique_id emitted by `rules/ttp/`
(R0001..R0058 → 69 IDs in the v0 pack).
- `IdentityTechniqueRow` / `TechniqueRollupRow` / `CampaignTechniqueRow`
/ `TTPTagDetailRow` gain optional `technique_name` /
`sub_technique_name` fields. Repo + router populate them from the
catalogue at row-construction time. None when an ID isn't in the
catalogue — UI falls back to the bare ID.
- Coverage test (`tests/ttp/test_attack_catalog.py`) walks every
YAML rule and asserts every emitted ID has a catalogue entry, so
a future rule author who forgets to update the catalogue gets a
loud failure rather than a silent UI fallback.
Frontend:
- `TTPsObservedSection` shows "T1595.002 — Active Scanning:
Vulnerability Scanning" instead of just the ID, with overflow
ellipsis + tooltip for narrow viewports. Inspector header /
TECHNIQUE row also surface the names.
The TTPsObservedSection rollup tells the operator "we saw T1059" but
not why. Click any technique row → side drawer opens listing every
ttp_tag row in scope with the persisted evidence JSON, firing
rule_id / rule_version, source_kind / source_id, confidence, and
created_at. Mirrors the CredentialReuseInspector / BountyInspector
pattern (drawer-backdrop + bd-head/bd-body + kvs grid).
Backend:
- New `GET /api/v1/ttp/tags/by-{scope}/{uuid}/{technique_id}`
(`scope ∈ {identity, attacker, session}`, optional
`?sub_technique_id=`, `?limit=` capped to 1000). Returns raw
TTPTag rows newest-first.
- New `TTPTagDetailRow` Pydantic model + re-export.
- New repo method `list_tags_by_scope_and_technique` on
TTPMixin (+ abstract on BaseRepository) — single query branched
on scope; identity scope projects through `Attacker.identity_id`
the same way `list_techniques_by_identity` does.
- Tests: evidence round-trips, sub_technique filter, JWT-required,
empty scope, unknown scope rejected.
Frontend:
- New `TTPInspector.tsx` + `TTPInspector.css` (violet accent, slide
animation, focus-trapped panel matching the existing inspector
family).
- `TTPsObservedSection`'s TechniqueBar is now click+keyboard
activatable; clicking opens the inspector for that
(technique, sub_technique) tuple.
mypy clean. 532 passed in the targeted sweep.
The collector's `attacker.session.ended` envelope carries
`attacker_uuid: null` and `attacker_ip: <ip>` because the collector
doesn't talk to the DB. The TTP worker passed that null straight
through, and `TTPTag.__init__` raised the documented invariant:
ValueError: ttp_tag requires at least one of attacker_uuid /
identity_uuid; both NULL is not a valid anchor.
The worker now resolves `attacker_uuid` from `attacker_ip` via
`BaseRepository.get_attacker_uuid_by_ip` before fanning out the
event. When the IP isn't in the DB yet (profiler hasn't ingested
the row), the event is dropped with one log line — better than
exploding mid-tag.
- New `get_attacker_uuid_by_ip(ip) -> str | None` on the repo
(BaseRepository abstract + AttackersCoreMixin impl).
- `_resolve_attacker_uuid` helper in `decnet/ttp/worker.py` runs
before `_build_events`. Short-circuits when the payload already
has either anchor; drops the event when neither anchor is
resolvable.
- Tests pin: short-circuit on existing uuid/identity, repo lookup,
drop on unknown IP, drop on "Unknown" sentinel, drop on
no-anchor payload, drop on repo failure.
Add a "Producer wiring" subsection under TTP_TAGGING.md §"Bus
topics" mapping every topic the TTP worker subscribes to onto the
file:line that publishes it. Calls out the gap (`email.received`
has no producer today) and the new `attacker.session.ended`
payload shape from the collector aggregator.
Also lists the four producer regression tests added in this series
so a future contributor sees the safety net before staring at the
silent rule engine.
DEBT.md gets the `attacker.email.received` follow-up entry — wire
the producer when SMTP-receive persistence lands, since today the
honeypot relay path doesn't store received emails anywhere a
publisher could read from.
Three producer-side regression guards. Each drives the worker's run
loop with a fake bus + stubbed repo and asserts the documented topic
fires when the producer has data:
- reuse correlator → credential.reuse.detected (one finding row)
- clusterer → identity.formed + identity.merged (one ClusterResult)
- intel worker → attacker.intel.enriched (one unenriched attacker
+ a fake provider returning a "malicious" verdict)
These complement commit 1's attacker.session.ended producer test —
together the four cover every TTP-relevant publisher in the tree
(modulo email.received, which has no producer yet; tracked in
DEBT.md).
The TTP worker subscribes to attacker.session.ended but no upstream
component published it — the rule pack (R0001–R0030) therefore never
fired on live SSH traffic even after the consume-side wiring landed
in E.3.18a/b/c.
The collector now hosts a per-attacker_ip command index
(_SessionAggregator) that watches the same parsed-event stream as
_publish_log. Shell `command` events are appended to a per-IP list;
on `session_recorded` the aggregator slices the list to commands
inside the [ended_at - duration_s, ended_at] window and publishes
attacker.session.ended with the session metadata + commands list.
The TTP worker's _build_events fan-out (E.3.18b) turns each command
into a source_kind="command" TaggerEvent that the RuleEngineTagger
(E.3.18c) matches against R0001–R0030.
Memory bound: per-IP entries TTL-evict at DECNET_COLLECTOR_SESSION_AGG_TTL_SEC
(default 3600 s). Publish failures are swallowed in the aggregator —
a misbehaving bus cannot stall the per-container stream threads.
Honeypot SSH containers run `PROMPT_COMMAND` that calls
`logger --rfc5424 --msgid command -t bash "CMD …"`. The Docker-stdout
reader prepends an outer RFC5424 envelope (HOSTNAME=<decky>,
APP-NAME=1, MSGID=NIL) around that inner syslog line. Both the
collector parser (`parse_rfc5424`) and the correlation parser
(`parse_line`) saw the outer NIL MSGID and emitted `event_type="-"`
for every shell command — which:
- kept `Attacker.commands` rows missing `command_text`
- left R0001–R0030 (the pattern rule pack that matches shell
commands) with no haystack
- made `decnet.collector.log` show `event written … type=-`
for the very lines that should be `type=command`
Both parsers now detect the inner-RFC5424 shape (`<TS> <HOST> <APP>
<PROCID> <MSGID> <rest>`) when the outer MSGID is NIL and the SD-arm
is also NIL, and re-extract HOSTNAME / APP-NAME / MSGID / remainder
from the body. The collector parser also recovers the post-SD msg
tail when the SD block isn't `relay@55555` (the bash CMD line carries
a `[timeQuality …]` block) so the kv-fallback can find `src_ip`.
Mirroring tests in tests/collector and tests/correlation pin both
the unwrap and the regression guard for non-double-wrapped lines.
The endpoint was a contract-phase stub returning `[]` even though the
RuleStore loaded all 58 YAML rules at worker startup. UI saw an empty
table; operators couldn't tell whether anything was wired up.
- `api_list_rules` now calls `get_rule_store().load_compiled()` and
serializes each CompiledRule + its operational state into a
RuleCatalogueRow. Sorted by rule_id for stable golden snapshots.
- Add `description: str` to RuleSchema (pydantic) and CompiledRule
(NamedTuple, defaulted) + propagate through `_compile_one` so the
catalogue surfaces the human-readable YAML description, not just
the slug-style `name`.
- Update `tests/ttp/test_rule_engine.py` _fields assertion for the
new column; new `tests/api/ttp/test_rules_catalogue.py` pins the
catalogue contents (R0001/R0014 presence, row shape, sort order).
Worker behaviour is unchanged: it was already loading rules
correctly. This is purely a read-side wiring fix on the operator API.
Closes the CDD design phase. Records:
- §E.1 contract inventory (every file exists, compileall clean).
- Targeted pytest pass: 604 passed, 1 skipped, 10 xfailed
(all xfails are `xfail(strict=True)` with reason= pointing to the
impl step that flips them; carry-overs, not flakes).
- Strict mypy over decnet/ttp + decnet/cli/ttp.py +
decnet/web/router/ttp + decnet/web/db/sqlmodel_repo/ttp.py: clean.
- Stranger-readability spot check on tests/ttp/: no doc bugs.
Notes the three pre-E.4 wiring fixes (E.3.18a/b/c) and the E.4
backfill CLI / DEBT entries that landed in this series.
Quarterly TTP provider mapping review for AbuseIPDB / GreyNoise /
abuse.ch (Feodo Tracker, ThreatFox) catalogue drift against
`rules/ttp/R0054..R0058`, and the post-v1 trigger for the Sigma rule
adapter. Both items reference TTP_TAGGING.md sections so the
rationale stays linked to the design doc.
The TTP worker entry moved out of decnet/cli/workers.py into its own
module so the TTP CLI surface (worker + admin verbs) is colocated,
mirroring decnet/cli/canary.py / webhook.py / swarm.py.
- New `decnet/cli/ttp.py` with `decnet ttp` (worker, ExecStart-stable
for decnet-ttp.service) and `decnet ttp-backfill --since-days N`.
- `decnet ttp-backfill` walks Attacker.commands and CanaryTrigger
history, dispatches each row through the live CompositeTagger,
persists tags via repo.insert_tags (idempotent INSERT OR IGNORE).
--dry-run / --source command|canary|all / --batch-size supported.
- Backfill deliberately bypasses bus publish — historical replay
must not re-trigger SIEM/webhook fan-out per TTP_TAGGING.md
§"Bus topics" loop-prevention invariant.
- Added `iter_attacker_commands_since` / `iter_canary_triggers_since`
read-only iterators on TTPMixin + abstract bindings on
BaseRepository.
- Master-only via gating; both `ttp` and `ttp-backfill` listed in
MASTER_ONLY_COMMANDS.
The canonical rule-based engine from §"Tagging engines, layered §1"
of TTP_TAGGING.md was fully implemented but never instantiated as a
composite child — pure pattern rules (R0014/R0017/R0023/... 23 rules
total) had no tagger to dispatch them.
- Add `RuleEngineTagger(Tagger)` adapter in rule_engine.py wrapping
`RuleEngine.evaluate()`. `HANDLES = {command, http_request,
auth_attempt, payload}` — the source kinds whose rules typically
live outside any per-source lifter.
- Adapter's `watch_store()` filters via `_is_engine_owned` so the
engine's dispatch index excludes lifter-claimed rules
(`match.kind: lifter:*`) and stays disjoint from per-lifter ownership.
- Prepend `RuleEngineTagger` to the `CompositeTagger` lifter list so
generic pattern rules dispatch before per-source cross-event logic.
- Composes with E.3.18a (worker hydrates `watch_store`) and E.3.18b
(worker fans session payloads into per-`command` events) — together
these three commits make R0001–R0030 actually fire at runtime.
R0001–R0030 declare `applies_to: [command]` and match per command, not
per session. The worker now translates one `attacker.session.ended`
payload carrying a `commands: list` into:
- one source_kind="session" event (behavioral / cross-event lifters)
- one source_kind="command" event per command (RuleEngineTagger)
Both string and dict command shapes are accepted; dicts contribute
their `id` / `uuid` / `command_id` as the per-command source_id so
the deterministic `compute_tag_uuid` keeps replays idempotent. Tags
from session + per-command dispatch are aggregated into a single
`ttp.tagged` envelope per upstream session.
Each per-source lifter holds its own RuleIndex and exposes an
`async watch_store()` that loads the corpus and drains store change
events forever. Until this commit nothing called `watch_store()` in
production — every dispatch index stayed empty and no rule fired.
- Add `WatchableTagger` runtime-checkable Protocol in `decnet.ttp.base`.
- `CompositeTagger.iter_watchables()` yields lifters that satisfy it.
- `run_ttp_worker_loop` fans out one task per watchable, cancelled
and awaited alongside pump/heartbeat/control in the existing finally.
- Watch failures log and exit the watch task without taking the
worker down — mirrors the pump-task tolerance contract.
Wires decnet-ttp as a first-class worker:
* `decnet ttp` CLI command (master-only via MASTER_ONLY_COMMANDS)
* deploy/decnet-ttp.service.j2 systemd unit (After= identity / intel
/ reuse-correlator workers; ProtectHome=read-only since
FilesystemRuleStore only reads ./rules/ttp/)
* deploy/decnet.target Wants= chain extended with decnet-ttp.service
* `ttp` was already in web/worker_registry.KNOWN_WORKERS
tests/api/test_schemathesis_ttp.py: TTP-routes-only schemathesis
suite, filtered via the OpenAPI tags=["TTP Tagging"] annotation
shared by the eight TTP routes. Reuses the live uvicorn subprocess
the wider test_schemathesis spawns; max_examples=400 keeps the
focused gate fast for E.3.13–E.3.16 iteration.
wiki-checkout/Service-Bus.md committed in its own repo: ttp.tagged
and ttp.rule.fired.<id> flipped from "reserved (TTP worker)" to
"decnet.ttp.worker" now that the worker publishes them.
TTPsObservedSection.tsx: shared analyst-facing rollup. scope=
identity drives /ttp/by-identity/{uuid} (primary, with Navigator
export download); scope=attacker drives /ttp/by-attacker/{uuid}
(per-IP slice). Tactic → technique tree in fixed UKC-aligned order,
counts and confidence-weighted bars. Literal "NO TECHNIQUES
OBSERVED YET" empty state per TTP_TAGGING.md §"UI surface — Empty
state": no spinner, no fallback list.
RuleStateControls.tsx: admin-only rule operational state panel
backed by POST/DELETE /ttp/rules/{rule_id}/state. Server-gated by
require_admin AND client-gated on /config?.role so a non-admin
never sees the controls (per feedback_serverside_ui.md the client
gate is UX, not security — the server returns 403 either way).
Wired into Config.tsx as a new "TTP RULES" admin tab.
Wired TTPsObservedSection into IdentityDetail (above fingerprints)
and AttackerDetail (above TIMELINE). DeckyFleet/PersonaGeneration
vocabulary throughout (logs-section / section-header / btn /
matrix-text / dim-chip).
tsc --noEmit and vite build clean.
The dev-server browser smoke is deferred per the "can't reliably
exercise UI from this harness" reality — typecheck + build is the
correctness gate, not feature verification.
Add BaseRepository.list_ttp_decky_phases(identity_uuid) returning
per-decky tag observations as (decky_id, tactic, created_at_ts) rows
ordered by creation time. Rewrite from_identity_row() to project
tactic → UKCPhase via tactic_to_ukc_phase and populate the four
phase-handoff maps (first/last_phase_per_decky,
first/last_seen_per_decky) so combined_campaign_weight finally lights
up on real DB rows — not just synthetic fixtures.
ConnectedComponentsCampaignClusterer.tick() pulls each active
identity's per-decky phase observations before projecting features.
Repo failures are non-fatal: a partial repo falls back to the empty
phase-handoff signal (legacy behavior) so the worker stays up.
tests/clustering/test_ttp_phase_handoff.py pins the production-row
pair clearing CAMPAIGN_EDGE_THRESHOLD on a C2 → DISCOVERY hand-off —
the trip-wire that says the whole project paid off.
commands_by_phase_on_decky itself stays empty on the production path:
it is consumed only by the synthetic-fixture similarity surface, and
the phase-handoff edge does not use it. Synthetic fixtures still
populate it directly via from_synthetic_identity.
Inner loop drains a per-process asyncio.Queue populated by one pump
task per topic in _TOPICS, dispatches each event through
CompositeTagger, persists via repo.insert_tags(), and publishes
ttp.tagged + per-technique ttp.rule.fired.<id> only when the insert
returned a non-zero rowcount.
CompositeTagger seeded with all six lifters (Behavioral, Intel,
CanaryFingerprint, Email, Identity, Credential).
Loop-prevention invariant from TTP_TAGGING.md §"Bus topics" enforced:
N replays of the same upstream event publish exactly one ttp.tagged
event. test_worker_bus covers both the direct invocation path and
the idempotency replay path.
Intel catch-up via attacker.session.ended is intentionally deferred
to E.3.14b — needs a session→intel join the repo doesn't expose yet.
IdentityLifter owns lifter:identity_* — currently R0003 (password
spraying). CredentialLifter owns lifter:credential_* — R0001 generic
auth brute, R0002 password guessing, R0004 credential reuse, R0005
valid-account use, R0006 default credentials.
YAMLs R0001/R0002/R0003/R0005/R0006 had their match.kind normalised
to fit the lifter prefix scheme — the design doc's promised "YAMLs
normalised in a separate refactor commit" lands here.
Identity-rollup tags null out attacker_uuid on emit so the worked-
example invariant holds (the tag belongs to the Identity, never to
one member IP).
Tests: test_identity_lifter.py + test_credential_lifter.py cover
each predicate's positive/negative path, state modulation
(disabled/clipped/expired), source-kind gating, and idempotent
replay. test_lifter_absence and test_lifters updated for the new
ctor signature.
Records the RuleIndex extraction prerequisite, the lifter:<owner>_
prefix routing convention, per-provider technique fan-out logic for
intel rules, the canary identity-merge guard rail, and the email PII
allowlist + R0042 simhash requirement.
SMTP message-level technique tagger per Appendix A.6: open relay abuse
(rcpt_count + foreign From), mass phishing (rcpt_count + body simhash),
phishing-kit X-Mailer, IDN/punycode URL, sender masquerade composite
(From/Return-Path/DKIM/SPF), malicious attachment (macro/.lnk/.iso/.img/
hash match), BEC subject+body composite, encoded payload in body.
PII discipline (TTP_TAGGING.md §'Hard parts §6') is enforced at the
lifter layer via _filter_evidence(): emitted TTPTag.evidence is
restricted to the EmailEvidence-allowed allowlist (body_sha256,
matched_headers — names only, rcpt_domain_set — domains only,
attachment_sha256s, rcpt_count) plus PII-safe match discriminators
(matched_kit, matched_trigger, matched_url_host, etc). Raw addresses,
raw body bytes, full URLs, and decoded base64 previews NEVER appear in
evidence — defense-in-depth over the YAML evidence_fields hint.
Tests: tests/ttp/test_email_lifter.py per-rule positive + negative +
PII allowlist guard + state modulation. tests/ttp/rule_precision/
test_email_rules.py xfail flipped to real precision (R0041-R0048
H-band ≥95%). Corpus rows updated to acknowledge that R0045 (masquerade)
co-fires with R0041 / R0047 when the sender-masquerade signals are
present alongside open-relay or BEC patterns — overlap is by design,
not a precision bug.
Browser-payload derivations per Appendix A.9: navigator.webdriver flag,
canvas/audio/WebGL automation hash matches (Puppeteer/Playwright/
Selenium/curl-impersonate), WebRTC IP leak, TZ/language vs source-IP
geo mismatch, navigator.platform vs userAgent vs WebGL renderer
inconsistency.
Evidence shape pinned to CanaryFingerprintEvidence (metric +
matched_signature) — raw fingerprint blobs (canvas hashes, full UAs,
navigator.platform values) explicitly NOT carried into TTPTag.evidence
per TTP_TAGGING.md §'Hard parts §7' (enrichment vs tag boundary). The
identity-merge guard rail is preserved: composite fp.id matches across
IPs are NOT a TTP, so no rule fires on the bare hash.
Tests: tests/ttp/test_canary_fingerprint_lifter.py per-rule positive +
negative + evidence-shape guard + state modulation.
tests/ttp/rule_precision/test_canary_rules.py xfail flipped to real
precision (R0049/R0050/R0051/R0053 H-band ≥95%; R0052 M-band ≥80%).
Per-provider verdict translator for AbuseIPDB, GreyNoise, Feodo Tracker,
and ThreatFox per Appendix A.10. Each rule's predicate inspects payload
fields produced by the enrich worker (no DB I/O, no decnet.intel.*
imports — E.2.7 decoupling guard preserved). AbuseIPDB confidence is
scaled by abuse_confidence_score / 100; categories drive per-technique
fan-out. R0058 aggregate-bump is a no-op in v0 (cross-tag bump deferred
to E.3.14 worker bootstrap).
Per-provider null tolerance is the steady state — a missing provider
column produces zero tags from that rule, never an error.
Tests:
- tests/ttp/test_intel_lifter.py — per-provider positive + negative +
state modulation + decoupling source-import guard.
- tests/ttp/rule_precision/test_intel_rules.py — xfail flipped, real
precision driven over seed_intel.jsonl (R0054-R0057 H-band ≥95%;
R0058 skipped as bump-only).
- tests/ttp/test_lifter_absence.py — IntelLifter all-populated test
flipped from xfail-strict to real assertion with realistic payload.
- tests/ttp/test_lifters.py — partial-null xfail flipped to real
assertion.
Reads pre-shaped session aggregates from TaggerEvent.payload and emits
techniques per Appendix A behavior tables. Per-rule predicates dispatch
on match.kind (lifter:behavioral_<name>); the lifter holds its own
RuleIndex watching the same RuleStore as the engine, so disable / clip /
TTL state reaches lifter-bound rules through the same atomic-swap path.
R0032/R0036/R0037/R0040 YAMLs had over-escaped regex strings (\\
instead of \\) — fixed in place.
Factory wired so default get_tagger() returns CompositeTagger with
BehavioralLifter shipped; remaining three lifters (E.3.10-E.3.12) land
in subsequent commits.
E.2.6 contract preserved via TolerantTagger: empty payload steady-state
yields [] with zero ERROR records. Disabled / clipped / expired state
verified.
E.3.9.1 prerequisite. Rules R0031-R0040 now use lifter:behavioral_*,
R0041 (open_relay) uses lifter:email_open_relay; the rest of the email,
canary, and intel cohorts already conformed. Each lifter at E.3.9-E.3.12
will claim its rules via str.startswith('lifter:<owner>_'), keeping the
ownership routing explicit and trivially extensible.
R0001-R0006 / R0030 lifter:* rules are E.3.13 (Identity/Credential)
territory and stay as-is.
E.3.9.0 prerequisite for the per-source lifters (E.3.9-E.3.13). The
dispatch index, install/evict/apply_change atomic-swap protocol, and
state-modulation helpers (is_active / apply_ceiling) move out of
rule_engine.py into _rule_index.py and _state.py. RuleEngine wraps a
RuleIndex; back-compat shims preserve _by_kind / _by_rule / _install
attribute access for tests poking at the dispatch internals.
Lifters in E.3.9-E.3.12 will each hold their own RuleIndex, watching
the same RuleStore via subscribe_changes() fan-out. Hot-reload
semantics (disable / clip / TTL via set_state API) now reach
lifter-bound rules through the same atomic-swap path the engine uses,
not a future composite-rebuild compromise.
5 YAMLs for the intel-verdict cohort per Appendix B / A.10:
AbuseIPDB category mapping, GreyNoise classification, Feodo
Tracker hit, ThreatFox IOC type, aggregate-malicious bump-only.
IntelLifter (E.3.10) consumes by rule_id and tolerates absence
silently (null provider column → no tag).
R0058 is the meta bump-only rule — emits a single confidence=0.0
sentinel so it validates and surfaces in the catalogue, but the
repository's sub-0.3 drop ensures no fresh tag persists if the
fanout fires accidentally. test_intel_rules.py pins that
zero-confidence invariant.
Marks E.3.8 done in development/TTP_TAGGING.md with the cohort-
split summary.
5 YAMLs for the canary-fingerprint cohort per Appendix B / A.9:
navigator.webdriver flag, automation canvas/audio/WebGL hash match,
WebRTC IP leak, TZ/lang vs geo mismatch, platform inconsistency.
CanaryFingerprintLifter (E.3.11) consumes by rule_id.
test_canary_rules.py: YAML-present + inert-in-v0 + xfail(strict)
gated on E.3.11.
10 YAMLs for the behavioral / cross-event cohort per Appendix B:
beaconing, data destruction, ransom note, web exfil, DB mass-read,
credentials-in-files, k8s SA token harvest, Docker host escape,
LLMNR poisoning, TFTP router-config retrieval.
Every rule is lifter-bound (BehavioralLifter / IdentityLifter) —
the v0 RuleEngine cannot count, aggregate, or compose cross-event
signals, so these YAMLs declare the technique mappings the lifter
will consume by rule_id at E.3.9. Their match specs use a
'kind: lifter:*' shape inert to the regex matcher.
test_behavioral_rules.py asserts each YAML compiles, none fire
from the v0 engine (FP regression guard against a YAML drifting
into a regex), and an xfail(strict=True, reason='impl phase E.3.9')
precision case that will flip green when the lifter lands.
30 YAMLs for the shell/command rule cohort per Appendix B (rules/ttp/).
Splits into engine-active (R0007-R0029, regex on command_text /
raw_url / user_agent) and lifter-bound (R0001-R0006, R0030 — the
v0 RuleEngine cannot count auth attempts, do identity rollups, or
parse fingerprint blobs; the BehavioralLifter / IdentityLifter /
CredentialLifter consume them by rule_id at E.3.9 / E.3.13).
test_command_rules.py asserts:
- every R000N has a YAML that compiles
- lifter-bound rules NEVER fire from the v0 engine (regression
guard against a YAML drifting into a regex match.spec)
- engine-active rules meet their Appendix-C precision target
against the seed corpus (≥0.95 high-conf, ≥0.80 medium)
Conftest fixes: precision_engine moved to module-scope so module-
scope precomputed dispatch fixture (fired_by_label) can request it;
_RULES_DIR path bumped from parents[2] to parents[3] so the loader
resolves the project root regardless of pytest cwd; make_event
synthesizes attacker_uuid so TTPTag's anchor invariant is satisfied.
Seed corpus broadened: positive examples for every regex rule plus
6 negative examples across innocuous shell verbs (ls, echo, cd, ps,
df, free) so FPs surface in precision rather than passing vacuously.
Sub-step preceding the rule-pack commits per TTP_TAGGING.md:2967.
Adds the per-rule precision suite scaffolding under
tests/ttp/rule_precision/:
- conftest.py: precision_engine fixture (RuleEngine populated from
./rules/ttp/), corpus_loader (real → seed → empty fallback),
precision_for() helper for TP/FP accounting.
- _build_corpus.py: extractor for a real prod corpus pull. Mandatory
--exclude-ip / DECNET_TTP_CORPUS_EXCLUDE_IPS — operator IPs never
end up in the committed exclusion list. Pulls both 'command' and
'unknown_command' event types.
- corpus/seed_*.jsonl: synthetic seed rows for each cohort so the
harness exercises in clean checkouts.
- corpus/*.jsonl (operator-built) is gitignored.
- test_corpus_loads.py: sentinel that every seed file parses.
Implements the rule engine body left empty at contract phase: evaluate()
dispatches by source_kind through self._by_kind, runs the rule's match
spec against event.payload, and emits one TTPTag per emits entry.
watch_store() loads the initial corpus from RuleStore.load_compiled,
then drains subscribe_changes, applying definition changes via
single-statement dict assignment (atomic swap, GIL-atomic to readers)
and state changes via NamedTuple._replace on the existing CompiledRule.
Why: with the FS + DB stores in place (E.3.5/E.3.6), the engine is the
last piece of the rule plane. Lifters (E.3.9–E.3.13) consume the
engine; the worker bootstrap (E.3.14) wires watch_store into the
asyncio event loop. After this commit a CompositeTagger constructed
with a RuleEngine + a populated rules dir will produce real tags.
Notes:
- CompiledRule.emits extended to 4-tuple
(technique_id, sub_technique_id, tactic, confidence). Tactic + confidence
ride per-emit so a single rule can carry multiple precision targets
(the "one event maps to many techniques" property). Compile helpers in
both backends extract them from the YAML emits dict; missing tactic
or confidence is a deploy-time error.
- v0 match operator is "pattern" (regex). The field defaults per
source_kind (command_text / raw_url / subject / verdict / …) and is
overridable via match.field. Future ops (contains, equals, in_set)
extend _match_event without touching the engine surface.
- Confidence model: rules with state="clipped" + confidence_max set
cap the per-emit confidence downward; clipped is a soft suppress, not
a hard skip. Disabled rules are skipped wholly; expires_at past is
re-checked at evaluate as defense-in-depth (the store auto-reverts,
but a racing read between expiry and revert must not fire the rule).
- _span(name, **attrs) helper in engine + both stores short-circuits on
decnet.telemetry._ENABLED — matches the project's @traced /
wrap_repository zero-overhead-when-disabled pattern instead of relying
solely on the no-op tracer indirection.
- Late-bound tracer (telemetry.get_tracer called per-span, not at
module load) so test_tracing's monkeypatch reaches the production
code path.
xfails flipped: tests/ttp/test_rule_engine.py multi-emit fan-out +
rule_version-collision-via-engine; tests/ttp/test_multi_mapping.py
N×M engine fan-out + idempotent replay; tests/ttp/test_tracing.py
ttp.eval span hierarchy + ttp.rule.fire span attributes.
Tests: 214 passed, 19 xfailed (gated on E.3.8 lifters / rule pack /
worker bootstrap).
mypy: clean on prod code; pre-existing test-stub arg-type warnings
unchanged.
Implements the DB-backed rule store body left empty at contract phase:
load_compiled reads from ttp_rule + ttp_rule_state; get_state /
set_state hit ttp_rule_state with the same expires_at auto-revert and
bus-event semantics as the FS backend; subscribe_changes returns a
per-subscriber queue. State persists across process restarts — the
swarm property the FS backend deliberately doesn't have.
Also lands two swarm-mode helpers:
- sync_from_filesystem(fs_store) — master-side, subscribes to a
FilesystemRuleStore and projects each RuleChange onto a ttp_rule
upsert/delete.
- tail_db(poll_interval) — worker-side, watermark poll over
ttp_rule.updated_at; emits RuleChange("definition", ...) for each
row that moved.
Why: swarm mode needs rule definitions and operator state to
propagate across hosts. The filesystem backend (E.3.5) was the
single-host-dev variant; this one survives restart and serves N
workers from a shared DB.
Notes:
- DatabaseRuleStore() with no args lazy-inits an in-memory SQLite
repo so the conformance fixture works without test plumbing. In
production the worker bootstrap (E.3.14) passes an explicit repo.
- The conftest.py rule_store fixture became async (pytest_asyncio),
per-backend creates/initializes a SQLite repo for the DB run.
- Adds a `seed_rule(store, rule_id, yaml)` helper to bridge backend
semantics: drop a YAML file (FS) vs insert a ttp_rule row (DB).
Used by the parametrized load_compiled conformance test.
- Late-bound _tracer() in both backends (was module-level get_tracer
binding) so test_tracing's monkeypatch of decnet.telemetry.get_tracer
actually affects span output.
xfails flipped: tests/ttp/store/test_database.py set_state-writes-to-
ttp_rule_state + filesystem-to-DB sync; tests/ttp/store/test_conformance.py
DB-side load_compiled / set_state isolation / round-trip / per-rule
fan-out / expired-state revert / set_state failure / get_state default
(was xfail-only-on-DB); tests/ttp/test_tracing.py set_state span
hierarchy.
Tests: 208 passed, 25 xfailed (gated on E.3.7 + lifters).
mypy: clean on all touched files.
Implements the filesystem-backed rule store body left empty at contract
phase: YAML parse + Pydantic validation, asyncinotify watch over
./rules/ttp/, in-process state cache with auto-revert on expires_at,
and a subscribe_changes() async iterator yielding one RuleChange per
per-rule edit. Bus topic builders ttp_rule_reloaded / ttp_rule_state
ship alongside.
Why: the rule plane needed a store before the engine (E.3.7) could
consume RuleChange events and atomically swap compiled rules into its
dispatch index.
Notes:
- Linux-only by construction (asyncinotify wheel gated by sys_platform
marker; FilesystemRuleStore.__init__ raises on non-Linux).
- Filename allowlist is the FIRST check on every inotify event.
- Content-hash dedup so a single write firing IN_CREATE + IN_CLOSE_WRITE
produces exactly one RuleChange.
- All compile work serializes on a single asyncio.Lock.
- Subscribers register their queue eagerly so events fired between
subscribe_changes() and the first __anext__() are buffered.
xfails flipped: per-save-style + filter-ordering + atomic-swap in
test_filesystem.py; load_compiled / set_state isolation / round-trip /
per-rule fan-out / expired-state revert / set_state failure semantics
in test_conformance.py (FS side; DB side stays xfail until E.3.6);
malformed-YAML compile-time check in test_rule_engine.py.
Tests: 197 passed, 35 xfailed (gated on E.3.6 / E.3.7 / lifters).
mypy + bandit: clean on all touched files.
Wiki update for the per-rule reload + state-change topics lands in a
matching wiki-checkout/Service-Bus.md edit (separate repo).
Five GET rollup endpoints (techniques, by-identity, by-attacker,
by-campaign, by-session) and the Navigator export (fleet +
per-identity) now call into the TTPMixin methods. Rule catalogue
endpoint still returns [] — backed by the RuleStore which lands
at E.3.5/E.3.6.
Dialect-split: portable rollup queries on TTPMixin; bulk insert with
ON CONFLICT DO NOTHING / INSERT IGNORE in the per-dialect repos.
Confidence-floor (< 0.3) drop applied at mixin layer before the
dialect hook. BaseRepository now declares the six TTP methods abstract.
Tests in tests/web/db/test_ttp_repo.py flipped from pytest.fail stubs
to real dual-backend behavioral tests; tests/ttp/test_confidence.py
drop-below-floor xfail removed.
Each section gets a Status: ✅ done block summarising what's GREEN
today vs xfail-gated and noting any divergence from the doc's
original wording (E.2.9 lossy observable phases; E.2.13 db_backends
fixture landed alongside; E.2.14a Jaeger-skip + tracing-enabled
plumbing; E.2.14b NamedTuple AttributeError vs FrozenInstanceError).
Session-scoped autouse fixture in tests/ttp/conftest.py sets
DECNET_DEVELOPER_TRACING=true and forces decnet.telemetry._ENABLED
so the no-op tracer doesn't silently swallow emitted spans. The
span_exporter fixture also monkeypatches decnet.telemetry.get_tracer
so production code under test lands spans in the in-memory
exporter. Tracing tests skip when DECNET_OTEL_ENDPOINT (default
localhost:4317) isn't reachable so the dev loop stays green
without lying about coverage.
In-memory span exporter fixture wired to a per-test TracerProvider
(OTEL global is locked once set, so each test gets its own).
ttp.eval / ttp.lifter.{name} / ttp.rule.fire / ttp.rule.state.change
hierarchy + no-PII canary battery xfail-gated behind E.3.5–E.3.13.
Hypothesis property: N rule_ids × M technique_ids on one event yield
N×M distinct tag UUIDs. Worked example pinned: one rule emitting
(T1110, None) and (T1078, None) → two distinct UUIDs. Engine-level
fan-out + replay xfail-gated behind E.3.7.
Pre-target phases (RECONNAISSANCE/RESOURCE_DEVELOPMENT/WEAPONIZATION/
SOCIAL_ENGINEERING) and observable-but-unmappable phases (EXPLOITATION/
PIVOTING/OBJECTIVES, UKC-only concepts ATT&CK lacks tactics for) are
pinned as lossy via _LOSSY_INVERSE_REFERENCE so a future contributor
cannot 'fix' the asymmetry without tripping the suite.
Adds decnet/ttp/store/ subpackage:
- base.py: RuleState frozen dataclass, RuleChange NamedTuple, RuleStore ABC
- factory.py: get_rule_store() reading DECNET_TTP_RULE_STORE_TYPE
- impl/filesystem.py: FilesystemRuleStore with sys.platform=='linux'
fail-fast guard, allowlist filename regex, raw inotify mask bits
(lib import deferred to E.3 so contract phase compiles without the
asyncinotify dep installed)
- impl/database.py: DatabaseRuleStore stub (no platform guard)
TTPRule + TTPRuleState SQLModels were already shipped at E.1.1; this
commit closes the type-only TYPE_CHECKING forward-ref in
rule_engine.py via real runtime imports through the new package.
Empty NotImplementedError bodies; the SQL lands at E.3 implementation.
Mixin composed onto SQLModelRepository alongside the existing domain
mixins. Dialect-specific INSERT-OR-IGNORE syntax overrides land in
the per-backend subclasses at E.3 per the dual-DB-backend convention.
Mounts /api/v1/ttp/* with empty-list / empty-Navigator responses.
GET endpoints viewer-gated; POST/DELETE /rules/{rule_id}/state
admin-gated server-side. POST parses JSON manually so a malformed
body returns the documented 400 (per feedback_schemathesis_400).
Drops xfail-strict markers from E.2.8 tests now that the router is
mounted; 26 tests pass against the contract handlers.
Third and fourth TTP-tagging contract commits, plus a scoped subset
of the E.2.4 conformance tests covering the contract surface shipped
here (full hypothesis-fuzz suite still lands with E.2.4).
E.1.3 — decnet/ttp/base.py
- TaggerEvent NamedTuple: source_kind, source_id, attacker_uuid,
identity_uuid, session_id, decky_id, opaque payload.
- Tagger(ABC) with abstract async tag(); class-level name and
HANDLES: frozenset[str] (default empty so a misconfigured subclass
is loudly idle, not loudly noisy).
- TolerantTagger(Tagger): concrete tag() wraps abstract _tag_impl()
in try/except Exception (deliberately not BaseException — so
KeyboardInterrupt / SystemExit / asyncio.CancelledError propagate
and the worker can shut down cleanly). Swallowed exceptions log
at WARNING with exc_info, never ERROR — absence is the steady
state, not a bug. Subclasses override _tag_impl, never tag — the
tolerance contract is enforced in the base class, not on trust.
- KNOWN_SOURCE_KINDS: Final[frozenset[str]] enumerating every
source_kind a producer is allowed to emit. Closed-by-enumeration
at the runtime layer; the composite tagger keys its WARNING/INFO
bridge off this constant to surface the silent-drop trap from
the design doc (lines 160–195).
E.1.4 — decnet/ttp/factory.py
- get_tagger() reads DECNET_TTP_TAGGER_TYPE (default 'composite');
unknown values raise ValueError with the known-list. Mirrors
decnet.intel.factory and decnet.clustering.factory.
- _KNOWN = ('composite',). Per-lifter classes (E.1.6) are children
of the composite, not standalone tagger types.
- CompositeTagger(Tagger): pre-computes a dict[str, list[Tagger]]
dispatch index from each lifter's HANDLES; fans events out
concurrently with asyncio.gather and concatenates results.
Empty lifters=[] is the legal contract-phase state — E.1.6
wires the real lifters in.
- Unhandled-event observability: source_kind in KNOWN_SOURCE_KINDS
but no lifter claims it -> WARNING once per kind per process
(missed E.1.6 update). Unknown kind -> INFO once per kind per
process (future-feature telemetry, by design). Per-process dedup
via plain set; E.1.6 may swap in a proper rate-limiter once
production traffic shapes are known.
Tests — tests/ttp/test_base.py, tests/ttp/test_factory.py
- Tagger / TolerantTagger abstractness, missing-tag-impl rejection,
WARNING-not-ERROR log level, propagation of KeyboardInterrupt /
SystemExit / asyncio.CancelledError.
- Factory env-var routing, unknown-name ValueError, dispatch-index
correctness, only-claiming-lifter invocation, WARNING-once for
known-but-unclaimed kinds, INFO-once for unknown kinds, result
concatenation across lifters.
Mypy clean under .311/bin/mypy --ignore-missing-imports.
Second TTP-tagging contract commit. Constants only — no publishers,
no subscribers, no tests. (E.2.3 ships the bus-topic naming tests.)
- New roots: EMAIL, TTP.
- New leaves: EMAIL_RECEIVED ('received', single-token under EMAIL),
TTP_TAGGED ('tagged'), TTP_RULE_FIRED ('rule.fired'),
TTP_RULE_SUPPRESSED ('rule.suppressed'). Per-rule reload + state
topics ship with the RuleStore (E.1.11) — co-located with
producer.
- New builders: email_topic(event_type), ttp(event_type),
ttp_rule_fired(technique_id). The ttp_rule_fired builder validates
technique_id as a single segment so sub-techniques like T1110.001
are rejected at construction; topic key is the parent technique,
sub_technique lives in the payload.
- email_topic is named with the _topic suffix to avoid shadowing the
Python email stdlib at import sites that pull both.
- TTP_TAGGING.md E.1.2 entry corrected: the spec referenced
'ATTACKER_ENRICHED' but the actual constant is
ATTACKER_INTEL_ENRICHED ('intel.enriched'). The existing constant
covers the design intent (TTP intel_lifter wakes on
attacker.intel.enriched). No rename — would break every existing
subscriber.
Wiki update for the four new topics ships in a sibling commit in
wiki-checkout (separate repo per project layout).
First contract commit of TTP tagging. Shapes only — no behavior.
- TTPTag SQLModel: deterministic UUIDv5 PK; (source_kind, source_id)
discriminated provenance; nullable attacker_uuid + identity_uuid
with ON DELETE CASCADE; native sqlalchemy.JSON evidence column;
required attack_release; CheckConstraint('attacker_uuid IS NOT
NULL OR identity_uuid IS NOT NULL'); composite indexes for the
primary query patterns (identity_uuid+technique_id,
attacker_uuid+technique_id, technique_id+created_at); __init__
guard raising ValueError with both anchor names in the message
(belt-and-braces for MySQL <8.0.16 where CHECK is silent).
- compute_tag_uuid(): RFC-4122 UUIDv5 over the six tag-identity
fields under a fixed _TTP_TAG_NS. Pure, deterministic, replay-safe.
- Per-source_kind evidence TypedDicts (CommandEvidence,
IntelEvidence, EmailEvidence, CanaryFingerprintEvidence) — PII
rule lives in the type: EmailEvidence has no field for raw rcpt
addresses or body bytes.
- TTPRule + TTPRuleState tables for the DatabaseRuleStore (E.1.11).
- All symbols re-exported from decnet.web.db.models per the
package's existing convention.
Tests for invariants (CHECK behavior, evidence round-trip across
SQLite+MySQL, idempotency property, init-guard ordering) land in
E.2.1/E.2.2 with xfail-strict markers per Appendix E discipline.
Pre-implementation spec for the TTP-tagging worker. Defines the
ATT&CK-canonical vocabulary, schema (ttp_tag + ttp_rule[_state]),
bus topics, worker shape, lifter layering (rule-based v0,
behavioral/intel/email v0.5, sigma/biometric later), confidence
model, API surface, UI surface, observability, performance targets,
and a CDD plan (Appendix E) that splits contracts from tests with
xfail discipline so CI stays green between steps.
- Add _MixinBase abstract class to _helpers.py: declares _session(),
_deserialize_attacker(), _assert_pending(), _check_and_bump_version(),
and list_running_topology_deckies() so mypy can see cross-mixin contracts
- Add _require(val, msg) helper for narrowing T | None → T
- Inherit _MixinBase in all 26 leaf mixin classes
- Wrap SQLAlchemy column method calls (.is_(), .like(), .notin_(), .in_(),
.contains()) with col() from sqlmodel — fixes attr-defined false positives
caused by pydantic plugin typing class-level fields as Python value types
- Wrap select(Model.field) with select(col(Model.field)) for column projections
- Add pyproject.toml [[tool.mypy.overrides]] to disable arg-type in
sqlmodel_repo.*: pydantic plugin resolves .where(Model.field == v) as
where(bool), a false positive; call-arg still catches real argument errors
- Remove 9 stale # type: ignore comments (logging, helpers, credentials)
- Fix telemetry.py traced() overload no-redef + misc
- Fix logs.py datetime/str operator and nullable PK comparison with col()
- sqlmodel_repo/ now has 0 mypy errors
syslog_bridge.py: base64.binascii is not a public mypy-visible attribute;
import binascii directly and reference binascii.Error at the except clause.
Propagated to all 26 template subdirectory copies (all were drift-free).
ntlmssp.py: `principal = username or None` widened the type to str | None
for no runtime reason — _decode_str() always returns str. Drop the `or None`.
Propagated to smb/ and rdp/ copies.
762 → 722 mypy errors (-40).
Replace repo: BaseRepository with a structural TopologyRepository protocol
in persistence.py and allocator.py. All read methods now return typed DTOs
(TopologySummary, LANRow, DeckyRow, EdgeRow) instead of raw dicts, eliminating
silent field-shape regressions across the topology subsystem.
TopologySummary gains email_personas and language_default so api_personas.py
can continue reading those fields via attribute access. hydrate() converts
DTOs to dicts before passing to _backfill_decky_configs, keeping the mutable
working-state function dict-based at its boundary. All production callers
(router handlers, mutator, CLI, heartbeat) migrated from dict/get access to
attribute access. 134 tests pass.
MutationRow.op was str despite _MUTATION_OPS existing; Topology.mode/status,
TopologyDecky.state, TopologyMutation.op/state carried valid values only in
comments; deferred json import had no justification.
- Promote _MUTATION_OPS before table classes so table fields can reference it
- Add sa_column=Column(String) on each Literal-annotated table field to satisfy
SQLModel 0.0.38 column-type inference
- Move import json to module top; remove deferred import inside _decode_json_payload
- MutationRow.op: str -> _MUTATION_OPS
[swarm] swarmctl-host → DECNET_SWARMCTL_HOST so operators set the bind
address once in decnet.ini; `decnet swarmctl` and the systemd unit both
resolve it via envvar — no --host/--port pinned on ExecStart.
ApiError: defined once in utils/api.ts, replaces 9 ad-hoc anonymous casts
across MazeNET, Inspector, DeckyFleet, SwarmHosts, Webhooks, PersonaGeneration,
ServiceConfigFields, CanaryTokens.
hex4 renamed to tempIdSuffix — the name now matches the comment that already
explained its purpose.
NET_GRID_{W,H,GAP,COLS} extracted from inline magic numbers to module-level
constants in MazeNET.tsx.
onPaletteDrop (130-line useCallback) split into three module-level handlers
(_dropNetwork, _dropArchetype, _dropService); the callback becomes a 10-line
router.
The 17-line block comment at _RULES was prose covering for a design wart.
The explanation belongs on the function itself — moved there and condensed.
_RULES now has a 2-line pointer instead of an essay.
topology_id[:8] appeared twice with no explanation. 8 chars is the
git short-SHA convention; collision-safe within a single deployment's
network namespace.
fix(generator): correct service pool count in _SVC_MIN/_SVC_MAX comment
BLE001 is not in ruff.toml select (F/ANN/RUF/E/W only); the suppressions
were whispering apologies to a linter that wasn't listening. Generator
comment now cites the actual ~28-entry non-singleton service pool.
apply() was an 85-line function handling hash verification, validation,
superseding teardown, bridge/compose provisioning, and store persistence.
Extracted _check_hash_and_validate(), _teardown_superseded(), and _materialise()
so each step is independently readable and testable.
_take_ip and _new_decky were closures capturing outer-scope state. Promoted to
module-level with explicit parameters. seen_service_pairs name makes the intent
clear — it prevents the same service frozenset from being assigned repeatedly.
The original except Exception silently disabled port collision detection for
any runtime error — not just a missing package. Now only ImportError degrades
gracefully; real psutil failures propagate.
_host_set is computed once in __init__ — reserve() and is_free() were rebuilding
the full host frozenset on every call. BaseRepository already existed; the Any
annotations were just never updated.
await inside a threading.Lock yields to the event loop while the OS
thread still holds the lock — potential deadlock under FastAPI thread
pool dispatch. asyncio.Lock is the correct primitive for async
critical sections. Also fixed stale diurnal.py docstring that had the
delegation direction backwards.
_parse_weights was silently dropping content_class values that don't
belong on their target list with no operator feedback. Changed it to
return (weights, dropped), apply_payload to collect and return all
dropped names, and put_config to include dropped_entries in the
response when non-empty.
The initial stat and read happened without a lock between them. A file
change mid-window stored the mtime of the pre-change stat against the
post-change content, suppressing the next reload. Re-stat after
read_text; fall back to the pre-read stat only on OSError.
The persona arg was typed Any to avoid a circular import. Added a
TYPE_CHECKING guard to import EmailPersona annotation-only so mypy
has the type without a runtime import cycle.
get_config was calling planner.apply_payload on every GET request, racing
concurrent reads on module-level globals. Added a _hydrated flag + lock
so DB hydration runs at most once per process lifetime; put_config marks
it done too. Test fixture resets the flag between tests.
Concurrent PUT requests could observe a half-updated planner between
the four sequential global assignments. Added _planner_lock so the
rebind is atomic; same lock wraps reset_to_defaults.
personas.in_active_hours was discarding the minute component of the
active-hours window, making "09:30-17:45" behave as "09:00-17:00".
Rewrote it to delegate to diurnal.in_work_hours (which uses full
minute arithmetic) and updated the scheduler caller to pass the full
datetime instead of now_dt.hour.
ServiceNotFoundError (→ 404) and ServiceConflictError (→ 409) replace the
"not found" / "already on" / "not on" substring checks in _map_mutation_error;
base ServiceMutationError still maps to 422. Fixes three pre-existing test
status-code assertions (201 vs 200 on POST endpoints).
Pure tarball construction (_build_tarball, _render_*, _iter_included,
_SYSTEMD_UNITS) moved to decnet/swarm/bundle_builder.py — no FastAPI
dependency, independently testable. EnrollBundleRequest/Response moved
to decnet/web/db/models/swarm.py alongside the other swarm DTOs.
Router drops from 504 to 260 lines; keeps only the in-memory token
registry, sweeper, and endpoints.
MailDrawer was reading fields.date / from_addr / message_id —
all wrong; actual log field names are date_hdr, from_hdr,
message_id_hdr, to_hdr. The mail table in AttackerDetail
showed only DECNET capture time and used from_addr instead
of from_hdr. Add a DATE (attacker) column so the attacker-
supplied Date header (including timezone) is visible at a
glance — useful for correlating campaigns like the Tiscali
run where IPs used distinct TZs (+0800 vs -0700).
Relay-test scripts send minimal DATA with no headers. Without a From:
header the mail client falls back to displaying the envelope sender
(upstream_sender). Inject From: <attacker MAIL FROM> before forwarding
when the message has no existing From: header.
bus.subscribe() is sync and returns an async iterator, not a coroutine.
Awaiting it caused an immediate crash at startup; bus.next_message() does
not exist either. Rewrote _run_smtp_probe_listener to use the standard
pattern: sub = bus.subscribe(...) / async with sub / async for event in sub.
SERVICE_NAME was hardcoded to 'smtp' in server.py; the ingester's probe
publish guard checked service == 'smtp_relay' and never matched.
Read SMTP_SERVICE_NAME from env (default 'smtp'); smtp_relay compose
fragment sets it to 'smtp_relay' so the two services are distinguishable.
The bind-mounted quarantine dir is owned by the host decnet user; the
logrelay process had no write access because the Dockerfile USER directive
pre-applied before the entrypoint could fix permissions.
Run entrypoint as root, chmod 0777 the quarantine dir, then exec the
server under logrelay via su.
Attacker probe emails are now forwarded by the master (realism worker)
rather than inside the MACVLAN container, which has no internet gateway.
- New smtp.probe.pending bus topic: ingester publishes when smtp_relay
message_stored fires; worker subscribes and does the actual delivery
- decnet/orchestrator/drivers/smtp_relay.py: pure-sync forward_probe()
reads the .eml from disk and sends via smtplib on a thread executor
- worker.py: _run_smtp_probe_listener + _handle_probe_pending subtask;
limit enforced via count_probe_relays() (DB-backed, restart-safe)
- bounties.py: count_probe_relays() query on probe_relay bounty type
- fleet.py: get_fleet_decky_by_name() to pull service config from DB
- services/smtp_relay.py: upstream_* and probe_limit fields defined in
config_schema but NOT injected into container env (credentials stay
out of docker env vars)
- ingester.py: stripped of smtplib; publishes probe.pending and exits
- tests: assert upstream keys absent from container environment
forwarded=0 was silent — now fwd_error carries the exception string so
you can see exactly why the upstream refused (auth failure, connection
refused, timeout, etc).
Docker Compose tracks the previous container by internal ID. When that
container was already removed or renamed, --force-recreate fails with
"No such container". Remove by name first so Compose always starts clean.
Override the envelope MAIL FROM with a domain we own when talking to the
upstream relay. SPF passes at the recipient; the attacker's From: header
inside the message body is untouched so they see their own address in their
inbox and believe the relay is real.
Adds probe_forwarded to meaningful event kinds and stores it in the
bounty table as bounty_type=probe_relay with forwarded=true/false, so
the dashboard shows whether the upstream actually accepted the test email.
First SMTP_PROBE_LIMIT messages per source IP are forwarded via a real
upstream relay (SMTP_UPSTREAM_HOST/PORT/USER/PASS) so the attacker's
test email actually lands in their inbox. All subsequent messages from
the same IP get 250 Ok but only hit the quarantine — campaign content
captured, nothing delivered.
The Dockerfile PROMPT_COMMAND logger uses --msgid command, so the MSGID
field arrives as 'command' not '-'. The CMD rewrite block was guarded by
event_type == '-' so it never fired, leaving fields['command'] unpopulated
and cmd_text=None for every SSH session command.
Broaden the guard to also match event_type == 'command' with no existing
'command' field, which covers both the intended (MSGID=NIL) and actual
(MSGID=command) wire formats.
.content-viewport is overflow-y: auto so flex:1 on dash-grid grew to
content height. Fix: dashboard uses height:100% instead of min-height,
and :has(>.dashboard) disables content-viewport scroll only on that
route — all other pages keep their normal scroll.
Sticky thead was floating mid-content when the container auto-scrolled
as new log entries arrived. Pinning scrollTop to 0 on each logs update
keeps the thead at position 0 where it belongs.
Use flex: 1 on dash-grid instead of height: 480px so the panels
consume all remaining space below the stat cards; dash-side uses
height: 100% to fill its grid cell
- Use height: 480px on .dash-grid so both columns are the same height;
side panels split that height via flex instead of their own max-height
- Add LayoutDashboard icon to the DASHBOARD page header
- Dashboard: fix invisible bar at bottom of LIVE FEED by constraining
max-height on the section instead of the inner container; same fix
for side panels
- Page icons: add violet-accent icon beside h1 on all 9 missing pages
(CanaryTokens, RealismConfig, SyntheticFiles, PersonaGeneration,
Attackers, Webhooks, LiveLogs, Topologies, DecoyFleet)
- Attackers filter chips: replace ad-hoc chip buttons with seg-group
tabs (ALL / ACTIVE N / PASSIVE N / INACTIVE N) matching Credential
Vault style; country chips use same seg-group treatment
- Credential Vault: add sortable headers to REUSE tab (LAST SEEN,
PRINCIPAL, KIND, TARGETS, ATTEMPTS); reuses same SortTh pattern
- Bounty: remove CREDENTIALS and PAYLOADS tabs; keep ALL, ARTIFACTS,
FINGERPRINTS; add EMAIL (artifact subtype, filtered client-side)
- DeckyFleet: card click opens inspect side-drawer instead of
auto-filtering (localSearch filter behavior removed)
- Dashboard: LIVE FEED / DECKIES UNDER SIEGE / TOP ATTACKERS panels
now have fixed max-height with overflow scroll instead of growing
- parseEventBody: defensive RFC 5424 header strip so raw syslog lines
from the collector render as k=v pills instead of raw text
- Attackers: search placeholder updated; activity (Active/Passive/
Inactive) and country chip filters added on top of existing IP search
- Credentials + Bounty: sortable column headers (click to asc/desc/clear)
- SwarmHosts + RemoteUpdates: icon extracted from <h1> into flex div
with violet-accent class, matching site-wide Identities pattern
- Swarm.css: fix --panel-border undefined variable → --border so the
title border-bottom line is visible on SwarmHosts and RemoteUpdates
Monkeypatching services_live.add_service had no effect because api_services
already held a local reference to the name. Patch api_services.add_service
and update fake stubs to accept the config kwarg added to the real signature.
Once a fingerprint canary's HTTP beacon passes all 4 validation layers
and the trigger row lands, the token is immediately set to state=revoked
and canary.<id>.revoked is published on the bus. The slug lookup is
tightened to only return planted tokens, so subsequent requests to the
same URL silently return the transparent GIF without persisting anything
(stealth posture preserved). Plain http/dns canaries with no
fingerprint_nonce are not affected.
Changes:
- sqlmodel_repo/canary.py: add state == "planted" filter to
get_canary_token_by_slug so revoked slugs resolve to None
- worker.py: after record_canary_trigger, if parsed_fp survived all
layers and token has a fingerprint_nonce, call
update_canary_token_state("revoked") + publish CANARY_REVOKED; errors
are best-effort (trigger row already landed)
- test_worker_http.py: assert state=revoked in test_fp_valid_nonce_persists;
new test_fp_deregisters_slug_after_valid_hit (second hit records nothing);
new test_plain_http_canary_not_deregistered (env_file stays planted)
Adds per-mint nonce gating, structural shape validation, mint UUID
consistency checks, and a per-(token, IP) rate limiter to the canary
worker so attackers who extract a canary from a decky filesystem cannot
poison fingerprint forensics by replaying or forging ?d= submissions.
Changes:
base.py
fingerprint_nonce: Optional[str] added to CanaryArtifact so generators
can surface the nonce to the cultivator without coupling the generator
directly to DB code.
obfuscator.py
nonce_for(callback_token, mint_uuid): HMAC-SHA256 keyed on
DECNET_CANARY_FINGERPRINT_SECRET, truncated to 16 hex chars.
FingerprintSecretMissing raised at mint time if env var is unset.
render_fingerprint_js() now accepts nonce= and substitutes MINT_NONCE.
fingerprint_payload.js
New MINT_NONCE placeholder. Appended as &k= on all beacon URLs (bare-open,
single-shot, chunked). Using &k= avoids colliding with &n= (chunk total).
fingerprint_html.py / fingerprint_svg.py
Derive nonce via nonce_for() and pass to render_fingerprint_js(). Set
artifact.fingerprint_nonce so the cultivator can persist it.
cultivator.py
Passes fingerprint_nonce into create_canary_token() when present on the
artifact; NULL for all non-fingerprint generators.
canary.py (model)
fingerprint_nonce: Optional[str] = Field(default=None, max_length=16)
added to CanaryToken. None for non-fingerprint tokens.
worker.py
_extract_fingerprint now returns (meta_dict, parsed_fp) tuple.
_record_hit accepts parsed_fp + raw_nonce and runs 4 layers after
token lookup: nonce match, shape check, mint UUID consistency, rate limit.
Each failure sets _fp_invalid_* flag and drops structured _fp.
Trigger row always lands regardless.
tests/canary/conftest.py
Session-scoped autouse fixture sets DECNET_CANARY_FINGERPRINT_SECRET so
fingerprint generator and worker tests work offline.
tests
5 new worker HTTP tests and 2 new generator tests covering each
validation layer.
The fingerprint canaries' obfuscator shells out to a Node helper that
require()s javascript-obfuscator. Without this commit, a fresh
pip install decnet would land the .py modules but not the .js helper /
package.json, and there'd be no documented way to provision Node side.
* pyproject.toml - extend tool.setuptools.package-data to ship
canary/_obfuscate_helper.js, canary/fingerprint_payload.js, and
canary/package.json with the wheel.
* decnet/cli/canary.py - new "decnet canary-install-toolchain"
subcommand. Resolves decnet.canary.__file__'s dir, runs
npm install --omit=dev there, exits non-zero with a clear message
if npm is missing or install fails. Idempotent - safe to call
every API service start.
* deploy/decnet-api.service.j2 - non-fatal ExecStartPre that calls
the new subcommand. Leading '-' so a missing Node toolchain only
degrades fingerprint canaries (loud at mint time) without keeping
the API from booting.
* tests/canary/test_cli.py - registration smoke test, missing-npm
exit path, and a mocked-subprocess test asserting the right argv
and cwd land on npm.
Realism cultivator already has a broad except Exception around
cultivate() in scheduler.py:195-211, so a missing toolchain on a
host running the realism tick degrades to an inert noise file with
no extra plumbing.
The two new fingerprint canary generators existed at the API level
since f64e78f but weren't visible to the realism engine or the
operator-facing dashboard. Threads them through every place that
enumerates canary content classes.
Backend:
* realism/taxonomy.py - two new ContentClass members
(CANARY_FINGERPRINT_HTML, CANARY_FINGERPRINT_SVG); enum is
wire-visible (synthetic_files.content_class column + bus discrim)
so we add at the bottom, never reorder.
* canary/cultivator.py - class-to-generator dispatch, kind mapping
(both http), and default placement paths
(~/Documents/asset_directory.html and network_topology.svg).
* realism/naming.py + bodies.py - _name_canary / _body_canary entries.
* realism/planner.py - added to _DEFAULT_CANARY_CLASS_WEIGHTS and
the _CANARY_CLASSES classification set.
Frontend:
* decnet_web/src/realism/labels.ts - display labels.
* decnet_web/src/components/RealismConfig/RealismConfig.tsx - default
canary weight rows so operators see them in the realism config UI.
* decnet_web/src/components/SyntheticFiles/SyntheticFiles.tsx - added
to the CONTENT_CLASSES allow-list so filter dropdowns show them.
Also: re-applied the nosec B404/B603 markers on canary/obfuscator.py;
the first commit's pre-commit autoformatter stripped them.
Tests: extended tests/realism/test_taxonomy.py's stability assertion
to include the two new values. Full canary + realism suites pass
(362 / 2 skipped).
The fingerprint payload beacons fingerprint data as base64url JSON in
GET query params: ?o=1 for the bare-open beacon, ?d=<blob> for a
single-shot dump, or ?s/i/n/d=<chunk> for chunked dumps. Until now
those params were buried inside request_path; consumers had to parse
the URL themselves.
Worker now extracts them in _extract_fingerprint and merges into
raw_headers under reserved _fp* keys:
* _fp_open — bare-open marker
* _fp — decoded fingerprint dict (single-shot path)
* _fp_sid/idx/total/chunk — chunked metadata + raw base64 (reassembly
is a downstream concern, not the worker's job)
* _fp_decode_error / _fp_oversize — failure markers for trash dumps
Per-chunk size capped at 8KB so an attacker spamming /c/<known_slug>
can't inflate trigger rows indefinitely. Decode failures degrade
gracefully — the trigger row still records the hit, just with a
_fp_decode_error flag instead of structured fingerprint data.
Tests cover the single-shot decode, bare-open flag, chunked metadata,
malformed input, and oversize drop paths.
Two new synthesised-artifact generators that bake the obfuscated
fingerprint payload into plausible-looking decoy files:
* fingerprint_html — a mundane "Internal Asset Directory" page with a
small table of fake hosts; the obfuscated payload is inlined at the
bottom of <body>. Visible content (row pool slice, sync timestamp)
also varies per mint via SHA-256-derived stable ints, so two
extracted canaries don't diff to zero even on the rendered surface.
* fingerprint_svg — standalone SVG with an embedded <script> CDATA
block. SVG <script> only fires for top-level loads / <object> /
<iframe>; <img>-referenced renders are safely inert.
Both derive the mint UUID via uuid.uuid5 from the callback token, so
re-mints are byte-identical (preserving the generator determinism
contract) AND the same token produces the same mint UUID across HTML
and SVG variants — the worker can correlate beacons across artifact
shapes.
Wired into the factory + KNOWN_GENERATORS, default placement paths
under ~/Documents/asset_directory.html and ~/Documents/network_topology.svg
for both linux and windows personas. Tests cover determinism, per-token
divergence, structural validity (DOCTYPE/SVG headers), and that the
beacon URL stays inside the obfuscated string array (not in plaintext).
The two new entries skip in test_generators.py when Node toolchain is
absent so bare CI checkouts still pass.
Adds the load-bearing primitives for obfuscated browser-fingerprinting
canaries. Step 3 (HTML/SVG generators) and step 4 (worker-side
fingerprint ingestion) build on top of these.
* decnet/canary/obfuscator.py - javascript-obfuscator wrapper. Seed
and polymorphic config bits both derive from the callback token, so
output is byte-identical for the same mint (preserving the generator
determinism contract from base.py) and structurally distinct across
mints.
* decnet/canary/fingerprint_payload.js - port of canary-self-test.html
with the rendering UI stripped. Two placeholders (BEACON_URL,
MINT_UUID) substituted before obfuscation. MVP beacon strategy:
bare-open GET pixel first, then base64url-encoded fingerprint as
query params on subsequent GETs (chunked above ~6KB) so the existing
worker records hits before step-4 lands.
* decnet/canary/_obfuscate_helper.js - Node subprocess helper that
reads code+options JSON from stdin and writes obfuscated JS to
stdout. Vendored javascript-obfuscator under decnet/canary/.
* tests/canary/test_obfuscator.py - determinism, per-mint divergence,
template substitution, Node syntax check, error path.
Add/remove/update_config on a fleet decky living on a swarm worker — and on
an agent-pinned topology — used to run the master's local docker-compose only,
which has no containers for the remote decky. The mutation persisted on master
and silently no-op'd on the worker.
- Fleet swarm: lookup DeckyShard.host_uuid; if found, rebuild a single-host
shard from master state and call dispatch_decnet_config — same proven path
as POST /swarm/deploy. Skip local _compose (no containers to touch).
- Topology agent-pinned: call decnet.engine.deployer.resync_agent_topology
(existing helper) to push the latest hydrated blob to the worker.
- Local-only deckies: behaviour unchanged.
- Tests: 5 new in tests/engine/test_services_live_swarm.py covering all
three mutations on a swarm fleet decky (no local _compose, dispatch fires
with the right host's deckies), plus apply=False save-only path (no
dispatch), plus regression that local-only fleet add still runs local compose.
Bus signal `decky.{name}.service_config_changed` keeps publishing as an
audit trail; it is not the propagation trigger.
- DeckyServiceAddRequest gains an optional `config: dict` field, validated
against the service's config_schema before any state mutation (400 on
bad type, no half-written rows).
- Engine: add_service threads `config` into _add_topology_service /
_add_fleet_service, persisting validated cfg to decky_config.service_config
BEFORE compose regen so the first `up -d --build` materialises the env on
the new container. No follow-up apply needed.
- Frontend: shared AddServiceConfigModal — same wizard accordion shape, used by:
* DeckyCard's ADD SERVICE picker (Fleet & MazeNET inspectors via shared component)
* MazeNET Inspector's ADD SERVICE picker
* MazeNET palette drag-drop onto a deployed decky
Empty-schema services short-circuit to a one-click add (no modal flash).
Operator can cancel; errors surface in the modal.
- Tests: add_service config plumbing — persist, drop unknown keys, 400-equivalent
on bad types, back-compat empty-config.
- Drive-by: fix stale repo-method names in test_services_live.py
(create_topology_decky → add_topology_decky, get_topology_decky → list+pick helper,
service.added → service_added topic).
- Declarative config_schema on RDP, Telnet, MySQL, Redis, SMTP, SMTP_Relay
matching the keys each service already reads at compose time.
- TODO marker on the 19 services that accept service_cfg but never read it,
so future contributors know where to plug schemas in.
- Wizard base64-wraps all textarea values at INI emit (DeckyFleet
buildIni); validate_cfg detects the b64: sentinel and decodes back to
UTF-8. Plain raw strings still pass through for direct API submitters.
- HTTPS image entrypoint accepts PEM content or path in TLS_CERT/TLS_KEY:
detects a BEGIN header, writes content to /opt/tls/, and re-exports
the on-disk path so server.py keeps reading paths.
- Tests cover schema/compose alignment for each new service plus
textarea base64 round-trip (incl. UTF-8) and HTTPS PEM end-to-end.
Setting a password, banner or TLS material AFTER deployment forces a
container recreate on every change. The deploy wizard now lets the
operator set service config up-front so the initial build has the
right env from the start.
Mechanics:
- Extracted the schema-driven field rendering out of ServiceConfigForm
into a standalone ServiceConfigFields component (no API/buttons,
just inputs + onChange). ServiceConfigForm now delegates to it.
- Wizard step 2 (CONFIGURATION) renders one accordion block per
selected service; clicking a service reveals its schema-driven
inputs and a 'N set' badge tracks how many overrides are populated.
Removing a service (back to step 1) drops its config so the INI
doesn't carry orphans.
- _buildIni emits one [<prefix>.<svc>] group subsection per service
with at least one override. The INI loader's prefix-matcher
applies it to every ${prefix}-NN decky in the batch, so one block
covers all clones.
- Multi-line string values (PEM textareas etc.) are escaped as \n
on the way into INI; downstream consumers re-expand.
PersonaGeneration.css scopes .info-banner under .persona-gen-root,
which doesn't match elements rendered inside the Modal portal —
so the wizard's CONFIGURATION-step banner I just added rendered
as plain text.
Add a page-unscoped .info-banner rule in DeckyFleet.css with the
same visual treatment (faint bg, violet left rule) so any modal
context picks it up.
The systemd unit grants AmbientCapabilities=CAP_NET_ADMIN so the API
service can program host-side macvlan/ipvlan interfaces without
running as root, but setup_host_macvlan/_ipvlan rejected with euid!=0
before even trying — making web-driven 'decnet deploy' impossible
under the privilege model the unit advertises.
Replace _require_root with _require_net_admin, which reads CapEff
from /proc/self/status and accepts the cap (bit 12) as well as
euid==0. No libcap dep — pure /proc parse.
The CONFIGURATION step had a stale disabled placeholder textarea
("per-service overrides") from before the schema-driven Inspector
landed. Replaced with a one-line info banner pointing at the Inspector,
which is now where per-service config actually lives.
The DEPLOY step's CLI preview was rendering '--archetype custom' when
pickMode==='services', but no such archetype is registered — only the
preset archetypes plus 'services' (free-form list). Drop the
--archetype line entirely in the services-mode preview so the rendered
command reflects what the API actually receives.
Previous CSS lived in DeckyFleet.css only, so when the form rendered
inside MazeNET Inspector the inputs fell back to browser defaults
(white-on-white, oversized labels, mismatched buttons).
New ServiceConfigForm.css ships with the component itself: small
uppercase tracking-1 labels at 0.6rem (matches kvs .k), dark
transparent inputs with violet focus, matrix-green text inside
inputs, custom select chevron, dedicated svc-cfg-btn that visually
mirrors maze-btn.small, password reveal toggle, and a 96px label
column so labels never wrap into the input. Help text drops to
0.58rem dim under the input. Works identically in both surfaces.
The schema useEffect depended on currentConfig, which the parent
passes as a fresh `{}` literal on every render — referentially new
each time, so the effect re-ran and the GET /services/.../schema
hammered the server.
Schema fetch now only depends on serviceSlug; form seeding from
currentConfig moved to a separate effect keyed on JSON-stringified
config so a real change reseeds but referential churn doesn't.
ServiceConfigForm.tsx fetches /topologies/services/{slug}/schema and renders
typed inputs (string/password/int/bool/textarea/enum) with reveal toggles for
secrets. SAVE persists via PUT (no restart); APPLY persists + force-recreates
the service container after a confirm dialog (matches the forwards_l3 pattern).
Mounts:
- DeckyFleet DeckyCard: clicking a service tag toggles the form below the
EXPOSED row, gated on liveServicesEnabled (admin + non-swarm).
- MazeNET Inspector: renders the form above REMOVE SERVICE when a service
is selected on a non-observed decky.
UI test plan is manual — no jsdom test infra in decnet_web yet.
- GET /topologies/services/{name}/schema serves the declared ServiceConfigField
metadata so the Inspector can auto-render forms.
- PUT /(topologies/{id}/)deckies/{decky}/services/{svc}/config persists the
validated dict (DB + compose); container untouched (Save).
- POST /(topologies/{id}/)deckies/{decky}/services/{svc}/apply persists then
force-recreates <decky>-<svc> so the new env takes effect (Apply, destructive).
- New engine helper update_service_config wires both fleet and topology paths
through the existing _persist_fleet_change / _rerender_topology_compose
machinery; emits decky.<name>.service_config_changed on the bus.
ServiceConfigField dataclass + BaseService.validate_cfg coerce/drop submitted
service_cfg dicts against per-service typed schemas. SSH/HTTP/HTTPS now declare
the keys they already read in compose_fragment, so the upcoming Inspector form
has metadata to render from instead of hardcoded inputs per service.
Dashboard's ACTIVE DECKIES (active_deckies in get_stats_summary) counts
TopologyDecky rows where state='running'. No code path was flipping
that state away from the default 'pending', so the count read 0/N
even when every container was running fine — the dashboard was lying.
Two complementary fixes:
1. deploy_topology — after the post-deploy compose ps verification,
reconcile each TopologyDecky.state from the corresponding base
container's docker state. running → 'running'; anything else →
'failed'. Reuses the ps_rows already gathered for the
ACTIVE-vs-DEGRADED status decision; no extra docker hit.
2. apply_add_decky — _materialise_decky_spawn now returns True/False;
on True the row is updated to state='running' before
_assert_valid_after. Catches the case where a decky added via the
live mutator queue stays at 'pending' indefinitely (the deployer's
reconcile only runs on a fresh deploy_topology pass).
Existing topology deckies in active topologies will still read as
'pending' until the next deploy_topology runs, since this is
forward-only. An operator-side fix is to teardown + redeploy or run
the (forthcoming) reconcile-on-startup pass.
apply_add_decky's compose-up was hard-failing whenever the operator's
~/.docker/buildx/activity/ landed on a read-only mount — the wedge
detection in _compose_with_retry correctly refuses to retry (would
just leak more mounts), but for live materialisation we don't want a
wedged buildx state to abort an admin's mutation. ANTI hit it on
adding decky-a977: 'failed to update builder last activity time: ...
read-only file system → buildx wedge detected → returned non-zero'.
_compose_up_with_buildkit_fallback wraps _compose_with_retry: on a
CalledProcessError whose stderr matches both wedge signatures
(_BUILDX_WEDGE_SIGNATURE + _BUILDX_EROFS_SIGNATURE), it logs a
warning with the manual recovery steps + retries once with
DOCKER_BUILDKIT=0 set. The legacy non-buildx builder doesn't use
the activity dir and isn't affected.
Wired into the two paths that pass --build:
* _materialise_decky_spawn (apply_add_decky)
* _materialise_decky_services_diff (apply_update_decky service add)
_materialise_decky_recreate_base doesn't build — it just recreates a
container from an existing image — so it's not affected.
Operator-facing log message points at the manual fix
(rm -rf ~/.docker/buildx/activity + docker buildx create) so they
can recover at their leisure; we don't ATTEMPT the recovery because
the activity dir might be RO for a reason (zfs/btrfs snapshot, etc.)
that an automated rm would be wrong to fight.
apply_update_decky's flip path now refuses to promote a decky to
gateway unless its home LAN is a DMZ. The compose generator publishes
host ports for forwards_l3=True; a non-DMZ gateway would shadow the
host's port space without anything legitimately able to reach the
service. Same posture as the existing 'forwards_l3 flip on live
requires force=true' guard — refused before any DB write so a bad
mutation leaves zero side-effects.
The check is intentionally NOT a standing _RULES invariant — the
codebase uses forwards_l3 for two semantics:
1. Generic L3 forwarding (internal bridge deckies routing between
their multi-home LANs). The generator writes this on internal
bridges via bridge_forward_probability; legitimately non-DMZ.
2. DMZ gateway (host-port publisher). Only meaningful on DMZ.
Standing validation can't enforce DMZ-homing without breaking case 1.
The guard fires only on the explicit user-driven flip path where the
operator's intent is unambiguously case 2. Generator output and
internal-bridge attachments bypass the check.
check_gateway_homed_in_dmz lives in validate.py for callers that want
the explicit form (and for the test surface), but is not a standing
rule — comment in _RULES explains the asymmetry.
W5's apply_update_decky now accepts a forwards_l3 flip on a live
topology only when payload['force'] is true (the unforced flip raises
MutationError to keep half-thinking operators from killing
in-container state). Until this commit there was no UI surface that
could even submit such a flip.
Inspector grows a 'PROMOTE TO GATEWAY' / 'DEMOTE GATEWAY' button when
a (non-observed) decky is selected. The handler:
* On pending topologies → submits via editor.updateDecky immediately.
No confirm dialog; no live containers to disturb.
* On active/degraded topologies → window.confirm() explaining the
destructive base recreate ('In-container state is lost; active
sessions to it drop'), then submits with extras.force=true.
useTopologyEditor.updateDecky grows an optional extras arg that
threads force: true into the queued mutation payload. The pending
CRUD path ignores it (no force needed when no containers exist).
MazeNET.tsx wires a toggleGateway callback that handles the
optimistic local state update, surfaces an enqueue toast on the
active path, and lets the SSE forwarder reconcile when
mutation.applied lands.
Two related fixes that came out of running the W5 tests locally:
1. tests/__init__.py — empty file, makes 'tests/' a package so pytest
stops inserting it into sys.path. Without it, 'tests/docker/'
(the docker-image test category) shadowed the installed docker SDK
on every engine-touching test in the repo:
module 'docker' has no attribute 'DockerClient'
Pytest's default --import-mode=prepend was the culprit; making
tests/ a package is the cheapest fix and doesn't change
--import-mode for the whole tree.
2. delete_topology_decky / delete_topology_edge / delete_lan grow an
'enforce_pending: bool = True' kwarg. Default preserves the HTTP
CRUD guard (api_decky_crud / api_edge_crud / api_lan_crud get the
409 for free). apply_remove_decky / apply_detach_decky /
apply_remove_lan now pass enforce_pending=False — the mutator
queue is the live-editing surface and has its own active-topology
gating; the repo's pending-only guard was for design-time CRUD
that mustn't bypass it. Without this, apply_remove_decky was
silently broken on active topologies pre-W5; W5's new test
surfaced it on first run.
10/10 new W5 tests pass; 58/58 across mutator + topology suites.
apply_update_decky now discriminates three sub-cases:
* services list changed → diff old vs new and call
_materialise_decky_services_diff (compose up -d for added,
stop + rm -f for removed). Mirrors services_live's pattern but
doesn't import it — mutator-routed mutations carry a different bus
surface (mutation.applied) than the direct API path
(decky.<name>.service_added).
* forwards_l3 flipped → port publishing changes, which docker can
only apply at container-create time. Gated on payload['force'] is
true; default raises MutationError so a half-thinking operator
can't stomp a live decky. When force=true,
_materialise_decky_recreate_base does compose up -d --no-deps
--force-recreate. Pre-checked BEFORE the DB write so a refused
mutation leaves zero side-effects.
* coord-only (x/y) → DB only, no docker work.
Ships tests/mutator/test_ops_materialisation.py with focused coverage
for every new helper: add_decky/remove_decky/attach_decky/
detach_decky/update_decky/update_lan paths against an active
topology, with compose primitives + docker SDK mocked at the source
modules so the helpers' lazy imports pick up the stubs. Also covers
the pending-topology skip and the force-flag gating.
Symmetric to apply_attach_decky — after deleting the multi-home edge
from the DB, calls the docker SDK to drop the base container's
interface in the now-detached LAN. Service containers lose
visibility automatically (they share the base's netns).
Idempotency: 'not connected' / 'no such' APIError is logged at info
and treated as success.
After the DB writes that record the multi-home edge, calls the docker
SDK directly to add an interface to the base container's netns:
client.networks.get(<topology bridge>).connect(<base>, ipv4_address=ip)
Non-destructive — the base keeps running, no recreate. Service
containers automatically see the new interface because they share
the base's netns via network_mode: service:<base>.
Idempotency: docker APIError with 'already' / 'endpoint exists' is
logged at info and treated as success. Other errors log + leave the
DB row in place; an operator retry will hit the same path.
Captures the decky's name and services list before delete_topology_decky
runs (the helper needs both as compose targets even though the DB row
is gone), then calls _materialise_decky_remove which stops + rm -f's
the base + per-service containers via 'docker compose stop / rm -f'.
Re-renders the per-topology compose AFTER the stop/rm so a future
'compose up -d' on the file doesn't try to bring the decky back.
Adds _materialise_decky_{spawn,remove,connect,disconnect,services_diff,recreate_base}
helpers alongside the existing _materialise_lan_change. Each follows
the same skip rules: bail when topology is not active/degraded, when
agent-pinned, or when docker calls fail (logged, not re-raised — DB
remains source of truth).
apply_add_decky now calls _materialise_decky_spawn after the DB writes.
The helper:
* re-renders the per-topology compose so it lists the new decky;
* runs 'compose up -d --no-deps --build <decky_base> <decky>-<svc>...'
in a worker thread (matches engine/services_live's pattern).
Service container targets are filtered through get_service() so
fleet_singleton services are skipped — they don't have per-decky
compose entries. Gateway (forwards_l3=True) deckies need no
special-case here; the compose generator already emits the host
'ports:' block for them.
Subsequent commits wire the other apply_* ops to the matching
helpers. Tests for the full set ship in the workstream's last
commit.
subnet and is_dmz are pinned at deploy time — live deckies bind to
the bridge with IPs allocated from the old subnet, and is_dmz flips
the docker network's internal flag which can't be changed while
containers are attached. Today the op happily wrote the new value
into the DB and left docker on the old one, drifting the two surfaces.
apply_update_lan now raises MutationError when topology status is
active or degraded and the patch touches subnet or is_dmz. Coord
(x/y) and rename updates still pass through; renames don't currently
have a live caller and the bridge's docker name keys off the lan name
in the renderer, so the next deploy will reconcile.
This matches the posture taken by _materialise_lan_change for live
LAN add/remove (commit 472c84b).
list_topology_edges has no ORDER BY, so SQL row order is undefined.
After apply_attach_decky added a bridge edge to a second LAN, on
refetch the bridge edge could come back first — firstLanFor then
picked it as the decky's home and the visualization 'teleported' the
decky into the other LAN (the bug ANTI saw immediately after
connecting two deckies across LANs).
Hydration now prefers the non-bridge edge (is_bridge=false) as home.
apply_add_decky writes is_bridge=false for the original edge;
apply_attach_decky writes is_bridge=true for subsequent multi-homing
edges. Picking the non-bridge edge is stable across row reordering.
Two-pass implementation: pass 1 sets pinned homes (DMZ for gateways,
non-bridge for others); pass 2 fills any gap with the first edge
(legacy rows where is_bridge was never written).
apply_add_lan and apply_remove_lan were DB-only — they wrote/deleted
the topology_lans row but never created or destroyed the docker bridge
network. Adding a LAN to a deployed topology silently did nothing on
the substrate side; any decky later attached to it had nowhere to bind.
Both ops now call a shared _materialise_lan_change helper after the DB
write. When the topology is active/degraded and not pinned to a swarm
agent, the helper:
* creates / removes the docker bridge network (internal=True for
non-DMZ LANs, mirroring engine/deployer.deploy_topology),
* re-renders the per-topology compose file so future redeploys reflect
the change.
Failures are logged, not re-raised — the DB row stays as source of
truth so an operator can retry without leaking inconsistent state.
Agent-pinned topologies are skipped; the next agent push reconciles.
apply_add_decky / apply_attach_decky have the same gap and are not
fixed here — multi-homing a running container needs careful
recreate-vs-network-connect handling and is its own commit. Without
those, dropping a decky into a freshly-added LAN still won't spawn a
container; only the LAN itself is now live.
Bus topic segments are NATS-style tokens and the validator at
bus/topics.py:402 rejects '.', '*', '>', whitespace. My W3 constants
'service.added' / 'service.removed' tripped this on every live
add/remove call:
ValueError: topic segment 'service.added' may not contain '.', ...
Renamed both to underscore form: DECKY_SERVICE_ADDED = 'service_added'.
Aligned the SSE forwarder's name mapping (decky.<name>.service_added →
SSE event 'decky.service_added') and the frontend's
useTopologyStream listener + MazeNET.tsx event handler. Also updated
the wiki entry with a note about the underscore.
deploy_topology was flipping to ACTIVE the moment 'compose up -d'
returned 0, but compose returns 0 as soon as containers are *started*.
A service that crashes on boot (port bind failure, bad image, missing
entrypoint) left the topology row sitting at ACTIVE indefinitely while
half the substrate was dead.
After compose returns, we now run 'compose ps --all --format json',
parse the newline-delimited per-container rows, and downgrade to
DEGRADED with a reason listing the first eight unhealthy containers if
anything isn't in state='running'. Operators see real state on the
topology page instead of an optimistic flag.
_compose_ps swallows compose-level errors (returns []) so an unrelated
docker hiccup doesn't gate the success path — the existing in-flight
exception path still catches genuine deploy failures with FAILED.
When topoStatus is active/degraded, editor.updateDecky enqueues into
the mutator queue and returns {kind:'enqueued'}. The palette-drop
handler then short-circuits on that and never updates local state, so
a service dragged onto a deployed decky just vanishes — what ANTI saw
as 'no way to APPLY'.
Same gap on the design-time 'REMOVE SERVICE' button in the Inspector's
service detail panel: enqueue + no local update = chip stays.
Both now route through liveAddService / liveRemoveService when the
topology is active, hitting POST/DELETE /topologies/{id}/deckies/{name}/services
directly and patching local state from the response. Pending
topologies still queue through the mutator (correct: no live
containers to mutate).
Hoisted serviceRegistry / liveAddService / liveRemoveService above
the palette-drop callback so the deps array doesn't trip the const
TDZ at render time.
FastAPI's redirect_slashes=True 307s /topologies → /topologies/, and
the browser drops Authorization on the redirected URL — the topology
picker in the canary create modal was landing as 401 even for admins.
Hit the canonical (trailing-slash) path so the request resolves on the
first hop.
The /topologies/{id}/events SSE proxy now subscribes to two bus
patterns concurrently and merges them through a bounded asyncio.Queue:
* topology.{id}.> — lifecycle (status, mutation.*) — unchanged.
* decky.> — per-decky events, filtered by payload.topology_id
so a fleet decky sharing a name with a topology
decky doesn't leak across.
_sse_name_for routes 'decky.<name>.service.added' to the SSE event
name 'decky.service.added' (kept the prefix so the frontend doesn't
collide with topology lifecycle events that share leaf names like
'status').
useTopologyStream surfaces the two new event names; MazeNET.tsx's
onStreamEvent optimistically patches the matching node's services
list so a second tab reflects shape changes without a refetch.
DeckyCard grows the same per-chip × + dashed '+ ADD' affordances we
just shipped on the MazeNET Inspector. Wired to POST/DELETE
/api/v1/deckies/{name}/services{,/svc}; the response's services list
flows back through onServicesChanged to update the parent's deckies
state without a refetch.
Gated on isAdmin && !decky.swarm — swarm deckies live on a remote
agent and the W3 endpoint runs docker compose locally, same gap as
the canary planter has for agent-pinned topologies. Out of scope
here; flagged as a known limitation.
stopPropagation on the inline buttons + add-row container keeps the
card-level click (which selects the decky for inspection) from firing
on intra-row interactions.
ObservedNode.services is the literal tuple ['*']; narrowing inside the
.filter() callback was tripping TS2345. We already gate the live
controls on node.kind !== 'observed', so casting to readonly string[]
inside the filter is safe and keeps the discriminated union strict
elsewhere.
When the topology is active/degraded the Inspector switches services
chips into live controls: each chip gets a × button that DELETEs to
the W3 endpoint, and a dashed '+ ADD' chip opens a typeahead picker
fed by useServiceRegistry().perDecky.
Pending topologies still use the existing design-time path
(onRemoveService → editor.updateDecky); the Inspector picks based on
topologyStatus, so an operator never accidentally hits a live API
call against a topology that isn't deployed yet.
The mutation handlers in MazeNET.tsx hit POST/DELETE
/api/v1/topologies/{id}/deckies/{name}/services{,/svc} and
optimistically apply the response's services list to local state.
Cross-tab reconciliation rides on the SSE forwarder shipped in the
follow-up commit.
Adds a fleet_singletons array to ServiceCatalogResponse so per-decky
add UIs can filter out services like LLMNR that run once fleet-wide
(and would 422 server-side at the live add endpoint).
The existing 'services: list[str]' field is unchanged for back-compat
with MazeNET/useMazeApi.ts:257; the new field is additive.
decnet_web/src/hooks/useServiceRegistry.ts wraps the endpoint with a
module-scoped cache (registry only changes on BYOS install / plugin
drop, neither of which happens mid-session) and exposes a precomputed
.perDecky list so consumers don't need to re-derive the diff.
CanaryTokens.tsx grows a third tab — File drops — alongside Tokens
and Blobs. The page now covers every 'admin landed bytes on a decky'
operation in one place.
FileDropModal mirrors the canary CreateModal's shape: Fleet/MazeNET
toggle, topology+decky picker, absolute-path validation matching the
backend (DeckyFileDropRequest rejects relative + ..-traversal), mode
+ mtime offset inputs, and a -1w preset for backdating. FileReader →
data URL → strip prefix → POST /api/v1/deckies/files.
The list is local-only (localStorage, capped at 200 entries). W2's
backend doesn't persist drops by design — the endpoint is for staging
payloads, not as an audit trail. CLEAR LIST button on the tab; no
DELETE button on rows since the local entry doesn't track whether the
file is still there (an attacker may have moved it).
Alt+D shortcut joins Alt+C; alt-key only per the Linux-meta-key rule.
CanaryTokens.tsx grows a Fleet/MazeNET toggle in the create modal. In
topology mode we hydrate /topologies?status=active for the topology
picker, then GET /topologies/{id} on selection to repopulate the decky
picker — topology deckies have a different shape than fleet's /deckies
endpoint.
The tokens table gains a SCOPE column (chip: 'fleet' / 'topology'),
and a third filter dropdown alongside state. The drawer's metadata
section shows a Scope row with a clickable jump-link back to the
MazeNET view at the right topology.
CanaryTokenRow grows a topology_id field so the drawer/list can
discriminate without re-fetching.
decnet.engine.services_live exposes add_service / remove_service for
both fleet and topology decky scopes. The host's _compose() wrapper
already supported per-service targeting (up --no-deps -d <svc>,
stop, rm -f); what was missing was the orchestration around it:
* add: validate against decnet.services.registry (rejects unknown +
fleet_singleton); persist the new services list; re-render the
per-scope compose file (so future redeploys reflect the change);
run docker compose up -d --no-deps --build <decky>-<svc>.
* remove: stop + rm -f the service container; persist; re-render
compose so a future up -d doesn't bring it back.
Both publish decky.<name>.service.added / .removed on the bus, with
the post-mutation services list. Topic constants added to
decnet.bus.topics; the matching wiki entry in wiki-checkout/Service-Bus.md
ships in a separate commit on the wiki repo (wiki-checkout/ is gitignored).
Four new admin endpoints:
* POST/DELETE /api/v1/deckies/{name}/services{,/svc}
* POST/DELETE /api/v1/topologies/{id}/deckies/{name}/services{,/svc}
ServiceMutationError messages are mapped at the API boundary to 404
(decky/topology missing), 409 (idempotency violation), 422 (unknown
or fleet_singleton service).
Extracts the docker-exec-with-base64-stdin pattern out of canary/planter
and orchestrator/drivers/ssh into a shared decnet.decky_io package.
Both consumers now delegate; the canary planter test still proves the
contract end-to-end.
Adds POST/DELETE /api/v1/deckies/files for arbitrary file drops.
Container resolution is shared with the canary path: topology_id absent
means fleet (<name>-ssh), present routes through resolve_decky_container
which picks <name>-ssh when the topology decky exposes ssh, else the
topology base container decnet_t_<id8>_<name>.
Path validation rejects relative paths and '..' traversal at the request
model layer. Bad base64 → 400; unknown topology → 404; decky not in
topology → 422; docker exec failure → 409.
POST /api/v1/canary/tokens grows an optional topology_id field. When
present, the server hydrates the topology, validates the named decky is
in it, and resolves the docker container via
planter.resolve_topology_container — <name>-ssh if the decky exposes ssh,
else the topology base container. Absent ⇒ fleet semantics, unchanged.
The token row gets a nullable topology_id column (no migration helper
per pre-v1 policy). GET /api/v1/canary/tokens accepts ?topology_id= as
a filter. DELETE re-resolves the container at revoke time so a
redeployed topology is still reachable.
422 when the named decky isn't in the topology; 404 when the topology
itself doesn't exist.
Topology deploys now plant the configured canary baseline set on every
decky in the topology, mirroring the fleet-deploy hook. Containers are
resolved via resolve_topology_container — <decky>-ssh when the decky
exposes an ssh service, else the topology base container
decnet_t_<id8>_<decky>.
The planter's plant/revoke/seed_baseline grow an optional container=
kwarg; default preserves the fleet <name>-ssh resolution.
The Vault page already shows file drops and stored mail (e3ddeb0) but
the inspector drawer had no download button — only the live-feed
ArtifactDrawer/MailDrawer offered raw byte retrieval. Add a DOWNLOAD
RAW action to BountyInspector that fires when bounty_type=artifact,
hitting /artifacts/{decky}/{stored_as}?service=<svc> with the bounty's
own service field (ssh or smtp). Mirrors ArtifactDrawer's blob handling
and 400/403/404 error mapping.
Also widen the icon/label vocabulary: artifact bounties get FileText
(file drops) or Mail (message_stored) instead of the generic Package,
and the inspector header chip mirrors the change.
The Bounty Vault page only read from the Bounty table, but
inotifywait-captured file drops (event_type=file_captured) and SMTP
quarantined messages (event_type=message_stored) were only landing in
the Logs table. AttackerDetail's tabs queried logs directly, so they
showed up per-attacker but were invisible on the global Vault page.
Mirror both events into Bounty as bounty_type=artifact with
payload.kind ∈ {file, mail} so the existing dedup
(bounty_type, attacker_ip, payload) collapses repeats by sha256. Add an
ARTIFACTS segment to the Vault filter row, plus dedicated render
branches: file drops show orig_path + size + writer attribution; mail
shows subject + From + attachment count + size, with the Mail icon
distinguishing them from FileText for file drops.
Forward-only — existing logs stay where they are. A backfill pass would
be straightforward (read Log WHERE event_type IN ('file_captured',
'message_stored') and feed each row through _extract_bounty) but is out
of scope here.
sshd, pam_unix, sudo, CRON, systemd, kernel, rsyslogd, and dbus-daemon
all share the SSH/telnet decky containers and write to the same syslog
socket as DECNET's own emitters. Their output was being parsed and
ingested into the JSON stream, the dashboard, and the profiler — pure
noise: sshd's "Failed password for root from X" duplicates the
auth-helper's structured auth_attempt event, pam_unix repeats it again,
CRON/systemd say nothing about attacker behavior.
Drop these APP-NAMEs in _should_ingest before the JSON write and bus
publish. Raw .log file still captures everything for forensics. The
denylist is overridable with DECNET_COLLECTOR_DROP_APPS so operators
can extend it without code changes.
Add --rfc5424 --msgid command to the logger invocation in SSH and telnet
decky bashrc. MSGID arrives as "command" instead of NIL, which is what
the profiler's _COMMAND_EVENT_TYPES filter expects. The parser heuristic
shipped in d4591b3 stays as a safety net for any future emitter that
forgets the flags or for inflight pre-rebuild containers.
SSH/telnet decky containers emit shell commands via `logger -t bash "CMD …"`
which produces RFC 5424 lines with MSGID=NIL. Both parsers were leaving
event_type="-", so the behavioral profiler's `_COMMAND_EVENT_TYPES` filter
silently dropped them — the IP profile existed but no command transcripts
or artifacts. Confirmed in the wild: 44/48 events from one attacker were
event_type="-".
Rewrite event_type to "command" in both parsers when MSGID=NIL and the
msg starts with "CMD ". Correlation parser also extracts the cmd= payload
into fields["command"] so the profiler can build the transcript; collector
parser leaves fields={} to avoid duplicate pills in the dashboard.
Default operator-supplied email seed for IMAP/POP3 deckies. Drop `*.eml` and/or `*.json` files here; the IMAP/POP3 services bind-mount this dir read-only at `/var/spool/decnet-emails/seed` when no per-decky `email_seed` is configured. Entries concatenate onto the hardcoded bait baseline (additive to realism-engine output, never replacing).
JSON shape: list of dicts with required `from_addr`, `to_addr`, `subject`, `body`; optional `from_name`, `date`, `flags`. See `decnet/templates/imap/server.py` for the loader.
# is colocated there, mirroring the per-feature CLI split used by
# :mod:`decnet.cli.canary`, :mod:`decnet.cli.webhook`, etc. The
# ``decnet-ttp.service`` systemd unit's ExecStart still resolves to
# ``decnet ttp`` because the command name is unchanged.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
