Commit Graph

141 Commits

Author SHA1 Message Date
f211d394e6 feat(ttp): E.3.11 CanaryFingerprintLifter (R0049-R0053)
Browser-payload derivations per Appendix A.9: navigator.webdriver flag,
canvas/audio/WebGL automation hash matches (Puppeteer/Playwright/
Selenium/curl-impersonate), WebRTC IP leak, TZ/language vs source-IP
geo mismatch, navigator.platform vs userAgent vs WebGL renderer
inconsistency.

Evidence shape pinned to CanaryFingerprintEvidence (metric +
matched_signature) — raw fingerprint blobs (canvas hashes, full UAs,
navigator.platform values) explicitly NOT carried into TTPTag.evidence
per TTP_TAGGING.md §'Hard parts §7' (enrichment vs tag boundary). The
identity-merge guard rail is preserved: composite fp.id matches across
IPs are NOT a TTP, so no rule fires on the bare hash.

Tests: tests/ttp/test_canary_fingerprint_lifter.py per-rule positive +
negative + evidence-shape guard + state modulation.
tests/ttp/rule_precision/test_canary_rules.py xfail flipped to real
precision (R0049/R0050/R0051/R0053 H-band ≥95%; R0052 M-band ≥80%).
2026-05-01 20:25:57 -04:00
7865e71aa9 feat(ttp): E.3.10 IntelLifter (R0054-R0058)
Per-provider verdict translator for AbuseIPDB, GreyNoise, Feodo Tracker,
and ThreatFox per Appendix A.10. Each rule's predicate inspects payload
fields produced by the enrich worker (no DB I/O, no decnet.intel.*
imports — E.2.7 decoupling guard preserved). AbuseIPDB confidence is
scaled by abuse_confidence_score / 100; categories drive per-technique
fan-out. R0058 aggregate-bump is a no-op in v0 (cross-tag bump deferred
to E.3.14 worker bootstrap).

Per-provider null tolerance is the steady state — a missing provider
column produces zero tags from that rule, never an error.

Tests:
- tests/ttp/test_intel_lifter.py — per-provider positive + negative +
  state modulation + decoupling source-import guard.
- tests/ttp/rule_precision/test_intel_rules.py — xfail flipped, real
  precision driven over seed_intel.jsonl (R0054-R0057 H-band ≥95%;
  R0058 skipped as bump-only).
- tests/ttp/test_lifter_absence.py — IntelLifter all-populated test
  flipped from xfail-strict to real assertion with realistic payload.
- tests/ttp/test_lifters.py — partial-null xfail flipped to real
  assertion.
2026-05-01 20:23:42 -04:00
eff3e4bce7 feat(ttp): E.3.9 BehavioralLifter (R0031-R0040)
Reads pre-shaped session aggregates from TaggerEvent.payload and emits
techniques per Appendix A behavior tables. Per-rule predicates dispatch
on match.kind (lifter:behavioral_<name>); the lifter holds its own
RuleIndex watching the same RuleStore as the engine, so disable / clip /
TTL state reaches lifter-bound rules through the same atomic-swap path.

R0032/R0036/R0037/R0040 YAMLs had over-escaped regex strings (\\
instead of \\) — fixed in place.

Factory wired so default get_tagger() returns CompositeTagger with
BehavioralLifter shipped; remaining three lifters (E.3.10-E.3.12) land
in subsequent commits.

E.2.6 contract preserved via TolerantTagger: empty payload steady-state
yields [] with zero ERROR records. Disabled / clipped / expired state
verified.
2026-05-01 20:17:59 -04:00
e7531ee756 refactor(ttp): extract RuleIndex from RuleEngine
E.3.9.0 prerequisite for the per-source lifters (E.3.9-E.3.13). The
dispatch index, install/evict/apply_change atomic-swap protocol, and
state-modulation helpers (is_active / apply_ceiling) move out of
rule_engine.py into _rule_index.py and _state.py. RuleEngine wraps a
RuleIndex; back-compat shims preserve _by_kind / _by_rule / _install
attribute access for tests poking at the dispatch internals.

Lifters in E.3.9-E.3.12 will each hold their own RuleIndex, watching
the same RuleStore via subscribe_changes() fan-out. Hot-reload
semantics (disable / clip / TTL via set_state API) now reach
lifter-bound rules through the same atomic-swap path the engine uses,
not a future composite-rebuild compromise.
2026-05-01 20:09:18 -04:00
b819dfefa3 feat(ttp): E.3.8 R0054-R0058 intel cohort + mark step done
5 YAMLs for the intel-verdict cohort per Appendix B / A.10:
AbuseIPDB category mapping, GreyNoise classification, Feodo
Tracker hit, ThreatFox IOC type, aggregate-malicious bump-only.
IntelLifter (E.3.10) consumes by rule_id and tolerates absence
silently (null provider column → no tag).

R0058 is the meta bump-only rule — emits a single confidence=0.0
sentinel so it validates and surfaces in the catalogue, but the
repository's sub-0.3 drop ensures no fresh tag persists if the
fanout fires accidentally. test_intel_rules.py pins that
zero-confidence invariant.

Marks E.3.8 done in development/TTP_TAGGING.md with the cohort-
split summary.
2026-05-01 09:22:48 -04:00
dc1867315d feat(ttp): E.3.8 R0049-R0053 canary fingerprint cohort
5 YAMLs for the canary-fingerprint cohort per Appendix B / A.9:
navigator.webdriver flag, automation canvas/audio/WebGL hash match,
WebRTC IP leak, TZ/lang vs geo mismatch, platform inconsistency.
CanaryFingerprintLifter (E.3.11) consumes by rule_id.

test_canary_rules.py: YAML-present + inert-in-v0 + xfail(strict)
gated on E.3.11.
2026-05-01 09:21:01 -04:00
1ad15470a1 feat(ttp): E.3.8 R0041-R0048 email cohort
8 YAMLs for the email cohort per Appendix B: open-relay abuse,
mass phishing, phishing-kit X-Mailer signatures, IDN/punycode
URLs, sender masquerade, malicious attachment, BEC, encoded
payload in body. EmailLifter (E.3.12) consumes by rule_id.

test_email_rules.py: YAML-present + inert-in-v0 + xfail(strict)
precision case gated on E.3.12.
2026-05-01 09:19:56 -04:00
806301e179 feat(ttp): E.3.8 R0031-R0040 behavioral cohort
10 YAMLs for the behavioral / cross-event cohort per Appendix B:
beaconing, data destruction, ransom note, web exfil, DB mass-read,
credentials-in-files, k8s SA token harvest, Docker host escape,
LLMNR poisoning, TFTP router-config retrieval.

Every rule is lifter-bound (BehavioralLifter / IdentityLifter) —
the v0 RuleEngine cannot count, aggregate, or compose cross-event
signals, so these YAMLs declare the technique mappings the lifter
will consume by rule_id at E.3.9. Their match specs use a
'kind: lifter:*' shape inert to the regex matcher.

test_behavioral_rules.py asserts each YAML compiles, none fire
from the v0 engine (FP regression guard against a YAML drifting
into a regex), and an xfail(strict=True, reason='impl phase E.3.9')
precision case that will flip green when the lifter lands.
2026-05-01 09:18:27 -04:00
b1fe1f9403 feat(ttp): E.3.8 R0001-R0030 command cohort
30 YAMLs for the shell/command rule cohort per Appendix B (rules/ttp/).
Splits into engine-active (R0007-R0029, regex on command_text /
raw_url / user_agent) and lifter-bound (R0001-R0006, R0030 — the
v0 RuleEngine cannot count auth attempts, do identity rollups, or
parse fingerprint blobs; the BehavioralLifter / IdentityLifter /
CredentialLifter consume them by rule_id at E.3.9 / E.3.13).

test_command_rules.py asserts:
- every R000N has a YAML that compiles
- lifter-bound rules NEVER fire from the v0 engine (regression
  guard against a YAML drifting into a regex match.spec)
- engine-active rules meet their Appendix-C precision target
  against the seed corpus (≥0.95 high-conf, ≥0.80 medium)

Conftest fixes: precision_engine moved to module-scope so module-
scope precomputed dispatch fixture (fired_by_label) can request it;
_RULES_DIR path bumped from parents[2] to parents[3] so the loader
resolves the project root regardless of pytest cwd; make_event
synthesizes attacker_uuid so TTPTag's anchor invariant is satisfied.

Seed corpus broadened: positive examples for every regex rule plus
6 negative examples across innocuous shell verbs (ls, echo, cd, ps,
df, free) so FPs surface in precision rather than passing vacuously.
2026-05-01 09:16:38 -04:00
c635478442 feat(ttp): E.3.8 corpus + harness — labelled holdout fixture
Sub-step preceding the rule-pack commits per TTP_TAGGING.md:2967.
Adds the per-rule precision suite scaffolding under
tests/ttp/rule_precision/:

- conftest.py: precision_engine fixture (RuleEngine populated from
  ./rules/ttp/), corpus_loader (real → seed → empty fallback),
  precision_for() helper for TP/FP accounting.
- _build_corpus.py: extractor for a real prod corpus pull. Mandatory
  --exclude-ip / DECNET_TTP_CORPUS_EXCLUDE_IPS — operator IPs never
  end up in the committed exclusion list. Pulls both 'command' and
  'unknown_command' event types.
- corpus/seed_*.jsonl: synthetic seed rows for each cohort so the
  harness exercises in clean checkouts.
- corpus/*.jsonl (operator-built) is gitignored.
- test_corpus_loads.py: sentinel that every seed file parses.
2026-05-01 09:08:07 -04:00
ed3f340ea8 feat(ttp): E.3.7 RuleEngine — evaluate + atomic-swap watch_store
Implements the rule engine body left empty at contract phase: evaluate()
dispatches by source_kind through self._by_kind, runs the rule's match
spec against event.payload, and emits one TTPTag per emits entry.
watch_store() loads the initial corpus from RuleStore.load_compiled,
then drains subscribe_changes, applying definition changes via
single-statement dict assignment (atomic swap, GIL-atomic to readers)
and state changes via NamedTuple._replace on the existing CompiledRule.

Why: with the FS + DB stores in place (E.3.5/E.3.6), the engine is the
last piece of the rule plane. Lifters (E.3.9–E.3.13) consume the
engine; the worker bootstrap (E.3.14) wires watch_store into the
asyncio event loop. After this commit a CompositeTagger constructed
with a RuleEngine + a populated rules dir will produce real tags.

Notes:
- CompiledRule.emits extended to 4-tuple
  (technique_id, sub_technique_id, tactic, confidence). Tactic + confidence
  ride per-emit so a single rule can carry multiple precision targets
  (the "one event maps to many techniques" property). Compile helpers in
  both backends extract them from the YAML emits dict; missing tactic
  or confidence is a deploy-time error.
- v0 match operator is "pattern" (regex). The field defaults per
  source_kind (command_text / raw_url / subject / verdict / …) and is
  overridable via match.field. Future ops (contains, equals, in_set)
  extend _match_event without touching the engine surface.
- Confidence model: rules with state="clipped" + confidence_max set
  cap the per-emit confidence downward; clipped is a soft suppress, not
  a hard skip. Disabled rules are skipped wholly; expires_at past is
  re-checked at evaluate as defense-in-depth (the store auto-reverts,
  but a racing read between expiry and revert must not fire the rule).
- _span(name, **attrs) helper in engine + both stores short-circuits on
  decnet.telemetry._ENABLED — matches the project's @traced /
  wrap_repository zero-overhead-when-disabled pattern instead of relying
  solely on the no-op tracer indirection.
- Late-bound tracer (telemetry.get_tracer called per-span, not at
  module load) so test_tracing's monkeypatch reaches the production
  code path.

xfails flipped: tests/ttp/test_rule_engine.py multi-emit fan-out +
rule_version-collision-via-engine; tests/ttp/test_multi_mapping.py
N×M engine fan-out + idempotent replay; tests/ttp/test_tracing.py
ttp.eval span hierarchy + ttp.rule.fire span attributes.

Tests: 214 passed, 19 xfailed (gated on E.3.8 lifters / rule pack /
worker bootstrap).
mypy: clean on prod code; pre-existing test-stub arg-type warnings
unchanged.
2026-05-01 08:49:15 -04:00
8a93ee3129 feat(ttp): E.3.6 DatabaseRuleStore — ttp_rule/ttp_rule_state + master sync
Implements the DB-backed rule store body left empty at contract phase:
load_compiled reads from ttp_rule + ttp_rule_state; get_state /
set_state hit ttp_rule_state with the same expires_at auto-revert and
bus-event semantics as the FS backend; subscribe_changes returns a
per-subscriber queue. State persists across process restarts — the
swarm property the FS backend deliberately doesn't have.

Also lands two swarm-mode helpers:
- sync_from_filesystem(fs_store) — master-side, subscribes to a
  FilesystemRuleStore and projects each RuleChange onto a ttp_rule
  upsert/delete.
- tail_db(poll_interval) — worker-side, watermark poll over
  ttp_rule.updated_at; emits RuleChange("definition", ...) for each
  row that moved.

Why: swarm mode needs rule definitions and operator state to
propagate across hosts. The filesystem backend (E.3.5) was the
single-host-dev variant; this one survives restart and serves N
workers from a shared DB.

Notes:
- DatabaseRuleStore() with no args lazy-inits an in-memory SQLite
  repo so the conformance fixture works without test plumbing. In
  production the worker bootstrap (E.3.14) passes an explicit repo.
- The conftest.py rule_store fixture became async (pytest_asyncio),
  per-backend creates/initializes a SQLite repo for the DB run.
- Adds a `seed_rule(store, rule_id, yaml)` helper to bridge backend
  semantics: drop a YAML file (FS) vs insert a ttp_rule row (DB).
  Used by the parametrized load_compiled conformance test.
- Late-bound _tracer() in both backends (was module-level get_tracer
  binding) so test_tracing's monkeypatch of decnet.telemetry.get_tracer
  actually affects span output.

xfails flipped: tests/ttp/store/test_database.py set_state-writes-to-
ttp_rule_state + filesystem-to-DB sync; tests/ttp/store/test_conformance.py
DB-side load_compiled / set_state isolation / round-trip / per-rule
fan-out / expired-state revert / set_state failure / get_state default
(was xfail-only-on-DB);  tests/ttp/test_tracing.py set_state span
hierarchy.

Tests: 208 passed, 25 xfailed (gated on E.3.7 + lifters).
mypy: clean on all touched files.
2026-05-01 08:39:46 -04:00
f41995a229 feat(ttp): E.3.5 FilesystemRuleStore — inotify hot-reload + per-rule events
Implements the filesystem-backed rule store body left empty at contract
phase: YAML parse + Pydantic validation, asyncinotify watch over
./rules/ttp/, in-process state cache with auto-revert on expires_at,
and a subscribe_changes() async iterator yielding one RuleChange per
per-rule edit. Bus topic builders ttp_rule_reloaded / ttp_rule_state
ship alongside.

Why: the rule plane needed a store before the engine (E.3.7) could
consume RuleChange events and atomically swap compiled rules into its
dispatch index.

Notes:
- Linux-only by construction (asyncinotify wheel gated by sys_platform
  marker; FilesystemRuleStore.__init__ raises on non-Linux).
- Filename allowlist is the FIRST check on every inotify event.
- Content-hash dedup so a single write firing IN_CREATE + IN_CLOSE_WRITE
  produces exactly one RuleChange.
- All compile work serializes on a single asyncio.Lock.
- Subscribers register their queue eagerly so events fired between
  subscribe_changes() and the first __anext__() are buffered.

xfails flipped: per-save-style + filter-ordering + atomic-swap in
test_filesystem.py; load_compiled / set_state isolation / round-trip /
per-rule fan-out / expired-state revert / set_state failure semantics
in test_conformance.py (FS side; DB side stays xfail until E.3.6);
malformed-YAML compile-time check in test_rule_engine.py.

Tests: 197 passed, 35 xfailed (gated on E.3.6 / E.3.7 / lifters).
mypy + bandit: clean on all touched files.

Wiki update for the per-rule reload + state-change topics lands in a
matching wiki-checkout/Service-Bus.md edit (separate repo).
2026-05-01 08:31:05 -04:00
fee697694d feat(ttp): E.3.3 repository — insert_tags + listing rollups (dual backend)
Dialect-split: portable rollup queries on TTPMixin; bulk insert with
ON CONFLICT DO NOTHING / INSERT IGNORE in the per-dialect repos.
Confidence-floor (< 0.3) drop applied at mixin layer before the
dialect hook. BaseRepository now declares the six TTP methods abstract.

Tests in tests/web/db/test_ttp_repo.py flipped from pytest.fail stubs
to real dual-backend behavioral tests; tests/ttp/test_confidence.py
drop-below-floor xfail removed.
2026-05-01 08:04:46 -04:00
0217319423 test(ttp): E.2.14b RuleStore conformance — cross-backend + filesystem-specific + database-specific
tests/ttp/store/conftest.py — parametrized rule_store fixture over
FilesystemRuleStore (skipped on non-Linux) + DatabaseRuleStore.

test_conformance.py — shared assertions (default-state, set_state
isolation/round-trip, subscribe_changes per-rule fan-out, expires_at
auto-revert, set_state failure semantics) parametrize over both.
get_state-default GREEN today on FS (returns RuleState() for empty
cache); rest xfail-gated behind E.3.5/E.3.6.

test_filesystem.py — inotify mask + canonical kernel values + 9
scratch-filename rejections + 4 valid-filename acceptances +
fullmatch anchor + tmp_path construction + CompiledRule frozen
property GREEN today; per-save-style + filter-ordering +
atomic-swap concurrency xfail-gated.

test_database.py — class-level surface (no platform guard, ABC
methods concrete, async coroutines) GREEN today; ttp_rule_state
write + filesystem→DB sync xfail-gated behind E.3.6.
2026-05-01 07:45:32 -04:00
bf5414c0d1 test(ttp): E.2.14a follow-up — force DECNET_DEVELOPER_TRACING=true, skip when Jaeger unreachable
Session-scoped autouse fixture in tests/ttp/conftest.py sets
DECNET_DEVELOPER_TRACING=true and forces decnet.telemetry._ENABLED
so the no-op tracer doesn't silently swallow emitted spans. The
span_exporter fixture also monkeypatches decnet.telemetry.get_tracer
so production code under test lands spans in the in-memory
exporter. Tracing tests skip when DECNET_OTEL_ENDPOINT (default
localhost:4317) isn't reachable so the dev loop stays green
without lying about coverage.
2026-05-01 07:42:22 -04:00
f4fe6fe6e4 test(ttp): E.2.14a observability tracing — span hierarchy + no-PII property
In-memory span exporter fixture wired to a per-test TracerProvider
(OTEL global is locked once set, so each test gets its own).
ttp.eval / ttp.lifter.{name} / ttp.rule.fire / ttp.rule.state.change
hierarchy + no-PII canary battery xfail-gated behind E.3.5–E.3.13.
2026-05-01 07:40:58 -04:00
4a93e16407 test(ttp): E.2.13 repository tests — TTPMixin idempotency + identity-rollup projection on dual backends
Adds tests/web/db/conftest.py with a db_backends fixture
parametrizing SQLite (always) + MySQL (gated on
DECNET_TEST_MYSQL_URL). Surface assertions (mixin methods present
+ async) GREEN today; insert_tags idempotency, identity rollup
projection, attacker-rollup exclusion of NULL-attacker tags
xfail-gated behind E.3.3.
2026-05-01 07:39:16 -04:00
6814949bc0 test(ttp): E.2.12 worker bus integration — _TOPICS equality, loop-prevention, delivery asymmetry
Pin _TOPICS frozenset against documented set (single source of
truth). Worker→engine invocation, loop-prevention invariant,
attacker.enriched/email.received catch-up asymmetry xfail-gated
behind E.3.14.
2026-05-01 07:37:58 -04:00
c276b5696e test(ttp): E.2.11 multi-mapping property — N×M fan-out, idempotent UUID, replay-safety
Hypothesis property: N rule_ids × M technique_ids on one event yield
N×M distinct tag UUIDs. Worked example pinned: one rule emitting
(T1110, None) and (T1078, None) → two distinct UUIDs. Engine-level
fan-out + replay xfail-gated behind E.3.7.
2026-05-01 07:36:19 -04:00
fd81be0bb1 test(ttp): E.2.10 confidence model — downward-only multiplier property, drop-below-0.3, AbuseIPDB-30 worked example
Pure-arithmetic adjustment formula pinned via Hypothesis property
test (multiplier ∈ [0,1] cannot raise base). Drop-at-floor and
provider-score multiplier xfail-gated behind E.3.3 / E.3.10.
2026-05-01 07:34:58 -04:00
79e6df8343 test(ttp): E.2.9 UKC bridge bijection — pin tactic↔phase mapping, observable round-trip, lossy phases
Pre-target phases (RECONNAISSANCE/RESOURCE_DEVELOPMENT/WEAPONIZATION/
SOCIAL_ENGINEERING) and observable-but-unmappable phases (EXPLOITATION/
PIVOTING/OBJECTIVES, UKC-only concepts ATT&CK lacks tactics for) are
pinned as lossy via _LOSSY_INVERSE_REFERENCE so a future contributor
cannot 'fix' the asymmetry without tripping the suite.
2026-05-01 07:33:47 -04:00
b7f206c8c5 feat(ttp): E.1.9 API contract — seven router endpoints, admin-gated state mutations, response models
Mounts /api/v1/ttp/* with empty-list / empty-Navigator responses.
GET endpoints viewer-gated; POST/DELETE /rules/{rule_id}/state
admin-gated server-side. POST parses JSON manually so a malformed
body returns the documented 400 (per feedback_schemathesis_400).

Drops xfail-strict markers from E.2.8 tests now that the router is
mounted; 26 tests pass against the contract handlers.
2026-05-01 07:20:13 -04:00
b5a19301a2 test(ttp): E.2.8 API shape + auth — GET 200/401 + admin-only POST/DELETE 401/403/200/400 contract 2026-05-01 07:00:41 -04:00
0cdf8d90da test(ttp): E.2.7 decoupling lint — TTP code may not import decnet.intel.* providers or decnet.profiler.keystroke 2026-05-01 06:58:12 -04:00
e2078c868d test(ttp): E.2.6 lifter tolerates absence — six lifters return [] on empty joins, no ERROR logs 2026-05-01 06:57:29 -04:00
1ffaa3df41 test(ttp): E.2.5 RuleEngine behavior — empty store, malformed YAML, multi-emit fan-out, version collisions 2026-05-01 06:56:28 -04:00
5accf8f1b1 test(ttp): E.2.4 Tagger ABC conformance — hypothesis fuzz over swallowed Exception types 2026-05-01 06:54:29 -04:00
cce84f23dc test(bus): E.2.3 TTP topic naming — constants, builders, wildcard match 2026-05-01 06:53:05 -04:00
e58aa4fe3a test(ttp): E.2.2 idempotency — determinism, golden value, replay-safety signature lock 2026-05-01 06:45:49 -04:00
e6f1da2344 test(ttp): E.2.1b evidence shape — TypedDict keys, PII §6 type-level assertion 2026-05-01 06:45:35 -04:00
c3a799726f test(ttp): E.2.1 schema invariant tests — CHECK, ValueError guard, UUIDv5, JSON round-trip 2026-05-01 06:44:57 -04:00
19cc8aa859 feat(ttp): E.1.7 worker contract — run_ttp_worker_loop, _TOPICS, registry entry 2026-05-01 06:33:34 -04:00
208ffd8f4f feat(ttp): E.1.6 per-lifter contracts — six TolerantTagger subclasses 2026-05-01 06:31:31 -04:00
cb9d183c20 feat(ttp): E.1.5 RuleEngine contract — CompiledRule, RuleSchema, RuleEngine ABC 2026-05-01 06:30:12 -04:00
c3c5813211 feat(ttp): E.1.3+E.1.4 Tagger ABC and composite factory contract
Third and fourth TTP-tagging contract commits, plus a scoped subset
of the E.2.4 conformance tests covering the contract surface shipped
here (full hypothesis-fuzz suite still lands with E.2.4).

E.1.3 — decnet/ttp/base.py
- TaggerEvent NamedTuple: source_kind, source_id, attacker_uuid,
  identity_uuid, session_id, decky_id, opaque payload.
- Tagger(ABC) with abstract async tag(); class-level name and
  HANDLES: frozenset[str] (default empty so a misconfigured subclass
  is loudly idle, not loudly noisy).
- TolerantTagger(Tagger): concrete tag() wraps abstract _tag_impl()
  in try/except Exception (deliberately not BaseException — so
  KeyboardInterrupt / SystemExit / asyncio.CancelledError propagate
  and the worker can shut down cleanly). Swallowed exceptions log
  at WARNING with exc_info, never ERROR — absence is the steady
  state, not a bug. Subclasses override _tag_impl, never tag — the
  tolerance contract is enforced in the base class, not on trust.
- KNOWN_SOURCE_KINDS: Final[frozenset[str]] enumerating every
  source_kind a producer is allowed to emit. Closed-by-enumeration
  at the runtime layer; the composite tagger keys its WARNING/INFO
  bridge off this constant to surface the silent-drop trap from
  the design doc (lines 160–195).

E.1.4 — decnet/ttp/factory.py
- get_tagger() reads DECNET_TTP_TAGGER_TYPE (default 'composite');
  unknown values raise ValueError with the known-list. Mirrors
  decnet.intel.factory and decnet.clustering.factory.
- _KNOWN = ('composite',). Per-lifter classes (E.1.6) are children
  of the composite, not standalone tagger types.
- CompositeTagger(Tagger): pre-computes a dict[str, list[Tagger]]
  dispatch index from each lifter's HANDLES; fans events out
  concurrently with asyncio.gather and concatenates results.
  Empty lifters=[] is the legal contract-phase state — E.1.6
  wires the real lifters in.
- Unhandled-event observability: source_kind in KNOWN_SOURCE_KINDS
  but no lifter claims it -> WARNING once per kind per process
  (missed E.1.6 update). Unknown kind -> INFO once per kind per
  process (future-feature telemetry, by design). Per-process dedup
  via plain set; E.1.6 may swap in a proper rate-limiter once
  production traffic shapes are known.

Tests — tests/ttp/test_base.py, tests/ttp/test_factory.py
- Tagger / TolerantTagger abstractness, missing-tag-impl rejection,
  WARNING-not-ERROR log level, propagation of KeyboardInterrupt /
  SystemExit / asyncio.CancelledError.
- Factory env-var routing, unknown-name ValueError, dispatch-index
  correctness, only-claiming-lifter invocation, WARNING-once for
  known-but-unclaimed kinds, INFO-once for unknown kinds, result
  concatenation across lifters.

Mypy clean under .311/bin/mypy --ignore-missing-imports.
2026-05-01 06:20:10 -04:00
ee24a7551f fix(types): T7 — eliminate all remaining 38 mypy errors; fix DeckyRow subscript in engine tests 2026-05-01 02:07:53 -04:00
d637ff515e fix(types): T3 — narrow str|None at 12 sites; fix LANRow/DeckyRow subscript in mutator tests 2026-05-01 01:47:04 -04:00
fc1f0914b7 refactor(topology): introduce TopologyRepository protocol with DTO return types
Replace repo: BaseRepository with a structural TopologyRepository protocol
in persistence.py and allocator.py. All read methods now return typed DTOs
(TopologySummary, LANRow, DeckyRow, EdgeRow) instead of raw dicts, eliminating
silent field-shape regressions across the topology subsystem.

TopologySummary gains email_personas and language_default so api_personas.py
can continue reading those fields via attribute access. hydrate() converts
DTOs to dicts before passing to _backfill_decky_configs, keeping the mutable
working-state function dict-based at its boundary. All production callers
(router handlers, mutator, CLI, heartbeat) migrated from dict/get access to
attribute access. 134 tests pass.
2026-04-30 23:51:41 -04:00
0b5228eb94 feat(config): add swarmctl-host to INI, env, CLI; drop hardcoded bind from systemd unit
[swarm] swarmctl-host → DECNET_SWARMCTL_HOST so operators set the bind
address once in decnet.ini; `decnet swarmctl` and the systemd unit both
resolve it via envvar — no --host/--port pinned on ExecStart.
2026-04-30 22:16:00 -04:00
ebe15310ab fix(api): hydrate planner from DB exactly once on first GET, not on every read
get_config was calling planner.apply_payload on every GET request, racing
concurrent reads on module-level globals. Added a _hydrated flag + lock
so DB hydration runs at most once per process lifetime; put_config marks
it done too. Test fixture resets the flag between tests.
2026-04-30 21:17:03 -04:00
f597d70430 fix(realism): use minute-precision datetime in in_active_hours
personas.in_active_hours was discarding the minute component of the
active-hours window, making "09:30-17:45" behave as "09:00-17:00".
Rewrote it to delegate to diurnal.in_work_hours (which uses full
minute arithmetic) and updated the scheduler caller to pass the full
datetime instead of now_dt.hour.
2026-04-30 21:14:36 -04:00
542d129d6f refactor(services_live): replace string-sniffed error dispatch with typed exception subclasses
ServiceNotFoundError (→ 404) and ServiceConflictError (→ 409) replace the
"not found" / "already on" / "not on" substring checks in _map_mutation_error;
base ServiceMutationError still maps to 422. Fixes three pre-existing test
status-code assertions (201 vs 200 on POST endpoints).
2026-04-30 20:49:29 -04:00
761c23a07c fix(smtp_relay): emit service=smtp_relay in syslog so ingester can gate probe publish
SERVICE_NAME was hardcoded to 'smtp' in server.py; the ingester's probe
publish guard checked service == 'smtp_relay' and never matched.

Read SMTP_SERVICE_NAME from env (default 'smtp'); smtp_relay compose
fragment sets it to 'smtp_relay' so the two services are distinguishable.
2026-04-30 12:31:29 -04:00
8ae7b9636e feat(smtp_relay): move probe forwarding to realism worker via bus
Attacker probe emails are now forwarded by the master (realism worker)
rather than inside the MACVLAN container, which has no internet gateway.

- New smtp.probe.pending bus topic: ingester publishes when smtp_relay
  message_stored fires; worker subscribes and does the actual delivery
- decnet/orchestrator/drivers/smtp_relay.py: pure-sync forward_probe()
  reads the .eml from disk and sends via smtplib on a thread executor
- worker.py: _run_smtp_probe_listener + _handle_probe_pending subtask;
  limit enforced via count_probe_relays() (DB-backed, restart-safe)
- bounties.py: count_probe_relays() query on probe_relay bounty type
- fleet.py: get_fleet_decky_by_name() to pull service config from DB
- services/smtp_relay.py: upstream_* and probe_limit fields defined in
  config_schema but NOT injected into container env (credentials stay
  out of docker env vars)
- ingester.py: stripped of smtplib; publishes probe.pending and exits
- tests: assert upstream keys absent from container environment
2026-04-30 12:10:58 -04:00
9a4fe2677b feat(smtp_relay): forward probe emails upstream so attackers verify relay works
First SMTP_PROBE_LIMIT messages per source IP are forwarded via a real
upstream relay (SMTP_UPSTREAM_HOST/PORT/USER/PASS) so the attacker's
test email actually lands in their inbox. All subsequent messages from
the same IP get 250 Ok but only hit the quarantine — campaign content
captured, nothing delivered.
2026-04-30 11:21:04 -04:00
07b32e2abe fix(tests): patch add_service/remove_service at the router import, not the module
Monkeypatching services_live.add_service had no effect because api_services
already held a local reference to the name. Patch api_services.add_service
and update fake stubs to accept the config kwarg added to the real signature.
2026-04-29 18:50:21 -04:00
2fc5f1bdc5 feat(canary): auto-deregister fingerprint slug after first valid beacon
Once a fingerprint canary's HTTP beacon passes all 4 validation layers
and the trigger row lands, the token is immediately set to state=revoked
and canary.<id>.revoked is published on the bus. The slug lookup is
tightened to only return planted tokens, so subsequent requests to the
same URL silently return the transparent GIF without persisting anything
(stealth posture preserved). Plain http/dns canaries with no
fingerprint_nonce are not affected.

Changes:
- sqlmodel_repo/canary.py: add state == "planted" filter to
  get_canary_token_by_slug so revoked slugs resolve to None
- worker.py: after record_canary_trigger, if parsed_fp survived all
  layers and token has a fingerprint_nonce, call
  update_canary_token_state("revoked") + publish CANARY_REVOKED; errors
  are best-effort (trigger row already landed)
- test_worker_http.py: assert state=revoked in test_fp_valid_nonce_persists;
  new test_fp_deregisters_slug_after_valid_hit (second hit records nothing);
  new test_plain_http_canary_not_deregistered (env_file stays planted)
2026-04-29 17:49:31 -04:00
b26dd8f529 feat(canary): API-trashing defense — 4-layer fingerprint validation
Adds per-mint nonce gating, structural shape validation, mint UUID
consistency checks, and a per-(token, IP) rate limiter to the canary
worker so attackers who extract a canary from a decky filesystem cannot
poison fingerprint forensics by replaying or forging ?d= submissions.

Changes:

base.py
  fingerprint_nonce: Optional[str] added to CanaryArtifact so generators
  can surface the nonce to the cultivator without coupling the generator
  directly to DB code.

obfuscator.py
  nonce_for(callback_token, mint_uuid): HMAC-SHA256 keyed on
  DECNET_CANARY_FINGERPRINT_SECRET, truncated to 16 hex chars.
  FingerprintSecretMissing raised at mint time if env var is unset.
  render_fingerprint_js() now accepts nonce= and substitutes MINT_NONCE.

fingerprint_payload.js
  New MINT_NONCE placeholder. Appended as &k= on all beacon URLs (bare-open,
  single-shot, chunked). Using &k= avoids colliding with &n= (chunk total).

fingerprint_html.py / fingerprint_svg.py
  Derive nonce via nonce_for() and pass to render_fingerprint_js(). Set
  artifact.fingerprint_nonce so the cultivator can persist it.

cultivator.py
  Passes fingerprint_nonce into create_canary_token() when present on the
  artifact; NULL for all non-fingerprint generators.

canary.py (model)
  fingerprint_nonce: Optional[str] = Field(default=None, max_length=16)
  added to CanaryToken. None for non-fingerprint tokens.

worker.py
  _extract_fingerprint now returns (meta_dict, parsed_fp) tuple.
  _record_hit accepts parsed_fp + raw_nonce and runs 4 layers after
  token lookup: nonce match, shape check, mint UUID consistency, rate limit.
  Each failure sets _fp_invalid_* flag and drops structured _fp.
  Trigger row always lands regardless.

tests/canary/conftest.py
  Session-scoped autouse fixture sets DECNET_CANARY_FINGERPRINT_SECRET so
  fingerprint generator and worker tests work offline.

tests
  5 new worker HTTP tests and 2 new generator tests covering each
  validation layer.
2026-04-29 17:41:04 -04:00
f86dc79990 feat(canary): ship Node helper with wheel + install-toolchain CLI
The fingerprint canaries' obfuscator shells out to a Node helper that
require()s javascript-obfuscator. Without this commit, a fresh
pip install decnet would land the .py modules but not the .js helper /
package.json, and there'd be no documented way to provision Node side.

* pyproject.toml - extend tool.setuptools.package-data to ship
  canary/_obfuscate_helper.js, canary/fingerprint_payload.js, and
  canary/package.json with the wheel.
* decnet/cli/canary.py - new "decnet canary-install-toolchain"
  subcommand. Resolves decnet.canary.__file__'s dir, runs
  npm install --omit=dev there, exits non-zero with a clear message
  if npm is missing or install fails. Idempotent - safe to call
  every API service start.
* deploy/decnet-api.service.j2 - non-fatal ExecStartPre that calls
  the new subcommand. Leading '-' so a missing Node toolchain only
  degrades fingerprint canaries (loud at mint time) without keeping
  the API from booting.
* tests/canary/test_cli.py - registration smoke test, missing-npm
  exit path, and a mocked-subprocess test asserting the right argv
  and cwd land on npm.

Realism cultivator already has a broad except Exception around
cultivate() in scheduler.py:195-211, so a missing toolchain on a
host running the realism tick degrades to an inert noise file with
no extra plumbing.
2026-04-29 16:53:27 -04:00