Commit Graph

136 Commits

Author SHA1 Message Date
dc1867315d feat(ttp): E.3.8 R0049-R0053 canary fingerprint cohort
5 YAMLs for the canary-fingerprint cohort per Appendix B / A.9:
navigator.webdriver flag, automation canvas/audio/WebGL hash match,
WebRTC IP leak, TZ/lang vs geo mismatch, platform inconsistency.
CanaryFingerprintLifter (E.3.11) consumes by rule_id.

test_canary_rules.py: YAML-present + inert-in-v0 + xfail(strict)
gated on E.3.11.
2026-05-01 09:21:01 -04:00
1ad15470a1 feat(ttp): E.3.8 R0041-R0048 email cohort
8 YAMLs for the email cohort per Appendix B: open-relay abuse,
mass phishing, phishing-kit X-Mailer signatures, IDN/punycode
URLs, sender masquerade, malicious attachment, BEC, encoded
payload in body. EmailLifter (E.3.12) consumes by rule_id.

test_email_rules.py: YAML-present + inert-in-v0 + xfail(strict)
precision case gated on E.3.12.
2026-05-01 09:19:56 -04:00
806301e179 feat(ttp): E.3.8 R0031-R0040 behavioral cohort
10 YAMLs for the behavioral / cross-event cohort per Appendix B:
beaconing, data destruction, ransom note, web exfil, DB mass-read,
credentials-in-files, k8s SA token harvest, Docker host escape,
LLMNR poisoning, TFTP router-config retrieval.

Every rule is lifter-bound (BehavioralLifter / IdentityLifter) —
the v0 RuleEngine cannot count, aggregate, or compose cross-event
signals, so these YAMLs declare the technique mappings the lifter
will consume by rule_id at E.3.9. Their match specs use a
'kind: lifter:*' shape inert to the regex matcher.

test_behavioral_rules.py asserts each YAML compiles, none fire
from the v0 engine (FP regression guard against a YAML drifting
into a regex), and an xfail(strict=True, reason='impl phase E.3.9')
precision case that will flip green when the lifter lands.
2026-05-01 09:18:27 -04:00
b1fe1f9403 feat(ttp): E.3.8 R0001-R0030 command cohort
30 YAMLs for the shell/command rule cohort per Appendix B (rules/ttp/).
Splits into engine-active (R0007-R0029, regex on command_text /
raw_url / user_agent) and lifter-bound (R0001-R0006, R0030 — the
v0 RuleEngine cannot count auth attempts, do identity rollups, or
parse fingerprint blobs; the BehavioralLifter / IdentityLifter /
CredentialLifter consume them by rule_id at E.3.9 / E.3.13).

test_command_rules.py asserts:
- every R000N has a YAML that compiles
- lifter-bound rules NEVER fire from the v0 engine (regression
  guard against a YAML drifting into a regex match.spec)
- engine-active rules meet their Appendix-C precision target
  against the seed corpus (≥0.95 high-conf, ≥0.80 medium)

Conftest fixes: precision_engine moved to module-scope so module-
scope precomputed dispatch fixture (fired_by_label) can request it;
_RULES_DIR path bumped from parents[2] to parents[3] so the loader
resolves the project root regardless of pytest cwd; make_event
synthesizes attacker_uuid so TTPTag's anchor invariant is satisfied.

Seed corpus broadened: positive examples for every regex rule plus
6 negative examples across innocuous shell verbs (ls, echo, cd, ps,
df, free) so FPs surface in precision rather than passing vacuously.
2026-05-01 09:16:38 -04:00
c635478442 feat(ttp): E.3.8 corpus + harness — labelled holdout fixture
Sub-step preceding the rule-pack commits per TTP_TAGGING.md:2967.
Adds the per-rule precision suite scaffolding under
tests/ttp/rule_precision/:

- conftest.py: precision_engine fixture (RuleEngine populated from
  ./rules/ttp/), corpus_loader (real → seed → empty fallback),
  precision_for() helper for TP/FP accounting.
- _build_corpus.py: extractor for a real prod corpus pull. Mandatory
  --exclude-ip / DECNET_TTP_CORPUS_EXCLUDE_IPS — operator IPs never
  end up in the committed exclusion list. Pulls both 'command' and
  'unknown_command' event types.
- corpus/seed_*.jsonl: synthetic seed rows for each cohort so the
  harness exercises in clean checkouts.
- corpus/*.jsonl (operator-built) is gitignored.
- test_corpus_loads.py: sentinel that every seed file parses.
2026-05-01 09:08:07 -04:00
ed3f340ea8 feat(ttp): E.3.7 RuleEngine — evaluate + atomic-swap watch_store
Implements the rule engine body left empty at contract phase: evaluate()
dispatches by source_kind through self._by_kind, runs the rule's match
spec against event.payload, and emits one TTPTag per emits entry.
watch_store() loads the initial corpus from RuleStore.load_compiled,
then drains subscribe_changes, applying definition changes via
single-statement dict assignment (atomic swap, GIL-atomic to readers)
and state changes via NamedTuple._replace on the existing CompiledRule.

Why: with the FS + DB stores in place (E.3.5/E.3.6), the engine is the
last piece of the rule plane. Lifters (E.3.9–E.3.13) consume the
engine; the worker bootstrap (E.3.14) wires watch_store into the
asyncio event loop. After this commit a CompositeTagger constructed
with a RuleEngine + a populated rules dir will produce real tags.

Notes:
- CompiledRule.emits extended to 4-tuple
  (technique_id, sub_technique_id, tactic, confidence). Tactic + confidence
  ride per-emit so a single rule can carry multiple precision targets
  (the "one event maps to many techniques" property). Compile helpers in
  both backends extract them from the YAML emits dict; missing tactic
  or confidence is a deploy-time error.
- v0 match operator is "pattern" (regex). The field defaults per
  source_kind (command_text / raw_url / subject / verdict / …) and is
  overridable via match.field. Future ops (contains, equals, in_set)
  extend _match_event without touching the engine surface.
- Confidence model: rules with state="clipped" + confidence_max set
  cap the per-emit confidence downward; clipped is a soft suppress, not
  a hard skip. Disabled rules are skipped wholly; expires_at past is
  re-checked at evaluate as defense-in-depth (the store auto-reverts,
  but a racing read between expiry and revert must not fire the rule).
- _span(name, **attrs) helper in engine + both stores short-circuits on
  decnet.telemetry._ENABLED — matches the project's @traced /
  wrap_repository zero-overhead-when-disabled pattern instead of relying
  solely on the no-op tracer indirection.
- Late-bound tracer (telemetry.get_tracer called per-span, not at
  module load) so test_tracing's monkeypatch reaches the production
  code path.

xfails flipped: tests/ttp/test_rule_engine.py multi-emit fan-out +
rule_version-collision-via-engine; tests/ttp/test_multi_mapping.py
N×M engine fan-out + idempotent replay; tests/ttp/test_tracing.py
ttp.eval span hierarchy + ttp.rule.fire span attributes.

Tests: 214 passed, 19 xfailed (gated on E.3.8 lifters / rule pack /
worker bootstrap).
mypy: clean on prod code; pre-existing test-stub arg-type warnings
unchanged.
2026-05-01 08:49:15 -04:00
8a93ee3129 feat(ttp): E.3.6 DatabaseRuleStore — ttp_rule/ttp_rule_state + master sync
Implements the DB-backed rule store body left empty at contract phase:
load_compiled reads from ttp_rule + ttp_rule_state; get_state /
set_state hit ttp_rule_state with the same expires_at auto-revert and
bus-event semantics as the FS backend; subscribe_changes returns a
per-subscriber queue. State persists across process restarts — the
swarm property the FS backend deliberately doesn't have.

Also lands two swarm-mode helpers:
- sync_from_filesystem(fs_store) — master-side, subscribes to a
  FilesystemRuleStore and projects each RuleChange onto a ttp_rule
  upsert/delete.
- tail_db(poll_interval) — worker-side, watermark poll over
  ttp_rule.updated_at; emits RuleChange("definition", ...) for each
  row that moved.

Why: swarm mode needs rule definitions and operator state to
propagate across hosts. The filesystem backend (E.3.5) was the
single-host-dev variant; this one survives restart and serves N
workers from a shared DB.

Notes:
- DatabaseRuleStore() with no args lazy-inits an in-memory SQLite
  repo so the conformance fixture works without test plumbing. In
  production the worker bootstrap (E.3.14) passes an explicit repo.
- The conftest.py rule_store fixture became async (pytest_asyncio),
  per-backend creates/initializes a SQLite repo for the DB run.
- Adds a `seed_rule(store, rule_id, yaml)` helper to bridge backend
  semantics: drop a YAML file (FS) vs insert a ttp_rule row (DB).
  Used by the parametrized load_compiled conformance test.
- Late-bound _tracer() in both backends (was module-level get_tracer
  binding) so test_tracing's monkeypatch of decnet.telemetry.get_tracer
  actually affects span output.

xfails flipped: tests/ttp/store/test_database.py set_state-writes-to-
ttp_rule_state + filesystem-to-DB sync; tests/ttp/store/test_conformance.py
DB-side load_compiled / set_state isolation / round-trip / per-rule
fan-out / expired-state revert / set_state failure / get_state default
(was xfail-only-on-DB);  tests/ttp/test_tracing.py set_state span
hierarchy.

Tests: 208 passed, 25 xfailed (gated on E.3.7 + lifters).
mypy: clean on all touched files.
2026-05-01 08:39:46 -04:00
f41995a229 feat(ttp): E.3.5 FilesystemRuleStore — inotify hot-reload + per-rule events
Implements the filesystem-backed rule store body left empty at contract
phase: YAML parse + Pydantic validation, asyncinotify watch over
./rules/ttp/, in-process state cache with auto-revert on expires_at,
and a subscribe_changes() async iterator yielding one RuleChange per
per-rule edit. Bus topic builders ttp_rule_reloaded / ttp_rule_state
ship alongside.

Why: the rule plane needed a store before the engine (E.3.7) could
consume RuleChange events and atomically swap compiled rules into its
dispatch index.

Notes:
- Linux-only by construction (asyncinotify wheel gated by sys_platform
  marker; FilesystemRuleStore.__init__ raises on non-Linux).
- Filename allowlist is the FIRST check on every inotify event.
- Content-hash dedup so a single write firing IN_CREATE + IN_CLOSE_WRITE
  produces exactly one RuleChange.
- All compile work serializes on a single asyncio.Lock.
- Subscribers register their queue eagerly so events fired between
  subscribe_changes() and the first __anext__() are buffered.

xfails flipped: per-save-style + filter-ordering + atomic-swap in
test_filesystem.py; load_compiled / set_state isolation / round-trip /
per-rule fan-out / expired-state revert / set_state failure semantics
in test_conformance.py (FS side; DB side stays xfail until E.3.6);
malformed-YAML compile-time check in test_rule_engine.py.

Tests: 197 passed, 35 xfailed (gated on E.3.6 / E.3.7 / lifters).
mypy + bandit: clean on all touched files.

Wiki update for the per-rule reload + state-change topics lands in a
matching wiki-checkout/Service-Bus.md edit (separate repo).
2026-05-01 08:31:05 -04:00
fee697694d feat(ttp): E.3.3 repository — insert_tags + listing rollups (dual backend)
Dialect-split: portable rollup queries on TTPMixin; bulk insert with
ON CONFLICT DO NOTHING / INSERT IGNORE in the per-dialect repos.
Confidence-floor (< 0.3) drop applied at mixin layer before the
dialect hook. BaseRepository now declares the six TTP methods abstract.

Tests in tests/web/db/test_ttp_repo.py flipped from pytest.fail stubs
to real dual-backend behavioral tests; tests/ttp/test_confidence.py
drop-below-floor xfail removed.
2026-05-01 08:04:46 -04:00
0217319423 test(ttp): E.2.14b RuleStore conformance — cross-backend + filesystem-specific + database-specific
tests/ttp/store/conftest.py — parametrized rule_store fixture over
FilesystemRuleStore (skipped on non-Linux) + DatabaseRuleStore.

test_conformance.py — shared assertions (default-state, set_state
isolation/round-trip, subscribe_changes per-rule fan-out, expires_at
auto-revert, set_state failure semantics) parametrize over both.
get_state-default GREEN today on FS (returns RuleState() for empty
cache); rest xfail-gated behind E.3.5/E.3.6.

test_filesystem.py — inotify mask + canonical kernel values + 9
scratch-filename rejections + 4 valid-filename acceptances +
fullmatch anchor + tmp_path construction + CompiledRule frozen
property GREEN today; per-save-style + filter-ordering +
atomic-swap concurrency xfail-gated.

test_database.py — class-level surface (no platform guard, ABC
methods concrete, async coroutines) GREEN today; ttp_rule_state
write + filesystem→DB sync xfail-gated behind E.3.6.
2026-05-01 07:45:32 -04:00
bf5414c0d1 test(ttp): E.2.14a follow-up — force DECNET_DEVELOPER_TRACING=true, skip when Jaeger unreachable
Session-scoped autouse fixture in tests/ttp/conftest.py sets
DECNET_DEVELOPER_TRACING=true and forces decnet.telemetry._ENABLED
so the no-op tracer doesn't silently swallow emitted spans. The
span_exporter fixture also monkeypatches decnet.telemetry.get_tracer
so production code under test lands spans in the in-memory
exporter. Tracing tests skip when DECNET_OTEL_ENDPOINT (default
localhost:4317) isn't reachable so the dev loop stays green
without lying about coverage.
2026-05-01 07:42:22 -04:00
f4fe6fe6e4 test(ttp): E.2.14a observability tracing — span hierarchy + no-PII property
In-memory span exporter fixture wired to a per-test TracerProvider
(OTEL global is locked once set, so each test gets its own).
ttp.eval / ttp.lifter.{name} / ttp.rule.fire / ttp.rule.state.change
hierarchy + no-PII canary battery xfail-gated behind E.3.5–E.3.13.
2026-05-01 07:40:58 -04:00
4a93e16407 test(ttp): E.2.13 repository tests — TTPMixin idempotency + identity-rollup projection on dual backends
Adds tests/web/db/conftest.py with a db_backends fixture
parametrizing SQLite (always) + MySQL (gated on
DECNET_TEST_MYSQL_URL). Surface assertions (mixin methods present
+ async) GREEN today; insert_tags idempotency, identity rollup
projection, attacker-rollup exclusion of NULL-attacker tags
xfail-gated behind E.3.3.
2026-05-01 07:39:16 -04:00
6814949bc0 test(ttp): E.2.12 worker bus integration — _TOPICS equality, loop-prevention, delivery asymmetry
Pin _TOPICS frozenset against documented set (single source of
truth). Worker→engine invocation, loop-prevention invariant,
attacker.enriched/email.received catch-up asymmetry xfail-gated
behind E.3.14.
2026-05-01 07:37:58 -04:00
c276b5696e test(ttp): E.2.11 multi-mapping property — N×M fan-out, idempotent UUID, replay-safety
Hypothesis property: N rule_ids × M technique_ids on one event yield
N×M distinct tag UUIDs. Worked example pinned: one rule emitting
(T1110, None) and (T1078, None) → two distinct UUIDs. Engine-level
fan-out + replay xfail-gated behind E.3.7.
2026-05-01 07:36:19 -04:00
fd81be0bb1 test(ttp): E.2.10 confidence model — downward-only multiplier property, drop-below-0.3, AbuseIPDB-30 worked example
Pure-arithmetic adjustment formula pinned via Hypothesis property
test (multiplier ∈ [0,1] cannot raise base). Drop-at-floor and
provider-score multiplier xfail-gated behind E.3.3 / E.3.10.
2026-05-01 07:34:58 -04:00
79e6df8343 test(ttp): E.2.9 UKC bridge bijection — pin tactic↔phase mapping, observable round-trip, lossy phases
Pre-target phases (RECONNAISSANCE/RESOURCE_DEVELOPMENT/WEAPONIZATION/
SOCIAL_ENGINEERING) and observable-but-unmappable phases (EXPLOITATION/
PIVOTING/OBJECTIVES, UKC-only concepts ATT&CK lacks tactics for) are
pinned as lossy via _LOSSY_INVERSE_REFERENCE so a future contributor
cannot 'fix' the asymmetry without tripping the suite.
2026-05-01 07:33:47 -04:00
b7f206c8c5 feat(ttp): E.1.9 API contract — seven router endpoints, admin-gated state mutations, response models
Mounts /api/v1/ttp/* with empty-list / empty-Navigator responses.
GET endpoints viewer-gated; POST/DELETE /rules/{rule_id}/state
admin-gated server-side. POST parses JSON manually so a malformed
body returns the documented 400 (per feedback_schemathesis_400).

Drops xfail-strict markers from E.2.8 tests now that the router is
mounted; 26 tests pass against the contract handlers.
2026-05-01 07:20:13 -04:00
b5a19301a2 test(ttp): E.2.8 API shape + auth — GET 200/401 + admin-only POST/DELETE 401/403/200/400 contract 2026-05-01 07:00:41 -04:00
0cdf8d90da test(ttp): E.2.7 decoupling lint — TTP code may not import decnet.intel.* providers or decnet.profiler.keystroke 2026-05-01 06:58:12 -04:00
e2078c868d test(ttp): E.2.6 lifter tolerates absence — six lifters return [] on empty joins, no ERROR logs 2026-05-01 06:57:29 -04:00
1ffaa3df41 test(ttp): E.2.5 RuleEngine behavior — empty store, malformed YAML, multi-emit fan-out, version collisions 2026-05-01 06:56:28 -04:00
5accf8f1b1 test(ttp): E.2.4 Tagger ABC conformance — hypothesis fuzz over swallowed Exception types 2026-05-01 06:54:29 -04:00
cce84f23dc test(bus): E.2.3 TTP topic naming — constants, builders, wildcard match 2026-05-01 06:53:05 -04:00
e58aa4fe3a test(ttp): E.2.2 idempotency — determinism, golden value, replay-safety signature lock 2026-05-01 06:45:49 -04:00
e6f1da2344 test(ttp): E.2.1b evidence shape — TypedDict keys, PII §6 type-level assertion 2026-05-01 06:45:35 -04:00
c3a799726f test(ttp): E.2.1 schema invariant tests — CHECK, ValueError guard, UUIDv5, JSON round-trip 2026-05-01 06:44:57 -04:00
19cc8aa859 feat(ttp): E.1.7 worker contract — run_ttp_worker_loop, _TOPICS, registry entry 2026-05-01 06:33:34 -04:00
208ffd8f4f feat(ttp): E.1.6 per-lifter contracts — six TolerantTagger subclasses 2026-05-01 06:31:31 -04:00
cb9d183c20 feat(ttp): E.1.5 RuleEngine contract — CompiledRule, RuleSchema, RuleEngine ABC 2026-05-01 06:30:12 -04:00
c3c5813211 feat(ttp): E.1.3+E.1.4 Tagger ABC and composite factory contract
Third and fourth TTP-tagging contract commits, plus a scoped subset
of the E.2.4 conformance tests covering the contract surface shipped
here (full hypothesis-fuzz suite still lands with E.2.4).

E.1.3 — decnet/ttp/base.py
- TaggerEvent NamedTuple: source_kind, source_id, attacker_uuid,
  identity_uuid, session_id, decky_id, opaque payload.
- Tagger(ABC) with abstract async tag(); class-level name and
  HANDLES: frozenset[str] (default empty so a misconfigured subclass
  is loudly idle, not loudly noisy).
- TolerantTagger(Tagger): concrete tag() wraps abstract _tag_impl()
  in try/except Exception (deliberately not BaseException — so
  KeyboardInterrupt / SystemExit / asyncio.CancelledError propagate
  and the worker can shut down cleanly). Swallowed exceptions log
  at WARNING with exc_info, never ERROR — absence is the steady
  state, not a bug. Subclasses override _tag_impl, never tag — the
  tolerance contract is enforced in the base class, not on trust.
- KNOWN_SOURCE_KINDS: Final[frozenset[str]] enumerating every
  source_kind a producer is allowed to emit. Closed-by-enumeration
  at the runtime layer; the composite tagger keys its WARNING/INFO
  bridge off this constant to surface the silent-drop trap from
  the design doc (lines 160–195).

E.1.4 — decnet/ttp/factory.py
- get_tagger() reads DECNET_TTP_TAGGER_TYPE (default 'composite');
  unknown values raise ValueError with the known-list. Mirrors
  decnet.intel.factory and decnet.clustering.factory.
- _KNOWN = ('composite',). Per-lifter classes (E.1.6) are children
  of the composite, not standalone tagger types.
- CompositeTagger(Tagger): pre-computes a dict[str, list[Tagger]]
  dispatch index from each lifter's HANDLES; fans events out
  concurrently with asyncio.gather and concatenates results.
  Empty lifters=[] is the legal contract-phase state — E.1.6
  wires the real lifters in.
- Unhandled-event observability: source_kind in KNOWN_SOURCE_KINDS
  but no lifter claims it -> WARNING once per kind per process
  (missed E.1.6 update). Unknown kind -> INFO once per kind per
  process (future-feature telemetry, by design). Per-process dedup
  via plain set; E.1.6 may swap in a proper rate-limiter once
  production traffic shapes are known.

Tests — tests/ttp/test_base.py, tests/ttp/test_factory.py
- Tagger / TolerantTagger abstractness, missing-tag-impl rejection,
  WARNING-not-ERROR log level, propagation of KeyboardInterrupt /
  SystemExit / asyncio.CancelledError.
- Factory env-var routing, unknown-name ValueError, dispatch-index
  correctness, only-claiming-lifter invocation, WARNING-once for
  known-but-unclaimed kinds, INFO-once for unknown kinds, result
  concatenation across lifters.

Mypy clean under .311/bin/mypy --ignore-missing-imports.
2026-05-01 06:20:10 -04:00
ee24a7551f fix(types): T7 — eliminate all remaining 38 mypy errors; fix DeckyRow subscript in engine tests 2026-05-01 02:07:53 -04:00
d637ff515e fix(types): T3 — narrow str|None at 12 sites; fix LANRow/DeckyRow subscript in mutator tests 2026-05-01 01:47:04 -04:00
fc1f0914b7 refactor(topology): introduce TopologyRepository protocol with DTO return types
Replace repo: BaseRepository with a structural TopologyRepository protocol
in persistence.py and allocator.py. All read methods now return typed DTOs
(TopologySummary, LANRow, DeckyRow, EdgeRow) instead of raw dicts, eliminating
silent field-shape regressions across the topology subsystem.

TopologySummary gains email_personas and language_default so api_personas.py
can continue reading those fields via attribute access. hydrate() converts
DTOs to dicts before passing to _backfill_decky_configs, keeping the mutable
working-state function dict-based at its boundary. All production callers
(router handlers, mutator, CLI, heartbeat) migrated from dict/get access to
attribute access. 134 tests pass.
2026-04-30 23:51:41 -04:00
0b5228eb94 feat(config): add swarmctl-host to INI, env, CLI; drop hardcoded bind from systemd unit
[swarm] swarmctl-host → DECNET_SWARMCTL_HOST so operators set the bind
address once in decnet.ini; `decnet swarmctl` and the systemd unit both
resolve it via envvar — no --host/--port pinned on ExecStart.
2026-04-30 22:16:00 -04:00
ebe15310ab fix(api): hydrate planner from DB exactly once on first GET, not on every read
get_config was calling planner.apply_payload on every GET request, racing
concurrent reads on module-level globals. Added a _hydrated flag + lock
so DB hydration runs at most once per process lifetime; put_config marks
it done too. Test fixture resets the flag between tests.
2026-04-30 21:17:03 -04:00
f597d70430 fix(realism): use minute-precision datetime in in_active_hours
personas.in_active_hours was discarding the minute component of the
active-hours window, making "09:30-17:45" behave as "09:00-17:00".
Rewrote it to delegate to diurnal.in_work_hours (which uses full
minute arithmetic) and updated the scheduler caller to pass the full
datetime instead of now_dt.hour.
2026-04-30 21:14:36 -04:00
542d129d6f refactor(services_live): replace string-sniffed error dispatch with typed exception subclasses
ServiceNotFoundError (→ 404) and ServiceConflictError (→ 409) replace the
"not found" / "already on" / "not on" substring checks in _map_mutation_error;
base ServiceMutationError still maps to 422. Fixes three pre-existing test
status-code assertions (201 vs 200 on POST endpoints).
2026-04-30 20:49:29 -04:00
761c23a07c fix(smtp_relay): emit service=smtp_relay in syslog so ingester can gate probe publish
SERVICE_NAME was hardcoded to 'smtp' in server.py; the ingester's probe
publish guard checked service == 'smtp_relay' and never matched.

Read SMTP_SERVICE_NAME from env (default 'smtp'); smtp_relay compose
fragment sets it to 'smtp_relay' so the two services are distinguishable.
2026-04-30 12:31:29 -04:00
8ae7b9636e feat(smtp_relay): move probe forwarding to realism worker via bus
Attacker probe emails are now forwarded by the master (realism worker)
rather than inside the MACVLAN container, which has no internet gateway.

- New smtp.probe.pending bus topic: ingester publishes when smtp_relay
  message_stored fires; worker subscribes and does the actual delivery
- decnet/orchestrator/drivers/smtp_relay.py: pure-sync forward_probe()
  reads the .eml from disk and sends via smtplib on a thread executor
- worker.py: _run_smtp_probe_listener + _handle_probe_pending subtask;
  limit enforced via count_probe_relays() (DB-backed, restart-safe)
- bounties.py: count_probe_relays() query on probe_relay bounty type
- fleet.py: get_fleet_decky_by_name() to pull service config from DB
- services/smtp_relay.py: upstream_* and probe_limit fields defined in
  config_schema but NOT injected into container env (credentials stay
  out of docker env vars)
- ingester.py: stripped of smtplib; publishes probe.pending and exits
- tests: assert upstream keys absent from container environment
2026-04-30 12:10:58 -04:00
9a4fe2677b feat(smtp_relay): forward probe emails upstream so attackers verify relay works
First SMTP_PROBE_LIMIT messages per source IP are forwarded via a real
upstream relay (SMTP_UPSTREAM_HOST/PORT/USER/PASS) so the attacker's
test email actually lands in their inbox. All subsequent messages from
the same IP get 250 Ok but only hit the quarantine — campaign content
captured, nothing delivered.
2026-04-30 11:21:04 -04:00
07b32e2abe fix(tests): patch add_service/remove_service at the router import, not the module
Monkeypatching services_live.add_service had no effect because api_services
already held a local reference to the name. Patch api_services.add_service
and update fake stubs to accept the config kwarg added to the real signature.
2026-04-29 18:50:21 -04:00
2fc5f1bdc5 feat(canary): auto-deregister fingerprint slug after first valid beacon
Once a fingerprint canary's HTTP beacon passes all 4 validation layers
and the trigger row lands, the token is immediately set to state=revoked
and canary.<id>.revoked is published on the bus. The slug lookup is
tightened to only return planted tokens, so subsequent requests to the
same URL silently return the transparent GIF without persisting anything
(stealth posture preserved). Plain http/dns canaries with no
fingerprint_nonce are not affected.

Changes:
- sqlmodel_repo/canary.py: add state == "planted" filter to
  get_canary_token_by_slug so revoked slugs resolve to None
- worker.py: after record_canary_trigger, if parsed_fp survived all
  layers and token has a fingerprint_nonce, call
  update_canary_token_state("revoked") + publish CANARY_REVOKED; errors
  are best-effort (trigger row already landed)
- test_worker_http.py: assert state=revoked in test_fp_valid_nonce_persists;
  new test_fp_deregisters_slug_after_valid_hit (second hit records nothing);
  new test_plain_http_canary_not_deregistered (env_file stays planted)
2026-04-29 17:49:31 -04:00
b26dd8f529 feat(canary): API-trashing defense — 4-layer fingerprint validation
Adds per-mint nonce gating, structural shape validation, mint UUID
consistency checks, and a per-(token, IP) rate limiter to the canary
worker so attackers who extract a canary from a decky filesystem cannot
poison fingerprint forensics by replaying or forging ?d= submissions.

Changes:

base.py
  fingerprint_nonce: Optional[str] added to CanaryArtifact so generators
  can surface the nonce to the cultivator without coupling the generator
  directly to DB code.

obfuscator.py
  nonce_for(callback_token, mint_uuid): HMAC-SHA256 keyed on
  DECNET_CANARY_FINGERPRINT_SECRET, truncated to 16 hex chars.
  FingerprintSecretMissing raised at mint time if env var is unset.
  render_fingerprint_js() now accepts nonce= and substitutes MINT_NONCE.

fingerprint_payload.js
  New MINT_NONCE placeholder. Appended as &k= on all beacon URLs (bare-open,
  single-shot, chunked). Using &k= avoids colliding with &n= (chunk total).

fingerprint_html.py / fingerprint_svg.py
  Derive nonce via nonce_for() and pass to render_fingerprint_js(). Set
  artifact.fingerprint_nonce so the cultivator can persist it.

cultivator.py
  Passes fingerprint_nonce into create_canary_token() when present on the
  artifact; NULL for all non-fingerprint generators.

canary.py (model)
  fingerprint_nonce: Optional[str] = Field(default=None, max_length=16)
  added to CanaryToken. None for non-fingerprint tokens.

worker.py
  _extract_fingerprint now returns (meta_dict, parsed_fp) tuple.
  _record_hit accepts parsed_fp + raw_nonce and runs 4 layers after
  token lookup: nonce match, shape check, mint UUID consistency, rate limit.
  Each failure sets _fp_invalid_* flag and drops structured _fp.
  Trigger row always lands regardless.

tests/canary/conftest.py
  Session-scoped autouse fixture sets DECNET_CANARY_FINGERPRINT_SECRET so
  fingerprint generator and worker tests work offline.

tests
  5 new worker HTTP tests and 2 new generator tests covering each
  validation layer.
2026-04-29 17:41:04 -04:00
f86dc79990 feat(canary): ship Node helper with wheel + install-toolchain CLI
The fingerprint canaries' obfuscator shells out to a Node helper that
require()s javascript-obfuscator. Without this commit, a fresh
pip install decnet would land the .py modules but not the .js helper /
package.json, and there'd be no documented way to provision Node side.

* pyproject.toml - extend tool.setuptools.package-data to ship
  canary/_obfuscate_helper.js, canary/fingerprint_payload.js, and
  canary/package.json with the wheel.
* decnet/cli/canary.py - new "decnet canary-install-toolchain"
  subcommand. Resolves decnet.canary.__file__'s dir, runs
  npm install --omit=dev there, exits non-zero with a clear message
  if npm is missing or install fails. Idempotent - safe to call
  every API service start.
* deploy/decnet-api.service.j2 - non-fatal ExecStartPre that calls
  the new subcommand. Leading '-' so a missing Node toolchain only
  degrades fingerprint canaries (loud at mint time) without keeping
  the API from booting.
* tests/canary/test_cli.py - registration smoke test, missing-npm
  exit path, and a mocked-subprocess test asserting the right argv
  and cwd land on npm.

Realism cultivator already has a broad except Exception around
cultivate() in scheduler.py:195-211, so a missing toolchain on a
host running the realism tick degrades to an inert noise file with
no extra plumbing.
2026-04-29 16:53:27 -04:00
907ade9142 feat(realism): wire fingerprint_html/svg through taxonomy + UI
The two new fingerprint canary generators existed at the API level
since f64e78f but weren't visible to the realism engine or the
operator-facing dashboard. Threads them through every place that
enumerates canary content classes.

Backend:
* realism/taxonomy.py - two new ContentClass members
  (CANARY_FINGERPRINT_HTML, CANARY_FINGERPRINT_SVG); enum is
  wire-visible (synthetic_files.content_class column + bus discrim)
  so we add at the bottom, never reorder.
* canary/cultivator.py - class-to-generator dispatch, kind mapping
  (both http), and default placement paths
  (~/Documents/asset_directory.html and network_topology.svg).
* realism/naming.py + bodies.py - _name_canary / _body_canary entries.
* realism/planner.py - added to _DEFAULT_CANARY_CLASS_WEIGHTS and
  the _CANARY_CLASSES classification set.

Frontend:
* decnet_web/src/realism/labels.ts - display labels.
* decnet_web/src/components/RealismConfig/RealismConfig.tsx - default
  canary weight rows so operators see them in the realism config UI.
* decnet_web/src/components/SyntheticFiles/SyntheticFiles.tsx - added
  to the CONTENT_CLASSES allow-list so filter dropdowns show them.

Also: re-applied the nosec B404/B603 markers on canary/obfuscator.py;
the first commit's pre-commit autoformatter stripped them.

Tests: extended tests/realism/test_taxonomy.py's stability assertion
to include the two new values. Full canary + realism suites pass
(362 / 2 skipped).
2026-04-29 16:44:03 -04:00
de6d5cd1a8 fix(canary): include fingerprint_* in KNOWN_GENERATORS stability test 2026-04-29 16:26:09 -04:00
dd807bc55e feat(canary): worker decodes ?d=/?o=/?s=&i=&n=&d= fingerprint params
The fingerprint payload beacons fingerprint data as base64url JSON in
GET query params: ?o=1 for the bare-open beacon, ?d=<blob> for a
single-shot dump, or ?s/i/n/d=<chunk> for chunked dumps. Until now
those params were buried inside request_path; consumers had to parse
the URL themselves.

Worker now extracts them in _extract_fingerprint and merges into
raw_headers under reserved _fp* keys:

* _fp_open       — bare-open marker
* _fp            — decoded fingerprint dict (single-shot path)
* _fp_sid/idx/total/chunk — chunked metadata + raw base64 (reassembly
  is a downstream concern, not the worker's job)
* _fp_decode_error / _fp_oversize — failure markers for trash dumps

Per-chunk size capped at 8KB so an attacker spamming /c/<known_slug>
can't inflate trigger rows indefinitely. Decode failures degrade
gracefully — the trigger row still records the hit, just with a
_fp_decode_error flag instead of structured fingerprint data.

Tests cover the single-shot decode, bare-open flag, chunked metadata,
malformed input, and oversize drop paths.
2026-04-29 16:25:17 -04:00
f64e78f78c feat(canary): fingerprint_html + fingerprint_svg generators
Two new synthesised-artifact generators that bake the obfuscated
fingerprint payload into plausible-looking decoy files:

* fingerprint_html — a mundane "Internal Asset Directory" page with a
  small table of fake hosts; the obfuscated payload is inlined at the
  bottom of <body>. Visible content (row pool slice, sync timestamp)
  also varies per mint via SHA-256-derived stable ints, so two
  extracted canaries don't diff to zero even on the rendered surface.
* fingerprint_svg — standalone SVG with an embedded <script> CDATA
  block. SVG <script> only fires for top-level loads / <object> /
  <iframe>; <img>-referenced renders are safely inert.

Both derive the mint UUID via uuid.uuid5 from the callback token, so
re-mints are byte-identical (preserving the generator determinism
contract) AND the same token produces the same mint UUID across HTML
and SVG variants — the worker can correlate beacons across artifact
shapes.

Wired into the factory + KNOWN_GENERATORS, default placement paths
under ~/Documents/asset_directory.html and ~/Documents/network_topology.svg
for both linux and windows personas. Tests cover determinism, per-token
divergence, structural validity (DOCTYPE/SVG headers), and that the
beacon URL stays inside the obfuscated string array (not in plaintext).
The two new entries skip in test_generators.py when Node toolchain is
absent so bare CI checkouts still pass.
2026-04-29 16:22:18 -04:00
12cd7ad9cb feat(canary): per-mint JS obfuscator wrapper + fingerprint payload
Adds the load-bearing primitives for obfuscated browser-fingerprinting
canaries. Step 3 (HTML/SVG generators) and step 4 (worker-side
fingerprint ingestion) build on top of these.

* decnet/canary/obfuscator.py - javascript-obfuscator wrapper. Seed
  and polymorphic config bits both derive from the callback token, so
  output is byte-identical for the same mint (preserving the generator
  determinism contract from base.py) and structurally distinct across
  mints.
* decnet/canary/fingerprint_payload.js - port of canary-self-test.html
  with the rendering UI stripped. Two placeholders (BEACON_URL,
  MINT_UUID) substituted before obfuscation. MVP beacon strategy:
  bare-open GET pixel first, then base64url-encoded fingerprint as
  query params on subsequent GETs (chunked above ~6KB) so the existing
  worker records hits before step-4 lands.
* decnet/canary/_obfuscate_helper.js - Node subprocess helper that
  reads code+options JSON from stdin and writes obfuscated JS to
  stdout. Vendored javascript-obfuscator under decnet/canary/.
* tests/canary/test_obfuscator.py - determinism, per-mint divergence,
  template substitution, Node syntax check, error path.
2026-04-29 16:16:37 -04:00