v0 Phase 1 of ATTRIBUTION-ENGINE.md:
* AttributionStateRow SQLModel keyed on (identity_uuid, primitive)
per ANTI direction — re-keying state rows when the v1 clusterer
merges attackers is the migration debt v0 should not bake in.
ATTRIBUTION-ENGINE.md updated with the deviation note.
* AttributionMixin: ensure_stub_identity_for_attacker, idempotent
upsert_attribution_state, get_attribution_state[_for_identity],
list_multi_actor_identities (the Phase 5 correlator's read).
* attribution.profile.{state_changed,multi_actor_suspected} bus
topics + builder; wiki Service-Bus.md updated separately.
* attribution_worker.py: subscribes to attacker.observation.>,
ensures stub identity per event, logs and continues. No merger,
no state writes, no derived events — Phase 4 wires those.
* attribution/{aggregate.py,_thresholds.py} skeletons: Phase 2
fills _aggregate_categorical, Phase 3 adds numeric+hash+dispatcher.
Two-half deliverable per BEHAVE-INTEGRATION.md §587-594:
* scripts/behave_shell/replay_calibration.py — Python helper that
drives the production handler against one asciinema shard, mints
a temp SQLite repo + an Attacker per session, captures bus
emissions in-process. Exits non-zero on zero-observation sessions.
* scripts/behave_shell/smoke.sh — bash entry that replays all five
2026-05-02 calibration shards (HUMAN / YOU-sim / LW-sim /
CLAUDE-FF / CLAUDE-CL). Auto-activates .311 venv, forces
DECNET_DB_TYPE=sqlite, prints per-class summary. Suitable for CI.
* scripts/behave_shell/README.md — runbook covering both halves.
Pins the manual live-decky procedure (one SSH session per class
against a deployed smoke-decky, expected dominant primitives table,
SQL verification query, AttackerDetail panel check, pass criteria).
* BEHAVE-INTEGRATION.md — Phase 6 completion log appended with
current corpus results table (15 sessions, 424 observations across
the five classes) and a note that the v0 tag (drop -pre) is gated
on the manual live-decky round-trip and lands as a separate
commit.
Live-decky run is intentionally NOT scripted — the integration doc
calls for manual SSH sessions per class so an operator confirms the
bus / collector / disk-reach plumbing under real PTY conditions.
Run the five-class calibration grid (HUMAN / YOU-sim / LW-sim /
CLAUDE-FF / CLAUDE-CL) against the 2026-05-02 shards.
* Hard gate green for 27 primitives across all 5 shards.
* environmental.keyboard_layout moved from hard gate to
PHASE_F_CONDITIONAL_PRIMITIVES — short SSH-recon corpus maxes at
~90 typed letters per session, well below the LAYOUT_MIN_TYPED_LETTERS
(200) floor. The 200-floor stays per the per-phase "v0 ships when
honest" rule; longer-text corpora will surface the layout signal.
* Three primitives never fire on the 2026-05-02 corpus, all already
conditional and all expected:
- cognitive.error_resilience.frustration_typing
- environmental.locale
- environmental.keyboard_layout
No D / F / G threshold re-tunes needed; only the keyboard_layout
binding-set move. Phase H step log appended to BEHAVE-EXTRACTOR.md
with per-class observation counts.
Widen calibration binding from PHASE_ABCDEF_PRIMITIVES (25) to
PHASE_ABCDEFG_PRIMITIVES (28 hard). Three Phase G primitives that
emit on any session-with-commands ride the hard gate:
* operational.opsec_discipline
* operational.cleanup_behavior
* emotional_valence.stress_response
The remaining five Phase G primitives ride a new
PHASE_G_CONDITIONAL_PRIMITIVES because their sample-size floors make
them legitimately absent from short shards:
* operational.objective (≥ 3 classified commands)
* operational.multi_actor_indicators (≥ 8 commands)
* emotional_valence.arousal (typing bursts)
* emotional_valence.valence (≥ 80 typed letters)
* emotional_valence.frustration_venting (≥ 30 typed letters)
Backwards-compat alias PHASE_ABCDEF_PRIMITIVES kept. Phase G
completion log + checkbox flips in BEHAVE-EXTRACTOR.md.
Tier-A corpus delta: all 37 Tier-A primitives now emit. Phase H
(full-corpus lockdown + v0 release) is next.
Widens the binding calibration set from PHASE_ABCDE_PRIMITIVES (20)
to PHASE_ABCDEF_PRIMITIVES (25). The five new entries:
* environmental.shell_type (per-shard hard gate)
* environmental.terminal_multiplexer (per-shard hard gate)
* environmental.keyboard_layout (per-shard hard gate; PII boundary
lifted by ANTI; emits all 4 registry values)
* environmental.numpad_usage (per-shard hard gate)
* temporal.lifecycle_markers.exit_behavior (resolution of the E.4
hold; uses Command.followed_by_prompt from F.0)
environmental.locale joins a new PHASE_F_CONDITIONAL_PRIMITIVES set
(only fires on shards with an env / locale dump in the output).
Phase F completion log appended to BEHAVE-EXTRACTOR.md. The original
F.0 row hinted at D.0 subsumption; reversed in the log — D.0 is
enriched, not subsumed (regex catches errors when PS1 is suppressed).
Tier-A corpus delta: 25 of 37 primitives now emit. Phase G is next.
F.0's row in BEHAVE-EXTRACTOR.md was forward-only — readers landing
on Phase F couldn't tell that F.0 also has a backlog (E.4 held, D.0
subsumption). Add a 'Carry-overs F.0 must unblock' section to the
Phase F prelude and a back-reference on the F.0 checkbox in the
implementation order checklist.
Widens the binding calibration set from PHASE_ABCD_PRIMITIVES (17) to
PHASE_ABCDE_PRIMITIVES (20). The three shipped Phase E primitives
(session_duration, escalation_pattern, landing_ritual) join the
per-shard hard gate.
E.4 (temporal.lifecycle_markers.exit_behavior) is held at ANTI's
direction pending Phase F.0's prompt parser — abrupt-vs-cleanup
needs exit-code visibility to be honest, and first-token membership
alone over-fires on benign rm / clear mid-session. E.4 picks up at
the tail of Phase F.
Phase E completion log appended to BEHAVE-EXTRACTOR.md; E.1-E.3
checkboxes flipped, E.4 left unchecked with a held note.
Widens the binding calibration set from PHASE_ABC_PRIMITIVES (13) to
PHASE_ABCD_PRIMITIVES (17). The four unconditional Phase D primitives
(cognitive_load, exploration_style, planning_depth, tool_vocabulary)
join the per-shard hard gate. The three error_resilience.* primitives
are conditional on at least one errored command in the shard and
tracked in PHASE_D_CONDITIONAL_PRIMITIVES — excluded from the
per-shard required-emission set, included in the cross-class
discrimination check.
cognitive_load empirical re-tune deferred to the next
BEHAVE_CALIBRATION_DIR run; v0.1 thresholds ship.
Phase D completion log appended to BEHAVE-EXTRACTOR.md; Phase D
checkboxes flipped to [x].
Tick the four Phase B checkboxes (B.1-B.4) and append a Phase B
completion log inline (per the "append phase logs to design docs"
memory rule). Captures per-primitive confidence ranges, source
signals, and the PII-discipline regression that all four
primitives uphold.
Phase A + Phase B = 10 primitives emitting on every shard;
PHASE_AB_PRIMITIVES is binding for every subsequent phase.
Phase C (motor.shell_mastery.*) lands next.
BEHAVE-EXTRACTOR.md Phase A Step 10. Closes the discriminative
floor: six primitives emit, the five-class calibration grid is the
binding regression test for every subsequent phase.
* Phase A checklist boxes (Steps 0-10) ticked in
development/BEHAVE-EXTRACTOR.md.
* Phase A completion log appended inline to the design doc per
the "append phase logs to design docs" memory rule — captures
per-primitive confidence ranges and the 2026-05-02 empirical
anchors that drove threshold calibration.
* Hard gate: tests/profiler/behave_shell/test_calibration_grid.py
parametrised over five class shards, all green; skips cleanly
on BEHAVE_CALIBRATION_DIR unset.
Phases B-G expand horizontally across the registry. Phase H is
the full-corpus lockdown + v0 release. Worker
(BEHAVE-INTEGRATION.md Phase 4) is unblocked at this milestone —
it can wire per-session production against the Phase A engine
without waiting for the rest of the Tier-A corpus.
Three sibling design docs plus DEBT.md updates that supersede the
stale DEBT-036 with a BEHAVE-aligned plan.
development/BEHAVE-INTEGRATION.md — five-phase rollout: storage
(observations table mirroring the BEHAVE Observation envelope plus
one DECNET-side denorm; UniqueConstraint(evidence_ref, primitive)
enforcing idempotency); engine (in decnet/profiler/behave_shell/
sublibrary, no new daemon, not in BEHAVE — DECNET is the engine);
BEHAVE pin; worker wire; UI panel + per-attacker SSE route; live
smoke. Bus payload merges id/ts/v back in to preserve sensor
identifiers across the bus envelope.
development/BEHAVE-EXTRACTOR.md — engine route in eight phases
(A–H). Phase A locks the 6-primitive calibration grid; Phases B–G
expand horizontally; Phase H is the full Tier-A corpus + v0
release. v0 ships every shell-extractable primitive (37 of them);
Tier B is cross-session and lives in the attribution engine; Tier
C is network-domain (toolchain.*) and lives elsewhere.
development/ATTRIBUTION-ENGINE.md — sublibrary inside
decnet/correlation/ that consumes attacker.observation.* events
and emits attribution.profile.* derived state. Five-state machine
(unknown / stable / drifting / conflicted / multi_actor) with per-
ValueKind merge functions. v0 closes DEBT-051; v1 adds the real
clusterer; v2 federation gossip. The bright line forbidding
attribution to natural persons is lifted directly from BEHAVE's
envelope docstring.
development/DEBT.md — DEBT-036 marked STALE; DEBT-050 and
DEBT-051 entries added; summary table + open list updated.
New MalHashProvider sibling ABC (decnet/intel/base.py) since SHA-256
is a different keyspace from IntelProvider's IPs. MalwareBazaarProvider
mirrors FeodoProvider's bulk-feed shape: 24h refresh via _ensure_fresh
/ _refresh, in-memory set[str] of hex-lowercased hashes, set-membership
lookup. Auth-keyed via DECNET_MALWAREBAZAAR_AUTH_KEY; absent key
silent-no-ops the lane (single warning, no HTTP traffic).
Per-hash observations persist to a new observed_attachments table.
DECNET is a honeypot platform — every attachment hash an attacker
delivers is intel, regardless of whether anyone classified it. Verdict
is sticky: True never downgrades to False/None on subsequent
observations. Out of scope: API surface, federation export, retention.
Ingester _publish_email_received calls the provider for each attachment
sha256, sets mal_hash_match on the bus payload (omitted entirely when
the message had no attachments — keeps R0046's `is True` predicate
silent on hash-less mail, matching pre-paydown behavior), and upserts
the row regardless of provider availability.
New GET /api/v1/orchestrator/events/stats?since=1h&success=false&kind=...
backed by repo.count_orchestrator_failures(since_ts, kind), which
counts failed rows across both orchestrator_events and
orchestrator_emails since the cutoff.
Window parser accepts ^\d+[smhd]$, capped at 7d. Today only
success=false is accepted on this surface so the endpoint isn't
accidentally repurposed before the next consumer is properly
designed.
Orchestrator.tsx polls the endpoint on mount + every 30 s and
renders the authoritative DB-derived count instead of deriving from
the in-memory SSE buffer + one paginated page (which silently
excluded failures older than the local window).
Wire vitest 4 + jsdom + @testing-library/{react,jest-dom,user-event}
+ @vitest/coverage-v8 through vite.config.ts (defineConfig from
vitest/config). src/test/setup.ts registers jest-dom matchers and
RTL cleanup. tsconfig.app.json picks up vitest/globals types.
Seed suite Orchestrator.test.tsx covers the three regressions
called out in DEBT-043: empty-state render, kind-filter toggling
triggers a scoped refetch, mocked stream callback prepends a row.
When the prober observes a NEW hash for an
(attacker_uuid, port, probe_type) triple it has seen before — VPS
rotation, SSH server rebuild, TLS cert swap — emit a derived
attacker.fingerprint_rotated event carrying both old and new hash.
Detection is a small library (decnet.correlation.fingerprint_rotation)
called inline from the prober at each of the three emit sites
(JARM/HASSH/TCPFP). No new daemon. New AttackerFingerprintState table
holds per-triple last-hash state; Attacker.rotation_count and
Attacker.last_rotation_at are stamped on every diff. Library is sync,
fully unit-tested via injected publish_fn / syslog_fn callbacks.
All base images (debian:bookworm-slim, ubuntu:22.04, ubuntu:20.04,
rockylinux:9-minimal, centos:7, alpine:3.19, fedora:39,
kalilinux/kali-rolling, archlinux:latest, honeynet/conpot:latest)
now carry their resolved sha256 digest so 'docker pull' is
deterministic. :tag retained for human readability; @sha256 is what
Docker actually resolves. Refresh procedure documented at the top of
decnet/distros.py.
When service_cfg["email_seed"] is absent, compose_fragment now falls
back to $PROJROOT/bait/ if that directory exists on the host. Lets
operators drop a deployment-wide bait corpus into one place without
threading email_seed through every decky's config. Missing dir keeps
old no-op behavior.
IMAP_EMAIL_SEED / POP3_EMAIL_SEED accept a directory (rglob *.eml +
*.json) or a single .json/.eml. Loaded entries CONCATENATE with the
hardcoded _BAIT_EMAILS — additive to the realism-engine emailgen
output rather than replacing it. JSON dicts require from_addr /
to_addr / subject / body; bare bodies are wrapped into RFC 5322 on
load. compose_fragment reads service_cfg["email_seed"] and bind-mounts
the host path read-only at /var/spool/decnet-emails/seed.
DEBT-035 (artifacts written as the container uid, not the API's) is
resolved by the two preceding commits:
* 39a298f6 — persists DECNET-service api-user/api-group as names in
decnet.ini for any future composer / worker that wants to resolve
the local uid via pwd.getpwnam.
* b2733216 — creates /var/lib/decnet/artifacts at init time with mode
0o2775 (setgid + group-write) owned by the DECNET-service
user:group.
The setgid bit is the load-bearing fix: Linux mkdir(2) propagates a
parent's group AND its setgid bit to every new subdirectory. Docker
auto-creates the per-decoy / per-service subtree as bind-mounts fire,
so those subdirs come up with group=decnet and setgid set; container
file writes (default umask 0o022 → mode 0o644) inherit the decnet
group; the API process and the local TTP worker (both running as the
DECNET-service user, primary group decnet) read via group-read.
The original recommendation of compose `user:` injection turned out
infeasible for SSH and Telnet — PAM's setuid(2) during login
fundamentally cannot run from a non-root container. Setgid covers
both root-internal and unprivileged-internal templates uniformly
without requiring per-template carve-outs.
DEBT-047 (R0047 BEC disk-reach) was gated on DEBT-035 for filesystem
access. That blocker is lifted — `decnet ttp` running on agents as
the local DECNET-service user can now read .eml files written by
the SMTP decoy. The remaining DEBT-047 work is the master-only gate
flip in decnet/cli/gating.py and the EmailLifter disk-reach helper
itself (factor _resolve_artifact_path out of the artifacts API
endpoint into a shared module).
Soft-fail paths in api_get_transcript.py and api_get_artifact.py
stay as defence-in-depth — option 2 should make them never fire on
a healthy install but a misconfigured deploy must not 500 the API.
A previous agent (and several of my own commits) wrote to a top-level
DEBT.md without seeing the existing development/DEBT.md — the
canonical register since DEBT-001. Resulted in two parallel files,
inconsistent numbering schemes, and references that resolved to the
wrong place.
Migrate the six entries that landed in the rogue file into the
canonical register as DEBT-044 through DEBT-049, preserving their
status (resolved / partial / open) and cross-references. The
TTP_TAGGING.md references to "DEBT.md" already resolve to
development/DEBT.md by virtue of being in the same directory; only
the comment in decnet/ttp/impl/intel_lifter.py needed disambiguation
to "development/DEBT.md DEBT-048".
* DEBT-044 — `attacker.email.received` producer wiring (✅ RESOLVED 2026-05-02)
* DEBT-045 — EmailLifter heavyweight feature extraction (PARTIAL PAID 2026-05-02)
* DEBT-046 — EmailLifter mal-hash feed integration (open)
* DEBT-047 — EmailLifter R0047 BEC unblock (open, gated on DEBT-035)
* DEBT-048 — TTP intel provider mapping review (recurring quarterly)
* DEBT-049 — TTP Sigma adapter — post-v1 (open)
Summary table extended; "Remaining open" line updated; root file
removed. The DEBT-047 entry now explicitly cross-references DEBT-035
as the gating dependency for the R0047 BEC unblock.
Mark the EmailLifter heavyweight follow-up as PARTIAL PAID — R0042 /
R0046 (macro / password / smuggling lanes) / R0048 fire end-to-end
after commits 291b78c1 (decky extractors) and the ingester producer
projection that follows.
Two narrower DEBT entries replace the lanes that remain gated:
* "EmailLifter mal-hash feed integration" — R0046's mal_hash_match
lane needs a curated bad-hash feed (MalwareBazaar SHA-256 dump as
the v0 candidate, mirroring the FeodoProvider bulk-feed pattern at
decnet/intel/feodo.py). Feed integration, not extraction. Lifter
predicate already reads `payload.get("mal_hash_match")` — silent
today only because the field is absent.
* "EmailLifter R0047 BEC — unblock when artifact disk-reach lands"
cross-references the agent UID/GID DEBT entry that blocks
`decnet ttp` from reading artifacts written by deckies on the
same host. Disk-reach is the intended solution; raw body_text on
the bus is rejected because the bus transport is abstracted (the
UNIX-socket implementation may swap to networked at any time, and
privacy decisions must hold regardless of transport).
Append to TTP_TAGGING.md §"Producer wiring": the email.received
producer pointer (was "none — DEBT"), the full per-message payload
shape with the new heavyweight fields, and an explanatory block on
why the bus is body-text-free + how R0047 / R0048 each handle their
body dependency (R0048 via the precomputed scalar; R0047 deferred).
Appendix A.10 corrected to match the post-2026-05-02-audit reality:
AbuseIPDB cat 7/13/16/17 land on their canonical AbuseIPDB names
(Phishing / VPN IP / SQL Injection / Spoofing); cats 4 and 10 carry
explicit "drop" annotations so the next reviewer sees the intent
rather than guessing. ThreatFox table re-keys on `threat_type` (the
canonical taxonomy field) and adds the `payload` and `cc_skimming`
rows. GreyNoise table promotes bare-malicious to a half-multiplier
emission of T1071.
§"Hard parts §9 Intel provider drift" replaces the prose handwave
with a runnable check: provider URLs, the ThreatFox curl invocation
that needs DECNET_THREATFOX_API_KEY, the rule_version + emits +
attack_catalog co-evolution rules, and the full chain of files to
exercise. Adds a "Ship-time audit log" subsection so future quarterly
runs have a known-good baseline to diff against.
DEBT.md item #1 records LAST_REVIEWED: 2026-05-02 / NEXT_REVIEW:
2026-08-02 and points at §9 for the runbook. DEBT.md item #3 (the
attacker.email.received producer) flags its gating premise as
potentially stale — ANTI noted SMTP honeypots already persist
received messages, contradicting the "no source row" claim that
deferred the wiring.
Add a "Producer wiring" subsection under TTP_TAGGING.md §"Bus
topics" mapping every topic the TTP worker subscribes to onto the
file:line that publishes it. Calls out the gap (`email.received`
has no producer today) and the new `attacker.session.ended`
payload shape from the collector aggregator.
Also lists the four producer regression tests added in this series
so a future contributor sees the safety net before staring at the
silent rule engine.
DEBT.md gets the `attacker.email.received` follow-up entry — wire
the producer when SMTP-receive persistence lands, since today the
honeypot relay path doesn't store received emails anywhere a
publisher could read from.
Closes the CDD design phase. Records:
- §E.1 contract inventory (every file exists, compileall clean).
- Targeted pytest pass: 604 passed, 1 skipped, 10 xfailed
(all xfails are `xfail(strict=True)` with reason= pointing to the
impl step that flips them; carry-overs, not flakes).
- Strict mypy over decnet/ttp + decnet/cli/ttp.py +
decnet/web/router/ttp + decnet/web/db/sqlmodel_repo/ttp.py: clean.
- Stranger-readability spot check on tests/ttp/: no doc bugs.
Notes the three pre-E.4 wiring fixes (E.3.18a/b/c) and the E.4
backfill CLI / DEBT entries that landed in this series.
Wires decnet-ttp as a first-class worker:
* `decnet ttp` CLI command (master-only via MASTER_ONLY_COMMANDS)
* deploy/decnet-ttp.service.j2 systemd unit (After= identity / intel
/ reuse-correlator workers; ProtectHome=read-only since
FilesystemRuleStore only reads ./rules/ttp/)
* deploy/decnet.target Wants= chain extended with decnet-ttp.service
* `ttp` was already in web/worker_registry.KNOWN_WORKERS
tests/api/test_schemathesis_ttp.py: TTP-routes-only schemathesis
suite, filtered via the OpenAPI tags=["TTP Tagging"] annotation
shared by the eight TTP routes. Reuses the live uvicorn subprocess
the wider test_schemathesis spawns; max_examples=400 keeps the
focused gate fast for E.3.13–E.3.16 iteration.
wiki-checkout/Service-Bus.md committed in its own repo: ttp.tagged
and ttp.rule.fired.<id> flipped from "reserved (TTP worker)" to
"decnet.ttp.worker" now that the worker publishes them.
TTPsObservedSection.tsx: shared analyst-facing rollup. scope=
identity drives /ttp/by-identity/{uuid} (primary, with Navigator
export download); scope=attacker drives /ttp/by-attacker/{uuid}
(per-IP slice). Tactic → technique tree in fixed UKC-aligned order,
counts and confidence-weighted bars. Literal "NO TECHNIQUES
OBSERVED YET" empty state per TTP_TAGGING.md §"UI surface — Empty
state": no spinner, no fallback list.
RuleStateControls.tsx: admin-only rule operational state panel
backed by POST/DELETE /ttp/rules/{rule_id}/state. Server-gated by
require_admin AND client-gated on /config?.role so a non-admin
never sees the controls (per feedback_serverside_ui.md the client
gate is UX, not security — the server returns 403 either way).
Wired into Config.tsx as a new "TTP RULES" admin tab.
Wired TTPsObservedSection into IdentityDetail (above fingerprints)
and AttackerDetail (above TIMELINE). DeckyFleet/PersonaGeneration
vocabulary throughout (logs-section / section-header / btn /
matrix-text / dim-chip).
tsc --noEmit and vite build clean.
The dev-server browser smoke is deferred per the "can't reliably
exercise UI from this harness" reality — typecheck + build is the
correctness gate, not feature verification.
Add BaseRepository.list_ttp_decky_phases(identity_uuid) returning
per-decky tag observations as (decky_id, tactic, created_at_ts) rows
ordered by creation time. Rewrite from_identity_row() to project
tactic → UKCPhase via tactic_to_ukc_phase and populate the four
phase-handoff maps (first/last_phase_per_decky,
first/last_seen_per_decky) so combined_campaign_weight finally lights
up on real DB rows — not just synthetic fixtures.
ConnectedComponentsCampaignClusterer.tick() pulls each active
identity's per-decky phase observations before projecting features.
Repo failures are non-fatal: a partial repo falls back to the empty
phase-handoff signal (legacy behavior) so the worker stays up.
tests/clustering/test_ttp_phase_handoff.py pins the production-row
pair clearing CAMPAIGN_EDGE_THRESHOLD on a C2 → DISCOVERY hand-off —
the trip-wire that says the whole project paid off.
commands_by_phase_on_decky itself stays empty on the production path:
it is consumed only by the synthetic-fixture similarity surface, and
the phase-handoff edge does not use it. Synthetic fixtures still
populate it directly via from_synthetic_identity.
Inner loop drains a per-process asyncio.Queue populated by one pump
task per topic in _TOPICS, dispatches each event through
CompositeTagger, persists via repo.insert_tags(), and publishes
ttp.tagged + per-technique ttp.rule.fired.<id> only when the insert
returned a non-zero rowcount.
CompositeTagger seeded with all six lifters (Behavioral, Intel,
CanaryFingerprint, Email, Identity, Credential).
Loop-prevention invariant from TTP_TAGGING.md §"Bus topics" enforced:
N replays of the same upstream event publish exactly one ttp.tagged
event. test_worker_bus covers both the direct invocation path and
the idempotency replay path.
Intel catch-up via attacker.session.ended is intentionally deferred
to E.3.14b — needs a session→intel join the repo doesn't expose yet.
IdentityLifter owns lifter:identity_* — currently R0003 (password
spraying). CredentialLifter owns lifter:credential_* — R0001 generic
auth brute, R0002 password guessing, R0004 credential reuse, R0005
valid-account use, R0006 default credentials.
YAMLs R0001/R0002/R0003/R0005/R0006 had their match.kind normalised
to fit the lifter prefix scheme — the design doc's promised "YAMLs
normalised in a separate refactor commit" lands here.
Identity-rollup tags null out attacker_uuid on emit so the worked-
example invariant holds (the tag belongs to the Identity, never to
one member IP).
Tests: test_identity_lifter.py + test_credential_lifter.py cover
each predicate's positive/negative path, state modulation
(disabled/clipped/expired), source-kind gating, and idempotent
replay. test_lifter_absence and test_lifters updated for the new
ctor signature.
Records the RuleIndex extraction prerequisite, the lifter:<owner>_
prefix routing convention, per-provider technique fan-out logic for
intel rules, the canary identity-merge guard rail, and the email PII
allowlist + R0042 simhash requirement.
5 YAMLs for the intel-verdict cohort per Appendix B / A.10:
AbuseIPDB category mapping, GreyNoise classification, Feodo
Tracker hit, ThreatFox IOC type, aggregate-malicious bump-only.
IntelLifter (E.3.10) consumes by rule_id and tolerates absence
silently (null provider column → no tag).
R0058 is the meta bump-only rule — emits a single confidence=0.0
sentinel so it validates and surfaces in the catalogue, but the
repository's sub-0.3 drop ensures no fresh tag persists if the
fanout fires accidentally. test_intel_rules.py pins that
zero-confidence invariant.
Marks E.3.8 done in development/TTP_TAGGING.md with the cohort-
split summary.
Implements the rule engine body left empty at contract phase: evaluate()
dispatches by source_kind through self._by_kind, runs the rule's match
spec against event.payload, and emits one TTPTag per emits entry.
watch_store() loads the initial corpus from RuleStore.load_compiled,
then drains subscribe_changes, applying definition changes via
single-statement dict assignment (atomic swap, GIL-atomic to readers)
and state changes via NamedTuple._replace on the existing CompiledRule.
Why: with the FS + DB stores in place (E.3.5/E.3.6), the engine is the
last piece of the rule plane. Lifters (E.3.9–E.3.13) consume the
engine; the worker bootstrap (E.3.14) wires watch_store into the
asyncio event loop. After this commit a CompositeTagger constructed
with a RuleEngine + a populated rules dir will produce real tags.
Notes:
- CompiledRule.emits extended to 4-tuple
(technique_id, sub_technique_id, tactic, confidence). Tactic + confidence
ride per-emit so a single rule can carry multiple precision targets
(the "one event maps to many techniques" property). Compile helpers in
both backends extract them from the YAML emits dict; missing tactic
or confidence is a deploy-time error.
- v0 match operator is "pattern" (regex). The field defaults per
source_kind (command_text / raw_url / subject / verdict / …) and is
overridable via match.field. Future ops (contains, equals, in_set)
extend _match_event without touching the engine surface.
- Confidence model: rules with state="clipped" + confidence_max set
cap the per-emit confidence downward; clipped is a soft suppress, not
a hard skip. Disabled rules are skipped wholly; expires_at past is
re-checked at evaluate as defense-in-depth (the store auto-reverts,
but a racing read between expiry and revert must not fire the rule).
- _span(name, **attrs) helper in engine + both stores short-circuits on
decnet.telemetry._ENABLED — matches the project's @traced /
wrap_repository zero-overhead-when-disabled pattern instead of relying
solely on the no-op tracer indirection.
- Late-bound tracer (telemetry.get_tracer called per-span, not at
module load) so test_tracing's monkeypatch reaches the production
code path.
xfails flipped: tests/ttp/test_rule_engine.py multi-emit fan-out +
rule_version-collision-via-engine; tests/ttp/test_multi_mapping.py
N×M engine fan-out + idempotent replay; tests/ttp/test_tracing.py
ttp.eval span hierarchy + ttp.rule.fire span attributes.
Tests: 214 passed, 19 xfailed (gated on E.3.8 lifters / rule pack /
worker bootstrap).
mypy: clean on prod code; pre-existing test-stub arg-type warnings
unchanged.
Implements the DB-backed rule store body left empty at contract phase:
load_compiled reads from ttp_rule + ttp_rule_state; get_state /
set_state hit ttp_rule_state with the same expires_at auto-revert and
bus-event semantics as the FS backend; subscribe_changes returns a
per-subscriber queue. State persists across process restarts — the
swarm property the FS backend deliberately doesn't have.
Also lands two swarm-mode helpers:
- sync_from_filesystem(fs_store) — master-side, subscribes to a
FilesystemRuleStore and projects each RuleChange onto a ttp_rule
upsert/delete.
- tail_db(poll_interval) — worker-side, watermark poll over
ttp_rule.updated_at; emits RuleChange("definition", ...) for each
row that moved.
Why: swarm mode needs rule definitions and operator state to
propagate across hosts. The filesystem backend (E.3.5) was the
single-host-dev variant; this one survives restart and serves N
workers from a shared DB.
Notes:
- DatabaseRuleStore() with no args lazy-inits an in-memory SQLite
repo so the conformance fixture works without test plumbing. In
production the worker bootstrap (E.3.14) passes an explicit repo.
- The conftest.py rule_store fixture became async (pytest_asyncio),
per-backend creates/initializes a SQLite repo for the DB run.
- Adds a `seed_rule(store, rule_id, yaml)` helper to bridge backend
semantics: drop a YAML file (FS) vs insert a ttp_rule row (DB).
Used by the parametrized load_compiled conformance test.
- Late-bound _tracer() in both backends (was module-level get_tracer
binding) so test_tracing's monkeypatch of decnet.telemetry.get_tracer
actually affects span output.
xfails flipped: tests/ttp/store/test_database.py set_state-writes-to-
ttp_rule_state + filesystem-to-DB sync; tests/ttp/store/test_conformance.py
DB-side load_compiled / set_state isolation / round-trip / per-rule
fan-out / expired-state revert / set_state failure / get_state default
(was xfail-only-on-DB); tests/ttp/test_tracing.py set_state span
hierarchy.
Tests: 208 passed, 25 xfailed (gated on E.3.7 + lifters).
mypy: clean on all touched files.
Implements the filesystem-backed rule store body left empty at contract
phase: YAML parse + Pydantic validation, asyncinotify watch over
./rules/ttp/, in-process state cache with auto-revert on expires_at,
and a subscribe_changes() async iterator yielding one RuleChange per
per-rule edit. Bus topic builders ttp_rule_reloaded / ttp_rule_state
ship alongside.
Why: the rule plane needed a store before the engine (E.3.7) could
consume RuleChange events and atomically swap compiled rules into its
dispatch index.
Notes:
- Linux-only by construction (asyncinotify wheel gated by sys_platform
marker; FilesystemRuleStore.__init__ raises on non-Linux).
- Filename allowlist is the FIRST check on every inotify event.
- Content-hash dedup so a single write firing IN_CREATE + IN_CLOSE_WRITE
produces exactly one RuleChange.
- All compile work serializes on a single asyncio.Lock.
- Subscribers register their queue eagerly so events fired between
subscribe_changes() and the first __anext__() are buffered.
xfails flipped: per-save-style + filter-ordering + atomic-swap in
test_filesystem.py; load_compiled / set_state isolation / round-trip /
per-rule fan-out / expired-state revert / set_state failure semantics
in test_conformance.py (FS side; DB side stays xfail until E.3.6);
malformed-YAML compile-time check in test_rule_engine.py.
Tests: 197 passed, 35 xfailed (gated on E.3.6 / E.3.7 / lifters).
mypy + bandit: clean on all touched files.
Wiki update for the per-rule reload + state-change topics lands in a
matching wiki-checkout/Service-Bus.md edit (separate repo).
Five GET rollup endpoints (techniques, by-identity, by-attacker,
by-campaign, by-session) and the Navigator export (fleet +
per-identity) now call into the TTPMixin methods. Rule catalogue
endpoint still returns [] — backed by the RuleStore which lands
at E.3.5/E.3.6.
Dialect-split: portable rollup queries on TTPMixin; bulk insert with
ON CONFLICT DO NOTHING / INSERT IGNORE in the per-dialect repos.
Confidence-floor (< 0.3) drop applied at mixin layer before the
dialect hook. BaseRepository now declares the six TTP methods abstract.
Tests in tests/web/db/test_ttp_repo.py flipped from pytest.fail stubs
to real dual-backend behavioral tests; tests/ttp/test_confidence.py
drop-below-floor xfail removed.
Each section gets a Status: ✅ done block summarising what's GREEN
today vs xfail-gated and noting any divergence from the doc's
original wording (E.2.9 lossy observable phases; E.2.13 db_backends
fixture landed alongside; E.2.14a Jaeger-skip + tracing-enabled
plumbing; E.2.14b NamedTuple AttributeError vs FrozenInstanceError).
Adds decnet/ttp/store/ subpackage:
- base.py: RuleState frozen dataclass, RuleChange NamedTuple, RuleStore ABC
- factory.py: get_rule_store() reading DECNET_TTP_RULE_STORE_TYPE
- impl/filesystem.py: FilesystemRuleStore with sys.platform=='linux'
fail-fast guard, allowlist filename regex, raw inotify mask bits
(lib import deferred to E.3 so contract phase compiles without the
asyncinotify dep installed)
- impl/database.py: DatabaseRuleStore stub (no platform guard)
TTPRule + TTPRuleState SQLModels were already shipped at E.1.1; this
commit closes the type-only TYPE_CHECKING forward-ref in
rule_engine.py via real runtime imports through the new package.
Empty NotImplementedError bodies; the SQL lands at E.3 implementation.
Mixin composed onto SQLModelRepository alongside the existing domain
mixins. Dialect-specific INSERT-OR-IGNORE syntax overrides land in
the per-backend subclasses at E.3 per the dual-DB-backend convention.
Mounts /api/v1/ttp/* with empty-list / empty-Navigator responses.
GET endpoints viewer-gated; POST/DELETE /rules/{rule_id}/state
admin-gated server-side. POST parses JSON manually so a malformed
body returns the documented 400 (per feedback_schemathesis_400).
Drops xfail-strict markers from E.2.8 tests now that the router is
mounted; 26 tests pass against the contract handlers.
