Files
DECNET/decnet/ttp/base.py
anti c3c5813211 feat(ttp): E.1.3+E.1.4 Tagger ABC and composite factory contract
Third and fourth TTP-tagging contract commits, plus a scoped subset
of the E.2.4 conformance tests covering the contract surface shipped
here (full hypothesis-fuzz suite still lands with E.2.4).

E.1.3 — decnet/ttp/base.py
- TaggerEvent NamedTuple: source_kind, source_id, attacker_uuid,
  identity_uuid, session_id, decky_id, opaque payload.
- Tagger(ABC) with abstract async tag(); class-level name and
  HANDLES: frozenset[str] (default empty so a misconfigured subclass
  is loudly idle, not loudly noisy).
- TolerantTagger(Tagger): concrete tag() wraps abstract _tag_impl()
  in try/except Exception (deliberately not BaseException — so
  KeyboardInterrupt / SystemExit / asyncio.CancelledError propagate
  and the worker can shut down cleanly). Swallowed exceptions log
  at WARNING with exc_info, never ERROR — absence is the steady
  state, not a bug. Subclasses override _tag_impl, never tag — the
  tolerance contract is enforced in the base class, not on trust.
- KNOWN_SOURCE_KINDS: Final[frozenset[str]] enumerating every
  source_kind a producer is allowed to emit. Closed-by-enumeration
  at the runtime layer; the composite tagger keys its WARNING/INFO
  bridge off this constant to surface the silent-drop trap from
  the design doc (lines 160–195).

E.1.4 — decnet/ttp/factory.py
- get_tagger() reads DECNET_TTP_TAGGER_TYPE (default 'composite');
  unknown values raise ValueError with the known-list. Mirrors
  decnet.intel.factory and decnet.clustering.factory.
- _KNOWN = ('composite',). Per-lifter classes (E.1.6) are children
  of the composite, not standalone tagger types.
- CompositeTagger(Tagger): pre-computes a dict[str, list[Tagger]]
  dispatch index from each lifter's HANDLES; fans events out
  concurrently with asyncio.gather and concatenates results.
  Empty lifters=[] is the legal contract-phase state — E.1.6
  wires the real lifters in.
- Unhandled-event observability: source_kind in KNOWN_SOURCE_KINDS
  but no lifter claims it -> WARNING once per kind per process
  (missed E.1.6 update). Unknown kind -> INFO once per kind per
  process (future-feature telemetry, by design). Per-process dedup
  via plain set; E.1.6 may swap in a proper rate-limiter once
  production traffic shapes are known.

Tests — tests/ttp/test_base.py, tests/ttp/test_factory.py
- Tagger / TolerantTagger abstractness, missing-tag-impl rejection,
  WARNING-not-ERROR log level, propagation of KeyboardInterrupt /
  SystemExit / asyncio.CancelledError.
- Factory env-var routing, unknown-name ValueError, dispatch-index
  correctness, only-claiming-lifter invocation, WARNING-once for
  known-but-unclaimed kinds, INFO-once for unknown kinds, result
  concatenation across lifters.

Mypy clean under .311/bin/mypy --ignore-missing-imports.
2026-05-01 06:20:10 -04:00

139 lines
4.8 KiB
Python
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
"""Tagger ABC — input shape, base class, tolerant mixin.
Contract step E.1.3 of ``development/TTP_TAGGING.md``. Defines the type
surface every lifter (E.1.6), the rule engine (E.1.5), the composite
tagger (E.1.4) and the worker (E.1.7) compile against. No behavior
beyond the tolerant-wrapper boundary lives here.
The design doc's "schema is forward-compat, code is not" trap (lines
160195) is mitigated *here*: :data:`KNOWN_SOURCE_KINDS` enumerates
every ``source_kind`` a producer is allowed to emit. Adding a new
producer means adding its kind to this set in the *same commit* that
ships the producer; the composite tagger's WARNING/INFO bridge in
:mod:`decnet.ttp.factory` keys off this constant to surface silent
drops.
"""
from __future__ import annotations
import logging
from abc import ABC, abstractmethod
from typing import Any, Final, NamedTuple
from decnet.web.db.models.ttp import TTPTag
_log = logging.getLogger(__name__)
# Every ``source_kind`` string a DECNET producer is allowed to emit.
# Closed-by-enumeration at the runtime layer even though the storage
# column is open. Producers MUST add their kind here in the same
# commit that starts emitting — see the design doc lines 160195 for
# the operational contract and the rationale.
KNOWN_SOURCE_KINDS: Final[frozenset[str]] = frozenset({
"command",
"intel",
"email",
"canary_fingerprint",
"identity",
"credential",
"auth_attempt",
"payload",
"session",
"http_request",
})
class TaggerEvent(NamedTuple):
"""Input shape for every tagger.
NamedTuple (not dataclass) so instances are hashable — downstream
dedup paths can put them in sets without a custom ``__hash__``.
``payload`` is opaque on purpose: each ``source_kind`` carries a
different shape, and the per-lifter contract owns the parse.
"""
source_kind: str
source_id: str
attacker_uuid: str | None
identity_uuid: str | None
session_id: str | None
decky_id: str | None
payload: dict[str, Any]
class Tagger(ABC):
"""Abstract tagger.
Every concrete tagger sets :attr:`name` and :attr:`HANDLES` at
class level. The composite tagger reads ``HANDLES`` to build its
dispatch index — a subclass that forgets to override it gets the
empty default and is therefore never invoked, which surfaces as a
test failure rather than a silent fan-out.
"""
#: Short tag used in logs and the ``DECNET_TTP_TAGGER_TYPE`` env
#: var. Subclasses override.
name: str = ""
#: ``source_kind`` strings this tagger consumes. Empty by default
#: so a misconfigured subclass is loudly idle, not loudly noisy.
HANDLES: frozenset[str] = frozenset()
@abstractmethod
async def tag(self, event: TaggerEvent) -> list[TTPTag]:
"""Produce zero or more tags for ``event``.
Implementations of :class:`Tagger` directly take responsibility
for their own error handling. Lifters that consume
sibling-worker output inherit from :class:`TolerantTagger`
instead, which enforces the "absence is not an error" contract
in the base class rather than on trust.
"""
class TolerantTagger(Tagger):
"""Tagger mixin that converts uncaught exceptions to ``[]``.
Every per-source lifter inherits from this. The rationale is
architectural, not stylistic: TTP tagging consumes outputs from
sibling workers (intel, behavioral, identity, …) that may not
have run yet, may have failed, or may simply have nothing to say
about a given event. "Absence" is the steady state, not the
exception, so a lifter blowing up on a missing join must not
cascade into a worker crash.
Subclasses override :meth:`_tag_impl`, never :meth:`tag` — the
tolerance contract is *enforced in the base class*, not on trust.
"""
async def tag(self, event: TaggerEvent) -> list[TTPTag]:
try:
return await self._tag_impl(event)
except Exception:
# ``Exception`` deliberately, not ``BaseException``:
# ``KeyboardInterrupt`` / ``SystemExit`` /
# ``asyncio.CancelledError`` propagate so the worker can
# shut down cleanly. E.2.4 conformance asserts this.
# WARNING, not ERROR: a sibling-worker absence is normal
# operation, not a bug. ERROR would page someone for the
# steady state.
_log.warning(
"tagger %r swallowed exception on source_kind=%r",
self.name,
event.source_kind,
exc_info=True,
)
return []
@abstractmethod
async def _tag_impl(self, event: TaggerEvent) -> list[TTPTag]:
"""Real tagging logic — subclasses override this, not :meth:`tag`."""
__all__ = [
"KNOWN_SOURCE_KINDS",
"TaggerEvent",
"Tagger",
"TolerantTagger",
]