The ~30-signature hand-rolled p0f-lite table in decnet/sniffer/p0f.py
misses most real-world attackers (yesterday's SLOW SCAN being a
textbook case — 9 hours of events, 19 hits, os_guess = NULL). The
375-sig vendored p0f v2 DB was already there; this commit actually
calls it.
New resolution chain in sniffer_rollup:
1. Enabled OS-fingerprint providers (p0f-v2 default, via
DECNET_OSFP_PROVIDERS) tried in declared order. Provider with
highest-confidence match across all enabled sources wins.
2. Modal os_guess label from the sniffer's hand-rolled p0f.py.
Kept as fallback because v2's DB predates post-2006 kernels.
3. TTL bucket (linux / windows / embedded). Coarse but never wrong.
Wiring details:
- _match_via_osfp_providers: never raises — factory / provider
failures collapse to None and the chain falls through to the
old modal-label / TTL path. A corrupt .fp file or misconfigured
DECNET_OSFP_PROVIDERS must never wedge a profile rebuild.
- tcp_fp_context tracks whether the LATEST tcp_fp snapshot came
from a passive SYN ('syn' → p0f.fp) or an active prober probe
('synack' → p0fa.fp). Routes to the right sig list.
- initial-TTL normalisation via decnet.sniffer.p0f.initial_ttl.
Observation's TTL may be N hops below the OS's initial; v2
signatures match on the canonical bucket.
Soft-field semantics on Signature.score(): df and total_len are now
skip-checked when the observation is missing them. Sniffer doesn't
currently emit either SD field; a literal-constraint sig
shouldn't hard-reject a match solely because of upstream
incompleteness. Hard fields (window, ttl, options_sig, quirks)
still hard-reject on absent/mismatched input — those are the real
discriminators. Promote df / total_len back to hard the moment the
sniffer starts emitting them.
+2 integration tests on TestSnifferRollup, +2 soft-field tests on
test_signature. Full regression: 166 tests across tests/prober/osfp
+ tests/profiler all green.
- decnet/prober/osfp/p0f/provider.py: P0fV2Provider loads the four
vendored .fp files into per-context signature lists (syn / synack /
rst / stray) and matches via highest-specificity score across the
relevant list. Also auto-picks up p0f-decnet.fp if present (GPL-3.0
additions land there later, empty for now).
- decnet/prober/osfp/factory.py: get_provider / get_all_providers /
reset_cache, mirrors decnet/geoip/factory exactly. Env-dispatched
via DECNET_OSFP_PROVIDERS (default "p0f-v2"). Reserved names
"nmap-osdb" (pending Fyodor's grant) and "decnet-observed" (our
future curated DB) raise NotImplementedError — visible on the
factory surface so a typo doesn't silently fall through.
- decnet/prober/osfp/__init__.py now re-exports the public API so
callers use `from decnet.prober.osfp import get_provider` without
reaching into submodules (upholds the provider-subpackage rule).
15 new provider+factory tests covering:
- All four DB contexts load (262/61/46/6 sigs per inventory).
- Known-good Linux 2.6 SYN + Linux 2.2 SYN-ACK match end-to-end.
- Unknown observations / contexts return None, not raise.
- Factory memoises, env override honoured, unsupported names raise.
- Reserved names raise NotImplementedError (not silent None).
`sniffer_rollup` wiring lands in the next commit.
First code layer of the OS-fingerprinting work on top of yesterday's
vendored p0f v2 database. Three new modules, all pure (no I/O outside
of the parser's file read):
- decnet/prober/osfp/base.py — Provider protocol + OsMatch dataclass
matching the established Provider convention in decnet/geoip and
decnet/bus. Docstring spells out the never-raise invariant: malformed
input returns None, so a single bad event can't wedge a whole
attacker-profile rebuild.
- decnet/prober/osfp/p0f/signature.py — Signature dataclass + three
predicate helpers (WindowSpec / IntSpec / OptionToken) encoding the
p0f v2 DSL's wildcard / modulo / MSS-multiple / MTU-multiple
semantics. Scoring is our extension on top of upstream p0f's
first-match-wins policy: each signature carries a precomputed
specificity in [0, 1] so the factory can pick the most-specific
match when multiple signatures fire against one observation.
- decnet/prober/osfp/p0f/format.py — .fp line parser. Every shipped
field variant from the DSL spec at the top of p0f.fp is covered
(Snn / Tnn / %nnn / * for window; T0 vs T; -/@/* os-genre prefixes;
quirks as concatenated single-letter flags; '.' sentinels for
no-options / no-quirks). Malformed lines log a warning and skip
instead of aborting the whole file — 1 bad row must not cost the
other 374.
20 parser tests + 14 scoring tests. Full vendored-DB smoke tests
confirm all 375 signatures parse round-trip (262 SYN + 61 SYN-ACK +
46 RST + 6 stray) and every computed specificity lands in [0, 1].
Ships the p0f v2.0.8 signature database for passive + active OS
fingerprinting. 375 total signatures across four probe contexts:
- p0f.fp (262 sigs) — passive SYN fingerprints
- p0fa.fp ( 61 sigs) — SYN-ACK response, for active probes
- p0fr.fp ( 46 sigs) — RST response quirks
- p0fo.fp ( 6 sigs) — "stray" packet fingerprints
Replaces reliance on the 10-signature hand-rolled p0f-lite table in
decnet/sniffer/p0f.py for any match job the upstream DB covers.
Keeping the hand-rolled table as a fallback for modern kernels the
v2 DB pre-dates — v2 froze in 2006 so post-Win10 / post-Linux-3.x
kernels won't match against upstream directly. DECNET-authored
additions will go in a sibling p0f-decnet.fp under GPLv3 (not yet
committed; added as the ingester observes real honeypot traffic).
Provenance (full chain in data/README.md):
- Source: Debian snapshot of p0f_2.0.8.orig.tar.gz
- SHA1 matches Debian-recorded 7b4d5b2f24af4b5a299979134bc7f6d7b1eaf875
- Files byte-identical to upstream tarball (verified by hash)
License chain:
- Upstream: LGPL-2.1 (doc/COPYING preserved verbatim as
data/LICENSE.p0f-upstream, Michal Zalewski's copyright intact).
- DECNET uses the LGPL-2.1 §3 explicit permission to convert to any
version of the GPL. These files, as consumed in DECNET, are
effectively GPL-3.0. Chain documented in data/README.md so an
auditor sees the full reasoning.
- LGPL-2.1 → GPL-3.0 §3 conversion is a settled compat path; same
mechanism the kernel uses for LGPL userland glue and many other
projects apply daily.
Rejected path — nmap-os-db under NPSL — because NPSL adds
restrictions GPLv3 §7 prohibits us from accepting. An email is out
to Fyodor requesting an open-source-author exception grant, but we
don't block on it: p0f v2 is a genuine accuracy improvement in
its own right, and adding nmap-osdb later (if granted) plugs into
the same provider interface with zero refactor.
Directory layout mirrors the established provider-subpackage pattern
(see decnet/geoip/, decnet/bus/) per the feedback_provider_
subpackages memory: base + factory + impl/ subpackages, no flat
files. Parser + matcher + factory wiring land in the next commit
sequence.
Ships the backend half of Config → Workers:
* Worker registry aggregates `system.*.health` + `system.bus.health`
heartbeats into a last-seen dict; OK / STALE / UNKNOWN tiers drop
out of a 90s window (3× the 30s heartbeat interval).
* `GET /api/v1/workers` returns the snapshot plus `bus_connected`
(so the UI can explain "all UNKNOWN" when the bus socket is down)
and a per-row `installed` flag populated from
`systemctl list-unit-files decnet-*.service` (cached 30s).
* `POST /api/v1/workers/{name}/stop` publishes a stop intent on
`system.<name>.control`; workers listen via the shared control
listener in `bus/publish.py`.
* Heartbeat + control listener wired into collector / profiler /
sniffer / prober / mutator worker loops. API self-heartbeats too
so the panel always has one ground-truth row.
* Topic helper `system_control(name)` + tests covering builder
validation, control listener shutdown path, and the API surface
(auth gating, bus-connected field, unknown-name 404).
Adds `StartFailure` / `StartAllResponse` models in anticipation of
the upcoming start endpoints (DEBT-034).
Each successful JARM / HASSH / TCPfp probe fans out an
attacker.fingerprinted event; the probe family goes in event.type so a
single subscription covers all three. Payload carries the attacker IP,
port, and probe-specific hash — enough for the MazeNET live map to
render fingerprint info on observed attackers.
Lifts the thread-safe publisher helper out of the sniffer worker into
decnet/bus/publish.py so the prober (and every future worker with a
to_thread hot path) can reuse it without copy-pasting the
run_coroutine_threadsafe dance. Sniffer rewires onto the shared helper
in passing.
Adds ATTACKER_FINGERPRINTED as a new leaf — distinct from
ATTACKER_OBSERVED (correlator's first-sight signal) because an active
probe result is additional evidence about an already-observed attacker.
Note: the plan's decky.{id}.state realism-probe publish path is
deferred — the current prober fingerprints attackers, not decky
realism. Will revisit when realism probes exist.
Rename the container-side logging module decnet_logging → syslog_bridge
(canonical at templates/syslog_bridge.py, synced into each template by
the deployer). Drop the stale per-template copies; setuptools find was
picking them up anyway. Swap useradd/USER/chown "decnet" for "logrelay"
so no obvious token appears in the rendered container image.
Apply the same cloaking pattern to the telnet template that SSH got:
syslog pipe moves to /run/systemd/journal/syslog-relay and the relay
is cat'd via exec -a "systemd-journal-fwd". rsyslog.d conf rename
99-decnet.conf → 50-journal-forward.conf. SSH capture script:
/var/decnet/captured → /var/lib/systemd/coredump (real systemd path),
logger tag decnet-capture → systemd-journal. Compose volume updated
to match the new in-container quarantine path.
SD element ID shifts decnet@55555 → relay@55555; synced across
collector, parser, sniffer, prober, formatter, tests, and docs so the
host-side pipeline still matches what containers emit.
Extends tracing to every remaining module: all 23 API route handlers,
correlation engine, sniffer (fingerprint/p0f/syslog), prober (jarm/hassh/tcpfp),
profiler behavioral analysis, logging subsystem, engine, and mutator.
Bridges the ingester→SSE trace gap by persisting trace_id/span_id columns on
the logs table and creating OTEL span links in the SSE endpoint. Adds log-trace
correlation via _TraceContextFilter injecting otel_trace_id into Python LogRecords.
Includes development/docs/TRACING.md with full span reference (76 spans),
pipeline propagation architecture, quick start guide, and troubleshooting.
Gated by DECNET_DEVELOPER_TRACING env var (default off, zero overhead).
When enabled, traces flow through FastAPI routes, background workers
(collector, ingester, profiler, sniffer, prober), engine/mutator
operations, and all DB calls via TracedRepository proxy.
Includes Jaeger docker-compose for local dev and 18 unit tests.
The active prober emits tcpfp_fingerprint events with TTL, window, MSS etc.
from the attacker's SYN-ACK. These were invisible to the behavioral profiler
for two reasons:
1. target_ip (prober's field name for attacker IP) was not in _IP_FIELDS in
collector/worker.py or correlation/parser.py, so the profiler re-parsed
raw_lines and got attacker_ip=None, never attributing prober events to
the attacker profile.
2. sniffer_rollup only handled tcp_syn_fingerprint (passive sniffer) and
ignored tcpfp_fingerprint (active prober). Prober events use different
field names: window_size/window_scale/sack_ok vs window/wscale/has_sack.
Changes:
- Add target_ip to _IP_FIELDS in collector and parser
- Add _PROBER_TCPFP_EVENT and _INITIAL_TTL table to behavioral.py
- sniffer_rollup now processes tcpfp_fingerprint: maps field names, derives
OS from TTL via _os_from_ttl, computes hop_distance = initial_ttl - observed
- Expand prober DEFAULT_TCPFP_PORTS to [22,80,443,8080,8443,445,3389] for
better SYN-ACK coverage on attacker machines
- Add 4 tests covering prober OS detection, hop distance, and field mapping
Remove unused imports (ruff F401), suppress B324 false positives on
spec-mandated MD5 in HASSH/JA3/JA3S fingerprinting, drop unused
record_version assignment in JARM parser, and pin pip>=26.0 in dev
deps to address CVE-2025-8869 and CVE-2026-1703.
Extends the prober with two new active probe types alongside JARM:
- HASSHServer: SSH server fingerprinting via KEX_INIT algorithm ordering
(MD5 hash of kex;enc_s2c;mac_s2c;comp_s2c, pure stdlib)
- TCP/IP stack: OS/tool fingerprinting via SYN-ACK analysis using scapy
(TTL, window size, DF bit, MSS, TCP options ordering, SHA256 hash)
Worker probe cycle now runs three phases per IP with independent
per-type port tracking. Ingester extracts bounties for all three
fingerprint types.
