wiki: recover Module-Reference Web/Services/Workers
88
Module-Reference-Services.md
Normal file
88
Module-Reference-Services.md
Normal file
@@ -0,0 +1,88 @@
|
|||||||
|
# Module Reference: Services
|
||||||
|
|
||||||
|
Code-level view of every service plugin under `decnet/services/`. For the user-facing description of what each honeypot does (personas, credentials, bait), see [Services catalog](Services-Catalog). For writing your own plugin, see [Writing a Service Plugin](Writing-a-Service-Plugin).
|
||||||
|
|
||||||
|
Citation format: `decnet/services/<file>::<symbol>`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Core
|
||||||
|
|
||||||
|
### `services/base.py::BaseService`
|
||||||
|
|
||||||
|
Abstract contract every plugin implements. Class attributes:
|
||||||
|
|
||||||
|
- `name: str` — unique slug (e.g. `"ssh"`).
|
||||||
|
- `ports: list[int]` — ports the service listens on inside the container.
|
||||||
|
- `default_image: str` — Docker image tag, or `"build"` when a Dockerfile context is needed.
|
||||||
|
- `fleet_singleton: bool = False` — `True` means the service runs once fleet-wide, not per-decky (used by the sniffer).
|
||||||
|
|
||||||
|
Methods:
|
||||||
|
|
||||||
|
- `compose_fragment(decky_name, log_target, service_cfg) -> dict` — **abstract**; returns the docker-compose service dict. Networking keys are injected by the composer and **must not** be set here.
|
||||||
|
- `dockerfile_context() -> Path | None` — return the build-context path if a custom image is needed; `None` means `default_image` is pulled directly.
|
||||||
|
|
||||||
|
### `services/registry.py`
|
||||||
|
|
||||||
|
Plugin auto-discovery.
|
||||||
|
|
||||||
|
- `registry.py::_load_plugins` — walks `decnet/services/`, imports every module, and collects all `BaseService` subclasses.
|
||||||
|
- `registry.py::register_custom_service` — register an out-of-tree plugin at runtime (used by tests and third-party integrations).
|
||||||
|
- `registry.py::get_service(name)` — fetch one plugin by slug.
|
||||||
|
- `registry.py::all_services()` — return `{slug: instance}` for every registered plugin.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Service plugins
|
||||||
|
|
||||||
|
All rows are subclasses of `BaseService`. The `Image` column is `build` for every in-tree plugin — each one ships a Dockerfile context under `templates/<slug>/`. See [Service Personas](Service-Personas) and [Archetypes](Archetypes) for how they're composed into deckies.
|
||||||
|
|
||||||
|
| Slug | Ports | Image | Class | One-line description |
|
||||||
|
|------|-------|-------|-------|----------------------|
|
||||||
|
| `conpot` | 502, 161 (UDP), 80 | build | `ConpotService` | ICS/SCADA honeypot covering Modbus, SNMP, and HTTP. |
|
||||||
|
| `docker_api` | 2375, 2376 | build | `DockerAPIService` | Fake Docker Engine API (unauthenticated + TLS ports). |
|
||||||
|
| `elasticsearch` | 9200 | build | `ElasticsearchService` | Exposed Elasticsearch REST endpoint with index bait. |
|
||||||
|
| `ftp` | 21 | build | `FTPService` | vsftpd honeypot with captured-upload persistence. |
|
||||||
|
| `http` | 80, 443 | build | `HTTPService` | Realistic HTTP/HTTPS web server with persona templates. |
|
||||||
|
| `https` | 443 | build | `HTTPSService` | TLS-terminated web persona (paired with `http` or standalone). |
|
||||||
|
| `imap` | 143, 993 | build | `IMAPService` | Dovecot IMAP/IMAPS with fake mailboxes. |
|
||||||
|
| `k8s` | 6443, 8080 | build | `KubernetesAPIService` | Mock Kubernetes API server surface. |
|
||||||
|
| `ldap` | 389, 636 | build | `LDAPService` | OpenLDAP with a seeded fake directory. |
|
||||||
|
| `llmnr` | 5355, 5353 | build | `LLMNRService` | LLMNR/mDNS/NBNS poisoning detector. |
|
||||||
|
| `mongodb` | 27017 | build | `MongoDBService` | Exposed MongoDB with unauthenticated collections. |
|
||||||
|
| `mqtt` | 1883 | build | `MQTTService` | Mosquitto broker with discoverable topics. |
|
||||||
|
| `mssql` | 1433 | build | `MSSQLService` | SQL Server persona emulating login + TDS banner. |
|
||||||
|
| `mysql` | 3306 | build | `MySQLService` | MySQL server with weak creds and seeded schema. |
|
||||||
|
| `pop3` | 110, 995 | build | `POP3Service` | Dovecot POP3/POP3S counterpart to `imap`. |
|
||||||
|
| `postgres` | 5432 | build | `PostgresService` | PostgreSQL server with bait databases. |
|
||||||
|
| `rdp` | 3389 | build | `RDPService` | RDP (xrdp) honeypot with credential capture. |
|
||||||
|
| `redis` | 6379 | build | `RedisService` | Unauthenticated Redis with bait keys. |
|
||||||
|
| `sip` | 5060 | build | `SIPService` | SIP/VoIP registrar bait. |
|
||||||
|
| `smb` | 445, 139 | build | `SMBService` | Samba server with seeded fake shares. |
|
||||||
|
| `smtp` | 25, 587 | build | `SMTPService` | Postfix SMTP endpoint with auth capture. |
|
||||||
|
| `smtp_relay` | 25, 587 | build | `SMTPRelayService` | Open-relay bait — accepts any RCPT TO and delivers. |
|
||||||
|
| `sniffer` | — | build | `SnifferService` | Fleet-wide passive MACVLAN sniffer (`fleet_singleton=True`). |
|
||||||
|
| `snmp` | 161 (UDP) | build | `SNMPService` | SNMP agent with fake MIB tree. |
|
||||||
|
| `ssh` | 22 | build | `SSHService` | Interactive OpenSSH (replaces Cowrie) for realistic fingerprints. |
|
||||||
|
| `telnet` | 23 | build | `TelnetService` | Real `busybox telnetd` with rsyslog logging pipeline. |
|
||||||
|
| `tftp` | 69 (UDP) | build | `TFTPService` | TFTP server bait for firmware-fetch probing. |
|
||||||
|
| `vnc` | 5900 | build | `VNCService` | VNC server with weak auth. |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## How they're assembled
|
||||||
|
|
||||||
|
- `engine/deployer.py` reads the archetype-selected slugs and calls each plugin's `compose_fragment()` to build `docker-compose.yml`. See [Module Reference: Workers](Module-Reference-Workers) for the deployer internals.
|
||||||
|
- The canonical `syslog_bridge.py` is copied into every active build context by `engine/deployer.py::_sync_logging_helper` so each container forwards logs identically. See [Logging](Logging-and-Syslog).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## See also
|
||||||
|
|
||||||
|
- [Services catalog](Services-Catalog) — user view: personas, creds, what each honeypot looks like to an attacker.
|
||||||
|
- [Writing a Service Plugin](Writing-a-Service-Plugin) — step-by-step guide.
|
||||||
|
- [Archetypes](Archetypes) — how slugs are bundled into OS personas.
|
||||||
|
- [Design overview](Design-Overview)
|
||||||
|
- [REST API](REST-API-Reference)
|
||||||
|
- [Logging](Logging-and-Syslog)
|
||||||
|
- [Developer guide](Developer-Guide)
|
||||||
161
Module-Reference-Web.md
Normal file
161
Module-Reference-Web.md
Normal file
@@ -0,0 +1,161 @@
|
|||||||
|
# Module Reference: Web
|
||||||
|
|
||||||
|
Code-level reference for every module under `decnet/web/`. For the user-facing API see [REST API](REST-API-Reference); for the dashboard UX see [Web Dashboard](Web-Dashboard); architecture is in [Design overview](Design-Overview).
|
||||||
|
|
||||||
|
Citation format: `decnet/web/<path>::<symbol>`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## `decnet/web/api.py`
|
||||||
|
|
||||||
|
FastAPI application factory and lifespan wiring. Instantiates the `FastAPI` app, mounts every router under `decnet/web/router/`, attaches exception handlers, and owns the background-task handles for the ingester, profiler, prober, sniffer, and correlation workers.
|
||||||
|
|
||||||
|
- `api.py::get_background_tasks` — returns the live `dict` of worker task handles so the `/health` endpoint can introspect liveness.
|
||||||
|
- `api.py::lifespan` — async context manager run by FastAPI on startup/shutdown: initializes the repository, seeds the admin user, spawns worker tasks, and cancels them on exit.
|
||||||
|
- `api.py::validation_exception_handler` — maps request-validation errors to the specific HTTP codes the contract tests require.
|
||||||
|
- `api.py::pydantic_validation_exception_handler` — handles Pydantic errors raised during manual model instantiation (e.g. state hydration).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## `decnet/web/auth.py`
|
||||||
|
|
||||||
|
Password hashing and JWT issuance primitives. No business logic — just stateless helpers shared by the login router and dependency injectors.
|
||||||
|
|
||||||
|
- `auth.py::verify_password` — sync bcrypt verify.
|
||||||
|
- `auth.py::get_password_hash` — sync bcrypt hash.
|
||||||
|
- `auth.py::averify_password` — async verify (offloaded to a threadpool).
|
||||||
|
- `auth.py::ahash_password` — async hash (threadpool).
|
||||||
|
- `auth.py::create_access_token` — issue a signed JWT with configured TTL and subject claim.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## `decnet/web/ingester.py`
|
||||||
|
|
||||||
|
Background worker that tails the DECNET ingest log (`DECNET_INGEST_LOG_FILE.json`) and writes rows into the `logs` and `bounties` tables. Checkpoints file offset via the `State` table so restarts are idempotent.
|
||||||
|
|
||||||
|
- `ingester.py::log_ingestion_worker` — asyncio task loop: tail + parse + flush batches.
|
||||||
|
- `ingester.py::_flush_batch` — commits a batch of log rows and returns the new file position (bulk-insert, one transaction).
|
||||||
|
- `ingester.py::_extract_bounty` — inspects a parsed log row and returns a bounty payload (creds, uploaded files, etc.) or `None`.
|
||||||
|
- Const: `_INGEST_STATE_KEY` — key under which the tail offset is stored in `State`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## `decnet/web/dependencies.py`
|
||||||
|
|
||||||
|
FastAPI dependencies for repository injection, authentication, RBAC, and user-cache management. Every router imports from here — never directly from the repo module.
|
||||||
|
|
||||||
|
- `dependencies.py::get_repo` — repository DI entrypoint; returns the configured `BaseRepository` singleton.
|
||||||
|
- `dependencies.py::invalidate_user_cache` — drop a single username (or all) from the auth caches (`_get_user_cached`, `get_user_by_username_cached`).
|
||||||
|
- `dependencies.py::get_user_by_username_cached` — TTL-cached lookup used on the login hot path.
|
||||||
|
- `dependencies.py::get_stream_user` — auth for SSE endpoints; accepts Bearer header **or** `?token=` query param (EventSource cannot set headers).
|
||||||
|
- `dependencies.py::get_current_user` — Bearer-token auth; enforces the `must_change_password` flag.
|
||||||
|
- `dependencies.py::get_current_user_unchecked` — same, but skips the change-password gate (used by the change-password endpoint itself).
|
||||||
|
- `dependencies.py::require_role` — factory returning an RBAC dependency enforcing role membership (`admin`, `analyst`, `viewer`).
|
||||||
|
- `dependencies.py::require_stream_role` — same, for SSE endpoints that take a query-param token.
|
||||||
|
- Private helpers: `_reset_user_cache`, `_get_user_cached`, `_decode_token` — cache reset (tests), cached user fetch by UUID, JWT decode.
|
||||||
|
- Consts: `_USER_TTL`, `_USERNAME_TTL` — cache TTLs in seconds.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## `decnet/web/db/`
|
||||||
|
|
||||||
|
### `db/factory.py`
|
||||||
|
Repository factory. Selects a `BaseRepository` implementation based on the `DECNET_DB_TYPE` env var (`sqlite` or `mysql`). Returns a module-level singleton.
|
||||||
|
|
||||||
|
- `db/factory.py::get_repository` — construct and cache the active repository.
|
||||||
|
|
||||||
|
### `db/repository.py`
|
||||||
|
Abstract base class declaring every storage operation the dashboard and workers rely on.
|
||||||
|
|
||||||
|
- `db/repository.py::BaseRepository` — abstract interface.
|
||||||
|
Notable methods: `initialize`, `add_log`, `add_logs`, `get_logs`, `get_total_logs`, `get_stats_summary`, `get_deckies`, `get_user_by_username`, `get_user_by_uuid`, `create_user`, `update_user_password`, `list_users`, `delete_user`, `update_user_role`, `purge_logs_and_bounties`, `add_bounty`, `get_bounties`, `get_total_bounties`, `get_state`, `set_state`, `get_max_log_id`, `get_logs_after_id`, `get_all_bounties_by_ip`, `get_bounties_for_ips`, `upsert_attacker`, `upsert_attacker_behavior`, `get_attacker_behavior`, `get_behaviors_for_ips`, `get_attacker_by_uuid`, `get_attackers`, `get_total_attackers`, `get_attacker_commands`, `get_attacker_artifacts`.
|
||||||
|
|
||||||
|
### `db/sqlmodel_repo.py`
|
||||||
|
Shared SQLModel/SQLAlchemy-async concrete implementation. Both the SQLite and MySQL backends subclass this and override only dialect-specific bits (DDL introspection, `get_log_histogram`).
|
||||||
|
|
||||||
|
- `db/sqlmodel_repo.py::SQLModelRepository` — concrete async repository.
|
||||||
|
- `db/sqlmodel_repo.py::_detach_close` — runs session close in a fresh asyncio task so cancellation in the caller never leaves sessions hanging.
|
||||||
|
- `db/sqlmodel_repo.py::_safe_session` — session context manager with cancellation-safe cleanup.
|
||||||
|
- Key methods: `initialize`, `reinitialize`, `add_log`, `add_logs`, `get_logs`, `get_log_histogram` (override per backend), `get_stats_summary`, `upsert_attacker`, `upsert_attacker_behavior`, `get_attacker_commands`, `get_attacker_artifacts`. Many private helpers for filter composition, JSON-field equality, and row normalization.
|
||||||
|
|
||||||
|
### `db/sqlite/database.py`
|
||||||
|
- `get_async_engine`, `get_sync_engine`, `init_db`, `get_session` — SQLite engine and session factories. `init_db` creates tables synchronously (used by CLI/tests).
|
||||||
|
|
||||||
|
### `db/sqlite/repository.py`
|
||||||
|
- `SQLiteRepository` — `SQLModelRepository` subclass using `aiosqlite`. Overrides `_migrate_attackers_table`, `_json_field_equals`, `get_log_histogram` for SQLite dialect.
|
||||||
|
|
||||||
|
### `db/mysql/database.py`
|
||||||
|
MySQL async engine factory.
|
||||||
|
|
||||||
|
- `build_mysql_url` — compose an async SQLAlchemy URL using `asyncmy`.
|
||||||
|
- `resolve_url` — explicit arg → `DECNET_DB_URL` env → built from components.
|
||||||
|
- `get_async_engine` — create the `AsyncEngine`.
|
||||||
|
- Consts: `DEFAULT_POOL_SIZE`, `DEFAULT_MAX_OVERFLOW`, `DEFAULT_POOL_RECYCLE`, `DEFAULT_POOL_PRE_PING`.
|
||||||
|
|
||||||
|
### `db/mysql/repository.py`
|
||||||
|
- `MySQLRepository` — `SQLModelRepository` subclass using `asyncmy`. Adds MySQL migrations (`_migrate_attackers_table`, `_migrate_column_types` to upgrade `TEXT` → `MEDIUMTEXT` on large-JSON columns) and dialect-specific histogram SQL.
|
||||||
|
|
||||||
|
### `db/models.py`
|
||||||
|
SQLModel table classes and Pydantic request/response DTOs.
|
||||||
|
|
||||||
|
- Tables: `User`, `Log`, `Bounty`, `State`, `Attacker`, `AttackerBehavior` (1:1 with `Attacker` by UUID).
|
||||||
|
- Request models: `LoginRequest`, `ChangePasswordRequest`, `MutateIntervalRequest`, `DeployIniRequest`, `CreateUserRequest`, `UpdateUserRoleRequest`, `ResetUserPasswordRequest`, `DeploymentLimitRequest`, `GlobalMutationIntervalRequest`.
|
||||||
|
- Response models: `Token`, `LogsResponse`, `BountyResponse`, `AttackersResponse`, `StatsResponse`, `UserResponse`, `ConfigResponse`, `AdminConfigResponse`, `ComponentHealth`, `HealthResponse`.
|
||||||
|
- Helpers: `_normalize_null` — collapses SQL `NULL` / empty-string variance.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## `decnet/web/router/`
|
||||||
|
|
||||||
|
All route modules follow one pattern: declare a module-level `APIRouter`, attach a single endpoint function. The aggregator `router/__init__.py` imports and mounts them. Response caches (`functools.lru_cache` with TTL) are used on read endpoints where invalidation is explicit.
|
||||||
|
|
||||||
|
### `router/logs/`
|
||||||
|
- `api_get_logs.py::get_logs` — paginated log feed with filters. Cached total-count via `_get_total_logs_cached` / `_reset_total_cache`.
|
||||||
|
- `api_get_histogram.py::get_logs_histogram` — binned log timeseries for the dashboard chart.
|
||||||
|
|
||||||
|
### `router/stats/`
|
||||||
|
- `api_get_stats.py::get_stats` — dashboard summary: event counts, attackers, bounties, deckies. Cached.
|
||||||
|
|
||||||
|
### `router/stream/`
|
||||||
|
- `api_stream_events.py::stream_events` — Server-Sent Events endpoint streaming new logs by id. Uses `get_stream_user` (query-param token). `_build_trace_links` builds OTEL span links from persisted `trace_id` / `span_id`.
|
||||||
|
|
||||||
|
### `router/bounty/`
|
||||||
|
- `api_get_bounties.py::get_bounties` — paginated bounty (captured credentials/files) feed. TTL-cached defaults.
|
||||||
|
|
||||||
|
### `router/attackers/`
|
||||||
|
- `api_get_attackers.py::get_attackers` — paginated attacker profile list. Cached total.
|
||||||
|
- `api_get_attacker_detail.py::get_attacker_detail` — single attacker profile joined with its `AttackerBehavior` row.
|
||||||
|
- `api_get_attacker_commands.py::get_attacker_commands` — paginated executed-commands list per attacker, optional service filter.
|
||||||
|
- `api_get_attacker_artifacts.py::get_attacker_artifacts` — captured file-drop artifacts for an attacker, newest first.
|
||||||
|
|
||||||
|
### `router/artifacts/`
|
||||||
|
- `api_get_artifact.py::get_artifact` — download a single captured artifact. `_resolve_artifact_path` validates inputs, resolves the on-disk path, and asserts it stays under `ARTIFACTS_ROOT` (path-traversal guard). Regex guards `_DECKY_RE`, `_STORED_AS_RE` sanitize path components.
|
||||||
|
|
||||||
|
### `router/fleet/`
|
||||||
|
- `api_deploy_deckies.py::api_deploy_deckies` — server-side deploy trigger (admin only). See [CLI Reference](CLI-Reference) for the same operation from shell.
|
||||||
|
- `api_get_deckies.py::get_deckies` — running decky inventory. Cached.
|
||||||
|
- `api_mutate_decky.py::api_mutate_decky` — force-mutate a single decky. See [Mutation and Randomization](Mutation-and-Randomization).
|
||||||
|
- `api_mutate_interval.py::api_update_mutate_interval` — set a decky's per-machine mutation interval. `_parse_duration` converts `5d`/`12h` strings to minutes.
|
||||||
|
|
||||||
|
### `router/config/`
|
||||||
|
- `api_get_config.py::api_get_config` — read the active config and user list (cached via `_get_list_users_cached` and `_get_state_cached`). `invalidate_list_users_cache` is called after any user mutation.
|
||||||
|
- `api_update_config.py::api_update_deployment_limit`, `api_update_global_mutation_interval` — write config knobs.
|
||||||
|
- `api_manage_users.py::api_create_user`, `api_delete_user`, `api_update_user_role`, `api_reset_user_password` — admin-only user CRUD.
|
||||||
|
- `api_reinit.py::api_reinit` — destructive reinit endpoint (drops logs/bounties/attackers, keeps users). Admin only.
|
||||||
|
|
||||||
|
### `router/auth/`
|
||||||
|
- `api_login.py::login` — username/password → JWT.
|
||||||
|
- `api_change_pass.py::change_password` — enforced when `must_change_password` is set; uses `get_current_user_unchecked`.
|
||||||
|
|
||||||
|
### `router/health/`
|
||||||
|
- `api_get_health.py::get_health` — aggregated health: DB liveness (`_check_database_cached`), Docker daemon reachable, background-worker statuses from `api.get_background_tasks()`. Cache reset helpers (`_reset_db_cache`, `_reset_docker_cache`) exist for tests.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## See also
|
||||||
|
|
||||||
|
- [Design overview](Design-Overview)
|
||||||
|
- [REST API](REST-API-Reference)
|
||||||
|
- [Logging](Logging-and-Syslog)
|
||||||
|
- [Services catalog](Services-Catalog)
|
||||||
|
- [Developer guide](Developer-Guide)
|
||||||
215
Module-Reference-Workers.md
Normal file
215
Module-Reference-Workers.md
Normal file
@@ -0,0 +1,215 @@
|
|||||||
|
# Module Reference: Workers
|
||||||
|
|
||||||
|
Code-level reference for every worker/subsystem module outside `decnet/web/`. For the pipeline overview see [Design overview](Design-Overview); for log format see [Logging](Logging-and-Syslog).
|
||||||
|
|
||||||
|
Each subsection lists: **entrypoint** (function spawned by the lifespan or CLI), **loop** (what it does on each tick), **reads** (input source), **writes** (output sink).
|
||||||
|
|
||||||
|
Citation format: `decnet/<path>::<symbol>`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## `decnet/collector/` — Host-side Docker log collector
|
||||||
|
|
||||||
|
Streams logs out of every DECNET-owned container on the host, parses the RFC 5424 line format, and appends to the shared JSON ingest file that `decnet/web/ingester.py` tails.
|
||||||
|
|
||||||
|
### `collector/worker.py`
|
||||||
|
|
||||||
|
- **Entrypoint:** `worker.py::log_collector_worker` — asyncio task started by the API lifespan.
|
||||||
|
- **Loop:** enumerate running service containers via the Docker SDK; per container, spawn `_stream_container` which opens a log stream and appends parsed events to the ingest file. Rate-limits duplicate events.
|
||||||
|
- **Reads:** Docker API container log streams; `decnet-state.json` (for the canonical service container name set).
|
||||||
|
- **Writes:** JSON ingest log file (consumed by `web/ingester.py`).
|
||||||
|
|
||||||
|
Key functions:
|
||||||
|
- `worker.py::parse_rfc5424` — parse a DECNET RFC 5424 line into a structured dict (SD params, IP fields, msg k/v).
|
||||||
|
- `worker.py::is_service_container`, `is_service_event` — gate which containers to ingest.
|
||||||
|
- `worker.py::_stream_container` — per-container streaming loop; `_reopen_if_needed` handles inode changes (log rotation).
|
||||||
|
- `worker.py::_should_ingest` — dedup + rate-limit filter.
|
||||||
|
- Consts: `_RFC5424_RE`, `_SD_BLOCK_RE`, `_PARAM_RE`, `_IP_FIELDS`, `_MSG_KV_RE`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## `decnet/profiler/` — Attacker profile builder
|
||||||
|
|
||||||
|
Incrementally joins raw log rows into per-attacker profiles in the `attackers` + `attacker_behavior` tables.
|
||||||
|
|
||||||
|
### `profiler/worker.py`
|
||||||
|
|
||||||
|
- **Entrypoint:** `worker.py::attacker_profile_worker` — long-running asyncio task.
|
||||||
|
- **Loop:** every N seconds, fetch logs with `id > last_id` (`_incremental_update`), bucket by attacker IP, call `_update_profiles` to upsert `Attacker` and `AttackerBehavior` rows. State key `_STATE_KEY` persists the last-seen log id.
|
||||||
|
- **Reads:** `logs` table (incremental via `get_logs_after_id`).
|
||||||
|
- **Writes:** `attackers`, `attacker_behavior`, `state` tables.
|
||||||
|
|
||||||
|
Helpers: `_build_record`, `_first_contact_deckies`, `_extract_commands_from_events`. Consts: `_BATCH_SIZE`, `_COMMAND_EVENT_TYPES`, `_COMMAND_FIELDS`.
|
||||||
|
|
||||||
|
### `profiler/behavioral.py`
|
||||||
|
|
||||||
|
Stateless analyzers used by `worker.py::_update_profiles`.
|
||||||
|
|
||||||
|
- `behavioral.py::timing_stats` — inter-arrival time mean/stddev/CV.
|
||||||
|
- `behavioral.py::classify_behavior` — coarse behavior bucket (scanner / interactive / automated).
|
||||||
|
- `behavioral.py::guess_tools` — match (mean IAT, CV) against C2 beacon profiles.
|
||||||
|
- `behavioral.py::detect_tools_from_headers` — scan HTTP `request` events for tool-identifying headers.
|
||||||
|
- `behavioral.py::phase_sequence` — derive recon→exfil transitions.
|
||||||
|
- `behavioral.py::sniffer_rollup` — roll up `tcp_syn_fingerprint` + `tcp_flow_timing` events.
|
||||||
|
- `behavioral.py::build_behavior_record` — compose the dict persisted to `attacker_behavior`.
|
||||||
|
- Helpers: `_os_from_ttl`, `_int_or_none`, `guess_tool` (deprecated, kept for BC).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## `decnet/sniffer/` — Fleet-wide passive MACVLAN sniffer
|
||||||
|
|
||||||
|
Runs as a `fleet_singleton` container on the MACVLAN (or falls back to the `ipvlan` host iface) and emits TLS/TCP fingerprints for every flow touching a decky.
|
||||||
|
|
||||||
|
### `sniffer/worker.py`
|
||||||
|
|
||||||
|
- **Entrypoint:** `worker.py::sniffer_worker` — started as `asyncio.create_task` by the API lifespan.
|
||||||
|
- **Loop:** `worker.py::_sniff_loop` (runs in a dedicated thread via `asyncio.to_thread`) drives `scapy.sniff` on the MACVLAN interface; each packet goes to `SnifferEngine.on_packet`.
|
||||||
|
- **Reads:** live packets on the sniffer interface; `decnet-state.json` → `_load_ip_to_decky`.
|
||||||
|
- **Writes:** RFC 5424 syslog lines via `sniffer/syslog.py::write_event` (both raw syslog and parsed JSON).
|
||||||
|
|
||||||
|
Helpers: `_interface_exists`, `_load_ip_to_decky`.
|
||||||
|
|
||||||
|
### `sniffer/fingerprint.py::SnifferEngine`
|
||||||
|
|
||||||
|
Stateful TLS + TCP fingerprint engine. Tracks sessions, TCP RTTs, flow aggregation, dedup. Emits `tls_handshake`, `tcp_syn_fingerprint`, `tcp_flow_timing` events.
|
||||||
|
|
||||||
|
Notable methods: `on_packet` (main dispatch), `_update_flow`, `_flush_flow`, `flush_all_flows`, `_flush_idle_flows`, `_resolve_decky`, `_is_duplicate`. Private parsers: `_parse_client_hello`, `_parse_server_hello`, `_parse_certificate`, `_parse_x509_der`, `_extract_sans`; fingerprint algorithms: `_ja3`, `_ja3s`, `_ja4`, `_ja4s`, `_ja4l`.
|
||||||
|
|
||||||
|
### `sniffer/p0f.py`
|
||||||
|
|
||||||
|
Passive OS fingerprinting (p0f-lite).
|
||||||
|
|
||||||
|
- `initial_ttl` — round observed TTL up to the nearest initial-TTL bucket.
|
||||||
|
- `hop_distance` — estimate hop count.
|
||||||
|
- `guess_os` — coarse OS bucket from SYN characteristics.
|
||||||
|
- `_match_signature` — predicate evaluator.
|
||||||
|
|
||||||
|
### `sniffer/syslog.py`
|
||||||
|
|
||||||
|
RFC 5424 emitter for the sniffer.
|
||||||
|
|
||||||
|
- `syslog.py::syslog_line` — build a line.
|
||||||
|
- `syslog.py::write_event` — append a syslog line to the raw log and its parsed JSON to the json log.
|
||||||
|
- Helpers: `_sd_escape`, `_sd_element`. Consts: `_FACILITY_LOCAL0`, `_SD_ID`, `_NILVALUE`, `SEVERITY_INFO`, `SEVERITY_WARNING`, `_MAX_HOSTNAME`, `_MAX_APPNAME`, `_MAX_MSGID`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## `decnet/prober/` — Active attacker fingerprinting
|
||||||
|
|
||||||
|
Standalone worker process (separate from the API) that actively probes IPs observed in the logs to collect JARM, HASSHServer, and TCP/IP fingerprints without ever impersonating DECNET.
|
||||||
|
|
||||||
|
### `prober/worker.py`
|
||||||
|
|
||||||
|
- **Entrypoint:** `worker.py::prober_worker` — main loop for the standalone prober process.
|
||||||
|
- **Loop:** tail the JSON ingest log (`_discover_attackers`), cycle through unique attacker IPs (`_probe_cycle`), run `_jarm_phase`, `_hassh_phase`, `_tcpfp_phase`, emit results as RFC 5424 events via `_syslog_line` + `_write_event`.
|
||||||
|
- **Reads:** JSON ingest log; configured probe targets.
|
||||||
|
- **Writes:** RFC 5424 events into the ingest file so the regular collector pipeline picks them up.
|
||||||
|
|
||||||
|
### `prober/jarm.py`
|
||||||
|
|
||||||
|
Pure-stdlib JARM TLS fingerprinting (10 ClientHello probes, reassembled into the 62-char JARM hash).
|
||||||
|
|
||||||
|
- `jarm.py::jarm_hash` — public entrypoint.
|
||||||
|
- Helpers: `_build_client_hello`, `_send_probe`, `_parse_server_hello`, `_compute_jarm`; per-extension builders (`_ext_*`).
|
||||||
|
|
||||||
|
### `prober/hassh.py`
|
||||||
|
|
||||||
|
HASSHServer SSH fingerprinting (MD5 of `kex;enc_s2c;mac_s2c;comp_s2c` name-lists).
|
||||||
|
|
||||||
|
- `hassh.py::hassh_server` — public entrypoint.
|
||||||
|
- Helpers: `_ssh_connect`, `_read_banner`, `_read_ssh_packet`, `_parse_kex_init`, `_compute_hassh`, `_recv_exact`.
|
||||||
|
|
||||||
|
### `prober/tcpfp.py`
|
||||||
|
|
||||||
|
TCP/IP stack fingerprinting via crafted SYN + SYN-ACK analysis (scapy).
|
||||||
|
|
||||||
|
- `tcpfp.py::tcp_fingerprint` — public entrypoint.
|
||||||
|
- Helpers: `_send_syn`, `_send_rst`, `_parse_synack`, `_extract_options_order`, `_compute_fingerprint`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## `decnet/mutator/` — Persona mutation engine
|
||||||
|
|
||||||
|
Periodically swaps a decky's services/personas (intra-archetype shuffle) so the same IP/MAC shows different attack surfaces over time. See [Mutation and Randomization](Mutation-and-Randomization).
|
||||||
|
|
||||||
|
### `mutator/engine.py`
|
||||||
|
|
||||||
|
- **Entrypoint:** `engine.py::run_watch_loop` — infinite loop checking for deckies past their mutation interval; spawned by the CLI or the API.
|
||||||
|
- **Loop:** `mutate_all` iterates due deckies → `mutate_decky` runs intra-archetype shuffle (rewrite compose, `docker compose up -d`).
|
||||||
|
- **Reads:** `decnet-state.json`, repository (per-decky mutation interval).
|
||||||
|
- **Writes:** rewritten `docker-compose.yml`, updated state, new containers.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## `decnet/correlation/` — Cross-decky attacker traversal
|
||||||
|
|
||||||
|
Builds attacker-traversal graphs by correlating events across multiple deckies.
|
||||||
|
|
||||||
|
### `correlation/engine.py::CorrelationEngine`
|
||||||
|
|
||||||
|
- **Entrypoint:** `ingest_file(path)` (batch) or `ingest(line)` (stream) — no background task by default; callable from CLI and tests.
|
||||||
|
- **Loop:** per-line: parse with `parser.parse_line`, append `TraversalHop` to per-IP `AttackerTraversal`.
|
||||||
|
- **Reads:** RFC 5424 syslog lines (file or stream).
|
||||||
|
- **Writes:** nothing by default; exposes `traversals(min_deckies)`, `all_attackers()`, `report_table()`, `report_json()`, `traversal_syslog_lines()`. Helper `_fmt_duration`.
|
||||||
|
|
||||||
|
### `correlation/graph.py`
|
||||||
|
Data classes. `TraversalHop` = a single event; `AttackerTraversal` = all activity from one IP across ≥ 2 deckies with `first_seen`, `last_seen`, `duration_seconds`, `deckies` (ordered), `decky_count`, `path` (`decky-01 → decky-03 → decky-07`), `to_dict`.
|
||||||
|
|
||||||
|
### `correlation/parser.py`
|
||||||
|
Stateless RFC 5424 parser shared by `correlation/engine.py` and ad-hoc tooling.
|
||||||
|
|
||||||
|
- `parser.py::parse_line` — parse one line into `LogEvent`.
|
||||||
|
- `parser.py::LogEvent` — parsed event record.
|
||||||
|
- Helpers: `_parse_sd_params`, `_extract_attacker_ip`. Consts: `_RFC5424_RE`, `_SD_BLOCK_RE`, `_PARAM_RE`, `_IP_FIELDS`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## `decnet/engine/` — Deploy/teardown orchestration
|
||||||
|
|
||||||
|
### `engine/deployer.py`
|
||||||
|
|
||||||
|
- **Entrypoint:** `deployer.py::deploy`, `teardown`, `status` — called from the CLI and the fleet router.
|
||||||
|
- **Loop (`deploy`):** render `docker-compose.yml` from archetypes + service plugins → sync the canonical `syslog_bridge.py` into every build context (`_sync_logging_helper`) → invoke `docker compose up -d` via `_compose` with retry on transient failures (`_compose_with_retry`).
|
||||||
|
- **Reads:** CLI args / INI config / archetype definitions / service plugin `compose_fragment`.
|
||||||
|
- **Writes:** `docker-compose.yml`, built images, running containers, `decnet-state.json`.
|
||||||
|
|
||||||
|
Consts: `COMPOSE_FILE`, `_CANONICAL_LOGGING`, `_PERMANENT_ERRORS`. Helper: `_print_status`.
|
||||||
|
|
||||||
|
See [Deployment Modes](Deployment-Modes) and [Teardown and State](Teardown-and-State).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## `decnet/logging/` — Application logging helpers
|
||||||
|
|
||||||
|
Standard-library `logging` integration that makes DECNET's own Python logs RFC 5424-compliant and mergeable with service logs.
|
||||||
|
|
||||||
|
### `logging/__init__.py`
|
||||||
|
- `__init__.py::get_logger(component)` — return a named logger that self-identifies as `component` in RFC 5424.
|
||||||
|
- `__init__.py::enable_trace_context` — install the OTEL trace-context filter on the root `decnet` logger.
|
||||||
|
- `__init__.py::_ComponentFilter`, `_TraceContextFilter` — inject `decnet_component`, `otel_trace_id`, `otel_span_id` onto LogRecords so the formatter can emit them as SD params.
|
||||||
|
|
||||||
|
### `logging/syslog_formatter.py`
|
||||||
|
- `syslog_formatter.py::format_rfc5424` — return a single RFC 5424-compliant syslog line (no trailing newline).
|
||||||
|
- Helpers: `_pri`, `_truncate`, `_sd_escape`, `_sd_element`. Consts: `FACILITY_LOCAL0`, `NILVALUE`, `_SD_ID`, `SEVERITY_INFO`, `SEVERITY_WARNING`, `SEVERITY_ERROR`, `_MAX_HOSTNAME`, `_MAX_APPNAME`, `_MAX_MSGID`.
|
||||||
|
|
||||||
|
### `logging/file_handler.py`
|
||||||
|
- `file_handler.py::write_syslog` — write a single RFC 5424 line to the rotating log file.
|
||||||
|
- `file_handler.py::get_log_path` — return the configured log file path.
|
||||||
|
- `file_handler.py::_init_file_handler`, `_get_logger`. Consts: `_LOG_FILE_ENV`, `_DEFAULT_LOG_FILE`, `_MAX_BYTES`, `_BACKUP_COUNT`.
|
||||||
|
|
||||||
|
### `logging/inode_aware_handler.py`
|
||||||
|
- `inode_aware_handler.py::InodeAwareRotatingFileHandler` — RotatingFileHandler that detects external deletion/rotation (logrotate) and reopens the target. Methods: `_should_reopen`, `emit`.
|
||||||
|
|
||||||
|
### `logging/forwarder.py`
|
||||||
|
- `forwarder.py::parse_log_target` — parse `"ip:port"` into `(host, port)`.
|
||||||
|
- `forwarder.py::probe_log_target` — TCP-connect probe for reachability.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## See also
|
||||||
|
|
||||||
|
- [Design overview](Design-Overview)
|
||||||
|
- [REST API](REST-API-Reference)
|
||||||
|
- [Logging](Logging-and-Syslog)
|
||||||
|
- [Services catalog](Services-Catalog)
|
||||||
|
- [Developer guide](Developer-Guide)
|
||||||
Reference in New Issue
Block a user