diff --git a/Module-Reference-Core.md b/Module-Reference-Core.md new file mode 100644 index 0000000..3799a1d --- /dev/null +++ b/Module-Reference-Core.md @@ -0,0 +1,218 @@ +# Module Reference — Core + +Every Python module in `decnet/` (top level, excluding sub-packages). This is the code view. For user-facing material see [Design overview](Design-Overview), [REST API](REST-API-Reference), and [Logging](Logging-and-Syslog). + +Citation format used throughout: `decnet/.py::`. + +--- + +## `decnet/__init__.py` + +Empty package marker. No exports. + +--- + +## `decnet/cli.py` + +Typer-based command-line entry point for DECNET. Defines every `decnet ` subcommand, process-group control for uvicorn workers, double-fork daemonisation, and a microservice registry used by `decnet status` / `decnet redeploy` to check and relaunch the supervisor processes (Collector, Mutator, Prober, Profiler, Sniffer, API). + +- `decnet/cli.py::app` — the root `typer.Typer` instance registered by the `decnet` console script. +- `decnet/cli.py::console` — module-level `rich.console.Console` used for all user-facing output. +- `decnet/cli.py::_daemonize` — Unix double-fork that redirects stdio to `/dev/null`; called by every command that accepts `--daemon`. +- `decnet/cli.py::_kill_all_services` — iterates the service registry and sends SIGTERM to every running DECNET microservice PID; invoked by `teardown --all`. +- `decnet/cli.py::api` — starts uvicorn for `decnet.web.api:app` in a new session group so Ctrl+C can tear down the worker tree. +- `decnet/cli.py::deploy` — main deploy command. Resolves interface/subnet/IP allocation, optionally parses an INI, builds the `DecnetConfig`, invokes `decnet.engine.deploy`, then spawns the mutator/collector/prober/profiler/sniffer/api children as requested. +- `decnet/cli.py::probe` — runs `decnet.prober.prober_worker` (JARM/HASSH/TCP fingerprint loop). +- `decnet/cli.py::collect` — runs `decnet.collector.log_collector_worker` to stream Docker container logs to the RFC 5424 file. +- `decnet/cli.py::mutate` — trigger mutation manually (`--decky`, `--all`) or in watch mode (`--watch`). +- `decnet/cli.py::status` — prints decky table via `decnet.engine.status` plus the DECNET-service table. +- `decnet/cli.py::teardown` — stops containers (`--all` / `--id`) and kills services when `--all`. +- `decnet/cli.py::list_services` — `decnet services` — prints registered service plugins. +- `decnet/cli.py::list_distros` — `decnet distros` — prints distro profiles. +- `decnet/cli.py::list_archetypes` — `decnet archetypes` — prints archetype profiles. +- `decnet/cli.py::correlate` — runs `CorrelationEngine` over a log file or stdin; emits table/json/syslog. +- `decnet/cli.py::redeploy` — health-checks each service in the registry and relaunches any that are down. +- `decnet/cli.py::serve_web` — `decnet web` — serves the Vite `dist/` build and reverse-proxies `/api/*` to the API port (with SSE-aware `read1()`/disabled socket timeout). +- `decnet/cli.py::profiler_cmd` — standalone `decnet profiler` worker. +- `decnet/cli.py::sniffer_cmd` — standalone `decnet sniffer` worker. +- `decnet/cli.py::db_reset` — `decnet db-reset` — MySQL-only destructive wipe (dry-run unless `--i-know-what-im-doing`). +- `decnet/cli.py::_db_reset_mysql_async` — coroutine that inspects row counts and optionally TRUNCATEs or DROPs the DECNET tables. Extracted from the CLI wrapper so tests can drive it without Typer. +- `decnet/cli.py::_is_running` — scans `psutil.process_iter` for a cmdline matching a predicate; returns PID or None. +- `decnet/cli.py::_service_registry` — returns the authoritative `(name, match_fn, launch_args)` triples used for status/redeploy. +- `decnet/cli.py::_DB_RESET_TABLES` — drop order tuple for the MySQL reset; `attacker_behavior` must be dropped before `attackers` because of the FK. + +--- + +## `decnet/models.py` + +Centralised Pydantic domain models — no web or DB imports. Used by the CLI, the INI loader, the fleet builder and the web API alike so that core logic stays isolated from adapters. + +- `decnet/models.py::validate_ini_string` — structural validator for INI text (≤ 512 KB, non-empty, must parse and contain at least one section). Raises `ValueError` with phrasing the tests assert against. +- `decnet/models.py::IniContent` — `Annotated[str, BeforeValidator(validate_ini_string)]` alias for fields that accept raw INI. +- `decnet/models.py::DeckySpec` — strict INI spec for a single decky (`name`, `ip`, `services`, `archetype`, `service_config`, `nmap_os`, `mutate_interval`). +- `decnet/models.py::CustomServiceSpec` — `[custom-*]` INI section (`name`, `image`, `exec_cmd`, `ports`). +- `decnet/models.py::IniConfig` — the parsed INI as a whole; enforces `at_least_one_decky`. +- `decnet/models.py::DeckyConfig` — runtime deployment record for one decky (`base_image`, `build_base`, `hostname`, `nmap_os`, `mutate_interval`, `last_mutated`, `last_login_attempt`). +- `decnet/models.py::DecnetConfig` — root config for a whole fleet (`mode`, `interface`, `subnet`, `gateway`, `deckies`, `log_file`, `ipvlan`, `mutate_interval`). + +--- + +## `decnet/ini_loader.py` + +INI parser for DECNET deployment files. Two-pass: first collects decky sections and `[custom-*]` service definitions, then walks again to attach `[decky.service]` per-service persona subsections — including expanding into `group-01`, `group-02`, … when the INI uses `amount=N`. + +- `decnet/ini_loader.py::load_ini` — read and parse an INI file, returning `IniConfig`. +- `decnet/ini_loader.py::load_ini_from_string` — normalise CRLF/CR to LF, validate, parse, return `IniConfig`. Used by the web API `update-config` endpoint. +- `decnet/ini_loader.py::_parse_configparser` — shared logic that walks a `configparser.ConfigParser` and emits `IniConfig`; implements the two-pass strategy and the `amount=` fan-out. + +--- + +## `decnet/composer.py` + +Generates a `docker-compose.yml` dict from a `DecnetConfig`. Each decky has one base container that owns the MACVLAN IP and runs `sleep infinity`; every service container attaches via `network_mode: "service:"` so all services on that decky share a single externally-visible IP. Service containers' Docker logs are captured by the json-file driver (10 MB × 5 rotation) and streamed by the host-side collector — no bind mounts. + +- `decnet/composer.py::generate_compose` — build and return the compose dict (services, networks). Injects `get_os_sysctls(decky.nmap_os)` into the base container so its namespace matches the claimed TTL/TCP-option profile; adds `NET_ADMIN` for the sysctl writes; seeds `build.args.BASE_IMAGE` so per-service Dockerfiles render the correct distro. +- `decnet/composer.py::write_compose` — dump the dict to YAML at a path and return the path. +- `decnet/composer.py::_DOCKER_LOGGING` — the json-file rotation options (`max-size: 10m`, `max-file: 5`). + +--- + +## `decnet/network.py` + +Host networking primitives: interface/subnet autodetection, IP allocation inside a subnet (skipping reserved + host + gateway), Docker MACVLAN and IPvlan network create/remove, and the host-side `decnet_macvlan0` / `decnet_ipvlan0` shim interfaces that fix the MACVLAN hairpin problem so the host can reach deckies it just spawned. + +- `decnet/network.py::MACVLAN_NETWORK_NAME` — `"decnet_lan"` — Docker network name used everywhere. +- `decnet/network.py::HOST_MACVLAN_IFACE` / `HOST_IPVLAN_IFACE` — host-side shim interface names. +- `decnet/network.py::detect_interface` — parse `ip route show default` and return the outbound `dev`. +- `decnet/network.py::detect_subnet` — parse `ip addr show ` + default route; returns `(cidr, gateway)`. +- `decnet/network.py::get_host_ip` — extract the host's IPv4 on a given interface. +- `decnet/network.py::allocate_ips` — yield `count` IPs from a subnet, skipping net/broadcast/gateway/host and honouring `ip_start`. +- `decnet/network.py::create_macvlan_network` / `create_ipvlan_network` — create the Docker driver network (no-op if it already exists). +- `decnet/network.py::remove_macvlan_network` — remove the Docker network. +- `decnet/network.py::setup_host_macvlan` / `setup_host_ipvlan` — build the host shim interface, assign a /32, add route to the decky range. Idempotent. Requires root. +- `decnet/network.py::teardown_host_macvlan` / `teardown_host_ipvlan` — inverse of the above. +- `decnet/network.py::ips_to_range` — compute the tightest CIDR that covers a given list of IPs. Used for `--ip-range` on MACVLAN. +- `decnet/network.py::_require_root` / `_run` — internal helpers (euid check, `subprocess.run` wrapper). + +--- + +## `decnet/config.py` + +Installs the RFC 5424 root logger used by every DECNET process, defines the state-file persistence helpers (`decnet-state.json`), and re-exports `DeckyConfig`/`DecnetConfig` from `models.py`. Imported very early so that merely doing `import decnet.config` configures logging side-effectfully. + +- `decnet/config.py::Rfc5424Formatter` — logging formatter emitting `1 TS HOST APP PID MSGID - MSG` with microsecond UTC timestamp; honours `record.decnet_component` to override the APP-NAME field. +- `decnet/config.py::_configure_logging` — idempotent root-logger setup: StreamHandler (stderr) plus `InodeAwareRotatingFileHandler` to `DECNET_SYSTEM_LOGS`; skips the file handler under pytest; chowns the file back to the sudo-invoking user. +- `decnet/config.py::STATE_FILE` — `Path` pointing at `/decnet-state.json`. +- `decnet/config.py::DEFAULT_MUTATE_INTERVAL` — `30` minutes. +- `decnet/config.py::random_hostname` — thin re-export of `decnet.distros.random_hostname`. +- `decnet/config.py::save_state` / `load_state` / `clear_state` — persist the `DecnetConfig` + compose path across CLI invocations. +- `decnet/config.py::_SYSLOG_SEVERITY` / `_FACILITY_LOCAL0` — RFC 5424 §6.2.1 mapping tables used by the formatter. + +--- + +## `decnet/env.py` + +Environment-variable loader. Resolves `.env.local` first then `.env`, exposes every `DECNET_*` setting as a module-level constant, and refuses to boot when required vars are missing or set to known-bad defaults. See [Environment Variables](Environment-Variables) for the full catalogue. + +- `decnet/env.py::_port` — parse an env var as an integer port (1–65535); raises with a clear message. +- `decnet/env.py::_require_env` — required-var guard; rejects empty values and the known-bad set `{admin, secret, password, changeme, fallback-secret-key-change-me}`; enforces ≥ 32-byte `DECNET_JWT_SECRET` outside developer mode. Pytest is exempt. +- `decnet/env.py::DECNET_SYSTEM_LOGS` — system-log file path (default `decnet.system.log`). +- `decnet/env.py::DECNET_EMBED_PROFILER` / `DECNET_EMBED_SNIFFER` — opt-in flags to run those workers inside the API process instead of as standalone daemons. +- `decnet/env.py::DECNET_PROFILE_REQUESTS` / `DECNET_PROFILE_DIR` — Pyinstrument ASGI middleware toggle. +- `decnet/env.py::DECNET_API_HOST` / `DECNET_API_PORT` — bind address for the FastAPI app. +- `decnet/env.py::DECNET_JWT_SECRET` — HS256 secret for bearer tokens (required, ≥ 32 bytes outside dev). +- `decnet/env.py::DECNET_INGEST_LOG_FILE` — canonical syslog path (default `/var/log/decnet/decnet.log`). +- `decnet/env.py::DECNET_BATCH_SIZE` / `DECNET_BATCH_MAX_WAIT_MS` — ingester batch knobs. +- `decnet/env.py::DECNET_WEB_HOST` / `DECNET_WEB_PORT` — dashboard bind. +- `decnet/env.py::DECNET_ADMIN_USER` / `DECNET_ADMIN_PASSWORD` — bootstrap admin credentials. +- `decnet/env.py::DECNET_DEVELOPER` / `DECNET_DEVELOPER_TRACING` / `DECNET_OTEL_ENDPOINT` — dev-mode and OTEL knobs. +- `decnet/env.py::DECNET_DB_TYPE` / `DECNET_DB_URL` / `DECNET_DB_HOST` / `DECNET_DB_PORT` / `DECNET_DB_NAME` / `DECNET_DB_USER` / `DECNET_DB_PASSWORD` — database selection (see [Database Drivers](Database-Drivers)). +- `decnet/env.py::DECNET_CORS_ORIGINS` — parsed list; defaults to the configured web host/port. + +--- + +## `decnet/privdrop.py` + +Drops root ownership on files created under `sudo`. When the deploy path (which needs root for MACVLAN) writes log files, they would otherwise be owned by root and block a later non-root `decnet api` from appending. `SUDO_UID` / `SUDO_GID` are used to chown them back to the invoking user. + +- `decnet/privdrop.py::chown_to_invoking_user` — best-effort chown on a path. No-op when not root, not launched via sudo, or the path doesn't exist. Failures are swallowed. +- `decnet/privdrop.py::chown_tree_to_invoking_user` — recursive variant for freshly-created parent directories. +- `decnet/privdrop.py::_sudo_ids` — read `SUDO_UID`/`SUDO_GID` from the environment, return `(uid, gid)` or `None`. + +--- + +## `decnet/distros.py` + +Distro registry that backs the heterogeneous-network look. Each `DistroProfile` pairs a Docker image (used for the base/IP-holder container) with a `build_base` used as `FROM ${BASE_IMAGE}` inside service Dockerfiles. Non-Debian distros pin their `build_base` to `debian:bookworm-slim` because the service Dockerfiles assume `apt-get`. + +- `decnet/distros.py::DistroProfile` — frozen dataclass (`slug`, `image`, `display_name`, `hostname_style`, `build_base`). +- `decnet/distros.py::DISTROS` — registry of built-in profiles: `debian`, `ubuntu22`, `ubuntu20`, `rocky9`, `centos7`, `alpine`, `fedora`, `kali`, `arch`. +- `decnet/distros.py::random_hostname` — generate a plausible hostname; shape depends on the profile's `hostname_style` (`generic` → `SRV-WORD-NN`, `rhel` → `wordNN.localdomain`, `minimal` → `word-NN`, `rolling` → `word-word`). +- `decnet/distros.py::get_distro` — lookup by slug or raise `ValueError`. +- `decnet/distros.py::random_distro` — pick a profile at random. +- `decnet/distros.py::all_distros` — copy of the registry dict. + +--- + +## `decnet/custom_service.py` + +Runtime wrapper for `[custom-*]` INI sections. Instantiated by the CLI/INI path and registered via `register_custom_service()`; not part of auto-discovery. + +- `decnet/custom_service.py::CustomService` — `BaseService` subclass that emits a compose fragment for an arbitrary user image. Ports come from the INI; `exec_cmd` becomes the container command; `LOG_TARGET` is injected when present. +- `decnet/custom_service.py::CustomService.compose_fragment` — build the dict for compose, including `NODE_NAME` env var for log tagging. +- `decnet/custom_service.py::CustomService.dockerfile_context` — always `None` — custom services never build. + +--- + +## `decnet/os_fingerprint.py` + +Namespace-scoped Linux sysctls applied to each decky's base container so that TCP/IP stack behaviour matches the claimed nmap OS family. Primary discriminator is `net.ipv4.ip_default_ttl` (64 Linux/BSD, 128 Windows, 255 Cisco/embedded). Secondary: `tcp_timestamps`, `tcp_window_scaling`, `tcp_sack`, `tcp_ecn`, `ip_no_pmtu_disc`, `tcp_fin_timeout`, `tcp_syn_retries`, `icmp_ratelimit`, `icmp_ratemask`. + +- `decnet/os_fingerprint.py::OS_SYSCTLS` — profile dict for `linux`, `windows`, `bsd`, `embedded`, `cisco`. +- `decnet/os_fingerprint.py::get_os_sysctls` — return the dict for a slug, falling back to `linux` on unknown input. + +```python +def get_os_sysctls(nmap_os: str) -> dict[str, str]: + return dict(OS_SYSCTLS.get(nmap_os, OS_SYSCTLS[_DEFAULT_OS])) +``` + +- `decnet/os_fingerprint.py::all_os_families` — list of all registered slugs. +- `decnet/os_fingerprint.py::_REQUIRED_SYSCTLS` — frozenset used by tests to assert every profile carries the full key set. + +--- + +## `decnet/archetypes.py` + +Pre-packaged machine identities: a service list, preferred distro pool, and nmap OS family. Lets users say `archetype=windows-workstation` instead of hand-picking `smb, rdp`. + +- `decnet/archetypes.py::Archetype` — frozen dataclass (`slug`, `display_name`, `description`, `services`, `preferred_distros`, `nmap_os`). +- `decnet/archetypes.py::ARCHETYPES` — registry: `windows-workstation`, `windows-server`, `domain-controller`, `linux-server`, `web-server`, `database-server`, `mail-server`, `file-server`, `printer`, `iot-device`, `industrial-control`, `voip-server`, `monitoring-node`, `devops-host`, `deaddeck`. +- `decnet/archetypes.py::get_archetype` / `all_archetypes` / `random_archetype` — standard lookup trio. + +--- + +## `decnet/fleet.py` + +Shared builder functions for constructing a list of `DeckyConfig` from either CLI flags or a parsed `IniConfig`. Lives outside `cli.py` so that the web API and the mutator can import it without dragging Typer. + +- `decnet/fleet.py::all_service_names` — sorted list of registered services that are not `fleet_singleton`. +- `decnet/fleet.py::resolve_distros` — decide the distro slug per decky (explicit list → randomised → archetype pool → all distros round-robin). +- `decnet/fleet.py::build_deckies` — CLI-style builder. Picks services from `services_explicit`, archetype, or random (with a dedup set so the first 20 attempts avoid duplicate service combos). +- `decnet/fleet.py::build_deckies_from_ini` — INI-style builder. Honours explicit `ip=`, auto-allocates the rest from the subnet (skipping network/broadcast/gateway/host/other-explicit IPs), resolves archetypes, and cascades `mutate_interval` (CLI > decky > global). + +--- + +## `decnet/telemetry.py` + +OpenTelemetry integration that is strictly opt-in via `DECNET_DEVELOPER_TRACING=true`. When disabled, every public export is a zero-cost no-op: `@traced` returns the unwrapped function, `get_tracer()` returns a `_NoOpTracer`, the repository wrapper returns the original repo, and injection/extraction of trace context is a nop. + +- `decnet/telemetry.py::setup_tracing` — initialise the TracerProvider, install `FastAPIInstrumentor`, enable log↔trace correlation. Called once from the FastAPI lifespan. +- `decnet/telemetry.py::shutdown_tracing` — best-effort provider flush+shutdown. +- `decnet/telemetry.py::get_tracer` — `get_tracer("db")`, `get_tracer("ingester")`, … cached per component. +- `decnet/telemetry.py::traced` — decorator that wraps async or sync functions in a span. Three call forms: `@traced`, `@traced("name")`, `@traced(name="name")`. +- `decnet/telemetry.py::wrap_repository` — dynamic `__getattr__` proxy that wraps every async repository method in a `db.` span. +- `decnet/telemetry.py::inject_context` / `extract_context` — W3C trace-context propagation into/out of JSON log records so the collector → ingester → profiler pipeline shows up as one trace in Jaeger. `extract_context` pops `_trace` from the dict so it never reaches the DB. +- `decnet/telemetry.py::start_span_with_context` — helper to continue a trace from an extracted context. +- `decnet/telemetry.py::_init_provider` — lazy OTEL SDK import + TracerProvider construction with an OTLP gRPC exporter to `DECNET_OTEL_ENDPOINT`. +- `decnet/telemetry.py::_NoOpTracer` / `_NoOpSpan` — stand-ins used when tracing is disabled. +- `decnet/telemetry.py::_wrap` — internal async-aware wrapper used by `traced`.