wiki: add Module-Reference-Core.md
218
Module-Reference-Core.md
Normal file
218
Module-Reference-Core.md
Normal file
@@ -0,0 +1,218 @@
|
|||||||
|
# Module Reference — Core
|
||||||
|
|
||||||
|
Every Python module in `decnet/` (top level, excluding sub-packages). This is the code view. For user-facing material see [Design overview](Design-Overview), [REST API](REST-API-Reference), and [Logging](Logging-and-Syslog).
|
||||||
|
|
||||||
|
Citation format used throughout: `decnet/<module>.py::<symbol>`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## `decnet/__init__.py`
|
||||||
|
|
||||||
|
Empty package marker. No exports.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## `decnet/cli.py`
|
||||||
|
|
||||||
|
Typer-based command-line entry point for DECNET. Defines every `decnet <verb>` subcommand, process-group control for uvicorn workers, double-fork daemonisation, and a microservice registry used by `decnet status` / `decnet redeploy` to check and relaunch the supervisor processes (Collector, Mutator, Prober, Profiler, Sniffer, API).
|
||||||
|
|
||||||
|
- `decnet/cli.py::app` — the root `typer.Typer` instance registered by the `decnet` console script.
|
||||||
|
- `decnet/cli.py::console` — module-level `rich.console.Console` used for all user-facing output.
|
||||||
|
- `decnet/cli.py::_daemonize` — Unix double-fork that redirects stdio to `/dev/null`; called by every command that accepts `--daemon`.
|
||||||
|
- `decnet/cli.py::_kill_all_services` — iterates the service registry and sends SIGTERM to every running DECNET microservice PID; invoked by `teardown --all`.
|
||||||
|
- `decnet/cli.py::api` — starts uvicorn for `decnet.web.api:app` in a new session group so Ctrl+C can tear down the worker tree.
|
||||||
|
- `decnet/cli.py::deploy` — main deploy command. Resolves interface/subnet/IP allocation, optionally parses an INI, builds the `DecnetConfig`, invokes `decnet.engine.deploy`, then spawns the mutator/collector/prober/profiler/sniffer/api children as requested.
|
||||||
|
- `decnet/cli.py::probe` — runs `decnet.prober.prober_worker` (JARM/HASSH/TCP fingerprint loop).
|
||||||
|
- `decnet/cli.py::collect` — runs `decnet.collector.log_collector_worker` to stream Docker container logs to the RFC 5424 file.
|
||||||
|
- `decnet/cli.py::mutate` — trigger mutation manually (`--decky`, `--all`) or in watch mode (`--watch`).
|
||||||
|
- `decnet/cli.py::status` — prints decky table via `decnet.engine.status` plus the DECNET-service table.
|
||||||
|
- `decnet/cli.py::teardown` — stops containers (`--all` / `--id`) and kills services when `--all`.
|
||||||
|
- `decnet/cli.py::list_services` — `decnet services` — prints registered service plugins.
|
||||||
|
- `decnet/cli.py::list_distros` — `decnet distros` — prints distro profiles.
|
||||||
|
- `decnet/cli.py::list_archetypes` — `decnet archetypes` — prints archetype profiles.
|
||||||
|
- `decnet/cli.py::correlate` — runs `CorrelationEngine` over a log file or stdin; emits table/json/syslog.
|
||||||
|
- `decnet/cli.py::redeploy` — health-checks each service in the registry and relaunches any that are down.
|
||||||
|
- `decnet/cli.py::serve_web` — `decnet web` — serves the Vite `dist/` build and reverse-proxies `/api/*` to the API port (with SSE-aware `read1()`/disabled socket timeout).
|
||||||
|
- `decnet/cli.py::profiler_cmd` — standalone `decnet profiler` worker.
|
||||||
|
- `decnet/cli.py::sniffer_cmd` — standalone `decnet sniffer` worker.
|
||||||
|
- `decnet/cli.py::db_reset` — `decnet db-reset` — MySQL-only destructive wipe (dry-run unless `--i-know-what-im-doing`).
|
||||||
|
- `decnet/cli.py::_db_reset_mysql_async` — coroutine that inspects row counts and optionally TRUNCATEs or DROPs the DECNET tables. Extracted from the CLI wrapper so tests can drive it without Typer.
|
||||||
|
- `decnet/cli.py::_is_running` — scans `psutil.process_iter` for a cmdline matching a predicate; returns PID or None.
|
||||||
|
- `decnet/cli.py::_service_registry` — returns the authoritative `(name, match_fn, launch_args)` triples used for status/redeploy.
|
||||||
|
- `decnet/cli.py::_DB_RESET_TABLES` — drop order tuple for the MySQL reset; `attacker_behavior` must be dropped before `attackers` because of the FK.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## `decnet/models.py`
|
||||||
|
|
||||||
|
Centralised Pydantic domain models — no web or DB imports. Used by the CLI, the INI loader, the fleet builder and the web API alike so that core logic stays isolated from adapters.
|
||||||
|
|
||||||
|
- `decnet/models.py::validate_ini_string` — structural validator for INI text (≤ 512 KB, non-empty, must parse and contain at least one section). Raises `ValueError` with phrasing the tests assert against.
|
||||||
|
- `decnet/models.py::IniContent` — `Annotated[str, BeforeValidator(validate_ini_string)]` alias for fields that accept raw INI.
|
||||||
|
- `decnet/models.py::DeckySpec` — strict INI spec for a single decky (`name`, `ip`, `services`, `archetype`, `service_config`, `nmap_os`, `mutate_interval`).
|
||||||
|
- `decnet/models.py::CustomServiceSpec` — `[custom-*]` INI section (`name`, `image`, `exec_cmd`, `ports`).
|
||||||
|
- `decnet/models.py::IniConfig` — the parsed INI as a whole; enforces `at_least_one_decky`.
|
||||||
|
- `decnet/models.py::DeckyConfig` — runtime deployment record for one decky (`base_image`, `build_base`, `hostname`, `nmap_os`, `mutate_interval`, `last_mutated`, `last_login_attempt`).
|
||||||
|
- `decnet/models.py::DecnetConfig` — root config for a whole fleet (`mode`, `interface`, `subnet`, `gateway`, `deckies`, `log_file`, `ipvlan`, `mutate_interval`).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## `decnet/ini_loader.py`
|
||||||
|
|
||||||
|
INI parser for DECNET deployment files. Two-pass: first collects decky sections and `[custom-*]` service definitions, then walks again to attach `[decky.service]` per-service persona subsections — including expanding into `group-01`, `group-02`, … when the INI uses `amount=N`.
|
||||||
|
|
||||||
|
- `decnet/ini_loader.py::load_ini` — read and parse an INI file, returning `IniConfig`.
|
||||||
|
- `decnet/ini_loader.py::load_ini_from_string` — normalise CRLF/CR to LF, validate, parse, return `IniConfig`. Used by the web API `update-config` endpoint.
|
||||||
|
- `decnet/ini_loader.py::_parse_configparser` — shared logic that walks a `configparser.ConfigParser` and emits `IniConfig`; implements the two-pass strategy and the `amount=` fan-out.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## `decnet/composer.py`
|
||||||
|
|
||||||
|
Generates a `docker-compose.yml` dict from a `DecnetConfig`. Each decky has one base container that owns the MACVLAN IP and runs `sleep infinity`; every service container attaches via `network_mode: "service:<base>"` so all services on that decky share a single externally-visible IP. Service containers' Docker logs are captured by the json-file driver (10 MB × 5 rotation) and streamed by the host-side collector — no bind mounts.
|
||||||
|
|
||||||
|
- `decnet/composer.py::generate_compose` — build and return the compose dict (services, networks). Injects `get_os_sysctls(decky.nmap_os)` into the base container so its namespace matches the claimed TTL/TCP-option profile; adds `NET_ADMIN` for the sysctl writes; seeds `build.args.BASE_IMAGE` so per-service Dockerfiles render the correct distro.
|
||||||
|
- `decnet/composer.py::write_compose` — dump the dict to YAML at a path and return the path.
|
||||||
|
- `decnet/composer.py::_DOCKER_LOGGING` — the json-file rotation options (`max-size: 10m`, `max-file: 5`).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## `decnet/network.py`
|
||||||
|
|
||||||
|
Host networking primitives: interface/subnet autodetection, IP allocation inside a subnet (skipping reserved + host + gateway), Docker MACVLAN and IPvlan network create/remove, and the host-side `decnet_macvlan0` / `decnet_ipvlan0` shim interfaces that fix the MACVLAN hairpin problem so the host can reach deckies it just spawned.
|
||||||
|
|
||||||
|
- `decnet/network.py::MACVLAN_NETWORK_NAME` — `"decnet_lan"` — Docker network name used everywhere.
|
||||||
|
- `decnet/network.py::HOST_MACVLAN_IFACE` / `HOST_IPVLAN_IFACE` — host-side shim interface names.
|
||||||
|
- `decnet/network.py::detect_interface` — parse `ip route show default` and return the outbound `dev`.
|
||||||
|
- `decnet/network.py::detect_subnet` — parse `ip addr show <iface>` + default route; returns `(cidr, gateway)`.
|
||||||
|
- `decnet/network.py::get_host_ip` — extract the host's IPv4 on a given interface.
|
||||||
|
- `decnet/network.py::allocate_ips` — yield `count` IPs from a subnet, skipping net/broadcast/gateway/host and honouring `ip_start`.
|
||||||
|
- `decnet/network.py::create_macvlan_network` / `create_ipvlan_network` — create the Docker driver network (no-op if it already exists).
|
||||||
|
- `decnet/network.py::remove_macvlan_network` — remove the Docker network.
|
||||||
|
- `decnet/network.py::setup_host_macvlan` / `setup_host_ipvlan` — build the host shim interface, assign a /32, add route to the decky range. Idempotent. Requires root.
|
||||||
|
- `decnet/network.py::teardown_host_macvlan` / `teardown_host_ipvlan` — inverse of the above.
|
||||||
|
- `decnet/network.py::ips_to_range` — compute the tightest CIDR that covers a given list of IPs. Used for `--ip-range` on MACVLAN.
|
||||||
|
- `decnet/network.py::_require_root` / `_run` — internal helpers (euid check, `subprocess.run` wrapper).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## `decnet/config.py`
|
||||||
|
|
||||||
|
Installs the RFC 5424 root logger used by every DECNET process, defines the state-file persistence helpers (`decnet-state.json`), and re-exports `DeckyConfig`/`DecnetConfig` from `models.py`. Imported very early so that merely doing `import decnet.config` configures logging side-effectfully.
|
||||||
|
|
||||||
|
- `decnet/config.py::Rfc5424Formatter` — logging formatter emitting `<PRI>1 TS HOST APP PID MSGID - MSG` with microsecond UTC timestamp; honours `record.decnet_component` to override the APP-NAME field.
|
||||||
|
- `decnet/config.py::_configure_logging` — idempotent root-logger setup: StreamHandler (stderr) plus `InodeAwareRotatingFileHandler` to `DECNET_SYSTEM_LOGS`; skips the file handler under pytest; chowns the file back to the sudo-invoking user.
|
||||||
|
- `decnet/config.py::STATE_FILE` — `Path` pointing at `<repo>/decnet-state.json`.
|
||||||
|
- `decnet/config.py::DEFAULT_MUTATE_INTERVAL` — `30` minutes.
|
||||||
|
- `decnet/config.py::random_hostname` — thin re-export of `decnet.distros.random_hostname`.
|
||||||
|
- `decnet/config.py::save_state` / `load_state` / `clear_state` — persist the `DecnetConfig` + compose path across CLI invocations.
|
||||||
|
- `decnet/config.py::_SYSLOG_SEVERITY` / `_FACILITY_LOCAL0` — RFC 5424 §6.2.1 mapping tables used by the formatter.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## `decnet/env.py`
|
||||||
|
|
||||||
|
Environment-variable loader. Resolves `.env.local` first then `.env`, exposes every `DECNET_*` setting as a module-level constant, and refuses to boot when required vars are missing or set to known-bad defaults. See [Environment Variables](Environment-Variables) for the full catalogue.
|
||||||
|
|
||||||
|
- `decnet/env.py::_port` — parse an env var as an integer port (1–65535); raises with a clear message.
|
||||||
|
- `decnet/env.py::_require_env` — required-var guard; rejects empty values and the known-bad set `{admin, secret, password, changeme, fallback-secret-key-change-me}`; enforces ≥ 32-byte `DECNET_JWT_SECRET` outside developer mode. Pytest is exempt.
|
||||||
|
- `decnet/env.py::DECNET_SYSTEM_LOGS` — system-log file path (default `decnet.system.log`).
|
||||||
|
- `decnet/env.py::DECNET_EMBED_PROFILER` / `DECNET_EMBED_SNIFFER` — opt-in flags to run those workers inside the API process instead of as standalone daemons.
|
||||||
|
- `decnet/env.py::DECNET_PROFILE_REQUESTS` / `DECNET_PROFILE_DIR` — Pyinstrument ASGI middleware toggle.
|
||||||
|
- `decnet/env.py::DECNET_API_HOST` / `DECNET_API_PORT` — bind address for the FastAPI app.
|
||||||
|
- `decnet/env.py::DECNET_JWT_SECRET` — HS256 secret for bearer tokens (required, ≥ 32 bytes outside dev).
|
||||||
|
- `decnet/env.py::DECNET_INGEST_LOG_FILE` — canonical syslog path (default `/var/log/decnet/decnet.log`).
|
||||||
|
- `decnet/env.py::DECNET_BATCH_SIZE` / `DECNET_BATCH_MAX_WAIT_MS` — ingester batch knobs.
|
||||||
|
- `decnet/env.py::DECNET_WEB_HOST` / `DECNET_WEB_PORT` — dashboard bind.
|
||||||
|
- `decnet/env.py::DECNET_ADMIN_USER` / `DECNET_ADMIN_PASSWORD` — bootstrap admin credentials.
|
||||||
|
- `decnet/env.py::DECNET_DEVELOPER` / `DECNET_DEVELOPER_TRACING` / `DECNET_OTEL_ENDPOINT` — dev-mode and OTEL knobs.
|
||||||
|
- `decnet/env.py::DECNET_DB_TYPE` / `DECNET_DB_URL` / `DECNET_DB_HOST` / `DECNET_DB_PORT` / `DECNET_DB_NAME` / `DECNET_DB_USER` / `DECNET_DB_PASSWORD` — database selection (see [Database Drivers](Database-Drivers)).
|
||||||
|
- `decnet/env.py::DECNET_CORS_ORIGINS` — parsed list; defaults to the configured web host/port.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## `decnet/privdrop.py`
|
||||||
|
|
||||||
|
Drops root ownership on files created under `sudo`. When the deploy path (which needs root for MACVLAN) writes log files, they would otherwise be owned by root and block a later non-root `decnet api` from appending. `SUDO_UID` / `SUDO_GID` are used to chown them back to the invoking user.
|
||||||
|
|
||||||
|
- `decnet/privdrop.py::chown_to_invoking_user` — best-effort chown on a path. No-op when not root, not launched via sudo, or the path doesn't exist. Failures are swallowed.
|
||||||
|
- `decnet/privdrop.py::chown_tree_to_invoking_user` — recursive variant for freshly-created parent directories.
|
||||||
|
- `decnet/privdrop.py::_sudo_ids` — read `SUDO_UID`/`SUDO_GID` from the environment, return `(uid, gid)` or `None`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## `decnet/distros.py`
|
||||||
|
|
||||||
|
Distro registry that backs the heterogeneous-network look. Each `DistroProfile` pairs a Docker image (used for the base/IP-holder container) with a `build_base` used as `FROM ${BASE_IMAGE}` inside service Dockerfiles. Non-Debian distros pin their `build_base` to `debian:bookworm-slim` because the service Dockerfiles assume `apt-get`.
|
||||||
|
|
||||||
|
- `decnet/distros.py::DistroProfile` — frozen dataclass (`slug`, `image`, `display_name`, `hostname_style`, `build_base`).
|
||||||
|
- `decnet/distros.py::DISTROS` — registry of built-in profiles: `debian`, `ubuntu22`, `ubuntu20`, `rocky9`, `centos7`, `alpine`, `fedora`, `kali`, `arch`.
|
||||||
|
- `decnet/distros.py::random_hostname` — generate a plausible hostname; shape depends on the profile's `hostname_style` (`generic` → `SRV-WORD-NN`, `rhel` → `wordNN.localdomain`, `minimal` → `word-NN`, `rolling` → `word-word`).
|
||||||
|
- `decnet/distros.py::get_distro` — lookup by slug or raise `ValueError`.
|
||||||
|
- `decnet/distros.py::random_distro` — pick a profile at random.
|
||||||
|
- `decnet/distros.py::all_distros` — copy of the registry dict.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## `decnet/custom_service.py`
|
||||||
|
|
||||||
|
Runtime wrapper for `[custom-*]` INI sections. Instantiated by the CLI/INI path and registered via `register_custom_service()`; not part of auto-discovery.
|
||||||
|
|
||||||
|
- `decnet/custom_service.py::CustomService` — `BaseService` subclass that emits a compose fragment for an arbitrary user image. Ports come from the INI; `exec_cmd` becomes the container command; `LOG_TARGET` is injected when present.
|
||||||
|
- `decnet/custom_service.py::CustomService.compose_fragment` — build the dict for compose, including `NODE_NAME` env var for log tagging.
|
||||||
|
- `decnet/custom_service.py::CustomService.dockerfile_context` — always `None` — custom services never build.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## `decnet/os_fingerprint.py`
|
||||||
|
|
||||||
|
Namespace-scoped Linux sysctls applied to each decky's base container so that TCP/IP stack behaviour matches the claimed nmap OS family. Primary discriminator is `net.ipv4.ip_default_ttl` (64 Linux/BSD, 128 Windows, 255 Cisco/embedded). Secondary: `tcp_timestamps`, `tcp_window_scaling`, `tcp_sack`, `tcp_ecn`, `ip_no_pmtu_disc`, `tcp_fin_timeout`, `tcp_syn_retries`, `icmp_ratelimit`, `icmp_ratemask`.
|
||||||
|
|
||||||
|
- `decnet/os_fingerprint.py::OS_SYSCTLS` — profile dict for `linux`, `windows`, `bsd`, `embedded`, `cisco`.
|
||||||
|
- `decnet/os_fingerprint.py::get_os_sysctls` — return the dict for a slug, falling back to `linux` on unknown input.
|
||||||
|
|
||||||
|
```python
|
||||||
|
def get_os_sysctls(nmap_os: str) -> dict[str, str]:
|
||||||
|
return dict(OS_SYSCTLS.get(nmap_os, OS_SYSCTLS[_DEFAULT_OS]))
|
||||||
|
```
|
||||||
|
|
||||||
|
- `decnet/os_fingerprint.py::all_os_families` — list of all registered slugs.
|
||||||
|
- `decnet/os_fingerprint.py::_REQUIRED_SYSCTLS` — frozenset used by tests to assert every profile carries the full key set.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## `decnet/archetypes.py`
|
||||||
|
|
||||||
|
Pre-packaged machine identities: a service list, preferred distro pool, and nmap OS family. Lets users say `archetype=windows-workstation` instead of hand-picking `smb, rdp`.
|
||||||
|
|
||||||
|
- `decnet/archetypes.py::Archetype` — frozen dataclass (`slug`, `display_name`, `description`, `services`, `preferred_distros`, `nmap_os`).
|
||||||
|
- `decnet/archetypes.py::ARCHETYPES` — registry: `windows-workstation`, `windows-server`, `domain-controller`, `linux-server`, `web-server`, `database-server`, `mail-server`, `file-server`, `printer`, `iot-device`, `industrial-control`, `voip-server`, `monitoring-node`, `devops-host`, `deaddeck`.
|
||||||
|
- `decnet/archetypes.py::get_archetype` / `all_archetypes` / `random_archetype` — standard lookup trio.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## `decnet/fleet.py`
|
||||||
|
|
||||||
|
Shared builder functions for constructing a list of `DeckyConfig` from either CLI flags or a parsed `IniConfig`. Lives outside `cli.py` so that the web API and the mutator can import it without dragging Typer.
|
||||||
|
|
||||||
|
- `decnet/fleet.py::all_service_names` — sorted list of registered services that are not `fleet_singleton`.
|
||||||
|
- `decnet/fleet.py::resolve_distros` — decide the distro slug per decky (explicit list → randomised → archetype pool → all distros round-robin).
|
||||||
|
- `decnet/fleet.py::build_deckies` — CLI-style builder. Picks services from `services_explicit`, archetype, or random (with a dedup set so the first 20 attempts avoid duplicate service combos).
|
||||||
|
- `decnet/fleet.py::build_deckies_from_ini` — INI-style builder. Honours explicit `ip=`, auto-allocates the rest from the subnet (skipping network/broadcast/gateway/host/other-explicit IPs), resolves archetypes, and cascades `mutate_interval` (CLI > decky > global).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## `decnet/telemetry.py`
|
||||||
|
|
||||||
|
OpenTelemetry integration that is strictly opt-in via `DECNET_DEVELOPER_TRACING=true`. When disabled, every public export is a zero-cost no-op: `@traced` returns the unwrapped function, `get_tracer()` returns a `_NoOpTracer`, the repository wrapper returns the original repo, and injection/extraction of trace context is a nop.
|
||||||
|
|
||||||
|
- `decnet/telemetry.py::setup_tracing` — initialise the TracerProvider, install `FastAPIInstrumentor`, enable log↔trace correlation. Called once from the FastAPI lifespan.
|
||||||
|
- `decnet/telemetry.py::shutdown_tracing` — best-effort provider flush+shutdown.
|
||||||
|
- `decnet/telemetry.py::get_tracer` — `get_tracer("db")`, `get_tracer("ingester")`, … cached per component.
|
||||||
|
- `decnet/telemetry.py::traced` — decorator that wraps async or sync functions in a span. Three call forms: `@traced`, `@traced("name")`, `@traced(name="name")`.
|
||||||
|
- `decnet/telemetry.py::wrap_repository` — dynamic `__getattr__` proxy that wraps every async repository method in a `db.<method>` span.
|
||||||
|
- `decnet/telemetry.py::inject_context` / `extract_context` — W3C trace-context propagation into/out of JSON log records so the collector → ingester → profiler pipeline shows up as one trace in Jaeger. `extract_context` pops `_trace` from the dict so it never reaches the DB.
|
||||||
|
- `decnet/telemetry.py::start_span_with_context` — helper to continue a trace from an extracted context.
|
||||||
|
- `decnet/telemetry.py::_init_provider` — lazy OTEL SDK import + TracerProvider construction with an OTLP gRPC exporter to `DECNET_OTEL_ENDPOINT`.
|
||||||
|
- `decnet/telemetry.py::_NoOpTracer` / `_NoOpSpan` — stand-ins used when tracing is disabled.
|
||||||
|
- `decnet/telemetry.py::_wrap` — internal async-aware wrapper used by `traced`.
|
||||||
Reference in New Issue
Block a user