diff --git a/Custom-Services.md b/Custom-Services.md new file mode 100644 index 0000000..70455ef --- /dev/null +++ b/Custom-Services.md @@ -0,0 +1,132 @@ +# Custom Services (Bring-Your-Own) + +DECNET ships 25+ first-class service plugins (see +[Services catalog](Services-Catalog)), but you can add your own **without +writing a plugin** by declaring a `[custom-]` section in the INI. + +The source of truth is `decnet/custom_service.py` (the `CustomService` +class) and the `[custom-*]` branch of `decnet/ini_loader.py`. + +--- + +## How it works + +At INI load time the parser scans every section starting with `custom-`: + +```python +# decnet/ini_loader.py +if section.startswith("custom-"): + s = cp[section] + svc_name = section[len("custom-"):] + image = s.get("binary", "") + exec_cmd = s.get("exec", "") + ports_raw = s.get("ports", "") + ports = [int(p.strip()) for p in ports_raw.split(",") if p.strip().isdigit()] + cfg.custom_services.append( + CustomServiceSpec(name=svc_name, image=image, exec_cmd=exec_cmd, ports=ports) + ) +``` + +Each resulting `CustomServiceSpec` is wrapped into a `CustomService` +(subclass of `BaseService`) and registered dynamically at deploy time +(`register_custom_service()` in the service registry). Once registered, the +custom slug behaves like any built-in service — you can list it in a decky's +`services=` line and reference it in the `--services` CLI flag. + +`CustomService.compose_fragment()` produces a Docker Compose fragment of the +shape: + +```yaml +image: +container_name: - +restart: unless-stopped +environment: + NODE_NAME: + LOG_TARGET: +command: [] # only emitted if exec= is non-empty +``` + +Underscores in the custom name are converted to dashes in the container +suffix (`slug = name.replace("_", "-")`). + +Note: `CustomService.dockerfile_context()` returns `None`, so DECNET will +**not** try to build the image. The image you name in `binary=` must be +pullable (public registry, or present in the local Docker daemon). + +--- + +## Accepted keys + +| Key | Required | Meaning | +|----------|----------|---------| +| `binary` | yes | Docker image reference, e.g. `myorg/weirdapp:1.2.3`. Becomes the container's `image:`. | +| `exec` | no | Command to run inside the container. Space-split into a list and emitted as Compose `command:`. If omitted, the image's default `CMD`/`ENTRYPOINT` is used. | +| `ports` | no | Comma-separated list of integer ports. Non-numeric tokens are silently dropped. Used by the deploy layer to expose/route the ports. | + +There are no other keys. Anything else in the section is ignored by +`ini_loader.py`. + +The custom service slug (the part after `custom-`) must not collide with a +built-in service name. + +--- + +## Minimal working example + +Ship a lightweight HTTP echo server as a decoy on port 9000: + +```ini +[general] +net = 192.168.1.0/24 +gw = 192.168.1.1 +interface = eth0 + +# Define the custom service +[custom-echoweb] +binary = ealen/echo-server:0.9.2 +exec = node index.js +ports = 80 + +# Use it on a decky, alongside a built-in service +[decky-app01] +ip = 192.168.1.130 +services = ssh, echoweb +nmap_os = linux + +[decky-app01.ssh] +ssh_version = OpenSSH_8.9p1 Ubuntu-3ubuntu0.6 +kernel_version = 5.15.0-91-generic +users = root:toor +``` + +Deploy: + +```bash +sudo decnet deploy --config echo.ini --interface eth0 \ + --log-target 192.168.1.200:5140 +``` + +At deploy time, `decky-app01` will get two containers: a standard `ssh` +decoy and a `decky-app01-echoweb` container running +`ealen/echo-server:0.9.2` with `command: ["node", "index.js"]`. + +--- + +## Caveats + +- No per-service persona subsection is honoured for custom services — the + `service_cfg` argument in `CustomService.compose_fragment()` is accepted + but ignored. If you need persona/override behaviour, write a real plugin. +- No Dockerfile build step. `binary=` is pulled as-is. +- If `exec=` is empty, no `command:` key is emitted — the image's default + entrypoint runs. +- `LOG_TARGET` is injected into the container env **only** if a CLI + `--log-target` was passed to `decnet deploy`; otherwise the custom service + is responsible for its own logging. + +--- + +See also: +- [INI Config Format](INI-Config-Format) +- [Services catalog](Services-Catalog) +- [Service personas](Service-Personas) diff --git a/Database-Drivers.md b/Database-Drivers.md new file mode 100644 index 0000000..824d2dc --- /dev/null +++ b/Database-Drivers.md @@ -0,0 +1,123 @@ +# Database Drivers + +DECNET persists dashboard state, deployment metadata, event/capture rows, and auth +data through a single `BaseRepository` interface. Two concrete backends implement +that interface: SQLite (default) and MySQL. Both extend the same +`SQLModelRepository` base and only override the dialect-specific bits (engine +construction, URL resolution, schema bootstrap). This is a load-bearing design +choice — adding a new backend means subclassing the shared repo, not reimplementing +the query surface. + +Source: `decnet/web/db/repository.py`, `decnet/web/db/sqlmodel_repo.py`, +`decnet/web/db/sqlite/repository.py`, `decnet/web/db/mysql/repository.py`. + +## Why two backends + +- **SQLite** fits single-host UNIHOST deploys. Zero setup, file-backed, + limited to one writer at a time. +- **MySQL** fits SWARM and any deployment with multi-process ingest or high + write volume (many sniffer/capture workers pushing concurrently). Scales + horizontally and survives multi-writer contention that SQLite cannot. + +The wire format, ORM models, and repository API are identical — only the engine +and connection URL change. + +## SQLite (default) + +Driver: `aiosqlite >= 0.20.0` (see `pyproject.toml`). Pulled in automatically +by `pip install -e .` — nothing else to install. + +Behavior: + +- WAL journal mode and `synchronous=NORMAL` are enabled on every connect + (`decnet/web/db/sqlite/database.py`). +- Default `busy_timeout` is 30s to absorb short write contention. +- The DB file lives at the path passed to `SQLiteRepository(db_path=...)` + (defaults set by the caller; repo bootstrap will `create_all` on first use). +- WAL sidecar files (`*-wal`, `*-shm`) are expected and are ignored by + `.gitignore`. + +Limits: + +- Single-writer. Concurrent writes serialize; under heavy load you will see + `SQLITE_BUSY`. Raise `DECNET_DB_POOL_SIZE` / timeouts only as a stopgap — + switch to MySQL if contention is real. +- Not safe to share across hosts over NFS or similar. + +## MySQL + +Driver: `asyncmy >= 0.2.9` (see `pyproject.toml`). Installed by +`pip install -e .`. No separate package step. + +### Setup + +Create the database and a dedicated user: + +```sql +CREATE DATABASE decnet CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci; +CREATE USER 'decnet'@'%' IDENTIFIED BY 'change-me'; +GRANT ALL PRIVILEGES ON decnet.* TO 'decnet'@'%'; +FLUSH PRIVILEGES; +``` + +Point DECNET at it with either a full URL or component vars (see +[Environment variables](Environment-Variables) for the complete list): + +```bash +# Option A — full URL +export DECNET_DB_TYPE=mysql +export DECNET_DB_URL="mysql+asyncmy://decnet:change-me@db.internal:3306/decnet" + +# Option B — components (password is percent-encoded automatically) +export DECNET_DB_TYPE=mysql +export DECNET_DB_HOST=db.internal +export DECNET_DB_PORT=3306 +export DECNET_DB_NAME=decnet +export DECNET_DB_USER=decnet +export DECNET_DB_PASSWORD='change-me' +``` + +Precedence: explicit `url=` kwarg to `get_async_engine` > `DECNET_DB_URL` > +components. An empty password outside pytest raises — this is intentional. + +Schema bootstrap: tables are created via `SQLModel.metadata.create_all` on the +first repository initialization. There is no migrator. + +### Tuning + +Pool sizing and recycle knobs (`DECNET_DB_POOL_SIZE`, `DECNET_DB_MAX_OVERFLOW`, +`DECNET_DB_POOL_RECYCLE`, `DECNET_DB_POOL_PRE_PING`) apply to both backends +and are documented on [Environment variables](Environment-Variables). MySQL +defaults `pool_pre_ping=true` to fail fast on dropped idle connections; SQLite +leaves it off since the "server" is a local file. + +## Switching backends + +Set `DECNET_DB_TYPE` to `sqlite` or `mysql` and restart the web/CLI process. +There is **no migration tool** — switching backends starts from an empty +schema. Export anything you care about beforehand. For MySQL teardown / +rebuild during development, `decnet db-reset` exists (MySQL-only; see +`decnet/cli.py`). + +## Factory and DI + +All code paths that need a repository must go through the factory: + +```python +from decnet.web.db.factory import get_repository + +repo = get_repository() # DECNET_DB_TYPE decides the subclass +``` + +In FastAPI handlers, depend on `get_repo` from `decnet/web/dependencies.py` +instead of constructing a repo inline. The factory also wraps the instance +with telemetry (`decnet.telemetry.wrap_repository`), so bypassing it loses +metrics. + +**Rule (from `CLAUDE.md`):** never import `SQLiteRepository` (or +`MySQLRepository`) directly in feature code. Use `get_repository()` or the +`get_repo` FastAPI dependency. Direct imports break backend switching and +silently drop telemetry wrapping. + +See the [Developer guide](Developer-Guide) for how repository-bound services +are wired into handlers and background workers. diff --git a/INI-Config-Format.md b/INI-Config-Format.md new file mode 100644 index 0000000..5d83dd5 --- /dev/null +++ b/INI-Config-Format.md @@ -0,0 +1,255 @@ +# INI Config Format + +DECNET deployments are described by an INI file passed via `--config`. The file +is parsed by `decnet/ini_loader.py` (structural validation in +`decnet/models.py::validate_ini_string`) and converted into an `IniConfig` of +`DeckySpec` and `CustomServiceSpec` objects. + +This page is an exhaustive reference. **Every key documented here is accepted +by the parser.** Keys not listed are ignored (or cause a validation error for +typed fields). + +Related pages: +- [Services catalog](Services-Catalog) +- [Archetypes](Archetypes) +- [Service personas](Service-Personas) +- [Custom services](Custom-Services) + +--- + +## Section types + +A DECNET INI file contains four kinds of sections: + +1. `[general]` — fleet-wide network settings (optional, but recommended). +2. `[]` — one decky (or a group of deckies if `amount=N`). +3. `[.]` — persona/override config for one service on + one decky (or group). +4. `[custom-]` — a bring-your-own service definition. See + [Custom services](Custom-Services). + +Section names must match `^[A-Za-z0-9\-_.]+$` (enforced on the decky `name`). + +### How the parser distinguishes a service subsection from a decky + +A section with a dot in its name is treated as a **per-service subsection** +if and only if the last dotted segment is a **known service name** registered +in `decnet.services.registry.all_services()`. Otherwise the whole section +name is treated as a decky name. + +Example: `[decky-01.ssh]` → subsection (because `ssh` is a registered service). +`[decky.webmail]` → decky section (because `webmail` is not a registered +service). + +--- + +## `[general]` section + +Fleet-wide settings. All keys optional. + +| Key | Type | Meaning | +|-------------|--------|------------------------------------------------------| +| `net` | string | LAN CIDR for the decoy network, e.g. `192.168.1.0/24`. Maps to `IniConfig.subnet`. | +| `gw` | string | LAN gateway IP. Maps to `IniConfig.gateway`. | +| `interface` | string | Host interface to bind MACVLAN/IPVLAN to, e.g. `eth0`, `wlp6s0`. | + +> **Note:** `log_target` is **not** an INI key. It is a CLI flag +> (`--log-target HOST:PORT`). You may leave a commented `#log_target=` reminder +> in your INI, but the parser will not read it. + +Example: + +```ini +[general] +net = 192.168.1.0/24 +gw = 192.168.1.1 +interface = eth0 +``` + +--- + +## Decky sections + +Each non-general, non-`custom-*` section that is not a service subsection +defines a decky (or a group). The section name becomes the decky's `name`. + +| Key | Type | Default | Meaning | +|--------------------|----------|---------|---------| +| `ip` | string | auto | Static IP inside the subnet. **Forbidden when `amount>1`** — raises `ValueError`. | +| `services` | csv list | `None` | Comma-separated service slugs. If omitted, falls back to the `--randomize-services` pool or the archetype's service list. | +| `archetype` | string | `None` | Archetype slug (e.g. `linux-server`, `windows-workstation`). Archetypes pre-seed services, distro, and `nmap_os`. See [Archetypes](Archetypes). | +| `amount` | int | `1` | Number of deckies to spawn from this section. Must be `1..100`. When `>1`, the section acts as a template and deckies are named `
-01`, `
-02`, …. | +| `nmap_os` | string | `linux` | TCP/IP stack family for fingerprint spoofing. Accepted: `linux`, `windows`, `embedded`, `bsd`, `cisco`. Also accepted as `nmap-os` (alias). | +| `mutate_interval` | int | global | Auto-rotation interval in minutes (`>=1`). Also accepted as `mutate-interval` (alias). | + +### `amount` expansion rules + +```ini +[corp-workstations] +archetype = windows-workstation +amount = 5 +``` + +Expands to `corp-workstations-01` … `corp-workstations-05`. Any +`[corp-workstations.]` subsection propagates to **all five** +expanded deckies (see Resolution order below). + +Setting `ip=` together with `amount>1` is rejected — one IP cannot be shared. + +--- + +## Per-service subsections `[.]` + +Persona overrides for a single service on one decky (or a group). The last +dotted segment must be a registered service slug — otherwise the subsection +is ignored. + +Every key inside such a subsection is passed through verbatim as a `dict` +into `DeckySpec.service_config[]` and then consumed by the service +plugin's `compose_fragment(..., service_cfg=...)`. The available keys are +therefore **service-specific** — see [Service personas](Service-Personas) for +the full per-service key list. + +Examples (excerpt from `development/test-full.ini`): + +```ini +[decky-webmail.http] +server_header = Apache/2.4.54 (Debian) +response_code = 200 +fake_app = wordpress + +[decky-webmail.smtp] +smtp_banner = 220 mail.corp.local ESMTP Postfix (Debian/GNU) +smtp_mta = mail.corp.local + +[decky-ldapdc.ssh] +ssh_version = OpenSSH_8.9p1 Ubuntu-3ubuntu0.6 +kernel_version = 5.15.0-91-generic +users = root:toor,admin:admin123,svc_backup:backup2024 +``` + +--- + +## Resolution order + +When the runtime configuration for a decky's service is built, values are +layered in this order (later wins): + +1. **Service defaults** — hard-coded inside the service plugin. +2. **Archetype** — if the decky's section had `archetype=…`, the archetype's + default services/distro/`nmap_os` apply. +3. **Per-decky keys** — `ip`, `services`, `nmap_os`, `mutate_interval` from + the decky section override archetype defaults. +4. **Per-service subsection** — keys in `[.]` override the + service's built-in defaults. For a group (`amount>N`), the subsection + `[.]` is copied into every expanded decky. + +--- + +## Full realistic example + +A 10-decky heterogeneous fleet, mixing an archetype pool with role-themed +boxes. This is a trimmed version of `development/test-full.ini`. + +```ini +# full.ini — 10-decky heterogeneous fleet +# +# sudo decnet deploy --config full.ini --interface eth0 \ +# --log-target 192.168.1.200:5140 + +[general] +net = 192.168.1.0/24 +gw = 192.168.1.1 +interface = eth0 + +# ── Archetype pool: 5 Windows workstations ─────────────────────────── +[windows-workstation] +archetype = windows-workstation +amount = 5 + +# ── Web / Mail stack ───────────────────────────────────────────────── +[decky-webmail] +ip = 192.168.1.110 +services = http, smtp, imap, pop3 +nmap_os = linux + +[decky-webmail.http] +server_header = Apache/2.4.54 (Debian) +fake_app = wordpress + +[decky-webmail.smtp] +smtp_banner = 220 mail.corp.local ESMTP Postfix (Debian/GNU) + +# ── Windows file server ────────────────────────────────────────────── +[decky-fileserv] +ip = 192.168.1.111 +services = smb, ftp, tftp +nmap_os = windows + +[decky-fileserv.smb] +workgroup = CORP +server_name = FILESERV01 +os_version = Windows Server 2019 + +# ── Postgres / Mongo / Elastic ─────────────────────────────────────── +[decky-dbsrv02] +ip = 192.168.1.113 +services = postgres, mongodb, elasticsearch +nmap_os = linux + +[decky-dbsrv02.postgres] +pg_version = 14.5 + +[decky-dbsrv02.elasticsearch] +es_version = 8.4.3 +cluster_name = prod-search + +# ── IoT / SCADA (embedded TCP stack) ───────────────────────────────── +[decky-iot] +ip = 192.168.1.117 +services = mqtt, snmp, conpot +nmap_os = embedded +mutate_interval = 60 + +[decky-iot.snmp] +snmp_community = public +sys_descr = Linux router 5.4.0 #1 SMP x86_64 + +# ── Legacy admin box ───────────────────────────────────────────────── +[decky-legacy] +ip = 192.168.1.119 +services = telnet, vnc, ssh +nmap_os = bsd + +[decky-legacy.ssh] +ssh_version = OpenSSH_7.4p1 Debian-10+deb9u7 +kernel_version = 4.9.0-19-amd64 +users = root:root,admin:password,pi:raspberry + +# ── Bring-your-own service (see Custom-Services) ───────────────────── +[custom-weirdapp] +binary = myorg/weirdapp:1.2.3 +exec = /opt/weirdapp/run --listen 0.0.0.0:9000 +ports = 9000 +``` + +Deploy with: + +```bash +sudo decnet deploy --config full.ini --interface eth0 \ + --log-target 192.168.1.200:5140 +``` + +--- + +## Validation and errors + +- Empty INI (no sections): `"The provided INI content must contain at least + one section"`. +- Total payload > 512 KB: rejected. +- At least one decky section is mandatory (`IniConfig.at_least_one_decky`). +- `amount` must be a positive integer `<=100`. +- `ip=` with `amount>1` is rejected. +- `mutate_interval` must be an integer. +- `DeckySpec` and `CustomServiceSpec` both have `extra="forbid"` — unknown + *typed* fields on the parser's output raise Pydantic errors.