merge: testing → main (reconcile 2-week divergence)
This commit is contained in:
63
development/docs/services/COLLECTOR.md
Normal file
63
development/docs/services/COLLECTOR.md
Normal file
@@ -0,0 +1,63 @@
|
||||
# DECNET Collector
|
||||
|
||||
The `decnet/collector` module is responsible for the background acquisition, normalization, and filtering of logs generated by the honeypot fleet. It acts as the bridge between the transient Docker container logs and the persistent analytical database.
|
||||
|
||||
## Architecture
|
||||
|
||||
The Collector runs as a host-side worker (typically managed by the CLI or a daemon). It employs a hybrid asynchronous and multi-threaded model to handle log streaming from a dynamic number of containers without blocking the main event loop.
|
||||
|
||||
### Log Pipeline Flow
|
||||
1. **Discovery**: Scans `decnet-state.json` to identify active Decky service containers.
|
||||
2. **Streaming**: Spawns a dedicated thread for every active container to tail its `stdout` via the Docker SDK.
|
||||
3. **Normalization**: Parses the raw RFC 5424 Syslog lines into structured JSON.
|
||||
4. **Filtering**: Applies a rate-limiter to deduplicate high-frequency connection events.
|
||||
5. **Storage**: Appends raw lines to `.log` and filtered JSON to `.json` for database ingestion.
|
||||
|
||||
---
|
||||
|
||||
## Core Components
|
||||
|
||||
### `worker.py`
|
||||
|
||||
#### `log_collector_worker(log_file: str)`
|
||||
The main asynchronous entry point.
|
||||
- **Initial Scan**: Identifies all running containers that match the DECNET service naming convention.
|
||||
- **Event Loop**: Uses the Docker `events` API to listen for `container:start` events, allowing it to automatically pick up new Deckies that are deployed after the collector has started.
|
||||
- **Task Management**: Manages a dictionary of active streaming tasks, ensuring no container is streamed more than once and cleaning up completed tasks.
|
||||
|
||||
---
|
||||
|
||||
## Log Normalization (RFC 5424)
|
||||
|
||||
DECNET services emit logs using a standardized RFC 5424 format with structured data. The `parse_rfc5424` function is the primary tool for extracting this information.
|
||||
|
||||
- **Structured Data**: Extracts parameters from the `relay@55555` SD-ELEMENT.
|
||||
- **Field Mapping**: Identifies the `attacker_ip` by scanning common source IP fields (`src_ip`, `client_ip`, etc.).
|
||||
- **Consistency**: Formats timestamps into a human-readable `%Y-%m-%d %H:%M:%S` format for the analytical stream.
|
||||
|
||||
---
|
||||
|
||||
## Ingestion Rate Limiter
|
||||
|
||||
To prevent the local SQLite database from being overwhelmed during credential-stuffing attacks or heavy port scanning, the Collector implements a window-based rate limiter for "lifecycle" events.
|
||||
|
||||
- **Scope**: By default, it limits: `connect`, `disconnect`, `connection`, `accept`, and `close`.
|
||||
- **Logic**: It groups events by `(attacker_ip, decky, service, event_type)`. If the same event occurs within the window, it is written to the raw `.log` file (for forensics) but **discarded** for the `.json` stream (ingestion).
|
||||
- **Configuration**:
|
||||
- `DECNET_COLLECTOR_RL_WINDOW_SEC`: The deduplication window size (default: 1.0s).
|
||||
- `DECNET_COLLECTOR_RL_EVENT_TYPES`: Comma-separated list of event types to limit.
|
||||
|
||||
---
|
||||
|
||||
## Resilience & Operational Stability
|
||||
|
||||
### Inode Tracking (`_reopen_if_needed`)
|
||||
Log files can be rotated by `logrotate` or manually deleted. The Collector tracks the **inode** of the log handles. If the file on disk changes (indicating rotation or deletion), the collector transparently closes and reopens the handle, ensuring no logs are lost and preventing "stale handle" errors.
|
||||
|
||||
### Docker SDK Integration
|
||||
The Collector uses `asyncio.to_thread` to run the blocking Docker SDK `logs(stream=True)` calls. This ensures that the high-latency network calls to the Docker daemon do not starve the asynchronous event loop responsible for monitoring container starts.
|
||||
|
||||
### Container Identification
|
||||
The Collector uses two layers of verification to ensure it only collects logs from DECNET honeypots:
|
||||
1. **Name Matching**: Checks if the container name matches the `{decky}-{service}` pattern.
|
||||
2. **State Verification**: Cross-references container names with the current `decnet-state.json`.
|
||||
61
development/docs/services/ENGINE.md
Normal file
61
development/docs/services/ENGINE.md
Normal file
@@ -0,0 +1,61 @@
|
||||
# DECNET Engine (Orchestrator)
|
||||
|
||||
The `decnet/engine` module is the central nervous system of DECNET. It acts as the primary orchestrator, responsible for bridging high-level configuration (user-defined deckies and archetypes) with the underlying infrastructure (Docker containers, MACVLAN/IPvlan networking, and host-level configurations).
|
||||
|
||||
## Role in the Ecosystem
|
||||
While the CLI manages user interaction and the Service Registry manages available honeypots, the **Engine** is what actually manifests these concepts into running containers on the network. It handles:
|
||||
- **Network Virtualization**: Dynamically setting up MACVLAN or IPvlan L2 interfaces.
|
||||
- **Container Lifecycle**: Orchestrating `docker compose` for building and running services.
|
||||
- **State Persistence**: Tracking active deployments to ensure clean teardowns.
|
||||
- **Unified Logging Injection**: Ensuring all honeypots share the same logging utilities.
|
||||
|
||||
---
|
||||
|
||||
## Core Components
|
||||
|
||||
### `deployer.py`
|
||||
This is the primary implementation file for the engine logic.
|
||||
|
||||
#### `deploy(config: DecnetConfig, ...)`
|
||||
The entry point for a deployment. It executes the following sequence:
|
||||
1. **Network Setup**: Identifies the IP range required for the requested deckies and initializes the Docker MACVLAN/IPvlan network.
|
||||
2. **Host Bridge**: Configures host-level routing (via `setup_host_macvlan` or `setup_host_ipvlan`) so the host can communicate with the decoys.
|
||||
3. **Logging Synchronization**: Copies the `decnet_logging.py` utility into every service's build context to ensure consistent log formatting.
|
||||
4. **Compose Generation**: Uses the `decnet.composer` to generate a `decnet-compose.yml` file.
|
||||
5. **State Management**: Saves the current configuration to `decnet-state.json`.
|
||||
6. **Orchestrated Build/Up**: Executes `docker compose up --build` with automatic retries for transient Docker daemon failures.
|
||||
|
||||
#### `teardown(decky_id: str | None = None)`
|
||||
Handles the cleanup of DECNET resources.
|
||||
- **Targeted Teardown**: If a `decky_id` is provided, it stops and removes only those specific containers.
|
||||
- **Full Teardown**: If no ID is provided, it:
|
||||
- Stops and removes all DECNET containers.
|
||||
- Tears down host-level virtual interfaces.
|
||||
- Removes the Docker MACVLAN/IPvlan network.
|
||||
- Clears the internal `decnet-state.json`.
|
||||
|
||||
#### `status()`
|
||||
Provides a real-time snapshot of the deployment.
|
||||
- Queries the Docker SDK for the current status of all containers associated with the active deployment.
|
||||
- Displays a `rich` table showing Decky names, IPs, Hostnames, and the health status of individual services.
|
||||
|
||||
---
|
||||
|
||||
## Internal Logic & Helpers
|
||||
|
||||
### Infrastructure Orchestration
|
||||
The Engine relies heavily on sub-processes to interface with `docker compose`, as it provides a robust abstraction for managing complex container groups (Deckies).
|
||||
|
||||
- **`_compose_with_retry`**: Docker operations (especially `pull` and `build`) can fail due to network timeouts or registry issues. This helper implements exponential backoff to ensure high reliability during deployment.
|
||||
- **`_compose`**: A direct wrapper for `docker compose` commands used during teardown where retries are less critical.
|
||||
|
||||
### The Logging Helper (`_sync_logging_helper`)
|
||||
One of the most critical parts of the engine is ensuring that every honeypot service, regardless of its unique implementation, speaks the same syslog "language." The engine iterates through every active service and copies `templates/decnet_logging.py` into their respective build contexts before the build starts. This allows service containers to import the standardized logging logic at runtime.
|
||||
|
||||
---
|
||||
|
||||
## Error Handling & Resilience
|
||||
The Engine is designed to handle "Permanent" vs "Transient" failures. It identifies errors such as `manifest unknown` or `repository does not exist` as terminal and will abort immediately, while others (connection resets, daemon timeouts) trigger a retry cycle.
|
||||
|
||||
## State Management
|
||||
The Engine maintains a `decnet-state.json` file. This file acts as the source of truth for what is currently "on the wire." Without this state, a proper `teardown` would be impossible, as the engine wouldn't know which virtual interfaces were created on the host NIC.
|
||||
58
development/docs/services/MODELS.md
Normal file
58
development/docs/services/MODELS.md
Normal file
@@ -0,0 +1,58 @@
|
||||
# DECNET Domain Models
|
||||
|
||||
> [!IMPORTANT]
|
||||
> **DEVELOPMENT DISCLAIMER**: DECNET is currently in active development. The models defined in `decnet/models.py` are subject to significant changes as the framework evolves.
|
||||
|
||||
## Overview
|
||||
|
||||
The `decnet/models.py` file serves as the centralized repository for all **Domain Models** used throughout the project. These are implemented using Pydantic v2 and ensure that the core business logic remains decoupled from the specific implementation details of the database (SQLAlchemy/SQLite) or the web layer (FastAPI).
|
||||
|
||||
---
|
||||
|
||||
## Model Hierarchy
|
||||
|
||||
DECNET categorizes its models into two primary functional groups: **INI Specifications** and **Runtime Configurations**.
|
||||
|
||||
### 1. INI Specifications (Input Validation)
|
||||
These models are designed to represent the structure of a `decnet.ini` file. They are primarily consumed by the `ini_loader.py` during the parsing of user-provided configuration files.
|
||||
|
||||
- **`IniConfig`**: The root model for a full deployment specification. It includes global settings like `subnet`, `gateway`, and `interface`, and contains a list of `DeckySpec` objects.
|
||||
- **`DeckySpec`**: A high-level description of a machine. It contains optional fields that the user *may* provide in an INI file (e.g., `ip`, `archetype`, `services`).
|
||||
- **`CustomServiceSpec`**: Defines external "Bring-Your-Own" services using Docker images and custom execution commands.
|
||||
|
||||
### 2. Runtime Configurations (Operational State)
|
||||
These models represent the **active, fully resolved state** of the deployment. Unlike the specifications, these models require all fields to be populated and valid.
|
||||
|
||||
- **`DecnetConfig`**: The operational root of a deployment. It includes the resolved network settings and the list of active `DeckyConfig` objects. It is used by the **Engine** for orchestration and is persisted in `decnet-state.json`.
|
||||
- **`DeckyConfig`**: A fully materialized decoy configuration. It includes generated hostnames, resolved distro images, and specific IP addresses.
|
||||
|
||||
---
|
||||
|
||||
## The Fleet Transformer (`fleet.py`)
|
||||
|
||||
The connection between the **Specifications** and the **Runtime Configurations** is handled by `decnet/fleet.py`.
|
||||
|
||||
The function `build_deckies_from_ini` takes an `IniConfig` as input and performs the following "up-conversion" logic:
|
||||
- **IP Allocation**: Auto-allocates free IPs from the subnet for any deckies missing an explicit IP in the INI.
|
||||
- **Service Resolution**: Validates that all requested services exist in the registry and assigns defaults from archetypes if needed.
|
||||
- **Environment Inheritance**: Inherits settings like rotation intervals (`mutate_interval`) from the global INI context down to individual deckies.
|
||||
|
||||
---
|
||||
|
||||
## Structural Validation: `IniContent`
|
||||
|
||||
To ensure that saved deployments in the database or provided by the API remain structurally sound, DECNET uses a specialized `IniContent` type.
|
||||
|
||||
- **`validate_ini_string`**: A pre-validator that uses Python's native `configparser`. It ensures that the content is a valid INI string, does not exceed 512KB, and contains at least one section.
|
||||
- **Standardized Errors**: It raises specifically formatted `ValueError` exceptions that are captured by both the CLI and the Web UI to provide clear feedback to the user.
|
||||
|
||||
---
|
||||
|
||||
## Key Consumer Modules
|
||||
|
||||
| Module | Usage |
|
||||
| :--- | :--- |
|
||||
| **`decnet/ini_loader.py`** | Uses `IniConfig` and `DeckySpec` to parse raw `.ini` files into structured objects. |
|
||||
| **`decnet/fleet.py`** | Transforms `IniConfig` specs into `DeckyConfig` operational models. |
|
||||
| **`decnet/config.py`** | Uses `DecnetConfig` and `DeckyConfig` to manage the lifecycle of `decnet-state.json`. |
|
||||
| **`decnet/web/db/models.py`** | Utilizes `IniContent` to enforce structural validity on INI strings stored in the database. |
|
||||
134
development/docs/services/WEB_MODELS.md
Normal file
134
development/docs/services/WEB_MODELS.md
Normal file
@@ -0,0 +1,134 @@
|
||||
# DECNET Web & Database Models: Architectural Deep Dive
|
||||
|
||||
> [!IMPORTANT]
|
||||
> **DEVELOPMENT DISCLAIMER**: DECNET is currently in active development. The storage schemas and API signatures defined in `decnet/web/db/models.py` are subject to radical change as the framework's analytical capabilities and distributed features expand.
|
||||
|
||||
## 1. Introduction & Philosophy
|
||||
|
||||
The `decnet/web/db/models.py` file represents the structural backbone of the DECNET web interface and its underlying analytical engine. It serves a dual purpose that is central to the project's architecture:
|
||||
|
||||
1. **Unified Source of Truth**: By utilizing **SQLModel**, DECNET collapses the traditional barrier between Pydantic data validation and SQLAlchemy ORM mapping. This allows a single class definition to act as both a database table and an API data object, drastically reducing the "boilerplate" associated with traditional web-database pipelines.
|
||||
2. **Analytical Scalability**: The models are designed to scale from small-scale local deployments using **SQLite** to large-scale, enterprise-ready environments backed by **MySQL**. This is achieved through clever usage of SQLAlchemy "Variants" and abstraction layers for large text blobs.
|
||||
|
||||
---
|
||||
|
||||
## 2. The Database Layer (SQLModel Entities)
|
||||
|
||||
These models define the physical tables within the DECNET infrastructure. Every class marked with `table=True` is interpreted by the repository layer to generate the corresponding DDL (Data Definition Language) for the target database.
|
||||
|
||||
### 2.1 Identity & Security: The `User` Entity
|
||||
|
||||
The `User` model handles dashboard access control and basic identity management.
|
||||
|
||||
* `uuid`: A unique string identifier. While integers are often used for IDs, DECNET uses strings to support potential future transitions to UUIDs without schema breakage.
|
||||
* `username`: The primary login handle. It is both `unique` and `indexed` for rapid authentication lookups.
|
||||
* `password_hash`: Stores the Argon2 or bcrypt hash. Length constraints in various routers ensure that raw passwords never exceed 72 characters, preventing "Long Password Denial of Service" attacks on various hashing algorithms.
|
||||
* `role`: A simple string-based permission field (e.g., `admin`, `viewer`).
|
||||
* `must_change_password`: A boolean flag used for fresh deployments or manual administrative resets, forcing the user to rotate their credentials upon their first authenticated session.
|
||||
|
||||
### 2.2 Intelligence & Attribution: `Attacker` and `AttackerBehavior`
|
||||
|
||||
These two tables form the core of DECNET's "Attacker Profiling" system. They are split into two tables to maintain "Narrow vs. Wide" performance characteristics.
|
||||
|
||||
#### The `Attacker` Entity (Broad Analytics)
|
||||
The `Attacker` table stores the "primary" record for every unique IP discovered by the honeypot fleet.
|
||||
|
||||
* `ip`: The source IP address. This is the primary key and is heavily indexed.
|
||||
* `first_seen` / `last_seen`: Tracking the lifecycle of an attacker's engagement with the network.
|
||||
* `event_count` / `service_count` / `decky_count`: Aggregated counters used by the stats dashboard to visualize the magnitude of an engagement.
|
||||
* `services` / `deckies`: JSON-serialized lists of every service and machine reached by the attacker. Using `_BIG_TEXT` here allows these lists to grow significantly during long-term campaigns.
|
||||
* `traversal_path`: A string representation (e.g., `omega → epsilon → zulu`) that helps analysts visualize lateral movement attempts recorded by the correlation engine.
|
||||
|
||||
#### The `AttackerBehavior` Entity (Granular Analytics)
|
||||
This "Wide" table stores behavioral signatures. It is separated from the main `Attacker` record so that high-frequency updates to timing stats or sniffer-derived packet signatures don't lock the primary attribution rows.
|
||||
|
||||
* `os_guess`: Derived from the `os_fingerprint` and `sniffer` engines, providing an estimate of the attacker's operating system based on TCP/IP stack nuances.
|
||||
* `tcp_fingerprint`: A JSON blob storing the raw TCP signature (Window size, MSS, Option sequence).
|
||||
* `behavior_class`: A classification (e.g., `beaconing`, `interactive`, `brute_force`) derived from log inter-arrival timing (IAT).
|
||||
* `timing_stats`: Stores a JSON dictionary of mean/median/stdev for event timing, used to detect automated tooling.
|
||||
|
||||
### 2.3 Telemetry: `Log` and `Bounty`
|
||||
|
||||
These tables store the "raw" data generated by the honeypots.
|
||||
|
||||
* **`Log` Table**: The primary event sink. Every line from the collector ends up here.
|
||||
* `event_type`: The MSGID from the RFC 5424 header (e.g., `connect`, `exploit`).
|
||||
* `raw_line`: The full, un-parsed syslog string for forensic verification.
|
||||
* `fields`: A JSON blob containing the structured data (SD-ELEMENTS) extracted during normalization.
|
||||
* **`Bounty` Table**: Specifically for high-value events. When a service detects "Gold" (like a plain-text password or a known PoC payload), it is mirrored here for rapid analyst review.
|
||||
|
||||
### 2.4 System State: The `State` Entity
|
||||
|
||||
The `State` table acts as the orchestrator's brain. It stores the `decnet-state.json` content within the database when the system is integrated with the web layer.
|
||||
|
||||
* `key`: The configuration key (e.g., `global_config`, `active_deployment`).
|
||||
* `value`: A `MEDIUMTEXT` JSON blob. This is potentially its largest field, storing the entire resolved configuration of every running Decky.
|
||||
|
||||
---
|
||||
|
||||
## 3. The API Layer (Pydantic DTOs)
|
||||
|
||||
These models define how data moves across the wire between the FastAPI backend and the frontend.
|
||||
|
||||
### 3.1 Authentication Pipeline
|
||||
* `LoginRequest`: Validates incoming credentials before passing them to the security middleware.
|
||||
* `Token`: The standard OAuth2 bearer token response, enriched with the `must_change_password` hint.
|
||||
* `ChangePasswordRequest`: Ensures the old password is provided and the new one meets the project's security constraints.
|
||||
|
||||
### 3.2 Reporting & Pagination
|
||||
DECNET uses a standardized "Envelope" pattern for broad analytical responses (`LogsResponse`, `AttackersResponse`, `BountyResponse`).
|
||||
|
||||
* `total`: The total count of matching records in the database (ignoring filters).
|
||||
* `limit` / `offset`: The specific slice of data returned, supporting "Infinite Scroll" or traditional pagination in the UI.
|
||||
* `data`: A list of dictionaries. By using `dict[str, Any]` here, the API remains flexible with SQLModel's dynamic attribute loading.
|
||||
|
||||
### 3.3 System Administration
|
||||
* **`DeployIniRequest`**: The most critical input model. It takes `ini_content` as a validated string. By using the `IniContent` annotated type, the API rejects malformed deployments before they ever touch the fleet builder.
|
||||
* **`MutateIntervalRequest`**: Uses a strict REGEX pattern (`^[1-9]\d*[mdMyY]$`) to ensure intervals like `30m` (30 minutes) or `2d` (2 days) are valid before being applied to the orchestrator.
|
||||
|
||||
---
|
||||
|
||||
## 4. Technical Foundations
|
||||
|
||||
### 4.1 Cross-DB Compatibility Logic
|
||||
The project uses a custom variant system to handle the discrepancies between SQLite (which has simplified typing) and MySQL (which has strict size constraints).
|
||||
|
||||
```python
|
||||
_BIG_TEXT = Text().with_variant(MEDIUMTEXT(), "mysql")
|
||||
```
|
||||
|
||||
This abstraction ensures that fields like `Attacker.services` (which can grow to thousands of items) are stored as `MEDIUMTEXT` (16 MiB) on MySQL, whereas standard SQLAlchemy `Text` (often 64 KiB on MySQL) would silently truncate the data, leading to analytical loss.
|
||||
|
||||
### 4.2 High-Fidelity Normalization
|
||||
Data arriving from distributed honeypots is often "dirty." The models include custom pre-validators like `_normalize_null`.
|
||||
|
||||
* **Null Coalescing**: Services often emit logging values as `"null"` or `"undefined"` strings. The `NullableString` type automatically converts these "noise" strings into actual Python `None` types during ingestion.
|
||||
* **Timestamp Integrity**: `NullableDatetime` ensures that various ISO formats or epoch timestamps provided by different service containers are normalized into standard UTC datetime objects.
|
||||
|
||||
---
|
||||
|
||||
## 5. Integration Case Studies (Deep Analysis)
|
||||
|
||||
To understand how these models function, we must examine their lifecycle across the web stack.
|
||||
|
||||
### 5.1 The Repository Layer (`decnet/web/db/sqlmodel_repo.py`)
|
||||
The repository is the primary consumer of the "Entities." It utilizes the metadata generated by SQLModel to:
|
||||
1. **Generate DDL**: On startup, the repository calls `SQLModel.metadata.create_all()`. This takes every `table=True` class and translates it into `CREATE TABLE` statements tailored to the active engine (SQLite or MySQL).
|
||||
2. **Translate DTOs**: When the repository fetches an `Attacker` from the DB, SQLModel automatically populates the Pydantic-style attributes, allowing the repository to return objects that are immediately serializeable by the routers.
|
||||
|
||||
### 5.2 The Dashboard Routers
|
||||
Specific endpoints rely on these models for boundary safety:
|
||||
|
||||
* **`api_deploy_deckies.py`**: Uses `DeployIniRequest`. This ensures that even if a user tries to POST a massive binary file instead of an INI, the Pydantic layer (powered by `decnet.models.validate_ini_string`) will intercept and reject the request with a `422 Unprocessable Entity` error before it reaches the orchestrator.
|
||||
* **`api_get_stats.py`**: Uses `StatsResponse`. This model serves as a "rollup" that aggregates data from the `Log`, `Attacker`, and `State` tables into a single unified JSON object for the dashboard's "At a Glance" view.
|
||||
* **`api_get_health.py`**: Uses `HealthResponse`. This model provides a nested view of the system, where each sub-component (Engine, Collector, DB) is represented as a `ComponentHealth` object, allowing the UI to show granular "Success" or "Failure" states.
|
||||
|
||||
---
|
||||
|
||||
## 6. Futureproofing & Guidelines
|
||||
|
||||
As the project grows, the following habits must be maintained:
|
||||
|
||||
1. **Keep the Row Narrow**: Always separate behavioral data that updates frequently into auxiliary tables like `AttackerBehavior`.
|
||||
2. **Use Variants**: Never use standard `String` or `Text` for JSON blobs; always use `_BIG_TEXT` to respect MySQL's storage limitations.
|
||||
3. **Validate at the Boundary**: Ensure every new API request model uses Pydantic's strict typing to prevent malicious payloads from reaching the database layer.
|
||||
Reference in New Issue
Block a user