Commit Graph

37 Commits

Author SHA1 Message Date
82ec7f3117 fix: gate embedded profiler behind DECNET_EMBED_PROFILER to prevent dual-instance cursor conflict
decnet deploy spawns a standalone profiler daemon AND api.py was also starting
attacker_profile_worker as an asyncio task inside the web server. Both instances
shared the same attacker_worker_cursor key in the state table, causing a race
where one instance could skip events already claimed by the other or overwrite
the cursor mid-batch.

Default is now OFF (embedded profiler disabled). The standalone daemon started
by decnet deploy is the single authoritative instance. Set DECNET_EMBED_PROFILER=true
only when running decnet api in isolation without a full deploy.
2026-04-15 17:49:18 -04:00
172a002d41 refactor: implement database backend factory for SQLite and MySQL
- Add `get_repository()` factory function to select DB implementation at runtime via DECNET_DB_TYPE env var
- Extract BaseRepository abstract interface from SQLiteRepository
- Update dependencies to use factory-based repository injection
- Add DECNET_DB_TYPE env var support (defaults to sqlite)
- Refactor models and repository base class for cross-dialect compatibility
2026-04-15 12:50:41 -04:00
a2ba7a7f3c feat: add /health endpoint for microservice monitoring
Checks database, background workers (ingestion, collector, attacker,
sniffer), and Docker daemon. Reports healthy/degraded/unhealthy status
with per-component details. Returns 503 when required services fail,
200 for healthy or degraded (only optional services down).
2026-04-14 16:56:20 -04:00
5a7ff285cd feat: fleet-wide MACVLAN sniffer microservice
Replace per-decky sniffer containers with a single host-side sniffer
that monitors all traffic on the MACVLAN interface. Runs as a background
task in the FastAPI lifespan alongside the collector, fully fault-isolated
so failures never crash the API.

- Add fleet_singleton flag to BaseService; sniffer marked as singleton
- Composer skips fleet_singleton services in compose generation
- Fleet builder excludes singletons from random service assignment
- Extract TLS fingerprinting engine from templates/sniffer/server.py
  into decnet/sniffer/ package (parameterized for fleet-wide use)
- Sniffer worker maps packets to deckies via IP→name state mapping
- Original templates/sniffer/server.py preserved for future use
2026-04-14 15:02:34 -04:00
3dc5b509f6 feat: Phase 1 — JA3/JA3S sniffer, Attacker model, profile worker
Add passive TLS fingerprinting via a sniffer container on the MACVLAN
interface, plus the Attacker table and periodic rebuild worker that
correlates per-IP profiles from Log + Bounty + CorrelationEngine.

- templates/sniffer/: Scapy sniffer with pure-Python TLS parser;
  emits tls_client_hello / tls_session RFC 5424 lines with ja3, ja3s,
  sni, alpn, raw_ciphers, raw_extensions; GREASE filtered per RFC 8701
- decnet/services/sniffer.py: service plugin (no ports, NET_RAW/NET_ADMIN)
- decnet/web/db/models.py: Attacker SQLModel table + AttackersResponse
- decnet/web/db/repository.py: 5 new abstract methods
- decnet/web/db/sqlite/repository.py: implement all 5 (upsert, pagination,
  sort by recent/active/traversals, bounty grouping)
- decnet/web/attacker_worker.py: 30s periodic rebuild via CorrelationEngine;
  extracts commands from log fields, merges fingerprint bounties
- decnet/web/api.py: wire attacker_profile_worker into lifespan
- decnet/web/ingester.py: extract JA3 bounty (fingerprint_type=ja3)
- development/DEVELOPMENT.md: full attacker intelligence collection roadmap
- pyproject.toml: scapy>=2.6.1 added to dev deps
- tests: test_sniffer_ja3.py (40+ vectors), test_attacker_worker.py,
  test_base_repo.py / test_web_api.py updated for new surface
2026-04-13 20:22:08 -04:00
035499f255 feat: add component-aware RFC 5424 application logging system
- Modify Rfc5424Formatter to read decnet_component from LogRecord
  and use it as RFC 5424 APP-NAME field (falls back to 'decnet')
- Add get_logger(component) factory in decnet/logging/__init__.py
  with _ComponentFilter that injects decnet_component on each record
- Wire all five layers to their component tag:
    cli -> 'cli', engine -> 'engine', api -> 'api' (api.py, ingester,
    routers), mutator -> 'mutator', collector -> 'collector'
- Add structured INFO/DEBUG/WARNING/ERROR log calls throughout each
  layer per the defined vocabulary; DEBUG calls are suppressed unless
  DECNET_DEVELOPER=true
- Add tests/test_logging.py covering factory, filter, formatter
  component-awareness, fallback behaviour, and level gating
2026-04-13 07:39:01 -04:00
f2cc585d72 fix: align tests with model validation and API error reporting 2026-04-13 01:43:52 -04:00
b2e4706a14 Refactor: implemented Repository Factory and Async Mutator Engine. Decoupled storage logic and enforced Dependency Injection across CLI and Web API. Updated documentation.
Some checks failed
CI / Lint (ruff) (push) Successful in 12s
CI / SAST (bandit) (push) Successful in 13s
CI / Dependency audit (pip-audit) (push) Successful in 22s
CI / Test (Standard) (3.11) (push) Failing after 54s
CI / Test (Standard) (3.12) (push) Successful in 1m35s
CI / Test (Live) (3.11) (push) Has been skipped
CI / Test (Fuzz) (3.11) (push) Has been skipped
CI / Merge dev → testing (push) Has been skipped
CI / Prepare Merge to Main (push) Has been skipped
CI / Finalize Merge to Main (push) Has been skipped
2026-04-12 07:48:17 -04:00
c384a3103a refactor: separate engine, collector, mutator, and fleet into independent subpackages
- decnet/engine/ — container lifecycle (deploy, teardown, status); _kill_api removed
- decnet/collector/ — Docker log streaming (moved from web/collector.py)
- decnet/mutator/ — mutation engine (no longer imports from cli or duplicates deployer code)
- decnet/fleet.py — shared decky-building logic extracted from cli.py

Cross-contamination eliminated:
- web router no longer imports from decnet.cli
- mutator no longer imports from decnet.cli
- cli no longer imports from decnet.web
- _kill_api() moved to cli (process management, not engine concern)
- _compose_with_retry duplicate removed from mutator
2026-04-12 00:26:22 -04:00
25ba3fb56a feat: replace bind-mount log pipeline with Docker log streaming
Services now print RFC 5424 to stdout; Docker captures via json-file driver.
A new host-side collector (decnet.web.collector) streams docker logs from all
running decky service containers and writes RFC 5424 + parsed JSON to the host
log file. The existing ingester continues to tail the .json file unchanged.
rsyslog can consume the .log file independently — no DECNET involvement needed.

Removes: bind-mount volume injection, _LOG_NETWORK bridge, log_target config
field and --log-target CLI flag, TCP syslog forwarding from service templates.
2026-04-10 00:14:14 -04:00
cea6279a08 fix: add Last-Event-ID to CORS allow_headers to unblock SSE reconnects 2026-04-09 19:26:24 -04:00
016115a523 fix: clear all addressable technical debt (DEBT-005 through DEBT-025)
Security:
- DEBT-008: remove query-string token auth; header-only Bearer now enforced
- DEBT-013: add regex constraint ^[a-z0-9\-]{1,64}$ on decky_name path param
- DEBT-015: stop leaking raw exception detail to API clients; log server-side
- DEBT-016: validate search (max_length=512) and datetime params with regex

Reliability:
- DEBT-014: wrap SSE event_generator in try/except; yield error frame on failure
- DEBT-017: emit log.warning/error on DB init retry; silent failures now visible

Observability / Docs:
- DEBT-020: add 401/422 response declarations to all route decorators

Infrastructure:
- DEBT-018: add HEALTHCHECK to all 24 template Dockerfiles
- DEBT-019: add USER decnet + setcap cap_net_bind_service to all 24 Dockerfiles
- DEBT-024: bump Redis template version 7.0.12 → 7.2.7

Config:
- DEBT-012: validate DECNET_API_PORT and DECNET_WEB_PORT range (1-65535)

Code quality:
- DEBT-010: delete 22 duplicate decnet_logging.py copies; deployer injects canonical
- DEBT-022: closed as false positive (print only in module docstring)
- DEBT-009: closed as false positive (templates already use structured syslog_line)

Build:
- DEBT-025: generate requirements.lock via pip freeze

Testing:
- DEBT-005/006/007: comprehensive test suite added across tests/api/
- conftest: in-memory SQLite + StaticPool + monkeypatched session_factory
- fuzz mark added; default run excludes fuzz; -n logical parallelism

DEBT.md updated: 23/25 items closed; DEBT-011 (Alembic) and DEBT-023 (digest pinning) remain
2026-04-09 19:02:51 -04:00
b6b046c90b fix: harden startup security — require strong secrets, restrict CORS
- decnet/env.py: DECNET_JWT_SECRET and DECNET_ADMIN_PASSWORD are now
  required env vars; startup raises ValueError if unset or set to a
  known-bad default ("admin", "password", etc.)
- decnet/env.py: add DECNET_CORS_ORIGINS (comma-separated, defaults to
  http://localhost:8080) replacing the previous allow_origins=["*"]
- decnet/web/api.py: use DECNET_CORS_ORIGINS and tighten allow_methods
  and allow_headers to explicit lists
- tests/conftest.py: set required env vars at module level so test
  collection works without real credentials
- tests/test_web_api.py, test_web_api_fuzz.py: use DECNET_ADMIN_PASSWORD
  from env instead of hardcoded "admin"

Closes DEBT-001, DEBT-002, DEBT-004
2026-04-09 12:13:22 -04:00
29a2cf2738 refactor: modularize API routes into separate files and clean up dependencies 2026-04-09 11:58:57 -04:00
551664bc43 fix: stabilize test suite by ensuring proper test DB isolation and initialization 2026-04-09 02:31:14 -04:00
a3b92d4dd6 docs: tag API endpoints for better organization 2026-04-09 01:58:54 -04:00
30edf9a55d feat: add DECNET_DEVELOPER toggle for API documentation 2026-04-09 01:55:31 -04:00
69626d705d feat: implement Bounty Vault for captured credentials and artifacts 2026-04-09 01:52:50 -04:00
0f86f883fe fix: resolve remaining bandit warnings and stabilize lifespan 2026-04-09 01:35:08 -04:00
13f3d15a36 fix: stabilize tests with synchronous DB init and handle Bandit security findings 2026-04-09 01:33:15 -04:00
8c7ec2953e fix: handle bcrypt 72-byte limit and increase JWT secret length 2026-04-09 01:11:32 -04:00
6c2478ede3 fix: restore missing API endpoints, fix chart rendering, and update date filter formatting 2026-04-08 21:25:59 -04:00
532a4e2dc5 fix: resolve SSE CORS issues and fix date filter format mismatch 2026-04-08 21:15:26 -04:00
ec503b9ec6 feat: implement advanced live logs with KQL search, histogram, and live/historical modes 2026-04-08 21:01:05 -04:00
32b06afef6 feat: add .env based configuration for API, Web, and Auth options 2026-04-08 01:27:11 -04:00
31e0c5151b fix: ensure API-deployed deckies inherit the correct log ingestion path 2026-04-08 01:09:48 -04:00
cc3d434c02 feat: add server-side validation for web-based INI deployments 2026-04-08 01:04:59 -04:00
168ecf14ab feat: add API-only mode and web-based INI deployment 2026-04-08 00:56:25 -04:00
c544964f57 feat: migrate dashboard live logs to Server-Sent Events (SSE) 2026-04-08 00:30:31 -04:00
18de381a43 feat: implement dynamic decky mutation and fix dot-separated INI sections 2026-04-08 00:16:57 -04:00
eb4be44c9a feat: add dedicated Decoy Fleet inventory page and API 2026-04-07 23:15:20 -04:00
ba2faba5d5 chore: enforce strict typing and internal naming conventions across web components 2026-04-07 19:56:15 -04:00
bad90dfb75 feat: implement background log ingestion from local file 2026-04-07 15:30:44 -04:00
52c26a2891 feat: backend support for mandatory password change on first login 2026-04-07 15:15:03 -04:00
697929a127 feat: implement Stats endpoints for web dashboard 2026-04-07 14:58:09 -04:00
b46934db46 feat: implement Logs endpoints for web dashboard 2026-04-07 14:56:25 -04:00
5b990743db feat: implement Auth endpoints for web dashboard 2026-04-07 14:54:36 -04:00