Auto PR: testing → main #3

Closed
anti wants to merge 0 commits from testing into main
Owner

All CI and security checks passed on both dev and testing. Review and merge when ready.

All CI and security checks passed on both dev and testing. Review and merge when ready.
anti added 136 commits 2026-04-12 09:56:30 +02:00
revert Added: modified notes. Finished CI/CD pipeline.
feat: add systemd service templates for API and Web Dashboard
Some checks failed
CI / Lint (ruff) (push) Successful in 15s
CI / Test (pytest) (3.11) (push) Failing after 21s
CI / Test (pytest) (3.12) (push) Failing after 22s
CI / SAST (bandit) (push) Failing after 13s
CI / Dependency audit (pip-audit) (push) Successful in 19s
CI / Open PR to main (push) Has been skipped
fc99375c62
chore: deleted trash
Some checks failed
CI / Lint (ruff) (push) Successful in 11s
CI / Test (pytest) (3.11) (push) Failing after 18s
CI / Test (pytest) (3.12) (push) Failing after 18s
CI / SAST (bandit) (push) Failing after 11s
CI / Dependency audit (pip-audit) (push) Successful in 18s
CI / Open PR to main (push) Has been skipped
6f10e7556f
modified: ci.yml, pyproject: added missing installs and modified pip install command
Some checks failed
CI / Lint (ruff) (push) Successful in 16s
CI / Test (pytest) (3.11) (push) Failing after 20s
CI / Test (pytest) (3.12) (push) Failing after 20s
CI / SAST (bandit) (push) Failing after 11s
CI / Dependency audit (pip-audit) (push) Successful in 19s
CI / Open PR to main (push) Has been skipped
65b220fdbe
modified: ci.yml, fucked up last time lol
Some checks failed
CI / Lint (ruff) (push) Successful in 11s
CI / Test (pytest) (3.11) (push) Successful in 1m42s
CI / Test (pytest) (3.12) (push) Successful in 1m45s
CI / SAST (bandit) (push) Failing after 12s
CI / Dependency audit (pip-audit) (push) Successful in 20s
CI / Open PR to main (push) Has been skipped
fe6b349e5e
- decnet/env.py: DECNET_JWT_SECRET and DECNET_ADMIN_PASSWORD are now
  required env vars; startup raises ValueError if unset or set to a
  known-bad default ("admin", "password", etc.)
- decnet/env.py: add DECNET_CORS_ORIGINS (comma-separated, defaults to
  http://localhost:8080) replacing the previous allow_origins=["*"]
- decnet/web/api.py: use DECNET_CORS_ORIGINS and tighten allow_methods
  and allow_headers to explicit lists
- tests/conftest.py: set required env vars at module level so test
  collection works without real credentials
- tests/test_web_api.py, test_web_api_fuzz.py: use DECNET_ADMIN_PASSWORD
  from env instead of hardcoded "admin"

Closes DEBT-001, DEBT-002, DEBT-004
- Rebuild repo.engine and repo.session_factory per-test using unique
  in-memory SQLite URIs — fixes KeyError: 'access_token' caused by
  stale session_factory pointing at production DB
- Add @pytest.mark.fuzz to all Hypothesis and Schemathesis tests;
  default run excludes them (addopts = -m 'not fuzz')
- Add missing fuzz tests to bounty, fleet, histogram, and repository
- Use tmp_path for state file in patch_state_file/mock_state_file to
  eliminate file-path race conditions under xdist parallelism
- Set default addopts: -v -q -x -n logical (26 tests in ~7s)
Security:
- DEBT-008: remove query-string token auth; header-only Bearer now enforced
- DEBT-013: add regex constraint ^[a-z0-9\-]{1,64}$ on decky_name path param
- DEBT-015: stop leaking raw exception detail to API clients; log server-side
- DEBT-016: validate search (max_length=512) and datetime params with regex

Reliability:
- DEBT-014: wrap SSE event_generator in try/except; yield error frame on failure
- DEBT-017: emit log.warning/error on DB init retry; silent failures now visible

Observability / Docs:
- DEBT-020: add 401/422 response declarations to all route decorators

Infrastructure:
- DEBT-018: add HEALTHCHECK to all 24 template Dockerfiles
- DEBT-019: add USER decnet + setcap cap_net_bind_service to all 24 Dockerfiles
- DEBT-024: bump Redis template version 7.0.12 → 7.2.7

Config:
- DEBT-012: validate DECNET_API_PORT and DECNET_WEB_PORT range (1-65535)

Code quality:
- DEBT-010: delete 22 duplicate decnet_logging.py copies; deployer injects canonical
- DEBT-022: closed as false positive (print only in module docstring)
- DEBT-009: closed as false positive (templates already use structured syslog_line)

Build:
- DEBT-025: generate requirements.lock via pip freeze

Testing:
- DEBT-005/006/007: comprehensive test suite added across tests/api/
- conftest: in-memory SQLite + StaticPool + monkeypatched session_factory
- fuzz mark added; default run excludes fuzz; -n logical parallelism

DEBT.md updated: 23/25 items closed; DEBT-011 (Alembic) and DEBT-023 (digest pinning) remain
Services now print RFC 5424 to stdout; Docker captures via json-file driver.
A new host-side collector (decnet.web.collector) streams docker logs from all
running decky service containers and writes RFC 5424 + parsed JSON to the host
log file. The existing ingester continues to tail the .json file unchanged.
rsyslog can consume the .log file independently — no DECNET involvement needed.

Removes: bind-mount volume injection, _LOG_NETWORK bridge, log_target config
field and --log-target CLI flag, TCP syslog forwarding from service templates.
- Add # nosec B104 to all intentional 0.0.0.0 binds in honeypot servers
  (hardcoded_bind_all_interfaces is by design — deckies must accept attacker connections)
- Add # nosec B101 to assert statements used for protocol validation in ldap/snmp
- Add # nosec B105 to fake SASL placeholder in ldap
- Add # nosec B108 to /tmp usage in smb template
- Exclude root-owned auto-generated decnet_logging.py copies from bandit scan
  via pyproject.toml [tool.bandit] config (synced by _sync_logging_helper at deploy)
- Buffer DATA body until CRLF.CRLF terminator — fixes 502-on-every-body-line bug
- SMTP_OPEN_RELAY=1: AUTH accepted (235), RCPT TO accepted for any domain,
  full DATA pipeline with queued-as message ID
- Default (SMTP_OPEN_RELAY=0): credential harvester — AUTH rejected (535)
  but connection stays open, RCPT TO returns 554 relay denied
- SASL PLAIN and LOGIN multi-step AUTH both decoded and logged
- RSET clears all per-transaction state
- Add development/SMTP_RELAY.md, IMAP_BAIT.md, ICS_SCADA.md, BUG_FIXES.md
  (live-tested service realism plans)
- decnet/services/smtp_relay.py: open relay variant of smtp, same template
  with SMTP_OPEN_RELAY=1 baked into the environment
- tests/service_testing/__init__.py: init so pytest discovers the subdirectory
- Add dynamic challenge nonces to Postgres, VNC, and SIP.
- Add basic keyspace lookup and mock data to Redis.
- Correct MSSQL TDS pre-login offset bounds.
- Support MongoDB OP_MSG handshake version checking.
- Suppress Werkzeug HTTP server headers and normalize FTPAnonymousShell response.
- Add tracking for Dynamic Bait Store (DEBT-027) via DEBT.md.
Add tcp_timestamps, tcp_window_scaling, tcp_sack, tcp_ecn, ip_no_pmtu_disc,
and tcp_fin_timeout to every OS profile in OS_SYSCTLS.

All 6 are network-namespace-scoped and safe to set per-container without
--privileged. They directly influence nmap's OPS, WIN, ECN, and T2-T6
probe groups, making OS family detection significantly more convincing.

Key changes:
- tcp_timestamps=0 for windows/embedded/cisco (strongest Windows discriminator)
- tcp_ecn=2 for linux (ECN offer), 0 for all others
- tcp_sack=0 / tcp_window_scaling=0 for embedded/cisco
- ip_no_pmtu_disc=1 for embedded/cisco (DF bit ICMP behaviour)
- Expose _REQUIRED_SYSCTLS frozenset for completeness assertions

Tests: 88 new test cases across all OS families and composer integration.
Total suite: 812 passed.
The entrypoint.sh was present in the build context but never COPYed into
the image, causing 'stat /entrypoint.sh: no such file or directory' at
container start. Added COPY+chmod before the USER decnet instruction so
the script is installed as root and is executable by all users.
When ip_no_pmtu_disc=0 the Linux kernel sets DF=1 on TCP packets and uses
IP ID=0 (RFC 6864). nmap's TI=Z fingerprint has no Windows match in its DB,
causing 91% confidence guesses of 'Linux 2.4/2.6 embedded' regardless of
TTL being 128. Setting ip_no_pmtu_disc=1 allows non-zero IP ID generation.

Trade-off: DF bit is not set on outgoing packets (slightly wrong for Windows)
but TI=Z is far more damaging to the spoof than losing DF accuracy.
ip_no_pmtu_disc controls PMTU discovery for UDP/ICMP paths only.
TI=Z originates from ip_select_ident() in the kernel TCP stack setting
IP ID=0 for DF=1 TCP packets — a namespace-scoped sysctl cannot change this.
The previous commit was based on incorrect root-cause analysis.
Phase 1 is complete. Live testing revealed:
- Window size (64240) is already correct — Phase 2 window mangling unnecessary
- TI=Z (IP ID = 0) is the single remaining blocker for Windows spoofing
- ip_no_pmtu_disc does NOT fix TI=Z (tested and confirmed)

Revised phase plan:
- Phase 2: ICMP tuning (icmp_ratelimit + icmp_ratemask sysctls)
- Phase 3: NFQUEUE daemon for IP ID rewriting (fixes TI=Z)
- Phase 4: diminishing returns, not recommended

Added detailed NFQUEUE architecture, TCPOPTSTRIP notes, and
note clarifying P= field in nmap output.
Windows: both 0 (no ICMP rate limiting — matches real Windows behavior)
Linux: 1000ms / mask 6168 (kernel defaults)
BSD: 250ms / mask 6168 (FreeBSD default is faster than Linux)
Embedded/Cisco: both 0 (most firmware doesn't rate-limit ICMP)

These affect nmap's IE and U1 probe groups which measure ICMP error
response timing to closed UDP ports. Windows responds to all probes
instantly while Linux throttles to ~1/sec.

Tests: 10 new cases (5 per sysctl). Suite: 822 passed.
IMAP: extended to full IMAP4rev1 — 10 bait emails (AWS keys, DB creds,
tokens, VPN config, root pw etc.), LIST/LSUB/STATUS/FETCH/UID FETCH/
SEARCH/CLOSE/NOOP, proper SELECT untagged responses (EXISTS, UIDNEXT,
FLAGS, PERMANENTFLAGS), CAPABILITY with IDLE/LITERAL+/AUTH=PLAIN.
FETCH correctly handles sequence sets (1:*, 1:3, *), item dispatch
(FLAGS, ENVELOPE, BODY[], RFC822, RFC822.SIZE), and places body literals
last per RFC 3501.

POP3: extended with same 10 bait emails, fixed banner env var key
(POP3_BANNER not IMAP_BANNER), CAPA fully populated (TOP/UIDL/USER/
RESP-CODES/SASL), TOP (headers + N body lines), UIDL (msg-N format),
DELE/RSET with _deleted set tracking, NOOP. _active_messages() helper
excludes DELE'd messages from STAT/LIST/UIDL.

Both: DEBT-026 stub added (_EMAIL_SEED_PATH env var, documented in
DEBT.md for next-session JSON seed file wiring).

Tests: test_imap.py expanded to 27 cases, test_pop3.py to 22 cases —
860 total tests passing.
The BASE_IMAGE build arg was being unconditionally overwritten by
composer.py with the decky's distro build_base (debian:bookworm-slim),
turning the conpot container into a bare Debian image with no conpot
installation — hence the silent restart loop.

Two fixes:
1. composer.py: use args.setdefault() so services that pre-declare
   BASE_IMAGE in their compose_fragment() win over the distro default.
2. conpot.py: pre-declare BASE_IMAGE=honeynet/conpot:latest in build
   args so it always uses the upstream image regardless of decky distro.

Also removed the USER decnet switch from the conpot Dockerfile. The
upstream image already runs as the non-root 'conpot' user; switching to
'decnet' broke pkg_resources because conpot's eggs live under
/home/conpot/.local and are only on sys.path for that user.
Conpot is a third-party app with its own Python logger — it never calls
decnet_logging. Added entrypoint.py as a subprocess wrapper that:
- Launches conpot and captures its stdout/stderr
- Classifies each line (startup/request/warning/error/log)
- Extracts source IPs via regex
- Emits RFC 5424 syslog lines to stdout for Docker/collector pickup

Entrypoint is self-contained (no import of shared decnet_logging.py)
because the conpot base image runs Python 3.6, which cannot parse the
dict[str, Any] / str | None type syntax used in the canonical file.
When --parallel is set:
- DOCKER_BUILDKIT=1 is injected into the subprocess environment to
  ensure BuildKit is active regardless of host daemon config
- docker compose build runs first (all images built concurrently)
- docker compose up -d follows without --build (no redundant checks)

Without --parallel the original up --build path is preserved.
--parallel and --no-cache compose correctly (build --no-cache).
Two bugs caused the log file to never be written:

1. is_service_container() used regex '^decky-\d+-\w' which only matched
   the old decky-01-smtp naming style. Actual containers are named
   omega-decky-smtp, relay-decky-smtp, etc. Fixed by using Docker Compose
   labels instead: com.docker.compose.project=decnet + non-empty
   depends_on discriminates service containers from base (sleep infinity)
   containers reliably regardless of decky naming convention.
   Added is_service_event() for the Docker events path.

2. The collector was only started when --api was used. Added a 'collect'
   CLI subcommand (decnet collect --log-file <path>) and wired it into
   deploy as an auto-started background process when --api is not in use.
   Default log path: /var/log/decnet/decnet.log
_load_service_container_names() reads decnet-state.json and builds the
exact set of expected container names ({decky}-{service}). is_service_container()
and is_service_event() do a direct set lookup — no regex, no label
inspection, no heuristics.
Scraps the Cowrie emulation layer. The real_ssh template now runs a
genuine sshd backed by a three-layer logging stack forwarded to stdout
as RFC 5424 for the DECNET collector:

  auth,authpriv.*  → rsyslogd → named pipe → stdout  (logins/failures)
  user.*           → rsyslogd → named pipe → stdout  (PROMPT_COMMAND cmds)
  sudo syslog=auth → rsyslogd → named pipe → stdout  (privilege escalation)
  sudo logfile     → /var/log/sudo.log               (local backup with I/O)

The ssh.py service plugin now points to templates/real_ssh and drops all
COWRIE_* / NODE_NAME env vars, sharing the same compose fragment shape as
real_ssh.py.
Collector and mutator watcher subprocesses were spawned without
start_new_session=True, leaving them in the parent's process group.
SIGHUP (sent when the controlling terminal closes) killed both
processes silently — stdout/stderr were DEVNULL so the crash was
invisible.

Also update test_services and test_composer to reflect the ssh plugin
no longer using Cowrie env vars (replaced with SSH_ROOT_PASSWORD /
SSH_HOSTNAME matching the real_ssh plugin).
The collector subprocess was spawned via 'python3 -m decnet.cli collect'
but cli.py had no 'if __name__ == __main__: app()' guard. Python executed
the module, defined all functions, then exited cleanly with code 0 without
ever calling the collect command. No output, no log file, exit 0 — silent
non-start every time.

Also route collector stderr to <log_file>.collector.log so future crashes
are visible instead of disappearing into DEVNULL.
real_ssh was a separate service name pointing to the same template and
behaviour as ssh. Merged them: ssh is now the single real-OpenSSH service.

- Rename templates/real_ssh/ → templates/ssh/
- Remove decnet/services/real_ssh.py
- Deaddeck archetype updated: services=["ssh"]
- Merge test_real_ssh.py into test_ssh.py (includes deaddeck + logging tests)
- Drop decnet.services.real_ssh from test_build module list
- decnet/engine/ — container lifecycle (deploy, teardown, status); _kill_api removed
- decnet/collector/ — Docker log streaming (moved from web/collector.py)
- decnet/mutator/ — mutation engine (no longer imports from cli or duplicates deployer code)
- decnet/fleet.py — shared decky-building logic extracted from cli.py

Cross-contamination eliminated:
- web router no longer imports from decnet.cli
- mutator no longer imports from decnet.cli
- cli no longer imports from decnet.web
- _kill_api() moved to cli (process management, not engine concern)
- _compose_with_retry duplicate removed from mutator
Cowrie was exposing an SSH daemon on port 22 alongside the telnet service
even when COWRIE_SSH_ENABLED=false, contaminating deployments that did not
request an SSH service.

New implementation mirrors the SSH service pattern:
- busybox telnetd in foreground mode on port 23
- /bin/login for real PAM authentication (brute-force attempts logged)
- rsyslog RFC 5424 bridge piped to stdout for Docker log capture
- Configurable root password and hostname via env vars
- No Cowrie dependency
MongoDB had the same infinite-loop bug as MSSQL (msg_len=0 → buffer never
shrinks in while loop). Postgres, MySQL, and MQTT had related length-field
issues (stuck state, resource exhaustion, overlong remaining-length).

Also fixes an existing MongoDB _op_reply struct.pack format bug (extra 'q'
specifier caused struct.error on any OP_QUERY response).

Adds 53 regression + protocol boundary tests across MSSQL, MongoDB,
Postgres, MySQL, and MQTT, including a _run_with_timeout threading harness
to catch infinite loops and @pytest.mark.fuzz hypothesis tests for each.
Spins up each service's server.py in a real subprocess via a free ephemeral
port (PORT env var), connects with real protocol clients, and asserts both
correct protocol behavior and RFC 5424 log output.

- 44 live tests across 10 services: http, ftp, smtp, redis, mqtt,
  mysql, postgres, mongodb, pop3, imap
- Shared conftest.py: _ServiceProcess (bg reader thread + queue),
  free_port, live_service fixture, assert_rfc5424 helper
- PORT env var added to all 10 targeted server.py templates
- New pytest marker `live`; excluded from default addopts run
- requirements-live-tests.txt: flask, twisted + protocol clients
modified: .gitignore
Some checks failed
CI / Lint (ruff) (push) Failing after 16s
CI / Test (pytest) (3.11) (push) Failing after 47s
CI / Test (pytest) (3.12) (push) Failing after 49s
CI / SAST (bandit) (push) Successful in 12s
CI / Dependency audit (pip-audit) (push) Successful in 23s
CI / Open PR to main (push) Has been skipped
c3c1cd2fa6
- Add merge-to-testing job: after all CI checks pass on dev, auto-merge
  into testing with --no-ff for clear merge history
- Move open-pr job to trigger on testing branch instead of dev
- PR now opens testing → main instead of dev → main
- Add bandit and pip-audit jobs to pr.yml PR gate for full suite coverage
- PR gate test job now installs dev dependencies consistently
Ruff fixes (20 errors → 0):
- F401: Remove unused imports (DeckyConfig, random_hostname, IniConfig,
  COMPOSE_FILE, sys, patch) across cli.py, mutator/engine.py,
  templates/ftp, templates/rdp, test_mysql.py, test_postgres.py
- F541: Remove extraneous f-prefixes on strings with no placeholders
  in templates/imap, test_ftp_live, test_http_live
- E741: Rename ambiguous variable 'l' to descriptive names (line, entry,
  part) across conftest.py, test_ftp_live, test_http_live,
  test_mongodb_live, test_pop3, test_ssh

SQLite fix:
- Change _initialize_sync() admin seeding from SELECT-then-INSERT to
  INSERT OR IGNORE, preventing IntegrityError when admin user already
  exists from a previous run
- Fixed CLI tests by patching local imports at source (psutil, os, Path).
- Fixed Collector tests by globalizing docker.from_env mock.
- Stabilized SSE stream tests via AsyncMock and immediate generator termination to prevent hangs.
- Achieved >80% coverage on CLI (84%), Collector (97%), and DB Repository (100%).
- Implemented SMTP Relay service tests (100%).
chore: fix unused imports in tests and update development roadmap
Some checks failed
CI / Lint (ruff) (push) Successful in 16s
CI / Test (pytest) (3.11) (push) Failing after 34s
CI / Test (pytest) (3.12) (push) Failing after 36s
CI / SAST (bandit) (push) Successful in 12s
CI / Merge dev → testing (push) Has been cancelled
CI / Open PR to main (push) Has been cancelled
CI / Dependency audit (pip-audit) (push) Has been cancelled
0f63820ee6
modified: pyproject, moved [live] deps to [dev] deps.
All checks were successful
CI / Lint (ruff) (push) Successful in 11s
CI / Test (pytest) (3.11) (push) Successful in 1m19s
CI / Test (pytest) (3.12) (push) Successful in 1m22s
CI / SAST (bandit) (push) Successful in 12s
CI / Dependency audit (pip-audit) (push) Successful in 21s
CI / Merge dev → testing (push) Successful in 10s
CI / Open PR to main (push) Has been skipped
fe18575a9c
ci: auto-merge dev → testing
All checks were successful
CI / Lint (ruff) (push) Successful in 11s
CI / Test (pytest) (3.11) (push) Successful in 1m9s
CI / Test (pytest) (3.12) (push) Successful in 1m14s
CI / SAST (bandit) (push) Successful in 12s
CI / Dependency audit (pip-audit) (push) Successful in 21s
CI / Merge dev → testing (push) Has been skipped
CI / Open PR to main (push) Successful in 6s
PR Gate / Lint (ruff) (pull_request) Successful in 11s
PR Gate / Test (pytest) (3.11) (pull_request) Successful in 1m13s
PR Gate / Test (pytest) (3.12) (pull_request) Successful in 1m12s
PR Gate / SAST (bandit) (pull_request) Successful in 13s
PR Gate / Dependency audit (pip-audit) (pull_request) Successful in 21s
ac4e5e1570
Author
Owner
fixed in: [Merge pull request 'fix/merge-testing-to-main' (#4) from fix/merge-testing-to-main into main](https://git.resacachile.cl/anti/DECNET/commit/23ec4709881b9c1c022700d65e61adc75a4db45a) closing.
anti closed this pull request 2026-04-12 10:18:09 +02:00
All checks were successful
CI / Lint (ruff) (push) Successful in 11s
CI / Test (pytest) (3.11) (push) Successful in 1m9s
CI / Test (pytest) (3.12) (push) Successful in 1m14s
CI / SAST (bandit) (push) Successful in 12s
CI / Dependency audit (pip-audit) (push) Successful in 21s
CI / Merge dev → testing (push) Has been skipped
CI / Open PR to main (push) Successful in 6s
PR Gate / Lint (ruff) (pull_request) Successful in 11s
PR Gate / Test (pytest) (3.11) (pull_request) Successful in 1m13s
PR Gate / Test (pytest) (3.12) (pull_request) Successful in 1m12s
PR Gate / SAST (bandit) (pull_request) Successful in 13s
PR Gate / Dependency audit (pip-audit) (pull_request) Successful in 21s

Pull request closed

Sign in to join this conversation.
No Reviewers
No Label
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: anti/DECNET#3