diff --git a/Database-Drivers.md b/Database-Drivers.md new file mode 100644 index 0000000..824d2dc --- /dev/null +++ b/Database-Drivers.md @@ -0,0 +1,123 @@ +# Database Drivers + +DECNET persists dashboard state, deployment metadata, event/capture rows, and auth +data through a single `BaseRepository` interface. Two concrete backends implement +that interface: SQLite (default) and MySQL. Both extend the same +`SQLModelRepository` base and only override the dialect-specific bits (engine +construction, URL resolution, schema bootstrap). This is a load-bearing design +choice — adding a new backend means subclassing the shared repo, not reimplementing +the query surface. + +Source: `decnet/web/db/repository.py`, `decnet/web/db/sqlmodel_repo.py`, +`decnet/web/db/sqlite/repository.py`, `decnet/web/db/mysql/repository.py`. + +## Why two backends + +- **SQLite** fits single-host UNIHOST deploys. Zero setup, file-backed, + limited to one writer at a time. +- **MySQL** fits SWARM and any deployment with multi-process ingest or high + write volume (many sniffer/capture workers pushing concurrently). Scales + horizontally and survives multi-writer contention that SQLite cannot. + +The wire format, ORM models, and repository API are identical — only the engine +and connection URL change. + +## SQLite (default) + +Driver: `aiosqlite >= 0.20.0` (see `pyproject.toml`). Pulled in automatically +by `pip install -e .` — nothing else to install. + +Behavior: + +- WAL journal mode and `synchronous=NORMAL` are enabled on every connect + (`decnet/web/db/sqlite/database.py`). +- Default `busy_timeout` is 30s to absorb short write contention. +- The DB file lives at the path passed to `SQLiteRepository(db_path=...)` + (defaults set by the caller; repo bootstrap will `create_all` on first use). +- WAL sidecar files (`*-wal`, `*-shm`) are expected and are ignored by + `.gitignore`. + +Limits: + +- Single-writer. Concurrent writes serialize; under heavy load you will see + `SQLITE_BUSY`. Raise `DECNET_DB_POOL_SIZE` / timeouts only as a stopgap — + switch to MySQL if contention is real. +- Not safe to share across hosts over NFS or similar. + +## MySQL + +Driver: `asyncmy >= 0.2.9` (see `pyproject.toml`). Installed by +`pip install -e .`. No separate package step. + +### Setup + +Create the database and a dedicated user: + +```sql +CREATE DATABASE decnet CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci; +CREATE USER 'decnet'@'%' IDENTIFIED BY 'change-me'; +GRANT ALL PRIVILEGES ON decnet.* TO 'decnet'@'%'; +FLUSH PRIVILEGES; +``` + +Point DECNET at it with either a full URL or component vars (see +[Environment variables](Environment-Variables) for the complete list): + +```bash +# Option A — full URL +export DECNET_DB_TYPE=mysql +export DECNET_DB_URL="mysql+asyncmy://decnet:change-me@db.internal:3306/decnet" + +# Option B — components (password is percent-encoded automatically) +export DECNET_DB_TYPE=mysql +export DECNET_DB_HOST=db.internal +export DECNET_DB_PORT=3306 +export DECNET_DB_NAME=decnet +export DECNET_DB_USER=decnet +export DECNET_DB_PASSWORD='change-me' +``` + +Precedence: explicit `url=` kwarg to `get_async_engine` > `DECNET_DB_URL` > +components. An empty password outside pytest raises — this is intentional. + +Schema bootstrap: tables are created via `SQLModel.metadata.create_all` on the +first repository initialization. There is no migrator. + +### Tuning + +Pool sizing and recycle knobs (`DECNET_DB_POOL_SIZE`, `DECNET_DB_MAX_OVERFLOW`, +`DECNET_DB_POOL_RECYCLE`, `DECNET_DB_POOL_PRE_PING`) apply to both backends +and are documented on [Environment variables](Environment-Variables). MySQL +defaults `pool_pre_ping=true` to fail fast on dropped idle connections; SQLite +leaves it off since the "server" is a local file. + +## Switching backends + +Set `DECNET_DB_TYPE` to `sqlite` or `mysql` and restart the web/CLI process. +There is **no migration tool** — switching backends starts from an empty +schema. Export anything you care about beforehand. For MySQL teardown / +rebuild during development, `decnet db-reset` exists (MySQL-only; see +`decnet/cli.py`). + +## Factory and DI + +All code paths that need a repository must go through the factory: + +```python +from decnet.web.db.factory import get_repository + +repo = get_repository() # DECNET_DB_TYPE decides the subclass +``` + +In FastAPI handlers, depend on `get_repo` from `decnet/web/dependencies.py` +instead of constructing a repo inline. The factory also wraps the instance +with telemetry (`decnet.telemetry.wrap_repository`), so bypassing it loses +metrics. + +**Rule (from `CLAUDE.md`):** never import `SQLiteRepository` (or +`MySQLRepository`) directly in feature code. Use `get_repository()` or the +`get_repo` FastAPI dependency. Direct imports break backend switching and +silently drop telemetry wrapping. + +See the [Developer guide](Developer-Guide) for how repository-bound services +are wired into handlers and background workers.