docs(wiki): add Database Drivers page (unit 9)
123
Database-Drivers.md
Normal file
123
Database-Drivers.md
Normal file
@@ -0,0 +1,123 @@
|
|||||||
|
# Database Drivers
|
||||||
|
|
||||||
|
DECNET persists dashboard state, deployment metadata, event/capture rows, and auth
|
||||||
|
data through a single `BaseRepository` interface. Two concrete backends implement
|
||||||
|
that interface: SQLite (default) and MySQL. Both extend the same
|
||||||
|
`SQLModelRepository` base and only override the dialect-specific bits (engine
|
||||||
|
construction, URL resolution, schema bootstrap). This is a load-bearing design
|
||||||
|
choice — adding a new backend means subclassing the shared repo, not reimplementing
|
||||||
|
the query surface.
|
||||||
|
|
||||||
|
Source: `decnet/web/db/repository.py`, `decnet/web/db/sqlmodel_repo.py`,
|
||||||
|
`decnet/web/db/sqlite/repository.py`, `decnet/web/db/mysql/repository.py`.
|
||||||
|
|
||||||
|
## Why two backends
|
||||||
|
|
||||||
|
- **SQLite** fits single-host UNIHOST deploys. Zero setup, file-backed,
|
||||||
|
limited to one writer at a time.
|
||||||
|
- **MySQL** fits SWARM and any deployment with multi-process ingest or high
|
||||||
|
write volume (many sniffer/capture workers pushing concurrently). Scales
|
||||||
|
horizontally and survives multi-writer contention that SQLite cannot.
|
||||||
|
|
||||||
|
The wire format, ORM models, and repository API are identical — only the engine
|
||||||
|
and connection URL change.
|
||||||
|
|
||||||
|
## SQLite (default)
|
||||||
|
|
||||||
|
Driver: `aiosqlite >= 0.20.0` (see `pyproject.toml`). Pulled in automatically
|
||||||
|
by `pip install -e .` — nothing else to install.
|
||||||
|
|
||||||
|
Behavior:
|
||||||
|
|
||||||
|
- WAL journal mode and `synchronous=NORMAL` are enabled on every connect
|
||||||
|
(`decnet/web/db/sqlite/database.py`).
|
||||||
|
- Default `busy_timeout` is 30s to absorb short write contention.
|
||||||
|
- The DB file lives at the path passed to `SQLiteRepository(db_path=...)`
|
||||||
|
(defaults set by the caller; repo bootstrap will `create_all` on first use).
|
||||||
|
- WAL sidecar files (`*-wal`, `*-shm`) are expected and are ignored by
|
||||||
|
`.gitignore`.
|
||||||
|
|
||||||
|
Limits:
|
||||||
|
|
||||||
|
- Single-writer. Concurrent writes serialize; under heavy load you will see
|
||||||
|
`SQLITE_BUSY`. Raise `DECNET_DB_POOL_SIZE` / timeouts only as a stopgap —
|
||||||
|
switch to MySQL if contention is real.
|
||||||
|
- Not safe to share across hosts over NFS or similar.
|
||||||
|
|
||||||
|
## MySQL
|
||||||
|
|
||||||
|
Driver: `asyncmy >= 0.2.9` (see `pyproject.toml`). Installed by
|
||||||
|
`pip install -e .`. No separate package step.
|
||||||
|
|
||||||
|
### Setup
|
||||||
|
|
||||||
|
Create the database and a dedicated user:
|
||||||
|
|
||||||
|
```sql
|
||||||
|
CREATE DATABASE decnet CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
|
||||||
|
CREATE USER 'decnet'@'%' IDENTIFIED BY 'change-me';
|
||||||
|
GRANT ALL PRIVILEGES ON decnet.* TO 'decnet'@'%';
|
||||||
|
FLUSH PRIVILEGES;
|
||||||
|
```
|
||||||
|
|
||||||
|
Point DECNET at it with either a full URL or component vars (see
|
||||||
|
[Environment variables](Environment-Variables) for the complete list):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Option A — full URL
|
||||||
|
export DECNET_DB_TYPE=mysql
|
||||||
|
export DECNET_DB_URL="mysql+asyncmy://decnet:change-me@db.internal:3306/decnet"
|
||||||
|
|
||||||
|
# Option B — components (password is percent-encoded automatically)
|
||||||
|
export DECNET_DB_TYPE=mysql
|
||||||
|
export DECNET_DB_HOST=db.internal
|
||||||
|
export DECNET_DB_PORT=3306
|
||||||
|
export DECNET_DB_NAME=decnet
|
||||||
|
export DECNET_DB_USER=decnet
|
||||||
|
export DECNET_DB_PASSWORD='change-me'
|
||||||
|
```
|
||||||
|
|
||||||
|
Precedence: explicit `url=` kwarg to `get_async_engine` > `DECNET_DB_URL` >
|
||||||
|
components. An empty password outside pytest raises — this is intentional.
|
||||||
|
|
||||||
|
Schema bootstrap: tables are created via `SQLModel.metadata.create_all` on the
|
||||||
|
first repository initialization. There is no migrator.
|
||||||
|
|
||||||
|
### Tuning
|
||||||
|
|
||||||
|
Pool sizing and recycle knobs (`DECNET_DB_POOL_SIZE`, `DECNET_DB_MAX_OVERFLOW`,
|
||||||
|
`DECNET_DB_POOL_RECYCLE`, `DECNET_DB_POOL_PRE_PING`) apply to both backends
|
||||||
|
and are documented on [Environment variables](Environment-Variables). MySQL
|
||||||
|
defaults `pool_pre_ping=true` to fail fast on dropped idle connections; SQLite
|
||||||
|
leaves it off since the "server" is a local file.
|
||||||
|
|
||||||
|
## Switching backends
|
||||||
|
|
||||||
|
Set `DECNET_DB_TYPE` to `sqlite` or `mysql` and restart the web/CLI process.
|
||||||
|
There is **no migration tool** — switching backends starts from an empty
|
||||||
|
schema. Export anything you care about beforehand. For MySQL teardown /
|
||||||
|
rebuild during development, `decnet db-reset` exists (MySQL-only; see
|
||||||
|
`decnet/cli.py`).
|
||||||
|
|
||||||
|
## Factory and DI
|
||||||
|
|
||||||
|
All code paths that need a repository must go through the factory:
|
||||||
|
|
||||||
|
```python
|
||||||
|
from decnet.web.db.factory import get_repository
|
||||||
|
|
||||||
|
repo = get_repository() # DECNET_DB_TYPE decides the subclass
|
||||||
|
```
|
||||||
|
|
||||||
|
In FastAPI handlers, depend on `get_repo` from `decnet/web/dependencies.py`
|
||||||
|
instead of constructing a repo inline. The factory also wraps the instance
|
||||||
|
with telemetry (`decnet.telemetry.wrap_repository`), so bypassing it loses
|
||||||
|
metrics.
|
||||||
|
|
||||||
|
**Rule (from `CLAUDE.md`):** never import `SQLiteRepository` (or
|
||||||
|
`MySQLRepository`) directly in feature code. Use `get_repository()` or the
|
||||||
|
`get_repo` FastAPI dependency. Direct imports break backend switching and
|
||||||
|
silently drop telemetry wrapping.
|
||||||
|
|
||||||
|
See the [Developer guide](Developer-Guide) for how repository-bound services
|
||||||
|
are wired into handlers and background workers.
|
||||||
Reference in New Issue
Block a user