docs(wiki): add Database Drivers page (unit 9)
123
Database-Drivers.md
Normal file
123
Database-Drivers.md
Normal file
@@ -0,0 +1,123 @@
|
||||
# Database Drivers
|
||||
|
||||
DECNET persists dashboard state, deployment metadata, event/capture rows, and auth
|
||||
data through a single `BaseRepository` interface. Two concrete backends implement
|
||||
that interface: SQLite (default) and MySQL. Both extend the same
|
||||
`SQLModelRepository` base and only override the dialect-specific bits (engine
|
||||
construction, URL resolution, schema bootstrap). This is a load-bearing design
|
||||
choice — adding a new backend means subclassing the shared repo, not reimplementing
|
||||
the query surface.
|
||||
|
||||
Source: `decnet/web/db/repository.py`, `decnet/web/db/sqlmodel_repo.py`,
|
||||
`decnet/web/db/sqlite/repository.py`, `decnet/web/db/mysql/repository.py`.
|
||||
|
||||
## Why two backends
|
||||
|
||||
- **SQLite** fits single-host UNIHOST deploys. Zero setup, file-backed,
|
||||
limited to one writer at a time.
|
||||
- **MySQL** fits SWARM and any deployment with multi-process ingest or high
|
||||
write volume (many sniffer/capture workers pushing concurrently). Scales
|
||||
horizontally and survives multi-writer contention that SQLite cannot.
|
||||
|
||||
The wire format, ORM models, and repository API are identical — only the engine
|
||||
and connection URL change.
|
||||
|
||||
## SQLite (default)
|
||||
|
||||
Driver: `aiosqlite >= 0.20.0` (see `pyproject.toml`). Pulled in automatically
|
||||
by `pip install -e .` — nothing else to install.
|
||||
|
||||
Behavior:
|
||||
|
||||
- WAL journal mode and `synchronous=NORMAL` are enabled on every connect
|
||||
(`decnet/web/db/sqlite/database.py`).
|
||||
- Default `busy_timeout` is 30s to absorb short write contention.
|
||||
- The DB file lives at the path passed to `SQLiteRepository(db_path=...)`
|
||||
(defaults set by the caller; repo bootstrap will `create_all` on first use).
|
||||
- WAL sidecar files (`*-wal`, `*-shm`) are expected and are ignored by
|
||||
`.gitignore`.
|
||||
|
||||
Limits:
|
||||
|
||||
- Single-writer. Concurrent writes serialize; under heavy load you will see
|
||||
`SQLITE_BUSY`. Raise `DECNET_DB_POOL_SIZE` / timeouts only as a stopgap —
|
||||
switch to MySQL if contention is real.
|
||||
- Not safe to share across hosts over NFS or similar.
|
||||
|
||||
## MySQL
|
||||
|
||||
Driver: `asyncmy >= 0.2.9` (see `pyproject.toml`). Installed by
|
||||
`pip install -e .`. No separate package step.
|
||||
|
||||
### Setup
|
||||
|
||||
Create the database and a dedicated user:
|
||||
|
||||
```sql
|
||||
CREATE DATABASE decnet CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
|
||||
CREATE USER 'decnet'@'%' IDENTIFIED BY 'change-me';
|
||||
GRANT ALL PRIVILEGES ON decnet.* TO 'decnet'@'%';
|
||||
FLUSH PRIVILEGES;
|
||||
```
|
||||
|
||||
Point DECNET at it with either a full URL or component vars (see
|
||||
[Environment variables](Environment-Variables) for the complete list):
|
||||
|
||||
```bash
|
||||
# Option A — full URL
|
||||
export DECNET_DB_TYPE=mysql
|
||||
export DECNET_DB_URL="mysql+asyncmy://decnet:change-me@db.internal:3306/decnet"
|
||||
|
||||
# Option B — components (password is percent-encoded automatically)
|
||||
export DECNET_DB_TYPE=mysql
|
||||
export DECNET_DB_HOST=db.internal
|
||||
export DECNET_DB_PORT=3306
|
||||
export DECNET_DB_NAME=decnet
|
||||
export DECNET_DB_USER=decnet
|
||||
export DECNET_DB_PASSWORD='change-me'
|
||||
```
|
||||
|
||||
Precedence: explicit `url=` kwarg to `get_async_engine` > `DECNET_DB_URL` >
|
||||
components. An empty password outside pytest raises — this is intentional.
|
||||
|
||||
Schema bootstrap: tables are created via `SQLModel.metadata.create_all` on the
|
||||
first repository initialization. There is no migrator.
|
||||
|
||||
### Tuning
|
||||
|
||||
Pool sizing and recycle knobs (`DECNET_DB_POOL_SIZE`, `DECNET_DB_MAX_OVERFLOW`,
|
||||
`DECNET_DB_POOL_RECYCLE`, `DECNET_DB_POOL_PRE_PING`) apply to both backends
|
||||
and are documented on [Environment variables](Environment-Variables). MySQL
|
||||
defaults `pool_pre_ping=true` to fail fast on dropped idle connections; SQLite
|
||||
leaves it off since the "server" is a local file.
|
||||
|
||||
## Switching backends
|
||||
|
||||
Set `DECNET_DB_TYPE` to `sqlite` or `mysql` and restart the web/CLI process.
|
||||
There is **no migration tool** — switching backends starts from an empty
|
||||
schema. Export anything you care about beforehand. For MySQL teardown /
|
||||
rebuild during development, `decnet db-reset` exists (MySQL-only; see
|
||||
`decnet/cli.py`).
|
||||
|
||||
## Factory and DI
|
||||
|
||||
All code paths that need a repository must go through the factory:
|
||||
|
||||
```python
|
||||
from decnet.web.db.factory import get_repository
|
||||
|
||||
repo = get_repository() # DECNET_DB_TYPE decides the subclass
|
||||
```
|
||||
|
||||
In FastAPI handlers, depend on `get_repo` from `decnet/web/dependencies.py`
|
||||
instead of constructing a repo inline. The factory also wraps the instance
|
||||
with telemetry (`decnet.telemetry.wrap_repository`), so bypassing it loses
|
||||
metrics.
|
||||
|
||||
**Rule (from `CLAUDE.md`):** never import `SQLiteRepository` (or
|
||||
`MySQLRepository`) directly in feature code. Use `get_repository()` or the
|
||||
`get_repo` FastAPI dependency. Direct imports break backend switching and
|
||||
silently drop telemetry wrapping.
|
||||
|
||||
See the [Developer guide](Developer-Guide) for how repository-bound services
|
||||
are wired into handlers and background workers.
|
||||
Reference in New Issue
Block a user