# Database Module

Dual-backend persistence layer. **All new persistence features must support both backends.**

## Quick Reference

```bash
# Default build (PostgreSQL)
cargo build

# libSQL/Turso build
cargo build --no-default-features --features libsql

# Both backends
cargo build --features "postgres,libsql"

# Test each backend in isolation
cargo check                                           # postgres (default)
cargo check --no-default-features --features libsql   # libsql only
cargo check --all-features                            # both
```

## Files

| File | Role |
|------|------|
| `mod.rs` | `Database` supertrait + 7 sub-traits (~78 async methods total) — add new ops here first |
| `postgres.rs` | PostgreSQL backend — delegates to `Store` + `Repository` in `history/` |
| `libsql/mod.rs` | libSQL/Turso backend struct, connection helpers, row parsing utilities |
| `libsql/conversations.rs` | `ConversationStore` impl |
| `libsql/jobs.rs` | `JobStore` impl |
| `libsql/sandbox.rs` | `SandboxStore` impl |
| `libsql/routines.rs` | `RoutineStore` impl |
| `libsql/settings.rs` | `SettingsStore` impl |
| `libsql/tool_failures.rs` | `ToolFailureStore` impl |
| `libsql/workspace.rs` | `WorkspaceStore` impl (FTS5 + vector search) |
| `libsql_migrations.rs` | Consolidated libSQL schema (CREATE IF NOT EXISTS, no ALTER TABLE) |
| `tls.rs` | TLS connector factory for PostgreSQL (`rustls` + system root certs) |

PostgreSQL schema: `migrations/V1__initial.sql` through `V9__flexible_embedding_dimension.sql` (managed by `refinery`). V1 is the base schema; later migrations add tables, columns, and rename `claude_code_events` → `job_events`.

## Trait Structure

The `Database` supertrait is composed of seven sub-traits. Leaf consumers can depend on the narrowest sub-trait they need rather than the full `Database`:

| Sub-trait | Methods | Covers |
|-----------|---------|--------|
| `ConversationStore` | 12 | Conversations, messages |
| `JobStore` | 13 | Agent jobs, actions, LLM calls, estimation |
| `SandboxStore` | 13 | Sandbox jobs, job events |
| `RoutineStore` | 15 | Routines, routine runs |
| `ToolFailureStore` | 4 | Self-repair tracking |
| `SettingsStore` | 8 | Per-user key-value settings |
| `WorkspaceStore` | 13 | Memory documents, chunks, hybrid search |

`Database` adds `run_migrations()` and combines all sub-traits.

## Adding a New Persistence Operation

1. Decide which sub-trait the method belongs to, or create a new sub-trait
2. Add the async method signature to that sub-trait in `mod.rs`
3. Implement in `postgres.rs` (delegate to `Store` or `Repository`)
4. Implement in `libsql/<module>.rs` (SQLite-dialect SQL, use `self.connect().await?` per operation)
5. Add migration if needed:
   - PostgreSQL: new `migrations/VN__description.sql`
   - libSQL: add `CREATE TABLE IF NOT EXISTS` to `libsql_migrations.rs`

## SQL Dialect Differences

| Feature | PostgreSQL | libSQL |
|---------|-----------|--------|
| UUIDs | `UUID` type | `TEXT` |
| Timestamps | `TIMESTAMPTZ` | `TEXT` (ISO-8601 RFC 3339 with ms precision) |
| JSON | `JSONB` | `TEXT` |
| Numeric/Decimal | `NUMERIC` | `TEXT` (preserves `rust_decimal` precision) |
| Arrays | `TEXT[]` | `TEXT` (JSON-encoded array) |
| Booleans | `BOOLEAN` | `INTEGER` (0/1) |
| Vector embeddings | `VECTOR` (any dim, V9 removed fixed 1536) | `F32_BLOB(1536)` via `libsql_vector_idx` |
| Full-text search | `tsvector` + `ts_rank_cd` | FTS5 virtual table + sync triggers |
| JSON path update | `jsonb_set(col, '{key}', val)` | `json_patch(col, '{"key": val}')` |
| PL/pgSQL | Functions | Triggers (no stored procs in SQLite) |
| Connection model | `deadpool-postgres` connection pool | New connection per operation (`self.connect()`) |
| Concurrency | Pool-based, fully concurrent | WAL mode + 5 s busy timeout; write serialized |
| Auto-timestamp | `DEFAULT NOW()` | `DEFAULT (datetime('now'))` |
| Timestamp parsing | Native type | Multi-format fallback in `parse_timestamp()` |

**JSON merge patch gotcha:** libSQL uses RFC 7396 JSON Merge Patch (`json_patch`) for metadata updates. This replaces top-level keys entirely — it **cannot** do partial nested updates. PostgreSQL uses `jsonb_set` which is path-targeted. Don't rely on partial nested metadata updates if you need libSQL compat.

**Boolean storage:** libSQL stores booleans as integers. When reading, use `get_i64(row, idx) != 0`; when writing, pass `1i64`/`0i64`. Never pass a Rust `bool` directly.

**Timestamp write format:** Always write timestamps with `fmt_ts(dt)` (RFC 3339, millisecond precision). Read with `get_ts()` / `get_opt_ts()` which handle legacy naive formats too.

**Vector dimension:** PostgreSQL V9 migration changed the column to unbounded `vector` (removing the HNSW index). libSQL still uses `F32_BLOB(1536)` — if you use a different-dimension embedding model, the libSQL schema needs updating too.

**Connection per operation:** `LibSqlBackend::connect()` creates a fresh connection for every operation, sets `PRAGMA busy_timeout = 5000`, and closes it when the `Connection` is dropped. This is intentional — the libSQL SDK does not offer a pool. Avoid holding connections open across `await` points.

## Schema: Key Tables

**Core:**
- `conversations` — multi-channel conversation tracking
- `conversation_messages` — individual messages within a conversation
- `agent_jobs` — job metadata and status
- `job_actions` — event-sourced tool executions
- `job_events` — sandbox job streaming events (renamed from `claude_code_events` in V7)
- `dynamic_tools` — agent-built tools
- `llm_calls` — cost/token tracking
- `estimation_snapshots` — learning data
- `repair_attempts` — self-repair action log (not exposed via Database trait yet)

**Workspace/Memory:**
- `memory_documents` — flexible path-based files
- `memory_chunks` — chunked content with FTS + vector indexes
- `memory_chunks_fts` — FTS5 virtual table (libSQL) / `tsvector` column (PostgreSQL)
- `heartbeat_state` — periodic execution tracking

**Security/Extensions:**
- `secrets` — AES-256-GCM encrypted credentials
- `wasm_tools` — installed WASM tool binaries
- `tool_capabilities` — per-tool HTTP allowlist, secret access, rate limits
- `leak_detection_patterns` — secret regex patterns (seed data in both backends)
- `leak_detection_events` — audit log of detected leaks
- `secret_usage_log` — per-request credential injection audit trail
- `tool_rate_limit_state` — sliding window rate limit counters

**Other:**
- `routines`, `routine_runs` — scheduled/reactive execution
- `settings` — per-user key-value
- `tool_failures` — broken tool tracking for self-repair
- `_migrations` — libSQL-only internal migration version tracking

## libSQL Current Limitations

- **Secrets store** — still requires `PostgresSecretsStore`; `LibSqlSecretsStore` exists but is not plumbed through the main startup path
- **Settings reload** — `Config::from_db` skipped (requires `Store`)
- **No incremental migrations** — schema is idempotent CREATE IF NOT EXISTS; no ALTER TABLE support; column additions require a new versioned approach
- **No encryption at rest** — only secrets (API tokens) are AES-256-GCM encrypted; all other data is plaintext SQLite
- **Hybrid search** — both FTS5 and vector search (`libsql_vector_idx`) are implemented; however, the vector index is fixed at `F32_BLOB(1536)` while PostgreSQL switched to unbounded `vector` in V9
- **Write serialization** — WAL mode allows concurrent readers but only one writer at a time; busy timeout is 5 s, which may cause timeouts under high write concurrency

## Running Locally with libSQL

```bash
# Use local SQLite file (default)
DATABASE_BACKEND=libsql LIBSQL_PATH=~/.ironclaw/test.db cargo run

# Use Turso cloud (embedded replica syncs local file to cloud)
DATABASE_BACKEND=libsql LIBSQL_URL=libsql://xxx.turso.io LIBSQL_AUTH_TOKEN=xxx cargo run

# In-memory (tests only — data is lost when the process exits)
# Use LibSqlBackend::new_memory() directly in test code
```

## Testing the libSQL Backend

Use `LibSqlBackend::new_memory()` in unit tests — no files, no cleanup required:

```rust
#[tokio::test]
async fn test_my_feature() {
    let backend = LibSqlBackend::new_memory().await.unwrap();
    backend.run_migrations().await.unwrap();
    // backend implements Database — call any trait method
}
```

For concurrency tests that require multiple connections sharing state, use `LibSqlBackend::new_local(&tmp_path)` with a `tempfile::tempdir()`. In-memory databases do not share state between connections.

## Sharing the libSQL Database Handle

`LibSqlBackend::shared_db()` returns an `Arc<LibSqlDatabase>` for passing to satellite stores (e.g., `LibSqlSecretsStore`, `LibSqlWasmToolStore`) that need their own connections per-operation but should share the same underlying database file. These stores call `.connect()` on the shared handle themselves. This is the correct pattern — do not pass a live `Connection` to satellite stores.

## Pattern: Fix the Pattern, Not the Instance

When fixing a bug in one backend's SQL, always grep for the same pattern in the other backend. A fix to `postgres.rs` that doesn't also fix the libSQL module (e.g., `libsql/jobs.rs`) is half a fix. The same applies to satellite types like `LibSqlSecretsStore` or `LibSqlWasmToolStore`.