From 633b234e44055fb2d46476752dff4121880c57b2 Mon Sep 17 00:00:00 2001 From: Henry Park Date: Sat, 7 Mar 2026 00:33:09 -0800 Subject: [PATCH] docs: add comprehensive subdirectory CLAUDE.md files and update root (#589) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * docs: add comprehensive subdirectory CLAUDE.md files and update root The repo has grown significantly. This adds module-level CLAUDE.md files for the five most complex subsystems, and updates the root CLAUDE.md to reflect the actual current state of the codebase. New files: - src/agent/CLAUDE.md — full module map (19 files), session/thread/turn model, agentic loop flow, compaction strategies with correct thresholds, scheduler invariants, self-repair details, complete submission command reference table - src/channels/web/CLAUDE.md — complete API route table (50+ endpoints), SSE event type reference, auth/rate limiting gotchas, connection limits, CORS headers, step-by-step endpoint addition guide - src/db/CLAUDE.md — dual-backend build commands, sub-trait structure (7 sub-traits, ~67 methods), SQL dialect differences, boolean/timestamp gotchas, complete schema table, in-memory test helper, shared handle pattern - src/llm/CLAUDE.md — corrected LlmProvider trait signatures, provider chain decorator order, NEAR AI dual-auth and session renewal details, circuit breaker thresholds, previously undocumented smart_routing.rs and recording.rs - tests/e2e/CLAUDE.md — conftest fixtures and async scoping, environment injected into the binary, mock_llm canned responses, writing guide with correct asyncio usage, gotchas section Root CLAUDE.md updates: - Added E2E test setup and integration test commands - Documented ~15 undocumented modules: cli/, registry/, hooks/, tunnel/, observability/, webhook_server.rs, cost_guard.rs, job_monitor.rs, etc. - Corrected libSQL backend path (libsql/ directory, 8 sub-modules) - Updated Database trait method count (~67, split across 7 sub-traits) - Fixed stale references: config.rs → config/channels.rs, main.rs → app.rs - Added Hook, Observer, Tunnel traits to extensibility section - Added tunnel and observability env vars to Configuration section - Removed resolved TODO (webhook trigger is now shipped) - Added Module Specifications entries for all 5 new CLAUDE.md files Co-Authored-By: Claude Sonnet 4.6 * docs: address PR review comments and reduce CLAUDE.md size - Fix 7-sub-trait count (was 6) and ~78 async methods (was ~60/~67) in both CLAUDE.md and src/db/CLAUDE.md - Add missing types.rs to secrets/ file tree (CLAUDE.md) - Add missing tls.rs to src/db/CLAUDE.md Files table - Fix method counts: ConversationStore 12, JobStore 13, RoutineStore 15 - Add Windows venv activation note to E2E setup commands - Collapse agent/, web/, llm/, db/ file trees to one-liners (detail lives in their respective CLAUDE.md files) - Replace verbose Database and LLM Providers sections with summaries linking to src/db/CLAUDE.md and src/llm/CLAUDE.md - Root CLAUDE.md: 43,868 → 35,270 chars (fixes >40k perf warning) [skip-regression-check] Co-Authored-By: Claude Sonnet 4.6 --------- Co-authored-by: Claude Sonnet 4.6 --- CLAUDE.md | 262 ++++++++++++++++++------------------- src/agent/CLAUDE.md | 171 ++++++++++++++++++++++++ src/channels/web/CLAUDE.md | 212 ++++++++++++++++++++++++++++++ src/db/CLAUDE.md | 174 ++++++++++++++++++++++++ src/llm/CLAUDE.md | 174 ++++++++++++++++++++++++ tests/e2e/CLAUDE.md | 174 ++++++++++++++++++++++++ 6 files changed, 1034 insertions(+), 133 deletions(-) create mode 100644 src/agent/CLAUDE.md create mode 100644 src/channels/web/CLAUDE.md create mode 100644 src/db/CLAUDE.md create mode 100644 src/llm/CLAUDE.md create mode 100644 tests/e2e/CLAUDE.md diff --git a/CLAUDE.md b/CLAUDE.md index c06c8537..d0e726ce 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -43,6 +43,20 @@ cargo test test_name # Run with logging RUST_LOG=ironclaw=debug cargo run + +# Run integration tests (may require running services/DB) +cargo test --test workspace_integration +cargo test --test ws_gateway_integration +cargo test --test heartbeat_integration + +# Run E2E tests (Python/Playwright — requires a running ironclaw instance) +# See tests/e2e/CLAUDE.md for full setup instructions +cd tests/e2e +python -m venv .venv && source .venv/bin/activate # On Windows: .venv\Scripts\activate +pip install -e . +playwright install chromium +pytest scenarios/ # all scenarios +pytest scenarios/test_chat.py # specific scenario ``` ### Test Tiers @@ -61,26 +75,21 @@ Run `bash scripts/check-boundaries.sh` to verify test tier gating and other arch src/ ├── lib.rs # Library root, module declarations ├── main.rs # Entry point, CLI args, startup -├── config.rs # Configuration from env vars +├── app.rs # App startup orchestration (channel wiring, DB init) +├── bootstrap.rs # Base directory resolution (~/.ironclaw), early .env loading +├── settings.rs # User settings persistence (~/.ironclaw/settings.json) +├── service.rs # OS service management (launchd/systemd daemon install) +├── tracing_fmt.rs # Custom tracing formatter +├── util.rs # Shared utilities +├── config/ # Configuration from env vars (split by subsystem) +│ ├── mod.rs # Re-exports all config types; top-level Config struct +│ ├── agent.rs, llm.rs, channels.rs, database.rs, sandbox.rs, skills.rs +│ ├── heartbeat.rs, routines.rs, safety.rs, embeddings.rs, wasm.rs +│ ├── tunnel.rs # Tunnel provider config (TUNNEL_PROVIDER, TUNNEL_URL, etc.) +│ └── secrets.rs, hygiene.rs, builder.rs, helpers.rs ├── error.rs # Error types (thiserror) │ -├── agent/ # Core agent logic -│ ├── agent_loop.rs # Main Agent struct, message handling loop -│ ├── router.rs # MessageIntent classification -│ ├── scheduler.rs # Parallel job scheduling -│ ├── worker.rs # Per-job execution with LLM reasoning -│ ├── self_repair.rs # Stuck job detection and recovery -│ ├── heartbeat.rs # Proactive periodic execution -│ ├── session.rs # Session/thread/turn model with state machine -│ ├── session_manager.rs # Thread/session lifecycle management -│ ├── compaction.rs # Context window management with turn summarization -│ ├── context_monitor.rs # Memory pressure detection -│ ├── undo.rs # Turn-based undo/redo with checkpoints -│ ├── submission.rs # Submission parsing (undo, redo, compact, clear, etc.) -│ ├── dispatcher.rs # Skill-aware job dispatching -│ ├── task.rs # Sub-task execution framework -│ ├── routine.rs # Routine types (Trigger, Action, Guardrails) -│ └── routine_engine.rs # Routine execution (cron ticker, event matcher) +├── agent/ # Core agent loop, dispatcher, scheduler, sessions — see src/agent/CLAUDE.md │ ├── channels/ # Multi-channel input │ ├── channel.rs # Channel trait, IncomingMessage, OutgoingResponse @@ -93,21 +102,60 @@ src/ │ │ ├── overlay.rs # Approval overlays │ │ └── composer.rs # Message composition │ ├── http.rs # HTTP webhook (axum) with secret validation +│ ├── webhook_server.rs # Unified HTTP server composing all webhook routes │ ├── repl.rs # Simple REPL (for testing) -│ ├── web/ # Web gateway (browser UI) -│ │ ├── mod.rs # Gateway builder, startup -│ │ ├── server.rs # Axum router, 40+ API endpoints -│ │ ├── sse.rs # SSE broadcast manager -│ │ ├── ws.rs # WebSocket gateway + connection tracking -│ │ ├── types.rs # Request/response types, SseEvent enum -│ │ ├── auth.rs # Bearer token auth middleware -│ │ ├── log_layer.rs # Tracing layer for log streaming -│ │ └── static/ # HTML, CSS, JS (single-page app) +│ ├── web/ # Web gateway (browser UI) — see src/channels/web/CLAUDE.md │ └── wasm/ # WASM channel runtime │ ├── mod.rs │ ├── bundled.rs # Bundled channel discovery +│ ├── capabilities.rs # Channel-specific capabilities (HTTP endpoint, emit rate) +│ ├── error.rs # WASM channel error types +│ ├── runtime.rs # WASM channel execution runtime │ └── wrapper.rs # Channel trait wrapper for WASM modules │ +├── cli/ # CLI subcommands (clap) +│ ├── mod.rs # Cli struct, Command enum (run/onboard/config/tool/registry/mcp/memory/pairing/service/doctor/status/completion) +│ ├── config.rs # config list/get/set subcommands +│ ├── tool.rs # tool install/list/remove subcommands +│ ├── registry.rs # registry list/install subcommands +│ ├── mcp.rs # mcp add/auth/list/test subcommands +│ ├── memory.rs # memory search/read/write subcommands +│ ├── pairing.rs # pairing list/approve subcommands +│ ├── service.rs # service install/start/stop subcommands +│ ├── doctor.rs # Active health diagnostics +│ ├── status.rs # System health/status display +│ ├── completion.rs # Shell completion script generation +│ └── oauth_defaults.rs # Default OAuth redirect URIs +│ +├── registry/ # Extension registry catalog +│ ├── mod.rs # Public API; re-exports RegistryCatalog, RegistryInstaller, manifest types +│ ├── manifest.rs # ExtensionManifest, ArtifactSpec, BundleDefinition types +│ ├── catalog.rs # RegistryCatalog: load from filesystem and embedded JSON +│ ├── installer.rs # RegistryInstaller: download, verify, install WASM artifacts +│ ├── artifacts.rs # Artifact download and caching +│ └── embedded.rs # Catalog compiled into binary at build time (via build.rs) +│ +├── hooks/ # Lifecycle hooks for intercepting agent operations +│ ├── mod.rs # 6 HookPoints: BeforeInbound, BeforeToolCall, BeforeOutbound, OnSessionStart, OnSessionEnd, TransformResponse +│ ├── hook.rs # Hook trait, HookContext, HookEvent, HookOutcome, HookFailureMode +│ ├── registry.rs # HookRegistry: register, prioritize, execute hooks +│ └── bundled.rs # Built-in hooks: rule-based filters, webhook forwarders, HookBundleConfig +│ +├── tunnel/ # Tunnel abstraction for public internet exposure +│ ├── mod.rs # Tunnel trait, TunnelProviderConfig, create_tunnel() factory +│ ├── cloudflare.rs # CloudflareTunnel (cloudflared binary) +│ ├── ngrok.rs # NgrokTunnel +│ ├── tailscale.rs # TailscaleTunnel (serve/funnel modes) +│ ├── custom.rs # CustomTunnel (arbitrary command with {host}/{port}) +│ └── none.rs # NoneTunnel (local-only, no exposure) +│ +├── observability/ # Pluggable event/metric recording +│ ├── mod.rs # create_observer() factory, ObservabilityConfig +│ ├── traits.rs # Observer trait, ObserverEvent, ObserverMetric +│ ├── noop.rs # NoopObserver (zero overhead, default) +│ ├── log.rs # LogObserver (tracing-based) +│ └── multi.rs # MultiObserver (fan-out to multiple backends) +│ ├── orchestrator/ # Internal HTTP API for sandbox containers │ ├── mod.rs │ ├── api.rs # Axum endpoints (LLM proxy, events, prompts) @@ -125,34 +173,30 @@ src/ │ ├── sanitizer.rs # Pattern detection, content escaping │ ├── validator.rs # Input validation (length, encoding, patterns) │ ├── policy.rs # PolicyRule system with severity/actions -│ └── leak_detector.rs # Secret detection (API keys, tokens, etc.) +│ ├── leak_detector.rs # Secret detection (API keys, tokens, etc.) +│ └── credential_detect.rs # HTTP request credential detection (headers, URL params) │ -├── llm/ # LLM integration (multi-provider) -│ ├── mod.rs # Provider factory, LlmBackend enum -│ ├── provider.rs # LlmProvider trait, message types -│ ├── nearai_chat.rs # NEAR AI Chat Completions provider (session token + API key auth) -│ ├── reasoning.rs # Planning, tool selection, evaluation -│ ├── session.rs # Session token management with auto-renewal -│ ├── circuit_breaker.rs # Circuit breaker for provider failures -│ ├── retry.rs # Retry with exponential backoff -│ ├── failover.rs # Multi-provider failover chain -│ ├── response_cache.rs # LLM response caching -│ ├── costs.rs # Token cost tracking -│ └── rig_adapter.rs # Rig framework adapter +├── llm/ # Multi-provider LLM integration — see src/llm/CLAUDE.md │ ├── tools/ # Extensible tool system │ ├── tool.rs # Tool trait, ToolOutput, ToolError │ ├── registry.rs # ToolRegistry for discovery │ ├── sandbox.rs # Process-based sandbox (stub, superseded by wasm/) +│ ├── rate_limiter.rs # Shared sliding-window rate limiter for built-in and WASM tools │ ├── builtin/ # Built-in tools │ │ ├── echo.rs, time.rs, json.rs, http.rs +│ │ ├── web_fetch.rs # GET URL → clean Markdown (readability + html-to-md conversion) │ │ ├── file.rs # ReadFile, WriteFile, ListDir, ApplyPatch │ │ ├── shell.rs # Shell command execution │ │ ├── memory.rs # Memory tools (search, write, read, tree) +│ │ ├── message.rs # MessageTool: agent proactively messages users on any channel │ │ ├── job.rs # CreateJob, ListJobs, JobStatus, CancelJob │ │ ├── routine.rs # routine_create/list/update/delete/history │ │ ├── extension_tools.rs # Extension install/auth/activate/remove │ │ ├── skill_tools.rs # skill_list/search/install/remove tools +│ │ ├── secrets_tools.rs # secret_list/secret_delete (zero-exposure: no values exposed) +│ │ ├── html_converter.rs # HTML→Markdown via readability + html-to-markdown-rs +│ │ ├── path_utils.rs # Shared path validation/canonicalization helpers │ │ └── marketplace.rs, ecommerce.rs, taskrabbit.rs, restaurant.rs (stubs) │ ├── builder/ # Dynamic tool building │ │ ├── core.rs # BuildRequirement, SoftwareType, Language @@ -161,7 +205,8 @@ src/ │ │ └── validation.rs # WASM validation │ ├── mcp/ # Model Context Protocol │ │ ├── client.rs # MCP client over HTTP -│ │ └── protocol.rs # JSON-RPC types +│ │ ├── protocol.rs # JSON-RPC types +│ │ └── session.rs # MCP session management (Mcp-Session-Id header, per-server state) │ └── wasm/ # Full WASM sandbox (wasmtime) │ ├── runtime.rs # Module compilation and caching │ ├── wrapper.rs # Tool trait wrapper for WASM modules @@ -171,13 +216,10 @@ src/ │ ├── credential_injector.rs # Safe credential injection │ ├── loader.rs # WASM tool discovery from filesystem │ ├── rate_limiter.rs # Per-tool rate limiting +│ ├── error.rs # WASM-specific error types │ └── storage.rs # Linear memory persistence │ -├── db/ # Database abstraction layer -│ ├── mod.rs # Database trait (~60 async methods) -│ ├── postgres.rs # PostgreSQL backend (delegates to Store + Repository) -│ ├── libsql_backend.rs # libSQL/Turso backend (embedded SQLite) -│ └── libsql_migrations.rs # SQLite-dialect schema (idempotent) +├── db/ # Dual-backend persistence (PostgreSQL + libSQL) — see src/db/CLAUDE.md │ ├── workspace/ # Persistent memory system (OpenClaw-inspired) │ ├── mod.rs # Workspace struct, memory operations @@ -215,9 +257,11 @@ src/ │ └── allowlist.rs # DomainAllowlist validation │ ├── secrets/ # Secrets management +│ ├── mod.rs # SecretsStore trait, public API +│ ├── types.rs # Core types (Secret, SecretRef, SecretMetadata) │ ├── crypto.rs # AES-256-GCM encryption -│ ├── store.rs # Secret storage -│ └── types.rs # Credential types +│ ├── keychain.rs # OS keychain integration (macOS Keychain, GNOME Keyring) for master key +│ └── store.rs # Encrypted secret storage │ ├── setup/ # Onboarding wizard (spec: src/setup/README.md) │ ├── mod.rs # Entry point, check_onboard_needed() @@ -237,6 +281,11 @@ src/ └── history/ # Persistence ├── store.rs # PostgreSQL repositories └── analytics.rs # Aggregation queries (JobStats, ToolStats) + +tests/ +├── *.rs # Integration tests (workspace, heartbeat, WS gateway, pairing, etc.) +├── test-pages/ # HTML→Markdown conversion fixtures (CNN, Medium, Yahoo) +└── e2e/ # Python/Playwright E2E scenarios (see tests/e2e/CLAUDE.md) ``` ## Key Patterns @@ -257,13 +306,16 @@ When designing new features or systems, always prefer generic/extensible archite - Use `RwLock` for concurrent read/write access ### Traits for Extensibility -- `Database` - Add new database backends (must implement all ~60 methods) +- `Database` - Add new database backends (must implement all ~78 methods) - `Channel` - Add new input sources - `Tool` - Add new capabilities - `LlmProvider` - Add new LLM backends - `SuccessEvaluator` - Custom evaluation logic - `EmbeddingProvider` - Add embedding backends (workspace search) - `NetworkPolicyDecider` - Custom network access policies for sandbox containers +- `Hook` - Lifecycle hook at 6 interception points (BeforeInbound, BeforeToolCall, BeforeOutbound, OnSessionStart, OnSessionEnd, TransformResponse) +- `Observer` - Observability backend (noop/log/multi; future: OpenTelemetry, Prometheus) +- `Tunnel` - Tunnel provider for public internet exposure ### Tool Implementation ```rust @@ -416,99 +468,38 @@ SKILLS_AUTO_DISCOVER=true # Scan skill directories on startup # Tinfoil private inference TINFOIL_API_KEY=... # Required when LLM_BACKEND=tinfoil TINFOIL_MODEL=kimi-k2-5 # Default model + +# Tunnel (public internet exposure for webhooks) +TUNNEL_URL=https://abc123.ngrok.io # Static public URL (manual tunnel) +# Or use a managed tunnel provider: +TUNNEL_PROVIDER=none # none (default), cloudflare, tailscale, ngrok, custom +TUNNEL_CF_TOKEN=... # Required for TUNNEL_PROVIDER=cloudflare +TUNNEL_NGROK_TOKEN=... # Required for TUNNEL_PROVIDER=ngrok +# TUNNEL_NGROK_DOMAIN=... # Custom domain (paid ngrok plan) +# TUNNEL_TS_FUNNEL=true # Use tailscale funnel (public) vs serve (tailnet) +TUNNEL_CUSTOM_COMMAND=... # Command with {host}/{port} for custom providers + +# Observability backend +OBSERVABILITY_BACKEND=none # none/noop (default) or log ``` ### LLM Providers -IronClaw supports multiple LLM backends via the `LLM_BACKEND` env var: `nearai` (default), `openai`, `anthropic`, `ollama`, `openai_compatible`, and `tinfoil`. - -**NEAR AI** -- Uses the Chat Completions API with dual auth support. Session token auth (default): authenticates with session tokens (`sess_xxx`) obtained via browser OAuth (GitHub/Google), base URL defaults to `https://private.near.ai`. API key auth: set `NEARAI_API_KEY` (from `cloud.near.ai`), base URL defaults to `https://cloud-api.near.ai`. Both modes use the same Chat Completions endpoint. Tool messages are flattened to plain text for compatibility. Set `NEARAI_SESSION_TOKEN` env var for hosting providers that inject tokens via environment. - -**NEAR AI Cloud** -- Uses the OpenAI-compatible Chat Completions API (`https://cloud-api.near.ai/v1/chat/completions`). Authenticates with API keys from `cloud.near.ai`. Auto-selected when `NEARAI_API_KEY` is set (or explicitly via `NEARAI_API_MODE=chat_completions`). Tool messages are flattened to plain text for compatibility. Configure with `NEARAI_API_KEY` and `NEARAI_BASE_URL` (default: `https://cloud-api.near.ai`). - -**OpenAI-compatible** -- Any endpoint that speaks the OpenAI API (vLLM, LiteLLM, OpenRouter, etc.). Configure with `LLM_BASE_URL`, `LLM_API_KEY` (optional), `LLM_MODEL`. Set `LLM_EXTRA_HEADERS` to inject custom HTTP headers into every request (format: `Key:Value,Key2:Value2`), useful for OpenRouter attribution headers like `HTTP-Referer` and `X-Title`. - -**Tinfoil** -- Private inference via `https://inference.tinfoil.sh/v1`. Runs models inside hardware-attested TEEs so neither Tinfoil nor the cloud provider can see prompts or responses. Uses the OpenAI-compatible Chat Completions API. Configure with `TINFOIL_API_KEY` and `TINFOIL_MODEL` (default: `kimi-k2-5`). +Backends: `nearai` (default), `openai`, `anthropic`, `ollama`, `openai_compatible`, `tinfoil` — set via `LLM_BACKEND`. See [src/llm/CLAUDE.md](src/llm/CLAUDE.md) for per-provider auth and configuration details. ## Database -IronClaw supports two database backends, selected at compile time via Cargo feature flags and at runtime via the `DATABASE_BACKEND` environment variable. - -**IMPORTANT: All new features that touch persistence MUST support both backends.** Implement the operation as a method on the `Database` trait in `src/db/mod.rs`, then add the implementation in both `src/db/postgres.rs` (delegate to Store/Repository) and `src/db/libsql_backend.rs` (native SQL). - -### Backends - -| Backend | Feature Flag | Default | Use Case | -|---------|-------------|---------|----------| -| PostgreSQL | `postgres` (default) | Yes | Production, existing deployments | -| libSQL/Turso | `libsql` | No | Zero-dependency local mode, edge, Turso cloud | +Dual-backend persistence (PostgreSQL + libSQL/Turso). **All new persistence features must support both backends** — see [src/db/CLAUDE.md](src/db/CLAUDE.md) for schema, SQL dialect differences, adding operations, and libSQL limitations. +Implement every new operation in both `src/db/postgres.rs` and `src/db/libsql/mod.rs`. Test in isolation: ```bash -# Build with PostgreSQL only (default) -cargo build - -# Build with libSQL only -cargo build --no-default-features --features libsql - -# Build with both backends available -cargo build --features "postgres,libsql" +cargo check # postgres (default) +cargo check --no-default-features --features libsql # libsql only +cargo check --all-features # both ``` -### Database Trait - -The `Database` trait (`src/db/mod.rs`) defines ~60 async methods covering all persistence: -- Conversations, messages, metadata -- Jobs, actions, LLM calls, estimation snapshots -- Sandbox jobs, job events -- Routines, routine runs -- Tool failures, settings -- Workspace: documents, chunks, hybrid search - -Both backends implement this trait. PostgreSQL delegates to the existing `Store` + `Repository`. libSQL implements native SQLite-dialect SQL. - -### Schema - -**PostgreSQL:** `migrations/V1__initial.sql` (351 lines). Uses pgvector for embeddings, tsvector for FTS, PL/pgSQL functions. Managed by `refinery`. - -**libSQL:** `src/db/libsql_migrations.rs` (consolidated schema, ~480 lines). Translates PG types: -- `UUID` -> `TEXT`, `TIMESTAMPTZ` -> `TEXT` (ISO-8601), `JSONB` -> `TEXT` -- `VECTOR(1536)` -> `F32_BLOB(1536)` with `libsql_vector_idx` -- `tsvector`/`ts_rank_cd` -> FTS5 virtual table with sync triggers -- PL/pgSQL functions -> SQLite triggers - -**Tables (both backends):** - -**Core:** -- `conversations` - Multi-channel conversation tracking -- `agent_jobs` - Job metadata and status -- `job_actions` - Event-sourced tool executions -- `dynamic_tools` - Agent-built tools -- `llm_calls` - Cost tracking -- `estimation_snapshots` - Learning data - -**Workspace/Memory:** -- `memory_documents` - Flexible path-based files (e.g., "context/vision.md", "daily/2024-01-15.md") -- `memory_chunks` - Chunked content with FTS and vector indexes -- `heartbeat_state` - Periodic execution tracking - -**Other:** -- `routines`, `routine_runs` - Scheduled/reactive execution -- `settings` - Per-user key-value settings -- `tool_failures` - Self-repair tracking -- `secrets`, `wasm_tools`, `tool_capabilities` - Extension infrastructure - Database configuration: see Configuration section above. -### Current Limitations (libSQL backend) - -- **Workspace/memory system** not yet wired through Database trait (requires Store migration) -- **Secrets store** not yet available (still requires PostgresSecretsStore) -- **Hybrid search** uses FTS5 only (vector search via libsql_vector_idx not yet implemented) -- **Settings reload from DB** skipped (Config::from_db requires Store) -- No incremental migration versioning (schema is CREATE IF NOT EXISTS, no ALTER TABLE support yet) -- **No encryption at rest** -- The local SQLite database file stores conversation content, job data, workspace memory, and other application data in plaintext. Only secrets (API tokens, credentials) are encrypted via AES-256-GCM before storage. Users handling sensitive data should use full-disk encryption (FileVault, LUKS, BitLocker) or consider the PostgreSQL backend with TDE/encrypted storage. -- **JSON merge patch vs path-targeted update** -- The libSQL backend uses RFC 7396 JSON Merge Patch (`json_patch`) for metadata updates, while PostgreSQL uses path-targeted `jsonb_set`. Merge patch replaces top-level keys entirely, which may drop nested keys not present in the patch. Callers should avoid relying on partial nested object updates in metadata fields. - ## Safety Layer All external tool output passes through `SafetyLayer`: @@ -638,8 +629,8 @@ Key test patterns: 4. **WIT bindgen integration** - Auto-extract tool description/schema from WASM modules (stubbed) 5. **Capability granting after tool build** - Built tools get empty capabilities; need UX for granting HTTP/secrets access 6. **Tool versioning workflow** - No version tracking or rollback for dynamically built tools -7. **Webhook trigger endpoint** - Routines webhook trigger not yet exposed in web gateway -8. **Full channel status view** - Gateway status widget exists, but no per-channel connection dashboard +7. **Full channel status view** - Gateway status widget exists, but no per-channel connection dashboard +8. **Observability backends** - Only `log` and `noop` implemented; OpenTelemetry/Prometheus not yet supported ## Tool Architecture @@ -653,8 +644,8 @@ See `src/tools/README.md` for full tool architecture, adding new tools (built-in 1. Create `src/channels/my_channel.rs` 2. Implement the `Channel` trait -3. Add config in `src/config.rs` -4. Wire up in `main.rs` channel setup section +3. Add config in `src/config/channels.rs` +4. Wire up in `src/app.rs` channel setup section ## Debugging @@ -686,6 +677,11 @@ for that module's behavior. When modifying code in a module that has a spec: | `src/setup/` | `src/setup/README.md` | | `src/workspace/` | `src/workspace/README.md` | | `src/tools/` | `src/tools/README.md` | +| `src/agent/` | `src/agent/CLAUDE.md` | +| `src/channels/web/` | `src/channels/web/CLAUDE.md` | +| `src/db/` | `src/db/CLAUDE.md` | +| `src/llm/` | `src/llm/CLAUDE.md` | +| `tests/e2e/` | `tests/e2e/CLAUDE.md` | ## Workspace & Memory System diff --git a/src/agent/CLAUDE.md b/src/agent/CLAUDE.md new file mode 100644 index 00000000..40221341 --- /dev/null +++ b/src/agent/CLAUDE.md @@ -0,0 +1,171 @@ +# Agent Module + +Core agent logic. This is the most complex subsystem — read this before working in `src/agent/`. + +## Module Map + +| File | Role | +|------|------| +| `agent_loop.rs` | `Agent` struct, `AgentDeps`, main `run()` event loop. Delegates to siblings. | +| `dispatcher.rs` | Agentic loop for conversational turns: LLM call → tool execution → repeat. Injects skill context. Returns `Response` or `NeedApproval`. | +| `thread_ops.rs` | Thread/session operations: `process_user_input`, undo/redo, approval, auth-mode interception, DB hydration, compaction. | +| `commands.rs` | System command handlers (`/help`, `/model`, `/status`, `/skills`, etc.) and job intent handlers. | +| `session.rs` | Data model: `Session` → `Thread` → `Turn`. State machines for threads and turns. | +| `session_manager.rs` | Lifecycle: create/lookup sessions, map external thread IDs to internal UUIDs, prune stale sessions, manage undo managers. | +| `router.rs` | Routes explicit `/commands` to `MessageIntent`. Natural language bypasses the router entirely. | +| `scheduler.rs` | Parallel job scheduling. Maintains `jobs` map (full LLM-driven) and `subtasks` map (tool-exec/background). | +| `worker.rs` | Per-job execution for background scheduler jobs: calls LLM, runs tools, handles the reasoning loop. Distinct from `dispatcher.rs`. | +| `compaction.rs` | Context window management: summarize old turns, write to workspace daily log, trim context. Three strategies. | +| `context_monitor.rs` | Detects memory pressure. Suggests `CompactionStrategy` based on usage level. | +| `self_repair.rs` | Detects stuck jobs and broken tools, attempts recovery. | +| `heartbeat.rs` | Proactive periodic execution. Reads `HEARTBEAT.md`, notifies via channel if findings. | +| `submission.rs` | Parses all user submissions into typed variants before routing. | +| `undo.rs` | Turn-based undo/redo with checkpoints. Checkpoints store message lists (max 20 by default). | +| `routine.rs` | `Routine` types: `Trigger` (cron/event/webhook/manual) + `RoutineAction` (lightweight/full_job) + `RoutineGuardrails`. | +| `routine_engine.rs` | Cron ticker and event matcher. Fires routines when triggers match. Lightweight runs inline; full_job dispatches to `Scheduler`. | +| `task.rs` | Task types for the scheduler: `Job`, `ToolExec`, `Background`. Used by `spawn_subtask` and `spawn_batch`. | +| `cost_guard.rs` | LLM spend and action-rate enforcement. Tracks daily budget (cents) and hourly call rate. Lives in `AgentDeps`. | +| `job_monitor.rs` | Subscribes to SSE broadcast and injects Claude Code (container) output back into the agent loop as `IncomingMessage`. | + +## Session / Thread / Turn Model + +``` +Session (per user) +└── Thread (per conversation — can have many) + └── Turn (per request/response pair) + ├── user_input: String + ├── response: Option + ├── tool_calls: Vec + └── state: TurnState (Pending | Running | Complete | Failed) +``` + +- A session has one **active thread** at a time; threads can be switched. +- Turns are append-only. Undo rolls back by restoring a prior checkpoint (message list, not a full thread snapshot). +- `UndoManager` is per-thread, stored in `SessionManager`, not on `Session` itself. Max 20 checkpoints (oldest dropped when exceeded). +- Group chat detection: if `metadata.chat_type` is `group`/`channel`/`supergroup`, `MEMORY.md` is excluded from the system prompt to prevent leaking personal context. +- **Auth mode**: if a thread has `pending_auth` set (e.g. from `tool_auth` returning `awaiting_token`), the next user message is intercepted before any turn creation, logging, or safety validation and sent directly to the credential store. Any control submission (undo, interrupt, etc.) cancels auth mode. +- `ThreadState` values: `Idle`, `Processing`, `AwaitingApproval`, `Completed`, `Interrupted`. +- `SessionManager` maps `(user_id, channel, external_thread_id)` → internal UUID. Prunes idle sessions every 10 minutes (warns at 1000 sessions). + +## Agentic Loop (dispatcher.rs) + +The `dispatcher.rs` module handles **direct conversational turns** (user messages processed inline by the main agent). Background scheduler jobs use `worker.rs` instead — these are two separate execution paths. + +``` +run_agentic_loop() [dispatcher.rs — conversational turns] + 1. Load workspace system prompt (identity files: AGENTS.md, SOUL.md, etc.) + 2. Detect group chat from metadata; exclude MEMORY.md if group chat + 3. Select active skills (keyword/pattern scoring against message content) + 4. Build skill context block (injected before user message) + 5. LLM call → text response OR tool calls + 6. If tool calls: + a. Check tool approval (session auto-approvals, pending approval queue) + b. Execute tools (parallel via JoinSet) + c. Sanitize results through SafetyLayer + d. Feed results back → goto 5 + 7. Return AgenticLoopResult::Response or NeedApproval +``` + +**Tool approval:** Tools flagged `requires_approval` pause the loop and return `NeedApproval`. The web gateway stores the `PendingApproval` in session state and sends an `approval_needed` SSE event. The user's approval/deny resumes the loop. + +**worker.rs vs dispatcher.rs:** `dispatcher.rs` runs the agentic loop for user-initiated conversational turns (holds session lock, tracks turns). `worker.rs` is spawned by the `Scheduler` for background jobs created via `CreateJob` / `/job` — it runs independently of the session and has its own LLM reasoning loop with planning support (`use_planning` flag). + +## Command Routing (router.rs) + +The `Router` handles explicit `/commands` (prefix `/`). It parses them into `MessageIntent` variants: `CreateJob`, `CheckJobStatus`, `CancelJob`, `ListJobs`, `HelpJob`, `Command`. Natural language messages bypass the router entirely — they go directly to `dispatcher.rs` via `process_user_input`. Note: most user-facing commands (undo, compact, etc.) are handled by `SubmissionParser` before the router runs, so `Router` only sees unrecognized `/xxx` patterns that haven't already been claimed by `submission.rs`. + +## Compaction + +Triggered by `ContextMonitor` when token usage approaches the model's context limit. + +**Token estimation**: Word-count × 1.3 + 4 overhead per message. Default context limit: 100,000 tokens. Compaction threshold: 80% (configurable). + +Three strategies, chosen by `ContextMonitor.suggest_compaction()` based on usage ratio: +- **MoveToWorkspace** — Writes full turn transcript to workspace daily log, keeps 10 recent turns. Used when usage is 80–85% (moderate). Falls back to `Truncate(5)` if no workspace. +- **Summarize** (`keep_recent: N`) — LLM generates a summary of old turns, writes it to workspace daily log (`daily/YYYY-MM-DD.md`), removes old turns. Used when usage is 85–95%. +- **Truncate** (`keep_recent: N`) — Removes oldest turns without summarization (fast path). Used when usage >95% (critical). + +If the LLM call for summarization fails, the error propagates — turns are **not** truncated on failure. + +Manual trigger: user sends `/compact` (parsed by `submission.rs`). + +## Scheduler + +`Scheduler` maintains two maps under `Arc>`: +- `jobs` — full LLM-driven jobs, each with a `Worker` and an `mpsc` channel for `WorkerMessage` (`Start`, `Stop`, `Ping`, `UserMessage`). +- `subtasks` — lightweight `ToolExec` or `Background` tasks spawned via `spawn_subtask()` / `spawn_batch()`. + +**Preferred entry point**: `dispatch_job()` — creates context, optionally sets metadata, persists to DB (so FK references from `job_actions`/`llm_calls` are valid immediately), then calls `schedule()`. Don't call `schedule()` directly unless you've already persisted. + +Check-insert is done under a single write lock to prevent TOCTOU races. A cleanup task polls every second for job completion and removes the entry from the map. + +`spawn_subtask()` returns a `oneshot::Receiver` — callers must await it to get the result. `spawn_batch()` runs all tasks concurrently and returns results in input order. + +## Self-Repair + +`DefaultSelfRepair` runs on `repair_check_interval` (from `AgentConfig`). It: +1. Calls `ContextManager::find_stuck_jobs()` to find jobs in `JobState::Stuck`. +2. Attempts `ctx.attempt_recovery()` (transitions back to `InProgress`). +3. Returns `ManualRequired` if `repair_attempts >= max_repair_attempts`. +4. Detects broken tools via `store.get_broken_tools(5)` (threshold: 5 failures). Requires `with_store()` to be called; returns empty without a store. +5. Attempts to rebuild broken tools via `SoftwareBuilder`. Requires `with_builder()` to be called; returns `ManualRequired` without a builder. + +Note: the `stuck_threshold` duration is stored but currently unused (marked `#[allow(dead_code)]`). Stuck detection relies on `JobState::Stuck` being set by the state machine, not wall-clock time comparison. + +Repair results: `Success`, `Retry`, `Failed`, `ManualRequired`. `Retry` does NOT notify the user (to avoid spam). + +## Key Invariants + +- Never call `.unwrap()` or `.expect()` — use `?` with proper error mapping. +- All state mutations on `Session`/`Thread` happen under `Arc>` lock. +- The agent loop is single-threaded per thread; parallel execution happens at the job/scheduler level. +- Skills are selected **deterministically** (no LLM call) — see `skills/selector.rs`. +- Tool results pass through `SafetyLayer` before returning to LLM (sanitizer → validator → policy → leak detector). +- `SessionManager` uses double-checked locking for session creation. Read lock first (fast path), then write lock with re-check to prevent duplicate sessions. +- `Scheduler.schedule()` holds the write lock for the entire check-insert sequence — don't hold any other locks when calling it. +- `cheap_llm` in `AgentDeps` is used for heartbeat and other lightweight tasks. Falls back to main `llm` if `None`. Use `agent.cheap_llm()` accessor, not `deps.cheap_llm` directly. +- `CostGuard.check_allowed()` must be called **before** LLM calls; `record_llm_call()` must be called **after**. Both calls are separate — the guard does not auto-record. +- `BeforeInbound` and `BeforeOutbound` hooks run for every user message and agent response respectively. Hooks can modify content or reject. Hook errors are logged but **fail-open** (processing continues). + +## Complete Submission Command Reference + +All commands parsed by `SubmissionParser::parse()`: + +| Input | Variant | Notes | +|-------|---------|-------| +| `/undo` | `Undo` | | +| `/redo` | `Redo` | | +| `/interrupt`, `/stop` | `Interrupt` | | +| `/compact` | `Compact` | | +| `/clear` | `Clear` | | +| `/heartbeat` | `Heartbeat` | | +| `/summarize`, `/summary` | `Summarize` | | +| `/suggest` | `Suggest` | | +| `/new`, `/thread new` | `NewThread` | | +| `/thread ` | `SwitchThread` | Must be valid UUID | +| `/resume ` | `Resume` | Must be valid UUID | +| `/status [id]`, `/progress [id]`, `/list` | `JobStatus` | `/list` = all jobs | +| `/cancel ` | `JobCancel` | | +| `/quit`, `/exit`, `/shutdown` | `Quit` | | +| `yes/y/approve/ok` and aliases | `ApprovalResponse { approved: true, always: false }` | | +| `always/a` and aliases | `ApprovalResponse { approved: true, always: true }` | | +| `no/n/deny/reject/cancel` and aliases | `ApprovalResponse { approved: false }` | | +| JSON `ExecApproval{...}` | `ExecApproval` | From web gateway approval endpoint | +| `/help`, `/?` | `SystemCommand { "help" }` | Bypasses thread-state checks | +| `/version` | `SystemCommand { "version" }` | | +| `/tools` | `SystemCommand { "tools" }` | | +| `/skills [search ]` | `SystemCommand { "skills" }` | | +| `/ping` | `SystemCommand { "ping" }` | | +| `/debug` | `SystemCommand { "debug" }` | | +| `/model [name]` | `SystemCommand { "model" }` | | +| Everything else | `UserInput` | Starts a new agentic turn | + +**`SystemCommand` vs control**: `SystemCommand` variants bypass thread-state checks entirely (no session lock, no turn creation). `Quit` returns `Ok(None)` from `handle_message` which breaks the main loop. + +## Adding a New Submission Command + +Submissions are special messages parsed in `submission.rs` before the agentic loop runs. To add a new one: +1. Add a variant to `Submission` enum in `submission.rs` +2. Add parsing in `SubmissionParser::parse()` +3. Handle in `agent_loop.rs` where `SubmissionResult` is matched (the `match submission { ... }` block in `handle_message`) +4. Implement the handler method (usually in `thread_ops.rs` for session operations, or `commands.rs` for system commands) diff --git a/src/channels/web/CLAUDE.md b/src/channels/web/CLAUDE.md new file mode 100644 index 00000000..df5cd6cf --- /dev/null +++ b/src/channels/web/CLAUDE.md @@ -0,0 +1,212 @@ +# Web Gateway Module + +Browser-facing HTTP API and SSE/WebSocket real-time streaming. Axum-based, single-user with bearer token auth. + +## File Map + +| File | Role | +|------|------| +| `mod.rs` | Gateway builder, startup, `WebChannel` implementation, `with_*` builder methods | +| `server.rs` | `GatewayState`, `start_server()`, all Axum route registrations, inline handlers | +| `types.rs` | Request/response DTOs and `SseEvent` enum (source of truth for SSE contract) | +| `sse.rs` | `SseManager` — broadcast channel that fans out `SseEvent` to all connected SSE clients | +| `ws.rs` | WebSocket handler (`handle_ws_connection`) + `WsConnectionTracker` | +| `auth.rs` | Bearer token middleware (`Authorization: Bearer `) | +| `log_layer.rs` | Tracing layer that tees log lines to the `/api/logs/events` SSE stream | +| `handlers/` | Handler functions split by domain: `chat`, `extensions`, `jobs`, `memory`, `routines`, `settings`, `skills`, `static_files` | +| `openai_compat.rs` | OpenAI-compatible proxy (`/v1/chat/completions`, `/v1/models`) | +| `util.rs` | Shared helpers (`build_turns_from_db_messages`, `truncate_preview`) | +| `static/` | Single-page app (HTML/CSS/JS) — embedded at compile time via `include_str!`/`include_bytes!` | + +## API Routes + +### Public (no auth) +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/health` | Health check | +| GET | `/oauth/callback` | OAuth callback for extension auth | + +### Chat +| Method | Path | Description | +|--------|------|-------------| +| POST | `/api/chat/send` | Send message → queues to agent loop | +| GET | `/api/chat/events` | SSE stream of agent events | +| GET | `/api/chat/ws` | WebSocket alternative to SSE | +| GET | `/api/chat/history` | Paginated turn history for a thread | +| GET | `/api/chat/threads` | List threads (returns `assistant_thread` + regular threads) | +| POST | `/api/chat/thread/new` | Create new thread | +| POST | `/api/chat/approval` | Approve/deny/always a pending tool call | +| POST | `/api/chat/auth-token` | Submit auth token for an extension | +| POST | `/api/chat/auth-cancel` | Cancel pending auth flow | + +### Memory +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/memory/tree` | Workspace directory tree | +| GET | `/api/memory/list` | List files at a path | +| GET | `/api/memory/read` | Read a workspace file | +| POST | `/api/memory/write` | Write a workspace file | +| POST | `/api/memory/search` | Hybrid FTS + vector search | + +### Jobs (sandbox) +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/jobs` | List sandbox jobs | +| GET | `/api/jobs/summary` | Aggregated stats | +| GET | `/api/jobs/{id}` | Job detail | +| POST | `/api/jobs/{id}/cancel` | Cancel a running job | +| POST | `/api/jobs/{id}/restart` | Restart a failed job | +| POST | `/api/jobs/{id}/prompt` | Send follow-up prompt to Claude Code bridge | +| GET | `/api/jobs/{id}/events` | SSE stream for a specific job | +| GET | `/api/jobs/{id}/files/list` | List files in job workspace | +| GET | `/api/jobs/{id}/files/read` | Read a file from job workspace | + +### Skills +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/skills` | List installed skills | +| POST | `/api/skills/search` | Search ClawHub registry + local skills | +| POST | `/api/skills/install` | Install a skill from ClawHub or by URL/content | +| DELETE | `/api/skills/{name}` | Remove an installed skill | + +### Extensions +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/extensions` | Installed extensions | +| GET | `/api/extensions/tools` | All registered tools (from tool registry) | +| POST | `/api/extensions/install` | Install extension | +| GET | `/api/extensions/registry` | Available extensions from registry manifests | +| POST | `/api/extensions/{name}/activate` | Activate installed extension | +| POST | `/api/extensions/{name}/remove` | Remove extension | +| GET/POST | `/api/extensions/{name}/setup` | Extension setup wizard | + +### Routines +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/routines` | List routines | +| GET | `/api/routines/summary` | Aggregated stats (total/enabled/disabled/failing/runs_today) | +| GET | `/api/routines/{id}` | Routine detail with recent run history | +| POST | `/api/routines/{id}/trigger` | Manually trigger a routine | +| POST | `/api/routines/{id}/toggle` | Enable/disable a routine | +| DELETE | `/api/routines/{id}` | Delete a routine | +| GET | `/api/routines/{id}/runs` | List runs for a specific routine | + +### Settings +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/settings` | List all settings | +| GET | `/api/settings/export` | Export all settings as a map | +| POST | `/api/settings/import` | Bulk-import settings from a map | +| GET | `/api/settings/{key}` | Get a single setting | +| PUT | `/api/settings/{key}` | Set a single setting | +| DELETE | `/api/settings/{key}` | Delete a setting | + +### Other +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/logs/events` | Live log stream (SSE) | +| GET/PUT | `/api/logs/level` | Get/set log level at runtime | +| GET | `/api/pairing/{channel}` | List pending pairing requests | +| POST | `/api/pairing/{channel}/approve` | Approve a pairing request | +| GET | `/api/gateway/status` | Server uptime, connected clients, config | +| POST | `/v1/chat/completions` | OpenAI-compatible LLM proxy | +| GET | `/v1/models` | OpenAI-compatible model list | + +### Static / Project files +| Method | Path | Description | +|--------|------|-------------| +| GET | `/` | Single-page app HTML | +| GET | `/style.css` | App stylesheet | +| GET | `/app.js` | App JavaScript | +| GET | `/favicon.ico` | Favicon (cached 1 day) | +| GET | `/projects/{project_id}/` | Job workspace browser (redirects) | +| GET | `/projects/{project_id}/{*path}` | Serve file from job workspace (auth required) | + +## SSE Event Types (`SseEvent` in `types.rs`) + +The SSE contract — every field is `#[serde(tag = "type")]`: + +| Type | When emitted | +|------|-------------| +| `response` | Final text response from agent | +| `stream_chunk` | Streaming token (partial response) | +| `thinking` | Agent status update during reasoning | +| `tool_started` | Tool call began | +| `tool_completed` | Tool call finished (includes success/error) | +| `tool_result` | Tool output preview | +| `status` | Generic status message | +| `job_started` | Sandbox job created | +| `job_message` | Message from sandbox worker | +| `job_tool_use` | Tool invoked inside sandbox | +| `job_tool_result` | Tool result from sandbox | +| `job_status` | Sandbox job status update | +| `job_result` | Sandbox job final result | +| `approval_needed` | Tool requires user approval (pauses agent) | +| `auth_required` | Extension needs auth credentials | +| `auth_completed` | Extension auth flow finished | +| `extension_status` | WASM channel activation status changed | +| `error` | Error from agent or gateway | +| `heartbeat` | SSE keepalive (empty payload) | + +**SSE serialization:** Events use `#[serde(tag = "type")]` — the wire format is `{"type":"", ...fields}`. The SSE frame's `event:` field is set to the same string as `type` for easy `addEventListener` use in the browser. + +**WebSocket envelope:** Over WebSocket, SSE events are wrapped as `{"type":"event","event_type":"","data":{...}}`. Ping/pong uses `{"type":"ping"}` / `{"type":"pong"}`. Client-to-server messages (`message`, `approval`, `auth_token`, `auth_cancel`) are defined in `WsClientMessage` in `types.rs`. + +**To add a new SSE event:** Use the `add-sse-event` skill (`/add-sse-event`). It scaffolds the Rust variant, serialization, broadcast call, and frontend handler. Also add a matching arm to `WsServerMessage::from_sse_event()` in `types.rs`. + +## Auth + +All protected routes require `Authorization: Bearer `. The token is set via `GATEWAY_AUTH_TOKEN` env var. Missing/wrong token → 401. The `Bearer` prefix is compared case-insensitively (RFC 6750). + +**Query-string token auth (`?token=xxx`):** Because `EventSource` and WebSocket upgrades cannot set custom headers from the browser, three endpoints also accept the token as a URL query parameter: `/api/chat/events`, `/api/logs/events`, and `/api/chat/ws`. All other endpoints reject query-string tokens. If you add a new SSE or WebSocket endpoint, register its path in `allows_query_token_auth()` in `auth.rs`. + +**If no `GATEWAY_AUTH_TOKEN` is configured**, a random 32-character alphanumeric token is generated at startup and printed to the console. + +Rate limiting: chat send endpoints are capped at **30 messages per 60 seconds** (sliding window, not per-IP). + +## GatewayState + +The shared state struct (`server.rs`) holds refs to all subsystems. Fields are `Option>` so the gateway can start even when optional subsystems (workspace, sandbox, skills) are disabled. Always null-check before use in handlers. + +Key fields: +- `msg_tx` — `RwLock>>` — sends messages to the agent loop; set when `start()` is called on the `Channel`. +- `sse` — `SseManager` — broadcast hub; call `state.sse.broadcast(event)` from any handler. +- `ws_tracker` — `Option>` — tracks WS connection count separately from SSE. +- `chat_rate_limiter` — `RateLimiter` — 30 req/60 s sliding window shared across all chat send callers. +- `scheduler` — `Option` — used to inject follow-up messages into running agent jobs. +- `cost_guard` — `Option>` — exposes token usage / cost totals in the status endpoint. +- `startup_time` — `Instant` — used to compute uptime in the gateway status response. +- `registry_entries` — `Vec` — loaded once at startup from registry manifests; used by the available extensions API without hitting the network. + +Subsystems are wired via `with_*` builder methods on `GatewayChannel` (`mod.rs`). Each call rebuilds `Arc` — safe to call before `start()`, not after. + +## SSE / WebSocket Connection Limits + +Both SSE and WebSocket share the same `SseManager` broadcast channel. Key characteristics: + +- **Broadcast buffer:** 256 events. A slow client that falls behind will miss events — the `BroadcastStream` silently drops lagged events. SSE clients are expected to reconnect and re-fetch history. +- **Max connections:** 100 total (SSE + WebSocket combined). Connections beyond the limit receive a 503 / are immediately dropped. +- **SSE keepalive:** Axum's `KeepAlive` sends an empty event every **30 seconds** to prevent proxy timeouts. +- **WebSocket:** Two tasks per connection — a sender task (broadcast → WS frames) and a receiver loop (WS frames → agent). When the client disconnects, the sender is aborted and both the SSE connection counter and WS tracker counter are decremented. + +## CORS and Security Headers + +CORS is restricted to the gateway's own origin (same IP+port and `localhost`+port). Allowed methods: GET, POST, PUT, DELETE. Allowed headers: `Content-Type`, `Authorization`. Credentials are allowed. + +All responses include: +- `X-Content-Type-Options: nosniff` +- `X-Frame-Options: DENY` + +**Request body limit:** 1 MB (`DefaultBodyLimit::max(1024 * 1024)`). Larger payloads return 413. + +## Pending Approvals + +Tool approval state is **in-memory only** (not persisted to DB). Server restart clears all pending approvals. The `pending_approval` field in `HistoryResponse` is re-populated on thread switch from in-memory state. + +## Adding a New API Endpoint + +1. Define request/response types in `types.rs`. +2. Implement the handler in the appropriate `handlers/*.rs` file (or inline in `server.rs` for simple handlers). +3. Register the route in `start_server()` in `server.rs` under the correct router (`public`, `protected`, or `statics`). +4. If it is an SSE or WebSocket endpoint, add its path to `allows_query_token_auth()` in `auth.rs`. +5. If it requires a new `GatewayState` field, add it to the struct and to both the `GatewayChannel::new()` initializer and `rebuild_state()` in `mod.rs`, then add a `with_*` builder method. diff --git a/src/db/CLAUDE.md b/src/db/CLAUDE.md new file mode 100644 index 00000000..123b9d95 --- /dev/null +++ b/src/db/CLAUDE.md @@ -0,0 +1,174 @@ +# Database Module + +Dual-backend persistence layer. **All new persistence features must support both backends.** + +## Quick Reference + +```bash +# Default build (PostgreSQL) +cargo build + +# libSQL/Turso build +cargo build --no-default-features --features libsql + +# Both backends +cargo build --features "postgres,libsql" + +# Test each backend in isolation +cargo check # postgres (default) +cargo check --no-default-features --features libsql # libsql only +cargo check --all-features # both +``` + +## Files + +| File | Role | +|------|------| +| `mod.rs` | `Database` supertrait + 7 sub-traits (~78 async methods total) — add new ops here first | +| `postgres.rs` | PostgreSQL backend — delegates to `Store` + `Repository` in `history/` | +| `libsql/mod.rs` | libSQL/Turso backend struct, connection helpers, row parsing utilities | +| `libsql/conversations.rs` | `ConversationStore` impl | +| `libsql/jobs.rs` | `JobStore` impl | +| `libsql/sandbox.rs` | `SandboxStore` impl | +| `libsql/routines.rs` | `RoutineStore` impl | +| `libsql/settings.rs` | `SettingsStore` impl | +| `libsql/tool_failures.rs` | `ToolFailureStore` impl | +| `libsql/workspace.rs` | `WorkspaceStore` impl (FTS5 + vector search) | +| `libsql_migrations.rs` | Consolidated libSQL schema (CREATE IF NOT EXISTS, no ALTER TABLE) | +| `tls.rs` | TLS connector factory for PostgreSQL (`rustls` + system root certs) | + +PostgreSQL schema: `migrations/V1__initial.sql` through `V9__flexible_embedding_dimension.sql` (managed by `refinery`). V1 is the base schema; later migrations add tables, columns, and rename `claude_code_events` → `job_events`. + +## Trait Structure + +The `Database` supertrait is composed of seven sub-traits. Leaf consumers can depend on the narrowest sub-trait they need rather than the full `Database`: + +| Sub-trait | Methods | Covers | +|-----------|---------|--------| +| `ConversationStore` | 12 | Conversations, messages | +| `JobStore` | 13 | Agent jobs, actions, LLM calls, estimation | +| `SandboxStore` | 13 | Sandbox jobs, job events | +| `RoutineStore` | 15 | Routines, routine runs | +| `ToolFailureStore` | 4 | Self-repair tracking | +| `SettingsStore` | 8 | Per-user key-value settings | +| `WorkspaceStore` | 13 | Memory documents, chunks, hybrid search | + +`Database` adds `run_migrations()` and combines all sub-traits. + +## Adding a New Persistence Operation + +1. Decide which sub-trait the method belongs to, or create a new sub-trait +2. Add the async method signature to that sub-trait in `mod.rs` +3. Implement in `postgres.rs` (delegate to `Store` or `Repository`) +4. Implement in `libsql/.rs` (SQLite-dialect SQL, use `self.connect().await?` per operation) +5. Add migration if needed: + - PostgreSQL: new `migrations/VN__description.sql` + - libSQL: add `CREATE TABLE IF NOT EXISTS` to `libsql_migrations.rs` + +## SQL Dialect Differences + +| Feature | PostgreSQL | libSQL | +|---------|-----------|--------| +| UUIDs | `UUID` type | `TEXT` | +| Timestamps | `TIMESTAMPTZ` | `TEXT` (ISO-8601 RFC 3339 with ms precision) | +| JSON | `JSONB` | `TEXT` | +| Numeric/Decimal | `NUMERIC` | `TEXT` (preserves `rust_decimal` precision) | +| Arrays | `TEXT[]` | `TEXT` (JSON-encoded array) | +| Booleans | `BOOLEAN` | `INTEGER` (0/1) | +| Vector embeddings | `VECTOR` (any dim, V9 removed fixed 1536) | `F32_BLOB(1536)` via `libsql_vector_idx` | +| Full-text search | `tsvector` + `ts_rank_cd` | FTS5 virtual table + sync triggers | +| JSON path update | `jsonb_set(col, '{key}', val)` | `json_patch(col, '{"key": val}')` | +| PL/pgSQL | Functions | Triggers (no stored procs in SQLite) | +| Connection model | `deadpool-postgres` connection pool | New connection per operation (`self.connect()`) | +| Concurrency | Pool-based, fully concurrent | WAL mode + 5 s busy timeout; write serialized | +| Auto-timestamp | `DEFAULT NOW()` | `DEFAULT (datetime('now'))` | +| Timestamp parsing | Native type | Multi-format fallback in `parse_timestamp()` | + +**JSON merge patch gotcha:** libSQL uses RFC 7396 JSON Merge Patch (`json_patch`) for metadata updates. This replaces top-level keys entirely — it **cannot** do partial nested updates. PostgreSQL uses `jsonb_set` which is path-targeted. Don't rely on partial nested metadata updates if you need libSQL compat. + +**Boolean storage:** libSQL stores booleans as integers. When reading, use `get_i64(row, idx) != 0`; when writing, pass `1i64`/`0i64`. Never pass a Rust `bool` directly. + +**Timestamp write format:** Always write timestamps with `fmt_ts(dt)` (RFC 3339, millisecond precision). Read with `get_ts()` / `get_opt_ts()` which handle legacy naive formats too. + +**Vector dimension:** PostgreSQL V9 migration changed the column to unbounded `vector` (removing the HNSW index). libSQL still uses `F32_BLOB(1536)` — if you use a different-dimension embedding model, the libSQL schema needs updating too. + +**Connection per operation:** `LibSqlBackend::connect()` creates a fresh connection for every operation, sets `PRAGMA busy_timeout = 5000`, and closes it when the `Connection` is dropped. This is intentional — the libSQL SDK does not offer a pool. Avoid holding connections open across `await` points. + +## Schema: Key Tables + +**Core:** +- `conversations` — multi-channel conversation tracking +- `conversation_messages` — individual messages within a conversation +- `agent_jobs` — job metadata and status +- `job_actions` — event-sourced tool executions +- `job_events` — sandbox job streaming events (renamed from `claude_code_events` in V7) +- `dynamic_tools` — agent-built tools +- `llm_calls` — cost/token tracking +- `estimation_snapshots` — learning data +- `repair_attempts` — self-repair action log (not exposed via Database trait yet) + +**Workspace/Memory:** +- `memory_documents` — flexible path-based files +- `memory_chunks` — chunked content with FTS + vector indexes +- `memory_chunks_fts` — FTS5 virtual table (libSQL) / `tsvector` column (PostgreSQL) +- `heartbeat_state` — periodic execution tracking + +**Security/Extensions:** +- `secrets` — AES-256-GCM encrypted credentials +- `wasm_tools` — installed WASM tool binaries +- `tool_capabilities` — per-tool HTTP allowlist, secret access, rate limits +- `leak_detection_patterns` — secret regex patterns (seed data in both backends) +- `leak_detection_events` — audit log of detected leaks +- `secret_usage_log` — per-request credential injection audit trail +- `tool_rate_limit_state` — sliding window rate limit counters + +**Other:** +- `routines`, `routine_runs` — scheduled/reactive execution +- `settings` — per-user key-value +- `tool_failures` — broken tool tracking for self-repair +- `_migrations` — libSQL-only internal migration version tracking + +## libSQL Current Limitations + +- **Secrets store** — still requires `PostgresSecretsStore`; `LibSqlSecretsStore` exists but is not plumbed through the main startup path +- **Settings reload** — `Config::from_db` skipped (requires `Store`) +- **No incremental migrations** — schema is idempotent CREATE IF NOT EXISTS; no ALTER TABLE support; column additions require a new versioned approach +- **No encryption at rest** — only secrets (API tokens) are AES-256-GCM encrypted; all other data is plaintext SQLite +- **Hybrid search** — both FTS5 and vector search (`libsql_vector_idx`) are implemented; however, the vector index is fixed at `F32_BLOB(1536)` while PostgreSQL switched to unbounded `vector` in V9 +- **Write serialization** — WAL mode allows concurrent readers but only one writer at a time; busy timeout is 5 s, which may cause timeouts under high write concurrency + +## Running Locally with libSQL + +```bash +# Use local SQLite file (default) +DATABASE_BACKEND=libsql LIBSQL_PATH=~/.ironclaw/test.db cargo run + +# Use Turso cloud (embedded replica syncs local file to cloud) +DATABASE_BACKEND=libsql LIBSQL_URL=libsql://xxx.turso.io LIBSQL_AUTH_TOKEN=xxx cargo run + +# In-memory (tests only — data is lost when the process exits) +# Use LibSqlBackend::new_memory() directly in test code +``` + +## Testing the libSQL Backend + +Use `LibSqlBackend::new_memory()` in unit tests — no files, no cleanup required: + +```rust +#[tokio::test] +async fn test_my_feature() { + let backend = LibSqlBackend::new_memory().await.unwrap(); + backend.run_migrations().await.unwrap(); + // backend implements Database — call any trait method +} +``` + +For concurrency tests that require multiple connections sharing state, use `LibSqlBackend::new_local(&tmp_path)` with a `tempfile::tempdir()`. In-memory databases do not share state between connections. + +## Sharing the libSQL Database Handle + +`LibSqlBackend::shared_db()` returns an `Arc` for passing to satellite stores (e.g., `LibSqlSecretsStore`, `LibSqlWasmToolStore`) that need their own connections per-operation but should share the same underlying database file. These stores call `.connect()` on the shared handle themselves. This is the correct pattern — do not pass a live `Connection` to satellite stores. + +## Pattern: Fix the Pattern, Not the Instance + +When fixing a bug in one backend's SQL, always grep for the same pattern in the other backend. A fix to `postgres.rs` that doesn't also fix the libSQL module (e.g., `libsql/jobs.rs`) is half a fix. The same applies to satellite types like `LibSqlSecretsStore` or `LibSqlWasmToolStore`. diff --git a/src/llm/CLAUDE.md b/src/llm/CLAUDE.md new file mode 100644 index 00000000..a1eb72be --- /dev/null +++ b/src/llm/CLAUDE.md @@ -0,0 +1,174 @@ +# LLM Module + +Multi-provider LLM integration with circuit breaker, retry, failover, and response caching. + +## File Map + +| File | Role | +|------|------| +| `mod.rs` | Provider factory (`create_llm_provider`, `build_provider_chain`); `LlmBackend` enum | +| `provider.rs` | `LlmProvider` trait, `ChatMessage`, `ToolCall`, `CompletionRequest`, `sanitize_tool_messages` | +| `nearai_chat.rs` | NEAR AI Chat Completions provider (dual auth: session token or API key) | +| `reasoning.rs` | `Reasoning` struct, `ReasoningContext`, `RespondResult`, `ActionPlan`, `ToolSelection`; thinking-tag stripping; `SILENT_REPLY_TOKEN` | +| `session.rs` | NEAR AI session token management with disk + DB persistence, OAuth login flow | +| `circuit_breaker.rs` | Circuit breaker: Closed → Open → HalfOpen state machine | +| `retry.rs` | Exponential backoff retry wrapper; `is_retryable()` classification | +| `failover.rs` | `FailoverProvider` — tries providers in order with per-provider cooldown | +| `response_cache.rs` | In-memory LLM response cache with TTL and LRU eviction (keyed by SHA-256) | +| `costs.rs` | Static per-model cost table (OpenAI, Anthropic, local/Ollama heuristics) | +| `rig_adapter.rs` | Adapter bridging rig-core `CompletionModel` → `LlmProvider`; used by OpenAI, Anthropic, Ollama, Tinfoil | +| `smart_routing.rs` | `SmartRoutingProvider` — 13-dimension complexity scorer routes cheap vs primary model | +| `recording.rs` | `RecordingLlm` — trace capture for E2E replay testing (`IRONCLAW_RECORD_TRACE`) | + +## Provider Selection + +Set via `LLM_BACKEND` env var: + +| Value | Provider | Key env vars | +|-------|----------|-------------| +| `nearai` (default) | NEAR AI Chat Completions | `NEARAI_SESSION_TOKEN` or `NEARAI_API_KEY` | +| `openai` | OpenAI | `OPENAI_API_KEY` | +| `anthropic` | Anthropic | `ANTHROPIC_API_KEY` | +| `ollama` | Ollama local | `OLLAMA_BASE_URL` | +| `openai_compatible` | Any OpenAI-compatible endpoint | `LLM_BASE_URL`, `LLM_API_KEY`, `LLM_MODEL` | +| `tinfoil` | Tinfoil TEE inference | `TINFOIL_API_KEY`, `TINFOIL_MODEL` | + +## NEAR AI Provider Gotchas + +**Dual auth modes:** +- **Session token** (default): `NEARAI_SESSION_TOKEN=sess_...`, base URL = `https://private.near.ai`. Tokens are persisted to `~/.ironclaw/session.json` (mode 0600) and optionally to the DB `settings` table (`nearai.session_token`). On 401 responses where the body contains "session" + "expired"/"invalid", `NearAiChatProvider` calls `session.handle_auth_failure()` which triggers the interactive OAuth login flow and retries once. Plain `AuthFailed` 401s are not retried. +- **API key**: Set `NEARAI_API_KEY` (from `cloud.near.ai`), base URL defaults to `https://cloud-api.near.ai`. 401s with API key auth are immediately returned as `LlmError::AuthFailed` — no renewal. + +**Session renewal is interactive:** When `SessionExpired` triggers renewal, it blocks and prompts the user in the terminal (GitHub/Google OAuth or manual API key entry). This is unsuitable for headless/hosted deployments — set `NEARAI_SESSION_TOKEN` env var instead. + +**Tool message flattening:** NEAR AI's API doesn't support `role: "tool"` messages in the standard format. `nearai_chat.rs` defaults `flatten_tool_messages = true`, converting tool results to user messages with `[Tool result from ]: ` format. Use `NearAiChatProvider::new_with_flatten(..., false)` to disable for compliant endpoints. + +**Pricing auto-fetch:** On startup, `NearAiChatProvider` fires a background task to fetch per-model pricing from `/v1/model/list`. If the fetch fails, it silently falls back to `costs::model_cost()` / `costs::default_cost()`. Pricing is stored in-memory only. + +**HTTP request timeout:** The NEAR AI HTTP client has a 120-second timeout per request. Rate limit `Retry-After` headers are parsed (both delay-seconds and HTTP-date formats) and forwarded as `LlmError::RateLimited { retry_after }` for the `RetryProvider` to honor. + +## Circuit Breaker + +State machine in `circuit_breaker.rs`: +``` +Closed (normal) + → Open (after failure_threshold consecutive transient failures; default: 5) + → HalfOpen (after recovery_timeout; default: 30s) + → Closed (after half_open_successes_needed probe successes; default: 2) + → Open (if any probe fails) +``` + +**Transient vs non-transient errors:** Only `RequestFailed`, `RateLimited`, `InvalidResponse`, `SessionExpired`, `SessionRenewalFailed`, `Http`, and `Io` count toward the threshold. `AuthFailed`, `ContextLengthExceeded`, `ModelNotAvailable`, and `Json` errors never trip the breaker — they indicate caller problems, not backend degradation. + +Configure via `NearAiConfig` fields: `circuit_breaker_threshold` (None = disabled), `circuit_breaker_recovery_secs` (default: 30). + +The circuit breaker wraps the entire provider chain. When open, it immediately returns `LlmError::RequestFailed` with a message including remaining cooldown seconds. The `FailoverProvider` sitting outside can then try a fallback model. + +## Failover Chain + +`FailoverProvider` in `failover.rs` wraps a list of `LlmProvider` instances. On a retryable error, it tries the next provider in the list. Providers that fail repeatedly enter a cooldown period and are skipped (unless all providers are in cooldown, in which case the least-recently-cooled one is tried). + +**Cooldown defaults:** `failure_threshold = 3` consecutive retryable failures → cooldown for `cooldown_duration = 300s`. Configure via `NearAiConfig` fields: `failover_cooldown_secs`, `failover_cooldown_threshold`. + +**Current wiring:** The failover is set up between primary model and `NEARAI_FALLBACK_MODEL` (a different model name on the same NEAR AI backend), not across different LLM provider types. Cross-provider failover (e.g., NEAR AI → Anthropic) requires manual construction. + +## Retry + +`RetryProvider` in `retry.rs` wraps any `LlmProvider` with exponential backoff. Retries on: `RequestFailed`, `RateLimited`, `InvalidResponse`, `SessionRenewalFailed`, `Http`, `Io`. Does **not** retry: `AuthFailed`, `SessionExpired`, `ContextLengthExceeded`, `ModelNotAvailable`, `Json`. + +**Backoff schedule:** base 1s doubled per attempt with ±25% jitter, minimum floor 100ms. Attempt 0: ~1s, attempt 1: ~2s, attempt 2: ~4s. For `RateLimited`, uses the `retry_after` duration from the error (provider-supplied) instead of backoff. + +Configure via `NearAiConfig.max_retries` (env: `NEARAI_MAX_RETRIES`; default: 3). Set to 0 to disable. + +## LlmProvider Trait + +The full trait (all methods must be implemented or rely on defaults): + +```rust +#[async_trait] +pub trait LlmProvider: Send + Sync { + // Required + fn model_name(&self) -> &str; + fn cost_per_token(&self) -> (Decimal, Decimal); // (input, output) per token + async fn complete(&self, request: CompletionRequest) -> Result; + async fn complete_with_tools(&self, request: ToolCompletionRequest) -> Result; + + // Optional (have defaults) + async fn list_models(&self) -> Result, LlmError> { Ok(vec![]) } + async fn model_metadata(&self) -> Result { /* name only */ } + fn effective_model_name(&self, requested_model: Option<&str>) -> String { /* uses active */ } + fn active_model_name(&self) -> String { self.model_name().to_string() } + fn set_model(&self, _model: &str) -> Result<(), LlmError> { /* Err: not supported */ } + fn calculate_cost(&self, input_tokens: u32, output_tokens: u32) -> Decimal { /* uses cost_per_token */ } +} +``` + +Key notes: +- `model_name()` returns the configured model name; `active_model_name()` returns the currently active model (may differ if `set_model()` was called — only `NearAiChatProvider` supports this). +- `cost_per_token()` returns `(Decimal, Decimal)` using `rust_decimal`. Look up via `costs::model_cost()` in your constructor; fall back to `costs::default_cost()` for unknowns. +- `RigAdapter` ignores per-request model overrides (logs a warning). Only `NearAiChatProvider` supports per-request model overrides via `CompletionRequest::model`. +- `complete_with_tools()` is never cached (tool calls can have side effects) — `CachedProvider` always passes them through. + +To add a new provider: +1. Create `src/llm/myprovider.rs` implementing `LlmProvider` +2. Add variant to `LlmBackend` in `mod.rs` +3. Wire into the factory match in `mod.rs` +4. Add env vars to `config/llm.rs` and `.env.example` + +## Response Cache + +`CachedProvider` in `response_cache.rs` caches `complete()` responses. `complete_with_tools()` is never cached (side effects). Cache key is SHA-256 of `(model_name, messages_json, max_tokens, temperature, stop_sequences)`. LRU eviction when `max_entries` is reached; TTL-based expiry on access. + +**Defaults:** TTL = 1 hour, max entries = 1000. Configure via `NearAiConfig` fields: `response_cache_enabled` (env: `NEARAI_RESPONSE_CACHE_ENABLED`), `response_cache_ttl_secs`, `response_cache_max_entries`. Cache is in-memory only — evicted on restart. + +## OpenAI-Compatible Custom Headers + +Set `LLM_EXTRA_HEADERS=Key:Value,Key2:Value2` to inject headers into every request. Useful for OpenRouter attribution (`HTTP-Referer`, `X-Title`). Invalid header names/values are skipped with a warning (not a fatal error). + +## Provider Chain Construction + +`build_provider_chain()` in `mod.rs` is the single source of truth for assembling decorators. The chain is: + +``` +Raw provider + → RetryProvider (per-provider backoff; wraps both primary and fallback) + → SmartRoutingProvider (cheap/primary split when NEARAI_CHEAP_MODEL is set) + → FailoverProvider (fallback model; only when NEARAI_FALLBACK_MODEL is set) + → CircuitBreakerProvider (fast-fail; only when NEARAI_CIRCUIT_BREAKER_THRESHOLD is set) + → CachedProvider (response cache; only when NEARAI_RESPONSE_CACHE_ENABLED=true) + → RecordingLlm (trace capture; only when IRONCLAW_RECORD_TRACE is set) +``` + +`build_provider_chain()` also returns a separate standalone cheap LLM provider (for heartbeat/evaluation tasks — not part of the decorator chain). + +## reasoning.rs Contents + +`reasoning.rs` does **not** contain an `IntentClassifier`. It contains: +- `Reasoning` struct — the main reasoning engine used by the agent worker; calls `complete_with_tools()` and handles tool dispatch +- `ReasoningContext` — carries messages, available tools, job description, and metadata into a reasoning call +- `RespondResult`, `ActionPlan`, `ToolSelection` — output types from the reasoning engine +- `TokenUsage` — input/output token counts +- `SILENT_REPLY_TOKEN` (`"NO_REPLY"`) and `is_silent_reply()` — used by the dispatcher to suppress empty responses in group chats +- Thinking-tag stripping — regex-based removal of ``, ``, ``, `<|think|>`, ``, etc. from model responses before returning to the user + +## costs.rs Details + +`costs.rs` provides a static lookup table (`model_cost(model_id)`) returning `(input_cost, output_cost)` per token as `rust_decimal::Decimal`. Provider prefixes like `"openai/gpt-4o"` are stripped before lookup. Returns `None` for unknown models — callers should fall back to `default_cost()` (roughly GPT-4o pricing). Local model heuristic (`is_local_model()`) returns zero cost for Ollama-style identifiers (llama*, mistral*, `:latest`, `:instruct`, etc.). + +## rig_adapter.rs Details + +`RigAdapter` bridges any rig-core `CompletionModel` to `LlmProvider`. It is actively used in production for all non-NEAR AI providers (OpenAI, Anthropic, Ollama, Tinfoil, OpenAI-compatible). Key behaviors: +- **Per-request model overrides are silently ignored** (warning logged); the model is baked at construction time. +- **OpenAI strict-mode schema normalization** is applied to all tool definitions: `additionalProperties: false`, all properties added to `required`, optional fields made nullable via `"type": ["T", "null"]`. This happens transparently at the provider boundary. +- **System messages** are extracted into the rig-core `preamble` field (concatenated with newlines if multiple). +- **Tool call IDs** are generated (`generated_tool_call_{seed}`) if the provider returns empty/whitespace IDs. +- **Tool name normalization**: strips `proxy_` prefix if it matches a known tool (handles some proxy implementations). +- **OpenAI uses Chat Completions API** (`completions_api()`), not the newer Responses API — the Responses API path panics when tool results are sent back (rig-core doesn't thread `call_id` through `ToolCall`). + +## Streaming Support + +No streaming support. All providers use non-streaming (blocking) Chat Completions requests. The `complete()` and `complete_with_tools()` methods return only after the full response is available. + +## Trace Recording + +Set `IRONCLAW_RECORD_TRACE=1` to enable live trace recording via `RecordingLlm`. Traces are JSON files containing: memory snapshot, HTTP exchanges from tools, and LLM steps (user inputs, text responses, tool call responses). Replay these in E2E tests via `TraceLlm`. Configure output path with `IRONCLAW_TRACE_OUTPUT` (default: `trace_{timestamp}.json`). diff --git a/tests/e2e/CLAUDE.md b/tests/e2e/CLAUDE.md new file mode 100644 index 00000000..c977b6fd --- /dev/null +++ b/tests/e2e/CLAUDE.md @@ -0,0 +1,174 @@ +# IronClaw E2E Tests + +Python/Playwright test suite that runs against a live ironclaw instance. Added in PR #553 ("Trajectory benchmarks and e2e trace test rig"). + +## Setup + +```bash +cd tests/e2e + +# Create virtualenv (one-time) +python -m venv .venv +source .venv/bin/activate # or .venv\Scripts\activate on Windows + +# Install dependencies +pip install -e . + +# Install browser binaries (one-time) +playwright install chromium +``` + +Dependencies: `pytest`, `pytest-asyncio`, `pytest-playwright`, `pytest-timeout`, `playwright`, `aiohttp`, `httpx`. Optional: `anthropic` (vision extras). Requires Python >= 3.11. + +## Running Tests + +```bash +# Activate venv first +source .venv/bin/activate + +# Run all scenarios (conftest.py builds the binary and starts all servers automatically) +pytest scenarios/ + +# Run a specific scenario +pytest scenarios/test_chat.py +pytest scenarios/test_sse_reconnect.py + +# Run with verbose output +pytest scenarios/ -v + +# Run with a specific timeout (default is 120s per test, set in pyproject.toml) +pytest scenarios/ --timeout=60 + +# Run with a headed browser (useful for debugging) +HEADED=1 pytest scenarios/ +``` + +## Test Scenarios + +| File | What it tests | +|------|--------------| +| `test_connection.py` | Gateway reachability, tab navigation, auth rejection (no token shows auth screen) | +| `test_chat.py` | Send message via browser UI, verify streamed response from mock LLM; also tests empty-message suppression | +| `test_html_injection.py` | XSS vectors injected directly via `page.evaluate("addMessage('assistant', ...)")` are sanitized by `renderMarkdown`; user messages are shown as escaped plain text | +| `test_skills.py` | Skills tab UI visibility, ClawHub search (skipped if registry unreachable), install + remove lifecycle | +| `test_sse_reconnect.py` | SSE reconnects after programmatic `eventSource.close()` + `connectSSE()`; history is reloaded after reconnect | +| `test_tool_approval.py` | Approval card appears, buttons disable on approve/deny, parameters toggle; all triggered via `page.evaluate("showApproval(...)")` — no real tool call needed | + +## `helpers.py` + +Shared constants and utilities imported by every test file and `conftest.py`. + +- **`SEL`** — dict of CSS/ID selectors for all DOM elements (chat input, message bubbles, approval card, tab buttons, skill search, etc.). Update this dict when frontend HTML changes; tests import selectors from here rather than hardcoding them. +- **`TABS`** — ordered list of tab names: `["chat", "memory", "jobs", "routines", "extensions", "skills"]`. +- **`AUTH_TOKEN`** — hardcoded to `"e2e-test-token"`. Used by `conftest.py` when starting the server (`GATEWAY_AUTH_TOKEN`) and by the `page` fixture when navigating (`/?token=e2e-test-token`). +- **`wait_for_ready(url, timeout, interval)`** — polls a URL until HTTP 200 or timeout; used to wait for the gateway and mock LLM to become available. +- **`wait_for_port_line(process, pattern, timeout)`** — reads a subprocess's stdout line-by-line until a regex match; used to extract the dynamically assigned mock LLM port from `MOCK_LLM_PORT=XXXX`. + +## `conftest.py` and Fixtures + +All fixtures are defined in `tests/e2e/conftest.py`. Running `pytest scenarios/` from the `tests/e2e/` directory picks up this conftest automatically (it is one level above `scenarios/`). + +### Session-scoped fixtures (run once per `pytest` invocation) + +| Fixture | What it does | +|---------|-------------| +| `ironclaw_binary` | Checks `target/debug/ironclaw`; if absent, runs `cargo build --no-default-features --features libsql` (timeout 600s). | +| `mock_llm_server` | Starts `mock_llm.py --port 0`, reads the assigned port from stdout, waits for `/v1/models` to return 200. Yields the base URL. | +| `ironclaw_server` | Starts the ironclaw binary with a minimal env (see below), waits for `/api/health` (timeout 60s). Yields the base URL. On teardown sends **SIGINT** (not SIGTERM) so the tokio ctrl_c handler triggers a graceful shutdown and LLVM coverage data is flushed. | +| `browser` | Launches a single Chromium instance (headless by default; set `HEADED=1` for headed). Shared across all tests. | + +### Function-scoped fixtures + +| Fixture | What it does | +|---------|-------------| +| `page` | Creates a fresh browser **context** (viewport 1280×720) and **page** per test, navigates to `/?token=e2e-test-token`, and waits for `#auth-screen` to become hidden before yielding. Closes the context after each test. | + +The function-scoped `page` fixture means **each test gets a clean browser context** (cookies, storage, etc.) but reuses the same ironclaw server and browser process. Tests that need the server URL directly (e.g., `test_auth_rejection`) accept `ironclaw_server` as an additional parameter. + +### Environment passed to ironclaw in tests + +The `ironclaw_server` fixture injects a minimal, deterministic environment: + +``` +GATEWAY_ENABLED=true, GATEWAY_HOST=127.0.0.1, GATEWAY_PORT= +GATEWAY_AUTH_TOKEN=e2e-test-token, GATEWAY_USER_ID=e2e-tester +CLI_ENABLED=false +LLM_BACKEND=openai_compatible, LLM_BASE_URL=, LLM_MODEL=mock-model +DATABASE_BACKEND=libsql, LIBSQL_PATH=/e2e.db +SANDBOX_ENABLED=false, ROUTINES_ENABLED=false, HEARTBEAT_ENABLED=false +EMBEDDING_ENABLED=false, SKILLS_ENABLED=true +ONBOARD_COMPLETED=true # prevents setup wizard +``` + +The binary is also started with `--no-onboard`. Coverage env vars (`CARGO_LLVM_COV*`, `LLVM_*`, `CARGO_ENCODED_RUSTFLAGS`, `CARGO_INCREMENTAL`) are forwarded from the outer environment when present. + +## Mock LLM (`mock_llm.py`) + +An `aiohttp`-based OpenAI-compatible server used by tests that need deterministic LLM responses without hitting a real provider. + +```bash +# Start manually (port auto-selected, printed as MOCK_LLM_PORT=XXXX) +python mock_llm.py --port 0 +``` + +It serves `POST /v1/chat/completions` (streaming + non-streaming) and `GET /v1/models`. Responses are pattern-matched from `CANNED_RESPONSES` against the last user message. Unmatched messages return `"I understand your request."`. The model name reported is always `"mock-model"`. + +To add a new canned response: +```python +# In mock_llm.py +CANNED_RESPONSES = [ + (re.compile(r"your pattern", re.IGNORECASE), "Your response"), + ... +] +``` + +## Configuration + +`conftest.py` handles all server startup automatically — you do not need to start ironclaw manually before running `pytest`. The conftest builds the binary (libsql feature), starts the mock LLM, and starts ironclaw with a fresh temp database on every `pytest` invocation. + +If you need to test against a manually started ironclaw, you can skip conftest by running pytest with `--co` (collect-only) to understand what would run, or by calling the httpx/REST helpers directly without the `page` fixture. + +## Writing New Scenarios + +1. Create `scenarios/test_my_feature.py`. +2. All async functions are automatically recognized as tests — `asyncio_mode = "auto"` is set globally in `pyproject.toml`. Do **not** add `@pytest.mark.asyncio`; it is redundant and raises a warning. +3. Use the `page` fixture for browser tests (function-scoped, fresh context each test). Use `ironclaw_server` directly for pure HTTP tests. +4. Import selectors from `helpers.SEL` and `helpers.AUTH_TOKEN` — do not hardcode selectors or tokens inline. +5. Use `httpx.AsyncClient` for REST calls; `aiohttp` for SSE streaming. +6. Keep new fixtures session-scoped where possible; server startup is expensive. Function-scoped fixtures (like `page`) are fine for browser state that must be clean per test. + +```python +import httpx +from helpers import AUTH_TOKEN + +async def test_my_endpoint(ironclaw_server): + headers = {"Authorization": f"Bearer {AUTH_TOKEN}"} + async with httpx.AsyncClient() as client: + r = await client.get(f"{ironclaw_server}/api/health", headers=headers) + assert r.status_code == 200 +``` + +For browser tests: +```python +from helpers import SEL + +async def test_my_ui_feature(page): + # page is already navigated and authenticated + chat_input = page.locator(SEL["chat_input"]) + await chat_input.wait_for(state="visible", timeout=5000) + # ... interact with the page ... +``` + +### Gotchas + +- **`asyncio_default_fixture_loop_scope = "session"`** — all async fixtures share one event loop. Do not use `asyncio.run()` inside fixtures; use `await` directly. +- **The `page` fixture navigates with `/?token=e2e-test-token` and waits for `#auth-screen` to be hidden.** Tests receive a page that is already past the auth screen and has SSE connected. +- **`test_skills.py` makes real network calls to ClawHub.** Tests skip (not fail) if the registry is unreachable via `pytest.skip()`. +- **`test_html_injection.py` and `test_tool_approval.py` inject state via `page.evaluate(...)`.** They test the browser-side rendering pipeline and do not depend on the LLM or backend tool execution. +- **Browser is Chromium only.** `conftest.py` uses `p.chromium.launch()`; there is no Firefox or WebKit variant. +- **Default timeout is 120 seconds** (pyproject.toml). Individual `wait_for` calls inside tests use shorter timeouts (5–20s) for faster failure messages. +- **The libsql database is a temp directory** created fresh per `pytest` invocation; tests do not share state across runs. + +## CI Integration + +E2E tests run in CI with `cargo-llvm-cov` for coverage collection. The CI workflow (`fix(ci): persist all cargo-llvm-cov env vars for E2E coverage` — PR #559) sets `LLVM_PROFILE_FILE` and related vars before spawning the ironclaw binary so coverage from E2E runs is captured.