gitops/optimclaw
1
0
Fork 0
mirror of https://github.com/outbackdingo/optimclaw.git synced 2026-08-31 16:49:34 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: add comprehensive subdirectory CLAUDE.md files and update root (#589)

Browse Source 
* docs: add comprehensive subdirectory CLAUDE.md files and update root

The repo has grown significantly. This adds module-level CLAUDE.md files
for the five most complex subsystems, and updates the root CLAUDE.md to
reflect the actual current state of the codebase.

New files:
- src/agent/CLAUDE.md — full module map (19 files), session/thread/turn
  model, agentic loop flow, compaction strategies with correct thresholds,
  scheduler invariants, self-repair details, complete submission command
  reference table
- src/channels/web/CLAUDE.md — complete API route table (50+ endpoints),
  SSE event type reference, auth/rate limiting gotchas, connection limits,
  CORS headers, step-by-step endpoint addition guide
- src/db/CLAUDE.md — dual-backend build commands, sub-trait structure
  (7 sub-traits, ~67 methods), SQL dialect differences, boolean/timestamp
  gotchas, complete schema table, in-memory test helper, shared handle pattern
- src/llm/CLAUDE.md — corrected LlmProvider trait signatures, provider
  chain decorator order, NEAR AI dual-auth and session renewal details,
  circuit breaker thresholds, previously undocumented smart_routing.rs
  and recording.rs
- tests/e2e/CLAUDE.md — conftest fixtures and async scoping, environment
  injected into the binary, mock_llm canned responses, writing guide with
  correct asyncio usage, gotchas section

Root CLAUDE.md updates:
- Added E2E test setup and integration test commands
- Documented ~15 undocumented modules: cli/, registry/, hooks/, tunnel/,
  observability/, webhook_server.rs, cost_guard.rs, job_monitor.rs, etc.
- Corrected libSQL backend path (libsql/ directory, 8 sub-modules)
- Updated Database trait method count (~67, split across 7 sub-traits)
- Fixed stale references: config.rs → config/channels.rs, main.rs → app.rs
- Added Hook, Observer, Tunnel traits to extensibility section
- Added tunnel and observability env vars to Configuration section
- Removed resolved TODO (webhook trigger is now shipped)
- Added Module Specifications entries for all 5 new CLAUDE.md files

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

* docs: address PR review comments and reduce CLAUDE.md size

- Fix 7-sub-trait count (was 6) and ~78 async methods (was ~60/~67) in
  both CLAUDE.md and src/db/CLAUDE.md
- Add missing types.rs to secrets/ file tree (CLAUDE.md)
- Add missing tls.rs to src/db/CLAUDE.md Files table
- Fix method counts: ConversationStore 12, JobStore 13, RoutineStore 15
- Add Windows venv activation note to E2E setup commands
- Collapse agent/, web/, llm/, db/ file trees to one-liners (detail
  lives in their respective CLAUDE.md files)
- Replace verbose Database and LLM Providers sections with summaries
  linking to src/db/CLAUDE.md and src/llm/CLAUDE.md
- Root CLAUDE.md: 43,868 → 35,270 chars (fixes >40k perf warning)

[skip-regression-check]

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

---------

Co-authored-by: Claude Sonnet 4.6 <[email protected]>
This commit is contained in:
Henry Park
2026-03-07 08:33:09 +00:00
committed by GitHub
co-authored by Claude Sonnet 4.6
parent 45ec691f4c
commit 633b234e44
6 changed files with 1034 additions and 133 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/channels/web/CLAUDE.md
+212
View File
@@ -0,0 +1,212 @@
# Web Gateway Module
Browser-facing HTTP API and SSE/WebSocket real-time streaming. Axum-based, single-user with bearer token auth.
## File Map
| File | Role |
|------|------|
| `mod.rs` | Gateway builder, startup, `WebChannel` implementation, `with_*` builder methods |
| `server.rs` | `GatewayState`, `start_server()`, all Axum route registrations, inline handlers |
| `types.rs` | Request/response DTOs and `SseEvent` enum (source of truth for SSE contract) |
| `sse.rs` | `SseManager` — broadcast channel that fans out `SseEvent` to all connected SSE clients |
| `ws.rs` | WebSocket handler (`handle_ws_connection`) + `WsConnectionTracker` |
| `auth.rs` | Bearer token middleware (`Authorization: Bearer <GATEWAY_AUTH_TOKEN>`) |
| `log_layer.rs` | Tracing layer that tees log lines to the `/api/logs/events` SSE stream |
| `handlers/` | Handler functions split by domain: `chat`, `extensions`, `jobs`, `memory`, `routines`, `settings`, `skills`, `static_files` |
| `openai_compat.rs` | OpenAI-compatible proxy (`/v1/chat/completions`, `/v1/models`) |
| `util.rs` | Shared helpers (`build_turns_from_db_messages`, `truncate_preview`) |
| `static/` | Single-page app (HTML/CSS/JS) — embedded at compile time via `include_str!`/`include_bytes!` |
## API Routes
### Public (no auth)
| Method | Path | Description |
|--------|------|-------------|
| GET | `/api/health` | Health check |
| GET | `/oauth/callback` | OAuth callback for extension auth |
### Chat
| Method | Path | Description |
|--------|------|-------------|
| POST | `/api/chat/send` | Send message → queues to agent loop |
| GET | `/api/chat/events` | SSE stream of agent events |
| GET | `/api/chat/ws` | WebSocket alternative to SSE |
| GET | `/api/chat/history` | Paginated turn history for a thread |
| GET | `/api/chat/threads` | List threads (returns `assistant_thread` + regular threads) |
| POST | `/api/chat/thread/new` | Create new thread |
| POST | `/api/chat/approval` | Approve/deny/always a pending tool call |
| POST | `/api/chat/auth-token` | Submit auth token for an extension |
| POST | `/api/chat/auth-cancel` | Cancel pending auth flow |
### Memory
| Method | Path | Description |
|--------|------|-------------|
| GET | `/api/memory/tree` | Workspace directory tree |
| GET | `/api/memory/list` | List files at a path |
| GET | `/api/memory/read` | Read a workspace file |
| POST | `/api/memory/write` | Write a workspace file |
| POST | `/api/memory/search` | Hybrid FTS + vector search |
### Jobs (sandbox)
| Method | Path | Description |
|--------|------|-------------|
| GET | `/api/jobs` | List sandbox jobs |
| GET | `/api/jobs/summary` | Aggregated stats |
| GET | `/api/jobs/{id}` | Job detail |
| POST | `/api/jobs/{id}/cancel` | Cancel a running job |
| POST | `/api/jobs/{id}/restart` | Restart a failed job |
| POST | `/api/jobs/{id}/prompt` | Send follow-up prompt to Claude Code bridge |
| GET | `/api/jobs/{id}/events` | SSE stream for a specific job |
| GET | `/api/jobs/{id}/files/list` | List files in job workspace |
| GET | `/api/jobs/{id}/files/read` | Read a file from job workspace |
### Skills
| Method | Path | Description |
|--------|------|-------------|
| GET | `/api/skills` | List installed skills |
| POST | `/api/skills/search` | Search ClawHub registry + local skills |
| POST | `/api/skills/install` | Install a skill from ClawHub or by URL/content |
| DELETE | `/api/skills/{name}` | Remove an installed skill |
### Extensions
| Method | Path | Description |
|--------|------|-------------|
| GET | `/api/extensions` | Installed extensions |
| GET | `/api/extensions/tools` | All registered tools (from tool registry) |
| POST | `/api/extensions/install` | Install extension |
| GET | `/api/extensions/registry` | Available extensions from registry manifests |
| POST | `/api/extensions/{name}/activate` | Activate installed extension |
| POST | `/api/extensions/{name}/remove` | Remove extension |
| GET/POST | `/api/extensions/{name}/setup` | Extension setup wizard |
### Routines
| Method | Path | Description |
|--------|------|-------------|
| GET | `/api/routines` | List routines |
| GET | `/api/routines/summary` | Aggregated stats (total/enabled/disabled/failing/runs_today) |
| GET | `/api/routines/{id}` | Routine detail with recent run history |
| POST | `/api/routines/{id}/trigger` | Manually trigger a routine |
| POST | `/api/routines/{id}/toggle` | Enable/disable a routine |
| DELETE | `/api/routines/{id}` | Delete a routine |
| GET | `/api/routines/{id}/runs` | List runs for a specific routine |
### Settings
| Method | Path | Description |
|--------|------|-------------|
| GET | `/api/settings` | List all settings |
| GET | `/api/settings/export` | Export all settings as a map |
| POST | `/api/settings/import` | Bulk-import settings from a map |
| GET | `/api/settings/{key}` | Get a single setting |
| PUT | `/api/settings/{key}` | Set a single setting |
| DELETE | `/api/settings/{key}` | Delete a setting |
### Other
| Method | Path | Description |
|--------|------|-------------|
| GET | `/api/logs/events` | Live log stream (SSE) |
| GET/PUT | `/api/logs/level` | Get/set log level at runtime |
| GET | `/api/pairing/{channel}` | List pending pairing requests |
| POST | `/api/pairing/{channel}/approve` | Approve a pairing request |
| GET | `/api/gateway/status` | Server uptime, connected clients, config |
| POST | `/v1/chat/completions` | OpenAI-compatible LLM proxy |
| GET | `/v1/models` | OpenAI-compatible model list |
### Static / Project files
| Method | Path | Description |
|--------|------|-------------|
| GET | `/` | Single-page app HTML |
| GET | `/style.css` | App stylesheet |
| GET | `/app.js` | App JavaScript |
| GET | `/favicon.ico` | Favicon (cached 1 day) |
| GET | `/projects/{project_id}/` | Job workspace browser (redirects) |
| GET | `/projects/{project_id}/{*path}` | Serve file from job workspace (auth required) |
## SSE Event Types (`SseEvent` in `types.rs`)
The SSE contract — every field is `#[serde(tag = "type")]`:
| Type | When emitted |
|------|-------------|
| `response` | Final text response from agent |
| `stream_chunk` | Streaming token (partial response) |
| `thinking` | Agent status update during reasoning |
| `tool_started` | Tool call began |
| `tool_completed` | Tool call finished (includes success/error) |
| `tool_result` | Tool output preview |
| `status` | Generic status message |
| `job_started` | Sandbox job created |
| `job_message` | Message from sandbox worker |
| `job_tool_use` | Tool invoked inside sandbox |
| `job_tool_result` | Tool result from sandbox |
| `job_status` | Sandbox job status update |
| `job_result` | Sandbox job final result |
| `approval_needed` | Tool requires user approval (pauses agent) |
| `auth_required` | Extension needs auth credentials |
| `auth_completed` | Extension auth flow finished |
| `extension_status` | WASM channel activation status changed |
| `error` | Error from agent or gateway |
| `heartbeat` | SSE keepalive (empty payload) |
**SSE serialization:** Events use `#[serde(tag = "type")]` — the wire format is `{"type":"<variant>", ...fields}`. The SSE frame's `event:` field is set to the same string as `type` for easy `addEventListener` use in the browser.
**WebSocket envelope:** Over WebSocket, SSE events are wrapped as `{"type":"event","event_type":"<variant>","data":{...}}`. Ping/pong uses `{"type":"ping"}` / `{"type":"pong"}`. Client-to-server messages (`message`, `approval`, `auth_token`, `auth_cancel`) are defined in `WsClientMessage` in `types.rs`.
**To add a new SSE event:** Use the `add-sse-event` skill (`/add-sse-event`). It scaffolds the Rust variant, serialization, broadcast call, and frontend handler. Also add a matching arm to `WsServerMessage::from_sse_event()` in `types.rs`.
## Auth
All protected routes require `Authorization: Bearer <GATEWAY_AUTH_TOKEN>`. The token is set via `GATEWAY_AUTH_TOKEN` env var. Missing/wrong token → 401. The `Bearer` prefix is compared case-insensitively (RFC 6750).
**Query-string token auth (`?token=xxx`):** Because `EventSource` and WebSocket upgrades cannot set custom headers from the browser, three endpoints also accept the token as a URL query parameter: `/api/chat/events`, `/api/logs/events`, and `/api/chat/ws`. All other endpoints reject query-string tokens. If you add a new SSE or WebSocket endpoint, register its path in `allows_query_token_auth()` in `auth.rs`.
**If no `GATEWAY_AUTH_TOKEN` is configured**, a random 32-character alphanumeric token is generated at startup and printed to the console.
Rate limiting: chat send endpoints are capped at **30 messages per 60 seconds** (sliding window, not per-IP).
## GatewayState
The shared state struct (`server.rs`) holds refs to all subsystems. Fields are `Option<Arc<T>>` so the gateway can start even when optional subsystems (workspace, sandbox, skills) are disabled. Always null-check before use in handlers.
Key fields:
- `msg_tx` — `RwLock<Option<mpsc::Sender<IncomingMessage>>>` — sends messages to the agent loop; set when `start()` is called on the `Channel`.
- `sse` — `SseManager` — broadcast hub; call `state.sse.broadcast(event)` from any handler.
- `ws_tracker` — `Option<Arc<WsConnectionTracker>>` — tracks WS connection count separately from SSE.
- `chat_rate_limiter` — `RateLimiter` — 30 req/60 s sliding window shared across all chat send callers.
- `scheduler` — `Option<SchedulerSlot>` — used to inject follow-up messages into running agent jobs.
- `cost_guard` — `Option<Arc<CostGuard>>` — exposes token usage / cost totals in the status endpoint.
- `startup_time` — `Instant` — used to compute uptime in the gateway status response.
- `registry_entries` — `Vec<RegistryEntry>` — loaded once at startup from registry manifests; used by the available extensions API without hitting the network.
Subsystems are wired via `with_*` builder methods on `GatewayChannel` (`mod.rs`). Each call rebuilds `Arc<GatewayState>` — safe to call before `start()`, not after.
## SSE / WebSocket Connection Limits
Both SSE and WebSocket share the same `SseManager` broadcast channel. Key characteristics:
- **Broadcast buffer:** 256 events. A slow client that falls behind will miss events — the `BroadcastStream` silently drops lagged events. SSE clients are expected to reconnect and re-fetch history.
- **Max connections:** 100 total (SSE + WebSocket combined). Connections beyond the limit receive a 503 / are immediately dropped.
- **SSE keepalive:** Axum's `KeepAlive` sends an empty event every **30 seconds** to prevent proxy timeouts.
- **WebSocket:** Two tasks per connection — a sender task (broadcast → WS frames) and a receiver loop (WS frames → agent). When the client disconnects, the sender is aborted and both the SSE connection counter and WS tracker counter are decremented.
## CORS and Security Headers
CORS is restricted to the gateway's own origin (same IP+port and `localhost`+port). Allowed methods: GET, POST, PUT, DELETE. Allowed headers: `Content-Type`, `Authorization`. Credentials are allowed.
All responses include:
- `X-Content-Type-Options: nosniff`
- `X-Frame-Options: DENY`
**Request body limit:** 1 MB (`DefaultBodyLimit::max(1024 * 1024)`). Larger payloads return 413.
## Pending Approvals
Tool approval state is **in-memory only** (not persisted to DB). Server restart clears all pending approvals. The `pending_approval` field in `HistoryResponse` is re-populated on thread switch from in-memory state.
## Adding a New API Endpoint
1. Define request/response types in `types.rs`.
2. Implement the handler in the appropriate `handlers/*.rs` file (or inline in `server.rs` for simple handlers).
3. Register the route in `start_server()` in `server.rs` under the correct router (`public`, `protected`, or `statics`).
4. If it is an SSE or WebSocket endpoint, add its path to `allows_query_token_auth()` in `auth.rs`.
5. If it requires a new `GatewayState` field, add it to the struct and to both the `GatewayChannel::new()` initializer and `rebuild_state()` in `mod.rs`, then add a `with_*` builder method.