mirror of
https://github.com/outbackdingo/optimclaw.git
synced 2026-08-25 14:53:34 +00:00
94d101924e
* refactor: encapsulate leaked abstractions from main.rs and app.rs into owning modules Move module-specific initialization logic out of main.rs (1222→665 lines, -46%) and app.rs (944→780 lines, -17%) into their respective owning modules as public factory functions. This enforces separation of concerns so that adding a new DB backend, MCP transport, or channel doesn't require editing main.rs/app.rs. Key changes: - Tracing init functions → src/tracing_fmt.rs - DB connection factory (connect_with_handles + DatabaseHandles) → src/db/mod.rs - Secrets store factory (create_secrets_store) → src/secrets/mod.rs - MCP transport dispatch factory (create_client_from_config) → src/tools/mcp/factory.rs - Orchestrator setup (setup_orchestrator + OrchestratorSetup) → src/orchestrator/mod.rs - WASM channel setup (setup_wasm_channels) → src/channels/wasm/setup.rs - Worker entry points (run_worker, run_claude_bridge) → src/worker/mod.rs - Shared CLI secrets init (init_secrets_store) → src/cli/mod.rs - Tunnel startup (start_managed_tunnel) → src/tunnel/mod.rs - Onboard check (check_onboard_needed) → src/setup/mod.rs - ExtensionManager unified MCP: uses create_client_from_config via McpProcessManager, enabling stdio/Unix transports for hot-activated MCP servers - Deduplicated ~130 lines of secrets store init across cli/mcp.rs and cli/tool.rs - CLAUDE.md updated with module-owned initialization guideline [skip-regression-check] Co-Authored-By: Claude Opus 4.6 <[email protected]> * refactor: address review feedback — deduplicate db factory, extract channel helper - connect_from_config() now delegates to connect_with_handles() to eliminate duplicated backend-matching logic (Copilot review feedback) - Extract register_channel() helper from setup_wasm_channels() loop body to improve readability (Gemini review feedback) [skip-regression-check] Co-Authored-By: Claude Opus 4.6 <[email protected]> * style: fix rustfmt line wrapping in setup_wasm_channels Co-Authored-By: Claude Opus 4.6 <[email protected]> * test: add integration test for module-owned initialization factories Exercises the full factory chain end-to-end to verify nothing was lost when initialization logic was moved from main.rs/app.rs into owning modules: - connect_with_handles returns Database + populated backend handles - connect_from_config delegates correctly (produces working Database) - secrets::create_secrets_store builds working store from DatabaseHandles - db::create_secrets_store standalone factory round-trips secrets - Both secrets factories produce compatible stores (cross-read works) - ExtensionManager constructs with McpProcessManager and is functional - DatabaseHandles default is empty All tests run without external services using libsql in-memory/tempfile. Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: wire cli/mcp.rs and cli/tool.rs to shared init_secrets_store() Both files had inline implementations identical to cli::init_secrets_store(). Replace with delegation to complete the claimed deduplication. [skip-regression-check] Co-Authored-By: Claude Opus 4.6 <[email protected]> * style: fix rustfmt line wrapping in integration test Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix(review): remove unused Config import and deduplicate Error Handling section - Remove `#[allow(unused_imports)]` and unused `use crate::config::Config` from cli/tool.rs (no longer needed after delegating to shared `cli::init_secrets_store()`) - Remove duplicate Error Handling subsection from CLAUDE.md Key Patterns (all four bullets already exist in Code Style section and review-discipline.md) Addresses Copilot review comments. [skip-regression-check] Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix(review): address remaining Copilot review comments - secrets/mod.rs: clarify docstring that None is a normal no-db condition - app.rs: add comment explaining the empty_handles fallback path - orchestrator/mod.rs: combine duplicated sandbox condition into single block - setup/mod.rs: document env var reads and thread-safety caveat [skip-regression-check] Co-Authored-By: Claude Opus 4.6 <[email protected]> --------- Co-authored-by: Claude Opus 4.6 <[email protected]> Co-authored-by: Henry Park <[email protected]>
240 lines
13 KiB
Markdown
240 lines
13 KiB
Markdown
# IronClaw Development Guide
|
|
|
|
**IronClaw** is a secure personal AI assistant — user-first security, self-expanding tools, defense in depth, multi-channel access with proactive background execution.
|
|
|
|
## Build & Test
|
|
|
|
```bash
|
|
cargo fmt # format
|
|
cargo clippy --all --benches --tests --examples --all-features # lint (zero warnings)
|
|
cargo test # unit tests
|
|
cargo test --features integration # + PostgreSQL tests
|
|
RUST_LOG=ironclaw=debug cargo run # run with logging
|
|
```
|
|
|
|
E2E tests: see `tests/e2e/CLAUDE.md`.
|
|
|
|
## Code Style
|
|
|
|
- Prefer `crate::` for cross-module imports; `super::` is fine in tests and intra-module refs
|
|
- No `pub use` re-exports unless exposing to downstream consumers
|
|
- No `.unwrap()` or `.expect()` in production code (tests are fine)
|
|
- Use `thiserror` for error types in `error.rs`
|
|
- Map errors with context: `.map_err(|e| SomeError::Variant { reason: e.to_string() })?`
|
|
- Prefer strong types over strings (enums, newtypes)
|
|
- Keep functions focused, extract helpers when logic is reused
|
|
- Comments for non-obvious logic only
|
|
|
|
## Architecture
|
|
|
|
Prefer generic/extensible architectures over hardcoding specific integrations. Ask clarifying questions about the desired abstraction level before implementing.
|
|
|
|
Key traits for extensibility: `Database`, `Channel`, `Tool`, `LlmProvider`, `SuccessEvaluator`, `EmbeddingProvider`, `NetworkPolicyDecider`, `Hook`, `Observer`, `Tunnel`.
|
|
|
|
All I/O is async with tokio. Use `Arc<T>` for shared state, `RwLock` for concurrent access.
|
|
|
|
## Project Structure
|
|
|
|
```
|
|
src/
|
|
├── lib.rs # Library root, module declarations
|
|
├── main.rs # Entry point, CLI args, startup
|
|
├── app.rs # App startup orchestration (channel wiring, DB init)
|
|
├── bootstrap.rs # Base directory resolution (~/.ironclaw), early .env loading
|
|
├── settings.rs # User settings persistence (~/.ironclaw/settings.json)
|
|
├── service.rs # OS service management (launchd/systemd daemon install)
|
|
├── tracing_fmt.rs # Custom tracing formatter
|
|
├── util.rs # Shared utilities
|
|
├── config/ # Configuration from env vars (split by subsystem)
|
|
│ ├── mod.rs # Re-exports all config types; top-level Config struct
|
|
│ ├── agent.rs, llm.rs, channels.rs, database.rs, sandbox.rs, skills.rs
|
|
│ ├── heartbeat.rs, routines.rs, safety.rs, embeddings.rs, wasm.rs
|
|
│ ├── tunnel.rs # Tunnel provider config (TUNNEL_PROVIDER, TUNNEL_URL, etc.)
|
|
│ └── secrets.rs, hygiene.rs, builder.rs, helpers.rs
|
|
├── error.rs # Error types (thiserror)
|
|
│
|
|
├── agent/ # Core agent loop, dispatcher, scheduler, sessions — see src/agent/CLAUDE.md
|
|
│
|
|
├── channels/ # Multi-channel input
|
|
│ ├── channel.rs # Channel trait, IncomingMessage, OutgoingResponse
|
|
│ ├── manager.rs # ChannelManager merges streams
|
|
│ ├── cli/ # Full TUI with Ratatui
|
|
│ ├── http.rs # HTTP webhook (axum) with secret validation
|
|
│ ├── webhook_server.rs # Unified HTTP server composing all webhook routes
|
|
│ ├── repl.rs # Simple REPL (for testing)
|
|
│ ├── web/ # Web gateway (browser UI) — see src/channels/web/CLAUDE.md
|
|
│ └── wasm/ # WASM channel runtime
|
|
│ ├── mod.rs
|
|
│ ├── bundled.rs # Bundled channel discovery
|
|
│ ├── capabilities.rs # Channel-specific capabilities (HTTP endpoint, emit rate)
|
|
│ ├── error.rs # WASM channel error types
|
|
│ ├── runtime.rs # WASM channel execution runtime
|
|
│ ├── setup.rs # WasmChannelSetup, setup_wasm_channels(), inject_channel_credentials()
|
|
│ └── wrapper.rs # Channel trait wrapper for WASM modules
|
|
│
|
|
├── cli/ # CLI subcommands (clap)
|
|
│ ├── mod.rs # Cli struct, Command enum (run/onboard/config/tool/registry/mcp/memory/pairing/service/doctor/status/completion)
|
|
│ └── config.rs, tool.rs, registry.rs, mcp.rs, memory.rs, pairing.rs, service.rs, doctor.rs, status.rs, completion.rs
|
|
│
|
|
├── registry/ # Extension registry catalog
|
|
│ ├── manifest.rs # ExtensionManifest, ArtifactSpec, BundleDefinition types
|
|
│ ├── catalog.rs # RegistryCatalog: load from filesystem and embedded JSON
|
|
│ └── installer.rs # RegistryInstaller: download, verify, install WASM artifacts
|
|
│
|
|
├── hooks/ # Lifecycle hooks (6 points: BeforeInbound, BeforeToolCall, BeforeOutbound, OnSessionStart, OnSessionEnd, TransformResponse)
|
|
│
|
|
├── tunnel/ # Tunnel abstraction for public internet exposure
|
|
│ ├── mod.rs # Tunnel trait, TunnelProviderConfig, create_tunnel(), start_managed_tunnel()
|
|
│ ├── cloudflare.rs # CloudflareTunnel (cloudflared binary)
|
|
│ ├── ngrok.rs # NgrokTunnel
|
|
│ ├── tailscale.rs # TailscaleTunnel (serve/funnel modes)
|
|
│ ├── custom.rs # CustomTunnel (arbitrary command with {host}/{port})
|
|
│ └── none.rs # NoneTunnel (local-only, no exposure)
|
|
│
|
|
├── observability/ # Pluggable event/metric recording (noop, log, multi)
|
|
│
|
|
├── orchestrator/ # Internal HTTP API for sandbox containers
|
|
│ ├── api.rs # Axum endpoints (LLM proxy, events, prompts)
|
|
│ ├── auth.rs # Per-job bearer token store
|
|
│ └── job_manager.rs # Container lifecycle (create, stop, cleanup)
|
|
│
|
|
├── worker/ # Runs inside Docker containers
|
|
│ ├── runtime.rs # Worker execution loop (tool calls, LLM)
|
|
│ ├── claude_bridge.rs # Claude Code bridge (spawns claude CLI)
|
|
│ └── proxy_llm.rs # LlmProvider that proxies through orchestrator
|
|
│
|
|
├── safety/ # Prompt injection defense
|
|
│ ├── sanitizer.rs # Pattern detection, content escaping
|
|
│ ├── validator.rs # Input validation (length, encoding, patterns)
|
|
│ ├── policy.rs # PolicyRule system with severity/actions
|
|
│ ├── leak_detector.rs # Secret detection (API keys, tokens, etc.)
|
|
│ └── credential_detect.rs # HTTP request credential detection
|
|
│
|
|
├── llm/ # Multi-provider LLM integration — see src/llm/CLAUDE.md
|
|
│
|
|
├── tools/ # Extensible tool system
|
|
│ ├── tool.rs # Tool trait, ToolOutput, ToolError
|
|
│ ├── registry.rs # ToolRegistry for discovery
|
|
│ ├── rate_limiter.rs # Shared sliding-window rate limiter
|
|
│ ├── builtin/ # Built-in tools (echo, time, json, http, web_fetch, file, shell, memory, message, job, routine, extension_tools, skill_tools, secrets_tools)
|
|
│ ├── builder/ # Dynamic tool building
|
|
│ │ ├── core.rs # BuildRequirement, SoftwareType, Language
|
|
│ │ ├── templates.rs # Project scaffolding
|
|
│ │ ├── testing.rs # Test harness integration
|
|
│ │ └── validation.rs # WASM validation
|
|
│ ├── mcp/ # Model Context Protocol
|
|
│ │ ├── client.rs # MCP client over HTTP
|
|
│ │ ├── factory.rs # create_client_from_config() — transport dispatch factory
|
|
│ │ ├── protocol.rs # JSON-RPC types
|
|
│ │ └── session.rs # MCP session management (Mcp-Session-Id header, per-server state)
|
|
│ └── wasm/ # Full WASM sandbox (wasmtime)
|
|
│ ├── runtime.rs # Module compilation and caching
|
|
│ ├── wrapper.rs # Tool trait wrapper for WASM modules
|
|
│ ├── host.rs # Host functions (logging, time, workspace)
|
|
│ ├── limits.rs # Fuel metering and memory limiting
|
|
│ ├── allowlist.rs # Network endpoint allowlisting
|
|
│ ├── credential_injector.rs # Safe credential injection
|
|
│ ├── loader.rs # WASM tool discovery from filesystem
|
|
│ ├── rate_limiter.rs # Per-tool rate limiting
|
|
│ ├── error.rs # WASM-specific error types
|
|
│ └── storage.rs # Linear memory persistence
|
|
│
|
|
├── db/ # Dual-backend persistence (PostgreSQL + libSQL) — see src/db/CLAUDE.md
|
|
│
|
|
├── workspace/ # Persistent memory system — see src/workspace/README.md
|
|
│
|
|
├── context/ # Job context isolation (JobState, JobContext, ContextManager)
|
|
├── estimation/ # Cost/time/value estimation with EMA learning
|
|
├── evaluation/ # Success evaluation (rule-based, LLM-based)
|
|
│
|
|
├── sandbox/ # Docker execution sandbox
|
|
│ ├── config.rs # SandboxConfig, SandboxPolicy enum (ReadOnly/WorkspaceWrite/FullAccess)
|
|
│ ├── manager.rs # SandboxManager orchestration
|
|
│ ├── container.rs # ContainerRunner, Docker lifecycle
|
|
│ └── proxy/ # Network proxy: domain allowlist, credential injection, CONNECT tunnel
|
|
│
|
|
├── secrets/ # Secrets management (AES-256-GCM, OS keychain for master key)
|
|
│
|
|
├── setup/ # 7-step onboarding wizard — see src/setup/README.md
|
|
│
|
|
├── skills/ # SKILL.md prompt extension system — see .claude/rules/skills.md
|
|
│
|
|
└── history/ # Persistence (PostgreSQL repositories, analytics)
|
|
|
|
tests/
|
|
├── *.rs # Integration tests (workspace, heartbeat, WS gateway, pairing, etc.)
|
|
├── test-pages/ # HTML→Markdown conversion fixtures
|
|
└── e2e/ # Python/Playwright E2E scenarios (see tests/e2e/CLAUDE.md)
|
|
```
|
|
|
|
## Database
|
|
|
|
Dual-backend: PostgreSQL + libSQL/Turso. **All new persistence features must support both backends.** See `src/db/CLAUDE.md` and `.claude/rules/database.md`.
|
|
|
|
## Module Specs
|
|
|
|
When modifying a module with a spec, read the spec first. Code follows spec; spec is the tiebreaker.
|
|
|
|
**Module-owned initialization:** Module-specific initialization logic (database connection, transport creation, channel setup) must live in the owning module as a public factory function — not in `main.rs` or `app.rs`. These entry-point files orchestrate calls to module factories. Feature-flag branching (`#[cfg(feature = ...)]`) must be confined to the module that owns the abstraction.
|
|
|
|
| Module | Spec |
|
|
|--------|------|
|
|
| `src/agent/` | `src/agent/CLAUDE.md` |
|
|
| `src/channels/web/` | `src/channels/web/CLAUDE.md` |
|
|
| `src/db/` | `src/db/CLAUDE.md` |
|
|
| `src/llm/` | `src/llm/CLAUDE.md` |
|
|
| `src/setup/` | `src/setup/README.md` |
|
|
| `src/tools/` | `src/tools/README.md` |
|
|
| `src/workspace/` | `src/workspace/README.md` |
|
|
| `tests/e2e/` | `tests/e2e/CLAUDE.md` |
|
|
|
|
## Job State Machine
|
|
|
|
```
|
|
Pending -> InProgress -> Completed -> Submitted -> Accepted
|
|
\-> Failed
|
|
\-> Stuck -> InProgress (recovery)
|
|
\-> Failed
|
|
```
|
|
|
|
## Skills System
|
|
|
|
SKILL.md files extend the agent's prompt with domain-specific instructions. See `.claude/rules/skills.md` for full details.
|
|
|
|
- **Trust model**: Trusted (user-placed in `~/.ironclaw/skills/` or workspace `skills/`, full tool access) vs Installed (registry, read-only tools)
|
|
- **Selection pipeline**: gating (check bin/env/config requirements) -> scoring (keywords/patterns/tags) -> budget (fit within `SKILLS_MAX_TOKENS`) -> attenuation (trust-based tool ceiling)
|
|
- **Skill tools**: `skill_list`, `skill_search`, `skill_install`, `skill_remove`
|
|
|
|
## Configuration
|
|
|
|
See `.env.example` for all environment variables. LLM backends (`nearai`, `openai`, `anthropic`, `ollama`, `openai_compatible`, `tinfoil`, `bedrock`) documented in `src/llm/CLAUDE.md`.
|
|
|
|
## Adding a New Channel
|
|
|
|
1. Create `src/channels/my_channel.rs`
|
|
2. Implement the `Channel` trait
|
|
3. Add config in `src/config/channels.rs`
|
|
4. Wire up in `src/app.rs` channel setup section
|
|
|
|
## Workspace & Memory
|
|
|
|
Persistent memory with hybrid search (FTS + vector via RRF). Four tools: `memory_search`, `memory_write`, `memory_read`, `memory_tree`. Identity files (AGENTS.md, SOUL.md, USER.md, IDENTITY.md) injected into system prompt. Heartbeat system runs proactive periodic execution (default: 30 minutes), reading `HEARTBEAT.md` and notifying via channel if findings. See `src/workspace/README.md`.
|
|
|
|
## Debugging
|
|
|
|
```bash
|
|
RUST_LOG=ironclaw=trace cargo run # verbose
|
|
RUST_LOG=ironclaw::agent=debug cargo run # agent module only
|
|
RUST_LOG=ironclaw=debug,tower_http=debug cargo run # + HTTP request logging
|
|
```
|
|
|
|
## Current Limitations
|
|
|
|
1. Domain-specific tools (`marketplace.rs`, `restaurant.rs`, etc.) are stubs
|
|
2. Integration tests need testcontainers for PostgreSQL
|
|
3. MCP: no streaming support; stdio/HTTP/Unix transports all use request-response
|
|
4. WIT bindgen: auto-extract tool schema from WASM is stubbed
|
|
5. Built tools get empty capabilities; need UX for granting access
|
|
6. No tool versioning or rollback
|
|
7. Observability: only `log` and `noop` backends (no OpenTelemetry)
|