Compare commits

...
Author SHA1 Message Date
Illia PolosukhinandClaude Opus 4.6 09e6a7e6d8 fix: review fixes for libSQL backend (shared connections, panics, indexes)
- Replace .expect() with proper error propagation in 3 call sites
- Share Arc<Database> between backend and stores instead of single Connection
- Add connect-per-operation pattern to LibSqlSecretsStore and LibSqlWasmToolStore
- Wrap store() INSERT + SELECT-back in a transaction
- Add ~22 missing indexes for parity with PostgreSQL schema
- Add 18 leak_detection_patterns seed rows matching PostgreSQL V2 migration
- Fix super:: import to use crate:: style
- Gate mask_password_in_url behind #[cfg(feature = "postgres")]
- Rewrite secrets store init with or_else chain for runtime backend selection

Co-Authored-By: Claude Opus 4.6 <[email protected]>
2026-02-13 17:03:58 -08:00
ZakiandClaude Opus 4.6 5814d77b16 fix: add missing JobContext fields and resolve fmt/clippy warnings
Add total_tokens_used and max_tokens fields to JobContext in
libsql_backend.rs, apply cargo fmt, and fix clippy warnings.

Co-Authored-By: Claude Opus 4.6 <[email protected]>
2026-02-13 08:24:03 -08:00
ZakiandClaude Opus 4.6 83773af997 merge: Resolve conflicts with main, add user-scoped DB methods
Merge main's security hardening (user-scoped job/conversation access,
cargo-dist config, CI improvements) into turso branch.

Add Database trait methods for user-scoped operations:
- list_sandbox_jobs_for_user
- sandbox_job_summary_for_user
- sandbox_job_belongs_to_user
- conversation_belongs_to_user

Implemented in both postgres and libsql backends.

Co-Authored-By: Claude Opus 4.6 <[email protected]>
2026-02-13 07:30:27 -08:00
bbb68f7490 Add OpenAI-compatible HTTP API (/v1/chat/completions, /v1/models) (#31)
* feat: add OpenAI-compatible HTTP API (/v1/chat/completions, /v1/models)

* - Reject model mismatches: validate req.model against the active model
    and return 404 model_not_found instead of silently ignoring it
  - Add x-ironclaw-streaming: simulated response header so clients know
    streaming is not true token-by-token delivery
  - Use SSE event type "error" for mid-stream LLM failures so clients can
    distinguish errors from content chunks
  - Mark docker-compose credentials as dev-only
  - Add integration tests for model mismatch, streaming header, and body
    size limit (axum's default 2MB)

* fix: address Copilot review feedback on OpenAI-compat API

- Wire chat_rate_limiter into /v1/chat/completions handler
- Execute LLM before starting SSE stream so failures return proper HTTP
  errors instead of SSE error events
- Validate tool-role messages require tool_call_id and name fields
- Surface list_models() errors in models_handler via map_llm_error
- Reject unknown roles with 400 instead of defaulting to User

Co-Authored-By: Claude Opus 4.6 <[email protected]>

---------

Co-authored-by: firat.sertgoz <[email protected]>
Co-authored-by: Claude Opus 4.6 <[email protected]>
2026-02-13 18:26:49 +04:00
b3dee13954 fix: flatten tool messages for NEAR AI cloud-api compatibility (#41)
* fix: flatten tool messages for NEAR AI cloud-api compatibility

NEAR AI cloud-api does not support the OpenAI multi-turn tool-calling
protocol (role:"tool" messages cause HTTP 400). This adds a
flatten_tool_messages() pass in NearAiChatProvider that rewrites
assistant tool_call messages and tool result messages into plain
assistant/user text before sending to the API. The model still sees
the tool execution history, just in a text format it can process.

Also includes a minor fix to telegram channel send_pairing_reply
for updated WASM host function signature.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: resolve CI failures in fmt, rate limiting, and test configuration

- Apply cargo fmt to nearai_chat.rs formatting violations
- Fix truncate(true) bug in record_failed_approve that cleared the
  attempts file before reading, preventing rate limit from ever
  triggering
- Skip bundled channel test when WASM build artifacts are unavailable
  (CI lacks wasm32-wasip2 target)
- Split CI test workflow to exclude workspace_integration tests that
  require PostgreSQL

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: resolve clippy unnecessary_unwrap lint (Rust 1.93)

Replace is_some() + unwrap() pattern with if-let binding to satisfy
clippy::unnecessary_unwrap which is now deny-by-default.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

---------

Co-authored-by: Claude Opus 4.6 <[email protected]>
Co-authored-by: firat.sertgoz <[email protected]>
Co-authored-by: Illia Polosukhin <[email protected]>
2026-02-13 06:18:43 +00:00
33ef0a6ea5 fix: security hardening across all layers (#35)
* fix: comprehensive security hardening across all layers

Critical:
- Replace --dangerously-skip-permissions with explicit tool allowlist
  via settings.json (Claude Code bridge)
- Constant-time token comparison (subtle crate) in web auth and
  orchestrator auth to prevent timing attacks

High:
- Revoke tokens and clean up handles on container creation failure
- Drop SETUID/SETGID capabilities from containers (keep only CHOWN)
- Disable redirect following in HTTP tool and WASM wrapper (SSRF)
- Reject URL userinfo (@) in WASM allowlist parser (host confusion)
- Fix binary body bypassing leak detection (from_utf8 -> from_utf8_lossy)
- Protect identity files from LLM overwrites (prompt injection defense)
- Prevent tool shadowing: built-in tools cannot be replaced dynamically
- User-scoped job APIs: list/detail/cancel/restart/prompt/events/files
- CORS restricted to localhost origins, WebSocket origin validation
- Sandbox shell fail-closed: no silent fallback to unsandboxed execution
- Scrub secrets from log broadcaster before SSE broadcast
- XSS sanitization on rendered markdown in web UI
- WASM epoch ticker thread so timeout deadlines actually fire

Medium:
- Cap state transition history at 200 entries
- SSE/WebSocket connection limit (100 max)
- Request body size limit (1MB)
- Response body size limit enforcement in WASM HTTP
- UTF-8 safe string truncation (routine engine, shell tool)
- Fix PolicyAction::Sanitize to actually run the sanitizer
- TOCTOU fix in scheduler and context manager (hold write lock)
- Project file serving moved behind auth
- Path traversal guard on project_id
- Session file permissions set to 0600 on unix
- AtomicUsize for routine running_count (panic-safe)
- Completion detection hardened against false positives and tool injection
- Tool output no longer drives job completion (only LLM response)

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: address security review findings across all layers

- Fix path traversal sandbox bypass via lexical normalization (file.rs)
- Fix SSRF via DNS rebinding with pre-request hostname resolution (http.rs)
- Add token budget enforcement on LLM calls (reasoning.rs, state.rs)
- Fix cross-user chat history leak with ownership verification (store.rs, server.rs)
- Add sliding-window rate limiter on gateway chat endpoint (server.rs)
- Harden extension install: HTTPS-only, 50MB cap, WASM magic validation (manager.rs)
- Add destructive command blocklist that overrides shell auto-approval (shell.rs)
- Add 5MB response body size cap to HTTP tool (http.rs)

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* refactor: deduplicate shared helpers and remove dead code

Extract floor_char_boundary and llm_signals_completion into src/util.rs,
unifying diverging phrase lists from agent/worker.rs and worker/runtime.rs.
Remove dead RespondResult::usage(), duplicate PROTECTED_IDENTITY_FILES
constant, double LeakDetector scanning in WebLogLayer, and invalid
0.0.0.0 origin from WebSocket allow list.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: address PR review findings and CI test failures

- Fix record_failed_approve: .truncate(true) wiped the attempts file
  before reading, so failed pairing attempts never accumulated and
  rate limiting never triggered.
- Guard wizard WASM test: skip gracefully when channel build artifacts
  are absent (CI doesn't compile wasm32-wasip2 targets).
- Fix DNS rebinding check: use port 0 instead of hardcoded 443, since
  the port is irrelevant for hostname resolution.
- Remove hardcoded CORS port 3001: the dynamic addr.port() entries
  already cover the actual server port.
- Require WebSocket Origin header: reject connections that omit it
  entirely, since browsers always send Origin for WS upgrades and a
  missing header indicates a non-browser client bypassing the check.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: address second round of PR review findings

- store.rs: reintroduce file locking around read-modify-write in
  record_failed_approve (concurrent callers could clobber each other).
- sse.rs: replace load+check+fetch_add with atomic fetch_update in both
  subscribe_raw() and subscribe() to prevent overshooting max_connections.
- ws.rs: decrement WS tracker before early return when subscribe_raw()
  returns None (connection limit reached), fixing a counter leak.
- server.rs: parse WS Origin host exactly instead of prefix matching,
  preventing bypass via crafted origins like http://localhost.evil.com.
- workspace_integration.rs: skip tests gracefully when Postgres is
  unreachable instead of panicking (fixes 10 CI failures).

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: add Origin header to WS integration tests

The Origin header requirement added in a3b0190 broke the WS gateway
integration tests. Test clients now send Origin: http://127.0.0.1:{port}
to match the server's localhost validation.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

---------

Co-authored-by: Claude Opus 4.6 <[email protected]>
2026-02-13 05:25:20 +00:00
github-actions[bot]GitHubgithub-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
e0a43c81f9 chore: release v0.1.3 (#56)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-02-13 00:28:13 +01:00
Vlad Frolov bada79ba4a ci: Enabled builds caching during CI/CD 2026-02-13 00:17:55 +01:00
Vlad Frolov a70c89d9e3 ci: Disabled npm publishing as the name is already taken 2026-02-13 00:17:55 +01:00
github-actions[bot]GitHubgithub-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
247445f819 chore: release v0.1.2 (#55)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-02-12 23:47:16 +01:00
Vlad Frolov 14254a699f docs: Added Installation instructions for the pre-built binaries 2026-02-12 23:42:46 +01:00
Vlad Frolov 2039442885 ci: Disabled Windows ARM64 builds as auto-updater [provided by cargo-dist] does not support this platform yet and it is not a common platform for us to support 2026-02-12 23:41:27 +01:00
Vlad Frolov e796b838fa ci: Skip creating GitHub Release with release-plz [cargo-dist will do it in the release workflow] 2026-02-12 23:11:30 +01:00
github-actions[bot]GitHubgithub-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
54ce7434fb chore: release v0.1.1 (#54)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-02-12 23:09:48 +01:00
Vlad Frolov e9e0374c85 ci: Renamed the secrets in release-plz.yml to match the configuration 2026-02-12 23:05:30 +01:00
Vlad Frolov 5582a0dfbf ci: Make sure that the binaries release CD it kicking in after release-plz 2026-02-12 22:55:08 +01:00
github-actions[bot]GitHubgithub-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
1cf08a4b42 chore: release v0.1.0 (#46)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-02-12 22:14:28 +01:00
ZakiandClaude Opus 4.6 7474fd4c52 fix: address PR review feedback for libSQL backend
- P0: Switch libsql_backend to connection-per-operation pattern to fix
  shared Connection concurrency issue across tokio tasks
- P0: Wrap secrets store INSERT+SELECT in transaction to fix TOCTOU race
- P0: Document encryption-at-rest limitations and json_patch divergence
- P1: Fix get_opt_text removing .filter(|s| !s.is_empty()) that conflated
  empty strings with NULL
- P1: Replace datetime('now') with fmt_ts(&Utc::now()) for consistent
  RFC 3339 timestamps across all queries
- P2: Use explicit _rowid column in FTS5 triggers and joins for stability
  across VACUUM operations
- P2: Add tracing::warn when embedding provided but vector search disabled
  in hybrid_search
- Extract shared connect_from_config() helper to deduplicate DB connection
  logic across main.rs, cli/config.rs, and cli/mcp.rs

Co-Authored-By: Claude Opus 4.6 <[email protected]>
2026-02-12 07:02:21 -08:00
ZakiandClaude Opus 4.6 64b6f559fd feat: enable onboarding wizard for libSQL builds
Refactor the setup wizard to work with both postgres and libsql feature
flags. Previously the wizard was gated behind #[cfg(feature = "postgres")]
only, so libsql-only builds would print an error on `ironclaw onboard`.

- Add libsql fields to Settings (database_backend, libsql_path, libsql_url)
- Split wizard database/migration/secrets methods into feature-gated variants
- Add step_database_libsql() with local path and Turso remote replica prompts
- Update setup/mod.rs and main.rs feature gates to any(postgres, libsql)
- Extend check_onboard_needed() to detect libsql database presence

Co-Authored-By: Claude Opus 4.6 <[email protected]>
2026-02-12 06:45:42 -08:00
ZakiandClaude Opus 4.6 0de6f6aabb feat: add libSQL/Turso database backend with full feature parity
Introduce a Database trait abstraction (~60 async methods) enabling
compile-time backend selection between PostgreSQL and libSQL/Turso.
Convert all modules from concrete Store to Arc<dyn Database>, add
LibSqlSecretsStore and LibSqlWasmToolStore implementations, wire
libsql stores throughout CLI and main entry points, and make the
setup wizard backend-agnostic.

Key changes:
- src/db/: Database trait, PostgresDatabase adapter, LibSqlBackend
  with native SQLite-dialect SQL, and idempotent migration system
- src/secrets/store.rs: LibSqlSecretsStore (all 8 trait methods)
- src/tools/wasm/storage.rs: LibSqlWasmToolStore (all 7 trait methods)
- src/main.rs, cli/tool.rs, cli/mcp.rs: backend-conditional wiring
- src/setup/channels.rs: SecretsContext uses Arc<dyn SecretsStore>
- Feature-gate postgres-only tests and examples

Co-Authored-By: Claude Opus 4.6 <[email protected]>
2026-02-12 04:35:05 -08:00
Vlad Frolov d55b302b39 ci: Skip release-plz on forks 2026-02-12 12:36:13 +01:00
Vlad Frolov 517be42ccc ci: Upgraded release-plz CD pipeline 2026-02-12 12:34:11 +01:00
Vlad FrolovandGitHub 09198c68ab ci: Added CI/CD and release pipelines (#45) 2026-02-12 12:25:36 +01:00
Ilgın KanatandGitHub 115b7f38fe DM pairing + Telegram channel improvements (#17)
* feat: Implement DM pairing for channels

- Introduced a new pairing system to manage direct messages from unknown senders.
- Added `PairingStore` to handle pending requests and allowlist management.
- Implemented CLI commands for listing and approving pairing requests.
- Updated Telegram channel to utilize the new pairing logic, including workspace paths for storing pairing data.
- Enhanced WASM channel integration to support pairing functionality.

This feature enhances security by requiring approval for unknown senders before they can interact with the agent.

* Enhance Telegram channel support with media captioning and DM pairing features

- Added support for media captions in Telegram messages, allowing for richer content handling.
- Updated message processing to utilize either text or caption, improving message flexibility.
- Enhanced DM pairing functionality to include approval and listing capabilities for direct messages.
- Updated feature parity documentation to reflect new capabilities and improvements in Telegram integration.

* Update README and BUILDING_CHANNELS documentation for Telegram channel integration

- Enhanced README with instructions for building and running the Telegram channel, including a note on running `./scripts/build-all.sh` for full releases.
- Added detailed steps in BUILDING_CHANNELS.md for building and deploying the Telegram channel, emphasizing the need to run `./channels-src/telegram/build.sh` before building the main crate to ensure updated WASM is included.
- Updated CLI module to expose a new command for pairing with store functionality.

* Implement build script for Telegram channel WASM and enhance pairing error handling

- Added a new `build.rs` script to automate the compilation of the Telegram channel's WASM binary from source, ensuring reproducible builds and emphasizing supply chain security by preventing committed binaries.
- Updated `BUILDING_CHANNELS.md` to reflect the new build process and the importance of not committing compiled binaries.
- Enhanced error handling in the pairing approval process to include rate limiting for failed attempts, improving security and user feedback.

* Remove Telegram channel WASM binary file as part of the build process cleanup, ensuring no committed binaries are present in the repository.
2026-02-12 00:46:47 +00:00
bb228f6315 feat: Add multi-provider LLM support via rig-core adapter (#36)
Add support for OpenAI, Anthropic, Ollama, and OpenAI-compatible
endpoints alongside the existing NEAR AI backend. Users can now
bring their own API keys via environment variables (LLM_BACKEND,
OPENAI_API_KEY, ANTHROPIC_API_KEY, etc.) while NEAR AI remains
the default.

Co-authored-by: Claude Opus 4.6 <[email protected]>
2026-02-12 00:37:51 +00:00
bkutasiandGitHub 45f547c711 fix: resolve runtime panic in Linux keychain integration (#32)
* fix: resolve runtime panic in Linux keychain integration

- Convert Linux keychain functions from sync (rt.block_on) to async
- Remove nested runtime panic when called from async context
- Make keychain API consistent across platforms (macOS, Linux, fallback)
- Propagate async through config loading and CLI commands

Fixes panic on Linux during 'ironclaw onboard' at Step 2 (Security).

* fix: await async Config::from_env in test_heartbeat example
2026-02-12 00:15:41 +00:00
firat.sertgozandGitHub 23de75d75b Merge pull request #13 from nearai/okta-tools
feat: Add Okta SSO WASM tool for profile management and app catalog
2026-02-11 16:35:08 +04:00
ced83d5b4d feat: Sandbox jobs (#4)
* Orchestrating jobs and running them in sandboxes

* Fix heartbeat: dynamic max_tokens, empty content guard, notification fallback

- Query /v1/models API for context_length and set max_tokens to half
  (floor 4096) instead of hardcoded 1024; reasoning models like GLM-4.7
  need much larger budgets
- Guard against empty LLM content (reasoning models can burn all tokens
  on chain-of-thought and return content: null)
- Simplify notification routing: try configured channel first, fall back
  to broadcast_all so heartbeat alerts always reach someone
- Add ModelMetadata struct and model_metadata() to LlmProvider trait
- Refactor NearAiChatProvider::list_models into shared fetch_models()
- Add standalone test_heartbeat example for isolated debugging

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* Add job detail view with drill-down from jobs list

Click a job row to see full details across four sub-tabs:
Overview (metadata grid, description, state transitions timeline),
Actions (expandable tool call cards with input/output JSON),
Thinking (conversation messages styled by role), and
Files (embedded workspace tree browser).

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* Strip model-internal XML tags from LLM responses, fix Telegram parse_mode 400

Some models (GLM-4.7, etc.) emit <tool_call>tool_list</tool_call> in the
content field instead of using the OpenAI tool_calls array. This XML leaks
through to channels as text, and Telegram's Markdown parser chokes on the
underscores, returning 400 "can't parse entities".

Two fixes:
- Generalize clean_response() to strip <tool_call>, <function_call>,
  <tool_calls>, and pipe-delimited variants (<|tool_call|>) alongside
  the existing <thinking> tag stripping
- Add Telegram send_message helper with parse_mode fallback: try Markdown
  first, retry as plain text on "can't parse entities" 400 errors

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* Add SystemCommand submission type for thread-state-independent commands

System commands (/help, /model, /version, /tools, /ping, /debug) now
bypass thread-state checks and safety validation via a dedicated
Submission::SystemCommand variant. Previously these flowed through
process_user_input() which blocked them during Processing/AwaitingApproval
/Completed states.

- Add /model [name] for runtime model switching with provider validation
- Add active_model_name()/set_model() to LlmProvider trait with RwLock
  hot-swap in both NEAR AI providers
- Rewrite /help with aligned columns grouped by category
- Expand REPL tab-completion from 10 to 23 slash commands
- Remove REPL-local /help interception (now handled by agent)

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* Add per-tool execution timeouts, auto-create sandbox project dirs, serve built files

The sandbox e2e pipeline (agent -> container -> built website -> browsable URL)
was broken by three gaps: hardcoded 60s timeouts killed sandbox jobs that need
minutes, no auto-created project directory meant container output vanished, and
no HTTP route to browse the built files.

- Add `execution_timeout()` to the `Tool` trait (default 60s), replace all four
  hardcoded `Duration::from_secs(60)` call sites (agent_loop, worker, scheduler,
  worker/runtime) with the per-tool value
- Override to 660s in `RunInSandboxTool` (10 min polling + 60s buffer)
- Auto-create `~/.ironclaw/projects/{uuid}/` when no `project_dir` is specified,
  so every sandbox job gets a persistent bind mount
- Include `project_dir` and `browse_url` in sandbox tool output JSON
- Add `/projects/{id}` and `/projects/{id}/{path}` static file serving routes
  to the web gateway with path traversal protection and MIME type detection
- Add `mime_guess` dependency for content-type detection

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* Apply cargo fmt to wizard.rs after merge

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* Persist sandbox jobs in DB, fix web UI, unify job model

Sandbox container jobs were invisible to the web UI because they lived
only in ContainerJobManager's in-memory HashMap while the API queried
ContextManager. This persists them to the agent_jobs table and fixes
all six front-end bugs (empty job list, broken back button, empty
actions/thinking tabs, wrong files tab, stuck status, no persistence).

Key changes:
- V4 migration adds project_dir and user_id columns to agent_jobs
- Embedded migrations via refinery (no external CLI needed)
- SandboxJobRecord CRUD in Store with fire-and-forget DB writes
- Unified job_id: sandbox tool generates UUID, passes to ContainerJobManager
- Web API queries DB for sandbox jobs, merges with ContextManager direct jobs
- New endpoints: restart, project file list/read with path traversal protection
- Front-end: rebuild DOM on back navigation, sandbox-aware tabs, job cards in
  chat stream, source badges, restart button for failed/interrupted jobs
- Gateway defaults to enabled, prints Web UI URL on startup
- Stale jobs marked "interrupted" on restart for visibility and restartability

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* Secure in-chat auth: tokens never touch the LLM or chat history

Remove the token parameter from tool_auth so the LLM cannot pass raw
API keys. Add dedicated REST (POST /api/chat/auth-token) and WebSocket
(auth_token) endpoints that route tokens directly to ext_mgr.auth(),
completely bypassing the message pipeline, turns, history, and compaction.

Web UI shows an auth card (password input + OAuth button) when the agent
enters auth mode, submitted via the dedicated endpoint. CLI auth mode
interception is unchanged (already secure).

New StatusUpdate::AuthRequired/AuthCompleted variants propagate through
all channels (SSE, WebSocket, REPL, WASM).

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* feat: Add Claude Code mode for sandbox jobs

Run Claude Code CLI inside Docker containers as an alternative to the
standard worker mode. The bridge spawns `claude -p` with stream-json
output, posts events to the orchestrator, and supports follow-up
prompts via `--resume`.

Key additions:
- `claude-bridge` CLI subcommand and ClaudeBridgeRuntime
- JobMode enum (Worker vs ClaudeCode) with per-mode container config
- Orchestrator endpoints for Claude events and prompt polling
- SSE event variants for real-time Claude Code streaming to frontend
- Claude Code sub-tab in web UI with terminal-style output and input bar
- Database migration for job_mode column and claude_code_events table
- ClaudeCodeConfig with env var support (CLAUDE_CODE_ENABLED, etc.)
- Mode parameter on run_in_sandbox tool schema

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: Skip create_job tool when sandbox is enabled to prevent duplicate jobs

When sandbox mode is on, the LLM would call create_job (creating a
pending "direct" entry) then run_in_sandbox (creating a second "sandbox"
entry), producing two jobs in the list for a single user request.

Now register_job_tools() skips create_job when sandbox is enabled since
run_in_sandbox already creates tracked jobs. Also improved the
run_in_sandbox description to guide the LLM to use it directly and to
mention wait=false for async execution.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* feat: Web gateway UI quality-of-life improvements

Phase 1: Send button disabled state to prevent double-sends, copy button
on code blocks, confirm() guards on destructive actions, SSE-driven job
list auto-refresh, log filters re-applied on tab switch, jobEvents memory
leak fix (cap at 500, cleanup after 60s).

Phase 2: Toast notification system replacing chat-based system messages,
memory search highlighting with centered snippets, keyboard shortcuts
(Ctrl+1-5 tabs, Ctrl+K focus, Ctrl+N new thread, Escape close/blur),
activity tab toolbar with event type filter and auto-scroll toggle.

Phase 3: Thread sidebar with load/switch/create, thread_id passed with
messages, collapsible to hamburger. Memory inline editing with textarea,
Save/Cancel, POST to /api/memory/write.

Phase 4: Gateway status popover on hover (polls every 30s), extension
install form (name/URL/kind), markdown rendering in memory viewer for
.md files, mobile responsive layout at 768px breakpoint.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* feat: Add routines system, remove non-sandbox job mode from web UI

Routines: scheduled & reactive job system with cron and event triggers,
lightweight (single LLM call) and full-job execution modes, guardrails
(cooldown, max concurrent, dedup), and LLM-facing tools for CRUD.

Web UI: remove ContextManager-backed "direct" job mode entirely. Jobs
are now exclusively sandbox-backed (DB + container). Simplify job detail
response, drop dead types (ActionInfo, MessageInfo, MessageToolCallInfo),
fix Browse Files CSS loading (trailing-slash redirect), fix Activity tab
event rendering.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: Re-enable chat input on agent completion, auto-auth on tool_activate, recover tool calls from content XML

Three fixes:

1. Chat input stays disabled after agent finishes: the "Done" status
   SSE event now calls enableChatInput() as a safety net when the
   response event is empty or lost. Same for auth_completed and
   cancelAuth().

2. tool_activate never triggers auth: when activation fails due to
   missing authentication, it now auto-initiates the auth flow
   (same pattern as the web API handler). detect_auth_awaiting()
   also matches tool_activate results now.

3. Models like GLM-4.7 emit tool calls as XML tags in content
   (<tool_call>tool_list</tool_call>) instead of using the structured
   tool_calls array. recover_tool_calls_from_content() extracts and
   validates these before falling back to plain text.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* feat: Add routines web UI tab, update docs for sandbox-jobs branch

Add full routines management to the web gateway (list, detail, trigger,
toggle, delete) with 7 new API endpoints, response types, and frontend
(HTML, JS, CSS). Update FEATURE_PARITY.md (~23 rows), CLAUDE.md (new
subsystems, config, TODOs), and README.md (architecture diagram,
features, components, fix onboard command).

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: Bind Telegram bot to owner account during setup

Without owner binding, anyone who discovers the bot can send it messages.
The setup wizard now prompts the user to message their bot, captures their
Telegram user ID via getUpdates, and persists it as telegram_owner_id in
settings. On startup, the owner_id is injected into the WASM channel config
so the existing owner restriction logic drops messages from non-owners.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* feat: Move settings from disk to PostgreSQL database

Settings previously lived in three JSON files on disk (settings.json,
mcp-servers.json, session.json). This made them inaccessible from the
web UI and caused redundant disk reads (Settings::load() called 8+
times during startup).

Now all settings live in a `settings` table (user_id + key -> JSONB)
with only 4 bootstrap fields remaining on disk (database_url, pool
size, secrets key source, onboard_completed) since they're needed
before the DB connection exists.

- Add V8 migration for settings table
- Add BootstrapConfig (thin disk file) and Settings DB round-trip
- Add Store CRUD methods for settings (get/set/delete/list/bulk)
- Refactor Config to load from DB (env > DB > default cascade)
- Add SessionManager DB persistence for session tokens
- Add DB-backed MCP server config load/save functions
- Add 6 settings web API endpoints (list/get/set/delete/export/import)
- Add one-time disk-to-DB migration on first boot
- Make CLI config commands async with DB access (disk fallback)

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* feat: Seed workspace on boot, fix gateway duplicate logs and URL auto-auth

- Add Workspace::seed_if_empty() to create core identity files (README,
  MEMORY, IDENTITY, SOUL, AGENTS, USER, HEARTBEAT) when missing, called
  on every boot without overwriting existing user edits
- Remove duplicate gateway log lines from web/mod.rs (main.rs has the
  useful clickable ?token= URL)
- Auto-authenticate from ?token= URL parameter in the web UI and strip
  the token from the address bar after successful auth

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: Harden sandbox security (path traversal + orchestrator auth)

Two vulnerabilities fixed:

1. project_dir path traversal: The create_job tool let the LLM specify
   arbitrary host paths for Docker bind mounts. Removed project_dir from
   the tool schema entirely, and added canonicalization + prefix validation
   at both resolve_project_dir() and the job_manager bind mount point.

2. Orchestrator API auth bypass: worker_auth_middleware was defined but
   never applied. Each handler manually called validate_token(), so any
   new endpoint that forgot would be publicly accessible. Applied the
   middleware as route_layer on all /worker/ routes, removed manual auth
   from all 7 handlers. Bind to 127.0.0.1 on macOS/Windows (Linux keeps
   0.0.0.0 since containers reach host via docker bridge, not loopback).

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* feat: Rework gateway chat with pinned assistant, pagination, and NEAR AI response chaining

Implements the 4-phase plan for overhauling the web gateway chat:

- Phase 1: Pinned "Assistant" thread at top of sidebar, regular threads below
- Phase 2: Cursor-based history pagination with infinite scroll
- Phase 3: NEAR AI previous_response_id chaining (delta-only messages),
  with fallback to full history on chain errors, and DB persistence of
  chain state across restarts
- Phase 4: SSE thread isolation (events filtered by thread_id)

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: Add per-request HTTP timeout to WASM host, redact credentials in errors

Three fixes for WASM channel reliability:

1. Per-request timeout: Add optional timeout-ms parameter to http-request
   in both channel and tool WIT interfaces. Telegram long-poll now specifies
   35s (outliving the 30s server-side hold), while regular API calls use
   the 30s default. Fixes the triple-30s timeout race that caused polling
   failures.

2. Credential redaction: reqwest::Error includes the full URL (with injected
   bot tokens) in its Display output. Scrub credential values from error
   messages before logging or returning to WASM.

3. Webhook route registration: Remove tunnel URL gate so webhook routes are
   always available when webhook channels exist, not only when TUNNEL_URL
   is configured.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* chore: Fix clippy warnings in WASM tools and channels

- slack channel: allow dead_code on signing_secret_name (forward compat field)
- gmail tool: use div_ceil() instead of manual (n+2)/3
- google-calendar tool: extract CreateEventParams/UpdateEventParams structs
  to fix too-many-arguments warnings

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* Fix approval flow

* fix: Rebuild bundled telegram.wasm with updated WIT interface

The bundled WASM binary must match the host's WIT definition.
Previous binary was compiled against the old 4-arg http-request;
this rebuild includes the new timeout-ms parameter.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* refactor: Load WASM channels from disk instead of bundling in binary

Remove include_bytes! embedding of telegram.wasm. Channels are now
loaded from their build output directories (channels-src/<name>/target/)
during onboarding, then from ~/.ironclaw/channels/ at runtime.

- bundled.rs: locate_channel_artifacts() finds WASM + capabilities from
  build output; IRONCLAW_CHANNELS_SRC env var overrides the default path
- available_channel_names(): only lists channels with build artifacts
- bundled_channel_names(): lists all known channels (manifest)
- Setup wizard uses available_channel_names() to offer installable channels
- Add *.wasm to .gitignore, remove tracked telegram.wasm

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: Persist gateway auth token, fix thread hydration race, polish auth screen

Three web gateway UX fixes:

1. Token persistence: Store auth token in sessionStorage so refreshing
   the page doesn't force re-authentication. Hide the auth screen
   immediately when a saved token exists to prevent flash.

2. Thread hydration: Remove the !msgs.is_empty() bail-out in
   maybe_hydrate_thread so that even brand-new (empty) assistant threads
   get hydrated with their correct DB UUID. Previously resolve_thread
   would mint a fresh UUID, causing messages to land in the wrong
   conversation and duplicate threads to appear.

3. Auth screen: Redesign as a centered card with brand, tagline, labeled
   input, and hint text.

Also adds 34 new tests covering session/thread lifecycle, thread
resolution isolation (user, channel, external ID), hydration edge cases,
serialization round-trips, approval flows, and stale mapping recovery.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: Use bindgen! for WASM tool wrapper, add dev tool loading

Three changes:

1. Rewrite src/tools/wasm/wrapper.rs to use wasmtime::component::bindgen!
   instead of manual linker.root().func_wrap(). This fixes the
   "component imports instance 'near:agent/host', but a matching
   implementation was not found in the linker" error. All 6 host functions
   (log, now-millis, workspace-read, http-request, secret-exists,
   tool-invoke) are now properly registered under the near:agent/host
   namespace. Also adds WASI support, credential injection, and leak
   detection for HTTP requests made by WASM tools.

2. Add dev tool loading to src/tools/wasm/loader.rs. During startup, the
   loader now also scans tools-src/*/target/wasm32-wasip2/release/ for
   build artifacts that are newer than installed copies. This means during
   development you just rebuild the WASM and restart the host; no manual
   copy step needed. Set IRONCLAW_TOOLS_SRC to override the source dir.

3. Wire up load_dev_tools() in main.rs alongside the existing
   load_from_dir() call.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* feat: Wire main startup and CLI to use DB-backed settings

main.rs now reloads Config from the database after connecting,
attaches the store to the session manager for dual-write tokens,
and loads MCP servers from DB instead of disk. ExtensionManager
and MCP CLI commands use DB when available with disk fallback.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

---------

Co-authored-by: Claude Opus 4.6 <[email protected]>
2026-02-11 08:31:25 +00:00
Illia PolosukhinandClaude Opus 4.6 810ba58fd2 feat: Add Okta SSO WASM tool for profile management and app catalog
Sandboxed WASM tool that integrates with Okta's Management API and
MyAccount API. Supports user profile CRUD, listing all SSO app
chiclets, searching apps by name, retrieving SSO launch links, and
fetching org info. Uses OAuth2 with PKCE against the Org Authorization
Server, with the domain stored in workspace at okta/domain.

Co-Authored-By: Claude Opus 4.6 <[email protected]>
2026-02-09 23:31:29 -08:00
Elliot BraemandGitHub 202665a55c Fixes build, adds missing sse event and correct command (#11)
* add missing type

* prune

* readme

* minor

* update to .ironclaw
2026-02-10 00:08:59 +00:00
157 changed files with 32574 additions and 1929 deletions
+8
View File
@@ -0,0 +1,8 @@
target/
.git/
.env
.env.*
*.md
!CLAUDE.md
node_modules/
tools-src/
+3 -3
View File
@@ -1,15 +1,15 @@
# Database Configuration
DATABASE_URL=postgres://ironclaw:password@localhost:5432/ironclaw
DATABASE_URL=postgres://localhost/ironclaw
DATABASE_POOL_SIZE=10
# LLM Provider (NEAR AI)
# NEAR AI provides a unified interface to all models with user authentication
# Session token is stored in ~/.near-agent/session.json and managed automatically.
# Session token is stored in ~/.ironclaw/session.json and managed automatically.
# On first run, the agent will open a browser for OAuth authentication.
NEARAI_MODEL=claude-3-5-sonnet-20241022
NEARAI_BASE_URL=https://cloud-api.near.ai
NEARAI_AUTH_URL=https://private.near.ai
# NEARAI_SESSION_PATH=~/.near-agent/session.json # optional, default shown
# NEARAI_SESSION_PATH=~/.ironclaw/session.json # optional, default shown
# Channel Configuration
# CLI is always enabled
+22
View File
@@ -0,0 +1,22 @@
name: Code Style
on:
pull_request:
jobs:
codestyle:
name: Code Style (fmt + clippy)
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v6
- name: Install Rust
uses: dtolnay/rust-toolchain@stable
with:
profile: minimal
components: rustfmt, clippy
- uses: Swatinem/rust-cache@v2
- name: Check formatting
run: |
cargo fmt --all -- --check
- name: Check lints (cargo clippy)
run: cargo clippy -- -D warnings
+67
View File
@@ -0,0 +1,67 @@
name: Release-plz
on:
push:
branches:
- main
jobs:
# Release unpublished packages.
release-plz-release:
if: ${{ github.repository_owner == 'nearai' }}
name: Release-plz release
runs-on: ubuntu-latest
permissions:
contents: write
steps:
- &checkout
name: Checkout repository
uses: actions/checkout@v6
with:
fetch-depth: 0
persist-credentials: false
- &install-rust
name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
- uses: Swatinem/rust-cache@v2
# Generating a GitHub token, so that PRs and tags created by
# the release-plz-action can trigger actions workflows.
- name: Generate GitHub token
uses: actions/create-github-app-token@v2
id: generate-token
with:
# GitHub App ID secret name
app-id: ${{ secrets.GH_RELEASES_MANAGER_APP_ID }}
# GitHub App private key secret name
private-key: ${{ secrets.GH_RELEASES_MANAGER_APP_PRIVATE_KEY }}
- name: Run release-plz
uses: release-plz/[email protected]
with:
command: release
env:
GITHUB_TOKEN: ${{ steps.generate-token.outputs.token }}
CARGO_REGISTRY_TOKEN: ${{ secrets.CARGO_REGISTRY_TOKEN }}
# Create a PR with the new versions and changelog, preparing the next release.
release-plz-pr:
if: ${{ github.repository_owner == 'nearai' }}
name: Release-plz PR
runs-on: ubuntu-latest
permissions:
contents: write
pull-requests: write
concurrency:
group: release-plz-${{ github.ref }}
cancel-in-progress: false
steps:
- *checkout
- *install-rust
- uses: Swatinem/rust-cache@v2
- name: Run release-plz
uses: release-plz/[email protected]
with:
command: release-pr
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
CARGO_REGISTRY_TOKEN: ${{ secrets.CARGO_REGISTRY_TOKEN }}
+300
View File
@@ -0,0 +1,300 @@
# This file was autogenerated by dist: https://axodotdev.github.io/cargo-dist
#
# Copyright 2022-2024, axodotdev
# SPDX-License-Identifier: MIT or Apache-2.0
#
# CI that:
#
# * checks for a Git Tag that looks like a release
# * builds artifacts with dist (archives, installers, hashes)
# * uploads those artifacts to temporary workflow zip
# * on success, uploads the artifacts to a GitHub Release
#
# Note that the GitHub Release will be created with a generated
# title/body based on your changelogs.
name: Release
permissions:
"contents": "write"
# This task will run whenever you push a git tag that looks like a version
# like "1.0.0", "v0.1.0-prerelease.1", "my-app/0.1.0", "releases/v1.0.0", etc.
# Various formats will be parsed into a VERSION and an optional PACKAGE_NAME, where
# PACKAGE_NAME must be the name of a Cargo package in your workspace, and VERSION
# must be a Cargo-style SemVer Version (must have at least major.minor.patch).
#
# If PACKAGE_NAME is specified, then the announcement will be for that
# package (erroring out if it doesn't have the given version or isn't dist-able).
#
# If PACKAGE_NAME isn't specified, then the announcement will be for all
# (dist-able) packages in the workspace with that version (this mode is
# intended for workspaces with only one dist-able package, or with all dist-able
# packages versioned/released in lockstep).
#
# If you push multiple tags at once, separate instances of this workflow will
# spin up, creating an independent announcement for each one. However, GitHub
# will hard limit this to 3 tags per commit, as it will assume more tags is a
# mistake.
#
# If there's a prerelease-style suffix to the version, then the release(s)
# will be marked as a prerelease.
on:
pull_request:
push:
tags:
- '**[0-9]+.[0-9]+.[0-9]+*'
jobs:
# Run 'dist plan' (or host) to determine what tasks we need to do
plan:
runs-on: "ubuntu-22.04"
outputs:
val: ${{ steps.plan.outputs.manifest }}
tag: ${{ !github.event.pull_request && github.ref_name || '' }}
tag-flag: ${{ !github.event.pull_request && format('--tag={0}', github.ref_name) || '' }}
publishing: ${{ !github.event.pull_request }}
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
steps:
- uses: actions/checkout@v4
with:
persist-credentials: false
submodules: recursive
- name: Install dist
# we specify bash to get pipefail; it guards against the `curl` command
# failing. otherwise `sh` won't catch that `curl` returned non-0
shell: bash
run: "curl --proto '=https' --tlsv1.2 -LsSf https://github.com/axodotdev/cargo-dist/releases/download/v0.30.3/cargo-dist-installer.sh | sh"
- name: Cache dist
uses: actions/upload-artifact@v4
with:
name: cargo-dist-cache
path: ~/.cargo/bin/dist
# sure would be cool if github gave us proper conditionals...
# so here's a doubly-nested ternary-via-truthiness to try to provide the best possible
# functionality based on whether this is a pull_request, and whether it's from a fork.
# (PRs run on the *source* but secrets are usually on the *target* -- that's *good*
# but also really annoying to build CI around when it needs secrets to work right.)
- id: plan
run: |
dist ${{ (!github.event.pull_request && format('host --steps=create --tag={0}', github.ref_name)) || 'plan' }} --output-format=json > plan-dist-manifest.json
echo "dist ran successfully"
cat plan-dist-manifest.json
echo "manifest=$(jq -c "." plan-dist-manifest.json)" >> "$GITHUB_OUTPUT"
- name: "Upload dist-manifest.json"
uses: actions/upload-artifact@v4
with:
name: artifacts-plan-dist-manifest
path: plan-dist-manifest.json
# Build and packages all the platform-specific things
build-local-artifacts:
name: build-local-artifacts (${{ join(matrix.targets, ', ') }})
# Let the initial task tell us to not run (currently very blunt)
needs:
- plan
if: ${{ fromJson(needs.plan.outputs.val).ci.github.artifacts_matrix.include != null && (needs.plan.outputs.publishing == 'true' || fromJson(needs.plan.outputs.val).ci.github.pr_run_mode == 'upload') }}
strategy:
fail-fast: false
# Target platforms/runners are computed by dist in create-release.
# Each member of the matrix has the following arguments:
#
# - runner: the github runner
# - dist-args: cli flags to pass to dist
# - install-dist: expression to run to install dist on the runner
#
# Typically there will be:
# - 1 "global" task that builds universal installers
# - N "local" tasks that build each platform's binaries and platform-specific installers
matrix: ${{ fromJson(needs.plan.outputs.val).ci.github.artifacts_matrix }}
runs-on: ${{ matrix.runner }}
container: ${{ matrix.container && matrix.container.image || null }}
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
BUILD_MANIFEST_NAME: target/distrib/${{ join(matrix.targets, '-') }}-dist-manifest.json
steps:
- name: enable windows longpaths
run: |
git config --global core.longpaths true
- uses: actions/checkout@v4
with:
persist-credentials: false
submodules: recursive
- name: Install Rust non-interactively if not already installed
if: ${{ matrix.container }}
run: |
if ! command -v cargo > /dev/null 2>&1; then
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
echo "$HOME/.cargo/bin" >> $GITHUB_PATH
fi
- uses: swatinem/rust-cache@v2
with:
key: ${{ join(matrix.targets, '-') }}
cache-provider: ${{ matrix.cache_provider }}
- name: Install dist
run: ${{ matrix.install_dist.run }}
# Get the dist-manifest
- name: Fetch local artifacts
uses: actions/download-artifact@v4
with:
pattern: artifacts-*
path: target/distrib/
merge-multiple: true
- name: Install dependencies
run: |
${{ matrix.packages_install }}
- name: Build artifacts
run: |
# Actually do builds and make zips and whatnot
dist build ${{ needs.plan.outputs.tag-flag }} --print=linkage --output-format=json ${{ matrix.dist_args }} > dist-manifest.json
echo "dist ran successfully"
- id: cargo-dist
name: Post-build
# We force bash here just because github makes it really hard to get values up
# to "real" actions without writing to env-vars, and writing to env-vars has
# inconsistent syntax between shell and powershell.
shell: bash
run: |
# Parse out what we just built and upload it to scratch storage
echo "paths<<EOF" >> "$GITHUB_OUTPUT"
dist print-upload-files-from-manifest --manifest dist-manifest.json >> "$GITHUB_OUTPUT"
echo "EOF" >> "$GITHUB_OUTPUT"
cp dist-manifest.json "$BUILD_MANIFEST_NAME"
- name: "Upload artifacts"
uses: actions/upload-artifact@v4
with:
name: artifacts-build-local-${{ join(matrix.targets, '_') }}
path: |
${{ steps.cargo-dist.outputs.paths }}
${{ env.BUILD_MANIFEST_NAME }}
# Build and package all the platform-agnostic(ish) things
build-global-artifacts:
needs:
- plan
- build-local-artifacts
runs-on: "ubuntu-22.04"
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
BUILD_MANIFEST_NAME: target/distrib/global-dist-manifest.json
steps:
- uses: actions/checkout@v4
with:
persist-credentials: false
submodules: recursive
- name: Install cached dist
uses: actions/download-artifact@v4
with:
name: cargo-dist-cache
path: ~/.cargo/bin/
- run: chmod +x ~/.cargo/bin/dist
# Get all the local artifacts for the global tasks to use (for e.g. checksums)
- name: Fetch local artifacts
uses: actions/download-artifact@v4
with:
pattern: artifacts-*
path: target/distrib/
merge-multiple: true
- id: cargo-dist
shell: bash
run: |
dist build ${{ needs.plan.outputs.tag-flag }} --output-format=json "--artifacts=global" > dist-manifest.json
echo "dist ran successfully"
# Parse out what we just built and upload it to scratch storage
echo "paths<<EOF" >> "$GITHUB_OUTPUT"
jq --raw-output ".upload_files[]" dist-manifest.json >> "$GITHUB_OUTPUT"
echo "EOF" >> "$GITHUB_OUTPUT"
cp dist-manifest.json "$BUILD_MANIFEST_NAME"
- name: "Upload artifacts"
uses: actions/upload-artifact@v4
with:
name: artifacts-build-global
path: |
${{ steps.cargo-dist.outputs.paths }}
${{ env.BUILD_MANIFEST_NAME }}
# Determines if we should publish/announce
host:
needs:
- plan
- build-local-artifacts
- build-global-artifacts
# Only run if we're "publishing", and only if plan, local and global didn't fail (skipped is fine)
if: ${{ always() && needs.plan.result == 'success' && needs.plan.outputs.publishing == 'true' && (needs.build-global-artifacts.result == 'skipped' || needs.build-global-artifacts.result == 'success') && (needs.build-local-artifacts.result == 'skipped' || needs.build-local-artifacts.result == 'success') }}
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
runs-on: "ubuntu-22.04"
outputs:
val: ${{ steps.host.outputs.manifest }}
steps:
- uses: actions/checkout@v4
with:
persist-credentials: false
submodules: recursive
- name: Install cached dist
uses: actions/download-artifact@v4
with:
name: cargo-dist-cache
path: ~/.cargo/bin/
- run: chmod +x ~/.cargo/bin/dist
# Fetch artifacts from scratch-storage
- name: Fetch artifacts
uses: actions/download-artifact@v4
with:
pattern: artifacts-*
path: target/distrib/
merge-multiple: true
- id: host
shell: bash
run: |
dist host ${{ needs.plan.outputs.tag-flag }} --steps=upload --steps=release --output-format=json > dist-manifest.json
echo "artifacts uploaded and released successfully"
cat dist-manifest.json
echo "manifest=$(jq -c "." dist-manifest.json)" >> "$GITHUB_OUTPUT"
- name: "Upload dist-manifest.json"
uses: actions/upload-artifact@v4
with:
# Overwrite the previous copy
name: artifacts-dist-manifest
path: dist-manifest.json
# Create a GitHub Release while uploading all files to it
- name: "Download GitHub Artifacts"
uses: actions/download-artifact@v4
with:
pattern: artifacts-*
path: artifacts
merge-multiple: true
- name: Cleanup
run: |
# Remove the granular manifests
rm -f artifacts/*-dist-manifest.json
- name: Create GitHub Release
env:
PRERELEASE_FLAG: "${{ fromJson(steps.host.outputs.manifest).announcement_is_prerelease && '--prerelease' || '' }}"
ANNOUNCEMENT_TITLE: "${{ fromJson(steps.host.outputs.manifest).announcement_title }}"
ANNOUNCEMENT_BODY: "${{ fromJson(steps.host.outputs.manifest).announcement_github_body }}"
RELEASE_COMMIT: "${{ github.sha }}"
run: |
# Write and read notes from a file to avoid quoting breaking things
echo "$ANNOUNCEMENT_BODY" > $RUNNER_TEMP/notes.txt
gh release create "${{ needs.plan.outputs.tag }}" --target "$RELEASE_COMMIT" $PRERELEASE_FLAG --title "$ANNOUNCEMENT_TITLE" --notes-file "$RUNNER_TEMP/notes.txt" artifacts/*
announce:
needs:
- plan
- host
# use "always() && ..." to allow us to wait for all publish jobs while
# still allowing individual publish jobs to skip themselves (for prereleases).
# "host" however must run to completion, no skipping allowed!
if: ${{ always() && needs.host.result == 'success' }}
runs-on: "ubuntu-22.04"
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
steps:
- uses: actions/checkout@v4
with:
persist-credentials: false
submodules: recursive
+21
View File
@@ -0,0 +1,21 @@
name: Run Tests
on:
pull_request:
push:
branches:
- main
jobs:
tests:
name: Run Tests
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v6
- name: Install Rust
uses: dtolnay/rust-toolchain@stable
with:
profile: minimal
- uses: Swatinem/rust-cache@v2
- name: Run Tests
run: cargo test --all-features -- --nocapture
+3
View File
@@ -4,3 +4,6 @@
target/
# WASM build artifacts (loaded from disk, not bundled)
*.wasm
+117
View File
@@ -0,0 +1,117 @@
# Changelog
All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [Unreleased]
## [0.1.3](https://github.com/nearai/ironclaw/compare/v0.1.2...v0.1.3) - 2026-02-12
### Other
- Enabled builds caching during CI/CD
- Disabled npm publishing as the name is already taken
## [0.1.2](https://github.com/nearai/ironclaw/compare/v0.1.1...v0.1.2) - 2026-02-12
### Other
- Added Installation instructions for the pre-built binaries
- Disabled Windows ARM64 builds as auto-updater [provided by cargo-dist] does not support this platform yet and it is not a common platform for us to support
## [0.1.1](https://github.com/nearai/ironclaw/compare/v0.1.0...v0.1.1) - 2026-02-12
### Other
- Renamed the secrets in release-plz.yml to match the configuration
- Make sure that the binaries release CD it kicking in after release-plz
## [0.1.0](https://github.com/nearai/ironclaw/releases/tag/v0.1.0) - 2026-02-12
### Added
- Add multi-provider LLM support via rig-core adapter ([#36](https://github.com/nearai/ironclaw/pull/36))
- Sandbox jobs ([#4](https://github.com/nearai/ironclaw/pull/4))
- Add Google Suite & Telegram WASM tools ([#9](https://github.com/nearai/ironclaw/pull/9))
- Improve CLI ([#5](https://github.com/nearai/ironclaw/pull/5))
### Fixed
- resolve runtime panic in Linux keychain integration ([#32](https://github.com/nearai/ironclaw/pull/32))
### Other
- Skip release-plz on forks
- Upgraded release-plz CD pipeline
- Added CI/CD and release pipelines ([#45](https://github.com/nearai/ironclaw/pull/45))
- DM pairing + Telegram channel improvements ([#17](https://github.com/nearai/ironclaw/pull/17))
- Fixes build, adds missing sse event and correct command ([#11](https://github.com/nearai/ironclaw/pull/11))
- Codex/feature parity pr hook ([#6](https://github.com/nearai/ironclaw/pull/6))
- Add WebSocket gateway and control plane ([#8](https://github.com/nearai/ironclaw/pull/8))
- select bundled Telegram channel and auto-install ([#3](https://github.com/nearai/ironclaw/pull/3))
- Adding skills for reusable work
- Fix MCP tool calls, approval loop, shutdown, and improve web UI
- Add auth mode, fix MCP token handling, and parallelize startup loading
- Merge remote-tracking branch 'origin/main' into ui
- Adding web UI
- Rename `setup` CLI command to `onboard` for compatibility
- Add in-chat extension discovery, auth, and activation system
- Add Telegram typing indicator via WIT on-status callback
- Add proactivity features: memory CLI, session pruning, self-repair notifications, slash commands, status diagnostics, context warnings
- Add hosted MCP server support with OAuth 2.1 and token refresh
- Add interactive setup wizard and persistent settings
- Rebrand to IronClaw with security-first mission
- Fix build_software tool stuck in planning mode loop
- Enable sandbox by default
- Fix Telegram Markdown formatting and clarify tool/memory distinctions
- Simplify Telegram channel config with host-injected tunnel/webhook settings
- Apply Telegram channel learnings to WhatsApp implementation
- Merge remote-tracking branch 'origin/main'
- Docker file for sandbox
- Replace hardcoded intent patterns with job tools
- Fix router test to match intentional job creation patterns
- Add Docker execution sandbox for secure shell command isolation
- Move setup wizard credentials to database storage
- Add interactive setup wizard for first-run configuration
- Add Telegram Bot API channel as WASM module
- Add OpenClaw feature parity tracking matrix
- Add Chat Completions API support and expand REPL debugging
- Implementing channels to be handled in wasm
- Support non interactive mode and model selection
- Implement tool approval, fix tool definition refresh, and wire embeddings
- Tool use
- Wiring more
- Add heartbeat integration, planning phase, and auto-repair
- Login flow
- Extend support for session management
- Adding builder capability
- Load tools at launch
- Fix multiline message rendering in TUI
- Parse NEAR AI alternative response format with output field
- Handle NEAR AI plain text responses
- Disable mouse capture to allow text selection in TUI
- Add verbose logging to debug empty NEAR AI responses
- Improve NEAR AI response parsing for varying response formats
- Show status/thinking messages in chat window, debug empty responses
- Add timeout and logging to NEAR AI provider
- Add status updates to show agent thinking/processing state
- Add CLI subcommands for WASM tool management
- Fix TUI shutdown: send /shutdown message and handle in agent loop
- Remove SimpleCliChannel, add Ctrl+D twice quit, redirect logs to TUI
- Fix TuiChannel integration and enable in main.rs
- Integrate Codex patterns: task scheduler, TUI, sessions, compaction
- Adding LICENSE
- Add README with IronClaw branding
- Add WASM sandbox secure API extension
- Wire database Store into agent loop
- Implementing WASM runtime
- Add workspace integration tests
- Compact memory_tree output format
- Replace memory_list with memory_tree tool
- Simplify workspace to path-based storage, remove legacy code
- Add NEAR AI chat-api as default LLM provider
- Add CLAUDE.md project documentation
- Add workspace and memory system (OpenClaw-inspired)
- Initial implementation of the agent framework
+173 -16
View File
@@ -11,8 +11,13 @@
- **Always available** - Multi-channel access with proactive background execution
### Features
- **Multi-channel input**: TUI (Ratatui), HTTP webhooks, Telegram, WhatsApp, Slack (WASM channels)
- **Multi-channel input**: TUI (Ratatui), HTTP webhooks, WASM channels (Telegram, Slack), web gateway
- **Parallel job execution** with state machine and self-repair for stuck jobs
- **Sandbox execution**: Docker container isolation with orchestrator/worker pattern
- **Claude Code mode**: Delegate jobs to Claude CLI inside containers
- **Routines**: Scheduled (cron) and reactive (event, webhook) task execution
- **Web gateway**: Browser UI with SSE/WebSocket real-time streaming
- **Extension management**: Install, auth, activate MCP/WASM extensions
- **Extensible tools**: Built-in tools, WASM sandbox, MCP client, dynamic builder
- **Persistent memory**: Workspace with hybrid search (FTS + vector via RRF)
- **Prompt injection defense**: Sanitizer, validator, policy rules, leak detection
@@ -59,7 +64,9 @@ src/
│ ├── context_monitor.rs # Memory pressure detection
│ ├── undo.rs # Turn-based undo/redo with checkpoints
│ ├── submission.rs # Submission parsing (undo, redo, compact, clear, etc.)
── task.rs # Sub-task execution framework
── task.rs # Sub-task execution framework
│ ├── routine.rs # Routine types (Trigger, Action, Guardrails)
│ └── routine_engine.rs # Routine execution (cron ticker, event matcher)
├── channels/ # Multi-channel input
│ ├── channel.rs # Channel trait, IncomingMessage, OutgoingResponse
@@ -72,8 +79,33 @@ src/
│ │ ├── overlay.rs # Approval overlays
│ │ └── composer.rs # Message composition
│ ├── http.rs # HTTP webhook (axum) with secret validation
│ ├── slack.rs # Stub
── telegram.rs # Stub
│ ├── repl.rs # Simple REPL (for testing)
── web/ # Web gateway (browser UI)
│ │ ├── mod.rs # Gateway builder, startup
│ │ ├── server.rs # Axum router, 40+ API endpoints
│ │ ├── sse.rs # SSE broadcast manager
│ │ ├── ws.rs # WebSocket gateway + connection tracking
│ │ ├── types.rs # Request/response types, SseEvent enum
│ │ ├── auth.rs # Bearer token auth middleware
│ │ ├── log_layer.rs # Tracing layer for log streaming
│ │ └── static/ # HTML, CSS, JS (single-page app)
│ └── wasm/ # WASM channel runtime
│ ├── mod.rs
│ ├── bundled.rs # Bundled channel discovery
│ └── wrapper.rs # Channel trait wrapper for WASM modules
├── orchestrator/ # Internal HTTP API for sandbox containers
│ ├── mod.rs
│ ├── api.rs # Axum endpoints (LLM proxy, events, prompts)
│ ├── auth.rs # Per-job bearer token store
│ └── job_manager.rs # Container lifecycle (create, stop, cleanup)
├── worker/ # Runs inside Docker containers
│ ├── mod.rs
│ ├── runtime.rs # Worker execution loop (tool calls, LLM)
│ ├── claude_bridge.rs # Claude Code bridge (spawns claude CLI)
│ ├── api.rs # HTTP client to orchestrator
│ └── proxy_llm.rs # LlmProvider that proxies through orchestrator
├── safety/ # Prompt injection defense
│ ├── sanitizer.rs # Pattern detection, content escaping
@@ -96,6 +128,9 @@ src/
│ │ ├── file.rs # ReadFile, WriteFile, ListDir, ApplyPatch
│ │ ├── shell.rs # Shell command execution
│ │ ├── memory.rs # Memory tools (search, write, read, tree)
│ │ ├── job.rs # CreateJob, ListJobs, JobStatus, CancelJob
│ │ ├── routine.rs # routine_create/list/update/delete/history
│ │ ├── extension_tools.rs # Extension install/auth/activate/remove
│ │ └── marketplace.rs, ecommerce.rs, taskrabbit.rs, restaurant.rs (stubs)
│ ├── builder/ # Dynamic tool building
│ │ ├── core.rs # BuildRequirement, SoftwareType, Language
@@ -116,6 +151,12 @@ src/
│ ├── rate_limiter.rs # Per-tool rate limiting
│ └── storage.rs # Linear memory persistence
├── db/ # Database abstraction layer
│ ├── mod.rs # Database trait (~60 async methods)
│ ├── postgres.rs # PostgreSQL backend (delegates to Store + Repository)
│ ├── libsql_backend.rs # libSQL/Turso backend (embedded SQLite)
│ └── libsql_migrations.rs # SQLite-dialect schema (idempotent)
├── workspace/ # Persistent memory system (OpenClaw-inspired)
│ ├── mod.rs # Workspace struct, memory operations
│ ├── document.rs # MemoryDocument, MemoryChunk, WorkspaceEntry
@@ -166,6 +207,7 @@ When designing new features or systems, always prefer generic/extensible archite
- Use `RwLock` for concurrent read/write access
### Traits for Extensibility
- `Database` - Add new database backends (must implement all ~60 methods)
- `Channel` - Add new input sources
- `Tool` - Add new capabilities
- `LlmProvider` - Add new LLM backends
@@ -213,7 +255,12 @@ Pending -> InProgress -> Completed -> Submitted -> Accepted
Environment variables (see `.env.example`):
```bash
# Database backend (default: postgres)
DATABASE_BACKEND=postgres # or "libsql" / "turso"
DATABASE_URL=postgres://user:pass@localhost/ironclaw
LIBSQL_PATH=~/.ironclaw/ironclaw.db # libSQL local path (default)
# LIBSQL_URL=libsql://xxx.turso.io # Turso cloud (optional)
# LIBSQL_AUTH_TOKEN=xxx # Required with LIBSQL_URL
# NEAR AI (required)
NEARAI_SESSION_TOKEN=sess_...
@@ -236,6 +283,30 @@ HEARTBEAT_ENABLED=true
HEARTBEAT_INTERVAL_SECS=1800 # 30 minutes
HEARTBEAT_NOTIFY_CHANNEL=tui
HEARTBEAT_NOTIFY_USER=default
# Web gateway
GATEWAY_ENABLED=true
GATEWAY_HOST=127.0.0.1
GATEWAY_PORT=3001
GATEWAY_AUTH_TOKEN=changeme # Required for API access
GATEWAY_USER_ID=default
# Docker sandbox
SANDBOX_ENABLED=true
SANDBOX_IMAGE=ironclaw-worker:latest
SANDBOX_MEMORY_LIMIT_MB=512
SANDBOX_TIMEOUT_SECS=1800
# Claude Code mode (runs inside sandbox containers)
CLAUDE_CODE_ENABLED=false
CLAUDE_CODE_MODEL=claude-sonnet-4-20250514
CLAUDE_CODE_MAX_TURNS=50
CLAUDE_CODE_CONFIG_DIR=/home/worker/.claude
# Routines (scheduled/reactive execution)
ROUTINES_ENABLED=true
ROUTINES_CRON_INTERVAL=60 # Tick interval in seconds
ROUTINES_MAX_CONCURRENT=3
```
### NEAR AI Provider
@@ -249,7 +320,51 @@ Session tokens have the format `sess_xxx` (37 characters). They are authenticate
## Database
Single migration in `migrations/V1__initial.sql`. Tables:
IronClaw supports two database backends, selected at compile time via Cargo feature flags and at runtime via the `DATABASE_BACKEND` environment variable.
**IMPORTANT: All new features that touch persistence MUST support both backends.** Implement the operation as a method on the `Database` trait in `src/db/mod.rs`, then add the implementation in both `src/db/postgres.rs` (delegate to Store/Repository) and `src/db/libsql_backend.rs` (native SQL).
### Backends
| Backend | Feature Flag | Default | Use Case |
|---------|-------------|---------|----------|
| PostgreSQL | `postgres` (default) | Yes | Production, existing deployments |
| libSQL/Turso | `libsql` | No | Zero-dependency local mode, edge, Turso cloud |
```bash
# Build with PostgreSQL only (default)
cargo build
# Build with libSQL only
cargo build --no-default-features --features libsql
# Build with both backends available
cargo build --features "postgres,libsql"
```
### Database Trait
The `Database` trait (`src/db/mod.rs`) defines ~60 async methods covering all persistence:
- Conversations, messages, metadata
- Jobs, actions, LLM calls, estimation snapshots
- Sandbox jobs, job events
- Routines, routine runs
- Tool failures, settings
- Workspace: documents, chunks, hybrid search
Both backends implement this trait. PostgreSQL delegates to the existing `Store` + `Repository`. libSQL implements native SQLite-dialect SQL.
### Schema
**PostgreSQL:** `migrations/V1__initial.sql` (351 lines). Uses pgvector for embeddings, tsvector for FTS, PL/pgSQL functions. Managed by `refinery`.
**libSQL:** `src/db/libsql_migrations.rs` (consolidated schema, ~480 lines). Translates PG types:
- `UUID` -> `TEXT`, `TIMESTAMPTZ` -> `TEXT` (ISO-8601), `JSONB` -> `TEXT`
- `VECTOR(1536)` -> `F32_BLOB(1536)` with `libsql_vector_idx`
- `tsvector`/`ts_rank_cd` -> FTS5 virtual table with sync triggers
- PL/pgSQL functions -> SQLite triggers
**Tables (both backends):**
**Core:**
- `conversations` - Multi-channel conversation tracking
@@ -261,12 +376,41 @@ Single migration in `migrations/V1__initial.sql`. Tables:
**Workspace/Memory:**
- `memory_documents` - Flexible path-based files (e.g., "context/vision.md", "daily/2024-01-15.md")
- `memory_chunks` - Chunked content with FTS (tsvector) and vector (pgvector) indexes
- `memory_chunks` - Chunked content with FTS and vector indexes
- `heartbeat_state` - Periodic execution tracking
Requires pgvector extension: `CREATE EXTENSION IF NOT EXISTS vector;`
**Other:**
- `routines`, `routine_runs` - Scheduled/reactive execution
- `settings` - Per-user key-value settings
- `tool_failures` - Self-repair tracking
- `secrets`, `wasm_tools`, `tool_capabilities` - Extension infrastructure
Run migrations: `refinery migrate -c refinery.toml`
### Configuration
```bash
# Backend selection (default: postgres)
DATABASE_BACKEND=libsql
# PostgreSQL
DATABASE_URL=postgres://user:pass@localhost/ironclaw
# libSQL (embedded)
LIBSQL_PATH=~/.ironclaw/ironclaw.db # Default path
# libSQL (Turso cloud sync)
LIBSQL_URL=libsql://your-db.turso.io
LIBSQL_AUTH_TOKEN=your-token # Required when LIBSQL_URL is set
```
### Current Limitations (libSQL backend)
- **Workspace/memory system** not yet wired through Database trait (requires Store migration)
- **Secrets store** not yet available (still requires PostgresSecretsStore)
- **Hybrid search** uses FTS5 only (vector search via libsql_vector_idx not yet implemented)
- **Settings reload from DB** skipped (Config::from_db requires Store)
- No incremental migration versioning (schema is CREATE IF NOT EXISTS, no ALTER TABLE support yet)
- **No encryption at rest** -- The local SQLite database file stores conversation content, job data, workspace memory, and other application data in plaintext. Only secrets (API tokens, credentials) are encrypted via AES-256-GCM before storage. Users handling sensitive data should use full-disk encryption (FileVault, LUKS, BitLocker) or consider the PostgreSQL backend with TDE/encrypted storage.
- **JSON merge patch vs path-targeted update** -- The libSQL backend uses RFC 7396 JSON Merge Patch (`json_patch`) for metadata updates, while PostgreSQL uses path-targeted `jsonb_set`. Merge patch replaces top-level keys entirely, which may drop nested keys not present in the patch. Callers should avoid relying on partial nested object updates in metadata fields.
## Safety Layer
@@ -297,13 +441,14 @@ Key test patterns:
## Current Limitations / TODOs
1. **Slack/Telegram channels** - Stubs only, need implementation
2. **Domain-specific tools** - `marketplace.rs`, `restaurant.rs`, `taskrabbit.rs`, `ecommerce.rs` return placeholder responses; need real API integrations
3. **Integration tests** - Need testcontainers setup for PostgreSQL
4. **MCP stdio transport** - Only HTTP transport implemented
5. **WIT bindgen integration** - Auto-extract tool description/schema from WASM modules (stubbed)
6. **Capability granting after tool build** - Built tools get empty capabilities; need UX for granting HTTP/secrets access
7. **Tool versioning workflow** - No version tracking or rollback for dynamically built tools
1. **Domain-specific tools** - `marketplace.rs`, `restaurant.rs`, `taskrabbit.rs`, `ecommerce.rs` return placeholder responses; need real API integrations
2. **Integration tests** - Need testcontainers setup for PostgreSQL
3. **MCP stdio transport** - Only HTTP transport implemented
4. **WIT bindgen integration** - Auto-extract tool description/schema from WASM modules (stubbed)
5. **Capability granting after tool build** - Built tools get empty capabilities; need UX for granting HTTP/secrets access
6. **Tool versioning workflow** - No version tracking or rollback for dynamically built tools
7. **Webhook trigger endpoint** - Routines webhook trigger not yet exposed in web gateway
8. **Full channel status view** - Gateway status widget exists, but no per-channel connection dashboard
### Completed
@@ -320,6 +465,14 @@ Key test patterns:
-**Tool approval enforcement** - Tools with `requires_approval()` (shell, http, file write/patch, build_software) now gate execution, track auto-approved tools per session
-**Tool definition refresh** - Tool definitions refreshed each iteration so newly built tools become visible in same session
-**Worker tool call handling** - Uses `respond_with_tools()` to properly execute tool calls when `select_tools()` returns empty
-**Gateway control plane** - Web gateway with 40+ API endpoints, SSE/WebSocket
-**Web Control UI** - Browser-based dashboard with chat, memory, jobs, logs, extensions, routines
-**Slack/Telegram channels** - Implemented as WASM tools
-**Docker sandbox** - Orchestrator/worker containers with per-job auth
-**Claude Code mode** - Delegate jobs to Claude CLI inside containers
-**Routines system** - Cron, event, webhook, and manual triggers with guardrails
-**Extension management** - Install, auth, activate MCP/WASM extensions via CLI and web UI
-**libSQL/Turso backend** - Database trait abstraction (`src/db/`), feature-gated dual backend support (postgres/libsql), embedded SQLite for zero-dependency local mode
## Adding a New Tool
@@ -558,7 +711,7 @@ Four tools for LLM use:
### Hybrid Search (RRF)
Combines full-text search (PostgreSQL `ts_rank_cd`) and vector similarity (pgvector cosine) using Reciprocal Rank Fusion:
Combines full-text search and vector similarity using Reciprocal Rank Fusion:
```
score(d) = Σ 1/(k + rank(d)) for each method where d appears
@@ -566,6 +719,10 @@ score(d) = Σ 1/(k + rank(d)) for each method where d appears
Default k=60. Results from both methods are combined, with documents appearing in both getting boosted scores.
**Backend differences:**
- **PostgreSQL:** `ts_rank_cd` for FTS, pgvector cosine distance for vectors, full RRF
- **libSQL:** FTS5 for keyword search only (vector search via `libsql_vector_idx` not yet wired)
### Heartbeat System
Proactive periodic execution (default: 30 minutes):
Generated
+1024 -61
View File
File diff suppressed because it is too large Load Diff
+84 -9
View File
@@ -1,10 +1,19 @@
[package]
name = "ironclaw"
version = "0.1.0"
version = "0.1.3"
edition = "2024"
rust-version = "1.85"
description = "Secure personal AI assistant that protects your data and expands its capabilities on the fly"
authors = ["NEAR AI <[email protected]>"]
license = "MIT OR Apache-2.0"
homepage = "https://github.com/nearai/ironclaw"
repository = "https://github.com/nearai/ironclaw"
[package.metadata.wix]
upgrade-guid = "D0156E61-BA37-451E-8AB9-1A2ECCCFA48F"
path-guid = "F90B6EA6-87F7-499B-BB19-CF55DE1EB339"
license = false
eula = false
[dependencies]
# Async runtime
@@ -19,11 +28,14 @@ reqwest = { version = "0.12", default-features = false, features = ["json", "rus
serde = { version = "1", features = ["derive"] }
serde_json = "1"
# Database
deadpool-postgres = "0.14"
tokio-postgres = { version = "0.7", features = ["with-uuid-1", "with-chrono-0_4", "with-serde_json-1"] }
postgres-types = { version = "0.2", features = ["with-serde_json-1"] }
refinery = { version = "0.8", features = ["tokio-postgres"] }
# Database - PostgreSQL (default, feature-gated)
deadpool-postgres = { version = "0.14", optional = true }
tokio-postgres = { version = "0.7", features = ["with-uuid-1", "with-chrono-0_4", "with-serde_json-1"], optional = true }
postgres-types = { version = "0.2", features = ["with-serde_json-1"], optional = true }
refinery = { version = "0.8", features = ["tokio-postgres"], optional = true }
# Database - libSQL/Turso (optional embedded database)
libsql = { version = "0.6", optional = true, default-features = false, features = ["core", "replication"] }
# Error handling
thiserror = "2"
@@ -39,7 +51,7 @@ dotenvy = "0.15"
# Core types
uuid = { version = "1", features = ["v4", "serde"] }
chrono = { version = "0.4", features = ["serde"] }
rust_decimal = { version = "1", features = ["serde", "serde-with-str", "db-tokio-postgres", "maths"] }
rust_decimal = { version = "1", features = ["serde", "serde-with-str", "maths"] }
rust_decimal_macros = "1"
# Async traits
@@ -58,12 +70,16 @@ axum = { version = "0.8", features = ["ws"] }
tower = "0.5"
tower-http = { version = "0.6", features = ["trace", "cors"] }
# Cron scheduling for routines
cron = "0.13"
# Safety/sanitization
regex = "1"
aho-corasick = "1"
# Filesystem paths
dirs = "6"
fs4 = "0.6"
# Secrecy for sensitive values
secrecy = { version = "0.10", features = ["serde"] }
@@ -76,7 +92,7 @@ open = "5"
# Vector embeddings for semantic search
# The postgres feature provides ToSql/FromSql for postgres-types (shared by tokio-postgres)
pgvector = { version = "0.4", features = ["postgres"] }
pgvector = { version = "0.4", features = ["postgres"], optional = true }
# WASM sandbox for untrusted tool execution
wasmtime = { version = "28", features = ["component-model"] }
@@ -89,6 +105,10 @@ hkdf = "0.12"
sha2 = "0.10"
blake3 = "1"
rand = "0.8"
subtle = "2" # Constant-time comparisons for token validation
# Multi-provider LLM support
rig-core = "0.30"
# Docker sandbox
bollard = "0.18"
@@ -99,6 +119,7 @@ hyper-util = { version = "0.1", features = ["server", "tokio", "http1", "http2"]
http-body-util = "0.1"
bytes = "1"
base64 = "0.22.1"
mime_guess = "2.0.5"
# macOS keychain
[target.'cfg(target_os = "macos")'.dependencies]
@@ -117,5 +138,59 @@ pretty_assertions = "1"
tempfile = "3"
[features]
default = []
default = ["postgres"]
postgres = [
"dep:deadpool-postgres",
"dep:tokio-postgres",
"dep:postgres-types",
"dep:refinery",
"dep:pgvector",
"rust_decimal/db-tokio-postgres",
]
libsql = ["dep:libsql"]
integration = []
[[example]]
name = "test_heartbeat"
required-features = ["postgres"]
# The profile that 'cargo dist' will build with
[profile.dist]
inherits = "release"
lto = "thin"
# Config for 'dist'
[workspace.metadata.dist]
# The preferred dist version to use in CI (Cargo.toml SemVer syntax)
cargo-dist-version = "0.30.3"
# CI backends to support
ci = "github"
# The installers to generate for each app
installers = ["shell", "powershell", "npm", "msi"]
# Publish jobs to run in CI
publish-jobs = []
# Target platforms to build apps for (Rust target-triple syntax)
targets = [
"aarch64-apple-darwin",
"aarch64-unknown-linux-gnu",
"x86_64-apple-darwin",
"x86_64-unknown-linux-gnu",
"x86_64-pc-windows-msvc",
]
# The archive format to use for windows builds (defaults .zip)
windows-archive = ".tar.gz"
# The archive format to use for non-windows builds (defaults .tar.xz)
unix-archive = ".tar.gz"
# Which actions to run on pull requests
pr-run-mode = "upload"
# Path that installers should place binaries in
install-path = "CARGO_HOME"
# Whether to install an updater program
install-updater = true
[workspace.metadata.dist.github-custom-runners]
aarch64-unknown-linux-gnu = "ubuntu-24.04-arm"
x86_64-unknown-linux-gnu = "ubuntu-22.04"
x86_64-pc-windows-msvc = "windows-2022"
x86_64-apple-darwin = "macos-15-intel"
aarch64-apple-darwin = "macos-14"
+63
View File
@@ -0,0 +1,63 @@
# Multi-stage Dockerfile for the IronClaw worker container.
#
# This image runs the ironclaw binary in worker mode inside Docker containers.
# The orchestrator creates instances of this image for sandboxed job execution.
#
# Build:
# docker build -f Dockerfile.worker -t ironclaw-worker .
#
# The image includes common development tools so workers can build software,
# run tests, and execute shell commands.
FROM rust:1.85-bookworm AS builder
WORKDIR /build
COPY . .
# Build only the ironclaw binary (release mode)
RUN cargo build --release --bin ironclaw
# ---
FROM debian:bookworm-slim
# Install common development tools
RUN apt-get update && apt-get install -y --no-install-recommends \
ca-certificates \
curl \
git \
build-essential \
pkg-config \
libssl-dev \
nodejs \
npm \
python3 \
python3-pip \
python3-venv \
&& rm -rf /var/lib/apt/lists/*
# Install Rust toolchain for the sandbox user
ENV RUSTUP_HOME=/usr/local/rustup \
CARGO_HOME=/usr/local/cargo \
PATH=/usr/local/cargo/bin:$PATH
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --default-toolchain 1.85.0 \
&& chmod -R a+r /usr/local/rustup /usr/local/cargo
# Install Claude Code CLI (for claude-bridge mode)
RUN npm install -g @anthropic-ai/claude-code@latest
# Copy the binary
COPY --from=builder /build/target/release/ironclaw /usr/local/bin/ironclaw
# Create non-root user (UID 1000 matches the orchestrator's container config)
RUN useradd -m -u 1000 -s /bin/bash sandbox \
&& mkdir -p /workspace \
&& chown sandbox:sandbox /workspace \
&& mkdir -p /home/sandbox/.claude \
&& chown sandbox:sandbox /home/sandbox/.claude
USER sandbox
WORKDIR /workspace
# The orchestrator passes the full command via Docker cmd.
ENTRYPOINT ["ironclaw"]
+45 -35
View File
@@ -16,8 +16,8 @@ This document tracks feature parity between IronClaw (Rust implementation) and O
| Feature | OpenClaw | IronClaw | Notes |
|---------|----------|----------|-------|
| Hub-and-spoke architecture | ✅ | 🚧 | IronClaw has channels but no central gateway |
| WebSocket control plane | ✅ | | Gateway with ws://127.0.0.1:18789 |
| Hub-and-spoke architecture | ✅ | | Web gateway as central hub |
| WebSocket control plane | ✅ | | Gateway with WebSocket + SSE |
| Single-user system | ✅ | ✅ | |
| Multi-agent routing | ✅ | ❌ | Workspace isolation per-agent |
| Session-based messaging | ✅ | ✅ | Per-sender sessions |
@@ -31,19 +31,19 @@ This document tracks feature parity between IronClaw (Rust implementation) and O
| Feature | OpenClaw | IronClaw | Notes |
|---------|----------|----------|-------|
| Gateway control plane | ✅ | | Central WebSocket server |
| HTTP endpoints for Control UI | ✅ | | Web dashboard |
| Channel connection lifecycle | ✅ | 🚧 | ChannelManager handles streams |
| Gateway control plane | ✅ | | Web gateway with 40+ API endpoints |
| HTTP endpoints for Control UI | ✅ | | Web dashboard with chat, memory, jobs, logs, extensions |
| Channel connection lifecycle | ✅ | | ChannelManager + WebSocket tracker |
| Session management/routing | ✅ | ✅ | SessionManager exists |
| Configuration hot-reload | ✅ | ❌ | |
| Network modes (loopback/LAN/remote) | ✅ | 🚧 | HTTP only |
| OpenAI-compatible HTTP API | ✅ | | /v1/chat/completions |
| OpenAI-compatible HTTP API | ✅ | | /v1/chat/completions |
| Canvas hosting | ✅ | ❌ | Agent-driven UI |
| Gateway lock (PID-based) | ✅ | ❌ | |
| launchd/systemd integration | ✅ | ❌ | |
| Bonjour/mDNS discovery | ✅ | ❌ | |
| Tailscale integration | ✅ | ❌ | |
| Health check endpoints | ✅ | | |
| Health check endpoints | ✅ | | /api/health + /api/gateway/status |
| `doctor` diagnostics | ✅ | ❌ | |
### Owner: _Unassigned_
@@ -59,14 +59,14 @@ This document tracks feature parity between IronClaw (Rust implementation) and O
| REPL (simple) | ✅ | ✅ | - | For testing |
| WASM channels | ❌ | ✅ | - | IronClaw innovation |
| WhatsApp | ✅ | ❌ | P1 | Baileys (Web) |
| Telegram | ✅ | | P1 | grammY (Bot API) |
| Telegram | ✅ | | - | WASM channel(MTProto), DM pairing, caption, /start, bot_username |
| Discord | ✅ | ❌ | P2 | discord.js |
| Signal | ✅ | ❌ | P2 | signal-cli |
| Slack | ✅ | 🚧 | P1 | Stub exists, needs implementation |
| Slack | ✅ | | - | WASM tool |
| iMessage | ✅ | ❌ | P3 | BlueBubbles recommended |
| Feishu/Lark | ✅ | ❌ | P3 | |
| LINE | ✅ | ❌ | P3 | |
| WebChat | ✅ | | P2 | Browser-based chat |
| WebChat | ✅ | | - | Web gateway chat |
| Matrix | ✅ | ❌ | P3 | E2EE support |
| Mattermost | ✅ | ❌ | P3 | |
| Google Chat | ✅ | ❌ | P3 | |
@@ -79,13 +79,13 @@ This document tracks feature parity between IronClaw (Rust implementation) and O
| Feature | OpenClaw | IronClaw | Notes |
|---------|----------|----------|-------|
| DM pairing codes | ✅ | | Verification for unknown senders |
| Allowlist/blocklist | ✅ | | Per-channel access control |
| DM pairing codes | ✅ | | `ironclaw pairing list/approve`, host APIs |
| Allowlist/blocklist | ✅ | 🚧 | allow_from + pairing store |
| Self-message bypass | ✅ | ❌ | Own messages skip pairing |
| Mention-based activation | ✅ | | Configurable patterns |
| Mention-based activation | ✅ | | bot_username + respond_to_all_group_messages |
| Per-group tool policies | ✅ | ❌ | Allow/deny specific tools |
| Thread isolation | ✅ | ✅ | Separate sessions per thread |
| Per-channel media limits | ✅ | | |
| Per-channel media limits | ✅ | 🚧 | Caption support for media; no size limits |
| Typing indicators | ✅ | 🚧 | TUI shows status |
### Owner: _Unassigned_
@@ -99,17 +99,17 @@ This document tracks feature parity between IronClaw (Rust implementation) and O
| `run` (agent) | ✅ | ✅ | - | Default command |
| `tool install/list/remove` | ✅ | ✅ | - | WASM tools |
| `gateway start/stop` | ✅ | ❌ | P2 | |
| `onboard` (wizard) | ✅ | | P2 | Interactive setup |
| `onboard` (wizard) | ✅ | | - | Interactive setup |
| `tui` | ✅ | ✅ | - | Ratatui TUI |
| `config` | ✅ | | P2 | Read/write config |
| `config` | ✅ | | - | Read/write config |
| `channels` | ✅ | ❌ | P2 | Channel management |
| `models` | ✅ | 🚧 | - | Model selector in TUI |
| `status` | ✅ | | P2 | System status |
| `status` | ✅ | | - | System status |
| `agents` | ✅ | ❌ | P3 | Multi-agent management |
| `sessions` | ✅ | ❌ | P3 | Session listing |
| `memory` | ✅ | | P2 | Memory search CLI |
| `memory` | ✅ | | - | Memory search CLI |
| `skills` | ✅ | ❌ | P3 | Agent skills |
| `pairing` | ✅ | | P3 | Node pairing |
| `pairing` | ✅ | | - | list/approve for channel DM pairing |
| `nodes` | ✅ | ❌ | P3 | Device management |
| `plugins` | ✅ | ❌ | P3 | Plugin management |
| `hooks` | ✅ | ❌ | P2 | Lifecycle hooks |
@@ -132,7 +132,7 @@ This document tracks feature parity between IronClaw (Rust implementation) and O
| Feature | OpenClaw | IronClaw | Notes |
|---------|----------|----------|-------|
| Pi agent runtime | ✅ | | IronClaw uses custom runtime |
| RPC-based execution | ✅ | 🚧 | Worker isolation |
| RPC-based execution | ✅ | | Orchestrator/worker pattern |
| Multi-provider failover | ✅ | ❌ | Provider fallback chains |
| Per-sender sessions | ✅ | ✅ | |
| Global sessions | ✅ | ❌ | Optional shared context |
@@ -303,13 +303,13 @@ This document tracks feature parity between IronClaw (Rust implementation) and O
| Feature | OpenClaw | IronClaw | Priority | Notes |
|---------|----------|----------|----------|-------|
| Control UI Dashboard | ✅ | | P2 | Web status/config |
| Channel status view | ✅ | | P2 | |
| Control UI Dashboard | ✅ | | - | Web gateway with chat, memory, jobs, logs, extensions |
| Channel status view | ✅ | 🚧 | P2 | Gateway status widget, full channel view pending |
| Agent management | ✅ | ❌ | P3 | |
| Model selection | ✅ | ✅ | - | TUI only |
| Config editing | ✅ | ❌ | P3 | |
| Debug/logs viewer | ✅ | | P3 | |
| WebChat interface | ✅ | | P2 | Browser chat |
| Debug/logs viewer | ✅ | | - | Real-time log streaming with level/target filters |
| WebChat interface | ✅ | | - | Web gateway chat with SSE/WebSocket |
| Canvas system (A2UI) | ✅ | ❌ | P3 | Agent-driven UI |
### Owner: _Unassigned_
@@ -320,13 +320,13 @@ This document tracks feature parity between IronClaw (Rust implementation) and O
| Feature | OpenClaw | IronClaw | Priority | Notes |
|---------|----------|----------|----------|-------|
| Cron jobs | ✅ | | P2 | Schedule-based tasks |
| Timezone support | ✅ | | P2 | |
| One-shot/recurring jobs | ✅ | | P2 | |
| Cron jobs | ✅ | | - | Routines with cron trigger |
| Timezone support | ✅ | | - | Via cron expressions |
| One-shot/recurring jobs | ✅ | | - | Manual + cron triggers |
| `beforeInbound` hook | ✅ | ❌ | P2 | |
| `beforeOutbound` hook | ✅ | ❌ | P2 | |
| `beforeToolCall` hook | ✅ | ❌ | P2 | |
| `onMessage` hook | ✅ | | P2 | |
| `onMessage` hook | ✅ | | - | Routines with event trigger |
| `onSessionStart` hook | ✅ | ❌ | P2 | |
| `onSessionEnd` hook | ✅ | ❌ | P2 | |
| `transcribeAudio` hook | ✅ | ❌ | P3 | |
@@ -346,18 +346,18 @@ This document tracks feature parity between IronClaw (Rust implementation) and O
| Feature | OpenClaw | IronClaw | Notes |
|---------|----------|----------|-------|
| Gateway token auth | ✅ | 🚧 | HTTP webhook secret |
| Gateway token auth | ✅ | | Bearer token auth on web gateway |
| Device pairing | ✅ | ❌ | |
| Tailscale identity | ✅ | ❌ | |
| OAuth flows | ✅ | 🚧 | NEAR AI OAuth |
| DM pairing verification | ✅ | | |
| Allowlist/blocklist | ✅ | | |
| DM pairing verification | ✅ | | ironclaw pairing approve, host APIs |
| Allowlist/blocklist | ✅ | 🚧 | allow_from + pairing store |
| Per-group tool policies | ✅ | ❌ | |
| Exec approvals | ✅ | ✅ | TUI overlay |
| TLS 1.3 minimum | ✅ | ✅ | reqwest rustls |
| SSRF protection | ✅ | ✅ | WASM allowlist |
| Loopback-first | ✅ | 🚧 | HTTP binds 0.0.0.0 |
| Docker sandbox | ✅ | | Uses WASM sandbox |
| Docker sandbox | ✅ | | Orchestrator/worker containers |
| WASM sandbox | ❌ | ✅ | IronClaw innovation |
| Tool policies | ✅ | ✅ | |
| Elevated mode | ✅ | ❌ | |
@@ -397,6 +397,7 @@ This document tracks feature parity between IronClaw (Rust implementation) and O
### P0 - Core (Already Done)
- ✅ TUI channel with approval overlays
- ✅ HTTP webhook channel
- ✅ DM pairing (ironclaw pairing list/approve, host APIs)
- ✅ WASM tool sandbox
- ✅ Workspace/memory with hybrid search
- ✅ Prompt injection defense
@@ -404,23 +405,32 @@ This document tracks feature parity between IronClaw (Rust implementation) and O
- ✅ Session management
- ✅ Context compaction
- ✅ Model selection
- ✅ Gateway control plane + WebSocket
- ✅ Web Control UI (chat, memory, jobs, logs, extensions, routines)
- ✅ WebChat channel (web gateway)
- ✅ Slack channel (WASM tool)
- ✅ Telegram channel (WASM tool, MTProto)
- ✅ Docker sandbox (orchestrator/worker)
- ✅ Cron job scheduling (routines)
- ✅ CLI subcommands (onboard, config, status, memory)
- ✅ Gateway token auth
### P1 - High Priority
- ❌ Slack channel (real implementation)
- Telegram channel
- Telegram channel (WASM, DM pairing, caption, /start)
- ❌ WhatsApp channel
- ❌ Multi-provider failover
- ❌ Gateway control plane + WebSocket
- ❌ Hooks system (beforeInbound, beforeToolCall, etc.)
### P2 - Medium Priority
- ❌ Cron job scheduling
- ❌ Web Control UI
- ❌ WebChat channel
- Media handling (images, PDFs)
- 🚧 Media handling (caption support; no image/PDF processing)
- ❌ CLI subcommands (config, status, memory, doctor)
- ❌ Ollama/local model support
- ❌ Configuration hot-reload
- ❌ Webhook trigger endpoint in web gateway
### P3 - Lower Priority
- ❌ Discord channel
+86 -36
View File
@@ -43,7 +43,10 @@ IronClaw is the AI assistant you can actually trust with your personal and profe
### Always Available
- **Multi-channel** - REPL, HTTP webhooks, and extensible WASM channels (Telegram, Slack, and more)
- **Multi-channel** - REPL, HTTP webhooks, WASM channels (Telegram, Slack), and web gateway
- **Docker Sandbox** - Isolated container execution with per-job tokens and orchestrator/worker pattern
- **Web Gateway** - Browser UI with real-time SSE/WebSocket streaming
- **Routines** - Cron schedules, event triggers, webhook handlers for background automation
- **Heartbeat System** - Proactive background execution for monitoring and maintenance tasks
- **Parallel Jobs** - Handle multiple requests concurrently with isolated contexts
- **Self-repair** - Automatic detection and recovery of stuck operations
@@ -65,10 +68,41 @@ IronClaw is the AI assistant you can actually trust with your personal and profe
### Prerequisites
- Rust 1.85+
- PostgreSQL 15+ with pgvector extension
- PostgreSQL 15+ with [pgvector](https://github.com/pgvector/pgvector) extension
- NEAR AI account (authentication handled via setup wizard)
### Build
## Download or Build
Visit [Releases page](https://github.com/nearai/ironclaw/releases/) to see the latest updates.
<details>
<summary>Install via Windows Installer (Windows)</summary>
Download the [Windows Installer](https://github.com/nearai/ironclaw/releases/latest/download/ironclaw-x86_64-pc-windows-msvc.msi) and run it.
</details>
<details>
<summary>Install via powershell script (Windows)</summary>
```sh
irm https://github.com/nearai/ironclaw/releases/latest/download/ironclaw-installer.ps1 | iex
```
</details>
<details>
<summary>Install via shell script (macOS, Linux, Windows/WSL)</summary>
```sh
curl --proto '=https' --tlsv1.2 -LsSf https://github.com/nearai/ironclaw/releases/latest/download/ironclaw-installer.sh | sh
```
</details>
<details>
<summary>Compile the source code (Cargo on Windows, Linux, macOS)</summary>
Install it with `cargo`, just make sure you have [Rust](https://rustup.rs) installed on your computer.
```bash
# Clone the repository
@@ -82,6 +116,10 @@ cargo build --release
cargo test
```
For **full release** (after modifying channel sources), run `./scripts/build-all.sh` to rebuild channels first.
</details>
### Database Setup
```bash
@@ -97,7 +135,7 @@ psql ironclaw -c "CREATE EXTENSION IF NOT EXISTS vector;"
Run the setup wizard to configure IronClaw:
```bash
ironclaw setup
ironclaw onboard
```
The wizard handles database connection, NEAR AI authentication (via browser OAuth),
@@ -143,37 +181,42 @@ External content passes through multiple security layers:
## Architecture
```
┌─────────────────────────────────────────────────────────────────┐
│ Channels │
│ ┌──────┐ ┌──────┐ ┌─────────────
│ │ REPL │ │ HTTP │ │ WASM Channels│
│ └──┬───┘ └──┬───┘ └──────┬──────
└─────────┴─────────────┘
┌────▼────┐
│ Router │ Intent classification
└────┬────┘
┌──────────▼──────────┐
│ Scheduler Parallel job management
└──────────┬──────────┘
──────────────────────────────
│ ┌─────────┐ ┌─────────┐ ┌─────────
│ Worker │ │ Worker Worker LLM reasoning
────────┘ ────────┘ └────────
└───────────────┼───────────────┘
┌──────────▼──────────┐
│ Tool Registry │ │
│ │ ┌───────────────┐ │
│ │ │ Built-in │ │
│ │ │ MCP │ │
│ │ │ WASM Sandbox │ │
└───────────────┘
└─────────────────────┘
└─────────────────────────────────────────────────────────────────┘
┌────────────────────────────────────────────────────────────────────
Channels
│ ┌──────┐ ┌──────┐ ┌─────────────┐ ┌─────────────┐
│ │ REPL │ │ HTTP │ │WASM Channels│ │ Web Gateway │
│ └──┬───┘ └──┬───┘ └──────┬──────┘ │ (SSE + WS) │
│ │ └──────┬──────┘
└─────────┴──────────────┴────────────────┘
┌─────────▼─────────┐
│ Agent Loop Intent routing
└────┬─────────┬────┘
│ │
┌──────────▼───┐ ┌──▼──────────────┐
│ Scheduler │ Routines Engine
│(parallel jobs)│ │(cron, event, wh) │
─────────────┘ └─────────────────
┌─────────────┼───────────────────
─────── ────────────────────
│ Local │ │ Orchestrator │
│Workers │ ┌───────────────┐
│(in-proc)│ │ Docker Sandbox│ │
└───┬────┘ │ │ Containers
┌───────────┐ │ │
│ │ │Worker / CC│ │ │ │
│ │ └───────────┘ │ │ │
└───────────────┘ │
────────┬───────────┘ │
│ └──────────────────
│ │ │
│ ┌───────────▼──────────┐ │
│ │ Tool Registry │ │
│ │ Built-in, MCP, WASM │ │
│ └──────────────────────┘ │
└────────────────────────────────────────────────────────────────────┘
```
### Core Components
@@ -184,6 +227,9 @@ External content passes through multiple security layers:
| **Router** | Classifies user intent (command, query, task) |
| **Scheduler** | Manages parallel job execution with priorities |
| **Worker** | Executes jobs with LLM reasoning and tool calls |
| **Orchestrator** | Container lifecycle, LLM proxying, per-job auth |
| **Web Gateway** | Browser UI with chat, memory, jobs, logs, extensions, routines |
| **Routines Engine** | Scheduled (cron) and reactive (event, webhook) background tasks |
| **Workspace** | Persistent memory with hybrid search |
| **Safety Layer** | Prompt injection defense and content sanitization |
@@ -191,7 +237,7 @@ External content passes through multiple security layers:
```bash
# First-time setup (configures database, auth, etc.)
ironclaw setup
ironclaw onboard
# Start interactive REPL
cargo run
@@ -210,12 +256,16 @@ cargo fmt
cargo clippy --all --benches --tests --examples --all-features
# Run tests
createdb ironclaw_test
cargo test
# Run specific test
cargo test test_name
```
- **Telegram channel**: See [docs/TELEGRAM_SETUP.md](docs/TELEGRAM_SETUP.md) for setup and DM pairing.
- **Changing channel sources**: Run `./channels-src/telegram/build.sh` before `cargo build` so the updated WASM is bundled.
## OpenClaw Heritage
IronClaw is a Rust reimplementation inspired by [OpenClaw](https://github.com/openclaw/openclaw). See [FEATURE_PARITY.md](FEATURE_PARITY.md) for the complete tracking matrix.
+106
View File
@@ -0,0 +1,106 @@
//! Build script: compile Telegram channel WASM from source.
//!
//! Do not commit compiled WASM binaries — they are a supply chain risk.
//! This script builds telegram.wasm from channels-src/telegram before the main crate compiles.
//!
//! Reproducible build:
//! cargo build --release
//! (build.rs invokes the channel build automatically)
//!
//! Prerequisites: rustup target add wasm32-wasip2, cargo install wasm-tools
use std::env;
use std::path::PathBuf;
use std::process::Command;
fn main() {
let manifest_dir = env::var("CARGO_MANIFEST_DIR").unwrap();
let root = PathBuf::from(&manifest_dir);
let channel_dir = root.join("channels-src/telegram");
let wasm_out = channel_dir.join("telegram.wasm");
// Rerun when channel source or build script changes
println!("cargo:rerun-if-changed=channels-src/telegram/src");
println!("cargo:rerun-if-changed=channels-src/telegram/Cargo.toml");
println!("cargo:rerun-if-changed=wit/channel.wit");
if !channel_dir.is_dir() {
return;
}
// Build WASM module
let status = match Command::new("cargo")
.args([
"build",
"--release",
"--target",
"wasm32-wasip2",
"--manifest-path",
channel_dir.join("Cargo.toml").to_str().unwrap(),
])
.current_dir(&root)
.status()
{
Ok(s) => s,
Err(_) => {
eprintln!(
"cargo:warning=Telegram channel build failed. Run: ./channels-src/telegram/build.sh"
);
return;
}
};
if !status.success() {
eprintln!(
"cargo:warning=Telegram channel build failed. Run: ./channels-src/telegram/build.sh"
);
return;
}
let raw_wasm = channel_dir.join("target/wasm32-wasip2/release/telegram_channel.wasm");
if !raw_wasm.exists() {
eprintln!(
"cargo:warning=Telegram WASM output not found at {:?}",
raw_wasm
);
return;
}
// Convert to component and strip (wasm-tools)
let component_ok = Command::new("wasm-tools")
.args([
"component",
"new",
raw_wasm.to_str().unwrap(),
"-o",
wasm_out.to_str().unwrap(),
])
.current_dir(&root)
.status()
.map(|s| s.success())
.unwrap_or(false);
if !component_ok {
// Fallback: copy raw module if wasm-tools unavailable
if std::fs::copy(&raw_wasm, &wasm_out).is_err() {
eprintln!("cargo:warning=wasm-tools not found. Run: cargo install wasm-tools");
}
} else {
// Strip debug info (use temp file to avoid clobbering)
let stripped = wasm_out.with_extension("wasm.stripped");
let strip_ok = Command::new("wasm-tools")
.args([
"strip",
wasm_out.to_str().unwrap(),
"-o",
stripped.to_str().unwrap(),
])
.current_dir(&root)
.status()
.map(|s| s.success())
.unwrap_or(false);
if strip_ok {
let _ = std::fs::rename(&stripped, &wasm_out);
}
}
}
+1 -1
View File
@@ -30,7 +30,7 @@ if [ -f "$WASM_PATH" ]; then
wasm-tools strip slack.wasm -o slack.wasm
echo "Built: slack.wasm ($(du -h slack.wasm | cut -f1))"
echo "Copy slack.wasm and slack.capabilities.json to ~/.near-agent/channels/"
echo "Copy slack.wasm and slack.capabilities.json to ~/.ironclaw/channels/"
else
echo "Error: WASM output not found at $WASM_PATH"
exit 1
+24 -22
View File
@@ -108,7 +108,10 @@ struct SlackPostMessageResponse {
#[derive(Debug, Deserialize)]
struct SlackConfig {
/// Name of secret containing signing secret (for verification by host).
/// Parsed from config for forward compatibility; not yet used in WASM
/// (host handles signature verification).
#[serde(default = "default_signing_secret_name")]
#[allow(dead_code)]
signing_secret_name: String,
}
@@ -175,11 +178,7 @@ impl Guest for SlackChannel {
// Actual event callback
"event_callback" => {
if let Some(event) = event_wrapper.event {
handle_slack_event(
event,
event_wrapper.team_id,
event_wrapper.event_id,
);
handle_slack_event(event, event_wrapper.team_id, event_wrapper.event_id);
}
// Always respond 200 quickly to Slack (they have a 3s timeout)
json_response(200, serde_json::json!({"ok": true}))
@@ -230,6 +229,7 @@ impl Guest for SlackChannel {
"https://slack.com/api/chat.postMessage",
&headers.to_string(),
Some(&payload_bytes),
None,
);
match result {
@@ -243,14 +243,15 @@ impl Guest for SlackChannel {
// Parse Slack response
let slack_response: SlackPostMessageResponse =
serde_json::from_slice(&http_response.body).map_err(|e| {
format!("Failed to parse Slack response: {}", e)
})?;
serde_json::from_slice(&http_response.body)
.map_err(|e| format!("Failed to parse Slack response: {}", e))?;
if !slack_response.ok {
return Err(format!(
"Slack API error: {}",
slack_response.error.unwrap_or_else(|| "unknown".to_string())
slack_response
.error
.unwrap_or_else(|| "unknown".to_string())
));
}
@@ -277,17 +278,16 @@ impl Guest for SlackChannel {
}
/// Handle a Slack event and emit message if applicable.
fn handle_slack_event(
event: SlackEvent,
team_id: Option<String>,
_event_id: Option<String>,
) {
fn handle_slack_event(event: SlackEvent, team_id: Option<String>, _event_id: Option<String>) {
match event.event_type.as_str() {
// Direct mention of the bot
"app_mention" => {
if let (Some(user), Some(channel), Some(text), Some(ts)) =
(event.user, event.channel.clone(), event.text, event.ts.clone())
{
if let (Some(user), Some(channel), Some(text), Some(ts)) = (
event.user,
event.channel.clone(),
event.text,
event.ts.clone(),
) {
emit_message(user, text, channel, event.thread_ts.or(Some(ts)), team_id);
}
}
@@ -299,9 +299,12 @@ fn handle_slack_event(
return;
}
if let (Some(user), Some(channel), Some(text), Some(ts)) =
(event.user, event.channel.clone(), event.text, event.ts.clone())
{
if let (Some(user), Some(channel), Some(text), Some(ts)) = (
event.user,
event.channel.clone(),
event.text,
event.ts.clone(),
) {
// Only process DMs (channel IDs starting with D)
if channel.starts_with('D') {
emit_message(user, text, channel, event.thread_ts.or(Some(ts)), team_id);
@@ -335,8 +338,7 @@ fn emit_message(
team_id,
};
let metadata_json =
serde_json::to_string(&metadata).unwrap_or_else(|_| "{}".to_string());
let metadata_json = serde_json::to_string(&metadata).unwrap_or_else(|_| "{}".to_string());
// Strip @ mentions of the bot from the text for cleaner messages
let cleaned_text = strip_bot_mention(&text);
+2 -2
View File
@@ -32,8 +32,8 @@ if [ -f "$WASM_PATH" ]; then
echo "Built: telegram.wasm ($(du -h telegram.wasm | cut -f1))"
echo ""
echo "To install:"
echo " mkdir -p ~/.near-agent/channels"
echo " cp telegram.wasm telegram.capabilities.json ~/.near-agent/channels/"
echo " mkdir -p ~/.ironclaw/channels"
echo " cp telegram.wasm telegram.capabilities.json ~/.ironclaw/channels/"
echo ""
echo "Then add your bot token to secrets:"
echo " # Set TELEGRAM_BOT_TOKEN in your environment or secrets store"
+403 -101
View File
@@ -72,6 +72,10 @@ struct TelegramMessage {
/// Message text.
text: Option<String>,
/// Caption for media (photo, video, document, etc.).
#[serde(default)]
caption: Option<String>,
/// Original message if this is a reply.
reply_to_message: Option<Box<TelegramMessage>>,
@@ -160,6 +164,21 @@ const POLLING_STATE_PATH: &str = "state/last_update_id";
/// Workspace path for persisting owner_id across WASM callbacks.
const OWNER_ID_PATH: &str = "state/owner_id";
/// Workspace path for persisting dm_policy across WASM callbacks.
const DM_POLICY_PATH: &str = "state/dm_policy";
/// Workspace path for persisting allow_from (JSON array) across WASM callbacks.
const ALLOW_FROM_PATH: &str = "state/allow_from";
/// Channel name for pairing store (used by pairing host APIs).
const CHANNEL_NAME: &str = "telegram";
/// Workspace path for persisting bot_username for mention detection in groups.
const BOT_USERNAME_PATH: &str = "state/bot_username";
/// Workspace path for persisting respond_to_all_group_messages flag.
const RESPOND_TO_ALL_GROUP_PATH: &str = "state/respond_to_all_group_messages";
// ============================================================================
// Channel Metadata
// ============================================================================
@@ -196,6 +215,14 @@ struct TelegramConfig {
#[serde(default)]
owner_id: Option<i64>,
/// DM policy: "pairing" (default), "allowlist", or "open".
#[serde(default)]
dm_policy: Option<String>,
/// Allowed sender IDs/usernames from config (merged with pairing-approved store).
#[serde(default)]
allow_from: Option<Vec<String>>,
/// Whether to respond to all group messages (not just mentions).
#[serde(default)]
respond_to_all_group_messages: bool,
@@ -257,6 +284,28 @@ impl Guest for TelegramChannel {
);
}
// Persist dm_policy and allow_from for DM pairing in handle_message
let dm_policy = config
.dm_policy
.as_deref()
.unwrap_or("pairing")
.to_string();
let _ = channel_host::workspace_write(DM_POLICY_PATH, &dm_policy);
let allow_from_json = serde_json::to_string(&config.allow_from.unwrap_or_default())
.unwrap_or_else(|_| "[]".to_string());
let _ = channel_host::workspace_write(ALLOW_FROM_PATH, &allow_from_json);
// Persist bot_username and respond_to_all_group_messages for group handling
let _ = channel_host::workspace_write(
BOT_USERNAME_PATH,
&config.bot_username.unwrap_or_default(),
);
let _ = channel_host::workspace_write(
RESPOND_TO_ALL_GROUP_PATH,
&config.respond_to_all_group_messages.to_string(),
);
// Mode is determined by whether the host injected a tunnel_url
// If tunnel is configured, use webhooks. Otherwise, use polling.
let webhook_mode = config.tunnel_url.is_some();
@@ -388,7 +437,9 @@ impl Guest for TelegramChannel {
let headers = serde_json::json!({});
let result = channel_host::http_request("GET", &url, &headers.to_string(), None);
// 35s HTTP timeout outlives Telegram's 30s server-side long-poll
let result =
channel_host::http_request("GET", &url, &headers.to_string(), None, Some(35_000));
match result {
Ok(response) => {
@@ -461,72 +512,52 @@ impl Guest for TelegramChannel {
}
fn on_respond(response: AgentResponse) -> Result<(), String> {
// Parse metadata to get chat info
let metadata: TelegramMessageMetadata = serde_json::from_str(&response.metadata_json)
.map_err(|e| format!("Failed to parse metadata: {}", e))?;
// Build sendMessage payload
let mut payload = serde_json::json!({
"chat_id": metadata.chat_id,
"text": response.content,
"parse_mode": "Markdown",
});
// Reply to the original message for context
payload["reply_to_message_id"] = serde_json::Value::Number(metadata.message_id.into());
let payload_bytes = serde_json::to_vec(&payload)
.map_err(|e| format!("Failed to serialize payload: {}", e))?;
// Make HTTP request to Telegram API
// The bot token is injected into the URL by the host
let headers = serde_json::json!({
"Content-Type": "application/json"
});
let result = channel_host::http_request(
"POST",
"https://api.telegram.org/bot{TELEGRAM_BOT_TOKEN}/sendMessage",
&headers.to_string(),
Some(&payload_bytes),
// Try sending with Markdown first; fall back to plain text if Telegram
// can't parse the entities (e.g. model leaked <tool_call> with underscores).
let result = send_message(
metadata.chat_id,
&response.content,
metadata.message_id,
Some("Markdown"),
);
match result {
Ok(http_response) => {
if http_response.status != 200 {
let body_str = String::from_utf8_lossy(&http_response.body);
return Err(format!(
"Telegram API returned status {}: {}",
http_response.status, body_str
));
}
// Parse Telegram response
let api_response: TelegramApiResponse<SentMessage> =
serde_json::from_slice(&http_response.body)
.map_err(|e| format!("Failed to parse Telegram response: {}", e))?;
if !api_response.ok {
return Err(format!(
"Telegram API error: {}",
api_response
.description
.unwrap_or_else(|| "unknown".to_string())
));
}
Ok(msg_id) => {
channel_host::log(
channel_host::LogLevel::Debug,
&format!(
"Sent message to chat {}: message_id={}",
metadata.chat_id,
api_response.result.map(|r| r.message_id).unwrap_or(0)
metadata.chat_id, msg_id
),
);
Ok(())
}
Err(e) => Err(format!("HTTP request failed: {}", e)),
Err(SendError::ParseEntities(detail)) => {
channel_host::log(
channel_host::LogLevel::Warn,
&format!("Markdown parse failed ({}), retrying as plain text", detail),
);
let msg_id = send_message(
metadata.chat_id,
&response.content,
metadata.message_id,
None,
)
.map_err(|e| format!("Plain-text retry also failed: {}", e))?;
channel_host::log(
channel_host::LogLevel::Debug,
&format!(
"Sent plain-text message to chat {}: message_id={}",
metadata.chat_id, msg_id
),
);
Ok(())
}
Err(e) => Err(e.to_string()),
}
}
@@ -568,6 +599,7 @@ impl Guest for TelegramChannel {
"https://api.telegram.org/bot{TELEGRAM_BOT_TOKEN}/sendChatAction",
&headers.to_string(),
Some(&payload_bytes),
None,
);
if let Err(e) = result {
@@ -586,6 +618,101 @@ impl Guest for TelegramChannel {
}
}
// ============================================================================
// Send Message Helper
// ============================================================================
/// Errors from send_message, split so callers can match on parse-entity failures.
enum SendError {
/// Telegram returned 400 with "can't parse entities" (Markdown issue).
ParseEntities(String),
/// Any other failure.
Other(String),
}
impl std::fmt::Display for SendError {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
SendError::ParseEntities(detail) => write!(f, "parse entities error: {}", detail),
SendError::Other(msg) => write!(f, "{}", msg),
}
}
}
/// Send a message via the Telegram Bot API.
///
/// Returns the sent message_id on success. When `parse_mode` is set and
/// Telegram returns a 400 "can't parse entities" error, returns
/// `SendError::ParseEntities` so the caller can retry without formatting.
fn send_message(
chat_id: i64,
text: &str,
reply_to_message_id: i64,
parse_mode: Option<&str>,
) -> Result<i64, SendError> {
let mut payload = serde_json::json!({
"chat_id": chat_id,
"text": text,
"reply_to_message_id": reply_to_message_id,
});
if let Some(mode) = parse_mode {
payload["parse_mode"] = serde_json::Value::String(mode.to_string());
}
let payload_bytes = serde_json::to_vec(&payload)
.map_err(|e| SendError::Other(format!("Failed to serialize payload: {}", e)))?;
let headers = serde_json::json!({ "Content-Type": "application/json" });
let result = channel_host::http_request(
"POST",
"https://api.telegram.org/bot{TELEGRAM_BOT_TOKEN}/sendMessage",
&headers.to_string(),
Some(&payload_bytes),
None,
);
match result {
Ok(http_response) => {
if http_response.status == 400 {
let body_str = String::from_utf8_lossy(&http_response.body);
if body_str.contains("can't parse entities") {
return Err(SendError::ParseEntities(body_str.to_string()));
}
return Err(SendError::Other(format!(
"Telegram API returned 400: {}",
body_str
)));
}
if http_response.status != 200 {
let body_str = String::from_utf8_lossy(&http_response.body);
return Err(SendError::Other(format!(
"Telegram API returned status {}: {}",
http_response.status, body_str
)));
}
let api_response: TelegramApiResponse<SentMessage> =
serde_json::from_slice(&http_response.body)
.map_err(|e| SendError::Other(format!("Failed to parse response: {}", e)))?;
if !api_response.ok {
return Err(SendError::Other(format!(
"Telegram API error: {}",
api_response
.description
.unwrap_or_else(|| "unknown".to_string())
)));
}
Ok(api_response.result.map(|r| r.message_id).unwrap_or(0))
}
Err(e) => Err(SendError::Other(format!("HTTP request failed: {}", e))),
}
}
// ============================================================================
// Webhook Management
// ============================================================================
@@ -604,6 +731,7 @@ fn delete_webhook() -> Result<(), String> {
"https://api.telegram.org/bot{TELEGRAM_BOT_TOKEN}/deleteWebhook",
&headers.to_string(),
None,
None,
);
match result {
@@ -666,6 +794,7 @@ fn register_webhook(tunnel_url: &str, webhook_secret: Option<&str>) -> Result<()
"https://api.telegram.org/bot{TELEGRAM_BOT_TOKEN}/setWebhook",
&headers.to_string(),
Some(&body_bytes),
None,
);
match result {
@@ -700,6 +829,48 @@ fn register_webhook(tunnel_url: &str, webhook_secret: Option<&str>) -> Result<()
}
}
// ============================================================================
// Pairing Reply
// ============================================================================
/// Send a pairing code message to a chat. Used when an unknown user DMs the bot.
fn send_pairing_reply(chat_id: i64, code: &str) -> Result<(), String> {
let payload = serde_json::json!({
"chat_id": chat_id,
"text": format!(
"To pair with this bot, run: `ironclaw pairing approve telegram {}`",
code
),
"parse_mode": "Markdown",
});
let payload_bytes = serde_json::to_vec(&payload)
.map_err(|e| format!("Failed to serialize payload: {}", e))?;
let headers = serde_json::json!({
"Content-Type": "application/json"
});
let result = channel_host::http_request(
"POST",
"https://api.telegram.org/bot{TELEGRAM_BOT_TOKEN}/sendMessage",
&headers.to_string(),
Some(&payload_bytes),
None,
);
match result {
Ok(response) => {
if response.status != 200 {
let body_str = String::from_utf8_lossy(&response.body);
return Err(format!("HTTP {}: {}", response.status, body_str));
}
Ok(())
}
Err(e) => Err(format!("HTTP request failed: {}", e)),
}
}
// ============================================================================
// Update Handling
// ============================================================================
@@ -719,11 +890,16 @@ fn handle_update(update: TelegramUpdate) {
/// Process a single message.
fn handle_message(message: TelegramMessage) {
// Skip messages without text
let text = match message.text {
Some(t) if !t.is_empty() => t,
_ => return,
};
// Use text or caption (for media messages)
let content = message
.text
.filter(|t| !t.is_empty())
.or_else(|| message.caption.filter(|c| !c.is_empty()))
.unwrap_or_default();
if content.is_empty() {
return;
}
// Skip messages without a sender (channel posts)
let from = match message.from {
@@ -736,41 +912,111 @@ fn handle_message(message: TelegramMessage) {
return;
}
// Owner validation: silently drop messages from non-owner users
if let Some(owner_id_str) = channel_host::workspace_read(OWNER_ID_PATH) {
if !owner_id_str.is_empty() {
if let Ok(owner_id) = owner_id_str.parse::<i64>() {
if from.id != owner_id {
channel_host::log(
channel_host::LogLevel::Debug,
&format!(
"Dropping message from non-owner user {} (owner: {})",
from.id, owner_id
),
);
return;
let is_private = message.chat.chat_type == "private";
// Owner validation: when owner_id is set, only that user can message
let owner_configured = channel_host::workspace_read(OWNER_ID_PATH)
.map(|s| !s.is_empty())
.unwrap_or(false);
if owner_configured {
if let Ok(owner_id) = channel_host::workspace_read(OWNER_ID_PATH)
.unwrap()
.parse::<i64>()
{
if from.id != owner_id {
channel_host::log(
channel_host::LogLevel::Debug,
&format!(
"Dropping message from non-owner user {} (owner: {})",
from.id, owner_id
),
);
return;
}
}
} else if is_private {
// No owner_id: apply dm_policy for private chats
let dm_policy = channel_host::workspace_read(DM_POLICY_PATH)
.unwrap_or_else(|| "pairing".to_string());
if dm_policy != "open" {
// Build effective allow list: config allow_from + pairing store
let mut allowed: Vec<String> = channel_host::workspace_read(ALLOW_FROM_PATH)
.and_then(|s| serde_json::from_str(&s).ok())
.unwrap_or_default();
if let Ok(store_allowed) = channel_host::pairing_read_allow_from(CHANNEL_NAME) {
allowed.extend(store_allowed);
}
let id_str = from.id.to_string();
let username_opt = from.username.as_deref();
let is_allowed = allowed.contains(&"*".to_string())
|| allowed.contains(&id_str)
|| username_opt.map_or(false, |u| allowed.contains(&u.to_string()));
if !is_allowed {
if dm_policy == "pairing" {
// Upsert pairing request and send reply
let meta = serde_json::json!({
"chat_id": message.chat.id,
"user_id": from.id,
"username": username_opt,
})
.to_string();
match channel_host::pairing_upsert_request(CHANNEL_NAME, &id_str, &meta) {
Ok(result) => {
channel_host::log(
channel_host::LogLevel::Info,
&format!(
"Pairing request for user {} (chat {}): code {}",
from.id, message.chat.id, result.code
),
);
if result.created {
let _ = send_pairing_reply(message.chat.id, &result.code);
}
}
Err(e) => {
channel_host::log(
channel_host::LogLevel::Error,
&format!("Pairing upsert failed: {}", e),
);
}
}
}
return;
}
}
}
let is_private = message.chat.chat_type == "private";
// For group chats, check if the bot was mentioned
// TODO: Read bot_username from config and check mentions
// For now, process all messages in private chats and groups
// For group chats, only respond if bot was mentioned or respond_to_all is enabled
if !is_private {
// In groups, only respond if there's a bot mention or command
// This is a simplified check - proper implementation would use entities
let has_command = text.starts_with('/');
let has_mention = text.contains('@');
let respond_to_all = channel_host::workspace_read(RESPOND_TO_ALL_GROUP_PATH)
.as_deref()
.unwrap_or("false")
== "true";
if !has_command && !has_mention {
channel_host::log(
channel_host::LogLevel::Debug,
&format!("Ignoring group message without mention: {}", text),
);
return;
if !respond_to_all {
let has_command = content.starts_with('/');
let bot_username = channel_host::workspace_read(BOT_USERNAME_PATH)
.unwrap_or_default();
let has_bot_mention = if bot_username.is_empty() {
content.contains('@')
} else {
let mention = format!("@{}", bot_username);
content.to_lowercase().contains(&mention.to_lowercase())
};
if !has_command && !has_bot_mention {
channel_host::log(
channel_host::LogLevel::Debug,
&format!("Ignoring group message without mention: {}", content),
);
return;
}
}
}
@@ -792,17 +1038,30 @@ fn handle_message(message: TelegramMessage) {
let metadata_json = serde_json::to_string(&metadata).unwrap_or_else(|_| "{}".to_string());
// Clean the message text (strip bot mentions and commands)
let cleaned_text = clean_message_text(&text);
let bot_username = channel_host::workspace_read(BOT_USERNAME_PATH).unwrap_or_default();
let cleaned_text = clean_message_text(
&content,
if bot_username.is_empty() {
None
} else {
Some(bot_username.as_str())
},
);
if cleaned_text.is_empty() {
// For /start with no args, emit placeholder so agent can respond with welcome
let content_to_emit = if cleaned_text.is_empty() && content.trim().starts_with('/') {
"[User started the bot]".to_string()
} else if cleaned_text.is_empty() {
return;
}
} else {
cleaned_text
};
// Emit the message to the agent
channel_host::emit_message(&EmittedMessage {
user_id: from.id.to_string(),
user_name: Some(user_name),
content: cleaned_text,
content: content_to_emit,
thread_id: None, // Telegram doesn't have threads in the same way
metadata_json,
});
@@ -817,7 +1076,8 @@ fn handle_message(message: TelegramMessage) {
}
/// Clean message text by removing bot commands and @mentions at the start.
fn clean_message_text(text: &str) -> String {
/// When bot_username is set, only strips that specific mention; otherwise strips any leading @mention.
fn clean_message_text(text: &str, bot_username: Option<&str>) -> String {
let mut result = text.trim().to_string();
// Remove leading /command
@@ -832,11 +1092,30 @@ fn clean_message_text(text: &str) -> String {
// Remove leading @mention
if result.starts_with('@') {
if let Some(space_idx) = result.find(' ') {
result = result[space_idx..].trim_start().to_string();
if let Some(bot) = bot_username {
let mention = format!("@{}", bot);
let mention_lower = mention.to_lowercase();
let result_lower = result.to_lowercase();
if result_lower.starts_with(&mention_lower) {
let rest = result[mention.len()..].trim_start();
if rest.is_empty() {
return String::new();
}
result = rest.to_string();
} else if let Some(space_idx) = result.find(' ') {
// Different leading @mention - only strip if it's the bot
let first_word = &result[..space_idx];
if first_word.eq_ignore_ascii_case(&mention) {
result = result[space_idx..].trim_start().to_string();
}
}
} else {
// Just a mention with no text
return String::new();
// No bot_username: strip any leading @mention
if let Some(space_idx) = result.find(' ') {
result = result[space_idx..].trim_start().to_string();
} else {
return String::new();
}
}
}
@@ -872,12 +1151,22 @@ mod tests {
#[test]
fn test_clean_message_text() {
assert_eq!(clean_message_text("/start hello"), "hello");
assert_eq!(clean_message_text("@bot hello world"), "hello world");
assert_eq!(clean_message_text("/start"), "");
assert_eq!(clean_message_text("@botname"), "");
assert_eq!(clean_message_text("just text"), "just text");
assert_eq!(clean_message_text(" spaced "), "spaced");
// Without bot_username: strips any leading @mention
assert_eq!(clean_message_text("/start hello", None), "hello");
assert_eq!(clean_message_text("@bot hello world", None), "hello world");
assert_eq!(clean_message_text("/start", None), "");
assert_eq!(clean_message_text("@botname", None), "");
assert_eq!(clean_message_text("just text", None), "just text");
assert_eq!(clean_message_text(" spaced ", None), "spaced");
// With bot_username: only strips @MyBot, not @alice
assert_eq!(clean_message_text("@MyBot hello", Some("MyBot")), "hello");
assert_eq!(clean_message_text("@mybot hi", Some("MyBot")), "hi");
assert_eq!(
clean_message_text("@alice hello", Some("MyBot")),
"@alice hello"
);
assert_eq!(clean_message_text("@MyBot", Some("MyBot")), "");
}
#[test]
@@ -945,4 +1234,17 @@ mod tests {
assert_eq!(from.id, 789);
assert_eq!(from.first_name, "John");
}
#[test]
fn test_parse_message_with_caption() {
let json = r#"{
"message_id": 1,
"from": {"id": 1, "is_bot": false, "first_name": "A"},
"chat": {"id": 1, "type": "private"},
"caption": "What's in this image?"
}"#;
let msg: TelegramMessage = serde_json::from_str(json).unwrap();
assert_eq!(msg.text, None);
assert_eq!(msg.caption.as_deref(), Some("What's in this image?"));
}
}
@@ -1,43 +1 @@
{
"type": "channel",
"name": "telegram",
"description": "Telegram Bot API channel for receiving and responding to Telegram messages",
"capabilities": {
"http": {
"allowlist": [
{ "host": "api.telegram.org", "path_prefix": "/bot" }
],
"credentials": {
"telegram_bot": {
"secret_name": "telegram_bot_token",
"location": { "type": "url_path", "placeholder": "{TELEGRAM_BOT_TOKEN}" },
"host_patterns": ["api.telegram.org"]
}
},
"rate_limit": {
"requests_per_minute": 30,
"requests_per_hour": 1000
}
},
"secrets": {
"allowed_names": ["telegram_*"]
},
"channel": {
"allowed_paths": ["/webhook/telegram"],
"allow_polling": true,
"min_poll_interval_ms": 30000,
"workspace_prefix": "channels/telegram/",
"emit_rate_limit": {
"messages_per_minute": 100,
"messages_per_hour": 5000
}
}
},
"config": {
"bot_username": null,
"owner_id": null,
"respond_to_all_group_messages": false,
"polling_enabled": false,
"poll_interval_ms": 30000
}
}
{"type":"channel","name":"telegram","description":"Telegram Bot API channel for receiving and responding to Telegram messages","capabilities":{"http":{"allowlist":[{"host":"api.telegram.org","path_prefix":"/bot"}],"credentials":{"telegram_bot":{"secret_name":"telegram_bot_token","location":{"type":"url_path","placeholder":"{TELEGRAM_BOT_TOKEN}"},"host_patterns":["api.telegram.org"]}},"rate_limit":{"requests_per_minute":30,"requests_per_hour":1000}},"secrets":{"allowed_names":["telegram_*"]},"channel":{"allowed_paths":["/webhook/telegram"],"allow_polling":true,"min_poll_interval_ms":30000,"workspace_prefix":"channels/telegram/","emit_rate_limit":{"messages_per_minute":100,"messages_per_hour":5000}}},"config":{"bot_username":null,"owner_id":null,"respond_to_all_group_messages":false,"polling_enabled":false,"poll_interval_ms":30000,"dm_policy":"pairing","allow_from":[]}}
Binary file not shown.
+1
View File
@@ -361,6 +361,7 @@ impl Guest for WhatsAppChannel {
&api_url,
&headers.to_string(),
Some(&payload_bytes),
None,
);
match result {
+20
View File
@@ -0,0 +1,20 @@
# Local development only — do NOT use these credentials in production.
services:
postgres:
image: pgvector/pgvector:pg16
ports:
- "5432:5432"
environment:
POSTGRES_DB: ironclaw
POSTGRES_USER: ironclaw
POSTGRES_PASSWORD: ironclaw # dev-only, change for any non-local deployment
volumes:
- pgdata:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U ironclaw"]
interval: 5s
timeout: 3s
retries: 5
volumes:
pgdata:
+36 -3
View File
@@ -246,13 +246,46 @@ Create `my-channel.capabilities.json`:
## Building and Deploying
### Supply Chain Security: No Committed Binaries
**Do not commit compiled WASM binaries.** They are a supply chain risk — the binary in a PR may not match the source. IronClaw builds channels from source:
- `cargo build` automatically builds `telegram.wasm` via `build.rs`
- The built binary is in `.gitignore` and is not committed
- CI should run `cargo build` (or `./scripts/build-all.sh`) to produce releases
**Reproducible build:**
```bash
cargo build --release
```
Prerequisites: `rustup target add wasm32-wasip2`, `cargo install wasm-tools` (optional; fallback copies raw WASM if unavailable).
### Telegram Channel (Manual Build)
```bash
# Add WASM target if needed
rustup target add wasm32-wasip2
# Build Telegram channel
./channels-src/telegram/build.sh
# Install (or use ironclaw onboard to install bundled channel)
mkdir -p ~/.ironclaw/channels
cp channels-src/telegram/telegram.wasm channels-src/telegram/telegram.capabilities.json ~/.ironclaw/channels/
```
**Note**: The main IronClaw binary bundles `telegram.wasm` via `include_bytes!`. When modifying the Telegram channel source, run `./channels-src/telegram/build.sh` **before** building the main crate, so the updated WASM is included.
### Other Channels
```bash
# Build the WASM component
cd channels/my-channel
cargo component build --release
cd channels-src/my-channel
cargo build --release --target wasm32-wasip2
# Deploy to ~/.ironclaw/channels/
cp target/wasm32-wasip1/release/my_channel.wasm ~/.ironclaw/channels/my-channel.wasm
cp target/wasm32-wasip2/release/my_channel.wasm ~/.ironclaw/channels/my-channel.wasm
cp my-channel.capabilities.json ~/.ironclaw/channels/
```
+135
View File
@@ -0,0 +1,135 @@
# Telegram Channel Setup
This guide covers configuring the Telegram channel for IronClaw, including DM pairing for access control.
## Overview
The Telegram channel lets you interact with IronClaw via Telegram DMs and groups. It supports:
- **Webhook mode** (recommended): Instant delivery via tunnel
- **Polling mode**: No tunnel required; ~30s delay
- **DM pairing**: Approve unknown users before they can message the agent
- **Group mentions**: `@YourBot` or `/command` to trigger in groups
## Prerequisites
- IronClaw installed and configured (`ironclaw onboard`)
- A Telegram bot token from [@BotFather](https://t.me/BotFather)
## Quick Start
### 1. Create a Bot
1. Message [@BotFather](https://t.me/BotFather) on Telegram
2. Send `/newbot` and follow the prompts
3. Copy the bot token (e.g., `123456789:ABCdefGHIjklMNOpqrsTUVwxyz`)
### 2. Configure via Setup Wizard
```bash
ironclaw onboard
```
When prompted, enable the Telegram channel and paste your bot token. The wizard will:
- Validate the token
- Optionally configure a webhook secret
- Set up tunnel (if you want webhook mode)
### 3. (Optional) Configure Tunnel for Webhooks
For instant message delivery, expose your agent via a tunnel:
```bash
# ngrok
ngrok http 8080
# Cloudflare
cloudflared tunnel --url http://localhost:8080
```
Set the tunnel URL in settings or via `TUNNEL_URL` env var. Without a tunnel, the channel uses polling (~30s delay).
## DM Pairing
When an unknown user DMs your bot, they receive a pairing code. You must approve them before they can message the agent.
### Flow
1. Unknown user sends a message to your bot
2. Bot replies: `To pair with this bot, run: ironclaw pairing approve telegram ABC12345`
3. You run: `ironclaw pairing approve telegram ABC12345`
4. User is added to the allow list; future messages are delivered
### Commands
```bash
# List pending pairing requests
ironclaw pairing list telegram
# List as JSON
ironclaw pairing list telegram --json
# Approve a user by code
ironclaw pairing approve telegram ABC12345
```
### Configuration
Edit `~/.ironclaw/channels/telegram.capabilities.json` (or the config injected by the host):
| Option | Values | Default | Description |
|--------|--------|---------|-------------|
| `dm_policy` | `open`, `allowlist`, `pairing` | `pairing` | `open` = allow all; `allowlist` = config + approved only; `pairing` = allowlist + send pairing reply to unknown |
| `allow_from` | `["user_id", "username", "*"]` | `[]` | Pre-approved IDs/usernames. `*` allows everyone. |
| `owner_id` | Telegram user ID | `null` | When set, only this user can message (overrides dm_policy) |
| `bot_username` | Bot username (no @) | `null` | Used for mention detection in groups; when set, only strips this mention from messages |
| `respond_to_all_group_messages` | `true`/`false` | `false` | When true, respond to all group messages; when false, only @mentions and /commands |
## Manual Installation
If the channel isn't installed via the wizard:
```bash
# Build the Telegram channel (requires wasm32-wasip2 target)
rustup target add wasm32-wasip2
./channels-src/telegram/build.sh
# Install
mkdir -p ~/.ironclaw/channels
cp channels-src/telegram/telegram.wasm channels-src/telegram/telegram.capabilities.json ~/.ironclaw/channels/
```
## Secrets
The channel expects a secret named `telegram_bot_token`. Configure via:
- **Setup wizard**: Saves to encrypted secrets store
- **Environment**: `TELEGRAM_BOT_TOKEN=your_token`
- **Secrets store**: `ironclaw` CLI (if available)
## Webhook Secret (Optional)
For webhook validation, set `telegram_webhook_secret` in secrets. Telegram will send `X-Telegram-Bot-Api-Secret-Token` with each request; the host validates it before forwarding.
## Troubleshooting
### Messages not delivered
- **Polling mode**: Check logs for `getUpdates` errors. Ensure the bot token is valid.
- **Webhook mode**: Verify tunnel is running and `TUNNEL_URL` is correct. Telegram requires HTTPS.
### Pairing code not received
- Verify the channel can send messages (HTTP allowlist includes `api.telegram.org`)
- Check `dm_policy` is `pairing` (not `allowlist` which blocks without reply)
### Group mentions not working
- Set `bot_username` in config to your bot's username (e.g., `MyIronClawBot`)
- Ensure the message contains `@YourBot` or starts with `/`
### "Connection refused" when starting
- For webhook mode: Start your tunnel before `ironclaw run`
- For polling only: No tunnel needed; ignore tunnel-related warnings
+122
View File
@@ -0,0 +1,122 @@
//! Standalone heartbeat test.
//!
//! Exercises the heartbeat system in isolation: connects to the real
//! database, reads the real HEARTBEAT.md, calls the real LLM, and prints
//! every step so you can see exactly where it breaks.
//!
//! Usage:
//! cargo run --example test_heartbeat
use std::sync::Arc;
use ironclaw::{
agent::HeartbeatRunner,
config::Config,
history::Store,
llm::{SessionConfig, create_llm_provider, create_session_manager},
workspace::Workspace,
};
#[tokio::main]
async fn main() -> anyhow::Result<()> {
// Load .env and set up logging
let _ = dotenvy::dotenv();
tracing_subscriber::fmt()
.with_env_filter("ironclaw=debug")
.init();
println!("=== Heartbeat Integration Test ===\n");
// 1. Load config
let config = Config::from_env()
.await
.map_err(|e| anyhow::anyhow!("Config: {}", e))?;
println!("[1/6] Config loaded");
println!(" heartbeat.enabled = {}", config.heartbeat.enabled);
println!(
" heartbeat.interval_secs = {}",
config.heartbeat.interval_secs
);
println!(
" heartbeat.notify_channel = {:?}",
config.heartbeat.notify_channel
);
println!(
" heartbeat.notify_user = {:?}",
config.heartbeat.notify_user
);
// 2. Connect to database
let store = Store::new(&config.database).await?;
store.run_migrations().await?;
println!("[2/6] Database connected");
// 3. Create workspace
let workspace = Arc::new(Workspace::new("default", store.pool()));
println!("[3/6] Workspace created");
// 4. Read HEARTBEAT.md
let checklist = workspace.heartbeat_checklist().await;
match &checklist {
Ok(Some(content)) => {
let preview: String = content.chars().take(200).collect();
println!("[4/6] HEARTBEAT.md found ({} chars)", content.len());
println!(" Preview: {}...", preview);
}
Ok(None) => {
println!("[4/6] HEARTBEAT.md is None (no file, no seed fallback)");
println!(" Heartbeat will return Skipped.");
}
Err(e) => {
println!("[4/6] HEARTBEAT.md read error: {}", e);
}
}
// Check if the checklist would be considered "effectively empty"
if let Ok(Some(_)) = checklist {
println!(" (Will verify via runner below)");
}
// 5. Create LLM provider
let session = create_session_manager(SessionConfig {
auth_base_url: config.llm.nearai.auth_base_url.clone(),
session_path: config.llm.nearai.session_path.clone(),
..Default::default()
})
.await;
let llm = create_llm_provider(&config.llm, session)?;
println!("[5/6] LLM provider created (model: {})", llm.model_name());
// 6. Run heartbeat check
println!("[6/6] Running check_heartbeat()...\n");
let hb_config = ironclaw::agent::HeartbeatConfig::default();
let runner = HeartbeatRunner::new(hb_config, workspace, llm);
let result = runner.check_heartbeat().await;
println!("=== Result ===\n");
match &result {
ironclaw::agent::HeartbeatResult::Ok => {
println!("HeartbeatResult::Ok");
println!(" LLM responded HEARTBEAT_OK, nothing needs attention.");
}
ironclaw::agent::HeartbeatResult::NeedsAttention(msg) => {
println!("HeartbeatResult::NeedsAttention");
println!(" Message:\n{}", msg);
}
ironclaw::agent::HeartbeatResult::Skipped => {
println!("HeartbeatResult::Skipped");
println!(" No checklist found, or checklist was effectively empty.");
println!(" This means the HEARTBEAT.md either:");
println!(" - Does not exist in the workspace database");
println!(" - Contains only headers, comments, and empty checkboxes");
}
ironclaw::agent::HeartbeatResult::Failed(err) => {
println!("HeartbeatResult::Failed");
println!(" Error: {}", err);
}
}
Ok(())
}
+10
View File
@@ -0,0 +1,10 @@
-- Add project_dir and user_id columns for sandbox job tracking.
-- user_id was previously hardcoded to "default" in the Rust layer;
-- now it's persisted so we can filter per-user.
ALTER TABLE agent_jobs ADD COLUMN IF NOT EXISTS project_dir TEXT;
ALTER TABLE agent_jobs ADD COLUMN IF NOT EXISTS user_id TEXT NOT NULL DEFAULT 'default';
CREATE INDEX IF NOT EXISTS idx_agent_jobs_source ON agent_jobs(source);
CREATE INDEX IF NOT EXISTS idx_agent_jobs_user ON agent_jobs(user_id);
CREATE INDEX IF NOT EXISTS idx_agent_jobs_created ON agent_jobs(created_at DESC);
+14
View File
@@ -0,0 +1,14 @@
-- Track which mode a sandbox job uses (worker vs claude_code).
ALTER TABLE agent_jobs ADD COLUMN IF NOT EXISTS job_mode TEXT NOT NULL DEFAULT 'worker';
-- Persist Claude Code streaming events so they survive restarts and can be
-- loaded when the frontend opens a job detail view after the fact.
CREATE TABLE IF NOT EXISTS claude_code_events (
id BIGSERIAL PRIMARY KEY,
job_id UUID NOT NULL REFERENCES agent_jobs(id),
event_type TEXT NOT NULL,
data JSONB NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX IF NOT EXISTS idx_cc_events_job ON claude_code_events(job_id, id);
+73
View File
@@ -0,0 +1,73 @@
-- Routines: scheduled and reactive job system.
--
-- A routine is a named, persistent, user-owned task with a trigger and an action.
-- Triggers fire independently (cron, event, webhook, manual) so only the
-- relevant routine's prompt hits the LLM, not the whole checklist.
CREATE TABLE routines (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
name TEXT NOT NULL,
description TEXT NOT NULL DEFAULT '',
user_id TEXT NOT NULL,
enabled BOOLEAN NOT NULL DEFAULT true,
-- Trigger definition
trigger_type TEXT NOT NULL, -- 'cron', 'event', 'webhook', 'manual'
trigger_config JSONB NOT NULL, -- type-specific config (schedule, pattern, etc.)
-- Action definition
action_type TEXT NOT NULL, -- 'lightweight', 'full_job'
action_config JSONB NOT NULL, -- prompt, context_paths, max_tokens / title, max_iterations
-- Guardrails
cooldown_secs INTEGER NOT NULL DEFAULT 300,
max_concurrent INTEGER NOT NULL DEFAULT 1,
dedup_window_secs INTEGER, -- NULL = no dedup
-- Notification preferences
notify_channel TEXT, -- NULL = use default
notify_user TEXT NOT NULL DEFAULT 'default',
notify_on_success BOOLEAN NOT NULL DEFAULT false,
notify_on_failure BOOLEAN NOT NULL DEFAULT true,
notify_on_attention BOOLEAN NOT NULL DEFAULT true,
-- Runtime state (updated by engine)
state JSONB NOT NULL DEFAULT '{}',
last_run_at TIMESTAMPTZ,
next_fire_at TIMESTAMPTZ, -- pre-computed for cron triggers
run_count BIGINT NOT NULL DEFAULT 0,
consecutive_failures INTEGER NOT NULL DEFAULT 0,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
UNIQUE (user_id, name)
);
-- Fast lookup: "which cron routines need to fire right now?"
CREATE INDEX idx_routines_next_fire
ON routines (next_fire_at)
WHERE enabled AND next_fire_at IS NOT NULL;
-- Fast lookup: event triggers for a user
CREATE INDEX idx_routines_event_triggers
ON routines (user_id)
WHERE enabled AND trigger_type = 'event';
-- Audit log of individual routine executions.
CREATE TABLE routine_runs (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
routine_id UUID NOT NULL REFERENCES routines(id) ON DELETE CASCADE,
trigger_type TEXT NOT NULL,
trigger_detail TEXT, -- e.g. matched message preview, cron expression
started_at TIMESTAMPTZ NOT NULL DEFAULT now(),
completed_at TIMESTAMPTZ,
status TEXT NOT NULL DEFAULT 'running', -- running, ok, attention, failed
result_summary TEXT,
tokens_used INTEGER,
job_id UUID REFERENCES agent_jobs(id), -- non-NULL for full_job runs
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX idx_routine_runs_routine ON routine_runs (routine_id);
CREATE INDEX idx_routine_runs_status ON routine_runs (status) WHERE status = 'running';
+3
View File
@@ -0,0 +1,3 @@
-- Rename claude_code_events to job_events (generic for all sandbox job types).
ALTER TABLE claude_code_events RENAME TO job_events;
ALTER INDEX idx_cc_events_job RENAME TO idx_job_events_job;
+16
View File
@@ -0,0 +1,16 @@
-- Settings table: key-value store for all user configuration.
--
-- Replaces ~/.ironclaw/settings.json, session.json, and mcp-servers.json.
-- Keys use dotted paths matching the existing Settings.get()/set() convention
-- (e.g., "agent.name", "sandbox.enabled", "mcp_servers").
-- One row per setting so individual values can be updated atomically.
CREATE TABLE IF NOT EXISTS settings (
user_id TEXT NOT NULL,
key TEXT NOT NULL,
value JSONB NOT NULL,
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
PRIMARY KEY (user_id, key)
);
CREATE INDEX IF NOT EXISTS idx_settings_user ON settings (user_id);
+2
View File
@@ -0,0 +1,2 @@
[workspace]
git_release_enable = false
+21
View File
@@ -0,0 +1,21 @@
#!/usr/bin/env bash
# Build IronClaw and all bundled channels.
#
# Run this before release or when channel sources have changed.
# The main binary bundles telegram.wasm via include_bytes!; it must exist.
set -euo pipefail
cd "$(dirname "$0")/.."
echo "Building bundled channels..."
if [ -d "channels-src/telegram" ]; then
./channels-src/telegram/build.sh
fi
echo ""
echo "Building IronClaw..."
cargo build --release
echo ""
echo "Done. Binary: target/release/ironclaw"
+681 -123
View File
File diff suppressed because it is too large Load Diff
+34 -3
View File
@@ -29,7 +29,7 @@ use std::time::Duration;
use tokio::sync::mpsc;
use crate::channels::OutgoingResponse;
use crate::llm::{ChatMessage, CompletionRequest, LlmProvider};
use crate::llm::{ChatMessage, CompletionRequest, FinishReason, LlmProvider};
use crate::workspace::Workspace;
/// Configuration for the heartbeat runner.
@@ -217,9 +217,26 @@ impl HeartbeatRunner {
]
};
// Use the model's context_length to set max_tokens. The API returns
// the total context window; we cap output at half of that (the rest is
// the prompt) with a floor of 4096.
let max_tokens = match self.llm.model_metadata().await {
Ok(meta) => {
let from_api = meta.context_length.map(|ctx| ctx / 2).unwrap_or(4096);
from_api.max(4096)
}
Err(e) => {
tracing::warn!(
"Could not fetch model metadata, using default max_tokens: {}",
e
);
4096
}
};
let request = CompletionRequest::new(messages)
.with_max_tokens(1024)
.with_temperature(0.3); // Lower temperature for more focused responses
.with_max_tokens(max_tokens)
.with_temperature(0.3);
let response = match self.llm.complete(request).await {
Ok(r) => r,
@@ -228,6 +245,20 @@ impl HeartbeatRunner {
let content = response.content.trim();
// Guard against empty content. Reasoning models (e.g. GLM-4.7) may
// burn all output tokens on chain-of-thought and return content: null.
if content.is_empty() {
return if response.finish_reason == FinishReason::Length {
HeartbeatResult::Failed(
"LLM response was truncated (finish_reason=length) with no content. \
The model may have exhausted its token budget on reasoning."
.to_string(),
)
} else {
HeartbeatResult::Failed("LLM returned empty content.".to_string())
};
}
// Check if nothing needs attention
if content == "HEARTBEAT_OK" || content.contains("HEARTBEAT_OK") {
return HeartbeatResult::Ok;
+5
View File
@@ -6,6 +6,7 @@
//! - Tool invocation with safety
//! - Self-repair for stuck jobs
//! - Proactive heartbeat execution
//! - Routine-based scheduled and reactive jobs
//! - Turn-based session management with undo
//! - Context compaction for long conversations
@@ -14,6 +15,8 @@ pub mod compaction;
pub mod context_monitor;
mod heartbeat;
mod router;
pub mod routine;
pub mod routine_engine;
mod scheduler;
mod self_repair;
pub mod session;
@@ -28,6 +31,8 @@ pub use compaction::{CompactionResult, ContextCompactor};
pub use context_monitor::{CompactionStrategy, ContextBreakdown, ContextMonitor};
pub use heartbeat::{HeartbeatConfig, HeartbeatResult, HeartbeatRunner, spawn_heartbeat};
pub use router::{MessageIntent, Router};
pub use routine::{Routine, RoutineAction, RoutineRun, Trigger};
pub use routine_engine::RoutineEngine;
pub use scheduler::Scheduler;
pub use self_repair::{BrokenTool, RepairResult, RepairTask, SelfRepair, StuckJob};
pub use session::{PendingApproval, PendingAuth, Session, Thread, ThreadState, Turn, TurnState};
+509
View File
@@ -0,0 +1,509 @@
//! Core types for the routines system.
//!
//! A routine is a named, persistent, user-owned task with a trigger and an action.
//! Each routine fires independently when its trigger condition is met, with only
//! that routine's prompt and context sent to the LLM.
//!
//! ```text
//! ┌──────────┐ ┌─────────┐ ┌──────────────────┐
//! │ Trigger │────▶│ Engine │────▶│ Execution Mode │
//! │ cron/event│ │guardrail│ │lightweight│full_job│
//! │ webhook │ │ check │ └──────────────────┘
//! │ manual │ └─────────┘ │
//! └──────────┘ ▼
//! ┌──────────────┐
//! │ Notify user │
//! │ if needed │
//! └──────────────┘
//! ```
use std::collections::hash_map::DefaultHasher;
use std::hash::{Hash, Hasher};
use std::str::FromStr;
use std::time::Duration;
use chrono::{DateTime, Utc};
use serde::{Deserialize, Serialize};
use uuid::Uuid;
/// A routine is a named, persistent, user-owned task with a trigger and an action.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct Routine {
pub id: Uuid,
pub name: String,
pub description: String,
pub user_id: String,
pub enabled: bool,
pub trigger: Trigger,
pub action: RoutineAction,
pub guardrails: RoutineGuardrails,
pub notify: NotifyConfig,
// Runtime state (DB-managed)
pub last_run_at: Option<DateTime<Utc>>,
pub next_fire_at: Option<DateTime<Utc>>,
pub run_count: u64,
pub consecutive_failures: u32,
pub state: serde_json::Value,
pub created_at: DateTime<Utc>,
pub updated_at: DateTime<Utc>,
}
/// When a routine should fire.
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(tag = "type", rename_all = "snake_case")]
pub enum Trigger {
/// Fire on a cron schedule (e.g. "0 9 * * MON-FRI" or "every 2h").
Cron { schedule: String },
/// Fire when a channel message matches a pattern.
Event {
/// Optional channel filter (e.g. "telegram", "slack").
channel: Option<String>,
/// Regex pattern to match against message content.
pattern: String,
},
/// Fire on incoming webhook POST to /hooks/routine/{id}.
Webhook {
/// Optional webhook path suffix (defaults to routine id).
path: Option<String>,
/// Optional shared secret for HMAC validation.
secret: Option<String>,
},
/// Only fires via tool call or CLI.
Manual,
}
impl Trigger {
/// The string tag stored in the DB trigger_type column.
pub fn type_tag(&self) -> &'static str {
match self {
Trigger::Cron { .. } => "cron",
Trigger::Event { .. } => "event",
Trigger::Webhook { .. } => "webhook",
Trigger::Manual => "manual",
}
}
/// Parse a trigger from its DB representation.
pub fn from_db(trigger_type: &str, config: serde_json::Value) -> Result<Self, String> {
match trigger_type {
"cron" => {
let schedule = config
.get("schedule")
.and_then(|v| v.as_str())
.ok_or("cron trigger missing 'schedule'")?
.to_string();
Ok(Trigger::Cron { schedule })
}
"event" => {
let pattern = config
.get("pattern")
.and_then(|v| v.as_str())
.ok_or("event trigger missing 'pattern'")?
.to_string();
let channel = config
.get("channel")
.and_then(|v| v.as_str())
.map(String::from);
Ok(Trigger::Event { channel, pattern })
}
"webhook" => {
let path = config
.get("path")
.and_then(|v| v.as_str())
.map(String::from);
let secret = config
.get("secret")
.and_then(|v| v.as_str())
.map(String::from);
Ok(Trigger::Webhook { path, secret })
}
"manual" => Ok(Trigger::Manual),
other => Err(format!("unknown trigger type: {other}")),
}
}
/// Serialize trigger-specific config to JSON for DB storage.
pub fn to_config_json(&self) -> serde_json::Value {
match self {
Trigger::Cron { schedule } => serde_json::json!({ "schedule": schedule }),
Trigger::Event { channel, pattern } => serde_json::json!({
"pattern": pattern,
"channel": channel,
}),
Trigger::Webhook { path, secret } => serde_json::json!({
"path": path,
"secret": secret,
}),
Trigger::Manual => serde_json::json!({}),
}
}
}
/// What happens when a routine fires.
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(tag = "type", rename_all = "snake_case")]
pub enum RoutineAction {
/// Single LLM call, no tools. Cheap and fast.
Lightweight {
/// The prompt sent to the LLM.
prompt: String,
/// Workspace paths to load as context (e.g. ["context/priorities.md"]).
#[serde(default)]
context_paths: Vec<String>,
/// Max output tokens (default: 4096).
#[serde(default = "default_max_tokens")]
max_tokens: u32,
},
/// Full multi-turn worker job with tool access.
FullJob {
/// Job title for the scheduler.
title: String,
/// Job description / initial prompt.
description: String,
/// Max reasoning iterations (default: 10).
#[serde(default = "default_max_iterations")]
max_iterations: u32,
},
}
fn default_max_tokens() -> u32 {
4096
}
fn default_max_iterations() -> u32 {
10
}
impl RoutineAction {
/// The string tag stored in the DB action_type column.
pub fn type_tag(&self) -> &'static str {
match self {
RoutineAction::Lightweight { .. } => "lightweight",
RoutineAction::FullJob { .. } => "full_job",
}
}
/// Parse an action from its DB representation.
pub fn from_db(action_type: &str, config: serde_json::Value) -> Result<Self, String> {
match action_type {
"lightweight" => {
let prompt = config
.get("prompt")
.and_then(|v| v.as_str())
.ok_or("lightweight action missing 'prompt'")?
.to_string();
let context_paths = config
.get("context_paths")
.and_then(|v| v.as_array())
.map(|arr| {
arr.iter()
.filter_map(|v| v.as_str().map(String::from))
.collect()
})
.unwrap_or_default();
let max_tokens = config
.get("max_tokens")
.and_then(|v| v.as_u64())
.unwrap_or(default_max_tokens() as u64) as u32;
Ok(RoutineAction::Lightweight {
prompt,
context_paths,
max_tokens,
})
}
"full_job" => {
let title = config
.get("title")
.and_then(|v| v.as_str())
.ok_or("full_job action missing 'title'")?
.to_string();
let description = config
.get("description")
.and_then(|v| v.as_str())
.ok_or("full_job action missing 'description'")?
.to_string();
let max_iterations = config
.get("max_iterations")
.and_then(|v| v.as_u64())
.unwrap_or(default_max_iterations() as u64)
as u32;
Ok(RoutineAction::FullJob {
title,
description,
max_iterations,
})
}
other => Err(format!("unknown action type: {other}")),
}
}
/// Serialize action config to JSON for DB storage.
pub fn to_config_json(&self) -> serde_json::Value {
match self {
RoutineAction::Lightweight {
prompt,
context_paths,
max_tokens,
} => serde_json::json!({
"prompt": prompt,
"context_paths": context_paths,
"max_tokens": max_tokens,
}),
RoutineAction::FullJob {
title,
description,
max_iterations,
} => serde_json::json!({
"title": title,
"description": description,
"max_iterations": max_iterations,
}),
}
}
}
/// Guardrails to prevent runaway execution.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct RoutineGuardrails {
/// Minimum time between fires.
pub cooldown: Duration,
/// Max simultaneous runs of this routine.
pub max_concurrent: u32,
/// Window for content-hash dedup (event triggers). None = no dedup.
pub dedup_window: Option<Duration>,
}
impl Default for RoutineGuardrails {
fn default() -> Self {
Self {
cooldown: Duration::from_secs(300),
max_concurrent: 1,
dedup_window: None,
}
}
}
/// Notification preferences for a routine.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct NotifyConfig {
/// Channel to notify on (None = default/broadcast all).
pub channel: Option<String>,
/// User to notify.
pub user: String,
/// Notify when routine produces actionable output.
pub on_attention: bool,
/// Notify when routine errors.
pub on_failure: bool,
/// Notify when routine runs with no findings.
pub on_success: bool,
}
impl Default for NotifyConfig {
fn default() -> Self {
Self {
channel: None,
user: "default".to_string(),
on_attention: true,
on_failure: true,
on_success: false,
}
}
}
/// Status of a routine run.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
pub enum RunStatus {
Running,
Ok,
Attention,
Failed,
}
impl std::fmt::Display for RunStatus {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
RunStatus::Running => write!(f, "running"),
RunStatus::Ok => write!(f, "ok"),
RunStatus::Attention => write!(f, "attention"),
RunStatus::Failed => write!(f, "failed"),
}
}
}
impl FromStr for RunStatus {
type Err = String;
fn from_str(s: &str) -> Result<Self, Self::Err> {
match s {
"running" => Ok(RunStatus::Running),
"ok" => Ok(RunStatus::Ok),
"attention" => Ok(RunStatus::Attention),
"failed" => Ok(RunStatus::Failed),
other => Err(format!("unknown run status: {other}")),
}
}
}
/// A single execution of a routine.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct RoutineRun {
pub id: Uuid,
pub routine_id: Uuid,
pub trigger_type: String,
pub trigger_detail: Option<String>,
pub started_at: DateTime<Utc>,
pub completed_at: Option<DateTime<Utc>>,
pub status: RunStatus,
pub result_summary: Option<String>,
pub tokens_used: Option<i32>,
pub job_id: Option<Uuid>,
pub created_at: DateTime<Utc>,
}
/// Compute a content hash for event dedup.
pub fn content_hash(content: &str) -> u64 {
let mut hasher = DefaultHasher::new();
content.hash(&mut hasher);
hasher.finish()
}
/// Parse a cron expression and compute the next fire time from now.
pub fn next_cron_fire(schedule: &str) -> Result<Option<DateTime<Utc>>, String> {
let cron_schedule =
cron::Schedule::from_str(schedule).map_err(|e| format!("invalid cron: {e}"))?;
Ok(cron_schedule.upcoming(Utc).next())
}
#[cfg(test)]
mod tests {
use crate::agent::routine::{
RoutineAction, RoutineGuardrails, RunStatus, Trigger, content_hash, next_cron_fire,
};
#[test]
fn test_trigger_roundtrip() {
let trigger = Trigger::Cron {
schedule: "0 9 * * MON-FRI".to_string(),
};
let json = trigger.to_config_json();
let parsed = Trigger::from_db("cron", json).expect("parse cron");
assert!(matches!(parsed, Trigger::Cron { schedule } if schedule == "0 9 * * MON-FRI"));
}
#[test]
fn test_event_trigger_roundtrip() {
let trigger = Trigger::Event {
channel: Some("telegram".to_string()),
pattern: r"deploy\s+\w+".to_string(),
};
let json = trigger.to_config_json();
let parsed = Trigger::from_db("event", json).expect("parse event");
assert!(matches!(parsed, Trigger::Event { channel, pattern }
if channel == Some("telegram".to_string()) && pattern == r"deploy\s+\w+"));
}
#[test]
fn test_action_lightweight_roundtrip() {
let action = RoutineAction::Lightweight {
prompt: "Check PRs".to_string(),
context_paths: vec!["context/priorities.md".to_string()],
max_tokens: 2048,
};
let json = action.to_config_json();
let parsed = RoutineAction::from_db("lightweight", json).expect("parse lightweight");
assert!(
matches!(parsed, RoutineAction::Lightweight { prompt, context_paths, max_tokens }
if prompt == "Check PRs" && context_paths.len() == 1 && max_tokens == 2048)
);
}
#[test]
fn test_action_full_job_roundtrip() {
let action = RoutineAction::FullJob {
title: "Deploy review".to_string(),
description: "Review and deploy pending changes".to_string(),
max_iterations: 5,
};
let json = action.to_config_json();
let parsed = RoutineAction::from_db("full_job", json).expect("parse full_job");
assert!(
matches!(parsed, RoutineAction::FullJob { title, max_iterations, .. }
if title == "Deploy review" && max_iterations == 5)
);
}
#[test]
fn test_run_status_display_parse() {
for status in [
RunStatus::Running,
RunStatus::Ok,
RunStatus::Attention,
RunStatus::Failed,
] {
let s = status.to_string();
let parsed: RunStatus = s.parse().expect("parse status");
assert_eq!(parsed, status);
}
}
#[test]
fn test_content_hash_deterministic() {
let h1 = content_hash("deploy production");
let h2 = content_hash("deploy production");
assert_eq!(h1, h2);
let h3 = content_hash("deploy staging");
assert_ne!(h1, h3);
}
#[test]
fn test_next_cron_fire_valid() {
// Every minute should always have a next fire
let next = next_cron_fire("* * * * * *").expect("valid cron");
assert!(next.is_some());
}
#[test]
fn test_next_cron_fire_invalid() {
let result = next_cron_fire("not a cron");
assert!(result.is_err());
}
#[test]
fn test_guardrails_default() {
let g = RoutineGuardrails::default();
assert_eq!(g.cooldown.as_secs(), 300);
assert_eq!(g.max_concurrent, 1);
assert!(g.dedup_window.is_none());
}
#[test]
fn test_trigger_type_tag() {
assert_eq!(
Trigger::Cron {
schedule: String::new()
}
.type_tag(),
"cron"
);
assert_eq!(
Trigger::Event {
channel: None,
pattern: String::new()
}
.type_tag(),
"event"
);
assert_eq!(
Trigger::Webhook {
path: None,
secret: None
}
.type_tag(),
"webhook"
);
assert_eq!(Trigger::Manual.type_tag(), "manual");
}
}
+602
View File
@@ -0,0 +1,602 @@
//! Routine execution engine.
//!
//! Handles loading routines, checking triggers, enforcing guardrails,
//! and executing both lightweight (single LLM call) and full-job routines.
//!
//! The engine runs two independent loops:
//! - A **cron ticker** that polls the DB every N seconds for due cron routines
//! - An **event matcher** called synchronously from the agent main loop
//!
//! Lightweight routines execute inline (single LLM call, no scheduler slot).
//! Full-job routines are delegated to the existing `Scheduler`.
use std::sync::Arc;
use std::sync::atomic::{AtomicUsize, Ordering};
use std::time::Duration;
use chrono::Utc;
use regex::Regex;
use tokio::sync::{RwLock, mpsc};
use uuid::Uuid;
use crate::agent::routine::{
NotifyConfig, Routine, RoutineAction, RoutineRun, RunStatus, Trigger, next_cron_fire,
};
use crate::channels::{IncomingMessage, OutgoingResponse};
use crate::config::RoutineConfig;
use crate::db::Database;
use crate::llm::{ChatMessage, CompletionRequest, FinishReason, LlmProvider};
use crate::workspace::Workspace;
/// The routine execution engine.
pub struct RoutineEngine {
config: RoutineConfig,
store: Arc<dyn Database>,
llm: Arc<dyn LlmProvider>,
workspace: Arc<Workspace>,
/// Sender for notifications (routed to channel manager).
notify_tx: mpsc::Sender<OutgoingResponse>,
/// Currently running routine count (across all routines).
running_count: Arc<AtomicUsize>,
/// Compiled event regex cache: routine_id -> compiled regex.
event_cache: Arc<RwLock<Vec<(Uuid, Routine, Regex)>>>,
}
impl RoutineEngine {
pub fn new(
config: RoutineConfig,
store: Arc<dyn Database>,
llm: Arc<dyn LlmProvider>,
workspace: Arc<Workspace>,
notify_tx: mpsc::Sender<OutgoingResponse>,
) -> Self {
Self {
config,
store,
llm,
workspace,
notify_tx,
running_count: Arc::new(AtomicUsize::new(0)),
event_cache: Arc::new(RwLock::new(Vec::new())),
}
}
/// Refresh the in-memory event trigger cache from DB.
pub async fn refresh_event_cache(&self) {
match self.store.list_event_routines().await {
Ok(routines) => {
let mut cache = Vec::new();
for routine in routines {
if let Trigger::Event { ref pattern, .. } = routine.trigger {
match Regex::new(pattern) {
Ok(re) => cache.push((routine.id, routine.clone(), re)),
Err(e) => {
tracing::warn!(
routine = %routine.name,
"Invalid event regex '{}': {}",
pattern, e
);
}
}
}
}
let count = cache.len();
*self.event_cache.write().await = cache;
tracing::debug!("Refreshed event cache: {} routines", count);
}
Err(e) => {
tracing::error!("Failed to refresh event cache: {}", e);
}
}
}
/// Check incoming message against event triggers. Returns number of routines fired.
///
/// Called synchronously from the main loop after handle_message(). The actual
/// execution is spawned async so this returns quickly.
pub async fn check_event_triggers(&self, message: &IncomingMessage) -> usize {
let cache = self.event_cache.read().await;
let mut fired = 0;
for (_, routine, re) in cache.iter() {
// Channel filter
if let Trigger::Event {
channel: Some(ch), ..
} = &routine.trigger
{
if ch != &message.channel {
continue;
}
}
// Regex match
if !re.is_match(&message.content) {
continue;
}
// Cooldown check
if !self.check_cooldown(routine) {
tracing::debug!(routine = %routine.name, "Skipped: cooldown active");
continue;
}
// Concurrent run check
if !self.check_concurrent(routine).await {
tracing::debug!(routine = %routine.name, "Skipped: max concurrent reached");
continue;
}
// Global capacity check
if self.running_count.load(Ordering::Relaxed) >= self.config.max_concurrent_routines {
tracing::warn!(routine = %routine.name, "Skipped: global max concurrent reached");
continue;
}
let detail = truncate(&message.content, 200);
self.spawn_fire(routine.clone(), "event", Some(detail));
fired += 1;
}
fired
}
/// Check all due cron routines and fire them. Called by the cron ticker.
pub async fn check_cron_triggers(&self) {
let routines = match self.store.list_due_cron_routines().await {
Ok(r) => r,
Err(e) => {
tracing::error!("Failed to load due cron routines: {}", e);
return;
}
};
for routine in routines {
if self.running_count.load(Ordering::Relaxed) >= self.config.max_concurrent_routines {
tracing::warn!("Global max concurrent routines reached, skipping remaining");
break;
}
if !self.check_cooldown(&routine) {
continue;
}
if !self.check_concurrent(&routine).await {
continue;
}
let detail = if let Trigger::Cron { ref schedule } = routine.trigger {
Some(schedule.clone())
} else {
None
};
self.spawn_fire(routine, "cron", detail);
}
}
/// Fire a routine manually (from tool call or CLI).
pub async fn fire_manual(&self, routine_id: Uuid) -> Result<Uuid, String> {
let routine = self
.store
.get_routine(routine_id)
.await
.map_err(|e| format!("DB error: {e}"))?
.ok_or_else(|| format!("routine {routine_id} not found"))?;
if !routine.enabled {
return Err(format!("routine '{}' is disabled", routine.name));
}
if !self.check_concurrent(&routine).await {
return Err(format!(
"routine '{}' already at max concurrent runs",
routine.name
));
}
let run_id = Uuid::new_v4();
let run = RoutineRun {
id: run_id,
routine_id: routine.id,
trigger_type: "manual".to_string(),
trigger_detail: None,
started_at: Utc::now(),
completed_at: None,
status: RunStatus::Running,
result_summary: None,
tokens_used: None,
job_id: None,
created_at: Utc::now(),
};
if let Err(e) = self.store.create_routine_run(&run).await {
return Err(format!("failed to create run record: {e}"));
}
// Execute inline for manual triggers (caller wants to wait)
let engine = EngineContext {
store: self.store.clone(),
llm: self.llm.clone(),
workspace: self.workspace.clone(),
notify_tx: self.notify_tx.clone(),
running_count: self.running_count.clone(),
max_lightweight_tokens: self.config.max_lightweight_tokens,
};
tokio::spawn(async move {
execute_routine(engine, routine, run).await;
});
Ok(run_id)
}
/// Spawn a fire in a background task.
fn spawn_fire(&self, routine: Routine, trigger_type: &str, trigger_detail: Option<String>) {
let run = RoutineRun {
id: Uuid::new_v4(),
routine_id: routine.id,
trigger_type: trigger_type.to_string(),
trigger_detail,
started_at: Utc::now(),
completed_at: None,
status: RunStatus::Running,
result_summary: None,
tokens_used: None,
job_id: None,
created_at: Utc::now(),
};
let engine = EngineContext {
store: self.store.clone(),
llm: self.llm.clone(),
workspace: self.workspace.clone(),
notify_tx: self.notify_tx.clone(),
running_count: self.running_count.clone(),
max_lightweight_tokens: self.config.max_lightweight_tokens,
};
// Record the run in DB, then spawn execution
let store = self.store.clone();
tokio::spawn(async move {
if let Err(e) = store.create_routine_run(&run).await {
tracing::error!(routine = %routine.name, "Failed to record run: {}", e);
return;
}
execute_routine(engine, routine, run).await;
});
}
fn check_cooldown(&self, routine: &Routine) -> bool {
if let Some(last_run) = routine.last_run_at {
let elapsed = Utc::now().signed_duration_since(last_run);
let cooldown = chrono::Duration::from_std(routine.guardrails.cooldown)
.unwrap_or(chrono::Duration::seconds(300));
if elapsed < cooldown {
return false;
}
}
true
}
async fn check_concurrent(&self, routine: &Routine) -> bool {
match self.store.count_running_routine_runs(routine.id).await {
Ok(count) => count < routine.guardrails.max_concurrent as i64,
Err(e) => {
tracing::error!(
routine = %routine.name,
"Failed to check concurrent runs: {}", e
);
false
}
}
}
}
/// Shared context passed to the execution function.
struct EngineContext {
store: Arc<dyn Database>,
llm: Arc<dyn LlmProvider>,
workspace: Arc<Workspace>,
notify_tx: mpsc::Sender<OutgoingResponse>,
running_count: Arc<AtomicUsize>,
max_lightweight_tokens: u32,
}
/// Execute a routine run. Handles both lightweight and full_job modes.
async fn execute_routine(ctx: EngineContext, routine: Routine, run: RoutineRun) {
// Increment running count (atomic: survives panics in the execution below)
ctx.running_count.fetch_add(1, Ordering::Relaxed);
let result = match &routine.action {
RoutineAction::Lightweight {
prompt,
context_paths,
max_tokens,
} => execute_lightweight(&ctx, &routine, prompt, context_paths, *max_tokens).await,
RoutineAction::FullJob { description, .. } => {
// Full job mode: for now, execute as lightweight with the description
// as prompt. Full scheduler integration will come as a follow-up.
tracing::info!(
routine = %routine.name,
"FullJob mode executing as lightweight (scheduler integration pending)"
);
execute_lightweight(&ctx, &routine, description, &[], ctx.max_lightweight_tokens).await
}
};
// Decrement running count
ctx.running_count.fetch_sub(1, Ordering::Relaxed);
// Process result
let (status, summary, tokens) = match result {
Ok(execution) => execution,
Err(e) => {
tracing::error!(routine = %routine.name, "Execution failed: {}", e);
(RunStatus::Failed, Some(e), None)
}
};
// Complete the run record
if let Err(e) = ctx
.store
.complete_routine_run(run.id, status, summary.as_deref(), tokens)
.await
{
tracing::error!(routine = %routine.name, "Failed to complete run record: {}", e);
}
// Update routine runtime state
let now = Utc::now();
let next_fire = if let Trigger::Cron { ref schedule } = routine.trigger {
next_cron_fire(schedule).unwrap_or(None)
} else {
None
};
let new_failures = if status == RunStatus::Failed {
routine.consecutive_failures + 1
} else {
0
};
if let Err(e) = ctx
.store
.update_routine_runtime(
routine.id,
now,
next_fire,
routine.run_count + 1,
new_failures,
&routine.state,
)
.await
{
tracing::error!(routine = %routine.name, "Failed to update runtime state: {}", e);
}
// Send notifications based on config
send_notification(
&ctx.notify_tx,
&routine.notify,
&routine.name,
status,
summary.as_deref(),
)
.await;
}
/// Execute a lightweight routine (single LLM call).
async fn execute_lightweight(
ctx: &EngineContext,
routine: &Routine,
prompt: &str,
context_paths: &[String],
max_tokens: u32,
) -> Result<(RunStatus, Option<String>, Option<i32>), String> {
// Load context from workspace
let mut context_parts = Vec::new();
for path in context_paths {
match ctx.workspace.read(path).await {
Ok(doc) => {
context_parts.push(format!("## {}\n\n{}", path, doc.content));
}
Err(e) => {
tracing::debug!(
routine = %routine.name,
"Failed to read context path {}: {}", path, e
);
}
}
}
// Load routine state from workspace
let state_path = format!("routines/{}/state.md", routine.name);
let state_content = match ctx.workspace.read(&state_path).await {
Ok(doc) => Some(doc.content),
Err(_) => None,
};
// Build the prompt
let mut full_prompt = String::new();
full_prompt.push_str(prompt);
if !context_parts.is_empty() {
full_prompt.push_str("\n\n---\n\n# Context\n\n");
full_prompt.push_str(&context_parts.join("\n\n"));
}
if let Some(state) = &state_content {
full_prompt.push_str("\n\n---\n\n# Previous State\n\n");
full_prompt.push_str(state);
}
full_prompt.push_str(
"\n\n---\n\nIf nothing needs attention, reply EXACTLY with: ROUTINE_OK\n\
If something needs attention, provide a concise summary.",
);
// Get system prompt
let system_prompt = match ctx.workspace.system_prompt().await {
Ok(p) => p,
Err(e) => {
tracing::warn!(routine = %routine.name, "Failed to get system prompt: {}", e);
String::new()
}
};
let messages = if system_prompt.is_empty() {
vec![ChatMessage::user(&full_prompt)]
} else {
vec![
ChatMessage::system(&system_prompt),
ChatMessage::user(&full_prompt),
]
};
// Determine max_tokens from model metadata with fallback
let effective_max_tokens = match ctx.llm.model_metadata().await {
Ok(meta) => {
let from_api = meta.context_length.map(|ctx| ctx / 2).unwrap_or(max_tokens);
from_api.max(max_tokens)
}
Err(_) => max_tokens,
};
let request = CompletionRequest::new(messages)
.with_max_tokens(effective_max_tokens)
.with_temperature(0.3);
let response = ctx
.llm
.complete(request)
.await
.map_err(|e| format!("LLM call failed: {e}"))?;
let content = response.content.trim();
let tokens_used = Some((response.input_tokens + response.output_tokens) as i32);
// Empty content guard (same as heartbeat)
if content.is_empty() {
return if response.finish_reason == FinishReason::Length {
Err(
"LLM response truncated (finish_reason=length) with no content. \
Model may have exhausted token budget on reasoning."
.to_string(),
)
} else {
Err("LLM returned empty content.".to_string())
};
}
// Check for the "nothing to do" sentinel
if content == "ROUTINE_OK" || content.contains("ROUTINE_OK") {
return Ok((RunStatus::Ok, None, tokens_used));
}
Ok((RunStatus::Attention, Some(content.to_string()), tokens_used))
}
/// Send a notification based on the routine's notify config and run status.
async fn send_notification(
tx: &mpsc::Sender<OutgoingResponse>,
notify: &NotifyConfig,
routine_name: &str,
status: RunStatus,
summary: Option<&str>,
) {
let should_notify = match status {
RunStatus::Ok => notify.on_success,
RunStatus::Attention => notify.on_attention,
RunStatus::Failed => notify.on_failure,
RunStatus::Running => false,
};
if !should_notify {
return;
}
let icon = match status {
RunStatus::Ok => "",
RunStatus::Attention => "🔔",
RunStatus::Failed => "",
RunStatus::Running => "",
};
let message = match summary {
Some(s) => format!("{} *Routine '{}'*: {}\n\n{}", icon, routine_name, status, s),
None => format!("{} *Routine '{}'*: {}", icon, routine_name, status),
};
let response = OutgoingResponse {
content: message,
thread_id: None,
metadata: serde_json::json!({
"source": "routine",
"routine_name": routine_name,
"status": status.to_string(),
}),
};
if let Err(e) = tx.send(response).await {
tracing::error!(routine = %routine_name, "Failed to send notification: {}", e);
}
}
/// Spawn the cron ticker background task.
pub fn spawn_cron_ticker(
engine: Arc<RoutineEngine>,
interval: Duration,
) -> tokio::task::JoinHandle<()> {
tokio::spawn(async move {
let mut ticker = tokio::time::interval(interval);
// Skip immediate first tick
ticker.tick().await;
loop {
ticker.tick().await;
engine.check_cron_triggers().await;
}
})
}
fn truncate(s: &str, max: usize) -> String {
if s.len() <= max {
s.to_string()
} else {
let end = crate::util::floor_char_boundary(s, max);
format!("{}...", &s[..end])
}
}
#[cfg(test)]
mod tests {
use crate::agent::routine::{NotifyConfig, RunStatus};
#[test]
fn test_notification_gating() {
let config = NotifyConfig {
on_success: false,
on_failure: true,
on_attention: true,
..Default::default()
};
// on_success = false means Ok status should not notify
assert!(!config.on_success);
assert!(config.on_failure);
assert!(config.on_attention);
}
#[test]
fn test_run_status_icons() {
// Just verify the mapping doesn't panic
for status in [
RunStatus::Ok,
RunStatus::Attention,
RunStatus::Failed,
RunStatus::Running,
] {
let _ = status.to_string();
}
}
}
+73 -73
View File
@@ -12,8 +12,8 @@ use crate::agent::task::{Task, TaskContext, TaskOutput};
use crate::agent::worker::{Worker, WorkerDeps};
use crate::config::AgentConfig;
use crate::context::{ContextManager, JobContext, JobState};
use crate::db::Database;
use crate::error::{Error, JobError};
use crate::history::Store;
use crate::llm::LlmProvider;
use crate::safety::SafetyLayer;
use crate::tools::ToolRegistry;
@@ -48,7 +48,7 @@ pub struct Scheduler {
llm: Arc<dyn LlmProvider>,
safety: Arc<SafetyLayer>,
tools: Arc<ToolRegistry>,
store: Option<Arc<Store>>,
store: Option<Arc<dyn Database>>,
/// Running jobs (main LLM-driven jobs).
jobs: Arc<RwLock<HashMap<Uuid, ScheduledJob>>>,
/// Running sub-tasks (tool executions, background tasks).
@@ -63,7 +63,7 @@ impl Scheduler {
llm: Arc<dyn LlmProvider>,
safety: Arc<SafetyLayer>,
tools: Arc<ToolRegistry>,
store: Option<Arc<Store>>,
store: Option<Arc<dyn Database>>,
) -> Self {
Self {
config,
@@ -79,63 +79,63 @@ impl Scheduler {
/// Schedule a job for execution.
pub async fn schedule(&self, job_id: Uuid) -> Result<(), JobError> {
// Check if already scheduled
if self.jobs.read().await.contains_key(&job_id) {
return Ok(());
}
// Hold write lock for the entire check-insert sequence to prevent
// TOCTOU races where two concurrent calls both pass the checks.
{
let mut jobs = self.jobs.write().await;
// Check capacity
let current_count = self.jobs.read().await.len();
if current_count >= self.config.max_parallel_jobs {
return Err(JobError::MaxJobsExceeded {
max: self.config.max_parallel_jobs,
});
}
// Transition job to in_progress
self.context_manager
.update_context(job_id, |ctx| {
ctx.transition_to(
JobState::InProgress,
Some("Scheduled for execution".to_string()),
)
})
.await?
.map_err(|s| JobError::ContextError {
id: job_id,
reason: s,
})?;
// Create worker channel
let (tx, rx) = mpsc::channel(16);
// Create worker with shared dependencies
let deps = WorkerDeps {
context_manager: self.context_manager.clone(),
llm: self.llm.clone(),
safety: self.safety.clone(),
tools: self.tools.clone(),
store: self.store.clone(),
timeout: self.config.job_timeout,
use_planning: self.config.use_planning,
};
let worker = Worker::new(job_id, deps);
// Spawn worker task
let handle = tokio::spawn(async move {
if let Err(e) = worker.run(rx).await {
tracing::error!("Worker for job {} failed: {}", job_id, e);
if jobs.contains_key(&job_id) {
return Ok(());
}
});
// Start the worker
let _ = tx.send(WorkerMessage::Start).await;
if jobs.len() >= self.config.max_parallel_jobs {
return Err(JobError::MaxJobsExceeded {
max: self.config.max_parallel_jobs,
});
}
// Store the scheduled job
self.jobs
.write()
.await
.insert(job_id, ScheduledJob { handle, tx });
// Transition job to in_progress
self.context_manager
.update_context(job_id, |ctx| {
ctx.transition_to(
JobState::InProgress,
Some("Scheduled for execution".to_string()),
)
})
.await?
.map_err(|s| JobError::ContextError {
id: job_id,
reason: s,
})?;
// Create worker channel
let (tx, rx) = mpsc::channel(16);
// Create worker with shared dependencies
let deps = WorkerDeps {
context_manager: self.context_manager.clone(),
llm: self.llm.clone(),
safety: self.safety.clone(),
tools: self.tools.clone(),
store: self.store.clone(),
timeout: self.config.job_timeout,
use_planning: self.config.use_planning,
};
let worker = Worker::new(job_id, deps);
// Spawn worker task
let handle = tokio::spawn(async move {
if let Err(e) = worker.run(rx).await {
tracing::error!("Worker for job {} failed: {}", job_id, e);
}
});
// Start the worker
let _ = tx.send(WorkerMessage::Start).await;
// Insert while still holding the write lock
jobs.insert(job_id, ScheduledJob { handle, tx });
}
// Cleanup task for this job to avoid capacity leaks
let jobs = Arc::clone(&self.jobs);
@@ -373,23 +373,23 @@ impl Scheduler {
.into());
}
// Execute with timeout
let result = tokio::time::timeout(Duration::from_secs(60), async {
tool.execute(params, &job_ctx).await
})
.await
.map_err(|_| {
Error::Tool(crate::error::ToolError::Timeout {
name: tool_name.to_string(),
timeout: Duration::from_secs(60),
})
})?
.map_err(|e| {
Error::Tool(crate::error::ToolError::ExecutionFailed {
name: tool_name.to_string(),
reason: e.to_string(),
})
})?;
// Execute with per-tool timeout
let tool_timeout = tool.execution_timeout();
let result =
tokio::time::timeout(tool_timeout, async { tool.execute(params, &job_ctx).await })
.await
.map_err(|_| {
Error::Tool(crate::error::ToolError::Timeout {
name: tool_name.to_string(),
timeout: tool_timeout,
})
})?
.map_err(|e| {
Error::Tool(crate::error::ToolError::ExecutionFailed {
name: tool_name.to_string(),
reason: e.to_string(),
})
})?;
Ok(TaskOutput::new(result.result, start.elapsed()))
}
+3 -3
View File
@@ -8,8 +8,8 @@ use chrono::{DateTime, Utc};
use uuid::Uuid;
use crate::context::{ContextManager, JobState};
use crate::db::Database;
use crate::error::RepairError;
use crate::history::Store;
use crate::tools::{BuildRequirement, Language, SoftwareBuilder, SoftwareType, ToolRegistry};
/// A job that has been detected as stuck.
@@ -69,7 +69,7 @@ pub struct DefaultSelfRepair {
#[allow(dead_code)] // Will be used for time-based stuck detection
stuck_threshold: Duration,
max_repair_attempts: u32,
store: Option<Arc<Store>>,
store: Option<Arc<dyn Database>>,
builder: Option<Arc<dyn SoftwareBuilder>>,
#[allow(dead_code)] // Will be used for tool hot-reload after repair
tools: Option<Arc<ToolRegistry>>,
@@ -94,7 +94,7 @@ impl DefaultSelfRepair {
/// Add a Store for tool failure tracking.
#[allow(dead_code)] // Public API for configuring repair with persistence
pub fn with_store(mut self, store: Arc<Store>) -> Self {
pub fn with_store(mut self, store: Arc<dyn Database>) -> Self {
self.store = Some(store);
self
}
+404
View File
@@ -173,6 +173,10 @@ pub struct Thread {
/// Pending auth token request (thread is in auth mode).
#[serde(default)]
pub pending_auth: Option<PendingAuth>,
/// Last NEAR AI response ID for response chaining. Persisted to DB
/// metadata so we can resume chaining across restarts.
#[serde(default)]
pub last_response_id: Option<String>,
}
impl Thread {
@@ -189,6 +193,24 @@ impl Thread {
metadata: serde_json::Value::Null,
pending_approval: None,
pending_auth: None,
last_response_id: None,
}
}
/// Create a thread with a specific ID (for DB hydration).
pub fn with_id(id: Uuid, session_id: Uuid) -> Self {
let now = Utc::now();
Self {
id,
session_id,
state: ThreadState::Idle,
turns: Vec::new(),
created_at: now,
updated_at: now,
metadata: serde_json::Value::Null,
pending_approval: None,
pending_auth: None,
last_response_id: None,
}
}
@@ -593,4 +615,386 @@ mod tests {
let restored: Thread = serde_json::from_str(&json).expect("should deserialize");
assert!(restored.pending_auth.is_none());
}
#[test]
fn test_thread_with_id() {
let specific_id = Uuid::new_v4();
let session_id = Uuid::new_v4();
let thread = Thread::with_id(specific_id, session_id);
assert_eq!(thread.id, specific_id);
assert_eq!(thread.session_id, session_id);
assert_eq!(thread.state, ThreadState::Idle);
assert!(thread.turns.is_empty());
}
#[test]
fn test_thread_with_id_restore_messages() {
let thread_id = Uuid::new_v4();
let session_id = Uuid::new_v4();
let mut thread = Thread::with_id(thread_id, session_id);
let messages = vec![
ChatMessage::user("Hello from DB"),
ChatMessage::assistant("Restored response"),
];
thread.restore_from_messages(messages);
assert_eq!(thread.id, thread_id);
assert_eq!(thread.turns.len(), 1);
assert_eq!(thread.turns[0].user_input, "Hello from DB");
assert_eq!(
thread.turns[0].response,
Some("Restored response".to_string())
);
}
#[test]
fn test_restore_from_messages_empty() {
let mut thread = Thread::new(Uuid::new_v4());
// Add a turn first, then restore with empty vec
thread.start_turn("hello");
thread.complete_turn("hi");
assert_eq!(thread.turns.len(), 1);
thread.restore_from_messages(Vec::new());
// Should clear all turns and stay idle
assert!(thread.turns.is_empty());
assert_eq!(thread.state, ThreadState::Idle);
}
#[test]
fn test_restore_from_messages_only_assistant_messages() {
let mut thread = Thread::new(Uuid::new_v4());
// Only assistant messages (no user messages to anchor turns)
let messages = vec![
ChatMessage::assistant("I'm here"),
ChatMessage::assistant("Still here"),
];
thread.restore_from_messages(messages);
// Assistant-only messages have no user turn to attach to, so
// they should be skipped entirely.
assert!(thread.turns.is_empty());
}
#[test]
fn test_restore_from_messages_multiple_user_messages_in_a_row() {
let mut thread = Thread::new(Uuid::new_v4());
// Two user messages with no assistant response between them
let messages = vec![
ChatMessage::user("first"),
ChatMessage::user("second"),
ChatMessage::assistant("reply to second"),
];
thread.restore_from_messages(messages);
// First user message becomes a turn with no response,
// second user message pairs with the assistant response.
assert_eq!(thread.turns.len(), 2);
assert_eq!(thread.turns[0].user_input, "first");
assert!(thread.turns[0].response.is_none());
assert_eq!(thread.turns[1].user_input, "second");
assert_eq!(
thread.turns[1].response,
Some("reply to second".to_string())
);
}
#[test]
fn test_thread_switch() {
let mut session = Session::new("user-1");
let t1_id = session.create_thread().id;
let t2_id = session.create_thread().id;
// After creating two threads, active should be the last one
assert_eq!(session.active_thread, Some(t2_id));
// Switch back to the first
assert!(session.switch_thread(t1_id));
assert_eq!(session.active_thread, Some(t1_id));
// Switching to a nonexistent thread should fail
let fake_id = Uuid::new_v4();
assert!(!session.switch_thread(fake_id));
// Active thread should remain unchanged
assert_eq!(session.active_thread, Some(t1_id));
}
#[test]
fn test_get_or_create_thread_idempotent() {
let mut session = Session::new("user-1");
let tid1 = session.get_or_create_thread().id;
let tid2 = session.get_or_create_thread().id;
// Should return the same thread (not create a new one each time)
assert_eq!(tid1, tid2);
assert_eq!(session.threads.len(), 1);
}
#[test]
fn test_truncate_turns() {
let mut thread = Thread::new(Uuid::new_v4());
for i in 0..5 {
thread.start_turn(format!("msg-{}", i));
thread.complete_turn(format!("resp-{}", i));
}
assert_eq!(thread.turns.len(), 5);
thread.truncate_turns(3);
assert_eq!(thread.turns.len(), 3);
// Should keep the most recent turns
assert_eq!(thread.turns[0].user_input, "msg-2");
assert_eq!(thread.turns[1].user_input, "msg-3");
assert_eq!(thread.turns[2].user_input, "msg-4");
// Turn numbers should be re-indexed
assert_eq!(thread.turns[0].turn_number, 0);
assert_eq!(thread.turns[1].turn_number, 1);
assert_eq!(thread.turns[2].turn_number, 2);
}
#[test]
fn test_truncate_turns_noop_when_fewer() {
let mut thread = Thread::new(Uuid::new_v4());
thread.start_turn("only one");
thread.complete_turn("response");
thread.truncate_turns(10);
assert_eq!(thread.turns.len(), 1);
assert_eq!(thread.turns[0].user_input, "only one");
}
#[test]
fn test_thread_interrupt_and_resume() {
let mut thread = Thread::new(Uuid::new_v4());
thread.start_turn("do something");
assert_eq!(thread.state, ThreadState::Processing);
thread.interrupt();
assert_eq!(thread.state, ThreadState::Interrupted);
let last_turn = thread.last_turn().unwrap();
assert_eq!(last_turn.state, TurnState::Interrupted);
assert!(last_turn.completed_at.is_some());
thread.resume();
assert_eq!(thread.state, ThreadState::Idle);
}
#[test]
fn test_resume_only_from_interrupted() {
let mut thread = Thread::new(Uuid::new_v4());
// Idle thread: resume should be a no-op
assert_eq!(thread.state, ThreadState::Idle);
thread.resume();
assert_eq!(thread.state, ThreadState::Idle);
// Processing thread: resume should not change state
thread.start_turn("work");
assert_eq!(thread.state, ThreadState::Processing);
thread.resume();
assert_eq!(thread.state, ThreadState::Processing);
}
#[test]
fn test_turn_fail() {
let mut thread = Thread::new(Uuid::new_v4());
thread.start_turn("risky operation");
thread.fail_turn("connection timed out");
assert_eq!(thread.state, ThreadState::Idle);
let turn = thread.last_turn().unwrap();
assert_eq!(turn.state, TurnState::Failed);
assert_eq!(turn.error, Some("connection timed out".to_string()));
assert!(turn.response.is_none());
assert!(turn.completed_at.is_some());
}
#[test]
fn test_messages_with_incomplete_last_turn() {
let mut thread = Thread::new(Uuid::new_v4());
thread.start_turn("first");
thread.complete_turn("first reply");
thread.start_turn("second (in progress)");
let messages = thread.messages();
// Should have 3 messages: user, assistant, user (no assistant for in-progress)
assert_eq!(messages.len(), 3);
assert_eq!(messages[0].content, "first");
assert_eq!(messages[1].content, "first reply");
assert_eq!(messages[2].content, "second (in progress)");
}
#[test]
fn test_thread_serialization_round_trip() {
let mut thread = Thread::new(Uuid::new_v4());
thread.start_turn("hello");
thread.complete_turn("world");
thread.last_response_id = Some("resp_abc123".to_string());
let json = serde_json::to_string(&thread).unwrap();
let restored: Thread = serde_json::from_str(&json).unwrap();
assert_eq!(restored.id, thread.id);
assert_eq!(restored.session_id, thread.session_id);
assert_eq!(restored.turns.len(), 1);
assert_eq!(restored.turns[0].user_input, "hello");
assert_eq!(restored.turns[0].response, Some("world".to_string()));
assert_eq!(restored.last_response_id, Some("resp_abc123".to_string()));
}
#[test]
fn test_session_serialization_round_trip() {
let mut session = Session::new("user-ser");
session.create_thread();
session.auto_approve_tool("echo");
let json = serde_json::to_string(&session).unwrap();
let restored: Session = serde_json::from_str(&json).unwrap();
assert_eq!(restored.user_id, "user-ser");
assert_eq!(restored.threads.len(), 1);
assert!(restored.is_tool_auto_approved("echo"));
assert!(!restored.is_tool_auto_approved("shell"));
}
#[test]
fn test_auto_approved_tools() {
let mut session = Session::new("user-1");
assert!(!session.is_tool_auto_approved("shell"));
session.auto_approve_tool("shell");
assert!(session.is_tool_auto_approved("shell"));
// Idempotent
session.auto_approve_tool("shell");
assert_eq!(session.auto_approved_tools.len(), 1);
}
#[test]
fn test_turn_tool_call_error() {
let mut turn = Turn::new(0, "test");
turn.record_tool_call("http", serde_json::json!({"url": "example.com"}));
turn.record_tool_error("timeout");
assert_eq!(turn.tool_calls.len(), 1);
assert_eq!(turn.tool_calls[0].error, Some("timeout".to_string()));
assert!(turn.tool_calls[0].result.is_none());
}
#[test]
fn test_turn_number_increments() {
let mut thread = Thread::new(Uuid::new_v4());
// Before any turns, turn_number() is 1 (1-indexed for display)
assert_eq!(thread.turn_number(), 1);
thread.start_turn("first");
thread.complete_turn("done");
assert_eq!(thread.turn_number(), 2);
thread.start_turn("second");
assert_eq!(thread.turn_number(), 3);
}
#[test]
fn test_complete_turn_on_empty_thread() {
let mut thread = Thread::new(Uuid::new_v4());
// Completing a turn when there are no turns should be a safe no-op
thread.complete_turn("phantom response");
assert_eq!(thread.state, ThreadState::Idle);
assert!(thread.turns.is_empty());
}
#[test]
fn test_fail_turn_on_empty_thread() {
let mut thread = Thread::new(Uuid::new_v4());
// Failing a turn when there are no turns should be a safe no-op
thread.fail_turn("phantom error");
assert_eq!(thread.state, ThreadState::Idle);
assert!(thread.turns.is_empty());
}
#[test]
fn test_pending_approval_flow() {
let mut thread = Thread::new(Uuid::new_v4());
let approval = PendingApproval {
request_id: Uuid::new_v4(),
tool_name: "shell".to_string(),
parameters: serde_json::json!({"command": "rm -rf /"}),
description: "dangerous command".to_string(),
tool_call_id: "call_123".to_string(),
context_messages: vec![ChatMessage::user("do it")],
};
thread.await_approval(approval);
assert_eq!(thread.state, ThreadState::AwaitingApproval);
assert!(thread.pending_approval.is_some());
let taken = thread.take_pending_approval();
assert!(taken.is_some());
assert_eq!(taken.unwrap().tool_name, "shell");
assert!(thread.pending_approval.is_none());
}
#[test]
fn test_clear_pending_approval() {
let mut thread = Thread::new(Uuid::new_v4());
let approval = PendingApproval {
request_id: Uuid::new_v4(),
tool_name: "http".to_string(),
parameters: serde_json::json!({}),
description: "test".to_string(),
tool_call_id: "call_456".to_string(),
context_messages: vec![],
};
thread.await_approval(approval);
thread.clear_pending_approval();
assert_eq!(thread.state, ThreadState::Idle);
assert!(thread.pending_approval.is_none());
}
#[test]
fn test_active_thread_accessors() {
let mut session = Session::new("user-1");
assert!(session.active_thread().is_none());
assert!(session.active_thread_mut().is_none());
let tid = session.create_thread().id;
assert!(session.active_thread().is_some());
assert_eq!(session.active_thread().unwrap().id, tid);
// Mutably modify through accessor
session.active_thread_mut().unwrap().start_turn("test");
assert_eq!(
session.active_thread().unwrap().state,
ThreadState::Processing
);
}
}
+375
View File
@@ -110,6 +110,41 @@ impl SessionManager {
(session, thread_id)
}
/// Register a hydrated thread so subsequent `resolve_thread` calls find it.
///
/// Inserts into the thread_map and creates an undo manager for the thread.
pub async fn register_thread(
&self,
user_id: &str,
channel: &str,
thread_id: Uuid,
session: Arc<Mutex<Session>>,
) {
let key = ThreadKey {
user_id: user_id.to_string(),
channel: channel.to_string(),
external_thread_id: Some(thread_id.to_string()),
};
{
let mut thread_map = self.thread_map.write().await;
thread_map.insert(key, thread_id);
}
{
let mut undo_managers = self.undo_managers.write().await;
undo_managers
.entry(thread_id)
.or_insert_with(|| Arc::new(Mutex::new(UndoManager::new())));
}
// Ensure the session is tracked
{
let mut sessions = self.sessions.write().await;
sessions.entry(user_id.to_string()).or_insert(session);
}
}
/// Get undo manager for a thread.
pub async fn get_undo_manager(&self, thread_id: Uuid) -> Arc<Mutex<UndoManager>> {
// Fast path
@@ -296,4 +331,344 @@ mod tests {
.await;
assert_eq!(pruned, 0);
}
#[tokio::test]
async fn test_register_thread() {
use crate::agent::session::{Session, Thread};
let manager = SessionManager::new();
let thread_id = Uuid::new_v4();
// Create a session with a hydrated thread
let session = Arc::new(Mutex::new(Session::new("user-hydrate")));
{
let mut sess = session.lock().await;
let thread = Thread::with_id(thread_id, sess.id);
sess.threads.insert(thread_id, thread);
sess.active_thread = Some(thread_id);
}
// Register the thread
manager
.register_thread("user-hydrate", "gateway", thread_id, Arc::clone(&session))
.await;
// resolve_thread should find it (using the UUID as external_thread_id)
let (resolved_session, resolved_tid) = manager
.resolve_thread("user-hydrate", "gateway", Some(&thread_id.to_string()))
.await;
assert_eq!(resolved_tid, thread_id);
// Should be the same session object
let sess = resolved_session.lock().await;
assert!(sess.threads.contains_key(&thread_id));
}
#[tokio::test]
async fn test_resolve_thread_with_explicit_external_id() {
let manager = SessionManager::new();
// Two calls with the same explicit external thread ID should resolve
// to the same internal thread.
let (_, t1) = manager
.resolve_thread("user-1", "gateway", Some("ext-abc"))
.await;
let (_, t2) = manager
.resolve_thread("user-1", "gateway", Some("ext-abc"))
.await;
assert_eq!(t1, t2);
// A different external ID on the same channel/user gets a new thread.
let (_, t3) = manager
.resolve_thread("user-1", "gateway", Some("ext-xyz"))
.await;
assert_ne!(t1, t3);
}
#[tokio::test]
async fn test_resolve_thread_none_vs_some_external_id() {
let manager = SessionManager::new();
// None external_thread_id is a distinct key from Some("ext-1").
let (_, t_none) = manager.resolve_thread("user-1", "cli", None).await;
let (_, t_some) = manager.resolve_thread("user-1", "cli", Some("ext-1")).await;
assert_ne!(t_none, t_some);
}
#[tokio::test]
async fn test_resolve_thread_different_users_isolated() {
let manager = SessionManager::new();
let (_, t1) = manager
.resolve_thread("user-a", "gateway", Some("same-ext"))
.await;
let (_, t2) = manager
.resolve_thread("user-b", "gateway", Some("same-ext"))
.await;
// Same channel + same external ID but different users = different threads
assert_ne!(t1, t2);
}
#[tokio::test]
async fn test_resolve_thread_different_channels_isolated() {
let manager = SessionManager::new();
let (_, t1) = manager
.resolve_thread("user-1", "gateway", Some("thread-x"))
.await;
let (_, t2) = manager
.resolve_thread("user-1", "telegram", Some("thread-x"))
.await;
// Same user + same external ID but different channels = different threads
assert_ne!(t1, t2);
}
#[tokio::test]
async fn test_resolve_thread_stale_mapping_creates_new_thread() {
let manager = SessionManager::new();
// Create a thread normally
let (session, original_tid) = manager
.resolve_thread("user-1", "gateway", Some("ext-1"))
.await;
// Simulate the thread being removed from the session (e.g. pruned)
{
let mut sess = session.lock().await;
sess.threads.remove(&original_tid);
}
// Next resolve should detect the stale mapping and create a fresh thread
let (_, new_tid) = manager
.resolve_thread("user-1", "gateway", Some("ext-1"))
.await;
assert_ne!(original_tid, new_tid);
// The new thread should actually exist in the session
let sess = session.lock().await;
assert!(sess.threads.contains_key(&new_tid));
}
#[tokio::test]
async fn test_register_thread_preserves_uuid_on_resolve() {
use crate::agent::session::{Session, Thread};
let manager = SessionManager::new();
let known_uuid = Uuid::new_v4();
let session = Arc::new(Mutex::new(Session::new("user-web")));
let session_id = {
let sess = session.lock().await;
sess.id
};
// Simulate hydration: create thread with a known UUID
{
let mut sess = session.lock().await;
let thread = Thread::with_id(known_uuid, session_id);
sess.threads.insert(known_uuid, thread);
}
// Register it
manager
.register_thread("user-web", "gateway", known_uuid, Arc::clone(&session))
.await;
// resolve_thread with UUID as external_thread_id MUST return the same UUID,
// not mint a new one (this was the root cause of the "wrong conversation" bug)
let (_, resolved) = manager
.resolve_thread("user-web", "gateway", Some(&known_uuid.to_string()))
.await;
assert_eq!(resolved, known_uuid);
}
#[tokio::test]
async fn test_register_thread_idempotent() {
use crate::agent::session::{Session, Thread};
let manager = SessionManager::new();
let tid = Uuid::new_v4();
let session = Arc::new(Mutex::new(Session::new("user-idem")));
{
let mut sess = session.lock().await;
let thread = Thread::with_id(tid, sess.id);
sess.threads.insert(tid, thread);
}
// Register twice
manager
.register_thread("user-idem", "gateway", tid, Arc::clone(&session))
.await;
manager
.register_thread("user-idem", "gateway", tid, Arc::clone(&session))
.await;
// Should still resolve to the same thread
let (_, resolved) = manager
.resolve_thread("user-idem", "gateway", Some(&tid.to_string()))
.await;
assert_eq!(resolved, tid);
}
#[tokio::test]
async fn test_register_thread_creates_undo_manager() {
use crate::agent::session::{Session, Thread};
let manager = SessionManager::new();
let tid = Uuid::new_v4();
let session = Arc::new(Mutex::new(Session::new("user-undo")));
{
let mut sess = session.lock().await;
let thread = Thread::with_id(tid, sess.id);
sess.threads.insert(tid, thread);
}
manager
.register_thread("user-undo", "gateway", tid, Arc::clone(&session))
.await;
// Undo manager should exist for the registered thread
let undo = manager.get_undo_manager(tid).await;
let undo2 = manager.get_undo_manager(tid).await;
assert!(Arc::ptr_eq(&undo, &undo2));
}
#[tokio::test]
async fn test_register_thread_stores_session() {
use crate::agent::session::{Session, Thread};
let manager = SessionManager::new();
let tid = Uuid::new_v4();
let session = Arc::new(Mutex::new(Session::new("user-new")));
{
let mut sess = session.lock().await;
let thread = Thread::with_id(tid, sess.id);
sess.threads.insert(tid, thread);
}
// The user has no session yet in the manager
{
let sessions = manager.sessions.read().await;
assert!(!sessions.contains_key("user-new"));
}
manager
.register_thread("user-new", "gateway", tid, Arc::clone(&session))
.await;
// Now the session should be tracked
{
let sessions = manager.sessions.read().await;
assert!(sessions.contains_key("user-new"));
}
}
#[tokio::test]
async fn test_multiple_threads_per_user() {
let manager = SessionManager::new();
let (_, t1) = manager
.resolve_thread("user-1", "gateway", Some("thread-a"))
.await;
let (_, t2) = manager
.resolve_thread("user-1", "gateway", Some("thread-b"))
.await;
let (session, t3) = manager
.resolve_thread("user-1", "gateway", Some("thread-c"))
.await;
// All three should be distinct
assert_ne!(t1, t2);
assert_ne!(t2, t3);
assert_ne!(t1, t3);
// All three should exist in the same session
let sess = session.lock().await;
assert!(sess.threads.contains_key(&t1));
assert!(sess.threads.contains_key(&t2));
assert!(sess.threads.contains_key(&t3));
}
#[tokio::test]
async fn test_prune_cleans_thread_map_and_undo_managers() {
let manager = SessionManager::new();
let (stale_session, stale_tid) = manager.resolve_thread("user-stale", "cli", None).await;
// Backdate the session
{
let mut sess = stale_session.lock().await;
sess.last_active_at = chrono::Utc::now() - chrono::TimeDelta::seconds(86400 * 30);
}
// Verify thread_map and undo_managers have entries
{
let tm = manager.thread_map.read().await;
assert!(!tm.is_empty());
}
{
let um = manager.undo_managers.read().await;
assert!(um.contains_key(&stale_tid));
}
let pruned = manager
.prune_stale_sessions(std::time::Duration::from_secs(86400 * 7))
.await;
assert_eq!(pruned, 1);
// Thread map and undo managers should be cleaned up
{
let tm = manager.thread_map.read().await;
assert!(tm.is_empty());
}
{
let um = manager.undo_managers.read().await;
assert!(!um.contains_key(&stale_tid));
}
}
#[tokio::test]
async fn test_resolve_thread_active_thread_set() {
let manager = SessionManager::new();
let (session, thread_id) = manager
.resolve_thread("user-1", "gateway", Some("ext-1"))
.await;
// The resolved thread should be set as the active thread
let sess = session.lock().await;
assert_eq!(sess.active_thread, Some(thread_id));
}
#[tokio::test]
async fn test_register_then_resolve_different_channel_creates_new() {
use crate::agent::session::{Session, Thread};
let manager = SessionManager::new();
let tid = Uuid::new_v4();
let session = Arc::new(Mutex::new(Session::new("user-cross")));
{
let mut sess = session.lock().await;
let thread = Thread::with_id(tid, sess.id);
sess.threads.insert(tid, thread);
}
// Register on "gateway" channel
manager
.register_thread("user-cross", "gateway", tid, Arc::clone(&session))
.await;
// Resolve on a different channel with the same UUID string should NOT
// find the registered thread (channel is part of the key)
let (_, resolved) = manager
.resolve_thread("user-cross", "telegram", Some(&tid.to_string()))
.await;
assert_ne!(resolved, tid);
}
}
+131
View File
@@ -43,6 +43,49 @@ impl SubmissionParser {
if lower == "/thread new" || lower == "/new" {
return Submission::NewThread;
}
// System commands (bypass thread-state checks)
if lower == "/help" || lower == "/?" {
return Submission::SystemCommand {
command: "help".to_string(),
args: vec![],
};
}
if lower == "/version" {
return Submission::SystemCommand {
command: "version".to_string(),
args: vec![],
};
}
if lower == "/tools" {
return Submission::SystemCommand {
command: "tools".to_string(),
args: vec![],
};
}
if lower == "/ping" {
return Submission::SystemCommand {
command: "ping".to_string(),
args: vec![],
};
}
if lower == "/debug" {
return Submission::SystemCommand {
command: "debug".to_string(),
args: vec![],
};
}
if lower.starts_with("/model") {
let args: Vec<String> = trimmed
.split_whitespace()
.skip(1)
.map(|s| s.to_string())
.collect();
return Submission::SystemCommand {
command: "model".to_string(),
args,
};
}
if lower == "/quit" || lower == "/exit" || lower == "/shutdown" {
return Submission::Quit;
}
@@ -172,6 +215,15 @@ pub enum Submission {
/// Quit the agent. Bypasses thread-state checks.
Quit,
/// System command (help, model, version, tools, ping, debug).
/// Bypasses thread-state checks and safety validation.
SystemCommand {
/// The command name (e.g. "help", "model", "version").
command: String,
/// Arguments to the command.
args: Vec<String>,
},
}
impl Submission {
@@ -238,6 +290,7 @@ impl Submission {
| Self::Heartbeat
| Self::Summarize
| Self::Suggest
| Self::SystemCommand { .. }
)
}
}
@@ -504,6 +557,84 @@ mod tests {
);
}
#[test]
fn test_parser_system_command_help() {
let submission = SubmissionParser::parse("/help");
assert!(
matches!(submission, Submission::SystemCommand { command, args } if command == "help" && args.is_empty())
);
let submission = SubmissionParser::parse("/?");
assert!(
matches!(submission, Submission::SystemCommand { command, .. } if command == "help")
);
let submission = SubmissionParser::parse("/HELP");
assert!(
matches!(submission, Submission::SystemCommand { command, .. } if command == "help")
);
}
#[test]
fn test_parser_system_command_model() {
// No args: show current model
let submission = SubmissionParser::parse("/model");
assert!(
matches!(submission, Submission::SystemCommand { command, args } if command == "model" && args.is_empty())
);
// With args: switch model
let submission = SubmissionParser::parse("/model gpt-4o");
assert!(
matches!(submission, Submission::SystemCommand { command, args } if command == "model" && args == vec!["gpt-4o"])
);
// Case insensitive command, preserves arg case
let submission = SubmissionParser::parse("/MODEL Claude-3.5");
assert!(
matches!(submission, Submission::SystemCommand { command, args } if command == "model" && args == vec!["Claude-3.5"])
);
}
#[test]
fn test_parser_system_command_version() {
let submission = SubmissionParser::parse("/version");
assert!(
matches!(submission, Submission::SystemCommand { command, args } if command == "version" && args.is_empty())
);
}
#[test]
fn test_parser_system_command_tools() {
let submission = SubmissionParser::parse("/tools");
assert!(
matches!(submission, Submission::SystemCommand { command, args } if command == "tools" && args.is_empty())
);
}
#[test]
fn test_parser_system_command_ping() {
let submission = SubmissionParser::parse("/ping");
assert!(
matches!(submission, Submission::SystemCommand { command, args } if command == "ping" && args.is_empty())
);
}
#[test]
fn test_parser_system_command_debug() {
let submission = SubmissionParser::parse("/debug");
assert!(
matches!(submission, Submission::SystemCommand { command, args } if command == "debug" && args.is_empty())
);
}
#[test]
fn test_parser_system_command_is_control() {
let submission = SubmissionParser::parse("/help");
assert!(submission.is_control());
assert!(!submission.starts_turn());
}
#[test]
fn test_parser_quit() {
assert!(matches!(SubmissionParser::parse("/quit"), Submission::Quit));
+132 -27
View File
@@ -10,8 +10,8 @@ use uuid::Uuid;
use crate::agent::scheduler::WorkerMessage;
use crate::agent::task::TaskOutput;
use crate::context::{ContextManager, JobState};
use crate::db::Database;
use crate::error::Error;
use crate::history::Store;
use crate::llm::{
ActionPlan, ChatMessage, LlmProvider, Reasoning, ReasoningContext, RespondResult, ToolSelection,
};
@@ -28,7 +28,7 @@ pub struct WorkerDeps {
pub llm: Arc<dyn LlmProvider>,
pub safety: Arc<SafetyLayer>,
pub tools: Arc<ToolRegistry>,
pub store: Option<Arc<Store>>,
pub store: Option<Arc<dyn Database>>,
pub timeout: Duration,
pub use_planning: bool,
}
@@ -67,7 +67,7 @@ impl Worker {
&self.deps.tools
}
fn store(&self) -> Option<&Arc<Store>> {
fn store(&self) -> Option<&Arc<dyn Database>> {
self.deps.store.as_ref()
}
@@ -248,16 +248,15 @@ Report when the job is complete or if you encounter issues you cannot resolve."#
if selections.is_empty() {
// No tools from select_tools, ask LLM directly (may still return tool calls)
let respond_result = reasoning.respond_with_tools(reason_ctx).await?;
let respond_output = reasoning.respond_with_tools(reason_ctx).await?;
match respond_result {
match respond_output.result {
RespondResult::Text(response) => {
// Check for completion keywords
let response_lower = response.to_lowercase();
if response_lower.contains("complete")
|| response_lower.contains("finished")
|| response_lower.contains("done")
{
// Check for explicit completion phrases. Use word-boundary
// aware checks to avoid false positives like "incomplete",
// "not done", or "unfinished". Only the LLM's own response
// (not tool output) can trigger this.
if crate::util::llm_signals_completion(&response) {
self.mark_completed().await?;
return Ok(());
}
@@ -272,7 +271,10 @@ Report when the job is complete or if you encounter issues you cannot resolve."#
));
}
}
RespondResult::ToolCalls(tool_calls) => {
RespondResult::ToolCalls {
tool_calls,
content,
} => {
// Model returned tool calls - execute them
tracing::debug!(
"Job {} respond_with_tools returned {} tool calls",
@@ -280,6 +282,14 @@ Report when the job is complete or if you encounter issues you cannot resolve."#
tool_calls.len()
);
// Add assistant message with tool_calls (OpenAI protocol)
reason_ctx
.messages
.push(ChatMessage::assistant_with_tool_calls(
content,
tool_calls.clone(),
));
for tc in tool_calls {
let result = self.execute_tool(&tc.name, &tc.arguments).await;
@@ -371,7 +381,7 @@ Report when the job is complete or if you encounter issues you cannot resolve."#
tools: Arc<ToolRegistry>,
context_manager: Arc<ContextManager>,
safety: Arc<SafetyLayer>,
store: Option<Arc<Store>>,
store: Option<Arc<dyn Database>>,
job_id: Uuid,
tool_name: &str,
params: &serde_json::Value,
@@ -417,14 +427,51 @@ Report when the job is complete or if you encounter issues you cannot resolve."#
.into());
}
// Execute with timeout and timing
tracing::debug!(
tool = %tool_name,
params = %params,
job = %job_id,
"Tool call started"
);
// Execute with per-tool timeout and timing
let tool_timeout = tool.execution_timeout();
let start = std::time::Instant::now();
let result = tokio::time::timeout(Duration::from_secs(60), async {
let result = tokio::time::timeout(tool_timeout, async {
tool.execute(params.clone(), &job_ctx).await
})
.await;
let elapsed = start.elapsed();
match &result {
Ok(Ok(output)) => {
let result_str = serde_json::to_string(&output.result)
.unwrap_or_else(|_| "<serialize error>".to_string());
tracing::debug!(
tool = %tool_name,
elapsed_ms = elapsed.as_millis() as u64,
result = %result_str,
"Tool call succeeded"
);
}
Ok(Err(e)) => {
tracing::debug!(
tool = %tool_name,
elapsed_ms = elapsed.as_millis() as u64,
error = %e,
"Tool call failed"
);
}
Err(_) => {
tracing::debug!(
tool = %tool_name,
elapsed_ms = elapsed.as_millis() as u64,
timeout_secs = tool_timeout.as_secs(),
"Tool call timed out"
);
}
}
// Record action in memory and get the ActionRecord for persistence
let action = match &result {
Ok(Ok(output)) => {
@@ -479,7 +526,7 @@ Report when the job is complete or if you encounter issues you cannot resolve."#
let output = result
.map_err(|_| crate::error::ToolError::Timeout {
name: tool_name.to_string(),
timeout: Duration::from_secs(60),
timeout: tool_timeout,
})?
.map_err(|e| crate::error::ToolError::ExecutionFailed {
name: tool_name.to_string(),
@@ -523,12 +570,9 @@ Report when the job is complete or if you encounter issues you cannot resolve."#
wrapped,
));
// Check if job is complete
if output.contains("TASK_COMPLETE") || output.contains("JOB_DONE") {
self.mark_completed().await?;
return Ok(true);
}
// Tool output never drives job completion. A malicious tool could
// emit "TASK_COMPLETE" to force premature completion. Only the LLM's
// own structured response (in execution_loop) can mark a job done.
Ok(false)
}
Err(e) => {
@@ -632,11 +676,7 @@ Report when the job is complete or if you encounter issues you cannot resolve."#
let response = reasoning.respond(reason_ctx).await?;
reason_ctx.messages.push(ChatMessage::assistant(&response));
let response_lower = response.to_lowercase();
if response_lower.contains("complete")
|| response_lower.contains("finished")
|| response_lower.contains("done")
{
if crate::util::llm_signals_completion(&response) {
self.mark_completed().await?;
} else {
// Job not complete, could re-plan or fall back to direct selection
@@ -731,3 +771,68 @@ impl From<TaskOutput> for Result<String, Error> {
})
}
}
#[cfg(test)]
mod tests {
use crate::util::llm_signals_completion;
#[test]
fn test_completion_positive_signals() {
assert!(llm_signals_completion("The job is complete."));
assert!(llm_signals_completion(
"I have completed the task successfully."
));
assert!(llm_signals_completion("The task is done."));
assert!(llm_signals_completion("The task is finished."));
assert!(llm_signals_completion(
"All steps are complete and verified."
));
assert!(llm_signals_completion(
"I've done all the work. The work is done."
));
assert!(llm_signals_completion(
"Successfully completed the migration."
));
}
#[test]
fn test_completion_negative_signals_block_false_positives() {
// These contain completion keywords but also negation, should NOT trigger.
assert!(!llm_signals_completion("The task is not complete yet."));
assert!(!llm_signals_completion("This is not done."));
assert!(!llm_signals_completion("The work is incomplete."));
assert!(!llm_signals_completion(
"The migration is not yet finished."
));
assert!(!llm_signals_completion("The job isn't done yet."));
assert!(!llm_signals_completion("This remains unfinished."));
}
#[test]
fn test_completion_does_not_match_bare_substrings() {
// Bare words embedded in other text should NOT trigger completion.
assert!(!llm_signals_completion(
"I need to complete more work first."
));
assert!(!llm_signals_completion(
"Let me finish the remaining steps."
));
assert!(!llm_signals_completion(
"I'm done analyzing, now let me fix it."
));
assert!(!llm_signals_completion(
"I completed step 1 but step 2 remains."
));
}
#[test]
fn test_completion_tool_output_injection() {
// A malicious tool output echoed by the LLM should not trigger
// completion unless it forms a genuine completion phrase.
assert!(!llm_signals_completion("TASK_COMPLETE"));
assert!(!llm_signals_completion("JOB_DONE"));
assert!(!llm_signals_completion(
"The tool returned: TASK_COMPLETE signal"
));
}
}
+325
View File
@@ -0,0 +1,325 @@
//! Bootstrap configuration for IronClaw.
//!
//! These are the only settings that MUST live on disk because they're needed
//! before the database connection is established. Everything else lives in the
//! `settings` table in PostgreSQL.
//!
//! File: `~/.ironclaw/bootstrap.json`
use std::path::PathBuf;
use serde::{Deserialize, Serialize};
use crate::settings::KeySource;
/// Minimal config needed to connect to the database and decrypt secrets.
///
/// This is the only JSON file IronClaw reads from disk at startup.
/// All other configuration lives in the `settings` table in PostgreSQL.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct BootstrapConfig {
/// Database connection URL (postgres://...).
#[serde(default)]
pub database_url: Option<String>,
/// Database connection pool size.
#[serde(default)]
pub database_pool_size: Option<usize>,
/// Source for the secrets master key.
#[serde(default)]
pub secrets_master_key_source: KeySource,
/// Whether onboarding wizard has been completed.
#[serde(default)]
pub onboard_completed: bool,
}
impl Default for BootstrapConfig {
fn default() -> Self {
Self {
database_url: None,
database_pool_size: None,
secrets_master_key_source: KeySource::None,
onboard_completed: false,
}
}
}
impl BootstrapConfig {
/// Default bootstrap file path: `~/.ironclaw/bootstrap.json`.
pub fn default_path() -> PathBuf {
dirs::home_dir()
.unwrap_or_else(|| PathBuf::from("."))
.join(".ironclaw")
.join("bootstrap.json")
}
/// Legacy settings.json path (for migration detection).
pub fn legacy_settings_path() -> PathBuf {
dirs::home_dir()
.unwrap_or_else(|| PathBuf::from("."))
.join(".ironclaw")
.join("settings.json")
}
/// Load from the default path, falling back to legacy settings.json,
/// then to defaults if neither exists.
pub fn load() -> Self {
let bootstrap_path = Self::default_path();
if bootstrap_path.exists() {
return Self::load_from(&bootstrap_path);
}
// Fall back to legacy settings.json (extract just the 4 bootstrap fields)
let legacy_path = Self::legacy_settings_path();
if legacy_path.exists() {
return Self::load_from_legacy(&legacy_path);
}
Self::default()
}
/// Load from a specific path.
pub fn load_from(path: &PathBuf) -> Self {
match std::fs::read_to_string(path) {
Ok(data) => serde_json::from_str(&data).unwrap_or_default(),
Err(_) => Self::default(),
}
}
/// Extract bootstrap fields from a legacy settings.json.
fn load_from_legacy(path: &PathBuf) -> Self {
match std::fs::read_to_string(path) {
Ok(data) => {
// The legacy Settings struct is a superset; serde will ignore extra fields.
serde_json::from_str(&data).unwrap_or_default()
}
Err(_) => Self::default(),
}
}
/// Save to the default path.
pub fn save(&self) -> std::io::Result<()> {
self.save_to(&Self::default_path())
}
/// Save to a specific path.
pub fn save_to(&self, path: &PathBuf) -> std::io::Result<()> {
if let Some(parent) = path.parent() {
std::fs::create_dir_all(parent)?;
}
let json = serde_json::to_string_pretty(self)
.map_err(|e| std::io::Error::new(std::io::ErrorKind::InvalidData, e.to_string()))?;
std::fs::write(path, json)
}
}
/// One-time migration from disk config files to the database settings table.
///
/// On first boot after upgrade, checks if:
/// 1. `~/.ironclaw/settings.json` exists
/// 2. The DB settings table is empty for this user
///
/// If both conditions hold, migrates settings, MCP servers, and session data
/// to the database, writes `bootstrap.json`, and renames old files to `.migrated`.
pub async fn migrate_disk_to_db(
store: &dyn crate::db::Database,
user_id: &str,
) -> Result<(), MigrationError> {
let legacy_settings_path = BootstrapConfig::legacy_settings_path();
if !legacy_settings_path.exists() {
tracing::debug!("No legacy settings.json found, skipping disk-to-DB migration");
return Ok(());
}
// Only migrate if DB is empty for this user
let has_settings = store.has_settings(user_id).await.map_err(|e| {
MigrationError::Database(format!("Failed to check existing settings: {}", e))
})?;
if has_settings {
tracing::debug!(
"DB already has settings for user '{}', skipping migration",
user_id
);
return Ok(());
}
tracing::info!("Migrating disk settings to database...");
// 1. Load and migrate settings.json
let settings = crate::settings::Settings::load_from(&legacy_settings_path);
let db_map = settings.to_db_map();
if !db_map.is_empty() {
store
.set_all_settings(user_id, &db_map)
.await
.map_err(|e| {
MigrationError::Database(format!("Failed to write settings to DB: {}", e))
})?;
tracing::info!("Migrated {} settings to database", db_map.len());
}
// 2. Write bootstrap.json with the 4 essential fields
let bootstrap = BootstrapConfig {
database_url: settings.database_url.clone(),
database_pool_size: settings.database_pool_size,
secrets_master_key_source: settings.secrets_master_key_source,
onboard_completed: settings.onboard_completed,
};
bootstrap
.save()
.map_err(|e| MigrationError::Io(format!("Failed to write bootstrap.json: {}", e)))?;
tracing::info!("Wrote bootstrap.json");
// 3. Migrate mcp-servers.json if it exists
let ironclaw_dir = dirs::home_dir()
.unwrap_or_else(|| PathBuf::from("."))
.join(".ironclaw");
let mcp_path = ironclaw_dir.join("mcp-servers.json");
if mcp_path.exists() {
match std::fs::read_to_string(&mcp_path) {
Ok(content) => match serde_json::from_str::<serde_json::Value>(&content) {
Ok(value) => {
store
.set_setting(user_id, "mcp_servers", &value)
.await
.map_err(|e| {
MigrationError::Database(format!(
"Failed to write MCP servers to DB: {}",
e
))
})?;
tracing::info!("Migrated mcp-servers.json to database");
rename_to_migrated(&mcp_path);
}
Err(e) => {
tracing::warn!("Failed to parse mcp-servers.json: {}", e);
}
},
Err(e) => {
tracing::warn!("Failed to read mcp-servers.json: {}", e);
}
}
}
// 4. Migrate session.json if it exists
let session_path = ironclaw_dir.join("session.json");
if session_path.exists() {
match std::fs::read_to_string(&session_path) {
Ok(content) => match serde_json::from_str::<serde_json::Value>(&content) {
Ok(value) => {
store
.set_setting(user_id, "nearai.session", &value)
.await
.map_err(|e| {
MigrationError::Database(format!(
"Failed to write session to DB: {}",
e
))
})?;
tracing::info!("Migrated session.json to database");
rename_to_migrated(&session_path);
}
Err(e) => {
tracing::warn!("Failed to parse session.json: {}", e);
}
},
Err(e) => {
tracing::warn!("Failed to read session.json: {}", e);
}
}
}
// 5. Rename settings.json to .migrated (don't delete, safety net)
rename_to_migrated(&legacy_settings_path);
tracing::info!("Disk-to-DB migration complete");
Ok(())
}
/// Rename a file to `<name>.migrated` as a safety net.
fn rename_to_migrated(path: &PathBuf) {
let mut migrated = path.as_os_str().to_owned();
migrated.push(".migrated");
if let Err(e) = std::fs::rename(path, &migrated) {
tracing::warn!("Failed to rename {} to .migrated: {}", path.display(), e);
}
}
/// Errors that can occur during disk-to-DB migration.
#[derive(Debug, thiserror::Error)]
pub enum MigrationError {
#[error("Database error: {0}")]
Database(String),
#[error("IO error: {0}")]
Io(String),
}
#[cfg(test)]
mod tests {
use super::*;
use tempfile::tempdir;
#[test]
fn test_bootstrap_save_load() {
let dir = tempdir().unwrap();
let path = dir.path().join("bootstrap.json");
let config = BootstrapConfig {
database_url: Some("postgres://localhost/test".to_string()),
database_pool_size: Some(5),
secrets_master_key_source: KeySource::Keychain,
onboard_completed: true,
};
config.save_to(&path).unwrap();
let loaded = BootstrapConfig::load_from(&path);
assert_eq!(
loaded.database_url,
Some("postgres://localhost/test".to_string())
);
assert_eq!(loaded.database_pool_size, Some(5));
assert_eq!(loaded.secrets_master_key_source, KeySource::Keychain);
assert!(loaded.onboard_completed);
}
#[test]
fn test_bootstrap_from_legacy_settings() {
let dir = tempdir().unwrap();
let path = dir.path().join("settings.json");
// Write a legacy settings.json with many extra fields
let legacy = serde_json::json!({
"database_url": "postgres://localhost/ironclaw",
"database_pool_size": 10,
"secrets_master_key_source": "keychain",
"onboard_completed": true,
"selected_model": "claude-3-5-sonnet",
"agent": { "name": "testbot", "max_parallel_jobs": 3 },
"heartbeat": { "enabled": true }
});
std::fs::write(&path, serde_json::to_string_pretty(&legacy).unwrap()).unwrap();
let config = BootstrapConfig::load_from_legacy(&path);
assert_eq!(
config.database_url,
Some("postgres://localhost/ironclaw".to_string())
);
assert_eq!(config.database_pool_size, Some(10));
assert_eq!(config.secrets_master_key_source, KeySource::Keychain);
assert!(config.onboard_completed);
}
#[test]
fn test_bootstrap_defaults() {
let config = BootstrapConfig::default();
assert!(config.database_url.is_none());
assert!(config.database_pool_size.is_none());
assert_eq!(config.secrets_master_key_source, KeySource::None);
assert!(!config.onboard_completed);
}
}
+19
View File
@@ -114,6 +114,12 @@ pub enum StatusUpdate {
StreamChunk(String),
/// General status message.
Status(String),
/// A sandbox job has started (shown as a clickable card in the UI).
JobStarted {
job_id: String,
title: String,
browse_url: String,
},
/// Tool requires user approval before execution.
ApprovalNeeded {
request_id: String,
@@ -121,6 +127,19 @@ pub enum StatusUpdate {
description: String,
parameters: serde_json::Value,
},
/// Extension needs user authentication (token or OAuth).
AuthRequired {
extension_name: String,
instructions: Option<String>,
auth_url: Option<String>,
setup_url: Option<String>,
},
/// Extension authentication completed.
AuthCompleted {
extension_name: String,
success: bool,
message: String,
},
}
/// Trait for message channels.
+52 -2
View File
@@ -42,12 +42,25 @@ const SLASH_COMMANDS: &[&str] = &[
"/quit",
"/exit",
"/debug",
"/model",
"/undo",
"/redo",
"/clear",
"/compact",
"/new",
"/interrupt",
"/version",
"/tools",
"/ping",
"/job",
"/status",
"/cancel",
"/list",
"/heartbeat",
"/summarize",
"/suggest",
"/thread",
"/resume",
];
/// Rustyline helper for slash-command tab completion.
@@ -295,10 +308,11 @@ impl Channel for ReplChannel {
continue;
}
// Handle local REPL commands
// Handle local REPL commands (only commands that need
// immediate local handling stay here)
match line.to_lowercase().as_str() {
"/quit" | "/exit" => break,
"/help" | "/?" => {
"/help" => {
print_help();
continue;
}
@@ -413,6 +427,15 @@ impl Channel for ReplChannel {
print!("{chunk}");
let _ = io::stdout().flush();
}
StatusUpdate::JobStarted {
job_id,
title,
browse_url,
} => {
eprintln!(
" \x1b[36m[job]\x1b[0m {title} \x1b[90m({job_id})\x1b[0m \x1b[4m{browse_url}\x1b[0m"
);
}
StatusUpdate::Status(msg) => {
if debug || msg.contains("approval") || msg.contains("Approval") {
eprintln!(" \x1b[90m{msg}\x1b[0m");
@@ -472,6 +495,33 @@ impl Channel for ReplChannel {
eprintln!(" {bot_border}");
eprintln!();
}
StatusUpdate::AuthRequired {
extension_name,
instructions,
setup_url,
..
} => {
eprintln!();
eprintln!("\x1b[33m Authentication required for {extension_name}\x1b[0m");
if let Some(ref instr) = instructions {
eprintln!(" {instr}");
}
if let Some(ref url) = setup_url {
eprintln!(" \x1b[4m{url}\x1b[0m");
}
eprintln!();
}
StatusUpdate::AuthCompleted {
extension_name,
success,
message,
} => {
if success {
eprintln!("\x1b[32m {extension_name}: {message}\x1b[0m");
} else {
eprintln!("\x1b[31m {extension_name}: {message}\x1b[0m");
}
}
}
Ok(())
}
+109 -48
View File
@@ -1,68 +1,125 @@
//! Bundled WASM channels that can be installed locally.
//! Known WASM channels that can be installed from build artifacts.
//!
//! Instead of embedding WASM binaries in the host binary via include_bytes!,
//! channels are compiled separately and installed from their build output
//! directories during onboarding.
//!
//! Channel source layout:
//! channels-src/<name>/
//! target/wasm32-wasip2/release/<name>_channel.wasm
//! <name>.capabilities.json
use std::path::Path;
use std::path::{Path, PathBuf};
use tokio::fs;
#[derive(Clone, Copy)]
struct BundledChannel {
name: &'static str,
wasm: &'static [u8],
capabilities: &'static [u8],
/// Compile-time project root, used to locate channels-src/ in dev builds.
const CARGO_MANIFEST_DIR: &str = env!("CARGO_MANIFEST_DIR");
/// Known channel names and their crate names (for locating build artifacts).
const KNOWN_CHANNELS: &[(&str, &str)] = &[
("telegram", "telegram_channel"),
("slack", "slack_channel"),
("whatsapp", "whatsapp_channel"),
];
/// Names of known channels that can be installed.
pub fn bundled_channel_names() -> Vec<&'static str> {
KNOWN_CHANNELS.iter().map(|(name, _)| *name).collect()
}
/// Names of bundled channels shipped with IronClaw.
pub fn bundled_channel_names() -> &'static [&'static str] {
&["telegram"]
/// Resolve the channels source directory.
///
/// Checks (in order):
/// 1. `IRONCLAW_CHANNELS_SRC` env var
/// 2. `<CARGO_MANIFEST_DIR>/channels-src/` (dev builds)
fn channels_src_dir() -> PathBuf {
if let Ok(dir) = std::env::var("IRONCLAW_CHANNELS_SRC") {
return PathBuf::from(dir);
}
PathBuf::from(CARGO_MANIFEST_DIR).join("channels-src")
}
/// Install a bundled channel into a channels directory.
/// Locate the build artifacts for a channel.
///
/// Returns (wasm_path, capabilities_path) or an error if files are missing.
fn locate_channel_artifacts(name: &str) -> Result<(PathBuf, PathBuf), String> {
let (_, crate_name) = KNOWN_CHANNELS
.iter()
.find(|(n, _)| *n == name)
.ok_or_else(|| format!("Unknown channel '{}'", name))?;
let src_dir = channels_src_dir();
let channel_dir = src_dir.join(name);
let wasm_path = channel_dir
.join("target/wasm32-wasip2/release")
.join(format!("{}.wasm", crate_name));
let caps_path = channel_dir.join(format!("{}.capabilities.json", name));
if !wasm_path.exists() {
return Err(format!(
"Channel '{}' WASM not found at {}. Build it first:\n \
cd {} && cargo build --target wasm32-wasip2 --release",
name,
wasm_path.display(),
channel_dir.display()
));
}
if !caps_path.exists() {
return Err(format!(
"Channel '{}' capabilities not found at {}",
name,
caps_path.display()
));
}
Ok((wasm_path, caps_path))
}
/// Install a channel from build artifacts into the channels directory.
pub async fn install_bundled_channel(
name: &str,
target_dir: &Path,
force: bool,
) -> Result<(), String> {
let channel = bundled_channel(name)
.ok_or_else(|| format!("Unknown bundled channel '{}'", name.to_lowercase()))?;
let (wasm_src, caps_src) = locate_channel_artifacts(name)?;
fs::create_dir_all(target_dir)
.await
.map_err(|e| format!("Failed to create channels directory: {}", e))?;
let wasm_path = target_dir.join(format!("{}.wasm", channel.name));
let caps_path = target_dir.join(format!("{}.capabilities.json", channel.name));
let wasm_dst = target_dir.join(format!("{}.wasm", name));
let caps_dst = target_dir.join(format!("{}.capabilities.json", name));
let has_existing = wasm_path.exists() || caps_path.exists();
let has_existing = wasm_dst.exists() || caps_dst.exists();
if has_existing && !force {
return Err(format!(
"Channel '{}' already exists at {}",
channel.name,
name,
target_dir.display()
));
}
fs::write(&wasm_path, channel.wasm)
fs::copy(&wasm_src, &wasm_dst)
.await
.map_err(|e| format!("Failed to write {}: {}", wasm_path.display(), e))?;
fs::write(&caps_path, channel.capabilities)
.map_err(|e| format!("Failed to copy {}: {}", wasm_src.display(), e))?;
fs::copy(&caps_src, &caps_dst)
.await
.map_err(|e| format!("Failed to write {}: {}", caps_path.display(), e))?;
.map_err(|e| format!("Failed to copy {}: {}", caps_src.display(), e))?;
Ok(())
}
fn bundled_channel(name: &str) -> Option<BundledChannel> {
if name.eq_ignore_ascii_case("telegram") {
Some(BundledChannel {
name: "telegram",
wasm: include_bytes!("../../../channels-src/telegram/telegram.wasm"),
capabilities: include_bytes!(
"../../../channels-src/telegram/telegram.capabilities.json"
),
})
} else {
None
}
/// Check which known channels have build artifacts available.
pub fn available_channel_names() -> Vec<&'static str> {
KNOWN_CHANNELS
.iter()
.filter(|(name, _)| locate_channel_artifacts(name).is_ok())
.map(|(name, _)| *name)
.collect()
}
#[cfg(test)]
@@ -73,31 +130,35 @@ mod tests {
use super::*;
#[test]
fn test_bundled_channel_names_contains_telegram() {
assert!(bundled_channel_names().contains(&"telegram"));
fn test_known_channels_includes_all_three() {
let names = bundled_channel_names();
assert!(names.contains(&"telegram"));
assert!(names.contains(&"slack"));
assert!(names.contains(&"whatsapp"));
}
#[test]
fn test_channels_src_dir_default() {
let dir = channels_src_dir();
assert!(dir.ends_with("channels-src"));
}
#[test]
fn test_locate_unknown_channel_errors() {
assert!(locate_channel_artifacts("nonexistent").is_err());
}
#[tokio::test]
async fn test_install_bundled_channel_writes_files() {
let dir = tempdir().unwrap();
install_bundled_channel("telegram", dir.path(), false)
.await
.unwrap();
assert!(dir.path().join("telegram.wasm").exists());
assert!(dir.path().join("telegram.capabilities.json").exists());
}
#[tokio::test]
async fn test_install_bundled_channel_refuses_overwrite_without_force() {
async fn test_install_refuses_overwrite_without_force() {
let dir = tempdir().unwrap();
let wasm_path = dir.path().join("telegram.wasm");
fs::write(&wasm_path, b"custom").await.unwrap();
let result = install_bundled_channel("telegram", dir.path(), false).await;
// Either fails because artifacts missing OR because file exists
assert!(result.is_err());
// Original file should be untouched
let existing = fs::read(&wasm_path).await.unwrap();
assert_eq!(existing, b"custom");
}
+17 -5
View File
@@ -16,16 +16,21 @@ use crate::channels::wasm::error::WasmChannelError;
use crate::channels::wasm::runtime::WasmChannelRuntime;
use crate::channels::wasm::schema::ChannelCapabilitiesFile;
use crate::channels::wasm::wrapper::WasmChannel;
use crate::pairing::PairingStore;
/// Loads WASM channels from the filesystem.
pub struct WasmChannelLoader {
runtime: Arc<WasmChannelRuntime>,
pairing_store: Arc<PairingStore>,
}
impl WasmChannelLoader {
/// Create a new loader with the given runtime.
pub fn new(runtime: Arc<WasmChannelRuntime>) -> Self {
Self { runtime }
/// Create a new loader with the given runtime and pairing store.
pub fn new(runtime: Arc<WasmChannelRuntime>, pairing_store: Arc<PairingStore>) -> Self {
Self {
runtime,
pairing_store,
}
}
/// Load a single WASM channel from a file pair.
@@ -114,7 +119,13 @@ impl WasmChannelLoader {
.await?;
// Create the channel
let channel = WasmChannel::new(self.runtime.clone(), prepared, capabilities, config_json);
let channel = WasmChannel::new(
self.runtime.clone(),
prepared,
capabilities,
config_json,
self.pairing_store.clone(),
);
tracing::info!(
name = name,
@@ -352,6 +363,7 @@ mod tests {
use crate::channels::wasm::loader::{WasmChannelLoader, discover_channels};
use crate::channels::wasm::runtime::{WasmChannelRuntime, WasmChannelRuntimeConfig};
use crate::pairing::PairingStore;
use std::sync::Arc;
#[tokio::test]
@@ -408,7 +420,7 @@ mod tests {
async fn test_loader_invalid_name() {
let config = WasmChannelRuntimeConfig::for_testing();
let runtime = Arc::new(WasmChannelRuntime::new(config).unwrap());
let loader = WasmChannelLoader::new(runtime);
let loader = WasmChannelLoader::new(runtime, Arc::new(PairingStore::new()));
let dir = TempDir::new().unwrap();
let wasm_path = dir.path().join("test.wasm");
+1 -1
View File
@@ -89,7 +89,7 @@ mod schema;
mod wrapper;
// Core types
pub use bundled::{bundled_channel_names, install_bundled_channel};
pub use bundled::{available_channel_names, bundled_channel_names, install_bundled_channel};
pub use capabilities::{ChannelCapabilities, EmitRateLimitConfig, HttpEndpointConfig, PollConfig};
pub use error::WasmChannelError;
pub use host::{ChannelEmitRateLimiter, ChannelHostState, EmittedMessage};
+2
View File
@@ -478,6 +478,7 @@ mod tests {
PreparedChannelModule, WasmChannelRuntime, WasmChannelRuntimeConfig,
};
use crate::channels::wasm::wrapper::WasmChannel;
use crate::pairing::PairingStore;
use crate::tools::wasm::ResourceLimits;
fn create_test_channel(name: &str) -> Arc<WasmChannel> {
@@ -499,6 +500,7 @@ mod tests {
prepared,
capabilities,
"{}".to_string(),
Arc::new(PairingStore::new()),
))
}
+301 -24
View File
@@ -48,6 +48,7 @@ use crate::channels::wasm::runtime::{PreparedChannelModule, WasmChannelRuntime};
use crate::channels::wasm::schema::ChannelConfig;
use crate::channels::{Channel, IncomingMessage, MessageStream, OutgoingResponse, StatusUpdate};
use crate::error::ChannelError;
use crate::pairing::PairingStore;
use crate::safety::LeakDetector;
use crate::tools::wasm::LogLevel;
use crate::tools::wasm::WasmResourceLimiter;
@@ -73,6 +74,8 @@ struct ChannelStoreData {
/// Injected credentials for URL substitution (e.g., bot tokens).
/// Keys are placeholder names like "TELEGRAM_BOT_TOKEN".
credentials: HashMap<String, String>,
/// Pairing store for DM pairing (guest access control).
pairing_store: Arc<PairingStore>,
}
impl ChannelStoreData {
@@ -81,6 +84,7 @@ impl ChannelStoreData {
channel_name: &str,
capabilities: ChannelCapabilities,
credentials: HashMap<String, String>,
pairing_store: Arc<PairingStore>,
) -> Self {
// Create a minimal WASI context (no filesystem, no env vars for security)
let wasi = WasiCtxBuilder::new().build();
@@ -91,6 +95,7 @@ impl ChannelStoreData {
wasi,
table: ResourceTable::new(),
credentials,
pairing_store,
}
}
@@ -141,6 +146,22 @@ impl ChannelStoreData {
result
}
/// Replace injected credential values with `[REDACTED]` in text.
///
/// Prevents credentials from leaking through error messages, logs, or
/// return values to WASM. reqwest::Error includes the full URL in its
/// Display output, so any error from an injected-URL request will
/// contain the raw credential unless we scrub it.
fn redact_credentials(&self, text: &str) -> String {
let mut result = text.to_string();
for (name, value) in &self.credentials {
if !value.is_empty() {
result = result.replace(value, &format!("[REDACTED:{}]", name));
}
}
result
}
}
// Implement WasiView to provide WASI context and resource table
@@ -187,6 +208,7 @@ impl near::agent::channel_host::Host for ChannelStoreData {
url: String,
headers_json: String,
body: Option<Vec<u8>>,
timeout_ms: Option<u32>,
) -> Result<near::agent::channel_host::HttpResponse, String> {
tracing::info!(
method = %method,
@@ -251,6 +273,16 @@ impl near::agent::channel_host::Host for ChannelStoreData {
.scan_http_request(&url, &header_vec, body.as_deref())
.map_err(|e| format!("Potential secret leak blocked: {}", e))?;
// Get the max response size from capabilities (default 10MB).
let max_response_bytes = self
.host_state
.capabilities()
.tool_capabilities
.http
.as_ref()
.map(|h| h.max_response_bytes)
.unwrap_or(10 * 1024 * 1024);
// Make the HTTP request using blocking I/O
// We're already in a spawn_blocking context, so we can use block_on
let result = tokio::runtime::Handle::current().block_on(async {
@@ -276,12 +308,21 @@ impl near::agent::channel_host::Host for ChannelStoreData {
request = request.body(body_bytes);
}
// Send request with timeout
let response = request
.timeout(std::time::Duration::from_secs(30))
.send()
.await
.map_err(|e| format!("HTTP request failed: {}", e))?;
// Send request with caller-specified timeout (default 30s).
// Cap at callback_timeout to prevent outliving the host wrapper.
let timeout = std::time::Duration::from_millis(timeout_ms.unwrap_or(30_000) as u64);
let response = request.timeout(timeout).send().await.map_err(|e| {
// Walk the full error chain so we get the actual root cause
// (DNS, TLS, connection refused, etc.) instead of just
// "error sending request for url (...)".
let mut chain = format!("HTTP request failed: {}", e);
let mut source = std::error::Error::source(&e);
while let Some(cause) = source {
chain.push_str(&format!(" -> {}", cause));
source = cause.source();
}
chain
})?;
let status = response.status().as_u16();
let response_headers: std::collections::HashMap<String, String> = response
@@ -294,11 +335,29 @@ impl near::agent::channel_host::Host for ChannelStoreData {
})
.collect();
let headers_json = serde_json::to_string(&response_headers).unwrap_or_default();
// Enforce max response body size to prevent memory exhaustion.
let max_response = max_response_bytes;
if let Some(cl) = response.content_length() {
if cl as usize > max_response {
return Err(format!(
"Response body too large: {} bytes exceeds limit of {} bytes",
cl, max_response
));
}
}
let body = response
.bytes()
.await
.map_err(|e| format!("Failed to read response body: {}", e))?
.to_vec();
.map_err(|e| format!("Failed to read response body: {}", e))?;
if body.len() > max_response {
return Err(format!(
"Response body too large: {} bytes exceeds limit of {} bytes",
body.len(),
max_response
));
}
let body = body.to_vec();
tracing::info!(
status = status,
@@ -330,6 +389,11 @@ impl near::agent::channel_host::Host for ChannelStoreData {
})
});
// Scrub credential values from error messages before logging or returning
// to WASM. reqwest::Error includes the full URL (with injected credentials)
// in its Display output.
let result = result.map_err(|e| self.redact_credentials(&e));
match &result {
Ok(resp) => {
tracing::info!(status = resp.status, "http_request completed successfully");
@@ -372,6 +436,43 @@ impl near::agent::channel_host::Host for ChannelStoreData {
}
}
}
fn pairing_upsert_request(
&mut self,
channel: String,
id: String,
meta_json: String,
) -> Result<near::agent::channel_host::PairingUpsertResult, String> {
let meta = if meta_json.is_empty() {
None
} else {
serde_json::from_str(&meta_json).ok()
};
match self.pairing_store.upsert_request(&channel, &id, meta) {
Ok(r) => Ok(near::agent::channel_host::PairingUpsertResult {
code: r.code,
created: r.created,
}),
Err(e) => Err(e.to_string()),
}
}
fn pairing_is_allowed(
&mut self,
channel: String,
id: String,
username: Option<String>,
) -> Result<bool, String> {
self.pairing_store
.is_sender_allowed(&channel, &id, username.as_deref())
.map_err(|e| e.to_string())
}
fn pairing_read_allow_from(&mut self, channel: String) -> Result<Vec<String>, String> {
self.pairing_store
.read_allow_from(&channel)
.map_err(|e| e.to_string())
}
}
/// A WASM-based channel implementing the Channel trait.
@@ -424,6 +525,9 @@ pub struct WasmChannel {
/// Background task that repeats typing indicators every 4 seconds.
/// Telegram's "typing..." indicator expires after ~5s, so we refresh it.
typing_task: RwLock<Option<tokio::task::JoinHandle<()>>>,
/// Pairing store for DM pairing (guest access control).
pairing_store: Arc<PairingStore>,
}
impl WasmChannel {
@@ -433,6 +537,7 @@ impl WasmChannel {
prepared: Arc<PreparedChannelModule>,
capabilities: ChannelCapabilities,
config_json: String,
pairing_store: Arc<PairingStore>,
) -> Self {
let name = prepared.name.clone();
let rate_limiter = ChannelEmitRateLimiter::new(capabilities.emit_rate_limit.clone());
@@ -452,6 +557,7 @@ impl WasmChannel {
endpoints: RwLock::new(Vec::new()),
credentials: Arc::new(RwLock::new(HashMap::new())),
typing_task: RwLock::new(None),
pairing_store,
}
}
@@ -533,6 +639,7 @@ impl WasmChannel {
prepared: &PreparedChannelModule,
capabilities: &ChannelCapabilities,
credentials: HashMap<String, String>,
pairing_store: Arc<PairingStore>,
) -> Result<Store<ChannelStoreData>, WasmChannelError> {
let engine = runtime.engine();
let limits = &prepared.limits;
@@ -543,6 +650,7 @@ impl WasmChannel {
&prepared.name,
capabilities.clone(),
credentials,
pairing_store,
);
let mut store = Store::new(engine, store_data);
@@ -643,12 +751,18 @@ impl WasmChannel {
let timeout = self.runtime.config().callback_timeout;
let channel_name = self.name.clone();
let credentials = self.get_credentials().await;
let pairing_store = self.pairing_store.clone();
// Execute in blocking task with timeout
let result = tokio::time::timeout(timeout, async move {
tokio::task::spawn_blocking(move || {
let mut store =
Self::create_store(&runtime, &prepared, &capabilities, credentials)?;
let mut store = Self::create_store(
&runtime,
&prepared,
&capabilities,
credentials,
pairing_store,
)?;
let instance = Self::instantiate_component(&runtime, &prepared, &mut store)?;
// Call on_start using the generated typed interface
@@ -753,6 +867,7 @@ impl WasmChannel {
let capabilities = self.capabilities.clone();
let timeout = self.runtime.config().callback_timeout;
let credentials = self.get_credentials().await;
let pairing_store = self.pairing_store.clone();
// Prepare request data
let method = method.to_string();
@@ -766,8 +881,13 @@ impl WasmChannel {
// Execute in blocking task with timeout
let result = tokio::time::timeout(timeout, async move {
tokio::task::spawn_blocking(move || {
let mut store =
Self::create_store(&runtime, &prepared, &capabilities, credentials)?;
let mut store = Self::create_store(
&runtime,
&prepared,
&capabilities,
credentials,
pairing_store,
)?;
let instance = Self::instantiate_component(&runtime, &prepared, &mut store)?;
// Build the WIT request type
@@ -840,12 +960,18 @@ impl WasmChannel {
let timeout = self.runtime.config().callback_timeout;
let channel_name = self.name.clone();
let credentials = self.get_credentials().await;
let pairing_store = self.pairing_store.clone();
// Execute in blocking task with timeout
let result = tokio::time::timeout(timeout, async move {
tokio::task::spawn_blocking(move || {
let mut store =
Self::create_store(&runtime, &prepared, &capabilities, credentials)?;
let mut store = Self::create_store(
&runtime,
&prepared,
&capabilities,
credentials,
pairing_store,
)?;
let instance = Self::instantiate_component(&runtime, &prepared, &mut store)?;
// Call on_poll using the generated typed interface
@@ -929,6 +1055,7 @@ impl WasmChannel {
let timeout = self.runtime.config().callback_timeout;
let channel_name = self.name.clone();
let credentials = self.get_credentials().await;
let pairing_store = self.pairing_store.clone();
// Prepare response data
let message_id_str = message_id.to_string();
@@ -942,8 +1069,13 @@ impl WasmChannel {
let result = tokio::time::timeout(timeout, async move {
tokio::task::spawn_blocking(move || {
tracing::info!("Creating WASM store for on_respond");
let mut store =
Self::create_store(&runtime, &prepared, &capabilities, credentials)?;
let mut store = Self::create_store(
&runtime,
&prepared,
&capabilities,
credentials,
pairing_store,
)?;
tracing::info!("Instantiating WASM component for on_respond");
let instance = Self::instantiate_component(&runtime, &prepared, &mut store)?;
@@ -1036,13 +1168,19 @@ impl WasmChannel {
let timeout = self.runtime.config().callback_timeout;
let channel_name = self.name.clone();
let credentials = self.get_credentials().await;
let pairing_store = self.pairing_store.clone();
let wit_update = status_to_wit(status, metadata);
let result = tokio::time::timeout(timeout, async move {
tokio::task::spawn_blocking(move || {
let mut store =
Self::create_store(&runtime, &prepared, &capabilities, credentials)?;
let mut store = Self::create_store(
&runtime,
&prepared,
&capabilities,
credentials,
pairing_store,
)?;
let instance = Self::instantiate_component(&runtime, &prepared, &mut store)?;
let channel_iface = instance.near_agent_channel();
@@ -1080,12 +1218,14 @@ impl WasmChannel {
///
/// Static method for use by the background typing repeat task (which
/// doesn't have access to `&self`).
#[allow(clippy::too_many_arguments)]
async fn execute_status(
channel_name: &str,
runtime: &Arc<WasmChannelRuntime>,
prepared: &Arc<PreparedChannelModule>,
capabilities: &ChannelCapabilities,
credentials: &RwLock<HashMap<String, String>>,
pairing_store: Arc<PairingStore>,
timeout: Duration,
wit_update: wit_channel::StatusUpdate,
) -> Result<(), WasmChannelError> {
@@ -1101,8 +1241,13 @@ impl WasmChannel {
let result = tokio::time::timeout(timeout, async move {
tokio::task::spawn_blocking(move || {
let mut store =
Self::create_store(&runtime, &prepared, &capabilities, credentials_snapshot)?;
let mut store = Self::create_store(
&runtime,
&prepared,
&capabilities,
credentials_snapshot,
pairing_store,
)?;
let instance = Self::instantiate_component(&runtime, &prepared, &mut store)?;
let channel_iface = instance.near_agent_channel();
@@ -1170,6 +1315,7 @@ impl WasmChannel {
let prepared = Arc::clone(&self.prepared);
let capabilities = self.capabilities.clone();
let credentials = self.credentials.clone();
let pairing_store = self.pairing_store.clone();
let callback_timeout = self.runtime.config().callback_timeout;
let wit_update = status_to_wit(&status, metadata);
@@ -1189,6 +1335,7 @@ impl WasmChannel {
&prepared,
&capabilities,
&credentials,
pairing_store.clone(),
callback_timeout,
wit_update_clone,
)
@@ -1319,6 +1466,7 @@ impl WasmChannel {
let message_tx = self.message_tx.clone();
let rate_limiter = self.rate_limiter.clone();
let credentials = self.credentials.clone();
let pairing_store = self.pairing_store.clone();
let callback_timeout = self.runtime.config().callback_timeout;
tokio::spawn(async move {
@@ -1340,6 +1488,7 @@ impl WasmChannel {
&prepared,
&capabilities,
&credentials,
pairing_store.clone(),
callback_timeout,
).await;
@@ -1391,6 +1540,7 @@ impl WasmChannel {
prepared: &Arc<PreparedChannelModule>,
capabilities: &ChannelCapabilities,
credentials: &RwLock<HashMap<String, String>>,
pairing_store: Arc<PairingStore>,
timeout: Duration,
) -> Result<Vec<EmittedMessage>, WasmChannelError> {
// Skip if no WASM bytes (testing mode)
@@ -1411,8 +1561,13 @@ impl WasmChannel {
// Execute in blocking task with timeout
let result = tokio::time::timeout(timeout, async move {
tokio::task::spawn_blocking(move || {
let mut store =
Self::create_store(&runtime, &prepared, &capabilities, credentials_snapshot)?;
let mut store = Self::create_store(
&runtime,
&prepared,
&capabilities,
credentials_snapshot,
pairing_store,
)?;
let instance = Self::instantiate_component(&runtime, &prepared, &mut store)?;
// Call on_poll using the generated typed interface
@@ -1858,6 +2013,29 @@ fn status_to_wit(status: &StatusUpdate, metadata: &serde_json::Value) -> wit_cha
message: format!("Approval needed: {} - {}", tool_name, description),
metadata_json,
},
StatusUpdate::JobStarted { job_id, title, .. } => wit_channel::StatusUpdate {
status: wit_channel::StatusType::Thinking,
message: format!("Job started: {} ({})", title, job_id),
metadata_json,
},
StatusUpdate::AuthRequired { extension_name, .. } => wit_channel::StatusUpdate {
status: wit_channel::StatusType::Thinking,
message: format!("Auth required: {}", extension_name),
metadata_json,
},
StatusUpdate::AuthCompleted {
extension_name,
success,
..
} => wit_channel::StatusUpdate {
status: wit_channel::StatusType::Thinking,
message: format!(
"Auth {}: {}",
if *success { "completed" } else { "failed" },
extension_name
),
metadata_json,
},
}
}
@@ -1929,6 +2107,7 @@ mod tests {
PreparedChannelModule, WasmChannelRuntime, WasmChannelRuntimeConfig,
};
use crate::channels::wasm::wrapper::{HttpResponse, WasmChannel};
use crate::pairing::PairingStore;
use crate::tools::wasm::ResourceLimits;
fn create_test_channel() -> WasmChannel {
@@ -1944,7 +2123,13 @@ mod tests {
let capabilities = ChannelCapabilities::for_channel("test").with_path("/webhook/test");
WasmChannel::new(runtime, prepared, capabilities, "{}".to_string())
WasmChannel::new(
runtime,
prepared,
capabilities,
"{}".to_string(),
Arc::new(PairingStore::new()),
)
}
#[test]
@@ -2019,6 +2204,7 @@ mod tests {
&prepared,
&capabilities,
&credentials,
Arc::new(PairingStore::new()),
timeout,
)
.await;
@@ -2112,7 +2298,13 @@ mod tests {
.with_path("/webhook/poll")
.with_polling(1000);
let channel = WasmChannel::new(runtime, prepared, capabilities, "{}".to_string());
let channel = WasmChannel::new(
runtime,
prepared,
capabilities,
"{}".to_string(),
Arc::new(PairingStore::new()),
);
// Start the channel
let _stream = channel.start().await.expect("Channel should start");
@@ -2350,4 +2542,89 @@ mod tests {
assert_eq!(cloned.message, "hello");
assert_eq!(cloned.metadata_json, "{\"a\":1}");
}
#[test]
fn test_redact_credentials_replaces_values() {
use super::ChannelStoreData;
let mut creds = std::collections::HashMap::new();
creds.insert(
"TELEGRAM_BOT_TOKEN".to_string(),
"8218490433:AAEZeUxwqZ5OO3mOCXv7fKvpdhDgsmBBNis".to_string(),
);
creds.insert("OTHER_SECRET".to_string(), "s3cret".to_string());
let store = ChannelStoreData::new(
1024 * 1024,
"test",
ChannelCapabilities::default(),
creds,
Arc::new(PairingStore::new()),
);
let error = "HTTP request failed: error sending request for url \
(https://api.telegram.org/bot8218490433:AAEZeUxwqZ5OO3mOCXv7fKvpdhDgsmBBNis/getUpdates)";
let redacted = store.redact_credentials(error);
assert!(
!redacted.contains("8218490433:AAEZeUxwqZ5OO3mOCXv7fKvpdhDgsmBBNis"),
"credential value should be redacted"
);
assert!(
redacted.contains("[REDACTED:TELEGRAM_BOT_TOKEN]"),
"redacted text should contain placeholder name"
);
assert!(
!redacted.contains("s3cret"),
"other credentials should also be redacted"
);
}
#[test]
fn test_redact_credentials_no_op_without_credentials() {
use super::ChannelStoreData;
let store = ChannelStoreData::new(
1024 * 1024,
"test",
ChannelCapabilities::default(),
std::collections::HashMap::new(),
Arc::new(PairingStore::new()),
);
let input = "some error message";
assert_eq!(store.redact_credentials(input), input);
}
#[test]
fn test_redact_credentials_skips_empty_values() {
use super::ChannelStoreData;
let mut creds = std::collections::HashMap::new();
creds.insert("EMPTY_TOKEN".to_string(), String::new());
let store = ChannelStoreData::new(
1024 * 1024,
"test",
ChannelCapabilities::default(),
creds,
Arc::new(PairingStore::new()),
);
let input = "should not match anything";
assert_eq!(store.redact_credentials(input), input);
}
/// Verify that the block_on-inside-spawn_blocking pattern used by the WASM
/// channel HTTP host function doesn't deadlock or panic.
#[tokio::test]
async fn test_block_on_inside_spawn_blocking_does_not_deadlock() {
let result = tokio::task::spawn_blocking(|| {
tokio::runtime::Handle::current().block_on(async { 42 })
})
.await
.expect("spawn_blocking panicked");
assert_eq!(result, 42);
}
}
+5 -4
View File
@@ -6,6 +6,7 @@ use axum::{
middleware::Next,
response::{IntoResponse, Response},
};
use subtle::ConstantTimeEq;
/// Shared auth state injected via axum middleware state.
#[derive(Clone)]
@@ -23,22 +24,22 @@ pub async fn auth_middleware(
request: Request,
next: Next,
) -> Response {
// Try Authorization header first
// Try Authorization header first (constant-time comparison)
if let Some(auth_header) = headers.get("authorization") {
if let Ok(value) = auth_header.to_str() {
if let Some(token) = value.strip_prefix("Bearer ") {
if token == auth.token {
if bool::from(token.as_bytes().ct_eq(auth.token.as_bytes())) {
return next.run(request).await;
}
}
}
}
// Fall back to query parameter (for SSE EventSource)
// Fall back to query parameter for SSE EventSource (constant-time comparison)
if let Some(query) = request.uri().query() {
for pair in query.split('&') {
if let Some(token) = pair.strip_prefix("token=") {
if token == auth.token {
if bool::from(token.as_bytes().ct_eq(auth.token.as_bytes())) {
return next.run(request).await;
}
}
+43 -1
View File
@@ -24,6 +24,8 @@ use tokio::sync::broadcast;
use tracing::field::{Field, Visit};
use tracing_subscriber::Layer;
use crate::safety::LeakDetector;
/// Maximum number of recent log entries kept for late-joining SSE subscribers.
const HISTORY_CAP: usize = 500;
@@ -46,6 +48,8 @@ pub struct LogEntry {
pub struct LogBroadcaster {
tx: broadcast::Sender<LogEntry>,
recent: Mutex<VecDeque<LogEntry>>,
/// Scrubs secrets from log messages before broadcasting to SSE clients.
leak_detector: LeakDetector,
}
impl LogBroadcaster {
@@ -54,10 +58,19 @@ impl LogBroadcaster {
Self {
tx,
recent: Mutex::new(VecDeque::with_capacity(HISTORY_CAP)),
leak_detector: LeakDetector::new(),
}
}
pub fn send(&self, entry: LogEntry) {
pub fn send(&self, mut entry: LogEntry) {
// Scrub secrets from the message before it reaches any subscriber.
// This is defense-in-depth: even if code elsewhere accidentally logs
// a secret, it won't be broadcast to SSE clients.
entry.message = self
.leak_detector
.scan_and_clean(&entry.message)
.unwrap_or_else(|_| "[log message redacted: contained blocked secret]".to_string());
// Stash in ring buffer (for late joiners)
if let Ok(mut buf) = self.recent.lock() {
if buf.len() >= HISTORY_CAP {
@@ -145,6 +158,9 @@ impl Visit for MessageVisitor {
///
/// Only forwards DEBUG and above. Attach to the tracing subscriber
/// alongside the existing fmt layer.
///
/// Log messages are scrubbed through `LeakDetector` in `LogBroadcaster::send()`
/// (the single funnel point for all log output, including late-joiner history).
pub struct WebLogLayer {
broadcaster: Arc<LogBroadcaster>,
}
@@ -178,6 +194,7 @@ impl<S: tracing::Subscriber> Layer<S> for WebLogLayer {
timestamp: chrono::Utc::now().to_rfc3339_opts(chrono::SecondsFormat::Millis, true),
};
// LeakDetector scrubbing happens inside broadcaster.send()
self.broadcaster.send(entry);
}
}
@@ -313,4 +330,29 @@ mod tests {
let v = MessageVisitor::new();
assert_eq!(v.finish(), "");
}
#[test]
fn test_broadcaster_has_leak_detector() {
let broadcaster = LogBroadcaster::new();
// Verify the leak detector is initialized with default patterns
assert!(broadcaster.leak_detector.pattern_count() > 0);
}
#[test]
fn test_leak_detector_scrubs_api_key_in_log() {
let detector = crate::safety::LeakDetector::new();
let msg = "Connecting with token sk-proj-test1234567890abcdefghij";
let result = detector.scan_and_clean(msg);
// Should be blocked (OpenAI key pattern)
assert!(result.is_err());
}
#[test]
fn test_leak_detector_passes_clean_log() {
let detector = crate::safety::LeakDetector::new();
let msg = "Request completed status=200 url=https://api.example.com/data";
let result = detector.scan_and_clean(msg);
assert!(result.is_ok());
assert_eq!(result.unwrap(), msg);
}
}
+109 -24
View File
@@ -10,12 +10,13 @@
//! ◄── GET /api/chat/events ── SSE stream
//! ─── GET /api/chat/ws ─────► WebSocket (bidirectional)
//! ─── GET /api/memory/* ────► Workspace
//! ─── GET /api/jobs/* ──────► ContextManager
//! ─── GET /api/jobs/* ──────► Database
//! ◄── GET / ───────────────── Static HTML/CSS/JS
//! ```
pub mod auth;
pub mod log_layer;
pub mod openai_compat;
pub mod server;
pub mod sse;
pub mod types;
@@ -31,9 +32,10 @@ use tokio_stream::wrappers::ReceiverStream;
use crate::agent::SessionManager;
use crate::channels::{Channel, IncomingMessage, MessageStream, OutgoingResponse, StatusUpdate};
use crate::config::GatewayConfig;
use crate::context::ContextManager;
use crate::db::Database;
use crate::error::ChannelError;
use crate::extensions::ExtensionManager;
use crate::orchestrator::job_manager::ContainerJobManager;
use crate::tools::ToolRegistry;
use crate::workspace::Workspace;
@@ -70,14 +72,18 @@ impl GatewayChannel {
msg_tx: tokio::sync::RwLock::new(None),
sse: SseManager::new(),
workspace: None,
context_manager: None,
session_manager: None,
log_broadcaster: None,
extension_manager: None,
tool_registry: None,
store: None,
job_manager: None,
prompt_queue: None,
user_id: config.user_id.clone(),
shutdown_tx: tokio::sync::RwLock::new(None),
ws_tracker: Some(Arc::new(ws::WsConnectionTracker::new())),
llm_provider: None,
chat_rate_limiter: server::RateLimiter::new(30, 60),
});
Self {
@@ -93,14 +99,18 @@ impl GatewayChannel {
msg_tx: tokio::sync::RwLock::new(None),
sse: SseManager::new(),
workspace: self.state.workspace.clone(),
context_manager: self.state.context_manager.clone(),
session_manager: self.state.session_manager.clone(),
log_broadcaster: self.state.log_broadcaster.clone(),
extension_manager: self.state.extension_manager.clone(),
tool_registry: self.state.tool_registry.clone(),
store: self.state.store.clone(),
job_manager: self.state.job_manager.clone(),
prompt_queue: self.state.prompt_queue.clone(),
user_id: self.state.user_id.clone(),
shutdown_tx: tokio::sync::RwLock::new(None),
ws_tracker: self.state.ws_tracker.clone(),
llm_provider: self.state.llm_provider.clone(),
chat_rate_limiter: server::RateLimiter::new(30, 60),
};
mutate(&mut new_state);
self.state = Arc::new(new_state);
@@ -112,12 +122,6 @@ impl GatewayChannel {
self
}
/// Inject the context manager for the jobs API.
pub fn with_context_manager(mut self, cm: Arc<ContextManager>) -> Self {
self.rebuild_state(|s| s.context_manager = Some(cm));
self
}
/// Inject the session manager for thread/session info.
pub fn with_session_manager(mut self, sm: Arc<SessionManager>) -> Self {
self.rebuild_state(|s| s.session_manager = Some(sm));
@@ -142,6 +146,40 @@ impl GatewayChannel {
self
}
/// Inject the database store for sandbox job persistence.
pub fn with_store(mut self, store: Arc<dyn Database>) -> Self {
self.rebuild_state(|s| s.store = Some(store));
self
}
/// Inject the container job manager for sandbox operations.
pub fn with_job_manager(mut self, jm: Arc<ContainerJobManager>) -> Self {
self.rebuild_state(|s| s.job_manager = Some(jm));
self
}
/// Inject the prompt queue for Claude Code follow-up prompts.
pub fn with_prompt_queue(
mut self,
pq: Arc<
tokio::sync::Mutex<
std::collections::HashMap<
uuid::Uuid,
std::collections::VecDeque<crate::orchestrator::api::PendingPrompt>,
>,
>,
>,
) -> Self {
self.rebuild_state(|s| s.prompt_queue = Some(pq));
self
}
/// Inject the LLM provider for OpenAI-compatible API proxy.
pub fn with_llm_provider(mut self, llm: Arc<dyn crate::llm::LlmProvider>) -> Self {
self.rebuild_state(|s| s.llm_provider = Some(llm));
self
}
/// Get the auth token (for printing to console on startup).
pub fn auth_token(&self) -> &str {
&self.auth_token
@@ -173,11 +211,7 @@ impl Channel for GatewayChannel {
),
})?;
let bound_addr =
server::start_server(addr, self.state.clone(), self.auth_token.clone()).await?;
tracing::info!("Web gateway listening on http://{}", bound_addr);
tracing::info!("Auth token: {}", self.auth_token);
server::start_server(addr, self.state.clone(), self.auth_token.clone()).await?;
Ok(Box::pin(ReceiverStream::new(rx)))
}
@@ -200,17 +234,48 @@ impl Channel for GatewayChannel {
async fn send_status(
&self,
status: StatusUpdate,
_metadata: &serde_json::Value,
metadata: &serde_json::Value,
) -> Result<(), ChannelError> {
let thread_id = metadata
.get("thread_id")
.and_then(|v| v.as_str())
.map(String::from);
let event = match status {
StatusUpdate::Thinking(msg) => SseEvent::Thinking { message: msg },
StatusUpdate::ToolStarted { name } => SseEvent::ToolStarted { name },
StatusUpdate::ToolCompleted { name, success } => {
SseEvent::ToolCompleted { name, success }
}
StatusUpdate::ToolResult { name, preview } => SseEvent::ToolResult { name, preview },
StatusUpdate::StreamChunk(content) => SseEvent::StreamChunk { content },
StatusUpdate::Status(msg) => SseEvent::Status { message: msg },
StatusUpdate::Thinking(msg) => SseEvent::Thinking {
message: msg,
thread_id: thread_id.clone(),
},
StatusUpdate::ToolStarted { name } => SseEvent::ToolStarted {
name,
thread_id: thread_id.clone(),
},
StatusUpdate::ToolCompleted { name, success } => SseEvent::ToolCompleted {
name,
success,
thread_id: thread_id.clone(),
},
StatusUpdate::ToolResult { name, preview } => SseEvent::ToolResult {
name,
preview,
thread_id: thread_id.clone(),
},
StatusUpdate::StreamChunk(content) => SseEvent::StreamChunk {
content,
thread_id: thread_id.clone(),
},
StatusUpdate::Status(msg) => SseEvent::Status {
message: msg,
thread_id: thread_id.clone(),
},
StatusUpdate::JobStarted {
job_id,
title,
browse_url,
} => SseEvent::JobStarted {
job_id,
title,
browse_url,
},
StatusUpdate::ApprovalNeeded {
request_id,
tool_name,
@@ -223,6 +288,26 @@ impl Channel for GatewayChannel {
parameters: serde_json::to_string_pretty(&parameters)
.unwrap_or_else(|_| parameters.to_string()),
},
StatusUpdate::AuthRequired {
extension_name,
instructions,
auth_url,
setup_url,
} => SseEvent::AuthRequired {
extension_name,
instructions,
auth_url,
setup_url,
},
StatusUpdate::AuthCompleted {
extension_name,
success,
message,
} => SseEvent::AuthCompleted {
extension_name,
success,
message,
},
};
self.state.sse.broadcast(event);
File diff suppressed because it is too large Load Diff
+1550 -98
View File
File diff suppressed because it is too large Load Diff
+71 -14
View File
@@ -13,10 +13,15 @@ use tokio_stream::wrappers::BroadcastStream;
use crate::channels::web::types::SseEvent;
/// Maximum number of concurrent SSE/WebSocket connections.
/// Prevents resource exhaustion from connection flooding.
const MAX_CONNECTIONS: u64 = 100;
/// Manages SSE broadcast to all connected browser tabs.
pub struct SseManager {
tx: broadcast::Sender<SseEvent>,
connection_count: Arc<AtomicU64>,
max_connections: u64,
}
impl SseManager {
@@ -27,6 +32,7 @@ impl SseManager {
Self {
tx,
connection_count: Arc::new(AtomicU64::new(0)),
max_connections: MAX_CONNECTIONS,
}
}
@@ -45,25 +51,50 @@ impl SseManager {
///
/// Returns a stream of `SseEvent` values and increments/decrements the
/// connection counter on creation/drop, just like `subscribe()` does for SSE.
pub fn subscribe_raw(&self) -> impl Stream<Item = SseEvent> + Send + 'static + use<> {
///
/// Returns `None` if the maximum connection limit has been reached.
pub fn subscribe_raw(&self) -> Option<impl Stream<Item = SseEvent> + Send + 'static + use<>> {
// Atomically increment only if below the limit. This prevents
// concurrent callers from overshooting max_connections.
let counter = Arc::clone(&self.connection_count);
counter.fetch_add(1, Ordering::Relaxed);
let max = self.max_connections;
counter
.fetch_update(Ordering::Relaxed, Ordering::Relaxed, |current| {
if current < max {
Some(current + 1)
} else {
None
}
})
.ok()?;
let rx = self.tx.subscribe();
let stream = BroadcastStream::new(rx).filter_map(|result| result.ok());
CountedStream {
Some(CountedStream {
inner: stream,
counter,
}
})
}
/// Create a new SSE stream for a client connection.
///
/// Returns `None` if the maximum connection limit has been reached.
pub fn subscribe(
&self,
) -> Sse<impl Stream<Item = Result<Event, Infallible>> + Send + 'static + use<>> {
) -> Option<Sse<impl Stream<Item = Result<Event, Infallible>> + Send + 'static + use<>>> {
// Atomically increment only if below the limit.
let counter = Arc::clone(&self.connection_count);
counter.fetch_add(1, Ordering::Relaxed);
let max = self.max_connections;
counter
.fetch_update(Ordering::Relaxed, Ordering::Relaxed, |current| {
if current < max {
Some(current + 1)
} else {
None
}
})
.ok()?;
let rx = self.tx.subscribe();
let stream = BroadcastStream::new(rx)
@@ -79,7 +110,15 @@ impl SseManager {
SseEvent::StreamChunk { .. } => "stream_chunk",
SseEvent::Status { .. } => "status",
SseEvent::ApprovalNeeded { .. } => "approval_needed",
SseEvent::AuthRequired { .. } => "auth_required",
SseEvent::AuthCompleted { .. } => "auth_completed",
SseEvent::Error { .. } => "error",
SseEvent::JobStarted { .. } => "job_started",
SseEvent::JobMessage { .. } => "job_message",
SseEvent::JobToolUse { .. } => "job_tool_use",
SseEvent::JobToolResult { .. } => "job_tool_result",
SseEvent::JobStatus { .. } => "job_status",
SseEvent::JobResult { .. } => "job_result",
SseEvent::Heartbeat => "heartbeat",
};
Ok(Event::default().event(event_type).data(data))
@@ -91,8 +130,10 @@ impl SseManager {
counter,
};
Sse::new(counted_stream)
.keep_alive(KeepAlive::new().interval(Duration::from_secs(30)).text(""))
Some(
Sse::new(counted_stream)
.keep_alive(KeepAlive::new().interval(Duration::from_secs(30)).text("")),
)
}
}
@@ -152,13 +193,14 @@ mod tests {
manager.broadcast(SseEvent::Status {
message: "test".to_string(),
thread_id: None,
});
let event = rx.next().await;
assert!(event.is_some());
let event = event.unwrap().unwrap();
match event {
SseEvent::Status { message } => assert_eq!(message, "test"),
SseEvent::Status { message, .. } => assert_eq!(message, "test"),
_ => panic!("unexpected event type"),
}
}
@@ -166,17 +208,18 @@ mod tests {
#[tokio::test]
async fn test_subscribe_raw_receives_events() {
let manager = SseManager::new();
let mut stream = Box::pin(manager.subscribe_raw());
let mut stream = Box::pin(manager.subscribe_raw().expect("should subscribe"));
assert_eq!(manager.connection_count(), 1);
manager.broadcast(SseEvent::Thinking {
message: "working".to_string(),
thread_id: None,
});
let event = stream.next().await.unwrap();
match event {
SseEvent::Thinking { message } => assert_eq!(message, "working"),
SseEvent::Thinking { message, .. } => assert_eq!(message, "working"),
_ => panic!("Expected Thinking event"),
}
}
@@ -185,7 +228,7 @@ mod tests {
async fn test_subscribe_raw_decrements_on_drop() {
let manager = SseManager::new();
{
let _stream = Box::pin(manager.subscribe_raw());
let _stream = Box::pin(manager.subscribe_raw().expect("should subscribe"));
assert_eq!(manager.connection_count(), 1);
}
// Stream dropped, counter should decrement
@@ -195,8 +238,8 @@ mod tests {
#[tokio::test]
async fn test_subscribe_raw_multiple_subscribers() {
let manager = SseManager::new();
let mut s1 = Box::pin(manager.subscribe_raw());
let mut s2 = Box::pin(manager.subscribe_raw());
let mut s1 = Box::pin(manager.subscribe_raw().expect("should subscribe"));
let mut s2 = Box::pin(manager.subscribe_raw().expect("should subscribe"));
assert_eq!(manager.connection_count(), 2);
manager.broadcast(SseEvent::Heartbeat);
@@ -211,4 +254,18 @@ mod tests {
drop(s2);
assert_eq!(manager.connection_count(), 0);
}
#[tokio::test]
async fn test_subscribe_raw_rejects_over_limit() {
let mut manager = SseManager::new();
manager.max_connections = 2; // Low limit for testing
let _s1 = Box::pin(manager.subscribe_raw().expect("first should succeed"));
let _s2 = Box::pin(manager.subscribe_raw().expect("second should succeed"));
assert_eq!(manager.connection_count(), 2);
// Third should be rejected
assert!(manager.subscribe_raw().is_none());
assert!(manager.subscribe().is_none());
}
}
File diff suppressed because it is too large Load Diff
+82 -7
View File
@@ -10,12 +10,19 @@
<body>
<!-- Auth Screen -->
<div id="auth-screen">
<h1>IronClaw</h1>
<div class="auth-form">
<input type="password" id="token-input" placeholder="Auth token" autofocus>
<button onclick="authenticate()">Connect</button>
<div class="auth-card-login">
<div class="auth-brand">
<h1>IronClaw</h1>
<p class="auth-tagline">Secure AI Assistant</p>
</div>
<div class="auth-form">
<label for="token-input">Gateway Token</label>
<input type="password" id="token-input" placeholder="Paste your auth token" autofocus>
<button onclick="authenticate()">Connect</button>
</div>
<div id="auth-error"></div>
<p class="auth-hint">Enter the GATEWAY_AUTH_TOKEN from your .env configuration.</p>
</div>
<div id="auth-error"></div>
</div>
<!-- Main App (hidden until authenticated) -->
@@ -26,16 +33,33 @@
<button data-tab="memory">Memory</button>
<button data-tab="jobs">Jobs</button>
<button data-tab="logs">Logs</button>
<button data-tab="routines">Routines</button>
<button data-tab="extensions">Extensions</button>
<div class="spacer"></div>
<div class="status">
<div class="status" id="gateway-status-trigger">
<div class="dot" id="sse-dot"></div>
<span id="sse-status">Connected</span>
<div class="gateway-popover" id="gateway-popover"></div>
</div>
</div>
<!-- Chat Tab -->
<div class="tab-panel active" id="tab-chat">
<div class="thread-sidebar" id="thread-sidebar">
<div class="thread-sidebar-header">
<span>Threads</span>
<button class="thread-new-btn" onclick="createNewThread()" title="New thread (Ctrl/Cmd+N)">+</button>
<button class="thread-toggle-btn" id="thread-toggle-btn" onclick="toggleThreadSidebar()" title="Toggle sidebar">&laquo;</button>
</div>
<div class="assistant-item" id="assistant-thread" onclick="switchToAssistant()">
<span class="assistant-label">Assistant</span>
<span class="assistant-meta" id="assistant-meta"></span>
</div>
<div class="threads-section-header">
<span>Conversations</span>
</div>
<div class="thread-list" id="thread-list"></div>
</div>
<div class="chat-container">
<div class="chat-messages" id="chat-messages"></div>
<div class="chat-status" id="chat-status"></div>
@@ -56,10 +80,20 @@
<div class="memory-tree" id="memory-tree"></div>
</div>
<div class="memory-content">
<div class="memory-breadcrumb" id="memory-breadcrumb">workspace /</div>
<div class="memory-breadcrumb" id="memory-breadcrumb">
<span id="memory-breadcrumb-path">workspace /</span>
<button class="memory-edit-btn" id="memory-edit-btn" style="display:none" onclick="startMemoryEdit()">Edit</button>
</div>
<div class="memory-viewer" id="memory-viewer">
<div class="empty">Select a file to view its contents</div>
</div>
<div class="memory-editor" id="memory-editor" style="display:none">
<textarea id="memory-edit-textarea"></textarea>
<div class="memory-editor-actions">
<button class="btn-save" onclick="saveMemoryEdit()">Save</button>
<button class="btn-cancel-edit" onclick="cancelMemoryEdit()">Cancel</button>
</div>
</div>
</div>
</div>
</div>
@@ -73,6 +107,7 @@
<tr>
<th>ID</th>
<th>Title</th>
<th>Source</th>
<th>Status</th>
<th>Created</th>
<th>Actions</th>
@@ -104,9 +139,48 @@
</div>
</div>
<!-- Routines Tab -->
<div class="tab-panel" id="tab-routines">
<div class="routines-container">
<div class="routines-summary" id="routines-summary"></div>
<table class="routines-table" id="routines-table">
<thead>
<tr>
<th>Name</th>
<th>Trigger</th>
<th>Action</th>
<th>Last Run</th>
<th>Next Run</th>
<th>Runs</th>
<th>Status</th>
<th>Actions</th>
</tr>
</thead>
<tbody id="routines-tbody"></tbody>
</table>
<div class="empty-state" id="routines-empty" style="display:none">
No routines configured. Ask the assistant to create one.
</div>
<div class="routine-detail" id="routine-detail" style="display:none"></div>
</div>
</div>
<!-- Extensions Tab -->
<div class="tab-panel" id="tab-extensions">
<div class="extensions-container">
<div class="extensions-section">
<h3>Install Extension</h3>
<div class="ext-install-form" id="ext-install-form">
<input type="text" id="ext-install-name" placeholder="Extension name (required)">
<input type="text" id="ext-install-url" placeholder="URL (optional)">
<select id="ext-install-kind">
<option value="mcp_server">MCP Server</option>
<option value="wasm_tool">WASM Tool</option>
<option value="wasm_channel">WASM Channel</option>
</select>
<button onclick="installExtension()">Install</button>
</div>
</div>
<div class="extensions-section">
<h3>Installed Extensions</h3>
<div class="extensions-list" id="extensions-list">
@@ -130,6 +204,7 @@
</div>
</div>
<div id="toasts"></div>
<script src="/app.js"></script>
</body>
</html>
File diff suppressed because it is too large Load Diff
+414 -9
View File
@@ -24,10 +24,17 @@ pub struct ThreadInfo {
pub turn_count: usize,
pub created_at: String,
pub updated_at: String,
#[serde(skip_serializing_if = "Option::is_none")]
pub title: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub thread_type: Option<String>,
}
#[derive(Debug, Serialize)]
pub struct ThreadListResponse {
/// The pinned assistant thread (always present after first load).
pub assistant_thread: Option<ThreadInfo>,
/// Regular conversation threads.
pub threads: Vec<ThreadInfo>,
pub active_thread: Option<Uuid>,
}
@@ -54,6 +61,12 @@ pub struct ToolCallInfo {
pub struct HistoryResponse {
pub thread_id: Uuid,
pub turns: Vec<TurnInfo>,
/// Whether there are older messages available.
#[serde(default)]
pub has_more: bool,
/// Cursor for the next page (ISO8601 timestamp of the oldest message returned).
#[serde(skip_serializing_if = "Option::is_none")]
pub oldest_timestamp: Option<String>,
}
// --- Approval ---
@@ -63,6 +76,8 @@ pub struct ApprovalRequest {
pub request_id: String,
/// "approve", "always", or "deny"
pub action: String,
/// Thread that owns the pending approval (so the agent loop finds the right session).
pub thread_id: Option<String>,
}
// --- SSE Event Types ---
@@ -73,17 +88,49 @@ pub enum SseEvent {
#[serde(rename = "response")]
Response { content: String, thread_id: String },
#[serde(rename = "thinking")]
Thinking { message: String },
Thinking {
message: String,
#[serde(skip_serializing_if = "Option::is_none")]
thread_id: Option<String>,
},
#[serde(rename = "tool_started")]
ToolStarted { name: String },
ToolStarted {
name: String,
#[serde(skip_serializing_if = "Option::is_none")]
thread_id: Option<String>,
},
#[serde(rename = "tool_completed")]
ToolCompleted { name: String, success: bool },
ToolCompleted {
name: String,
success: bool,
#[serde(skip_serializing_if = "Option::is_none")]
thread_id: Option<String>,
},
#[serde(rename = "tool_result")]
ToolResult { name: String, preview: String },
ToolResult {
name: String,
preview: String,
#[serde(skip_serializing_if = "Option::is_none")]
thread_id: Option<String>,
},
#[serde(rename = "stream_chunk")]
StreamChunk { content: String },
StreamChunk {
content: String,
#[serde(skip_serializing_if = "Option::is_none")]
thread_id: Option<String>,
},
#[serde(rename = "status")]
Status { message: String },
Status {
message: String,
#[serde(skip_serializing_if = "Option::is_none")]
thread_id: Option<String>,
},
#[serde(rename = "job_started")]
JobStarted {
job_id: String,
title: String,
browse_url: String,
},
#[serde(rename = "approval_needed")]
ApprovalNeeded {
request_id: String,
@@ -91,10 +138,59 @@ pub enum SseEvent {
description: String,
parameters: String,
},
#[serde(rename = "auth_required")]
AuthRequired {
extension_name: String,
#[serde(skip_serializing_if = "Option::is_none")]
instructions: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
auth_url: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
setup_url: Option<String>,
},
#[serde(rename = "auth_completed")]
AuthCompleted {
extension_name: String,
success: bool,
message: String,
},
#[serde(rename = "error")]
Error { message: String },
Error {
message: String,
#[serde(skip_serializing_if = "Option::is_none")]
thread_id: Option<String>,
},
#[serde(rename = "heartbeat")]
Heartbeat,
// Sandbox job streaming events (worker + Claude Code bridge)
#[serde(rename = "job_message")]
JobMessage {
job_id: String,
role: String,
content: String,
},
#[serde(rename = "job_tool_use")]
JobToolUse {
job_id: String,
tool_name: String,
input: serde_json::Value,
},
#[serde(rename = "job_tool_result")]
JobToolResult {
job_id: String,
tool_name: String,
output: String,
},
#[serde(rename = "job_status")]
JobStatus { job_id: String, message: String },
#[serde(rename = "job_result")]
JobResult {
job_id: String,
status: String,
#[serde(skip_serializing_if = "Option::is_none")]
session_id: Option<String>,
},
}
// --- Memory ---
@@ -188,6 +284,54 @@ pub struct JobSummaryResponse {
pub stuck: usize,
}
#[derive(Debug, Serialize)]
pub struct JobDetailResponse {
pub id: Uuid,
pub title: String,
pub description: String,
pub state: String,
pub user_id: String,
pub created_at: String,
pub started_at: Option<String>,
pub completed_at: Option<String>,
pub elapsed_secs: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub project_dir: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub browse_url: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub job_mode: Option<String>,
pub transitions: Vec<TransitionInfo>,
}
// --- Project Files ---
#[derive(Debug, Serialize)]
pub struct ProjectFileEntry {
pub name: String,
pub path: String,
pub is_dir: bool,
}
#[derive(Debug, Serialize)]
pub struct ProjectFilesResponse {
pub entries: Vec<ProjectFileEntry>,
}
#[derive(Debug, Serialize)]
pub struct ProjectFileReadResponse {
pub path: String,
pub content: String,
}
#[derive(Debug, Serialize)]
pub struct TransitionInfo {
pub from: String,
pub to: String,
pub timestamp: String,
pub reason: Option<String>,
}
// --- Extensions ---
#[derive(Debug, Serialize)]
@@ -262,6 +406,21 @@ impl ActionResponse {
}
}
// --- Auth Token ---
/// Request to submit an auth token for an extension (dedicated endpoint).
#[derive(Debug, Deserialize)]
pub struct AuthTokenRequest {
pub extension_name: String,
pub token: String,
}
/// Request to cancel an in-progress auth flow.
#[derive(Debug, Deserialize)]
pub struct AuthCancelRequest {
pub extension_name: String,
}
// --- WebSocket ---
/// Message sent by a WebSocket client to the server.
@@ -280,7 +439,18 @@ pub enum WsClientMessage {
request_id: String,
/// "approve", "always", or "deny"
action: String,
/// Thread that owns the pending approval.
thread_id: Option<String>,
},
/// Submit an auth token for an extension (bypasses message pipeline).
#[serde(rename = "auth_token")]
AuthToken {
extension_name: String,
token: String,
},
/// Cancel an in-progress auth flow.
#[serde(rename = "auth_cancel")]
AuthCancel { extension_name: String },
/// Client heartbeat ping.
#[serde(rename = "ping")]
Ping,
@@ -314,11 +484,20 @@ impl WsServerMessage {
SseEvent::Thinking { .. } => "thinking",
SseEvent::ToolStarted { .. } => "tool_started",
SseEvent::ToolCompleted { .. } => "tool_completed",
SseEvent::ToolResult { .. } => "tool_result",
SseEvent::StreamChunk { .. } => "stream_chunk",
SseEvent::Status { .. } => "status",
SseEvent::JobStarted { .. } => "job_started",
SseEvent::ApprovalNeeded { .. } => "approval_needed",
SseEvent::AuthRequired { .. } => "auth_required",
SseEvent::AuthCompleted { .. } => "auth_completed",
SseEvent::Error { .. } => "error",
SseEvent::Heartbeat => "heartbeat",
SseEvent::JobMessage { .. } => "job_message",
SseEvent::JobToolUse { .. } => "job_tool_use",
SseEvent::JobToolResult { .. } => "job_tool_result",
SseEvent::JobStatus { .. } => "job_status",
SseEvent::JobResult { .. } => "job_result",
};
let data = serde_json::to_value(event).unwrap_or(serde_json::Value::Null);
WsServerMessage::Event {
@@ -328,6 +507,96 @@ impl WsServerMessage {
}
}
// --- Routines ---
#[derive(Debug, Serialize)]
pub struct RoutineInfo {
pub id: Uuid,
pub name: String,
pub description: String,
pub enabled: bool,
pub trigger_type: String,
pub trigger_summary: String,
pub action_type: String,
pub last_run_at: Option<String>,
pub next_fire_at: Option<String>,
pub run_count: u64,
pub consecutive_failures: u32,
pub status: String,
}
#[derive(Debug, Serialize)]
pub struct RoutineListResponse {
pub routines: Vec<RoutineInfo>,
}
#[derive(Debug, Serialize)]
pub struct RoutineSummaryResponse {
pub total: u64,
pub enabled: u64,
pub disabled: u64,
pub failing: u64,
pub runs_today: u64,
}
#[derive(Debug, Serialize)]
pub struct RoutineDetailResponse {
pub id: Uuid,
pub name: String,
pub description: String,
pub enabled: bool,
pub trigger: serde_json::Value,
pub action: serde_json::Value,
pub guardrails: serde_json::Value,
pub notify: serde_json::Value,
pub last_run_at: Option<String>,
pub next_fire_at: Option<String>,
pub run_count: u64,
pub consecutive_failures: u32,
pub created_at: String,
pub recent_runs: Vec<RoutineRunInfo>,
}
#[derive(Debug, Serialize)]
pub struct RoutineRunInfo {
pub id: Uuid,
pub trigger_type: String,
pub started_at: String,
pub completed_at: Option<String>,
pub status: String,
pub result_summary: Option<String>,
pub tokens_used: Option<i32>,
}
// --- Settings ---
#[derive(Debug, Serialize)]
pub struct SettingResponse {
pub key: String,
pub value: serde_json::Value,
pub updated_at: String,
}
#[derive(Debug, Serialize)]
pub struct SettingsListResponse {
pub settings: Vec<SettingResponse>,
}
#[derive(Debug, Deserialize)]
pub struct SettingWriteRequest {
pub value: serde_json::Value,
}
#[derive(Debug, Deserialize)]
pub struct SettingsImportRequest {
pub settings: std::collections::HashMap<String, serde_json::Value>,
}
#[derive(Debug, Serialize)]
pub struct SettingsExportResponse {
pub settings: std::collections::HashMap<String, serde_json::Value>,
}
// --- Health ---
#[derive(Debug, Serialize)]
@@ -370,12 +639,36 @@ mod tests {
#[test]
fn test_ws_client_approval_parse() {
let json = r#"{"type":"approval","request_id":"abc-123","action":"approve"}"#;
let json =
r#"{"type":"approval","request_id":"abc-123","action":"approve","thread_id":"t1"}"#;
let msg: WsClientMessage = serde_json::from_str(json).unwrap();
match msg {
WsClientMessage::Approval { request_id, action } => {
WsClientMessage::Approval {
request_id,
action,
thread_id,
} => {
assert_eq!(request_id, "abc-123");
assert_eq!(action, "approve");
assert_eq!(thread_id.as_deref(), Some("t1"));
}
_ => panic!("Expected Approval variant"),
}
}
#[test]
fn test_ws_client_approval_parse_no_thread() {
let json = r#"{"type":"approval","request_id":"abc-123","action":"deny"}"#;
let msg: WsClientMessage = serde_json::from_str(json).unwrap();
match msg {
WsClientMessage::Approval {
request_id,
action,
thread_id,
} => {
assert_eq!(request_id, "abc-123");
assert_eq!(action, "deny");
assert!(thread_id.is_none());
}
_ => panic!("Expected Approval variant"),
}
@@ -436,6 +729,7 @@ mod tests {
fn test_ws_server_from_sse_thinking() {
let sse = SseEvent::Thinking {
message: "reasoning...".to_string(),
thread_id: None,
};
let ws = WsServerMessage::from_sse_event(&sse);
match ws {
@@ -476,4 +770,115 @@ mod tests {
_ => panic!("Expected Event variant"),
}
}
// ---- Auth type tests ----
#[test]
fn test_ws_client_auth_token_parse() {
let json = r#"{"type":"auth_token","extension_name":"notion","token":"sk-123"}"#;
let msg: WsClientMessage = serde_json::from_str(json).unwrap();
match msg {
WsClientMessage::AuthToken {
extension_name,
token,
} => {
assert_eq!(extension_name, "notion");
assert_eq!(token, "sk-123");
}
_ => panic!("Expected AuthToken variant"),
}
}
#[test]
fn test_ws_client_auth_cancel_parse() {
let json = r#"{"type":"auth_cancel","extension_name":"notion"}"#;
let msg: WsClientMessage = serde_json::from_str(json).unwrap();
match msg {
WsClientMessage::AuthCancel { extension_name } => {
assert_eq!(extension_name, "notion");
}
_ => panic!("Expected AuthCancel variant"),
}
}
#[test]
fn test_sse_auth_required_serialize() {
let event = SseEvent::AuthRequired {
extension_name: "notion".to_string(),
instructions: Some("Get your token from...".to_string()),
auth_url: None,
setup_url: Some("https://notion.so/integrations".to_string()),
};
let json = serde_json::to_string(&event).unwrap();
let parsed: serde_json::Value = serde_json::from_str(&json).unwrap();
assert_eq!(parsed["type"], "auth_required");
assert_eq!(parsed["extension_name"], "notion");
assert_eq!(parsed["instructions"], "Get your token from...");
assert!(parsed.get("auth_url").is_none());
assert_eq!(parsed["setup_url"], "https://notion.so/integrations");
}
#[test]
fn test_sse_auth_completed_serialize() {
let event = SseEvent::AuthCompleted {
extension_name: "notion".to_string(),
success: true,
message: "notion authenticated (3 tools loaded)".to_string(),
};
let json = serde_json::to_string(&event).unwrap();
let parsed: serde_json::Value = serde_json::from_str(&json).unwrap();
assert_eq!(parsed["type"], "auth_completed");
assert_eq!(parsed["extension_name"], "notion");
assert_eq!(parsed["success"], true);
}
#[test]
fn test_ws_server_from_sse_auth_required() {
let sse = SseEvent::AuthRequired {
extension_name: "openai".to_string(),
instructions: Some("Enter API key".to_string()),
auth_url: None,
setup_url: None,
};
let ws = WsServerMessage::from_sse_event(&sse);
match ws {
WsServerMessage::Event { event_type, data } => {
assert_eq!(event_type, "auth_required");
assert_eq!(data["extension_name"], "openai");
}
_ => panic!("Expected Event variant"),
}
}
#[test]
fn test_ws_server_from_sse_auth_completed() {
let sse = SseEvent::AuthCompleted {
extension_name: "slack".to_string(),
success: false,
message: "Invalid token".to_string(),
};
let ws = WsServerMessage::from_sse_event(&sse);
match ws {
WsServerMessage::Event { event_type, data } => {
assert_eq!(event_type, "auth_completed");
assert_eq!(data["success"], false);
}
_ => panic!("Expected Event variant"),
}
}
#[test]
fn test_auth_token_request_deserialize() {
let json = r#"{"extension_name":"telegram","token":"bot12345"}"#;
let req: AuthTokenRequest = serde_json::from_str(json).unwrap();
assert_eq!(req.extension_name, "telegram");
assert_eq!(req.token, "bot12345");
}
#[test]
fn test_auth_cancel_request_deserialize() {
let json = r#"{"extension_name":"telegram"}"#;
let req: AuthCancelRequest = serde_json::from_str(json).unwrap();
assert_eq!(req.extension_name, "telegram");
}
}
+86 -5
View File
@@ -71,8 +71,17 @@ pub async fn handle_ws_connection(socket: WebSocket, state: Arc<GatewayState>) {
}
let tracker_for_drop = state.ws_tracker.clone();
// Subscribe to broadcast events (same source as SSE)
let mut event_stream = Box::pin(state.sse.subscribe_raw());
// Subscribe to broadcast events (same source as SSE).
// Reject if we've hit the connection limit.
let Some(raw_stream) = state.sse.subscribe_raw() else {
tracing::warn!("WebSocket rejected: too many connections");
// Decrement the WS tracker we already incremented above.
if let Some(ref tracker) = tracker_for_drop {
tracker.decrement();
}
return;
};
let mut event_stream = Box::pin(raw_stream);
// Channel for the sender task to receive messages from both
// the broadcast stream and any direct sends (like Pong)
@@ -170,7 +179,11 @@ async fn handle_client_message(
.await;
}
}
WsClientMessage::Approval { request_id, action } => {
WsClientMessage::Approval {
request_id,
action,
thread_id,
} => {
let (approved, always) = match action.as_str() {
"approve" => (true, false),
"always" => (true, true),
@@ -214,12 +227,71 @@ async fn handle_client_message(
}
};
let msg = IncomingMessage::new("gateway", user_id, content);
let mut msg = IncomingMessage::new("gateway", user_id, content);
if let Some(ref tid) = thread_id {
msg = msg.with_thread(tid);
}
let tx_guard = state.msg_tx.read().await;
if let Some(ref tx) = *tx_guard {
let _ = tx.send(msg).await;
}
}
WsClientMessage::AuthToken {
extension_name,
token,
} => {
if let Some(ref ext_mgr) = state.extension_manager {
match ext_mgr.auth(&extension_name, Some(&token)).await {
Ok(result) if result.status == "authenticated" => {
let msg = match ext_mgr.activate(&extension_name).await {
Ok(r) => format!(
"{} authenticated ({} tools loaded)",
extension_name,
r.tools_loaded.len()
),
Err(e) => format!(
"{} authenticated but activation failed: {}",
extension_name, e
),
};
crate::channels::web::server::clear_auth_mode(state).await;
state
.sse
.broadcast(crate::channels::web::types::SseEvent::AuthCompleted {
extension_name,
success: true,
message: msg,
});
}
Ok(result) => {
state
.sse
.broadcast(crate::channels::web::types::SseEvent::AuthRequired {
extension_name,
instructions: result.instructions,
auth_url: result.auth_url,
setup_url: result.setup_url,
});
}
Err(e) => {
let _ = direct_tx
.send(WsServerMessage::Error {
message: format!("Auth failed: {}", e),
})
.await;
}
}
} else {
let _ = direct_tx
.send(WsServerMessage::Error {
message: "Extension manager not available".to_string(),
})
.await;
}
}
WsClientMessage::AuthCancel { .. } => {
crate::channels::web::server::clear_auth_mode(state).await;
}
WsClientMessage::Ping => {
let _ = direct_tx.send(WsServerMessage::Pong).await;
}
@@ -328,6 +400,7 @@ mod tests {
WsClientMessage::Approval {
request_id: request_id.to_string(),
action: "approve".to_string(),
thread_id: Some("thread-42".to_string()),
},
&state,
"user1",
@@ -338,6 +411,8 @@ mod tests {
let incoming = agent_rx.recv().await.unwrap();
// The content should be a serialized ExecApproval
assert!(incoming.content.contains("ExecApproval"));
// Thread should be forwarded onto the IncomingMessage.
assert_eq!(incoming.thread_id.as_deref(), Some("thread-42"));
}
#[tokio::test]
@@ -349,6 +424,7 @@ mod tests {
WsClientMessage::Approval {
request_id: Uuid::new_v4().to_string(),
action: "maybe".to_string(),
thread_id: None,
},
&state,
"user1",
@@ -374,6 +450,7 @@ mod tests {
WsClientMessage::Approval {
request_id: "not-a-uuid".to_string(),
action: "approve".to_string(),
thread_id: None,
},
&state,
"user1",
@@ -398,14 +475,18 @@ mod tests {
msg_tx: tokio::sync::RwLock::new(msg_tx),
sse: SseManager::new(),
workspace: None,
context_manager: None,
session_manager: None,
log_broadcaster: None,
extension_manager: None,
tool_registry: None,
store: None,
job_manager: None,
prompt_queue: None,
user_id: "test".to_string(),
shutdown_tx: tokio::sync::RwLock::new(None),
ws_tracker: Some(Arc::new(WsConnectionTracker::new())),
llm_provider: None,
chat_rate_limiter: crate::channels::web::server::RateLimiter::new(30, 60),
}
}
}
+124 -54
View File
@@ -1,6 +1,9 @@
//! Configuration management CLI commands.
//!
//! Commands for viewing and modifying settings.
//! Settings are stored in the database (env > DB > default).
use std::sync::Arc;
use clap::Subcommand;
@@ -36,41 +39,83 @@ pub enum ConfigCommand {
path: String,
},
/// Show the settings file path
/// Show the settings storage info
Path,
}
/// Run a config command.
pub fn run_config_command(cmd: ConfigCommand) -> anyhow::Result<()> {
///
/// Connects to the database to read/write settings. Falls back to disk
/// if the database is not available.
pub async fn run_config_command(cmd: ConfigCommand) -> anyhow::Result<()> {
let _ = dotenvy::dotenv();
// Try to connect to the DB for settings access
let db: Option<Arc<dyn crate::db::Database>> = match connect_db().await {
Ok(d) => Some(d),
Err(e) => {
eprintln!(
"Warning: Could not connect to database ({}), using disk fallback",
e
);
None
}
};
let db_ref = db.as_deref();
match cmd {
ConfigCommand::List { filter } => list_settings(filter),
ConfigCommand::Get { path } => get_setting(&path),
ConfigCommand::Set { path, value } => set_setting(&path, &value),
ConfigCommand::Reset { path } => reset_setting(&path),
ConfigCommand::Path => show_path(),
ConfigCommand::List { filter } => list_settings(db_ref, filter).await,
ConfigCommand::Get { path } => get_setting(db_ref, &path).await,
ConfigCommand::Set { path, value } => set_setting(db_ref, &path, &value).await,
ConfigCommand::Reset { path } => reset_setting(db_ref, &path).await,
ConfigCommand::Path => show_path(db_ref.is_some()),
}
}
/// Bootstrap a DB connection for config commands (backend-agnostic).
async fn connect_db() -> anyhow::Result<Arc<dyn crate::db::Database>> {
let config = crate::config::Config::from_env()
.await
.map_err(|e| anyhow::anyhow!("{}", e))?;
crate::db::connect_from_config(&config.database)
.await
.map_err(|e| anyhow::anyhow!("{}", e))
}
const DEFAULT_USER_ID: &str = "default";
/// Load settings: DB if available, else disk.
async fn load_settings(store: Option<&dyn crate::db::Database>) -> Settings {
if let Some(store) = store {
match store.get_all_settings(DEFAULT_USER_ID).await {
Ok(map) if !map.is_empty() => return Settings::from_db_map(&map),
_ => {}
}
}
Settings::load()
}
/// List all settings.
fn list_settings(filter: Option<String>) -> anyhow::Result<()> {
let settings = Settings::load();
async fn list_settings(
store: Option<&dyn crate::db::Database>,
filter: Option<String>,
) -> anyhow::Result<()> {
let settings = load_settings(store).await;
let all = settings.list();
// Find the longest key for alignment
let max_key_len = all.iter().map(|(k, _)| k.len()).max().unwrap_or(0);
println!("Settings:");
let source = if store.is_some() { "database" } else { "disk" };
println!("Settings (source: {}):", source);
println!();
for (key, value) in all {
// Skip if filter is set and doesn't match
if let Some(ref f) = filter {
if !key.starts_with(f) {
continue;
}
}
// Truncate long values for display
let display_value = if value.len() > 60 {
format!("{}...", &value[..57])
} else {
@@ -84,8 +129,8 @@ fn list_settings(filter: Option<String>) -> anyhow::Result<()> {
}
/// Get a specific setting.
fn get_setting(path: &str) -> anyhow::Result<()> {
let settings = Settings::load();
async fn get_setting(store: Option<&dyn crate::db::Database>, path: &str) -> anyhow::Result<()> {
let settings = load_settings(store).await;
match settings.get(path) {
Some(value) => {
@@ -99,67 +144,92 @@ fn get_setting(path: &str) -> anyhow::Result<()> {
}
/// Set a setting value.
fn set_setting(path: &str, value: &str) -> anyhow::Result<()> {
let mut settings = Settings::load();
async fn set_setting(
store: Option<&dyn crate::db::Database>,
path: &str,
value: &str,
) -> anyhow::Result<()> {
let mut settings = load_settings(store).await;
// Try to set the value
settings
.set(path, value)
.map_err(|e| anyhow::anyhow!("{}", e))?;
// Save to disk
settings.save()?;
// Save to DB if available, otherwise disk
if let Some(store) = store {
let json_value = match serde_json::from_str::<serde_json::Value>(value) {
Ok(v) => v,
Err(_) => serde_json::Value::String(value.to_string()),
};
store
.set_setting(DEFAULT_USER_ID, path, &json_value)
.await
.map_err(|e| anyhow::anyhow!("Failed to save to database: {}", e))?;
} else {
settings.save()?;
}
println!("Set {} = {}", path, value);
Ok(())
}
/// Reset a setting to default.
fn reset_setting(path: &str) -> anyhow::Result<()> {
let mut settings = Settings::load();
// Get the default value for display
async fn reset_setting(store: Option<&dyn crate::db::Database>, path: &str) -> anyhow::Result<()> {
let default = Settings::default();
let default_value = default
.get(path)
.ok_or_else(|| anyhow::anyhow!("Unknown setting: {}", path))?;
// Reset it
settings.reset(path).map_err(|e| anyhow::anyhow!("{}", e))?;
// Save to disk
settings.save()?;
// Delete from DB (falling back to default) or reset on disk
if let Some(store) = store {
store
.delete_setting(DEFAULT_USER_ID, path)
.await
.map_err(|e| anyhow::anyhow!("Failed to delete setting from database: {}", e))?;
} else {
let mut settings = Settings::load();
settings.reset(path).map_err(|e| anyhow::anyhow!("{}", e))?;
settings.save()?;
}
println!("Reset {} to default: {}", path, default_value);
Ok(())
}
/// Show the settings file path.
fn show_path() -> anyhow::Result<()> {
let path = Settings::default_path();
println!("{}", path.display());
if path.exists() {
let metadata = std::fs::metadata(&path)?;
println!(" Size: {} bytes", metadata.len());
if let Ok(modified) = metadata.modified() {
use std::time::SystemTime;
let duration = SystemTime::now()
.duration_since(modified)
.unwrap_or_default();
let secs = duration.as_secs();
if secs < 60 {
println!(" Modified: {} seconds ago", secs);
} else if secs < 3600 {
println!(" Modified: {} minutes ago", secs / 60);
} else if secs < 86400 {
println!(" Modified: {} hours ago", secs / 3600);
} else {
println!(" Modified: {} days ago", secs / 86400);
}
}
/// Show the settings storage info.
fn show_path(has_db: bool) -> anyhow::Result<()> {
if has_db {
println!("Settings stored in: database (settings table)");
println!(
"Bootstrap config: {}",
crate::bootstrap::BootstrapConfig::default_path().display()
);
} else {
println!(" (does not exist, using defaults)");
let path = Settings::default_path();
println!("Settings stored in: {} (disk fallback)", path.display());
if path.exists() {
let metadata = std::fs::metadata(&path)?;
println!(" Size: {} bytes", metadata.len());
if let Ok(modified) = metadata.modified() {
use std::time::SystemTime;
let duration = SystemTime::now()
.duration_since(modified)
.unwrap_or_default();
let secs = duration.as_secs();
if secs < 60 {
println!(" Modified: {} seconds ago", secs);
} else if secs < 3600 {
println!(" Modified: {} minutes ago", secs / 60);
} else if secs < 86400 {
println!(" Modified: {} hours ago", secs / 3600);
} else {
println!(" Modified: {} days ago", secs / 86400);
}
}
} else {
println!(" (does not exist, using defaults)");
}
}
Ok(())
+117 -21
View File
@@ -8,14 +8,14 @@ use std::sync::Arc;
use clap::Subcommand;
use crate::config::Config;
use crate::history::Store;
use crate::secrets::{PostgresSecretsStore, SecretsCrypto, SecretsStore};
use crate::db::Database;
#[cfg(feature = "postgres")]
use crate::secrets::PostgresSecretsStore;
use crate::secrets::{SecretsCrypto, SecretsStore};
use crate::tools::mcp::{
McpClient, McpServerConfig, McpSessionManager, OAuthConfig,
auth::{authorize_mcp_server, is_authenticated},
config::{
add_mcp_server, get_mcp_server, load_mcp_servers, remove_mcp_server, save_mcp_servers,
},
config::{self, McpServersFile},
};
#[derive(Subcommand, Debug, Clone)]
@@ -173,8 +173,11 @@ async fn add_server(
// Validate
config.validate()?;
// Save
add_mcp_server(config).await?;
// Save (DB if available, else disk)
let db = connect_db().await;
let mut servers = load_servers(db.as_deref()).await?;
servers.upsert(config);
save_servers(db.as_deref(), &servers).await?;
println!();
println!(" ✓ Added MCP server '{}'", name);
@@ -192,7 +195,12 @@ async fn add_server(
/// Remove an MCP server.
async fn remove_server(name: String) -> anyhow::Result<()> {
remove_mcp_server(&name).await?;
let db = connect_db().await;
let mut servers = load_servers(db.as_deref()).await?;
if !servers.remove(&name) {
anyhow::bail!("Server '{}' not found", name);
}
save_servers(db.as_deref(), &servers).await?;
println!();
println!(" ✓ Removed MCP server '{}'", name);
@@ -203,7 +211,8 @@ async fn remove_server(name: String) -> anyhow::Result<()> {
/// List configured MCP servers.
async fn list_servers(verbose: bool) -> anyhow::Result<()> {
let servers = load_mcp_servers().await?;
let db = connect_db().await;
let servers = load_servers(db.as_deref()).await?;
if servers.servers.is_empty() {
println!();
@@ -261,7 +270,12 @@ async fn list_servers(verbose: bool) -> anyhow::Result<()> {
/// Authenticate with an MCP server.
async fn auth_server(name: String, user_id: String) -> anyhow::Result<()> {
// Get server config
let server = get_mcp_server(&name).await?;
let db = connect_db().await;
let servers = load_servers(db.as_deref()).await?;
let server = servers
.get(&name)
.cloned()
.ok_or_else(|| anyhow::anyhow!("Server '{}' not found", name))?;
// Initialize secrets store
let secrets = get_secrets_store().await?;
@@ -329,7 +343,12 @@ async fn auth_server(name: String, user_id: String) -> anyhow::Result<()> {
/// Test connection to an MCP server.
async fn test_server(name: String, user_id: String) -> anyhow::Result<()> {
// Get server config
let server = get_mcp_server(&name).await?;
let db = connect_db().await;
let servers = load_servers(db.as_deref()).await?;
let server = servers
.get(&name)
.cloned()
.ok_or_else(|| anyhow::anyhow!("Server '{}' not found", name))?;
println!();
println!(" Testing connection to '{}'...", name);
@@ -420,7 +439,8 @@ async fn test_server(name: String, user_id: String) -> anyhow::Result<()> {
/// Toggle server enabled/disabled state.
async fn toggle_server(name: String, enable: bool, disable: bool) -> anyhow::Result<()> {
let mut servers = load_mcp_servers().await?;
let db = connect_db().await;
let mut servers = load_servers(db.as_deref()).await?;
let server = servers
.get_mut(&name)
@@ -435,7 +455,7 @@ async fn toggle_server(name: String, enable: bool, disable: bool) -> anyhow::Res
};
server.enabled = new_state;
save_mcp_servers(&servers).await?;
save_servers(db.as_deref(), &servers).await?;
let status = if new_state { "enabled" } else { "disabled" };
println!();
@@ -445,9 +465,38 @@ async fn toggle_server(name: String, enable: bool, disable: bool) -> anyhow::Res
Ok(())
}
const DEFAULT_USER_ID: &str = "default";
/// Try to connect to the database (backend-agnostic).
async fn connect_db() -> Option<Arc<dyn Database>> {
let config = Config::from_env().await.ok()?;
crate::db::connect_from_config(&config.database).await.ok()
}
/// Load MCP servers (DB if available, else disk).
async fn load_servers(db: Option<&dyn Database>) -> Result<McpServersFile, config::ConfigError> {
if let Some(db) = db {
config::load_mcp_servers_from_db(db, DEFAULT_USER_ID).await
} else {
config::load_mcp_servers().await
}
}
/// Save MCP servers (DB if available, else disk).
async fn save_servers(
db: Option<&dyn Database>,
servers: &McpServersFile,
) -> Result<(), config::ConfigError> {
if let Some(db) = db {
config::save_mcp_servers_to_db(db, DEFAULT_USER_ID, servers).await
} else {
config::save_mcp_servers(servers).await
}
}
/// Initialize and return the secrets store.
async fn get_secrets_store() -> anyhow::Result<Arc<dyn SecretsStore + Send + Sync>> {
let config = Config::from_env()?;
let config = Config::from_env().await?;
let master_key = config.secrets.master_key().ok_or_else(|| {
anyhow::anyhow!(
@@ -455,14 +504,61 @@ async fn get_secrets_store() -> anyhow::Result<Arc<dyn SecretsStore + Send + Syn
)
})?;
let store = Store::new(&config.database).await?;
store.run_migrations().await?;
let crypto = SecretsCrypto::new(master_key.clone())?;
Ok(Arc::new(PostgresSecretsStore::new(
store.pool(),
Arc::new(crypto),
)))
#[cfg(feature = "postgres")]
{
let store = crate::history::Store::new(&config.database).await?;
store.run_migrations().await?;
Ok(Arc::new(PostgresSecretsStore::new(
store.pool(),
Arc::new(crypto),
)))
}
#[cfg(all(feature = "libsql", not(feature = "postgres")))]
{
use crate::db::Database as _;
use crate::db::libsql_backend::LibSqlBackend;
use secrecy::ExposeSecret as _;
let default_path = crate::config::default_libsql_path();
let db_path = config
.database
.libsql_path
.as_deref()
.unwrap_or(&default_path);
let backend = if let Some(ref url) = config.database.libsql_url {
let token = config.database.libsql_auth_token.as_ref().ok_or_else(|| {
anyhow::anyhow!("LIBSQL_AUTH_TOKEN is required when LIBSQL_URL is set")
})?;
LibSqlBackend::new_remote_replica(db_path, url, token.expose_secret())
.await
.map_err(|e| anyhow::anyhow!("{}", e))?
} else {
LibSqlBackend::new_local(db_path)
.await
.map_err(|e| anyhow::anyhow!("{}", e))?
};
backend
.run_migrations()
.await
.map_err(|e| anyhow::anyhow!("{}", e))?;
return Ok(Arc::new(crate::secrets::LibSqlSecretsStore::new(
backend.shared_db(),
Arc::new(crypto),
)));
}
#[cfg(not(any(feature = "postgres", feature = "libsql")))]
{
let _ = crypto;
anyhow::bail!(
"No database backend available for secrets. Enable 'postgres' or 'libsql' feature."
);
}
}
#[cfg(test)]
+26 -1
View File
@@ -9,6 +9,30 @@ use clap::Subcommand;
use crate::workspace::{EmbeddingProvider, SearchConfig, Workspace};
/// Run a memory command using the Database trait (works with any backend).
pub async fn run_memory_command_with_db(
cmd: MemoryCommand,
db: std::sync::Arc<dyn crate::db::Database>,
embeddings: Option<Arc<dyn EmbeddingProvider>>,
) -> anyhow::Result<()> {
let mut workspace = Workspace::new_with_db("default", db);
if let Some(emb) = embeddings {
workspace = workspace.with_embeddings(emb);
}
match cmd {
MemoryCommand::Search { query, limit } => search(&workspace, &query, limit).await,
MemoryCommand::Read { path } => read(&workspace, &path).await,
MemoryCommand::Write {
path,
content,
append,
} => write(&workspace, &path, content, append).await,
MemoryCommand::Tree { path, depth } => tree(&workspace, &path, depth).await,
MemoryCommand::Status => status(&workspace).await,
}
}
#[derive(Subcommand, Debug, Clone)]
pub enum MemoryCommand {
/// Search workspace memory (hybrid full-text + semantic)
@@ -55,7 +79,8 @@ pub enum MemoryCommand {
Status,
}
/// Run a memory command.
/// Run a memory command (PostgreSQL backend).
#[cfg(feature = "postgres")]
pub async fn run_memory_command(
cmd: MemoryCommand,
pool: deadpool_postgres::Pool,
+46 -1
View File
@@ -12,12 +12,17 @@
mod config;
mod mcp;
pub mod memory;
mod pairing;
pub mod status;
mod tool;
pub use config::{ConfigCommand, run_config_command};
pub use mcp::{McpCommand, run_mcp_command};
pub use memory::{MemoryCommand, run_memory_command};
pub use memory::MemoryCommand;
#[cfg(feature = "postgres")]
pub use memory::run_memory_command;
pub use memory::run_memory_command_with_db;
pub use pairing::{PairingCommand, run_pairing_command, run_pairing_command_with_store};
pub use status::run_status_command;
pub use tool::{ToolCommand, run_tool_command};
@@ -86,8 +91,48 @@ pub enum Command {
#[command(subcommand)]
Memory(MemoryCommand),
/// DM pairing (approve inbound requests from unknown senders)
#[command(subcommand)]
Pairing(PairingCommand),
/// Show system health and diagnostics
Status,
/// Run as a sandboxed worker inside a Docker container (internal use).
/// This is invoked automatically by the orchestrator, not by users directly.
Worker {
/// Job ID to execute.
#[arg(long)]
job_id: uuid::Uuid,
/// URL of the orchestrator's internal API.
#[arg(long, default_value = "http://host.docker.internal:50051")]
orchestrator_url: String,
/// Maximum iterations before stopping.
#[arg(long, default_value = "50")]
max_iterations: u32,
},
/// Run as a Claude Code bridge inside a Docker container (internal use).
/// Spawns the `claude` CLI and streams output back to the orchestrator.
ClaudeBridge {
/// Job ID to execute.
#[arg(long)]
job_id: uuid::Uuid,
/// URL of the orchestrator's internal API.
#[arg(long, default_value = "http://host.docker.internal:50051")]
orchestrator_url: String,
/// Maximum agentic turns for Claude Code.
#[arg(long, default_value = "50")]
max_turns: u32,
/// Claude model to use (e.g. "sonnet", "opus").
#[arg(long, default_value = "sonnet")]
model: String,
},
}
impl Cli {
+187
View File
@@ -0,0 +1,187 @@
//! DM pairing CLI commands.
//!
//! Manage pairing requests for channels (Telegram, Slack, etc.).
use clap::Subcommand;
use crate::pairing::PairingStore;
/// Pairing subcommands.
#[derive(Subcommand, Debug, Clone)]
pub enum PairingCommand {
/// List pending pairing requests
List {
/// Channel name (e.g., telegram, slack)
#[arg(required = true)]
channel: String,
/// Output as JSON
#[arg(long)]
json: bool,
},
/// Approve a pairing request by code
Approve {
/// Channel name (e.g., telegram, slack)
#[arg(required = true)]
channel: String,
/// Pairing code (e.g., ABC12345)
#[arg(required = true)]
code: String,
},
}
/// Run pairing CLI command.
pub fn run_pairing_command(cmd: PairingCommand) -> Result<(), String> {
run_pairing_command_with_store(&PairingStore::new(), cmd)
}
/// Run pairing CLI command with a given store (for testing).
pub fn run_pairing_command_with_store(
store: &PairingStore,
cmd: PairingCommand,
) -> Result<(), String> {
match cmd {
PairingCommand::List { channel, json } => run_list(store, &channel, json),
PairingCommand::Approve { channel, code } => run_approve(store, &channel, &code),
}
}
fn run_list(store: &PairingStore, channel: &str, json: bool) -> Result<(), String> {
let requests = store.list_pending(channel).map_err(|e| e.to_string())?;
if json {
println!(
"{}",
serde_json::to_string_pretty(&requests).map_err(|e| e.to_string())?
);
return Ok(());
}
if requests.is_empty() {
println!("No pending {} pairing requests.", channel);
return Ok(());
}
println!("Pairing requests ({}):", requests.len());
for r in &requests {
let meta = r
.meta
.as_ref()
.and_then(|m| m.as_object())
.map(|o| {
o.iter()
.filter_map(|(k, v)| v.as_str().map(|s| format!("{}={}", k, s)))
.collect::<Vec<_>>()
.join(", ")
})
.unwrap_or_default();
println!(" {} {} {} {}", r.code, r.id, meta, r.created_at);
}
Ok(())
}
fn run_approve(store: &PairingStore, channel: &str, code: &str) -> Result<(), String> {
match store.approve(channel, code) {
Ok(Some(entry)) => {
println!("Approved {} sender {}.", channel, entry.id);
Ok(())
}
Ok(None) => Err(format!(
"No pending pairing request found for code: {}",
code
)),
Err(crate::pairing::PairingStoreError::ApproveRateLimited) => Err(
"Too many failed approve attempts. Wait a few minutes before trying again.".to_string(),
),
Err(e) => Err(e.to_string()),
}
}
#[cfg(test)]
mod tests {
use super::*;
use tempfile::TempDir;
fn test_store() -> (PairingStore, TempDir) {
let dir = TempDir::new().unwrap();
let store = PairingStore::with_base_dir(dir.path().to_path_buf());
(store, dir)
}
#[test]
fn test_list_empty_returns_ok() {
let (store, _) = test_store();
let result = run_pairing_command_with_store(
&store,
PairingCommand::List {
channel: "telegram".to_string(),
json: false,
},
);
assert!(result.is_ok());
}
#[test]
fn test_list_json_empty_returns_ok() {
let (store, _) = test_store();
let result = run_pairing_command_with_store(
&store,
PairingCommand::List {
channel: "telegram".to_string(),
json: true,
},
);
assert!(result.is_ok());
}
#[test]
fn test_approve_invalid_code_returns_err() {
let (store, _) = test_store();
// Create a pending request so the pairing file exists, then approve with wrong code
store.upsert_request("telegram", "user1", None).unwrap();
let result = run_pairing_command_with_store(
&store,
PairingCommand::Approve {
channel: "telegram".to_string(),
code: "BADCODE1".to_string(),
},
);
assert!(result.is_err());
assert!(result.unwrap_err().contains("No pending pairing request"));
}
#[test]
fn test_approve_valid_code_returns_ok() {
let (store, _) = test_store();
let r = store.upsert_request("telegram", "user1", None).unwrap();
assert!(r.created);
let result = run_pairing_command_with_store(
&store,
PairingCommand::Approve {
channel: "telegram".to_string(),
code: r.code,
},
);
assert!(result.is_ok());
}
#[test]
fn test_list_with_pending_returns_ok() {
let (store, _) = test_store();
store.upsert_request("telegram", "user1", None).unwrap();
let result = run_pairing_command_with_store(
&store,
PairingCommand::List {
channel: "telegram".to_string(),
json: false,
},
);
assert!(result.is_ok());
}
}
+8 -1
View File
@@ -47,7 +47,7 @@ pub async fn run_status_command() -> anyhow::Result<()> {
print!(" Secrets: ");
let secrets_configured = settings.secrets_master_key_source != crate::settings::KeySource::None
|| std::env::var("SECRETS_MASTER_KEY").is_ok()
|| crate::secrets::keychain::has_master_key();
|| crate::secrets::keychain::has_master_key().await;
if secrets_configured {
println!("configured ({:?})", settings.secrets_master_key_source);
} else {
@@ -135,6 +135,7 @@ pub async fn run_status_command() -> anyhow::Result<()> {
Ok(())
}
#[cfg(feature = "postgres")]
async fn check_database() -> anyhow::Result<()> {
let _ = dotenvy::dotenv();
let settings = Settings::load();
@@ -167,6 +168,12 @@ async fn check_database() -> anyhow::Result<()> {
Ok(())
}
#[cfg(not(feature = "postgres"))]
async fn check_database() -> anyhow::Result<()> {
// For non-postgres backends, just report configured
Ok(())
}
fn count_wasm_files(dir: &std::path::Path) -> usize {
std::fs::read_dir(dir)
.map(|entries| {
+64 -14
View File
@@ -11,8 +11,11 @@ use clap::Subcommand;
use tokio::fs;
use crate::config::Config;
use crate::history::Store;
use crate::secrets::{CreateSecretParams, PostgresSecretsStore, SecretsCrypto, SecretsStore};
#[allow(unused_imports)]
use crate::db::Database;
#[cfg(feature = "postgres")]
use crate::secrets::PostgresSecretsStore;
use crate::secrets::{CreateSecretParams, SecretsCrypto, SecretsStore};
use crate::tools::wasm::{CapabilitiesFile, compute_binary_hash};
/// Default tools directory.
@@ -715,18 +718,65 @@ async fn auth_tool(name: String, dir: Option<PathBuf>, user_id: String) -> anyho
println!();
// Initialize secrets store
let config = Config::from_env()?;
let config = Config::from_env().await?;
let master_key = config.secrets.master_key().ok_or_else(|| {
anyhow::anyhow!(
"SECRETS_MASTER_KEY not set. Run 'ironclaw onboard' first or set it in .env"
)
})?;
let store = Store::new(&config.database).await?;
store.run_migrations().await?;
let crypto = SecretsCrypto::new(master_key.clone())?;
let secrets_store = Arc::new(PostgresSecretsStore::new(store.pool(), Arc::new(crypto)));
let secrets_store: Arc<dyn SecretsStore + Send + Sync> = {
#[cfg(feature = "postgres")]
{
let store = crate::history::Store::new(&config.database).await?;
store.run_migrations().await?;
Arc::new(PostgresSecretsStore::new(store.pool(), Arc::new(crypto)))
}
#[cfg(all(feature = "libsql", not(feature = "postgres")))]
{
use crate::db::Database as _;
use crate::db::libsql_backend::LibSqlBackend;
use secrecy::ExposeSecret as _;
let default_path = crate::config::default_libsql_path();
let db_path = config
.database
.libsql_path
.as_deref()
.unwrap_or(&default_path);
let backend = if let Some(ref url) = config.database.libsql_url {
let token = config.database.libsql_auth_token.as_ref().ok_or_else(|| {
anyhow::anyhow!("LIBSQL_AUTH_TOKEN is required when LIBSQL_URL is set")
})?;
LibSqlBackend::new_remote_replica(db_path, url, token.expose_secret())
.await
.map_err(|e| anyhow::anyhow!("{}", e))?
} else {
LibSqlBackend::new_local(db_path)
.await
.map_err(|e| anyhow::anyhow!("{}", e))?
};
backend
.run_migrations()
.await
.map_err(|e| anyhow::anyhow!("{}", e))?;
Arc::new(crate::secrets::LibSqlSecretsStore::new(
backend.shared_db(),
Arc::new(crypto),
))
}
#[cfg(not(any(feature = "postgres", feature = "libsql")))]
{
let _ = crypto;
anyhow::bail!(
"No database backend available for secrets. Enable 'postgres' or 'libsql' feature."
);
}
};
// Check if already configured
let already_configured = secrets_store
@@ -772,13 +822,13 @@ async fn auth_tool(name: String, dir: Option<PathBuf>, user_id: String) -> anyho
println!(" Validation failed: {}", e);
println!();
println!(" Falling back to manual entry...");
return auth_tool_manual(&secrets_store, &user_id, &auth).await;
return auth_tool_manual(secrets_store.as_ref(), &user_id, &auth).await;
}
}
}
// Save the token
save_token(&secrets_store, &user_id, &auth, &token).await?;
save_token(secrets_store.as_ref(), &user_id, &auth, &token).await?;
print_success(display_name);
return Ok(());
}
@@ -787,16 +837,16 @@ async fn auth_tool(name: String, dir: Option<PathBuf>, user_id: String) -> anyho
// Check for OAuth configuration
if let Some(ref oauth) = auth.oauth {
return auth_tool_oauth(&secrets_store, &user_id, &auth, oauth).await;
return auth_tool_oauth(secrets_store.as_ref(), &user_id, &auth, oauth).await;
}
// Fall back to manual entry
auth_tool_manual(&secrets_store, &user_id, &auth).await
auth_tool_manual(secrets_store.as_ref(), &user_id, &auth).await
}
/// OAuth browser-based login flow.
async fn auth_tool_oauth(
store: &PostgresSecretsStore,
store: &(dyn SecretsStore + Send + Sync),
user_id: &str,
auth: &crate::tools::wasm::AuthCapabilitySchema,
oauth: &crate::tools::wasm::OAuthConfigSchema,
@@ -1044,7 +1094,7 @@ async fn auth_tool_oauth(
/// Manual token entry flow.
async fn auth_tool_manual(
store: &PostgresSecretsStore,
store: &(dyn SecretsStore + Send + Sync),
user_id: &str,
auth: &crate::tools::wasm::AuthCapabilitySchema,
) -> anyhow::Result<()> {
@@ -1217,7 +1267,7 @@ async fn validate_token(
/// Save token to secrets store.
async fn save_token(
store: &PostgresSecretsStore,
store: &(dyn SecretsStore + Send + Sync),
user_id: &str,
auth: &crate::tools::wasm::AuthCapabilitySchema,
token: &str,
+525 -122
View File
@@ -1,4 +1,9 @@
//! Configuration for IronClaw.
//!
//! Settings are loaded with priority: env var > database > default.
//! The database replaces the old `settings.json` file for all settings
//! except the 4 bootstrap fields (database_url, pool_size, secrets key
//! source, onboard_completed) which live in `~/.ironclaw/bootstrap.json`.
use std::path::PathBuf;
use std::time::Duration;
@@ -6,6 +11,7 @@ use std::time::Duration;
use secrecy::{ExposeSecret, SecretString};
use crate::error::ConfigError;
use crate::settings::Settings;
/// Main configuration for the agent.
#[derive(Debug, Clone)]
@@ -21,28 +27,67 @@ pub struct Config {
pub secrets: SecretsConfig,
pub builder: BuilderModeConfig,
pub heartbeat: HeartbeatConfig,
pub routines: RoutineConfig,
pub sandbox: SandboxModeConfig,
pub claude_code: ClaudeCodeConfig,
}
impl Config {
/// Load configuration from environment variables.
pub fn from_env() -> Result<Self, ConfigError> {
// Load .env file if present (ignore errors if not found)
/// Load configuration from environment variables and the database.
///
/// Priority: env var > DB settings > default.
/// This is the primary way to load config after DB is connected.
pub async fn from_db(
store: &dyn crate::db::Database,
user_id: &str,
bootstrap: &crate::bootstrap::BootstrapConfig,
) -> Result<Self, ConfigError> {
let _ = dotenvy::dotenv();
// Load all settings from DB into a Settings struct
let db_settings = match store.get_all_settings(user_id).await {
Ok(map) => Settings::from_db_map(&map),
Err(e) => {
tracing::warn!("Failed to load settings from DB, using defaults: {}", e);
Settings::default()
}
};
Self::build(bootstrap, &db_settings).await
}
/// Load configuration from environment variables only (no database).
///
/// Used during early startup before the database is connected,
/// and by CLI commands that don't have DB access.
/// Falls back to legacy `settings.json` on disk if present.
pub async fn from_env() -> Result<Self, ConfigError> {
let _ = dotenvy::dotenv();
let bootstrap = crate::bootstrap::BootstrapConfig::load();
let settings = Settings::load();
Self::build(&bootstrap, &settings).await
}
/// Build config from bootstrap + settings (shared by from_env and from_db).
async fn build(
bootstrap: &crate::bootstrap::BootstrapConfig,
settings: &Settings,
) -> Result<Self, ConfigError> {
Ok(Self {
database: DatabaseConfig::from_env()?,
llm: LlmConfig::from_env()?,
embeddings: EmbeddingsConfig::from_env()?,
tunnel: TunnelConfig::from_env()?,
channels: ChannelsConfig::from_env()?,
agent: AgentConfig::from_env()?,
safety: SafetyConfig::from_env()?,
wasm: WasmConfig::from_env()?,
secrets: SecretsConfig::from_env()?,
builder: BuilderModeConfig::from_env()?,
heartbeat: HeartbeatConfig::from_env()?,
sandbox: SandboxModeConfig::from_env()?,
database: DatabaseConfig::resolve(bootstrap)?,
llm: LlmConfig::resolve(settings)?,
embeddings: EmbeddingsConfig::resolve(settings)?,
tunnel: TunnelConfig::resolve(settings)?,
channels: ChannelsConfig::resolve(settings)?,
agent: AgentConfig::resolve(settings)?,
safety: SafetyConfig::resolve()?,
wasm: WasmConfig::resolve()?,
secrets: SecretsConfig::resolve(bootstrap).await?,
builder: BuilderModeConfig::resolve()?,
heartbeat: HeartbeatConfig::resolve(settings)?,
routines: RoutineConfig::resolve()?,
sandbox: SandboxModeConfig::resolve()?,
claude_code: ClaudeCodeConfig::resolve()?,
})
}
}
@@ -51,48 +96,17 @@ impl Config {
///
/// Used by channels and tools that need public webhook endpoints.
/// The tunnel URL is shared across all channels (Telegram, Slack, etc.).
///
/// # Security Notes
///
/// **Webhook endpoints** (e.g., `/webhook/telegram`) should NOT use tunnel-level
/// authentication because webhook providers (Telegram, Slack, GitHub) need
/// unauthenticated access to POST updates. Security for webhooks comes from:
/// - Webhook signature verification (provider-specific secrets)
/// - IP allowlisting (if supported by provider)
///
/// **Non-webhook endpoints** (admin APIs, health checks) CAN be protected using
/// tunnel provider features:
/// - ngrok: Basic Auth, OAuth, IP restrictions
/// - Cloudflare: Access policies, mTLS
///
/// These protections are configured in the tunnel provider, not here.
///
/// # Supported Providers
///
/// - **ngrok**: `ngrok http 8080` -> `https://abc123.ngrok.io`
/// - **Cloudflare Tunnel**: `cloudflared tunnel --url http://localhost:8080`
/// - **localtunnel**: `lt --port 8080`
/// - Any service that provides a public HTTPS URL to localhost
#[derive(Debug, Clone, Default)]
pub struct TunnelConfig {
/// Public URL from tunnel provider (e.g., "https://abc123.ngrok.io").
///
/// When set, channels that support webhooks will register their endpoints
/// with this base URL instead of using polling.
pub public_url: Option<String>,
}
impl TunnelConfig {
fn from_env() -> Result<Self, ConfigError> {
// Priority: env var > settings file
let public_url = optional_env("TUNNEL_URL")?.or_else(|| {
crate::settings::Settings::load()
.tunnel
.public_url
.filter(|s| !s.is_empty())
});
fn resolve(settings: &Settings) -> Result<Self, ConfigError> {
let public_url = optional_env("TUNNEL_URL")?
.or_else(|| settings.tunnel.public_url.clone().filter(|s| !s.is_empty()));
// Validate URL format if provided
if let Some(ref url) = public_url {
if !url.starts_with("https://") {
return Err(ConfigError::InvalidValue {
@@ -111,8 +125,6 @@ impl TunnelConfig {
}
/// Get the webhook URL for a given path.
///
/// Returns `None` if no tunnel is configured.
pub fn webhook_url(&self, path: &str) -> Option<String> {
self.public_url.as_ref().map(|base| {
let base = base.trim_end_matches('/');
@@ -122,26 +134,77 @@ impl TunnelConfig {
}
}
/// Which database backend to use.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
pub enum DatabaseBackend {
/// PostgreSQL via deadpool-postgres (default).
#[default]
Postgres,
/// libSQL/Turso embedded database.
LibSql,
}
impl std::str::FromStr for DatabaseBackend {
type Err = String;
fn from_str(s: &str) -> Result<Self, Self::Err> {
match s.to_lowercase().as_str() {
"postgres" | "postgresql" | "pg" => Ok(Self::Postgres),
"libsql" | "turso" | "sqlite" => Ok(Self::LibSql),
_ => Err(format!(
"invalid database backend '{}', expected 'postgres' or 'libsql'",
s
)),
}
}
}
/// Database configuration.
#[derive(Debug, Clone)]
pub struct DatabaseConfig {
/// Which backend to use (default: Postgres).
pub backend: DatabaseBackend,
// -- PostgreSQL fields --
pub url: SecretString,
pub pool_size: usize,
// -- libSQL fields --
/// Path to local libSQL database file (default: ~/.ironclaw/ironclaw.db).
pub libsql_path: Option<PathBuf>,
/// Turso cloud URL for remote sync (optional).
pub libsql_url: Option<String>,
/// Turso auth token (required when libsql_url is set).
pub libsql_auth_token: Option<SecretString>,
}
impl DatabaseConfig {
fn from_env() -> Result<Self, ConfigError> {
let settings = crate::settings::Settings::load();
fn resolve(bootstrap: &crate::bootstrap::BootstrapConfig) -> Result<Self, ConfigError> {
let backend: DatabaseBackend = if let Some(b) = optional_env("DATABASE_BACKEND")? {
b.parse().map_err(|e| ConfigError::InvalidValue {
key: "DATABASE_BACKEND".to_string(),
message: e,
})?
} else {
DatabaseBackend::default()
};
// Priority: env var > settings > error (required)
// PostgreSQL URL is required only when using the postgres backend.
// For libsql backend, default to an empty placeholder.
let url = optional_env("DATABASE_URL")?
.or(settings.database_url.clone())
.or_else(|| bootstrap.database_url.clone())
.or_else(|| {
if backend == DatabaseBackend::LibSql {
Some("unused://libsql".to_string())
} else {
None
}
})
.ok_or_else(|| ConfigError::MissingRequired {
key: "database_url".to_string(),
hint: "Run 'ironclaw onboard' or set DATABASE_URL environment variable".to_string(),
})?;
// Priority: env var > settings > default
let pool_size = optional_env("DATABASE_POOL_SIZE")?
.map(|s| s.parse())
.transpose()
@@ -149,12 +212,34 @@ impl DatabaseConfig {
key: "DATABASE_POOL_SIZE".to_string(),
message: format!("must be a positive integer: {e}"),
})?
.or(settings.database_pool_size)
.or(bootstrap.database_pool_size)
.unwrap_or(10);
let libsql_path = optional_env("LIBSQL_PATH")?.map(PathBuf::from).or_else(|| {
if backend == DatabaseBackend::LibSql {
Some(default_libsql_path())
} else {
None
}
});
let libsql_url = optional_env("LIBSQL_URL")?;
let libsql_auth_token = optional_env("LIBSQL_AUTH_TOKEN")?.map(SecretString::from);
if libsql_url.is_some() && libsql_auth_token.is_none() {
return Err(ConfigError::MissingRequired {
key: "LIBSQL_AUTH_TOKEN".to_string(),
hint: "LIBSQL_AUTH_TOKEN is required when LIBSQL_URL is set".to_string(),
});
}
Ok(Self {
backend,
url: SecretString::from(url),
pool_size,
libsql_path,
libsql_url,
libsql_auth_token,
})
}
@@ -164,10 +249,110 @@ impl DatabaseConfig {
}
}
/// LLM provider configuration (NEAR AI only).
/// Default libSQL database path (~/.ironclaw/ironclaw.db).
pub fn default_libsql_path() -> PathBuf {
dirs::home_dir()
.unwrap_or_else(|| PathBuf::from("."))
.join(".ironclaw")
.join("ironclaw.db")
}
/// Which LLM backend to use.
///
/// Defaults to `NearAi` to keep IronClaw close to the NEAR ecosystem.
/// Users can override with `LLM_BACKEND` env var to use their own API keys.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
pub enum LlmBackend {
/// NEAR AI proxy (default) -- session or API key auth
#[default]
NearAi,
/// Direct OpenAI API
OpenAi,
/// Direct Anthropic API
Anthropic,
/// Local Ollama instance
Ollama,
/// Any OpenAI-compatible endpoint (e.g. vLLM, LiteLLM, Together)
OpenAiCompatible,
}
impl std::str::FromStr for LlmBackend {
type Err = String;
fn from_str(s: &str) -> Result<Self, Self::Err> {
match s.to_lowercase().as_str() {
"nearai" | "near_ai" | "near" => Ok(Self::NearAi),
"openai" | "open_ai" => Ok(Self::OpenAi),
"anthropic" | "claude" => Ok(Self::Anthropic),
"ollama" => Ok(Self::Ollama),
"openai_compatible" | "openai-compatible" | "compatible" => Ok(Self::OpenAiCompatible),
_ => Err(format!(
"invalid LLM backend '{}', expected one of: nearai, openai, anthropic, ollama, openai_compatible",
s
)),
}
}
}
impl std::fmt::Display for LlmBackend {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
Self::NearAi => write!(f, "nearai"),
Self::OpenAi => write!(f, "openai"),
Self::Anthropic => write!(f, "anthropic"),
Self::Ollama => write!(f, "ollama"),
Self::OpenAiCompatible => write!(f, "openai_compatible"),
}
}
}
/// Configuration for direct OpenAI API access.
#[derive(Debug, Clone)]
pub struct OpenAiDirectConfig {
pub api_key: SecretString,
pub model: String,
}
/// Configuration for direct Anthropic API access.
#[derive(Debug, Clone)]
pub struct AnthropicDirectConfig {
pub api_key: SecretString,
pub model: String,
}
/// Configuration for local Ollama.
#[derive(Debug, Clone)]
pub struct OllamaConfig {
pub base_url: String,
pub model: String,
}
/// Configuration for any OpenAI-compatible endpoint.
#[derive(Debug, Clone)]
pub struct OpenAiCompatibleConfig {
pub base_url: String,
pub api_key: Option<SecretString>,
pub model: String,
}
/// LLM provider configuration.
///
/// NEAR AI remains the default backend. Users can switch to other providers
/// by setting `LLM_BACKEND` (e.g. `openai`, `anthropic`, `ollama`).
#[derive(Debug, Clone)]
pub struct LlmConfig {
/// Which backend to use (default: NearAi)
pub backend: LlmBackend,
/// NEAR AI config (always populated for NEAR AI embeddings, etc.)
pub nearai: NearAiConfig,
/// Direct OpenAI config (populated when backend=openai)
pub openai: Option<OpenAiDirectConfig>,
/// Direct Anthropic config (populated when backend=anthropic)
pub anthropic: Option<AnthropicDirectConfig>,
/// Ollama config (populated when backend=ollama)
pub ollama: Option<OllamaConfig>,
/// OpenAI-compatible config (populated when backend=openai_compatible)
pub openai_compatible: Option<OpenAiCompatibleConfig>,
}
/// API mode for NEAR AI.
@@ -215,42 +400,110 @@ pub struct NearAiConfig {
}
impl LlmConfig {
fn from_env() -> Result<Self, ConfigError> {
let api_key = optional_env("NEARAI_API_KEY")?.map(SecretString::from);
fn resolve(settings: &Settings) -> Result<Self, ConfigError> {
// Determine backend (default: NearAi)
let backend: LlmBackend = if let Some(b) = optional_env("LLM_BACKEND")? {
b.parse().map_err(|e| ConfigError::InvalidValue {
key: "LLM_BACKEND".to_string(),
message: e,
})?
} else {
LlmBackend::NearAi
};
// Always resolve NEAR AI config (used as fallback and for embeddings)
let nearai_api_key = optional_env("NEARAI_API_KEY")?.map(SecretString::from);
// Determine API mode: explicit setting, or infer from API key presence
let api_mode = if let Some(mode_str) = optional_env("NEARAI_API_MODE")? {
mode_str.parse().map_err(|e| ConfigError::InvalidValue {
key: "NEARAI_API_MODE".to_string(),
message: e,
})?
} else if api_key.is_some() {
// If API key is provided, default to chat_completions mode
} else if nearai_api_key.is_some() {
NearAiApiMode::ChatCompletions
} else {
NearAiApiMode::Responses
};
Ok(Self {
nearai: NearAiConfig {
// Load model from saved settings first, then env, then default
model: crate::settings::Settings::load()
.selected_model
.or_else(|| optional_env("NEARAI_MODEL").ok().flatten())
.unwrap_or_else(|| {
"fireworks::accounts/fireworks/models/llama4-maverick-instruct-basic"
.to_string()
}),
base_url: optional_env("NEARAI_BASE_URL")?
.unwrap_or_else(|| "https://cloud-api.near.ai".to_string()),
auth_base_url: optional_env("NEARAI_AUTH_URL")?
.unwrap_or_else(|| "https://private.near.ai".to_string()),
session_path: optional_env("NEARAI_SESSION_PATH")?
.map(PathBuf::from)
.unwrap_or_else(default_session_path),
api_mode,
let nearai = NearAiConfig {
model: optional_env("NEARAI_MODEL")?
.or_else(|| settings.selected_model.clone())
.unwrap_or_else(|| {
"fireworks::accounts/fireworks/models/llama4-maverick-instruct-basic"
.to_string()
}),
base_url: optional_env("NEARAI_BASE_URL")?
.unwrap_or_else(|| "https://cloud-api.near.ai".to_string()),
auth_base_url: optional_env("NEARAI_AUTH_URL")?
.unwrap_or_else(|| "https://private.near.ai".to_string()),
session_path: optional_env("NEARAI_SESSION_PATH")?
.map(PathBuf::from)
.unwrap_or_else(default_session_path),
api_mode,
api_key: nearai_api_key,
};
// Resolve provider-specific configs based on backend
let openai = if backend == LlmBackend::OpenAi {
let api_key = optional_env("OPENAI_API_KEY")?
.map(SecretString::from)
.ok_or_else(|| ConfigError::MissingRequired {
key: "OPENAI_API_KEY".to_string(),
hint: "Set OPENAI_API_KEY when LLM_BACKEND=openai".to_string(),
})?;
let model = optional_env("OPENAI_MODEL")?.unwrap_or_else(|| "gpt-4o".to_string());
Some(OpenAiDirectConfig { api_key, model })
} else {
None
};
let anthropic = if backend == LlmBackend::Anthropic {
let api_key = optional_env("ANTHROPIC_API_KEY")?
.map(SecretString::from)
.ok_or_else(|| ConfigError::MissingRequired {
key: "ANTHROPIC_API_KEY".to_string(),
hint: "Set ANTHROPIC_API_KEY when LLM_BACKEND=anthropic".to_string(),
})?;
let model = optional_env("ANTHROPIC_MODEL")?
.unwrap_or_else(|| "claude-sonnet-4-20250514".to_string());
Some(AnthropicDirectConfig { api_key, model })
} else {
None
};
let ollama = if backend == LlmBackend::Ollama {
let base_url = optional_env("OLLAMA_BASE_URL")?
.unwrap_or_else(|| "http://localhost:11434".to_string());
let model = optional_env("OLLAMA_MODEL")?.unwrap_or_else(|| "llama3".to_string());
Some(OllamaConfig { base_url, model })
} else {
None
};
let openai_compatible = if backend == LlmBackend::OpenAiCompatible {
let base_url =
optional_env("LLM_BASE_URL")?.ok_or_else(|| ConfigError::MissingRequired {
key: "LLM_BASE_URL".to_string(),
hint: "Set LLM_BASE_URL when LLM_BACKEND=openai_compatible".to_string(),
})?;
let api_key = optional_env("LLM_API_KEY")?.map(SecretString::from);
let model = optional_env("LLM_MODEL")?.unwrap_or_else(|| "default".to_string());
Some(OpenAiCompatibleConfig {
base_url,
api_key,
},
model,
})
} else {
None
};
Ok(Self {
backend,
nearai,
openai,
anthropic,
ollama,
openai_compatible,
})
}
}
@@ -265,8 +518,6 @@ pub struct EmbeddingsConfig {
/// OpenAI API key (for OpenAI provider).
pub openai_api_key: Option<SecretString>,
/// Model to use for embeddings.
/// For OpenAI: "text-embedding-3-small", "text-embedding-3-large", "text-embedding-ada-002"
/// For NEAR AI: Uses the configured session for auth.
pub model: String,
}
@@ -282,18 +533,15 @@ impl Default for EmbeddingsConfig {
}
impl EmbeddingsConfig {
fn from_env() -> Result<Self, ConfigError> {
let settings = crate::settings::Settings::load();
fn resolve(settings: &Settings) -> Result<Self, ConfigError> {
let openai_api_key = optional_env("OPENAI_API_KEY")?.map(SecretString::from);
// Priority: env var > settings > default
let provider = optional_env("EMBEDDING_PROVIDER")?
.unwrap_or_else(|| settings.embeddings.provider.clone());
let model =
optional_env("EMBEDDING_MODEL")?.unwrap_or_else(|| settings.embeddings.model.clone());
// Priority: env var > settings > auto-detect from API key
let enabled = optional_env("EMBEDDING_ENABLED")?
.map(|s| s.parse())
.transpose()
@@ -301,10 +549,7 @@ impl EmbeddingsConfig {
key: "EMBEDDING_ENABLED".to_string(),
message: format!("must be 'true' or 'false': {e}"),
})?
.unwrap_or_else(|| {
// Check settings, or auto-enable if API key present
settings.embeddings.enabled || openai_api_key.is_some()
});
.unwrap_or_else(|| settings.embeddings.enabled || openai_api_key.is_some());
Ok(Self {
enabled,
@@ -338,6 +583,8 @@ pub struct ChannelsConfig {
pub wasm_channels_dir: std::path::PathBuf,
/// Whether WASM channels are enabled.
pub wasm_channels_enabled: bool,
/// Telegram owner user ID. When set, the bot only responds to this user.
pub telegram_owner_id: Option<i64>,
}
#[derive(Debug, Clone)]
@@ -364,7 +611,7 @@ pub struct GatewayConfig {
}
impl ChannelsConfig {
fn from_env() -> Result<Self, ConfigError> {
fn resolve(settings: &Settings) -> Result<Self, ConfigError> {
let http = if optional_env("HTTP_PORT")?.is_some() || optional_env("HTTP_HOST")?.is_some() {
Some(HttpConfig {
host: optional_env("HTTP_HOST")?.unwrap_or_else(|| "0.0.0.0".to_string()),
@@ -385,7 +632,7 @@ impl ChannelsConfig {
let gateway = if optional_env("GATEWAY_ENABLED")?
.map(|s| s.to_lowercase() == "true" || s == "1")
.unwrap_or(false)
.unwrap_or(true)
{
Some(GatewayConfig {
host: optional_env("GATEWAY_HOST")?.unwrap_or_else(|| "127.0.0.1".to_string()),
@@ -425,6 +672,14 @@ impl ChannelsConfig {
message: format!("must be 'true' or 'false': {e}"),
})?
.unwrap_or(true),
telegram_owner_id: optional_env("TELEGRAM_OWNER_ID")?
.map(|s| s.parse())
.transpose()
.map_err(|e| ConfigError::InvalidValue {
key: "TELEGRAM_OWNER_ID".to_string(),
message: format!("must be an integer: {e}"),
})?
.or(settings.channels.telegram_owner_id),
})
}
}
@@ -450,14 +705,13 @@ pub struct AgentConfig {
pub use_planning: bool,
/// Session idle timeout. Sessions inactive longer than this are pruned.
pub session_idle_timeout: Duration,
/// Allow chat to use filesystem/shell tools directly (bypass sandbox).
pub allow_local_tools: bool,
}
impl AgentConfig {
fn from_env() -> Result<Self, ConfigError> {
let settings = crate::settings::Settings::load();
fn resolve(settings: &Settings) -> Result<Self, ConfigError> {
Ok(Self {
// Priority: env var > settings > default
name: optional_env("AGENT_NAME")?.unwrap_or_else(|| settings.agent.name.clone()),
max_parallel_jobs: optional_env("AGENT_MAX_PARALLEL_JOBS")?
.map(|s| s.parse())
@@ -523,6 +777,14 @@ impl AgentConfig {
})?
.unwrap_or(settings.agent.session_idle_timeout_secs),
),
allow_local_tools: optional_env("ALLOW_LOCAL_TOOLS")?
.map(|s| s.parse())
.transpose()
.map_err(|e| ConfigError::InvalidValue {
key: "ALLOW_LOCAL_TOOLS".to_string(),
message: format!("must be 'true' or 'false': {e}"),
})?
.unwrap_or(false),
})
}
}
@@ -535,7 +797,7 @@ pub struct SafetyConfig {
}
impl SafetyConfig {
fn from_env() -> Result<Self, ConfigError> {
fn resolve() -> Result<Self, ConfigError> {
Ok(Self {
max_output_length: parse_optional_env("SAFETY_MAX_OUTPUT_LENGTH", 100_000)?,
injection_check_enabled: optional_env("SAFETY_INJECTION_CHECK_ENABLED")?
@@ -573,7 +835,6 @@ pub struct WasmConfig {
#[derive(Clone, Default)]
pub struct SecretsConfig {
/// Master key for encrypting secrets.
/// Source determined by KeySource in settings.
pub master_key: Option<SecretString>,
/// Whether secrets management is enabled.
pub enabled: bool,
@@ -592,20 +853,16 @@ impl std::fmt::Debug for SecretsConfig {
}
impl SecretsConfig {
fn from_env() -> Result<Self, ConfigError> {
async fn resolve(bootstrap: &crate::bootstrap::BootstrapConfig) -> Result<Self, ConfigError> {
use crate::settings::KeySource;
let settings = crate::settings::Settings::load();
// Priority: env var > keychain (based on settings) > disabled
let (master_key, source) = if let Some(env_key) = optional_env("SECRETS_MASTER_KEY")? {
// Env var takes priority (for CI/Docker)
(Some(SecretString::from(env_key)), KeySource::Env)
} else {
match settings.secrets_master_key_source {
match bootstrap.secrets_master_key_source {
KeySource::Keychain => {
// Try to load from OS keychain
match crate::secrets::keychain::get_master_key() {
// Try to load from OS keychain (async on Linux)
match crate::secrets::keychain::get_master_key().await {
Ok(key_bytes) => {
let key_hex: String =
key_bytes.iter().map(|b| format!("{:02x}", b)).collect();
@@ -623,7 +880,6 @@ impl SecretsConfig {
}
}
KeySource::Env => {
// Settings say env, but no env var found
tracing::warn!(
"Secrets configured for env var but SECRETS_MASTER_KEY not set."
);
@@ -635,7 +891,6 @@ impl SecretsConfig {
let enabled = master_key.is_some();
// Validate master key length if provided
if let Some(ref key) = master_key {
if key.expose_secret().len() < 32 {
return Err(ConfigError::InvalidValue {
@@ -681,7 +936,7 @@ fn default_tools_dir() -> PathBuf {
}
impl WasmConfig {
fn from_env() -> Result<Self, ConfigError> {
fn resolve() -> Result<Self, ConfigError> {
Ok(Self {
enabled: optional_env("WASM_ENABLED")?
.map(|s| s.parse())
@@ -752,7 +1007,7 @@ pub struct BuilderModeConfig {
impl Default for BuilderModeConfig {
fn default() -> Self {
Self {
enabled: true, // Builder enabled by default
enabled: true,
build_dir: None,
max_iterations: 20,
timeout_secs: 600,
@@ -762,7 +1017,7 @@ impl Default for BuilderModeConfig {
}
impl BuilderModeConfig {
fn from_env() -> Result<Self, ConfigError> {
fn resolve() -> Result<Self, ConfigError> {
Ok(Self {
enabled: optional_env("BUILDER_ENABLED")?
.map(|s| s.parse())
@@ -771,7 +1026,7 @@ impl BuilderModeConfig {
key: "BUILDER_ENABLED".to_string(),
message: format!("must be 'true' or 'false': {e}"),
})?
.unwrap_or(true), // Builder enabled by default
.unwrap_or(true),
build_dir: optional_env("BUILDER_DIR")?.map(PathBuf::from),
max_iterations: parse_optional_env("BUILDER_MAX_ITERATIONS", 20)?,
timeout_secs: parse_optional_env("BUILDER_TIMEOUT_SECS", 600)?,
@@ -826,11 +1081,8 @@ impl Default for HeartbeatConfig {
}
impl HeartbeatConfig {
fn from_env() -> Result<Self, ConfigError> {
let settings = crate::settings::Settings::load();
fn resolve(settings: &Settings) -> Result<Self, ConfigError> {
Ok(Self {
// Priority: env var > settings > default
enabled: optional_env("HEARTBEAT_ENABLED")?
.map(|s| s.parse())
.transpose()
@@ -848,9 +1100,55 @@ impl HeartbeatConfig {
})?
.unwrap_or(settings.heartbeat.interval_secs),
notify_channel: optional_env("HEARTBEAT_NOTIFY_CHANNEL")?
.or(settings.heartbeat.notify_channel.clone()),
.or_else(|| settings.heartbeat.notify_channel.clone()),
notify_user: optional_env("HEARTBEAT_NOTIFY_USER")?
.or(settings.heartbeat.notify_user.clone()),
.or_else(|| settings.heartbeat.notify_user.clone()),
})
}
}
/// Routines configuration.
#[derive(Debug, Clone)]
pub struct RoutineConfig {
/// Whether the routines system is enabled.
pub enabled: bool,
/// How often (seconds) to poll for cron routines that need firing.
pub cron_check_interval_secs: u64,
/// Max routines executing concurrently across all users.
pub max_concurrent_routines: usize,
/// Default cooldown between fires (seconds).
pub default_cooldown_secs: u64,
/// Max output tokens for lightweight routine LLM calls.
pub max_lightweight_tokens: u32,
}
impl Default for RoutineConfig {
fn default() -> Self {
Self {
enabled: true,
cron_check_interval_secs: 15,
max_concurrent_routines: 10,
default_cooldown_secs: 300,
max_lightweight_tokens: 4096,
}
}
}
impl RoutineConfig {
fn resolve() -> Result<Self, ConfigError> {
Ok(Self {
enabled: optional_env("ROUTINES_ENABLED")?
.map(|s| s.parse())
.transpose()
.map_err(|e| ConfigError::InvalidValue {
key: "ROUTINES_ENABLED".to_string(),
message: format!("must be 'true' or 'false': {e}"),
})?
.unwrap_or(true),
cron_check_interval_secs: parse_optional_env("ROUTINES_CRON_INTERVAL", 15)?,
max_concurrent_routines: parse_optional_env("ROUTINES_MAX_CONCURRENT", 10)?,
default_cooldown_secs: parse_optional_env("ROUTINES_DEFAULT_COOLDOWN", 300)?,
max_lightweight_tokens: parse_optional_env("ROUTINES_MAX_TOKENS", 4096)?,
})
}
}
@@ -879,7 +1177,7 @@ pub struct SandboxModeConfig {
impl Default for SandboxModeConfig {
fn default() -> Self {
Self {
enabled: true, // Enabled by default
enabled: true,
policy: "readonly".to_string(),
timeout_secs: 120,
memory_limit_mb: 2048,
@@ -892,7 +1190,7 @@ impl Default for SandboxModeConfig {
}
impl SandboxModeConfig {
fn from_env() -> Result<Self, ConfigError> {
fn resolve() -> Result<Self, ConfigError> {
let extra_domains = optional_env("SANDBOX_EXTRA_DOMAINS")?
.map(|s| s.split(',').map(|d| d.trim().to_string()).collect())
.unwrap_or_default();
@@ -948,6 +1246,111 @@ impl SandboxModeConfig {
}
}
/// Claude Code sandbox configuration.
#[derive(Debug, Clone)]
pub struct ClaudeCodeConfig {
/// Whether Claude Code sandbox mode is available.
pub enabled: bool,
/// Host directory containing Claude auth session (mounted read-only).
pub config_dir: std::path::PathBuf,
/// Claude model to use (e.g. "sonnet", "opus").
pub model: String,
/// Maximum agentic turns before stopping.
pub max_turns: u32,
/// Memory limit in MB for Claude Code containers (heavier than workers).
pub memory_limit_mb: u64,
/// Allowed tool patterns for Claude Code permission settings.
///
/// Written to `/workspace/.claude/settings.json` before spawning the CLI.
/// Provides defense-in-depth: only explicitly listed tools are auto-approved.
/// Any new/unknown tools would require interactive approval (which times out
/// in the non-interactive container, failing safely).
///
/// Patterns follow Claude Code syntax: `"Bash(*)"`, `"Read"`, `"Edit(*)"`, etc.
pub allowed_tools: Vec<String>,
}
/// Default allowed tools for Claude Code inside containers.
///
/// These cover all standard Claude Code tools needed for autonomous operation.
/// The Docker container provides the primary security boundary; this allowlist
/// provides defense-in-depth by preventing any future unknown tools from being
/// silently auto-approved.
fn default_claude_code_allowed_tools() -> Vec<String> {
[
"Bash(*)",
"Read",
"Edit(*)",
"Glob",
"Grep",
"WebFetch(*)",
"Task(*)",
]
.into_iter()
.map(String::from)
.collect()
}
impl Default for ClaudeCodeConfig {
fn default() -> Self {
Self {
enabled: false,
config_dir: dirs::home_dir()
.unwrap_or_else(|| std::path::PathBuf::from("."))
.join(".claude"),
model: "sonnet".to_string(),
max_turns: 50,
memory_limit_mb: 4096,
allowed_tools: default_claude_code_allowed_tools(),
}
}
}
impl ClaudeCodeConfig {
/// Load from environment variables only (used inside containers where
/// there is no database or full config).
pub fn from_env() -> Self {
match Self::resolve() {
Ok(c) => c,
Err(e) => {
tracing::warn!("Failed to resolve ClaudeCodeConfig: {e}, using defaults");
Self::default()
}
}
}
fn resolve() -> Result<Self, ConfigError> {
let defaults = Self::default();
Ok(Self {
enabled: optional_env("CLAUDE_CODE_ENABLED")?
.map(|s| s.parse())
.transpose()
.map_err(|e| ConfigError::InvalidValue {
key: "CLAUDE_CODE_ENABLED".to_string(),
message: format!("must be 'true' or 'false': {e}"),
})?
.unwrap_or(defaults.enabled),
config_dir: optional_env("CLAUDE_CONFIG_DIR")?
.map(std::path::PathBuf::from)
.unwrap_or(defaults.config_dir),
model: optional_env("CLAUDE_CODE_MODEL")?.unwrap_or(defaults.model),
max_turns: parse_optional_env("CLAUDE_CODE_MAX_TURNS", defaults.max_turns)?,
memory_limit_mb: parse_optional_env(
"CLAUDE_CODE_MEMORY_LIMIT_MB",
defaults.memory_limit_mb,
)?,
allowed_tools: optional_env("CLAUDE_CODE_ALLOWED_TOOLS")?
.map(|s| {
s.split(',')
.map(|t| t.trim().to_string())
.filter(|t| !t.is_empty())
.collect()
})
.unwrap_or(defaults.allowed_tools),
})
}
}
// Helper functions
fn optional_env(key: &str) -> Result<Option<String>, ConfigError> {
+5 -4
View File
@@ -45,20 +45,21 @@ impl ContextManager {
title: impl Into<String>,
description: impl Into<String>,
) -> Result<Uuid, JobError> {
let contexts = self.contexts.read().await;
// Hold write lock for the entire check-insert to prevent TOCTOU races
// where two concurrent calls both pass the active_count check.
let mut contexts = self.contexts.write().await;
let active_count = contexts.values().filter(|c| c.state.is_active()).count();
if active_count >= self.max_jobs {
return Err(JobError::MaxJobsExceeded { max: self.max_jobs });
}
drop(contexts);
let context = JobContext::with_user(user_id, title, description);
let job_id = context.job_id;
contexts.insert(job_id, context);
drop(contexts);
let memory = Memory::new(job_id);
self.contexts.write().await.insert(job_id, context);
self.memories.write().await.insert(job_id, memory);
Ok(job_id)
+88
View File
@@ -119,6 +119,10 @@ pub struct JobContext {
pub estimated_duration: Option<Duration>,
/// Actual cost so far.
pub actual_cost: Decimal,
/// Total tokens consumed by LLM calls in this job.
pub total_tokens_used: u64,
/// Maximum tokens allowed per job (0 = unlimited).
pub max_tokens: u64,
/// When the job was created.
pub created_at: DateTime<Utc>,
/// When the job was started.
@@ -159,6 +163,8 @@ impl JobContext {
estimated_cost: None,
estimated_duration: None,
actual_cost: Decimal::ZERO,
total_tokens_used: 0,
max_tokens: 0,
created_at: Utc::now(),
started_at: None,
completed_at: None,
@@ -189,6 +195,14 @@ impl JobContext {
};
self.transitions.push(transition);
// Cap transition history to prevent unbounded memory growth
const MAX_TRANSITIONS: usize = 200;
if self.transitions.len() > MAX_TRANSITIONS {
let drain_count = self.transitions.len() - MAX_TRANSITIONS;
self.transitions.drain(..drain_count);
}
self.state = new_state;
// Update timestamps
@@ -210,6 +224,29 @@ impl JobContext {
self.actual_cost += cost;
}
/// Record token usage from an LLM call. Returns an error string if the
/// token budget has been exceeded after this addition.
pub fn add_tokens(&mut self, tokens: u64) -> Result<(), String> {
self.total_tokens_used += tokens;
if self.max_tokens > 0 && self.total_tokens_used > self.max_tokens {
Err(format!(
"Token budget exceeded: used {} of {} allowed tokens",
self.total_tokens_used, self.max_tokens
))
} else {
Ok(())
}
}
/// Check whether the monetary budget has been exceeded.
pub fn budget_exceeded(&self) -> bool {
if let Some(ref budget) = self.budget {
self.actual_cost > *budget
} else {
false
}
}
/// Get the duration since the job started.
pub fn elapsed(&self) -> Option<Duration> {
self.started_at.map(|start| {
@@ -274,6 +311,57 @@ mod tests {
assert_eq!(ctx.state, JobState::Completed);
}
#[test]
fn test_transition_history_capped() {
let mut ctx = JobContext::new("Test", "Transition cap test");
// Cycle through Pending -> InProgress -> Stuck -> InProgress -> Stuck ...
ctx.transition_to(JobState::InProgress, None).unwrap();
for i in 0..250 {
ctx.mark_stuck(format!("stuck {}", i)).unwrap();
ctx.attempt_recovery().unwrap();
}
// 1 initial + 250*2 = 501 transitions, should be capped at 200
assert!(
ctx.transitions.len() <= 200,
"transitions should be capped at 200, got {}",
ctx.transitions.len()
);
}
#[test]
fn test_add_tokens_enforces_budget() {
let mut ctx = JobContext::new("Test", "Budget test");
ctx.max_tokens = 1000;
assert!(ctx.add_tokens(500).is_ok());
assert_eq!(ctx.total_tokens_used, 500);
assert!(ctx.add_tokens(600).is_err());
assert_eq!(ctx.total_tokens_used, 1100); // tokens still recorded
}
#[test]
fn test_add_tokens_unlimited() {
let mut ctx = JobContext::new("Test", "No budget");
// max_tokens = 0 means unlimited
assert!(ctx.add_tokens(1_000_000).is_ok());
}
#[test]
fn test_budget_exceeded() {
let mut ctx = JobContext::new("Test", "Money test");
ctx.budget = Some(Decimal::new(100, 0)); // $100
assert!(!ctx.budget_exceeded());
ctx.add_cost(Decimal::new(50, 0));
assert!(!ctx.budget_exceeded());
ctx.add_cost(Decimal::new(60, 0));
assert!(ctx.budget_exceeded());
}
#[test]
fn test_budget_exceeded_none() {
let ctx = JobContext::new("Test", "No budget");
assert!(!ctx.budget_exceeded()); // No budget = never exceeded
}
#[test]
fn test_stuck_recovery() {
let mut ctx = JobContext::new("Test", "Test job");
File diff suppressed because it is too large Load Diff
+549
View File
@@ -0,0 +1,549 @@
//! SQLite-dialect migrations for the libSQL/Turso backend.
//!
//! Consolidates all PostgreSQL migrations (V1-V8) into a single SQLite-compatible
//! schema. Run once on database creation; idempotent via `IF NOT EXISTS`.
/// Consolidated schema for libSQL.
///
/// Translates PostgreSQL types and features:
/// - `UUID` -> `TEXT` (store as hex string)
/// - `TIMESTAMPTZ` -> `TEXT` (ISO-8601)
/// - `JSONB` -> `TEXT` (JSON encoded)
/// - `BYTEA` -> `BLOB`
/// - `NUMERIC` -> `TEXT` (preserve precision for rust_decimal)
/// - `TEXT[]` -> `TEXT` (JSON array)
/// - `VECTOR(1536)` -> `F32_BLOB(1536)` (libsql native)
/// - `TSVECTOR` -> FTS5 virtual table
/// - `BIGSERIAL` -> `INTEGER PRIMARY KEY AUTOINCREMENT`
/// - PL/pgSQL functions -> SQLite triggers
pub const SCHEMA: &str = r#"
-- ==================== Migration tracking ====================
CREATE TABLE IF NOT EXISTS _migrations (
version INTEGER PRIMARY KEY,
name TEXT NOT NULL,
applied_at TEXT NOT NULL DEFAULT (datetime('now'))
);
-- ==================== Conversations ====================
CREATE TABLE IF NOT EXISTS conversations (
id TEXT PRIMARY KEY,
channel TEXT NOT NULL,
user_id TEXT NOT NULL,
thread_id TEXT,
started_at TEXT NOT NULL DEFAULT (datetime('now')),
last_activity TEXT NOT NULL DEFAULT (datetime('now')),
metadata TEXT NOT NULL DEFAULT '{}'
);
CREATE INDEX IF NOT EXISTS idx_conversations_channel ON conversations(channel);
CREATE INDEX IF NOT EXISTS idx_conversations_user ON conversations(user_id);
CREATE INDEX IF NOT EXISTS idx_conversations_last_activity ON conversations(last_activity);
CREATE TABLE IF NOT EXISTS conversation_messages (
id TEXT PRIMARY KEY,
conversation_id TEXT NOT NULL REFERENCES conversations(id) ON DELETE CASCADE,
role TEXT NOT NULL,
content TEXT NOT NULL,
created_at TEXT NOT NULL DEFAULT (datetime('now'))
);
CREATE INDEX IF NOT EXISTS idx_conversation_messages_conversation
ON conversation_messages(conversation_id);
-- ==================== Agent Jobs ====================
CREATE TABLE IF NOT EXISTS agent_jobs (
id TEXT PRIMARY KEY,
marketplace_job_id TEXT,
conversation_id TEXT REFERENCES conversations(id),
title TEXT NOT NULL,
description TEXT NOT NULL,
category TEXT,
status TEXT NOT NULL,
source TEXT NOT NULL,
user_id TEXT NOT NULL DEFAULT 'default',
project_dir TEXT,
job_mode TEXT NOT NULL DEFAULT 'worker',
budget_amount TEXT,
budget_token TEXT,
bid_amount TEXT,
estimated_cost TEXT,
estimated_time_secs INTEGER,
estimated_value TEXT,
actual_cost TEXT,
actual_time_secs INTEGER,
success INTEGER,
failure_reason TEXT,
stuck_since TEXT,
repair_attempts INTEGER NOT NULL DEFAULT 0,
created_at TEXT NOT NULL DEFAULT (datetime('now')),
started_at TEXT,
completed_at TEXT
);
CREATE INDEX IF NOT EXISTS idx_agent_jobs_status ON agent_jobs(status);
CREATE INDEX IF NOT EXISTS idx_agent_jobs_marketplace ON agent_jobs(marketplace_job_id);
CREATE INDEX IF NOT EXISTS idx_agent_jobs_conversation ON agent_jobs(conversation_id);
CREATE INDEX IF NOT EXISTS idx_agent_jobs_source ON agent_jobs(source);
CREATE INDEX IF NOT EXISTS idx_agent_jobs_user ON agent_jobs(user_id);
CREATE INDEX IF NOT EXISTS idx_agent_jobs_created ON agent_jobs(created_at DESC);
CREATE TABLE IF NOT EXISTS job_actions (
id TEXT PRIMARY KEY,
job_id TEXT NOT NULL REFERENCES agent_jobs(id) ON DELETE CASCADE,
sequence_num INTEGER NOT NULL,
tool_name TEXT NOT NULL,
input TEXT NOT NULL,
output_raw TEXT,
output_sanitized TEXT,
sanitization_warnings TEXT,
cost TEXT,
duration_ms INTEGER,
success INTEGER NOT NULL,
error_message TEXT,
created_at TEXT NOT NULL DEFAULT (datetime('now')),
UNIQUE(job_id, sequence_num)
);
CREATE INDEX IF NOT EXISTS idx_job_actions_job_id ON job_actions(job_id);
CREATE INDEX IF NOT EXISTS idx_job_actions_tool ON job_actions(tool_name);
-- ==================== Dynamic Tools ====================
CREATE TABLE IF NOT EXISTS dynamic_tools (
id TEXT PRIMARY KEY,
name TEXT NOT NULL UNIQUE,
description TEXT NOT NULL,
parameters_schema TEXT NOT NULL,
code TEXT NOT NULL,
sandbox_config TEXT NOT NULL,
created_by_job_id TEXT REFERENCES agent_jobs(id),
success_count INTEGER NOT NULL DEFAULT 0,
failure_count INTEGER NOT NULL DEFAULT 0,
last_error TEXT,
status TEXT NOT NULL DEFAULT 'active',
created_at TEXT NOT NULL DEFAULT (datetime('now')),
updated_at TEXT NOT NULL DEFAULT (datetime('now'))
);
CREATE INDEX IF NOT EXISTS idx_dynamic_tools_status ON dynamic_tools(status);
CREATE INDEX IF NOT EXISTS idx_dynamic_tools_name ON dynamic_tools(name);
-- ==================== LLM Calls ====================
CREATE TABLE IF NOT EXISTS llm_calls (
id TEXT PRIMARY KEY,
job_id TEXT REFERENCES agent_jobs(id) ON DELETE CASCADE,
conversation_id TEXT REFERENCES conversations(id),
provider TEXT NOT NULL,
model TEXT NOT NULL,
input_tokens INTEGER NOT NULL,
output_tokens INTEGER NOT NULL,
cost TEXT NOT NULL,
purpose TEXT,
created_at TEXT NOT NULL DEFAULT (datetime('now'))
);
CREATE INDEX IF NOT EXISTS idx_llm_calls_job ON llm_calls(job_id);
CREATE INDEX IF NOT EXISTS idx_llm_calls_conversation ON llm_calls(conversation_id);
CREATE INDEX IF NOT EXISTS idx_llm_calls_provider ON llm_calls(provider);
-- ==================== Estimation ====================
CREATE TABLE IF NOT EXISTS estimation_snapshots (
id TEXT PRIMARY KEY,
job_id TEXT NOT NULL REFERENCES agent_jobs(id) ON DELETE CASCADE,
category TEXT NOT NULL,
tool_names TEXT NOT NULL DEFAULT '[]',
estimated_cost TEXT NOT NULL,
actual_cost TEXT,
estimated_time_secs INTEGER NOT NULL,
actual_time_secs INTEGER,
estimated_value TEXT NOT NULL,
actual_value TEXT,
created_at TEXT NOT NULL DEFAULT (datetime('now'))
);
CREATE INDEX IF NOT EXISTS idx_estimation_category ON estimation_snapshots(category);
CREATE INDEX IF NOT EXISTS idx_estimation_job ON estimation_snapshots(job_id);
-- ==================== Self Repair ====================
CREATE TABLE IF NOT EXISTS repair_attempts (
id TEXT PRIMARY KEY,
target_type TEXT NOT NULL,
target_id TEXT NOT NULL,
diagnosis TEXT NOT NULL,
action_taken TEXT NOT NULL,
success INTEGER NOT NULL,
error_message TEXT,
created_at TEXT NOT NULL DEFAULT (datetime('now'))
);
CREATE INDEX IF NOT EXISTS idx_repair_attempts_target ON repair_attempts(target_type, target_id);
CREATE INDEX IF NOT EXISTS idx_repair_attempts_created ON repair_attempts(created_at);
-- ==================== Workspace: Memory Documents ====================
CREATE TABLE IF NOT EXISTS memory_documents (
id TEXT PRIMARY KEY,
user_id TEXT NOT NULL,
agent_id TEXT,
path TEXT NOT NULL,
content TEXT NOT NULL,
created_at TEXT NOT NULL DEFAULT (datetime('now')),
updated_at TEXT NOT NULL DEFAULT (datetime('now')),
metadata TEXT NOT NULL DEFAULT '{}',
UNIQUE (user_id, agent_id, path)
);
CREATE INDEX IF NOT EXISTS idx_memory_documents_user ON memory_documents(user_id);
CREATE INDEX IF NOT EXISTS idx_memory_documents_path ON memory_documents(user_id, path);
CREATE INDEX IF NOT EXISTS idx_memory_documents_updated ON memory_documents(updated_at DESC);
-- Trigger to auto-update updated_at on memory_documents
CREATE TRIGGER IF NOT EXISTS update_memory_documents_updated_at
AFTER UPDATE ON memory_documents
FOR EACH ROW
WHEN NEW.updated_at = OLD.updated_at
BEGIN
UPDATE memory_documents SET updated_at = datetime('now') WHERE id = NEW.id;
END;
-- ==================== Workspace: Memory Chunks ====================
CREATE TABLE IF NOT EXISTS memory_chunks (
_rowid INTEGER PRIMARY KEY AUTOINCREMENT,
id TEXT NOT NULL UNIQUE,
document_id TEXT NOT NULL REFERENCES memory_documents(id) ON DELETE CASCADE,
chunk_index INTEGER NOT NULL,
content TEXT NOT NULL,
embedding F32_BLOB(1536),
created_at TEXT NOT NULL DEFAULT (datetime('now')),
UNIQUE (document_id, chunk_index)
);
CREATE INDEX IF NOT EXISTS idx_memory_chunks_document ON memory_chunks(document_id);
-- Vector index for semantic search (libSQL native)
CREATE INDEX IF NOT EXISTS idx_memory_chunks_embedding
ON memory_chunks (libsql_vector_idx(embedding));
-- FTS5 virtual table for full-text search
CREATE VIRTUAL TABLE IF NOT EXISTS memory_chunks_fts USING fts5(
content,
content='memory_chunks',
content_rowid='_rowid'
);
-- Triggers to keep FTS5 in sync with memory_chunks
CREATE TRIGGER IF NOT EXISTS memory_chunks_fts_insert AFTER INSERT ON memory_chunks BEGIN
INSERT INTO memory_chunks_fts(rowid, content) VALUES (new._rowid, new.content);
END;
CREATE TRIGGER IF NOT EXISTS memory_chunks_fts_delete AFTER DELETE ON memory_chunks BEGIN
INSERT INTO memory_chunks_fts(memory_chunks_fts, rowid, content)
VALUES ('delete', old._rowid, old.content);
END;
CREATE TRIGGER IF NOT EXISTS memory_chunks_fts_update AFTER UPDATE ON memory_chunks BEGIN
INSERT INTO memory_chunks_fts(memory_chunks_fts, rowid, content)
VALUES ('delete', old._rowid, old.content);
INSERT INTO memory_chunks_fts(rowid, content) VALUES (new._rowid, new.content);
END;
-- ==================== Workspace: Heartbeat State ====================
CREATE TABLE IF NOT EXISTS heartbeat_state (
id TEXT PRIMARY KEY,
user_id TEXT NOT NULL,
agent_id TEXT,
last_run TEXT,
next_run TEXT,
interval_seconds INTEGER NOT NULL DEFAULT 1800,
enabled INTEGER NOT NULL DEFAULT 1,
consecutive_failures INTEGER NOT NULL DEFAULT 0,
last_checks TEXT NOT NULL DEFAULT '{}',
UNIQUE (user_id, agent_id)
);
CREATE INDEX IF NOT EXISTS idx_heartbeat_user ON heartbeat_state(user_id);
-- ==================== Secrets ====================
CREATE TABLE IF NOT EXISTS secrets (
id TEXT PRIMARY KEY,
user_id TEXT NOT NULL,
name TEXT NOT NULL,
encrypted_value BLOB NOT NULL,
key_salt BLOB NOT NULL,
provider TEXT,
expires_at TEXT,
last_used_at TEXT,
usage_count INTEGER NOT NULL DEFAULT 0,
created_at TEXT NOT NULL DEFAULT (datetime('now')),
updated_at TEXT NOT NULL DEFAULT (datetime('now')),
UNIQUE (user_id, name)
);
CREATE INDEX IF NOT EXISTS idx_secrets_user ON secrets(user_id);
-- ==================== WASM Tools ====================
CREATE TABLE IF NOT EXISTS wasm_tools (
id TEXT PRIMARY KEY,
user_id TEXT NOT NULL,
name TEXT NOT NULL,
version TEXT NOT NULL DEFAULT '1.0.0',
description TEXT NOT NULL,
wasm_binary BLOB NOT NULL,
binary_hash BLOB NOT NULL,
parameters_schema TEXT NOT NULL,
source_url TEXT,
trust_level TEXT NOT NULL DEFAULT 'user',
status TEXT NOT NULL DEFAULT 'active',
created_at TEXT NOT NULL DEFAULT (datetime('now')),
updated_at TEXT NOT NULL DEFAULT (datetime('now')),
UNIQUE (user_id, name, version)
);
CREATE INDEX IF NOT EXISTS idx_wasm_tools_user ON wasm_tools(user_id);
CREATE INDEX IF NOT EXISTS idx_wasm_tools_name ON wasm_tools(user_id, name);
CREATE INDEX IF NOT EXISTS idx_wasm_tools_status ON wasm_tools(status);
-- ==================== Tool Capabilities ====================
CREATE TABLE IF NOT EXISTS tool_capabilities (
id TEXT PRIMARY KEY,
wasm_tool_id TEXT NOT NULL REFERENCES wasm_tools(id) ON DELETE CASCADE,
http_allowlist TEXT NOT NULL DEFAULT '[]',
allowed_secrets TEXT NOT NULL DEFAULT '[]',
tool_aliases TEXT NOT NULL DEFAULT '{}',
requests_per_minute INTEGER NOT NULL DEFAULT 60,
requests_per_hour INTEGER NOT NULL DEFAULT 1000,
max_request_body_bytes INTEGER NOT NULL DEFAULT 1048576,
max_response_body_bytes INTEGER NOT NULL DEFAULT 10485760,
workspace_read_prefixes TEXT NOT NULL DEFAULT '[]',
http_timeout_secs INTEGER NOT NULL DEFAULT 30,
created_at TEXT NOT NULL DEFAULT (datetime('now')),
updated_at TEXT NOT NULL DEFAULT (datetime('now')),
UNIQUE (wasm_tool_id)
);
-- ==================== Leak Detection Patterns ====================
CREATE TABLE IF NOT EXISTS leak_detection_patterns (
id TEXT PRIMARY KEY,
name TEXT NOT NULL UNIQUE,
pattern TEXT NOT NULL,
severity TEXT NOT NULL DEFAULT 'high',
action TEXT NOT NULL DEFAULT 'block',
enabled INTEGER NOT NULL DEFAULT 1,
created_at TEXT NOT NULL DEFAULT (datetime('now'))
);
-- ==================== Rate Limit State ====================
CREATE TABLE IF NOT EXISTS tool_rate_limit_state (
id TEXT PRIMARY KEY,
wasm_tool_id TEXT NOT NULL REFERENCES wasm_tools(id) ON DELETE CASCADE,
user_id TEXT NOT NULL,
minute_window_start TEXT NOT NULL DEFAULT (datetime('now')),
minute_count INTEGER NOT NULL DEFAULT 0,
hour_window_start TEXT NOT NULL DEFAULT (datetime('now')),
hour_count INTEGER NOT NULL DEFAULT 0,
UNIQUE (wasm_tool_id, user_id)
);
-- ==================== Secret Usage Audit Log ====================
CREATE TABLE IF NOT EXISTS secret_usage_log (
id TEXT PRIMARY KEY,
secret_id TEXT NOT NULL REFERENCES secrets(id) ON DELETE CASCADE,
wasm_tool_id TEXT REFERENCES wasm_tools(id) ON DELETE SET NULL,
user_id TEXT NOT NULL,
target_host TEXT NOT NULL,
target_path TEXT,
success INTEGER NOT NULL,
error_message TEXT,
created_at TEXT NOT NULL DEFAULT (datetime('now'))
);
CREATE INDEX IF NOT EXISTS idx_secret_usage_user ON secret_usage_log(user_id);
-- ==================== Leak Detection Events ====================
CREATE TABLE IF NOT EXISTS leak_detection_events (
id TEXT PRIMARY KEY,
pattern_id TEXT REFERENCES leak_detection_patterns(id) ON DELETE SET NULL,
wasm_tool_id TEXT REFERENCES wasm_tools(id) ON DELETE SET NULL,
user_id TEXT NOT NULL,
source TEXT NOT NULL,
action_taken TEXT NOT NULL,
context_preview TEXT,
created_at TEXT NOT NULL DEFAULT (datetime('now'))
);
-- ==================== Tool Failures ====================
CREATE TABLE IF NOT EXISTS tool_failures (
id TEXT PRIMARY KEY,
tool_name TEXT NOT NULL UNIQUE,
error_message TEXT,
error_count INTEGER DEFAULT 1,
first_failure TEXT DEFAULT (datetime('now')),
last_failure TEXT DEFAULT (datetime('now')),
last_build_result TEXT,
repaired_at TEXT,
repair_attempts INTEGER DEFAULT 0
);
CREATE INDEX IF NOT EXISTS idx_tool_failures_name ON tool_failures(tool_name);
-- ==================== Job Events ====================
CREATE TABLE IF NOT EXISTS job_events (
id INTEGER PRIMARY KEY AUTOINCREMENT,
job_id TEXT NOT NULL REFERENCES agent_jobs(id),
event_type TEXT NOT NULL,
data TEXT NOT NULL,
created_at TEXT NOT NULL DEFAULT (datetime('now'))
);
CREATE INDEX IF NOT EXISTS idx_job_events_job ON job_events(job_id, id);
-- ==================== Routines ====================
CREATE TABLE IF NOT EXISTS routines (
id TEXT PRIMARY KEY,
name TEXT NOT NULL,
description TEXT NOT NULL DEFAULT '',
user_id TEXT NOT NULL,
enabled INTEGER NOT NULL DEFAULT 1,
trigger_type TEXT NOT NULL,
trigger_config TEXT NOT NULL,
action_type TEXT NOT NULL,
action_config TEXT NOT NULL,
cooldown_secs INTEGER NOT NULL DEFAULT 300,
max_concurrent INTEGER NOT NULL DEFAULT 1,
dedup_window_secs INTEGER,
notify_channel TEXT,
notify_user TEXT NOT NULL DEFAULT 'default',
notify_on_success INTEGER NOT NULL DEFAULT 0,
notify_on_failure INTEGER NOT NULL DEFAULT 1,
notify_on_attention INTEGER NOT NULL DEFAULT 1,
state TEXT NOT NULL DEFAULT '{}',
last_run_at TEXT,
next_fire_at TEXT,
run_count INTEGER NOT NULL DEFAULT 0,
consecutive_failures INTEGER NOT NULL DEFAULT 0,
created_at TEXT NOT NULL DEFAULT (datetime('now')),
updated_at TEXT NOT NULL DEFAULT (datetime('now')),
UNIQUE (user_id, name)
);
CREATE INDEX IF NOT EXISTS idx_routines_user ON routines(user_id);
-- ==================== Routine Runs ====================
CREATE TABLE IF NOT EXISTS routine_runs (
id TEXT PRIMARY KEY,
routine_id TEXT NOT NULL REFERENCES routines(id) ON DELETE CASCADE,
trigger_type TEXT NOT NULL,
trigger_detail TEXT,
started_at TEXT NOT NULL DEFAULT (datetime('now')),
completed_at TEXT,
status TEXT NOT NULL DEFAULT 'running',
result_summary TEXT,
tokens_used INTEGER,
job_id TEXT REFERENCES agent_jobs(id),
created_at TEXT NOT NULL DEFAULT (datetime('now'))
);
CREATE INDEX IF NOT EXISTS idx_routine_runs_routine ON routine_runs(routine_id);
-- ==================== Settings ====================
CREATE TABLE IF NOT EXISTS settings (
user_id TEXT NOT NULL,
key TEXT NOT NULL,
value TEXT NOT NULL,
updated_at TEXT NOT NULL DEFAULT (datetime('now')),
PRIMARY KEY (user_id, key)
);
CREATE INDEX IF NOT EXISTS idx_settings_user ON settings(user_id);
-- ==================== Missing indexes (parity with PostgreSQL) ====================
-- agent_jobs
CREATE INDEX IF NOT EXISTS idx_agent_jobs_stuck ON agent_jobs(stuck_since);
-- secrets
CREATE INDEX IF NOT EXISTS idx_secrets_provider ON secrets(provider);
CREATE INDEX IF NOT EXISTS idx_secrets_expires ON secrets(expires_at);
-- wasm_tools
CREATE INDEX IF NOT EXISTS idx_wasm_tools_trust ON wasm_tools(trust_level);
-- tool_capabilities
CREATE INDEX IF NOT EXISTS idx_tool_capabilities_tool ON tool_capabilities(wasm_tool_id);
-- leak_detection_patterns
CREATE INDEX IF NOT EXISTS idx_leak_patterns_enabled ON leak_detection_patterns(enabled);
-- tool_rate_limit_state
CREATE INDEX IF NOT EXISTS idx_rate_limit_tool ON tool_rate_limit_state(wasm_tool_id);
-- secret_usage_log
CREATE INDEX IF NOT EXISTS idx_secret_usage_secret ON secret_usage_log(secret_id);
CREATE INDEX IF NOT EXISTS idx_secret_usage_tool ON secret_usage_log(wasm_tool_id);
CREATE INDEX IF NOT EXISTS idx_secret_usage_created ON secret_usage_log(created_at DESC);
-- leak_detection_events
CREATE INDEX IF NOT EXISTS idx_leak_events_pattern ON leak_detection_events(pattern_id);
CREATE INDEX IF NOT EXISTS idx_leak_events_tool ON leak_detection_events(wasm_tool_id);
CREATE INDEX IF NOT EXISTS idx_leak_events_user ON leak_detection_events(user_id);
CREATE INDEX IF NOT EXISTS idx_leak_events_created ON leak_detection_events(created_at DESC);
-- tool_failures
CREATE INDEX IF NOT EXISTS idx_tool_failures_count ON tool_failures(error_count DESC);
CREATE INDEX IF NOT EXISTS idx_tool_failures_unrepaired ON tool_failures(tool_name);
-- routines
CREATE INDEX IF NOT EXISTS idx_routines_next_fire ON routines(next_fire_at);
CREATE INDEX IF NOT EXISTS idx_routines_event_triggers ON routines(user_id);
-- routine_runs
CREATE INDEX IF NOT EXISTS idx_routine_runs_status ON routine_runs(status);
-- heartbeat_state
CREATE INDEX IF NOT EXISTS idx_heartbeat_next_run ON heartbeat_state(next_run);
-- ==================== Seed data ====================
-- Pre-populate leak detection patterns (matches PostgreSQL V2 migration).
INSERT OR IGNORE INTO leak_detection_patterns (id, name, pattern, severity, action, enabled, created_at) VALUES
('550e8400-e29b-41d4-a716-446655440001', 'openai_api_key', 'sk-(?:proj-)?[a-zA-Z0-9]{20,}(?:T3BlbkFJ[a-zA-Z0-9_-]*)?', 'critical', 'block', 1, datetime('now')),
('550e8400-e29b-41d4-a716-446655440002', 'anthropic_api_key', 'sk-ant-api[a-zA-Z0-9_-]{90,}', 'critical', 'block', 1, datetime('now')),
('550e8400-e29b-41d4-a716-446655440003', 'aws_access_key', 'AKIA[0-9A-Z]{16}', 'critical', 'block', 1, datetime('now')),
('550e8400-e29b-41d4-a716-446655440004', 'aws_secret_key', '(?<![A-Za-z0-9/+=])[A-Za-z0-9/+=]{40}(?![A-Za-z0-9/+=])', 'high', 'block', 1, datetime('now')),
('550e8400-e29b-41d4-a716-446655440005', 'github_token', 'gh[pousr]_[A-Za-z0-9_]{36,}', 'critical', 'block', 1, datetime('now')),
('550e8400-e29b-41d4-a716-446655440006', 'github_fine_grained_pat', 'github_pat_[a-zA-Z0-9]{22}_[a-zA-Z0-9]{59}', 'critical', 'block', 1, datetime('now')),
('550e8400-e29b-41d4-a716-446655440007', 'stripe_api_key', 'sk_(?:live|test)_[a-zA-Z0-9]{24,}', 'critical', 'block', 1, datetime('now')),
('550e8400-e29b-41d4-a716-446655440008', 'nearai_session', 'sess_[a-zA-Z0-9]{32,}', 'critical', 'block', 1, datetime('now')),
('550e8400-e29b-41d4-a716-446655440009', 'bearer_token', 'Bearer\s+[a-zA-Z0-9_-]{20,}', 'high', 'redact', 1, datetime('now')),
('550e8400-e29b-41d4-a716-44665544000a', 'pem_private_key', '-----BEGIN\s+(?:RSA\s+)?PRIVATE\s+KEY-----', 'critical', 'block', 1, datetime('now')),
('550e8400-e29b-41d4-a716-44665544000b', 'ssh_private_key', '-----BEGIN\s+(?:OPENSSH|EC|DSA)\s+PRIVATE\s+KEY-----', 'critical', 'block', 1, datetime('now')),
('550e8400-e29b-41d4-a716-44665544000c', 'google_api_key', 'AIza[0-9A-Za-z_-]{35}', 'high', 'block', 1, datetime('now')),
('550e8400-e29b-41d4-a716-44665544000d', 'slack_token', 'xox[baprs]-[0-9a-zA-Z-]{10,}', 'high', 'block', 1, datetime('now')),
('550e8400-e29b-41d4-a716-44665544000e', 'discord_token', '[MN][A-Za-z\d]{23,}\.[\w-]{6}\.[\w-]{27}', 'high', 'block', 1, datetime('now')),
('550e8400-e29b-41d4-a716-44665544000f', 'twilio_api_key', 'SK[a-fA-F0-9]{32}', 'high', 'block', 1, datetime('now')),
('550e8400-e29b-41d4-a716-446655440010', 'sendgrid_api_key', 'SG\.[a-zA-Z0-9_-]{22}\.[a-zA-Z0-9_-]{43}', 'high', 'block', 1, datetime('now')),
('550e8400-e29b-41d4-a716-446655440011', 'mailchimp_api_key', '[a-f0-9]{32}-us[0-9]{1,2}', 'medium', 'block', 1, datetime('now')),
('550e8400-e29b-41d4-a716-446655440012', 'high_entropy_hex', '(?<![a-fA-F0-9])[a-fA-F0-9]{64}(?![a-fA-F0-9])', 'medium', 'warn', 1, datetime('now'));
"#;
+538
View File
@@ -0,0 +1,538 @@
//! Database abstraction layer.
//!
//! Provides a backend-agnostic `Database` trait that unifies all persistence
//! operations. Two implementations exist behind feature flags:
//!
//! - `postgres` (default): Uses `deadpool-postgres` + `tokio-postgres`
//! - `libsql`: Uses libSQL (Turso's SQLite fork) for embedded/edge deployment
//!
//! The existing `Store`, `Repository`, `SecretsStore`, and `WasmToolStore`
//! types become thin wrappers that delegate to `Arc<dyn Database>`.
#[cfg(feature = "postgres")]
pub mod postgres;
#[cfg(feature = "libsql")]
pub mod libsql_backend;
#[cfg(feature = "libsql")]
pub mod libsql_migrations;
use std::collections::HashMap;
use std::sync::Arc;
use async_trait::async_trait;
use chrono::{DateTime, Utc};
use rust_decimal::Decimal;
use uuid::Uuid;
use crate::agent::BrokenTool;
use crate::agent::routine::{Routine, RoutineRun, RunStatus};
use crate::context::{ActionRecord, JobContext, JobState};
use crate::error::DatabaseError;
use crate::error::WorkspaceError;
use crate::history::{
ConversationMessage, ConversationSummary, JobEventRecord, LlmCallRecord, SandboxJobRecord,
SandboxJobSummary, SettingRow,
};
use crate::workspace::{MemoryChunk, MemoryDocument, WorkspaceEntry};
use crate::workspace::{SearchConfig, SearchResult};
/// Create a database backend from configuration, run migrations, and return it.
///
/// This is the shared helper for CLI commands and other call sites that need
/// a simple `Arc<dyn Database>` without retaining backend-specific handles
/// (e.g., `pg_pool` or `libsql_conn` for the secrets store). The main agent
/// startup in `main.rs` uses its own initialization block because it also
/// captures those backend-specific handles.
pub async fn connect_from_config(
config: &crate::config::DatabaseConfig,
) -> Result<Arc<dyn Database>, DatabaseError> {
match config.backend {
#[cfg(feature = "libsql")]
crate::config::DatabaseBackend::LibSql => {
use secrecy::ExposeSecret as _;
let default_path = crate::config::default_libsql_path();
let db_path = config.libsql_path.as_deref().unwrap_or(&default_path);
let backend = if let Some(ref url) = config.libsql_url {
let token = config.libsql_auth_token.as_ref().ok_or_else(|| {
DatabaseError::Pool(
"LIBSQL_AUTH_TOKEN required when LIBSQL_URL is set".to_string(),
)
})?;
libsql_backend::LibSqlBackend::new_remote_replica(
db_path,
url,
token.expose_secret(),
)
.await
.map_err(|e| DatabaseError::Pool(e.to_string()))?
} else {
libsql_backend::LibSqlBackend::new_local(db_path)
.await
.map_err(|e| DatabaseError::Pool(e.to_string()))?
};
backend.run_migrations().await?;
Ok(Arc::new(backend))
}
#[cfg(feature = "postgres")]
_ => {
let pg = postgres::PgBackend::new(config)
.await
.map_err(|e| DatabaseError::Pool(e.to_string()))?;
pg.run_migrations().await?;
Ok(Arc::new(pg))
}
#[cfg(not(feature = "postgres"))]
_ => Err(DatabaseError::Pool(
"No database backend available. Enable 'postgres' or 'libsql' feature.".to_string(),
)),
}
}
/// Backend-agnostic database trait.
///
/// Combines all persistence operations from Store, Repository, and related
/// stores into a single trait that can be implemented for different backends.
#[async_trait]
pub trait Database: Send + Sync {
/// Run schema migrations for this backend.
async fn run_migrations(&self) -> Result<(), DatabaseError>;
// ==================== Conversations ====================
/// Create a new conversation.
async fn create_conversation(
&self,
channel: &str,
user_id: &str,
thread_id: Option<&str>,
) -> Result<Uuid, DatabaseError>;
/// Update conversation last activity.
async fn touch_conversation(&self, id: Uuid) -> Result<(), DatabaseError>;
/// Add a message to a conversation.
async fn add_conversation_message(
&self,
conversation_id: Uuid,
role: &str,
content: &str,
) -> Result<Uuid, DatabaseError>;
/// Ensure a conversation row exists (upsert).
async fn ensure_conversation(
&self,
id: Uuid,
channel: &str,
user_id: &str,
thread_id: Option<&str>,
) -> Result<(), DatabaseError>;
/// List conversations with a title preview.
async fn list_conversations_with_preview(
&self,
user_id: &str,
channel: &str,
limit: i64,
) -> Result<Vec<ConversationSummary>, DatabaseError>;
/// Get or create the singleton assistant conversation.
async fn get_or_create_assistant_conversation(
&self,
user_id: &str,
channel: &str,
) -> Result<Uuid, DatabaseError>;
/// Create a conversation with specific metadata.
async fn create_conversation_with_metadata(
&self,
channel: &str,
user_id: &str,
metadata: &serde_json::Value,
) -> Result<Uuid, DatabaseError>;
/// Load messages with cursor-based pagination.
async fn list_conversation_messages_paginated(
&self,
conversation_id: Uuid,
before: Option<DateTime<Utc>>,
limit: i64,
) -> Result<(Vec<ConversationMessage>, bool), DatabaseError>;
/// Merge a single key into conversation metadata.
async fn update_conversation_metadata_field(
&self,
id: Uuid,
key: &str,
value: &serde_json::Value,
) -> Result<(), DatabaseError>;
/// Read conversation metadata.
async fn get_conversation_metadata(
&self,
id: Uuid,
) -> Result<Option<serde_json::Value>, DatabaseError>;
/// Load all messages for a conversation.
async fn list_conversation_messages(
&self,
conversation_id: Uuid,
) -> Result<Vec<ConversationMessage>, DatabaseError>;
/// Check if a conversation belongs to a specific user.
async fn conversation_belongs_to_user(
&self,
conversation_id: Uuid,
user_id: &str,
) -> Result<bool, DatabaseError>;
// ==================== Jobs ====================
/// Save a job context.
async fn save_job(&self, ctx: &JobContext) -> Result<(), DatabaseError>;
/// Get a job by ID.
async fn get_job(&self, id: Uuid) -> Result<Option<JobContext>, DatabaseError>;
/// Update job status.
async fn update_job_status(
&self,
id: Uuid,
status: JobState,
failure_reason: Option<&str>,
) -> Result<(), DatabaseError>;
/// Mark job as stuck.
async fn mark_job_stuck(&self, id: Uuid) -> Result<(), DatabaseError>;
/// Get stuck jobs.
async fn get_stuck_jobs(&self) -> Result<Vec<Uuid>, DatabaseError>;
// ==================== Actions ====================
/// Save a job action.
async fn save_action(&self, job_id: Uuid, action: &ActionRecord) -> Result<(), DatabaseError>;
/// Get actions for a job.
async fn get_job_actions(&self, job_id: Uuid) -> Result<Vec<ActionRecord>, DatabaseError>;
// ==================== LLM Calls ====================
/// Record an LLM call.
async fn record_llm_call(&self, record: &LlmCallRecord<'_>) -> Result<Uuid, DatabaseError>;
// ==================== Estimation Snapshots ====================
/// Save an estimation snapshot.
async fn save_estimation_snapshot(
&self,
job_id: Uuid,
category: &str,
tool_names: &[String],
estimated_cost: Decimal,
estimated_time_secs: i32,
estimated_value: Decimal,
) -> Result<Uuid, DatabaseError>;
/// Update estimation snapshot with actual values.
async fn update_estimation_actuals(
&self,
id: Uuid,
actual_cost: Decimal,
actual_time_secs: i32,
actual_value: Option<Decimal>,
) -> Result<(), DatabaseError>;
// ==================== Sandbox Jobs ====================
/// Insert a new sandbox job.
async fn save_sandbox_job(&self, job: &SandboxJobRecord) -> Result<(), DatabaseError>;
/// Get a sandbox job by ID.
async fn get_sandbox_job(&self, id: Uuid) -> Result<Option<SandboxJobRecord>, DatabaseError>;
/// List all sandbox jobs, most recent first.
async fn list_sandbox_jobs(&self) -> Result<Vec<SandboxJobRecord>, DatabaseError>;
/// Update sandbox job status.
async fn update_sandbox_job_status(
&self,
id: Uuid,
status: &str,
success: Option<bool>,
message: Option<&str>,
started_at: Option<DateTime<Utc>>,
completed_at: Option<DateTime<Utc>>,
) -> Result<(), DatabaseError>;
/// Mark stale sandbox jobs as interrupted.
async fn cleanup_stale_sandbox_jobs(&self) -> Result<u64, DatabaseError>;
/// Get sandbox job summary.
async fn sandbox_job_summary(&self) -> Result<SandboxJobSummary, DatabaseError>;
/// List sandbox jobs for a specific user, most recent first.
async fn list_sandbox_jobs_for_user(
&self,
user_id: &str,
) -> Result<Vec<SandboxJobRecord>, DatabaseError>;
/// Get sandbox job summary for a specific user.
async fn sandbox_job_summary_for_user(
&self,
user_id: &str,
) -> Result<SandboxJobSummary, DatabaseError>;
/// Check if a sandbox job belongs to a specific user.
async fn sandbox_job_belongs_to_user(
&self,
job_id: Uuid,
user_id: &str,
) -> Result<bool, DatabaseError>;
/// Update sandbox job mode.
async fn update_sandbox_job_mode(&self, id: Uuid, mode: &str) -> Result<(), DatabaseError>;
/// Get sandbox job mode.
async fn get_sandbox_job_mode(&self, id: Uuid) -> Result<Option<String>, DatabaseError>;
// ==================== Job Events ====================
/// Persist a job event.
async fn save_job_event(
&self,
job_id: Uuid,
event_type: &str,
data: &serde_json::Value,
) -> Result<(), DatabaseError>;
/// Load all job events.
async fn list_job_events(&self, job_id: Uuid) -> Result<Vec<JobEventRecord>, DatabaseError>;
// ==================== Routines ====================
/// Create a new routine.
async fn create_routine(&self, routine: &Routine) -> Result<(), DatabaseError>;
/// Get a routine by ID.
async fn get_routine(&self, id: Uuid) -> Result<Option<Routine>, DatabaseError>;
/// Get a routine by user_id and name.
async fn get_routine_by_name(
&self,
user_id: &str,
name: &str,
) -> Result<Option<Routine>, DatabaseError>;
/// List routines for a user.
async fn list_routines(&self, user_id: &str) -> Result<Vec<Routine>, DatabaseError>;
/// List all enabled event routines.
async fn list_event_routines(&self) -> Result<Vec<Routine>, DatabaseError>;
/// List due cron routines.
async fn list_due_cron_routines(&self) -> Result<Vec<Routine>, DatabaseError>;
/// Update a routine.
async fn update_routine(&self, routine: &Routine) -> Result<(), DatabaseError>;
/// Update runtime state after a routine fires.
async fn update_routine_runtime(
&self,
id: Uuid,
last_run_at: DateTime<Utc>,
next_fire_at: Option<DateTime<Utc>>,
run_count: u64,
consecutive_failures: u32,
state: &serde_json::Value,
) -> Result<(), DatabaseError>;
/// Delete a routine.
async fn delete_routine(&self, id: Uuid) -> Result<bool, DatabaseError>;
// ==================== Routine Runs ====================
/// Record a routine run starting.
async fn create_routine_run(&self, run: &RoutineRun) -> Result<(), DatabaseError>;
/// Complete a routine run.
async fn complete_routine_run(
&self,
id: Uuid,
status: RunStatus,
result_summary: Option<&str>,
tokens_used: Option<i32>,
) -> Result<(), DatabaseError>;
/// List recent runs for a routine.
async fn list_routine_runs(
&self,
routine_id: Uuid,
limit: i64,
) -> Result<Vec<RoutineRun>, DatabaseError>;
/// Count currently running runs for a routine.
async fn count_running_routine_runs(&self, routine_id: Uuid) -> Result<i64, DatabaseError>;
// ==================== Tool Failures ====================
/// Record a tool failure (upsert).
async fn record_tool_failure(
&self,
tool_name: &str,
error_message: &str,
) -> Result<(), DatabaseError>;
/// Get broken tools exceeding threshold.
async fn get_broken_tools(&self, threshold: i32) -> Result<Vec<BrokenTool>, DatabaseError>;
/// Mark a tool as repaired.
async fn mark_tool_repaired(&self, tool_name: &str) -> Result<(), DatabaseError>;
/// Increment repair attempts.
async fn increment_repair_attempts(&self, tool_name: &str) -> Result<(), DatabaseError>;
// ==================== Settings ====================
/// Get a single setting.
async fn get_setting(
&self,
user_id: &str,
key: &str,
) -> Result<Option<serde_json::Value>, DatabaseError>;
/// Get a single setting with metadata.
async fn get_setting_full(
&self,
user_id: &str,
key: &str,
) -> Result<Option<SettingRow>, DatabaseError>;
/// Set a single setting (upsert).
async fn set_setting(
&self,
user_id: &str,
key: &str,
value: &serde_json::Value,
) -> Result<(), DatabaseError>;
/// Delete a single setting.
async fn delete_setting(&self, user_id: &str, key: &str) -> Result<bool, DatabaseError>;
/// List all settings for a user.
async fn list_settings(&self, user_id: &str) -> Result<Vec<SettingRow>, DatabaseError>;
/// Get all settings as a flat map.
async fn get_all_settings(
&self,
user_id: &str,
) -> Result<HashMap<String, serde_json::Value>, DatabaseError>;
/// Bulk-write settings atomically.
async fn set_all_settings(
&self,
user_id: &str,
settings: &HashMap<String, serde_json::Value>,
) -> Result<(), DatabaseError>;
/// Check if settings exist for a user.
async fn has_settings(&self, user_id: &str) -> Result<bool, DatabaseError>;
// ==================== Workspace: Documents ====================
/// Get a document by path.
async fn get_document_by_path(
&self,
user_id: &str,
agent_id: Option<Uuid>,
path: &str,
) -> Result<MemoryDocument, WorkspaceError>;
/// Get a document by ID.
async fn get_document_by_id(&self, id: Uuid) -> Result<MemoryDocument, WorkspaceError>;
/// Get or create a document by path.
async fn get_or_create_document_by_path(
&self,
user_id: &str,
agent_id: Option<Uuid>,
path: &str,
) -> Result<MemoryDocument, WorkspaceError>;
/// Update a document's content.
async fn update_document(&self, id: Uuid, content: &str) -> Result<(), WorkspaceError>;
/// Delete a document by path.
async fn delete_document_by_path(
&self,
user_id: &str,
agent_id: Option<Uuid>,
path: &str,
) -> Result<(), WorkspaceError>;
/// List files and directories in a directory path.
async fn list_directory(
&self,
user_id: &str,
agent_id: Option<Uuid>,
directory: &str,
) -> Result<Vec<WorkspaceEntry>, WorkspaceError>;
/// List all file paths in the workspace.
async fn list_all_paths(
&self,
user_id: &str,
agent_id: Option<Uuid>,
) -> Result<Vec<String>, WorkspaceError>;
/// List all documents for a user.
async fn list_documents(
&self,
user_id: &str,
agent_id: Option<Uuid>,
) -> Result<Vec<MemoryDocument>, WorkspaceError>;
// ==================== Workspace: Chunks ====================
/// Delete all chunks for a document.
async fn delete_chunks(&self, document_id: Uuid) -> Result<(), WorkspaceError>;
/// Insert a chunk.
async fn insert_chunk(
&self,
document_id: Uuid,
chunk_index: i32,
content: &str,
embedding: Option<&[f32]>,
) -> Result<Uuid, WorkspaceError>;
/// Update a chunk's embedding.
async fn update_chunk_embedding(
&self,
chunk_id: Uuid,
embedding: &[f32],
) -> Result<(), WorkspaceError>;
/// Get chunks without embeddings for backfilling.
async fn get_chunks_without_embeddings(
&self,
user_id: &str,
agent_id: Option<Uuid>,
limit: usize,
) -> Result<Vec<MemoryChunk>, WorkspaceError>;
// ==================== Workspace: Search ====================
/// Perform hybrid search combining FTS and vector similarity.
async fn hybrid_search(
&self,
user_id: &str,
agent_id: Option<Uuid>,
query: &str,
embedding: Option<&[f32]>,
config: &SearchConfig,
) -> Result<Vec<SearchResult>, WorkspaceError>;
}
+627
View File
@@ -0,0 +1,627 @@
//! PostgreSQL backend for the Database trait.
//!
//! Delegates to the existing `Store` (history) and `Repository` (workspace)
//! implementations, avoiding SQL duplication.
use std::collections::HashMap;
use async_trait::async_trait;
use chrono::{DateTime, Utc};
use deadpool_postgres::Pool;
use rust_decimal::Decimal;
use uuid::Uuid;
use crate::agent::BrokenTool;
use crate::agent::routine::{Routine, RoutineRun, RunStatus};
use crate::config::DatabaseConfig;
use crate::context::{ActionRecord, JobContext, JobState};
use crate::db::Database;
use crate::error::{DatabaseError, WorkspaceError};
use crate::history::{
ConversationMessage, ConversationSummary, JobEventRecord, LlmCallRecord, SandboxJobRecord,
SandboxJobSummary, SettingRow, Store,
};
use crate::workspace::{
MemoryChunk, MemoryDocument, Repository, SearchConfig, SearchResult, WorkspaceEntry,
};
/// PostgreSQL database backend.
///
/// Wraps the existing `Store` (for history/conversations/jobs/routines/settings)
/// and `Repository` (for workspace documents/chunks/search) to implement the
/// unified `Database` trait.
pub struct PgBackend {
store: Store,
repo: Repository,
}
impl PgBackend {
/// Create a new PostgreSQL backend from configuration.
pub async fn new(config: &DatabaseConfig) -> Result<Self, DatabaseError> {
let store = Store::new(config).await?;
let repo = Repository::new(store.pool());
Ok(Self { store, repo })
}
/// Get a clone of the connection pool.
///
/// Useful for sharing with components that still need raw pool access.
pub fn pool(&self) -> Pool {
self.store.pool()
}
}
#[async_trait]
impl Database for PgBackend {
async fn run_migrations(&self) -> Result<(), DatabaseError> {
self.store.run_migrations().await
}
// ==================== Conversations ====================
async fn create_conversation(
&self,
channel: &str,
user_id: &str,
thread_id: Option<&str>,
) -> Result<Uuid, DatabaseError> {
self.store
.create_conversation(channel, user_id, thread_id)
.await
}
async fn touch_conversation(&self, id: Uuid) -> Result<(), DatabaseError> {
self.store.touch_conversation(id).await
}
async fn add_conversation_message(
&self,
conversation_id: Uuid,
role: &str,
content: &str,
) -> Result<Uuid, DatabaseError> {
self.store
.add_conversation_message(conversation_id, role, content)
.await
}
async fn ensure_conversation(
&self,
id: Uuid,
channel: &str,
user_id: &str,
thread_id: Option<&str>,
) -> Result<(), DatabaseError> {
self.store
.ensure_conversation(id, channel, user_id, thread_id)
.await
}
async fn list_conversations_with_preview(
&self,
user_id: &str,
channel: &str,
limit: i64,
) -> Result<Vec<ConversationSummary>, DatabaseError> {
self.store
.list_conversations_with_preview(user_id, channel, limit)
.await
}
async fn get_or_create_assistant_conversation(
&self,
user_id: &str,
channel: &str,
) -> Result<Uuid, DatabaseError> {
self.store
.get_or_create_assistant_conversation(user_id, channel)
.await
}
async fn create_conversation_with_metadata(
&self,
channel: &str,
user_id: &str,
metadata: &serde_json::Value,
) -> Result<Uuid, DatabaseError> {
self.store
.create_conversation_with_metadata(channel, user_id, metadata)
.await
}
async fn list_conversation_messages_paginated(
&self,
conversation_id: Uuid,
before: Option<DateTime<Utc>>,
limit: i64,
) -> Result<(Vec<ConversationMessage>, bool), DatabaseError> {
self.store
.list_conversation_messages_paginated(conversation_id, before, limit)
.await
}
async fn update_conversation_metadata_field(
&self,
id: Uuid,
key: &str,
value: &serde_json::Value,
) -> Result<(), DatabaseError> {
self.store
.update_conversation_metadata_field(id, key, value)
.await
}
async fn get_conversation_metadata(
&self,
id: Uuid,
) -> Result<Option<serde_json::Value>, DatabaseError> {
self.store.get_conversation_metadata(id).await
}
async fn list_conversation_messages(
&self,
conversation_id: Uuid,
) -> Result<Vec<ConversationMessage>, DatabaseError> {
self.store.list_conversation_messages(conversation_id).await
}
async fn conversation_belongs_to_user(
&self,
conversation_id: Uuid,
user_id: &str,
) -> Result<bool, DatabaseError> {
self.store
.conversation_belongs_to_user(conversation_id, user_id)
.await
}
// ==================== Jobs ====================
async fn save_job(&self, ctx: &JobContext) -> Result<(), DatabaseError> {
self.store.save_job(ctx).await
}
async fn get_job(&self, id: Uuid) -> Result<Option<JobContext>, DatabaseError> {
self.store.get_job(id).await
}
async fn update_job_status(
&self,
id: Uuid,
status: JobState,
failure_reason: Option<&str>,
) -> Result<(), DatabaseError> {
self.store
.update_job_status(id, status, failure_reason)
.await
}
async fn mark_job_stuck(&self, id: Uuid) -> Result<(), DatabaseError> {
self.store.mark_job_stuck(id).await
}
async fn get_stuck_jobs(&self) -> Result<Vec<Uuid>, DatabaseError> {
self.store.get_stuck_jobs().await
}
// ==================== Actions ====================
async fn save_action(&self, job_id: Uuid, action: &ActionRecord) -> Result<(), DatabaseError> {
self.store.save_action(job_id, action).await
}
async fn get_job_actions(&self, job_id: Uuid) -> Result<Vec<ActionRecord>, DatabaseError> {
self.store.get_job_actions(job_id).await
}
// ==================== LLM Calls ====================
async fn record_llm_call(&self, record: &LlmCallRecord<'_>) -> Result<Uuid, DatabaseError> {
self.store.record_llm_call(record).await
}
// ==================== Estimation Snapshots ====================
async fn save_estimation_snapshot(
&self,
job_id: Uuid,
category: &str,
tool_names: &[String],
estimated_cost: Decimal,
estimated_time_secs: i32,
estimated_value: Decimal,
) -> Result<Uuid, DatabaseError> {
self.store
.save_estimation_snapshot(
job_id,
category,
tool_names,
estimated_cost,
estimated_time_secs,
estimated_value,
)
.await
}
async fn update_estimation_actuals(
&self,
id: Uuid,
actual_cost: Decimal,
actual_time_secs: i32,
actual_value: Option<Decimal>,
) -> Result<(), DatabaseError> {
self.store
.update_estimation_actuals(id, actual_cost, actual_time_secs, actual_value)
.await
}
// ==================== Sandbox Jobs ====================
async fn save_sandbox_job(&self, job: &SandboxJobRecord) -> Result<(), DatabaseError> {
self.store.save_sandbox_job(job).await
}
async fn get_sandbox_job(&self, id: Uuid) -> Result<Option<SandboxJobRecord>, DatabaseError> {
self.store.get_sandbox_job(id).await
}
async fn list_sandbox_jobs(&self) -> Result<Vec<SandboxJobRecord>, DatabaseError> {
self.store.list_sandbox_jobs().await
}
async fn update_sandbox_job_status(
&self,
id: Uuid,
status: &str,
success: Option<bool>,
message: Option<&str>,
started_at: Option<DateTime<Utc>>,
completed_at: Option<DateTime<Utc>>,
) -> Result<(), DatabaseError> {
self.store
.update_sandbox_job_status(id, status, success, message, started_at, completed_at)
.await
}
async fn cleanup_stale_sandbox_jobs(&self) -> Result<u64, DatabaseError> {
self.store.cleanup_stale_sandbox_jobs().await
}
async fn sandbox_job_summary(&self) -> Result<SandboxJobSummary, DatabaseError> {
self.store.sandbox_job_summary().await
}
async fn list_sandbox_jobs_for_user(
&self,
user_id: &str,
) -> Result<Vec<SandboxJobRecord>, DatabaseError> {
self.store.list_sandbox_jobs_for_user(user_id).await
}
async fn sandbox_job_summary_for_user(
&self,
user_id: &str,
) -> Result<SandboxJobSummary, DatabaseError> {
self.store.sandbox_job_summary_for_user(user_id).await
}
async fn sandbox_job_belongs_to_user(
&self,
job_id: Uuid,
user_id: &str,
) -> Result<bool, DatabaseError> {
self.store
.sandbox_job_belongs_to_user(job_id, user_id)
.await
}
async fn update_sandbox_job_mode(&self, id: Uuid, mode: &str) -> Result<(), DatabaseError> {
self.store.update_sandbox_job_mode(id, mode).await
}
async fn get_sandbox_job_mode(&self, id: Uuid) -> Result<Option<String>, DatabaseError> {
self.store.get_sandbox_job_mode(id).await
}
// ==================== Job Events ====================
async fn save_job_event(
&self,
job_id: Uuid,
event_type: &str,
data: &serde_json::Value,
) -> Result<(), DatabaseError> {
self.store.save_job_event(job_id, event_type, data).await
}
async fn list_job_events(&self, job_id: Uuid) -> Result<Vec<JobEventRecord>, DatabaseError> {
self.store.list_job_events(job_id).await
}
// ==================== Routines ====================
async fn create_routine(&self, routine: &Routine) -> Result<(), DatabaseError> {
self.store.create_routine(routine).await
}
async fn get_routine(&self, id: Uuid) -> Result<Option<Routine>, DatabaseError> {
self.store.get_routine(id).await
}
async fn get_routine_by_name(
&self,
user_id: &str,
name: &str,
) -> Result<Option<Routine>, DatabaseError> {
self.store.get_routine_by_name(user_id, name).await
}
async fn list_routines(&self, user_id: &str) -> Result<Vec<Routine>, DatabaseError> {
self.store.list_routines(user_id).await
}
async fn list_event_routines(&self) -> Result<Vec<Routine>, DatabaseError> {
self.store.list_event_routines().await
}
async fn list_due_cron_routines(&self) -> Result<Vec<Routine>, DatabaseError> {
self.store.list_due_cron_routines().await
}
async fn update_routine(&self, routine: &Routine) -> Result<(), DatabaseError> {
self.store.update_routine(routine).await
}
async fn update_routine_runtime(
&self,
id: Uuid,
last_run_at: DateTime<Utc>,
next_fire_at: Option<DateTime<Utc>>,
run_count: u64,
consecutive_failures: u32,
state: &serde_json::Value,
) -> Result<(), DatabaseError> {
self.store
.update_routine_runtime(
id,
last_run_at,
next_fire_at,
run_count,
consecutive_failures,
state,
)
.await
}
async fn delete_routine(&self, id: Uuid) -> Result<bool, DatabaseError> {
self.store.delete_routine(id).await
}
// ==================== Routine Runs ====================
async fn create_routine_run(&self, run: &RoutineRun) -> Result<(), DatabaseError> {
self.store.create_routine_run(run).await
}
async fn complete_routine_run(
&self,
id: Uuid,
status: RunStatus,
result_summary: Option<&str>,
tokens_used: Option<i32>,
) -> Result<(), DatabaseError> {
self.store
.complete_routine_run(id, status, result_summary, tokens_used)
.await
}
async fn list_routine_runs(
&self,
routine_id: Uuid,
limit: i64,
) -> Result<Vec<RoutineRun>, DatabaseError> {
self.store.list_routine_runs(routine_id, limit).await
}
async fn count_running_routine_runs(&self, routine_id: Uuid) -> Result<i64, DatabaseError> {
self.store.count_running_routine_runs(routine_id).await
}
// ==================== Tool Failures ====================
async fn record_tool_failure(
&self,
tool_name: &str,
error_message: &str,
) -> Result<(), DatabaseError> {
self.store
.record_tool_failure(tool_name, error_message)
.await
}
async fn get_broken_tools(&self, threshold: i32) -> Result<Vec<BrokenTool>, DatabaseError> {
self.store.get_broken_tools(threshold).await
}
async fn mark_tool_repaired(&self, tool_name: &str) -> Result<(), DatabaseError> {
self.store.mark_tool_repaired(tool_name).await
}
async fn increment_repair_attempts(&self, tool_name: &str) -> Result<(), DatabaseError> {
self.store.increment_repair_attempts(tool_name).await
}
// ==================== Settings ====================
async fn get_setting(
&self,
user_id: &str,
key: &str,
) -> Result<Option<serde_json::Value>, DatabaseError> {
self.store.get_setting(user_id, key).await
}
async fn get_setting_full(
&self,
user_id: &str,
key: &str,
) -> Result<Option<SettingRow>, DatabaseError> {
self.store.get_setting_full(user_id, key).await
}
async fn set_setting(
&self,
user_id: &str,
key: &str,
value: &serde_json::Value,
) -> Result<(), DatabaseError> {
self.store.set_setting(user_id, key, value).await
}
async fn delete_setting(&self, user_id: &str, key: &str) -> Result<bool, DatabaseError> {
self.store.delete_setting(user_id, key).await
}
async fn list_settings(&self, user_id: &str) -> Result<Vec<SettingRow>, DatabaseError> {
self.store.list_settings(user_id).await
}
async fn get_all_settings(
&self,
user_id: &str,
) -> Result<HashMap<String, serde_json::Value>, DatabaseError> {
self.store.get_all_settings(user_id).await
}
async fn set_all_settings(
&self,
user_id: &str,
settings: &HashMap<String, serde_json::Value>,
) -> Result<(), DatabaseError> {
self.store.set_all_settings(user_id, settings).await
}
async fn has_settings(&self, user_id: &str) -> Result<bool, DatabaseError> {
self.store.has_settings(user_id).await
}
// ==================== Workspace: Documents ====================
async fn get_document_by_path(
&self,
user_id: &str,
agent_id: Option<Uuid>,
path: &str,
) -> Result<MemoryDocument, WorkspaceError> {
self.repo
.get_document_by_path(user_id, agent_id, path)
.await
}
async fn get_document_by_id(&self, id: Uuid) -> Result<MemoryDocument, WorkspaceError> {
self.repo.get_document_by_id(id).await
}
async fn get_or_create_document_by_path(
&self,
user_id: &str,
agent_id: Option<Uuid>,
path: &str,
) -> Result<MemoryDocument, WorkspaceError> {
self.repo
.get_or_create_document_by_path(user_id, agent_id, path)
.await
}
async fn update_document(&self, id: Uuid, content: &str) -> Result<(), WorkspaceError> {
self.repo.update_document(id, content).await
}
async fn delete_document_by_path(
&self,
user_id: &str,
agent_id: Option<Uuid>,
path: &str,
) -> Result<(), WorkspaceError> {
self.repo
.delete_document_by_path(user_id, agent_id, path)
.await
}
async fn list_directory(
&self,
user_id: &str,
agent_id: Option<Uuid>,
directory: &str,
) -> Result<Vec<WorkspaceEntry>, WorkspaceError> {
self.repo.list_directory(user_id, agent_id, directory).await
}
async fn list_all_paths(
&self,
user_id: &str,
agent_id: Option<Uuid>,
) -> Result<Vec<String>, WorkspaceError> {
self.repo.list_all_paths(user_id, agent_id).await
}
async fn list_documents(
&self,
user_id: &str,
agent_id: Option<Uuid>,
) -> Result<Vec<MemoryDocument>, WorkspaceError> {
self.repo.list_documents(user_id, agent_id).await
}
// ==================== Workspace: Chunks ====================
async fn delete_chunks(&self, document_id: Uuid) -> Result<(), WorkspaceError> {
self.repo.delete_chunks(document_id).await
}
async fn insert_chunk(
&self,
document_id: Uuid,
chunk_index: i32,
content: &str,
embedding: Option<&[f32]>,
) -> Result<Uuid, WorkspaceError> {
self.repo
.insert_chunk(document_id, chunk_index, content, embedding)
.await
}
async fn update_chunk_embedding(
&self,
chunk_id: Uuid,
embedding: &[f32],
) -> Result<(), WorkspaceError> {
self.repo.update_chunk_embedding(chunk_id, embedding).await
}
async fn get_chunks_without_embeddings(
&self,
user_id: &str,
agent_id: Option<Uuid>,
limit: usize,
) -> Result<Vec<MemoryChunk>, WorkspaceError> {
self.repo
.get_chunks_without_embeddings(user_id, agent_id, limit)
.await
}
// ==================== Workspace: Search ====================
async fn hybrid_search(
&self,
user_id: &str,
agent_id: Option<Uuid>,
query: &str,
embedding: Option<&[f32]>,
config: &SearchConfig,
) -> Result<Vec<SearchResult>, WorkspaceError> {
self.repo
.hybrid_search(user_id, agent_id, query, embedding, config)
.await
}
}
+60
View File
@@ -39,6 +39,12 @@ pub enum Error {
#[error("Workspace error: {0}")]
Workspace(#[from] WorkspaceError),
#[error("Orchestrator error: {0}")]
Orchestrator(#[from] OrchestratorError),
#[error("Worker error: {0}")]
Worker(#[from] WorkerError),
}
/// Configuration-related errors.
@@ -81,14 +87,21 @@ pub enum DatabaseError {
#[error("Serialization error: {0}")]
Serialization(String),
#[cfg(feature = "postgres")]
#[error("PostgreSQL error: {0}")]
Postgres(#[from] tokio_postgres::Error),
#[cfg(feature = "postgres")]
#[error("Pool build error: {0}")]
PoolBuild(#[from] deadpool_postgres::BuildError),
#[cfg(feature = "postgres")]
#[error("Pool runtime error: {0}")]
PoolRuntime(#[from] deadpool_postgres::PoolError),
#[cfg(feature = "libsql")]
#[error("LibSQL error: {0}")]
LibSql(#[from] libsql::Error),
}
/// Channel-related errors.
@@ -308,5 +321,52 @@ pub enum WorkspaceError {
HeartbeatError { reason: String },
}
/// Orchestrator errors (internal API, container management).
#[derive(Debug, thiserror::Error)]
pub enum OrchestratorError {
#[error("Container creation failed for job {job_id}: {reason}")]
ContainerCreationFailed { job_id: Uuid, reason: String },
#[error("Container not found for job {job_id}")]
ContainerNotFound { job_id: Uuid },
#[error("Container for job {job_id} is in unexpected state: {state}")]
InvalidContainerState { job_id: Uuid, state: String },
#[error("Worker authentication failed: {reason}")]
AuthFailed { reason: String },
#[error("Internal API error: {reason}")]
ApiError { reason: String },
#[error("Docker error: {reason}")]
Docker { reason: String },
#[error("Job {job_id} timed out in container")]
ContainerTimeout { job_id: Uuid },
}
/// Worker errors (container-side execution).
#[derive(Debug, thiserror::Error)]
pub enum WorkerError {
#[error("Failed to connect to orchestrator at {url}: {reason}")]
ConnectionFailed { url: String, reason: String },
#[error("LLM proxy request failed: {reason}")]
LlmProxyFailed { reason: String },
#[error("Secret resolution failed for {secret_name}: {reason}")]
SecretResolveFailed { secret_name: String, reason: String },
#[error("Orchestrator returned error for job {job_id}: {reason}")]
OrchestratorRejected { job_id: Uuid, reason: String },
#[error("Worker execution failed: {reason}")]
ExecutionFailed { reason: String },
#[error("Missing worker token (IRONCLAW_WORKER_TOKEN not set)")]
MissingToken,
}
/// Result type alias for the agent.
pub type Result<T> = std::result::Result<T, Error>;
+101 -12
View File
@@ -23,9 +23,7 @@ use crate::tools::mcp::auth::{
PkceChallenge, authorize_mcp_server, build_authorization_url, discover_full_oauth_metadata,
find_available_port, is_authenticated, register_client,
};
use crate::tools::mcp::config::{
McpServerConfig, add_mcp_server, get_mcp_server, load_mcp_servers, remove_mcp_server,
};
use crate::tools::mcp::config::McpServerConfig;
use crate::tools::mcp::session::McpSessionManager;
use crate::tools::wasm::{WasmToolLoader, WasmToolRuntime, discover_tools};
@@ -58,6 +56,8 @@ pub struct ExtensionManager {
/// Tunnel URL for remote OAuth callbacks (used in future iterations).
_tunnel_url: Option<String>,
user_id: String,
/// Optional database store for DB-backed MCP config.
store: Option<Arc<dyn crate::db::Database>>,
}
impl ExtensionManager {
@@ -71,6 +71,7 @@ impl ExtensionManager {
wasm_channels_dir: PathBuf,
tunnel_url: Option<String>,
user_id: String,
store: Option<Arc<dyn crate::db::Database>>,
) -> Self {
Self {
registry: ExtensionRegistry::new(),
@@ -85,6 +86,7 @@ impl ExtensionManager {
pending_auth: RwLock::new(HashMap::new()),
_tunnel_url: tunnel_url,
user_id,
store,
}
}
@@ -191,7 +193,7 @@ impl ExtensionManager {
// List MCP servers
if kind_filter.is_none() || kind_filter == Some(ExtensionKind::McpServer) {
match load_mcp_servers().await {
match self.load_mcp_servers().await {
Ok(servers) => {
for server in &servers.servers {
let authenticated =
@@ -304,7 +306,7 @@ impl ExtensionManager {
self.mcp_clients.write().await.remove(name);
// Remove from config
remove_mcp_server(name)
self.remove_mcp_server(name)
.await
.map_err(|e| ExtensionError::Config(e.to_string()))?;
@@ -342,6 +344,56 @@ impl ExtensionManager {
}
}
// ── MCP config helpers (DB with disk fallback) ─────────────────────
async fn load_mcp_servers(
&self,
) -> Result<crate::tools::mcp::config::McpServersFile, crate::tools::mcp::config::ConfigError>
{
if let Some(ref store) = self.store {
crate::tools::mcp::config::load_mcp_servers_from_db(store.as_ref(), &self.user_id).await
} else {
crate::tools::mcp::config::load_mcp_servers().await
}
}
async fn get_mcp_server(
&self,
name: &str,
) -> Result<McpServerConfig, crate::tools::mcp::config::ConfigError> {
let servers = self.load_mcp_servers().await?;
servers.get(name).cloned().ok_or_else(|| {
crate::tools::mcp::config::ConfigError::ServerNotFound {
name: name.to_string(),
}
})
}
async fn add_mcp_server(
&self,
config: McpServerConfig,
) -> Result<(), crate::tools::mcp::config::ConfigError> {
config.validate()?;
if let Some(ref store) = self.store {
crate::tools::mcp::config::add_mcp_server_db(store.as_ref(), &self.user_id, config)
.await
} else {
crate::tools::mcp::config::add_mcp_server(config).await
}
}
async fn remove_mcp_server(
&self,
name: &str,
) -> Result<(), crate::tools::mcp::config::ConfigError> {
if let Some(ref store) = self.store {
crate::tools::mcp::config::remove_mcp_server_db(store.as_ref(), &self.user_id, name)
.await
} else {
crate::tools::mcp::config::remove_mcp_server(name).await
}
}
// ── Private helpers ──────────────────────────────────────────────────
async fn install_from_entry(
@@ -381,7 +433,7 @@ impl ExtensionManager {
url: &str,
) -> Result<InstallResult, ExtensionError> {
// Check if already installed
if get_mcp_server(name).await.is_ok() {
if self.get_mcp_server(name).await.is_ok() {
return Err(ExtensionError::AlreadyInstalled(name.to_string()));
}
@@ -390,7 +442,7 @@ impl ExtensionManager {
.validate()
.map_err(|e| ExtensionError::InvalidUrl(e.to_string()))?;
add_mcp_server(config)
self.add_mcp_server(config)
.await
.map_err(|e| ExtensionError::Config(e.to_string()))?;
@@ -411,7 +463,16 @@ impl ExtensionManager {
name: &str,
url: &str,
) -> Result<InstallResult, ExtensionError> {
// Download the WASM binary
// Require HTTPS to prevent downgrade attacks
if !url.starts_with("https://") {
return Err(ExtensionError::InstallFailed(
"Only HTTPS URLs are allowed for extension downloads".to_string(),
));
}
// 50 MB cap to prevent disk-fill DoS
const MAX_WASM_SIZE: usize = 50 * 1024 * 1024;
let client = reqwest::Client::builder()
.timeout(std::time::Duration::from_secs(60))
.build()
@@ -430,11 +491,36 @@ impl ExtensionManager {
)));
}
// Check Content-Length header before downloading the full body
if let Some(len) = response.content_length() {
if len as usize > MAX_WASM_SIZE {
return Err(ExtensionError::InstallFailed(format!(
"WASM binary too large ({} bytes, max {} bytes)",
len, MAX_WASM_SIZE
)));
}
}
let bytes = response
.bytes()
.await
.map_err(|e| ExtensionError::DownloadFailed(e.to_string()))?;
if bytes.len() > MAX_WASM_SIZE {
return Err(ExtensionError::InstallFailed(format!(
"WASM binary too large ({} bytes, max {} bytes)",
bytes.len(),
MAX_WASM_SIZE
)));
}
// Basic WASM magic number check (\0asm)
if bytes.len() < 4 || &bytes[..4] != b"\0asm" {
return Err(ExtensionError::InstallFailed(
"Downloaded file is not a valid WASM binary (bad magic number)".to_string(),
));
}
// Ensure tools directory exists
tokio::fs::create_dir_all(&self.wasm_tools_dir)
.await
@@ -447,9 +533,10 @@ impl ExtensionManager {
.map_err(|e| ExtensionError::InstallFailed(e.to_string()))?;
tracing::info!(
"Installed WASM tool '{}' ({} bytes) to {}",
"Installed WASM tool '{}' ({} bytes) from {} to {}",
name,
bytes.len(),
url,
wasm_path.display()
);
@@ -465,7 +552,8 @@ impl ExtensionManager {
name: &str,
token: Option<&str>,
) -> Result<AuthResult, ExtensionError> {
let server = get_mcp_server(name)
let server = self
.get_mcp_server(name)
.await
.map_err(|e| ExtensionError::NotInstalled(e.to_string()))?;
@@ -784,7 +872,8 @@ impl ExtensionManager {
}
}
let server = get_mcp_server(name)
let server = self
.get_mcp_server(name)
.await
.map_err(|e| ExtensionError::NotInstalled(e.to_string()))?;
@@ -893,7 +982,7 @@ impl ExtensionManager {
/// Determine what kind of installed extension this is.
async fn determine_installed_kind(&self, name: &str) -> Result<ExtensionKind, ExtensionError> {
// Check MCP servers first
if get_mcp_server(name).await.is_ok() {
if self.get_mcp_server(name).await.is_ok() {
return Ok(ExtensionKind::McpServer);
}
+8 -1
View File
@@ -5,8 +5,15 @@
//! - Learning from past executions
//! - Analytics and metrics
#[cfg(feature = "postgres")]
mod analytics;
mod store;
#[cfg(feature = "postgres")]
pub use analytics::{JobStats, ToolStats};
pub use store::{LlmCallRecord, Store};
#[cfg(feature = "postgres")]
pub use store::Store;
pub use store::{
ConversationMessage, ConversationSummary, JobEventRecord, LlmCallRecord, SandboxJobRecord,
SandboxJobSummary, SettingRow,
};
+1242 -6
View File
File diff suppressed because it is too large Load Diff
+6
View File
@@ -39,22 +39,28 @@
//! - **Continuous learning** - Improve estimates from historical data
pub mod agent;
pub mod bootstrap;
pub mod channels;
pub mod cli;
pub mod config;
pub mod context;
pub mod db;
pub mod error;
pub mod estimation;
pub mod evaluation;
pub mod extensions;
pub mod history;
pub mod llm;
pub mod orchestrator;
pub mod pairing;
pub mod safety;
pub mod sandbox;
pub mod secrets;
pub mod settings;
pub mod setup;
pub mod tools;
pub mod util;
pub mod worker;
pub mod workspace;
pub use config::Config;
+124
View File
@@ -0,0 +1,124 @@
//! Per-model cost lookup table for multi-provider LLM support.
//!
//! Returns (input_cost_per_token, output_cost_per_token) as Decimal pairs.
//! Ollama and other local models return zero cost.
use rust_decimal::Decimal;
use rust_decimal_macros::dec;
/// Look up known per-token costs for a model by its identifier.
///
/// Returns `Some((input_cost, output_cost))` for known models, `None` otherwise.
pub fn model_cost(model_id: &str) -> Option<(Decimal, Decimal)> {
// Normalize: strip provider prefixes (e.g., "openai/gpt-4o" -> "gpt-4o")
let id = model_id
.rsplit_once('/')
.map(|(_, name)| name)
.unwrap_or(model_id);
match id {
// OpenAI models -- prices per token (USD)
"gpt-4o" | "gpt-4o-2024-11-20" | "gpt-4o-2024-08-06" => {
Some((dec!(0.0000025), dec!(0.00001)))
}
"gpt-4o-mini" | "gpt-4o-mini-2024-07-18" => Some((dec!(0.00000015), dec!(0.0000006))),
"gpt-4-turbo" | "gpt-4-turbo-2024-04-09" => Some((dec!(0.00001), dec!(0.00003))),
"gpt-4" | "gpt-4-0613" => Some((dec!(0.00003), dec!(0.00006))),
"gpt-3.5-turbo" | "gpt-3.5-turbo-0125" => Some((dec!(0.0000005), dec!(0.0000015))),
"o1" | "o1-2024-12-17" => Some((dec!(0.000015), dec!(0.00006))),
"o1-mini" | "o1-mini-2024-09-12" => Some((dec!(0.000003), dec!(0.000012))),
"o3-mini" | "o3-mini-2025-01-31" => Some((dec!(0.0000011), dec!(0.0000044))),
// Anthropic models
"claude-3-5-sonnet-20241022" | "claude-3-5-sonnet-latest" | "claude-sonnet-4-20250514" => {
Some((dec!(0.000003), dec!(0.000015)))
}
"claude-3-5-haiku-20241022" | "claude-3-5-haiku-latest" => {
Some((dec!(0.0000008), dec!(0.000004)))
}
"claude-3-opus-20240229" | "claude-3-opus-latest" | "claude-opus-4-20250514" => {
Some((dec!(0.000015), dec!(0.000075)))
}
"claude-3-haiku-20240307" => Some((dec!(0.00000025), dec!(0.00000125))),
// Ollama / local models -- free
_ if is_local_model(id) => Some((Decimal::ZERO, Decimal::ZERO)),
_ => None,
}
}
/// Default cost for unknown models.
pub fn default_cost() -> (Decimal, Decimal) {
// Conservative estimate: roughly GPT-4o pricing
(dec!(0.0000025), dec!(0.00001))
}
/// Heuristic to detect local/self-hosted models (Ollama, llama.cpp, etc.).
fn is_local_model(model_id: &str) -> bool {
let lower = model_id.to_lowercase();
lower.starts_with("llama")
|| lower.starts_with("mistral")
|| lower.starts_with("mixtral")
|| lower.starts_with("phi")
|| lower.starts_with("gemma")
|| lower.starts_with("qwen")
|| lower.starts_with("codellama")
|| lower.starts_with("deepseek")
|| lower.starts_with("starcoder")
|| lower.starts_with("vicuna")
|| lower.starts_with("yi")
|| lower.contains(":latest")
|| lower.contains(":instruct")
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_known_model_costs() {
let (input, output) = model_cost("gpt-4o").unwrap();
assert!(input > Decimal::ZERO);
assert!(output > input);
}
#[test]
fn test_claude_costs() {
let (input, output) = model_cost("claude-3-5-sonnet-20241022").unwrap();
assert!(input > Decimal::ZERO);
assert!(output > input);
}
#[test]
fn test_local_model_free() {
let (input, output) = model_cost("llama3").unwrap();
assert_eq!(input, Decimal::ZERO);
assert_eq!(output, Decimal::ZERO);
}
#[test]
fn test_ollama_tagged_model_free() {
let (input, output) = model_cost("mistral:latest").unwrap();
assert_eq!(input, Decimal::ZERO);
assert_eq!(output, Decimal::ZERO);
}
#[test]
fn test_unknown_model_returns_none() {
assert!(model_cost("some-totally-unknown-model-xyz").is_none());
}
#[test]
fn test_default_cost_nonzero() {
let (input, output) = default_cost();
assert!(input > Decimal::ZERO);
assert!(output > Decimal::ZERO);
}
#[test]
fn test_provider_prefix_stripped() {
// "openai/gpt-4o" should resolve to same as "gpt-4o"
assert_eq!(model_cost("openai/gpt-4o"), model_cost("gpt-4o"));
}
}
+138 -11
View File
@@ -1,48 +1,175 @@
//! LLM integration for the agent.
//!
//! Supports two API modes:
//! - **Responses API** (chat-api): Session-based auth, uses `/v1/responses` endpoint
//! - **Chat Completions API** (cloud-api): API key auth, uses `/v1/chat/completions` endpoint
//! Supports multiple backends:
//! - **NEAR AI** (default): Session-based or API key auth via NEAR AI proxy
//! - **OpenAI**: Direct API access with your own key
//! - **Anthropic**: Direct API access with your own key
//! - **Ollama**: Local model inference
//! - **OpenAI-compatible**: Any endpoint that speaks the OpenAI API
mod costs;
mod nearai;
mod nearai_chat;
mod provider;
mod reasoning;
mod rig_adapter;
pub mod session;
pub use nearai::{ModelInfo, NearAiProvider};
pub use nearai_chat::NearAiChatProvider;
pub use provider::{
ChatMessage, CompletionRequest, CompletionResponse, LlmProvider, Role, ToolCall,
ToolCompletionRequest, ToolCompletionResponse, ToolDefinition, ToolResult,
ChatMessage, CompletionRequest, CompletionResponse, FinishReason, LlmProvider, ModelMetadata,
Role, ToolCall, ToolCompletionRequest, ToolCompletionResponse, ToolDefinition, ToolResult,
};
pub use reasoning::{ActionPlan, Reasoning, ReasoningContext, RespondResult, ToolSelection};
pub use reasoning::{
ActionPlan, Reasoning, ReasoningContext, RespondOutput, RespondResult, TokenUsage,
ToolSelection,
};
pub use rig_adapter::RigAdapter;
pub use session::{SessionConfig, SessionManager, create_session_manager};
use std::sync::Arc;
use crate::config::{LlmConfig, NearAiApiMode};
use rig::client::CompletionClient;
use secrecy::ExposeSecret;
use crate::config::{LlmBackend, LlmConfig, NearAiApiMode};
use crate::error::LlmError;
/// Create an LLM provider based on configuration.
///
/// - For `Responses` mode: Requires a session manager for authentication
/// - For `ChatCompletions` mode: Uses API key from config (session not needed)
/// - `NearAi` backend: Uses session manager for authentication (Responses API)
/// or API key (Chat Completions API)
/// - Other backends: Use rig-core adapter with provider-specific clients
pub fn create_llm_provider(
config: &LlmConfig,
session: Arc<SessionManager>,
) -> Result<Arc<dyn LlmProvider>, LlmError> {
match config.backend {
LlmBackend::NearAi => create_nearai_provider(config, session),
LlmBackend::OpenAi => create_openai_provider(config),
LlmBackend::Anthropic => create_anthropic_provider(config),
LlmBackend::Ollama => create_ollama_provider(config),
LlmBackend::OpenAiCompatible => create_openai_compatible_provider(config),
}
}
fn create_nearai_provider(
config: &LlmConfig,
session: Arc<SessionManager>,
) -> Result<Arc<dyn LlmProvider>, LlmError> {
match config.nearai.api_mode {
NearAiApiMode::Responses => {
tracing::info!("Using Responses API (chat-api) with session auth");
tracing::info!("Using NEAR AI Responses API (chat-api) with session auth");
Ok(Arc::new(NearAiProvider::new(
config.nearai.clone(),
session,
)))
}
NearAiApiMode::ChatCompletions => {
tracing::info!("Using Chat Completions API (cloud-api) with API key auth");
tracing::info!("Using NEAR AI Chat Completions API (cloud-api) with API key auth");
Ok(Arc::new(NearAiChatProvider::new(config.nearai.clone())?))
}
}
}
fn create_openai_provider(config: &LlmConfig) -> Result<Arc<dyn LlmProvider>, LlmError> {
let oai = config.openai.as_ref().ok_or_else(|| LlmError::AuthFailed {
provider: "openai".to_string(),
})?;
use rig::providers::openai;
let client: openai::Client =
openai::Client::new(oai.api_key.expose_secret()).map_err(|e| LlmError::RequestFailed {
provider: "openai".to_string(),
reason: format!("Failed to create OpenAI client: {}", e),
})?;
let model = client.completion_model(&oai.model);
tracing::info!("Using OpenAI direct API (model: {})", oai.model);
Ok(Arc::new(RigAdapter::new(model, &oai.model)))
}
fn create_anthropic_provider(config: &LlmConfig) -> Result<Arc<dyn LlmProvider>, LlmError> {
let anth = config
.anthropic
.as_ref()
.ok_or_else(|| LlmError::AuthFailed {
provider: "anthropic".to_string(),
})?;
use rig::providers::anthropic;
let client: anthropic::Client =
anthropic::Client::new(anth.api_key.expose_secret()).map_err(|e| {
LlmError::RequestFailed {
provider: "anthropic".to_string(),
reason: format!("Failed to create Anthropic client: {}", e),
}
})?;
let model = client.completion_model(&anth.model);
tracing::info!("Using Anthropic direct API (model: {})", anth.model);
Ok(Arc::new(RigAdapter::new(model, &anth.model)))
}
fn create_ollama_provider(config: &LlmConfig) -> Result<Arc<dyn LlmProvider>, LlmError> {
let oll = config.ollama.as_ref().ok_or_else(|| LlmError::AuthFailed {
provider: "ollama".to_string(),
})?;
use rig::client::Nothing;
use rig::providers::ollama;
let client: ollama::Client = ollama::Client::builder()
.base_url(&oll.base_url)
.api_key(Nothing)
.build()
.map_err(|e| LlmError::RequestFailed {
provider: "ollama".to_string(),
reason: format!("Failed to create Ollama client: {}", e),
})?;
let model = client.completion_model(&oll.model);
tracing::info!(
"Using Ollama (base_url: {}, model: {})",
oll.base_url,
oll.model
);
Ok(Arc::new(RigAdapter::new(model, &oll.model)))
}
fn create_openai_compatible_provider(config: &LlmConfig) -> Result<Arc<dyn LlmProvider>, LlmError> {
let compat = config
.openai_compatible
.as_ref()
.ok_or_else(|| LlmError::AuthFailed {
provider: "openai_compatible".to_string(),
})?;
use rig::providers::openai;
let api_key = compat
.api_key
.as_ref()
.map(|k| k.expose_secret().to_string())
.unwrap_or_else(|| "no-key".to_string());
let client: openai::Client = openai::Client::builder()
.base_url(&compat.base_url)
.api_key(api_key)
.build()
.map_err(|e| LlmError::RequestFailed {
provider: "openai_compatible".to_string(),
reason: format!("Failed to create OpenAI-compatible client: {}", e),
})?;
let model = client.completion_model(&compat.model);
tracing::info!(
"Using OpenAI-compatible endpoint (base_url: {}, model: {})",
compat.base_url,
compat.model
);
Ok(Arc::new(RigAdapter::new(model, &compat.model)))
}
+308 -20
View File
@@ -3,6 +3,7 @@
//! This provider uses the NEAR AI chat-api which provides a unified interface
//! to multiple LLM models (OpenAI, Anthropic, etc.) with user authentication.
use std::collections::HashMap;
use std::sync::Arc;
use async_trait::async_trait;
@@ -31,11 +32,23 @@ pub struct ModelInfo {
pub provider: Option<String>,
}
/// Per-thread chaining state: the last response ID and how many input
/// messages were included in that request. This lets subsequent calls send
/// only the delta (new messages since last call).
struct ChainState {
response_id: String,
input_count: usize,
}
/// NEAR AI Chat API provider.
pub struct NearAiProvider {
client: Client,
config: NearAiConfig,
session: Arc<SessionManager>,
active_model: std::sync::RwLock<String>,
/// Per-thread response ID chaining state.
/// Key is thread_id from request metadata.
response_chains: std::sync::RwLock<HashMap<String, ChainState>>,
}
impl NearAiProvider {
@@ -46,13 +59,64 @@ impl NearAiProvider {
.build()
.unwrap_or_else(|_| Client::new());
let active_model = std::sync::RwLock::new(config.model.clone());
Self {
client,
config,
session,
active_model,
response_chains: std::sync::RwLock::new(HashMap::new()),
}
}
/// Seed a response chain for a thread (e.g. when restoring from DB).
pub fn seed_response_id(&self, thread_id: &str, response_id: String) {
let mut chains = self
.response_chains
.write()
.expect("response_chains lock poisoned");
chains.insert(
thread_id.to_string(),
ChainState {
response_id,
input_count: 0,
},
);
}
/// Get the last response ID for a thread (for persistence).
pub fn get_response_id(&self, thread_id: &str) -> Option<String> {
let chains = self
.response_chains
.read()
.expect("response_chains lock poisoned");
chains.get(thread_id).map(|c| c.response_id.clone())
}
/// Store a response chain state after a successful call.
fn store_chain(&self, thread_id: &str, response_id: String, input_count: usize) {
let mut chains = self
.response_chains
.write()
.expect("response_chains lock poisoned");
chains.insert(
thread_id.to_string(),
ChainState {
response_id,
input_count,
},
);
}
/// Clear the chain for a thread (on error / fallback).
fn clear_chain(&self, thread_id: &str) {
let mut chains = self
.response_chains
.write()
.expect("response_chains lock poisoned");
chains.remove(thread_id);
}
fn api_url(&self, path: &str) -> String {
format!(
"{}/v1/{}",
@@ -291,18 +355,34 @@ impl NearAiProvider {
}
}
/// Split messages into system instructions and non-system input messages.
/// Split messages into system instructions and non-system input items.
/// The OpenAI Responses API expects system prompts in an `instructions` field,
/// not as a message with role "system" in the input array.
fn split_messages(messages: Vec<ChatMessage>) -> (Option<String>, Vec<NearAiMessage>) {
///
/// When `chaining` is true, tool result messages (role=tool) are converted to
/// `NearAiInputItem::FunctionCallOutput` for the Responses API protocol.
fn split_messages(
messages: Vec<ChatMessage>,
chaining: bool,
) -> (Option<String>, Vec<NearAiInputItem>) {
let mut instructions: Vec<String> = Vec::new();
let mut input: Vec<NearAiMessage> = Vec::new();
let mut input: Vec<NearAiInputItem> = Vec::new();
for msg in messages {
if msg.role == Role::System {
instructions.push(msg.content);
} else if chaining && msg.role == Role::Tool {
if let Some(ref call_id) = msg.tool_call_id {
input.push(NearAiInputItem::FunctionCallOutput {
item_type: "function_call_output".to_string(),
call_id: call_id.clone(),
output: msg.content,
});
} else {
input.push(NearAiInputItem::Message(msg.into()));
}
} else {
input.push(msg.into());
input.push(NearAiInputItem::Message(msg.into()));
}
}
@@ -318,12 +398,14 @@ fn split_messages(messages: Vec<ChatMessage>) -> (Option<String>, Vec<NearAiMess
#[async_trait]
impl LlmProvider for NearAiProvider {
async fn complete(&self, req: CompletionRequest) -> Result<CompletionResponse, LlmError> {
let (instructions, input) = split_messages(req.messages);
let thread_id = req.metadata.get("thread_id").cloned();
let (instructions, input) = split_messages(req.messages, false);
let request = NearAiRequest {
model: self.config.model.clone(),
model: self.active_model_name(),
instructions,
input,
previous_response_id: None,
temperature: req.temperature,
max_output_tokens: req.max_tokens,
stream: Some(false),
@@ -350,6 +432,7 @@ impl LlmProvider for NearAiProvider {
finish_reason: FinishReason::Stop,
input_tokens: usage.input_tokens,
output_tokens: usage.output_tokens,
response_id: None,
});
}
@@ -367,6 +450,7 @@ impl LlmProvider for NearAiProvider {
finish_reason: FinishReason::Stop,
input_tokens: 0,
output_tokens: 0,
response_id: None,
});
}
Err(e) => return Err(e),
@@ -423,11 +507,17 @@ impl LlmProvider for NearAiProvider {
);
}
// Store response ID for chaining
if let Some(ref tid) = thread_id {
self.store_chain(tid, response.id.clone(), 0);
}
Ok(CompletionResponse {
content: text,
finish_reason: FinishReason::Stop,
input_tokens: response.usage.input_tokens,
output_tokens: response.usage.output_tokens,
response_id: Some(response.id),
})
}
@@ -435,7 +525,33 @@ impl LlmProvider for NearAiProvider {
&self,
req: ToolCompletionRequest,
) -> Result<ToolCompletionResponse, LlmError> {
let (instructions, input) = split_messages(req.messages);
let thread_id = req.metadata.get("thread_id").cloned();
// Look up chaining state for this thread
let chain_state = thread_id.as_ref().and_then(|tid| {
let chains = self
.response_chains
.read()
.expect("response_chains lock poisoned");
chains
.get(tid)
.map(|c| (c.response_id.clone(), c.input_count))
});
let chaining = chain_state.is_some();
let (previous_response_id, prev_input_count) = chain_state
.map(|(rid, count)| (Some(rid), count))
.unwrap_or((None, 0));
// When chaining, only send new messages (the delta since last call).
// Tool results are converted to function_call_output items.
let (instructions, all_input) = split_messages(req.messages, chaining);
let input = if chaining && all_input.len() > prev_input_count {
all_input[prev_input_count..].to_vec()
} else {
all_input.clone()
};
let total_input_count = all_input.len();
let tools: Vec<NearAiTool> = req
.tools
@@ -449,18 +565,58 @@ impl LlmProvider for NearAiProvider {
.collect();
let request = NearAiRequest {
model: self.config.model.clone(),
instructions,
model: self.active_model_name(),
instructions: if chaining { None } else { instructions.clone() },
input,
previous_response_id: previous_response_id.clone(),
temperature: req.temperature,
max_output_tokens: req.max_tokens,
stream: Some(false),
tools: if tools.is_empty() { None } else { Some(tools) },
tools: if tools.is_empty() {
None
} else {
Some(tools.clone())
},
};
// Try to get structured response, fall back to alternative formats
// Try to get structured response, fall back to alternative formats.
// If chaining fails (bad previous_response_id), retry with full history.
let response: NearAiResponse = match self.send_request("responses", &request).await {
Ok(r) => r,
Err(ref e) if chaining && is_chain_error(e) => {
tracing::warn!(
"Response chaining failed, retrying with full history: {}",
e
);
if let Some(ref tid) = thread_id {
self.clear_chain(tid);
}
let (instructions_full, input_full) = split_messages(
// Rebuild from the original input (non-chaining mode)
{
let mut msgs = Vec::new();
if let Some(ref instr) = instructions {
msgs.push(ChatMessage::system(instr.clone()));
}
for item in &all_input {
msgs.push(item.to_chat_message());
}
msgs
},
false,
);
let retry_request = NearAiRequest {
model: self.active_model_name(),
instructions: instructions_full,
input: input_full,
previous_response_id: None,
temperature: request.temperature,
max_output_tokens: request.max_output_tokens,
stream: Some(false),
tools: request.tools.clone(),
};
self.send_request("responses", &retry_request).await?
}
Err(LlmError::InvalidResponse { reason, .. }) if reason.contains("Raw: ") => {
let raw_text = reason.split("Raw: ").nth(1).unwrap_or("");
@@ -490,6 +646,7 @@ impl LlmProvider for NearAiProvider {
finish_reason,
input_tokens: usage.input_tokens,
output_tokens: usage.output_tokens,
response_id: None,
});
}
@@ -507,6 +664,7 @@ impl LlmProvider for NearAiProvider {
finish_reason: FinishReason::Stop,
input_tokens: 0,
output_tokens: 0,
response_id: None,
});
}
Err(e) => return Err(e),
@@ -560,12 +718,18 @@ impl LlmProvider for NearAiProvider {
FinishReason::ToolUse
};
// Store response ID for chaining on subsequent calls
if let Some(ref tid) = thread_id {
self.store_chain(tid, response.id.clone(), total_input_count);
}
Ok(ToolCompletionResponse {
content: if text.is_empty() { None } else { Some(text) },
tool_calls,
finish_reason,
input_tokens: response.usage.input_tokens,
output_tokens: response.usage.output_tokens,
response_id: Some(response.id),
})
}
@@ -584,6 +748,30 @@ impl LlmProvider for NearAiProvider {
let models = NearAiProvider::list_models(self).await?;
Ok(models.into_iter().map(|m| m.name).collect())
}
fn active_model_name(&self) -> String {
self.active_model
.read()
.expect("active_model lock poisoned")
.clone()
}
fn set_model(&self, model: &str) -> Result<(), LlmError> {
let mut guard = self
.active_model
.write()
.expect("active_model lock poisoned");
*guard = model.to_string();
Ok(())
}
fn seed_response_chain(&self, thread_id: &str, response_id: String) {
self.seed_response_id(thread_id, response_id);
}
fn get_response_chain_id(&self, thread_id: &str) -> Option<String> {
self.get_response_id(thread_id)
}
}
// NEAR AI API types
@@ -597,8 +785,11 @@ struct NearAiRequest {
/// System instructions (replaces sending system role in input)
#[serde(skip_serializing_if = "Option::is_none")]
instructions: Option<String>,
/// Input messages (user/assistant/tool only, NOT system)
input: Vec<NearAiMessage>,
/// Input items: messages and/or function_call_output entries.
input: Vec<NearAiInputItem>,
/// Chain this request to a previous response (avoids resending full context).
#[serde(skip_serializing_if = "Option::is_none")]
previous_response_id: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
temperature: Option<f32>,
#[serde(skip_serializing_if = "Option::is_none")]
@@ -609,7 +800,7 @@ struct NearAiRequest {
tools: Option<Vec<NearAiTool>>,
}
#[derive(Debug, Serialize, Deserialize)]
#[derive(Debug, Serialize, Deserialize, Clone)]
struct NearAiMessage {
role: String,
content: String,
@@ -630,7 +821,68 @@ impl From<ChatMessage> for NearAiMessage {
}
}
#[derive(Debug, Serialize)]
/// Input item for the Responses API. Either a regular message or a
/// function_call_output (for returning tool results when chaining).
#[derive(Debug, Serialize, Deserialize, Clone)]
#[serde(untagged)]
enum NearAiInputItem {
Message(NearAiMessage),
FunctionCallOutput {
#[serde(rename = "type")]
item_type: String,
call_id: String,
output: String,
},
}
impl NearAiInputItem {
/// Convert back to a ChatMessage (used for fallback retry).
fn to_chat_message(&self) -> ChatMessage {
match self {
NearAiInputItem::Message(msg) => {
let role = match msg.role.as_str() {
"system" => Role::System,
"user" => Role::User,
"assistant" => Role::Assistant,
"tool" => Role::Tool,
_ => Role::User,
};
ChatMessage {
role,
content: msg.content.clone(),
tool_call_id: None,
name: None,
tool_calls: None,
}
}
NearAiInputItem::FunctionCallOutput {
call_id, output, ..
} => ChatMessage {
role: Role::Tool,
content: output.clone(),
tool_call_id: Some(call_id.clone()),
name: None,
tool_calls: None,
},
}
}
}
/// Check if an LLM error is likely caused by an invalid previous_response_id.
fn is_chain_error(err: &LlmError) -> bool {
match err {
LlmError::RequestFailed { reason, .. } => {
let lower = reason.to_lowercase();
lower.contains("previous_response_id")
|| lower.contains("previous response")
|| lower.contains("not found")
|| lower.contains("invalid response id")
}
_ => false,
}
}
#[derive(Debug, Clone, Serialize)]
struct NearAiTool {
#[serde(rename = "type")]
tool_type: String,
@@ -833,14 +1085,17 @@ mod tests {
ChatMessage::user("Hello"),
ChatMessage::assistant("Hi there!"),
];
let (instructions, input) = split_messages(messages);
let (instructions, input) = split_messages(messages, false);
assert_eq!(
instructions,
Some("You are a helpful assistant".to_string())
);
assert_eq!(input.len(), 2);
assert_eq!(input[0].role, "user");
assert_eq!(input[1].role, "assistant");
// Verify the input items are messages
match &input[0] {
NearAiInputItem::Message(m) => assert_eq!(m.role, "user"),
_ => panic!("expected Message"),
}
}
#[test]
@@ -849,7 +1104,7 @@ mod tests {
ChatMessage::user("Hello"),
ChatMessage::assistant("Hi there!"),
];
let (instructions, input) = split_messages(messages);
let (instructions, input) = split_messages(messages, false);
assert!(instructions.is_none());
assert_eq!(input.len(), 2);
}
@@ -861,11 +1116,44 @@ mod tests {
ChatMessage::system("Second instruction"),
ChatMessage::user("Hello"),
];
let (instructions, input) = split_messages(messages);
let (instructions, input) = split_messages(messages, false);
assert_eq!(
instructions,
Some("First instruction\n\nSecond instruction".to_string())
);
assert_eq!(input.len(), 1);
}
#[test]
fn test_split_messages_chaining_converts_tool_results() {
let messages = vec![
ChatMessage::user("Hello"),
ChatMessage::tool_result("call_123", "my_tool", "result data"),
];
let (_, input) = split_messages(messages, true);
assert_eq!(input.len(), 2);
match &input[1] {
NearAiInputItem::FunctionCallOutput {
call_id, output, ..
} => {
assert_eq!(call_id, "call_123");
assert_eq!(output, "result data");
}
_ => panic!("expected FunctionCallOutput"),
}
}
#[test]
fn test_split_messages_no_chaining_keeps_tool_as_message() {
let messages = vec![
ChatMessage::user("Hello"),
ChatMessage::tool_result("call_123", "my_tool", "result data"),
];
let (_, input) = split_messages(messages, false);
assert_eq!(input.len(), 2);
match &input[1] {
NearAiInputItem::Message(m) => assert_eq!(m.role, "tool"),
_ => panic!("expected Message"),
}
}
}
+248 -18
View File
@@ -13,14 +13,15 @@ use serde::{Deserialize, Serialize};
use crate::config::NearAiConfig;
use crate::error::LlmError;
use crate::llm::provider::{
ChatMessage, CompletionRequest, CompletionResponse, FinishReason, LlmProvider, Role, ToolCall,
ToolCompletionRequest, ToolCompletionResponse,
ChatMessage, CompletionRequest, CompletionResponse, FinishReason, LlmProvider, ModelMetadata,
Role, ToolCall, ToolCompletionRequest, ToolCompletionResponse,
};
/// NEAR AI Chat Completions API provider.
pub struct NearAiChatProvider {
client: Client,
config: NearAiConfig,
active_model: std::sync::RwLock<String>,
}
impl NearAiChatProvider {
@@ -37,7 +38,12 @@ impl NearAiChatProvider {
.build()
.unwrap_or_else(|_| Client::new());
Ok(Self { client, config })
let active_model = std::sync::RwLock::new(config.model.clone());
Ok(Self {
client,
config,
active_model,
})
}
fn api_url(&self, path: &str) -> String {
@@ -65,6 +71,11 @@ impl NearAiChatProvider {
tracing::debug!("Sending request to NEAR AI Chat: {}", url);
// Log the request body for debugging tool call issues
if let Ok(json) = serde_json::to_string(body) {
tracing::debug!("NEAR AI Chat request body: {}", json);
}
let response = self
.client
.post(&url)
@@ -111,8 +122,8 @@ impl NearAiChatProvider {
})
}
/// Fetch available models.
pub async fn list_models(&self) -> Result<Vec<String>, LlmError> {
/// Fetch available models with full metadata from the `/v1/models` endpoint.
async fn fetch_models(&self) -> Result<Vec<ApiModelEntry>, LlmError> {
let url = self.api_url("models");
let response = self
@@ -138,12 +149,7 @@ impl NearAiChatProvider {
#[derive(Deserialize)]
struct ModelsResponse {
data: Vec<ModelEntry>,
}
#[derive(Deserialize)]
struct ModelEntry {
id: String,
data: Vec<ApiModelEntry>,
}
let resp: ModelsResponse =
@@ -152,10 +158,18 @@ impl NearAiChatProvider {
reason: format!("JSON parse error: {}", e),
})?;
Ok(resp.data.into_iter().map(|m| m.id).collect())
Ok(resp.data)
}
}
/// Model entry as returned by the `/v1/models` API.
#[derive(Debug, Deserialize)]
struct ApiModelEntry {
id: String,
#[serde(default)]
context_length: Option<u32>,
}
#[async_trait]
impl LlmProvider for NearAiChatProvider {
async fn complete(&self, req: CompletionRequest) -> Result<CompletionResponse, LlmError> {
@@ -163,7 +177,7 @@ impl LlmProvider for NearAiChatProvider {
req.messages.into_iter().map(|m| m.into()).collect();
let request = ChatCompletionRequest {
model: self.config.model.clone(),
model: self.active_model_name(),
messages,
temperature: req.temperature,
max_tokens: req.max_tokens,
@@ -197,6 +211,7 @@ impl LlmProvider for NearAiChatProvider {
finish_reason,
input_tokens: response.usage.prompt_tokens,
output_tokens: response.usage.completion_tokens,
response_id: None,
})
}
@@ -207,6 +222,12 @@ impl LlmProvider for NearAiChatProvider {
let messages: Vec<ChatCompletionMessage> =
req.messages.into_iter().map(|m| m.into()).collect();
// NEAR AI cloud-api does not support multi-turn tool calling (rejects
// any request containing role:"tool" messages with HTTP 400). Rewrite
// tool-call / tool-result pairs into plain text so the conversation
// history is preserved without using unsupported message roles.
let messages = flatten_tool_messages(messages);
let tools: Vec<ChatCompletionTool> = req
.tools
.into_iter()
@@ -221,7 +242,7 @@ impl LlmProvider for NearAiChatProvider {
.collect();
let request = ChatCompletionRequest {
model: self.config.model.clone(),
model: self.active_model_name(),
messages,
temperature: req.temperature,
max_tokens: req.max_tokens,
@@ -278,6 +299,7 @@ impl LlmProvider for NearAiChatProvider {
finish_reason,
input_tokens: response.usage.prompt_tokens,
output_tokens: response.usage.completion_tokens,
response_id: None,
})
}
@@ -291,7 +313,34 @@ impl LlmProvider for NearAiChatProvider {
}
async fn list_models(&self) -> Result<Vec<String>, LlmError> {
NearAiChatProvider::list_models(self).await
let models = self.fetch_models().await?;
Ok(models.into_iter().map(|m| m.id).collect())
}
async fn model_metadata(&self) -> Result<ModelMetadata, LlmError> {
let active = self.active_model_name();
let models = self.fetch_models().await?;
let current = models.iter().find(|m| m.id == active);
Ok(ModelMetadata {
id: active,
context_length: current.and_then(|m| m.context_length),
})
}
fn active_model_name(&self) -> String {
self.active_model
.read()
.expect("active_model lock poisoned")
.clone()
}
fn set_model(&self, model: &str) -> Result<(), crate::error::LlmError> {
let mut guard = self
.active_model
.write()
.expect("active_model lock poisoned");
*guard = model.to_string();
Ok(())
}
}
@@ -324,6 +373,64 @@ struct ChatCompletionMessage {
tool_calls: Option<Vec<ChatCompletionToolCall>>,
}
/// Rewrite tool-call / tool-result messages into plain assistant/user text.
///
/// NEAR AI cloud-api does not support the OpenAI multi-turn tool-calling
/// protocol (`role: "tool"` messages). This function converts:
/// - Assistant messages with `tool_calls` → assistant text describing the calls
/// - Tool result messages (`role: "tool"`) → user messages with the result
///
/// Non-tool messages pass through unchanged.
fn flatten_tool_messages(messages: Vec<ChatCompletionMessage>) -> Vec<ChatCompletionMessage> {
let has_tool_msgs = messages.iter().any(|m| m.role == "tool");
if !has_tool_msgs {
return messages;
}
tracing::debug!("Flattening tool messages for NEAR AI compatibility");
messages
.into_iter()
.map(|msg| {
if let (true, Some(calls)) = (msg.role == "assistant", &msg.tool_calls) {
// Convert assistant tool_calls into descriptive text
let mut parts: Vec<String> = Vec::new();
if let Some(ref text) = msg.content {
if !text.is_empty() {
parts.push(text.clone());
}
}
for tc in calls {
parts.push(format!(
"[Called tool `{}` with arguments: {}]",
tc.function.name, tc.function.arguments
));
}
ChatCompletionMessage {
role: "assistant".to_string(),
content: Some(parts.join("\n")),
tool_call_id: None,
name: None,
tool_calls: None,
}
} else if msg.role == "tool" {
// Convert tool result into a user message
let tool_name = msg.name.as_deref().unwrap_or("unknown");
let result = msg.content.as_deref().unwrap_or("");
ChatCompletionMessage {
role: "user".to_string(),
content: Some(format!("[Tool `{}` returned: {}]", tool_name, result)),
tool_call_id: None,
name: None,
tool_calls: None,
}
} else {
msg
}
})
.collect()
}
impl From<ChatMessage> for ChatCompletionMessage {
fn from(msg: ChatMessage) -> Self {
let role = match msg.role {
@@ -332,6 +439,7 @@ impl From<ChatMessage> for ChatCompletionMessage {
Role::Assistant => "assistant",
Role::Tool => "tool",
};
let tool_calls = msg.tool_calls.map(|calls| {
calls
.into_iter()
@@ -345,9 +453,16 @@ impl From<ChatMessage> for ChatCompletionMessage {
})
.collect()
});
let content = if role == "assistant" && tool_calls.is_some() && msg.content.is_empty() {
None
} else {
Some(msg.content)
};
Self {
role: role.to_string(),
content: Some(msg.content),
content,
tool_call_id: msg.tool_call_id,
name: msg.name,
tool_calls,
@@ -454,7 +569,7 @@ mod tests {
},
];
let msg = ChatMessage::assistant_with_tool_calls("", tool_calls);
let msg = ChatMessage::assistant_with_tool_calls(None, tool_calls);
let chat_msg: ChatCompletionMessage = msg.into();
assert_eq!(chat_msg.role, "assistant");
@@ -484,7 +599,7 @@ mod tests {
name: "test".to_string(),
arguments: serde_json::json!({"key": "value"}),
};
let msg = ChatMessage::assistant_with_tool_calls("", vec![tc]);
let msg = ChatMessage::assistant_with_tool_calls(None, vec![tc]);
let chat_msg: ChatCompletionMessage = msg.into();
let calls = chat_msg.tool_calls.unwrap();
@@ -493,4 +608,119 @@ mod tests {
serde_json::from_str(&calls[0].function.arguments).expect("valid JSON string");
assert_eq!(parsed["key"], "value");
}
#[test]
fn test_flatten_no_tool_messages_passthrough() {
let messages = vec![
ChatCompletionMessage {
role: "system".to_string(),
content: Some("You are helpful.".to_string()),
tool_call_id: None,
name: None,
tool_calls: None,
},
ChatCompletionMessage {
role: "user".to_string(),
content: Some("Hello".to_string()),
tool_call_id: None,
name: None,
tool_calls: None,
},
];
let result = flatten_tool_messages(messages);
assert_eq!(result.len(), 2);
assert_eq!(result[0].role, "system");
assert_eq!(result[1].role, "user");
}
#[test]
fn test_flatten_tool_call_and_result() {
let messages = vec![
ChatCompletionMessage {
role: "user".to_string(),
content: Some("test".to_string()),
tool_call_id: None,
name: None,
tool_calls: None,
},
ChatCompletionMessage {
role: "assistant".to_string(),
content: None,
tool_call_id: None,
name: None,
tool_calls: Some(vec![ChatCompletionToolCall {
id: "call_1".to_string(),
call_type: "function".to_string(),
function: ChatCompletionToolCallFunction {
name: "echo".to_string(),
arguments: r#"{"message":"hi"}"#.to_string(),
},
}]),
},
ChatCompletionMessage {
role: "tool".to_string(),
content: Some("hi".to_string()),
tool_call_id: Some("call_1".to_string()),
name: Some("echo".to_string()),
tool_calls: None,
},
];
let result = flatten_tool_messages(messages);
assert_eq!(result.len(), 3);
// Assistant tool_calls → plain assistant text
assert_eq!(result[1].role, "assistant");
assert!(result[1].tool_calls.is_none());
assert!(
result[1]
.content
.as_ref()
.unwrap()
.contains("[Called tool `echo`")
);
// Tool result → user message
assert_eq!(result[2].role, "user");
assert!(result[2].tool_call_id.is_none());
assert!(
result[2]
.content
.as_ref()
.unwrap()
.contains("[Tool `echo` returned: hi]")
);
}
#[test]
fn test_flatten_preserves_assistant_text_with_tool_calls() {
let messages = vec![
ChatCompletionMessage {
role: "assistant".to_string(),
content: Some("Let me check that.".to_string()),
tool_call_id: None,
name: None,
tool_calls: Some(vec![ChatCompletionToolCall {
id: "call_1".to_string(),
call_type: "function".to_string(),
function: ChatCompletionToolCallFunction {
name: "search".to_string(),
arguments: r#"{"q":"test"}"#.to_string(),
},
}]),
},
ChatCompletionMessage {
role: "tool".to_string(),
content: Some("found it".to_string()),
tool_call_id: Some("call_1".to_string()),
name: Some("search".to_string()),
tool_calls: None,
},
];
let result = flatten_tool_messages(messages);
let text = result[0].content.as_ref().unwrap();
assert!(text.starts_with("Let me check that."));
assert!(text.contains("[Called tool `search`"));
}
}
+64 -11
View File
@@ -27,9 +27,8 @@ pub struct ChatMessage {
/// Name of the tool for tool results.
#[serde(skip_serializing_if = "Option::is_none")]
pub name: Option<String>,
/// Tool calls requested by the assistant (for conversation replay).
/// OpenAI-compatible APIs require the assistant message to include
/// tool_calls when followed by tool result messages.
/// Tool calls made by the assistant (OpenAI protocol requires these
/// to appear on the assistant message preceding tool result messages).
#[serde(skip_serializing_if = "Option::is_none")]
pub tool_calls: Option<Vec<ToolCall>>,
}
@@ -68,17 +67,14 @@ impl ChatMessage {
}
}
/// Create an assistant message that requested tool calls.
/// Create an assistant message that includes tool calls.
///
/// OpenAI-compatible APIs require the assistant message to carry the
/// `tool_calls` array when followed by tool-result messages.
pub fn assistant_with_tool_calls(
content: impl Into<String>,
tool_calls: Vec<ToolCall>,
) -> Self {
/// Per the OpenAI protocol, an assistant message with tool_calls must
/// precede the corresponding tool result messages in the conversation.
pub fn assistant_with_tool_calls(content: Option<String>, tool_calls: Vec<ToolCall>) -> Self {
Self {
role: Role::Assistant,
content: content.into(),
content: content.unwrap_or_default(),
tool_call_id: None,
name: None,
tool_calls: if tool_calls.is_empty() {
@@ -112,6 +108,8 @@ pub struct CompletionRequest {
pub max_tokens: Option<u32>,
pub temperature: Option<f32>,
pub stop_sequences: Option<Vec<String>>,
/// Opaque metadata passed through to the provider (e.g. thread_id for chaining).
pub metadata: std::collections::HashMap<String, String>,
}
impl CompletionRequest {
@@ -122,6 +120,7 @@ impl CompletionRequest {
max_tokens: None,
temperature: None,
stop_sequences: None,
metadata: std::collections::HashMap::new(),
}
}
@@ -145,6 +144,8 @@ pub struct CompletionResponse {
pub input_tokens: u32,
pub output_tokens: u32,
pub finish_reason: FinishReason,
/// Provider-specific response ID (e.g. for NEAR AI response chaining).
pub response_id: Option<String>,
}
/// Why the completion finished.
@@ -191,6 +192,8 @@ pub struct ToolCompletionRequest {
pub temperature: Option<f32>,
/// How to handle tool use: "auto", "required", or "none".
pub tool_choice: Option<String>,
/// Opaque metadata passed through to the provider (e.g. thread_id for chaining).
pub metadata: std::collections::HashMap<String, String>,
}
impl ToolCompletionRequest {
@@ -202,6 +205,7 @@ impl ToolCompletionRequest {
max_tokens: None,
temperature: None,
tool_choice: None,
metadata: std::collections::HashMap::new(),
}
}
@@ -234,6 +238,16 @@ pub struct ToolCompletionResponse {
pub input_tokens: u32,
pub output_tokens: u32,
pub finish_reason: FinishReason,
/// Provider-specific response ID (e.g. for NEAR AI response chaining).
pub response_id: Option<String>,
}
/// Metadata about a model returned by the provider's API.
#[derive(Debug, Clone)]
pub struct ModelMetadata {
pub id: String,
/// Total context window size in tokens.
pub context_length: Option<u32>,
}
/// Trait for LLM providers.
@@ -260,6 +274,45 @@ pub trait LlmProvider: Send + Sync {
Ok(Vec::new())
}
/// Fetch metadata for the current model (context length, etc.).
/// Default returns the model name with no size info.
async fn model_metadata(&self) -> Result<ModelMetadata, LlmError> {
Ok(ModelMetadata {
id: self.model_name().to_string(),
context_length: None,
})
}
/// Get the currently active model name.
///
/// May differ from `model_name()` if the model was switched at runtime
/// via `set_model()`. Default returns `model_name()`.
fn active_model_name(&self) -> String {
self.model_name().to_string()
}
/// Switch the active model at runtime. Not all providers support this.
fn set_model(&self, _model: &str) -> Result<(), LlmError> {
Err(LlmError::RequestFailed {
provider: "unknown".to_string(),
reason: "Runtime model switching not supported by this provider".to_string(),
})
}
/// Seed a response chain for a thread (e.g. restoring from DB).
///
/// Providers that support response chaining (e.g. NEAR AI `previous_response_id`)
/// store this so subsequent calls send only delta messages.
fn seed_response_chain(&self, _thread_id: &str, _response_id: String) {}
/// Get the last response chain ID for a thread.
///
/// Returns `None` if the provider doesn't support chaining or has no
/// stored state for this thread.
fn get_response_chain_id(&self, _thread_id: &str) -> Option<String> {
None
}
/// Calculate cost for a completion.
fn calculate_cost(&self, input_tokens: u32, output_tokens: u32) -> Decimal {
let (input_cost, output_cost) = self.cost_per_token();
+400 -37
View File
@@ -21,6 +21,8 @@ pub struct ReasoningContext {
pub job_description: Option<String>,
/// Current state description.
pub current_state: Option<String>,
/// Opaque metadata forwarded to the LLM provider (e.g. thread_id for chaining).
pub metadata: std::collections::HashMap<String, String>,
}
impl ReasoningContext {
@@ -31,6 +33,7 @@ impl ReasoningContext {
available_tools: Vec::new(),
job_description: None,
current_state: None,
metadata: std::collections::HashMap::new(),
}
}
@@ -57,6 +60,12 @@ impl ReasoningContext {
self.job_description = Some(description.into());
self
}
/// Set metadata (forwarded to the LLM provider).
pub fn with_metadata(mut self, metadata: std::collections::HashMap<String, String>) -> Self {
self.metadata = metadata;
self
}
}
impl Default for ReasoningContext {
@@ -106,6 +115,19 @@ pub struct ToolSelection {
pub alternatives: Vec<String>,
}
/// Token usage from a single LLM call.
#[derive(Debug, Clone, Copy, Default)]
pub struct TokenUsage {
pub input_tokens: u32,
pub output_tokens: u32,
}
impl TokenUsage {
pub fn total(&self) -> u32 {
self.input_tokens + self.output_tokens
}
}
/// Result of a response with potential tool calls.
///
/// Used by the agent loop to handle tool execution before returning a final response.
@@ -114,7 +136,19 @@ pub enum RespondResult {
/// A text response (no tools needed).
Text(String),
/// The model wants to call tools. Caller should execute them and call back.
ToolCalls(Vec<ToolCall>),
/// Includes the optional content from the assistant message (some models
/// include explanatory text alongside tool calls).
ToolCalls {
tool_calls: Vec<ToolCall>,
content: Option<String>,
},
}
/// A `RespondResult` bundled with the token usage from the LLM call that produced it.
#[derive(Debug, Clone)]
pub struct RespondOutput {
pub result: RespondResult,
pub usage: TokenUsage,
}
/// Reasoning engine for the agent.
@@ -192,10 +226,11 @@ impl Reasoning {
return Ok(vec![]);
}
let request =
let mut request =
ToolCompletionRequest::new(context.messages.clone(), context.available_tools.clone())
.with_max_tokens(1024)
.with_tool_choice("auto");
request.metadata = context.metadata.clone();
let response = self.llm.complete_with_tools(request).await?;
@@ -269,9 +304,12 @@ Respond in JSON format:
/// tool calls as text for simple cases. Use `respond_with_tools()` when you
/// need to actually execute tool calls in an agentic loop.
pub async fn respond(&self, context: &ReasoningContext) -> Result<String, LlmError> {
match self.respond_with_tools(context).await? {
let output = self.respond_with_tools(context).await?;
match output.result {
RespondResult::Text(text) => Ok(text),
RespondResult::ToolCalls(calls) => {
RespondResult::ToolCalls {
tool_calls: calls, ..
} => {
// Format tool calls as text (legacy behavior for non-agentic callers)
let tool_info: Vec<String> = calls
.iter()
@@ -282,15 +320,14 @@ Respond in JSON format:
}
}
/// Generate a response that may include tool calls.
/// Generate a response that may include tool calls, with token usage tracking.
///
/// Returns `RespondResult::ToolCalls` if the model wants to call tools,
/// allowing the caller to execute them and continue the conversation.
/// Returns `RespondResult::Text` when the model has a final text response.
/// Returns `RespondOutput` containing the result and token usage from the LLM call.
/// The caller should use `usage` to track cost/budget against the job.
pub async fn respond_with_tools(
&self,
context: &ReasoningContext,
) -> Result<RespondResult, LlmError> {
) -> Result<RespondOutput, LlmError> {
let system_prompt = self.build_conversation_prompt(context);
let mut messages = vec![ChatMessage::system(system_prompt)];
@@ -298,31 +335,71 @@ Respond in JSON format:
// If we have tools, use tool completion mode
if !context.available_tools.is_empty() {
let request = ToolCompletionRequest::new(messages, context.available_tools.clone())
let mut request = ToolCompletionRequest::new(messages, context.available_tools.clone())
.with_max_tokens(4096)
.with_temperature(0.7)
.with_tool_choice("auto");
request.metadata = context.metadata.clone();
let response = self.llm.complete_with_tools(request).await?;
let usage = TokenUsage {
input_tokens: response.input_tokens,
output_tokens: response.output_tokens,
};
// If there were tool calls, return them for execution
if !response.tool_calls.is_empty() {
return Ok(RespondResult::ToolCalls(response.tool_calls));
return Ok(RespondOutput {
result: RespondResult::ToolCalls {
tool_calls: response.tool_calls,
content: response.content,
},
usage,
});
}
let content = response
.content
.unwrap_or_else(|| "I'm not sure how to respond to that.".to_string());
Ok(RespondResult::Text(clean_response(&content)))
// Some models (e.g. GLM-4.7) emit tool calls as XML tags in content
// instead of using the structured tool_calls field. Try to recover
// them before giving up and returning plain text.
let recovered = recover_tool_calls_from_content(&content, &context.available_tools);
if !recovered.is_empty() {
let cleaned = clean_response(&content);
return Ok(RespondOutput {
result: RespondResult::ToolCalls {
tool_calls: recovered,
content: if cleaned.is_empty() {
None
} else {
Some(cleaned)
},
},
usage,
});
}
Ok(RespondOutput {
result: RespondResult::Text(clean_response(&content)),
usage,
})
} else {
// No tools, use simple completion
let request = CompletionRequest::new(messages)
let mut request = CompletionRequest::new(messages)
.with_max_tokens(4096)
.with_temperature(0.7);
request.metadata = context.metadata.clone();
let response = self.llm.complete(request).await?;
Ok(RespondResult::Text(clean_response(&response.content)))
Ok(RespondOutput {
result: RespondResult::Text(clean_response(&response.content)),
usage: TokenUsage {
input_tokens: response.input_tokens,
output_tokens: response.output_tokens,
},
})
}
}
@@ -462,47 +539,178 @@ fn extract_json(text: &str) -> Option<&str> {
}
}
/// Clean up LLM response by stripping thinking tags and reasoning patterns.
/// Clean up LLM response by stripping model-internal tags and reasoning patterns.
///
/// Some models (GLM-4.7, etc.) emit XML-tagged internal state like
/// Try to extract tool calls from content text where the model emitted them
/// as XML tags instead of using the structured tool_calls field.
///
/// Handles these formats:
/// - `<tool_call>tool_name</tool_call>` (bare name)
/// - `<tool_call>{"name":"x","arguments":{}}</tool_call>` (JSON)
/// - `<|tool_call|>...<|/tool_call|>` (pipe-delimited variant)
/// - `<function_call>...</function_call>` (function_call variant)
///
/// Only returns calls whose name matches an available tool.
fn recover_tool_calls_from_content(
content: &str,
available_tools: &[ToolDefinition],
) -> Vec<ToolCall> {
let tool_names: std::collections::HashSet<&str> =
available_tools.iter().map(|t| t.name.as_str()).collect();
let mut calls = Vec::new();
for (open, close) in &[
("<tool_call>", "</tool_call>"),
("<|tool_call|>", "<|/tool_call|>"),
("<function_call>", "</function_call>"),
("<|function_call|>", "<|/function_call|>"),
] {
let mut remaining = content;
while let Some(start) = remaining.find(open) {
let inner_start = start + open.len();
let after = &remaining[inner_start..];
let Some(end) = after.find(close) else {
break;
};
let inner = after[..end].trim();
remaining = &after[end + close.len()..];
if inner.is_empty() {
continue;
}
// Try JSON first: {"name":"x","arguments":{}}
if let Ok(parsed) = serde_json::from_str::<serde_json::Value>(inner) {
if let Some(name) = parsed.get("name").and_then(|v| v.as_str()) {
if tool_names.contains(name) {
let arguments = parsed
.get("arguments")
.cloned()
.unwrap_or(serde_json::Value::Object(Default::default()));
calls.push(ToolCall {
id: format!("recovered_{}", calls.len()),
name: name.to_string(),
arguments,
});
continue;
}
}
}
// Bare tool name (e.g. "<tool_call>tool_list</tool_call>")
let name = inner.trim();
if tool_names.contains(name) {
calls.push(ToolCall {
id: format!("recovered_{}", calls.len()),
name: name.to_string(),
arguments: serde_json::Value::Object(Default::default()),
});
}
}
}
calls
}
/// `<tool_call>tool_list</tool_call>` or `<|tool_call|>` in the content field
/// instead of using the standard OpenAI tool_calls array. We strip all of
/// these before the response reaches channels/users.
fn clean_response(text: &str) -> String {
let text = strip_thinking_tags(text);
let text = strip_internal_tags(text);
strip_reasoning_patterns(&text)
}
/// Strip `<thinking>...</thinking>` blocks from LLM output.
/// Tags that are model-internal and should never reach users.
const INTERNAL_TAGS: &[&str] = &["thinking", "tool_call", "function_call", "tool_calls"];
/// Strip all model-internal XML tags from LLM output.
///
/// Some models (especially Claude with extended thinking) include internal
/// reasoning in thinking tags. We strip these before showing to users.
fn strip_thinking_tags(text: &str) -> String {
/// Handles standard XML tags (`<tag>...</tag>`) and pipe-delimited variants
/// (`<|tag|>...<|/tag|>`) used by some models (e.g. GLM-4.7).
fn strip_internal_tags(text: &str) -> String {
let mut result = text.to_string();
for tag in INTERNAL_TAGS {
result = strip_xml_tag(&result, tag);
result = strip_pipe_tag(&result, tag);
}
// Collapse triple+ newlines left behind by removed blocks
while result.contains("\n\n\n") {
result = result.replace("\n\n\n", "\n\n");
}
result.trim().to_string()
}
/// Strip `<tag>...</tag>` and `<tag ...>...</tag>` blocks from text.
fn strip_xml_tag(text: &str, tag: &str) -> String {
let open_exact = format!("<{}>", tag);
let open_prefix = format!("<{} ", tag); // for <tag attr="...">
let close = format!("</{}>", tag);
let mut result = String::with_capacity(text.len());
let mut remaining = text;
while let Some(start) = remaining.find("<thinking>") {
loop {
// Find the next opening tag (exact or with attributes)
let exact_pos = remaining.find(&open_exact);
let prefix_pos = remaining.find(&open_prefix);
let start = match (exact_pos, prefix_pos) {
(Some(a), Some(b)) => a.min(b),
(Some(a), None) => a,
(None, Some(b)) => b,
(None, None) => break,
};
// Add everything before the tag
result.push_str(&remaining[..start]);
// Find the end of the opening tag (the closing >)
let after_open = &remaining[start..];
let open_end = match after_open.find('>') {
Some(pos) => start + pos + 1,
None => break, // malformed, stop
};
// Find the closing tag
if let Some(end_offset) = remaining[start..].find("</thinking>") {
// Skip past the closing tag (start + offset + tag length)
let end = start + end_offset + "</thinking>".len();
if let Some(close_offset) = remaining[open_end..].find(&close) {
let end = open_end + close_offset + close.len();
remaining = &remaining[end..];
} else {
// No closing tag found, discard everything from here
// (malformed, but handle gracefully by not including the unclosed tag)
// No closing tag, discard from here (malformed)
remaining = "";
break;
}
}
// Add any remaining content after the last thinking block
result.push_str(remaining);
result
}
// Clean up any double newlines left behind
let mut cleaned = result.trim().to_string();
while cleaned.contains("\n\n\n") {
cleaned = cleaned.replace("\n\n\n", "\n\n");
/// Strip `<|tag|>...<|/tag|>` pipe-delimited blocks from text.
///
/// Some models (e.g. certain Chinese LLMs) use this format instead of
/// standard XML tags.
fn strip_pipe_tag(text: &str, tag: &str) -> String {
let open = format!("<|{}|>", tag);
let close = format!("<|/{}|>", tag);
let mut result = String::with_capacity(text.len());
let mut remaining = text;
while let Some(start) = remaining.find(&open) {
result.push_str(&remaining[..start]);
if let Some(close_offset) = remaining[start..].find(&close) {
let end = start + close_offset + close.len();
remaining = &remaining[end..];
} else {
remaining = "";
break;
}
}
cleaned
result.push_str(remaining);
result
}
/// Strip any remaining reasoning that wasn't in proper <thinking> tags.
@@ -574,7 +782,7 @@ That's my plan."#;
#[test]
fn test_strip_thinking_tags_basic() {
let input = "<thinking>Let me think about this...</thinking>Hello, user!";
let output = strip_thinking_tags(input);
let output = strip_internal_tags(input);
assert_eq!(output, "Hello, user!");
}
@@ -582,7 +790,7 @@ That's my plan."#;
fn test_strip_thinking_tags_multiple() {
let input =
"<thinking>First thought</thinking>Hello<thinking>Second thought</thinking> world!";
let output = strip_thinking_tags(input);
let output = strip_internal_tags(input);
assert_eq!(output, "Hello world!");
}
@@ -594,14 +802,14 @@ I need to consider:
2. How to respond
</thinking>
Here is my response to your question."#;
let output = strip_thinking_tags(input);
let output = strip_internal_tags(input);
assert_eq!(output, "Here is my response to your question.");
}
#[test]
fn test_strip_thinking_tags_no_tags() {
let input = "Just a normal response without thinking tags.";
let output = strip_thinking_tags(input);
let output = strip_internal_tags(input);
assert_eq!(output, "Just a normal response without thinking tags.");
}
@@ -609,10 +817,77 @@ Here is my response to your question."#;
fn test_strip_thinking_tags_unclosed() {
// Malformed: unclosed tag should strip from there to end
let input = "Hello <thinking>this never closes";
let output = strip_thinking_tags(input);
let output = strip_internal_tags(input);
assert_eq!(output, "Hello");
}
#[test]
fn test_strip_tool_call_tags() {
// GLM-4.7 emits this garbage instead of using the tool_calls array
let input = "<tool_call>tool_list</tool_call>";
let output = strip_internal_tags(input);
assert_eq!(output, "");
}
#[test]
fn test_strip_tool_call_with_surrounding_text() {
let input = "Here is my answer.\n\n<tool_call>\n{\"name\": \"search\", \"arguments\": {}}\n</tool_call>";
let output = strip_internal_tags(input);
assert_eq!(output, "Here is my answer.");
}
#[test]
fn test_strip_multiple_internal_tags() {
let input = "<thinking>Let me think</thinking>Hello!\n<tool_call>some_tool</tool_call>";
let output = strip_internal_tags(input);
assert_eq!(output, "Hello!");
}
#[test]
fn test_strip_function_call_tags() {
let input = "Response text<function_call>{\"name\": \"foo\"}</function_call>";
let output = strip_internal_tags(input);
assert_eq!(output, "Response text");
}
#[test]
fn test_strip_tool_calls_plural() {
let input = "<tool_calls>[{\"id\": \"1\"}]</tool_calls>Actual response.";
let output = strip_internal_tags(input);
assert_eq!(output, "Actual response.");
}
#[test]
fn test_strip_pipe_delimited_tags() {
let input = "<|tool_call|>{\"name\": \"search\"}<|/tool_call|>Hello!";
let output = strip_internal_tags(input);
assert_eq!(output, "Hello!");
}
#[test]
fn test_strip_pipe_delimited_thinking() {
let input = "<|thinking|>reasoning here<|/thinking|>The answer is 42.";
let output = strip_internal_tags(input);
assert_eq!(output, "The answer is 42.");
}
#[test]
fn test_strip_xml_tag_with_attributes() {
let input = "<tool_call type=\"function\">search()</tool_call>Done.";
let output = strip_internal_tags(input);
assert_eq!(output, "Done.");
}
#[test]
fn test_clean_response_preserves_normal_content() {
let input = "The function tool_call_handler works great. No tags here!";
let output = clean_response(input);
assert_eq!(
output,
"The function tool_call_handler works great. No tags here!"
);
}
#[test]
fn test_strip_reasoning_paragraph_break() {
// Content after paragraph break with "here" marker
@@ -660,4 +935,92 @@ Here is my response to your question."#;
let output = clean_response(input);
assert_eq!(output, "Here's the answer.");
}
// -- recover_tool_calls_from_content tests --
fn make_tools(names: &[&str]) -> Vec<ToolDefinition> {
names
.iter()
.map(|n| ToolDefinition {
name: n.to_string(),
description: String::new(),
parameters: serde_json::json!({}),
})
.collect()
}
#[test]
fn test_recover_bare_tool_name() {
let tools = make_tools(&["tool_list", "tool_auth"]);
let content = "<tool_call>tool_list</tool_call>";
let calls = recover_tool_calls_from_content(content, &tools);
assert_eq!(calls.len(), 1);
assert_eq!(calls[0].name, "tool_list");
assert_eq!(calls[0].arguments, serde_json::json!({}));
}
#[test]
fn test_recover_json_tool_call() {
let tools = make_tools(&["memory_search"]);
let content =
r#"<tool_call>{"name": "memory_search", "arguments": {"query": "test"}}</tool_call>"#;
let calls = recover_tool_calls_from_content(content, &tools);
assert_eq!(calls.len(), 1);
assert_eq!(calls[0].name, "memory_search");
assert_eq!(calls[0].arguments, serde_json::json!({"query": "test"}));
}
#[test]
fn test_recover_pipe_delimited() {
let tools = make_tools(&["tool_list"]);
let content = "<|tool_call|>tool_list<|/tool_call|>";
let calls = recover_tool_calls_from_content(content, &tools);
assert_eq!(calls.len(), 1);
assert_eq!(calls[0].name, "tool_list");
}
#[test]
fn test_recover_unknown_tool_ignored() {
let tools = make_tools(&["tool_list"]);
let content = "<tool_call>nonexistent_tool</tool_call>";
let calls = recover_tool_calls_from_content(content, &tools);
assert!(calls.is_empty());
}
#[test]
fn test_recover_no_tags() {
let tools = make_tools(&["tool_list"]);
let content = "Just a normal response.";
let calls = recover_tool_calls_from_content(content, &tools);
assert!(calls.is_empty());
}
#[test]
fn test_recover_multiple_tool_calls() {
let tools = make_tools(&["tool_list", "tool_auth"]);
let content = "<tool_call>tool_list</tool_call>\n<tool_call>tool_auth</tool_call>";
let calls = recover_tool_calls_from_content(content, &tools);
assert_eq!(calls.len(), 2);
assert_eq!(calls[0].name, "tool_list");
assert_eq!(calls[1].name, "tool_auth");
}
#[test]
fn test_recover_function_call_variant() {
let tools = make_tools(&["shell"]);
let content =
r#"<function_call>{"name": "shell", "arguments": {"cmd": "ls"}}</function_call>"#;
let calls = recover_tool_calls_from_content(content, &tools);
assert_eq!(calls.len(), 1);
assert_eq!(calls[0].name, "shell");
}
#[test]
fn test_recover_with_surrounding_text() {
let tools = make_tools(&["tool_list"]);
let content = "Let me check.\n\n<tool_call>tool_list</tool_call>\n\nDone.";
let calls = recover_tool_calls_from_content(content, &tools);
assert_eq!(calls.len(), 1);
assert_eq!(calls[0].name, "tool_list");
}
}
+451
View File
@@ -0,0 +1,451 @@
//! Generic adapter that bridges rig-core's `CompletionModel` trait to IronClaw's `LlmProvider`.
//!
//! This lets us use any rig-core provider (OpenAI, Anthropic, Ollama, etc.) as an
//! `Arc<dyn LlmProvider>` without changing any of the agent, reasoning, or tool code.
use async_trait::async_trait;
use rig::OneOrMany;
use rig::completion::{
AssistantContent, CompletionModel, CompletionRequest as RigRequest,
ToolDefinition as RigToolDefinition, Usage as RigUsage,
};
use rig::message::{
Message as RigMessage, ToolChoice as RigToolChoice, ToolFunction, ToolResult as RigToolResult,
ToolResultContent, UserContent,
};
use rust_decimal::Decimal;
use serde::Serialize;
use serde::de::DeserializeOwned;
use crate::error::LlmError;
use crate::llm::costs;
use crate::llm::provider::{
ChatMessage, CompletionRequest, CompletionResponse, FinishReason, LlmProvider,
ToolCall as IronToolCall, ToolCompletionRequest, ToolCompletionResponse,
ToolDefinition as IronToolDefinition,
};
/// Adapter that wraps a rig-core `CompletionModel` and implements `LlmProvider`.
pub struct RigAdapter<M: CompletionModel> {
model: M,
model_name: String,
input_cost: Decimal,
output_cost: Decimal,
}
impl<M: CompletionModel> RigAdapter<M> {
/// Create a new adapter wrapping the given rig-core model.
pub fn new(model: M, model_name: impl Into<String>) -> Self {
let name = model_name.into();
let (input_cost, output_cost) =
costs::model_cost(&name).unwrap_or_else(costs::default_cost);
Self {
model,
model_name: name,
input_cost,
output_cost,
}
}
}
// -- Type conversion helpers --
/// Convert IronClaw messages to rig-core format.
///
/// Returns `(preamble, chat_history)` where preamble is extracted from
/// any System message and chat_history contains the rest.
fn convert_messages(messages: &[ChatMessage]) -> (Option<String>, Vec<RigMessage>) {
let mut preamble: Option<String> = None;
let mut history = Vec::new();
for msg in messages {
match msg.role {
crate::llm::Role::System => {
// Concatenate system messages into preamble
match preamble {
Some(ref mut p) => {
p.push('\n');
p.push_str(&msg.content);
}
None => preamble = Some(msg.content.clone()),
}
}
crate::llm::Role::User => {
history.push(RigMessage::user(&msg.content));
}
crate::llm::Role::Assistant => {
if let Some(ref tool_calls) = msg.tool_calls {
// Assistant message with tool calls
let mut contents: Vec<AssistantContent> = Vec::new();
if !msg.content.is_empty() {
contents.push(AssistantContent::text(&msg.content));
}
for tc in tool_calls {
contents.push(AssistantContent::ToolCall(rig::message::ToolCall::new(
tc.id.clone(),
ToolFunction::new(tc.name.clone(), tc.arguments.clone()),
)));
}
if let Ok(many) = OneOrMany::many(contents) {
history.push(RigMessage::Assistant {
id: None,
content: many,
});
} else {
// Shouldn't happen but fall back to text
history.push(RigMessage::assistant(&msg.content));
}
} else {
history.push(RigMessage::assistant(&msg.content));
}
}
crate::llm::Role::Tool => {
// Tool result message: wrap as User { ToolResult }
let tool_id = msg.tool_call_id.clone().unwrap_or_default();
history.push(RigMessage::User {
content: OneOrMany::one(UserContent::ToolResult(RigToolResult {
id: tool_id,
call_id: None,
content: OneOrMany::one(ToolResultContent::text(&msg.content)),
})),
});
}
}
}
(preamble, history)
}
/// Convert IronClaw tool definitions to rig-core format.
fn convert_tools(tools: &[IronToolDefinition]) -> Vec<RigToolDefinition> {
tools
.iter()
.map(|t| RigToolDefinition {
name: t.name.clone(),
description: t.description.clone(),
parameters: t.parameters.clone(),
})
.collect()
}
/// Convert IronClaw tool_choice string to rig-core ToolChoice.
fn convert_tool_choice(choice: Option<&str>) -> Option<RigToolChoice> {
match choice.map(|s| s.to_lowercase()).as_deref() {
Some("auto") => Some(RigToolChoice::Auto),
Some("required") => Some(RigToolChoice::Required),
Some("none") => Some(RigToolChoice::None),
_ => None,
}
}
/// Extract text and tool calls from a rig-core completion response.
fn extract_response(
choice: &OneOrMany<AssistantContent>,
_usage: &RigUsage,
) -> (Option<String>, Vec<IronToolCall>, FinishReason) {
let mut text_parts: Vec<String> = Vec::new();
let mut tool_calls: Vec<IronToolCall> = Vec::new();
for content in choice.iter() {
match content {
AssistantContent::Text(t) => {
if !t.text.is_empty() {
text_parts.push(t.text.clone());
}
}
AssistantContent::ToolCall(tc) => {
tool_calls.push(IronToolCall {
id: tc.id.clone(),
name: tc.function.name.clone(),
arguments: tc.function.arguments.clone(),
});
}
// Reasoning and Image variants are not mapped to IronClaw types
_ => {}
}
}
let text = if text_parts.is_empty() {
None
} else {
Some(text_parts.join(""))
};
let finish = if !tool_calls.is_empty() {
FinishReason::ToolUse
} else {
FinishReason::Stop
};
(text, tool_calls, finish)
}
/// Saturate u64 to u32 for token counts.
fn saturate_u32(val: u64) -> u32 {
val.min(u32::MAX as u64) as u32
}
/// Build a rig-core CompletionRequest from our internal types.
fn build_rig_request(
preamble: Option<String>,
mut history: Vec<RigMessage>,
tools: Vec<RigToolDefinition>,
tool_choice: Option<RigToolChoice>,
temperature: Option<f32>,
max_tokens: Option<u32>,
) -> Result<RigRequest, LlmError> {
// rig-core requires at least one message in chat_history
if history.is_empty() {
history.push(RigMessage::user("Hello"));
}
let chat_history = OneOrMany::many(history).map_err(|e| LlmError::RequestFailed {
provider: "rig".to_string(),
reason: format!("Failed to build chat history: {}", e),
})?;
Ok(RigRequest {
preamble,
chat_history,
documents: Vec::new(),
tools,
temperature: temperature.map(|t| t as f64),
max_tokens: max_tokens.map(|t| t as u64),
tool_choice,
additional_params: None,
})
}
#[async_trait]
impl<M> LlmProvider for RigAdapter<M>
where
M: CompletionModel + Send + Sync + 'static,
M::Response: Send + Sync + Serialize + DeserializeOwned,
{
fn model_name(&self) -> &str {
&self.model_name
}
fn cost_per_token(&self) -> (Decimal, Decimal) {
(self.input_cost, self.output_cost)
}
async fn complete(&self, request: CompletionRequest) -> Result<CompletionResponse, LlmError> {
let (preamble, history) = convert_messages(&request.messages);
let rig_req = build_rig_request(
preamble,
history,
Vec::new(),
None,
request.temperature,
request.max_tokens,
)?;
let response =
self.model
.completion(rig_req)
.await
.map_err(|e| LlmError::RequestFailed {
provider: self.model_name.clone(),
reason: e.to_string(),
})?;
let (text, _tool_calls, finish) = extract_response(&response.choice, &response.usage);
Ok(CompletionResponse {
content: text.unwrap_or_default(),
input_tokens: saturate_u32(response.usage.input_tokens),
output_tokens: saturate_u32(response.usage.output_tokens),
finish_reason: finish,
response_id: None,
})
}
async fn complete_with_tools(
&self,
request: ToolCompletionRequest,
) -> Result<ToolCompletionResponse, LlmError> {
let (preamble, history) = convert_messages(&request.messages);
let tools = convert_tools(&request.tools);
let tool_choice = convert_tool_choice(request.tool_choice.as_deref());
let rig_req = build_rig_request(
preamble,
history,
tools,
tool_choice,
request.temperature,
request.max_tokens,
)?;
let response =
self.model
.completion(rig_req)
.await
.map_err(|e| LlmError::RequestFailed {
provider: self.model_name.clone(),
reason: e.to_string(),
})?;
let (text, tool_calls, finish) = extract_response(&response.choice, &response.usage);
Ok(ToolCompletionResponse {
content: text,
tool_calls,
input_tokens: saturate_u32(response.usage.input_tokens),
output_tokens: saturate_u32(response.usage.output_tokens),
finish_reason: finish,
response_id: None,
})
}
fn active_model_name(&self) -> String {
self.model_name.clone()
}
fn set_model(&self, _model: &str) -> Result<(), LlmError> {
// rig-core models are baked at construction time.
// Switching requires creating a new adapter.
Err(LlmError::RequestFailed {
provider: self.model_name.clone(),
reason: "Runtime model switching not supported for rig-core providers. \
Restart with a different model configured."
.to_string(),
})
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_convert_messages_system_to_preamble() {
let messages = vec![
ChatMessage::system("You are a helpful assistant."),
ChatMessage::user("Hello"),
];
let (preamble, history) = convert_messages(&messages);
assert_eq!(preamble, Some("You are a helpful assistant.".to_string()));
assert_eq!(history.len(), 1);
}
#[test]
fn test_convert_messages_multiple_systems_concatenated() {
let messages = vec![
ChatMessage::system("System 1"),
ChatMessage::system("System 2"),
ChatMessage::user("Hi"),
];
let (preamble, history) = convert_messages(&messages);
assert_eq!(preamble, Some("System 1\nSystem 2".to_string()));
assert_eq!(history.len(), 1);
}
#[test]
fn test_convert_messages_tool_result() {
let messages = vec![ChatMessage::tool_result(
"call_123",
"search",
"result text",
)];
let (preamble, history) = convert_messages(&messages);
assert!(preamble.is_none());
assert_eq!(history.len(), 1);
// Tool results become User messages in rig-core
match &history[0] {
RigMessage::User { .. } => {}
other => panic!("Expected User message, got: {:?}", other),
}
}
#[test]
fn test_convert_messages_assistant_with_tool_calls() {
let tc = IronToolCall {
id: "call_1".to_string(),
name: "search".to_string(),
arguments: serde_json::json!({"query": "test"}),
};
let msg = ChatMessage::assistant_with_tool_calls(Some("thinking".to_string()), vec![tc]);
let messages = vec![msg];
let (_preamble, history) = convert_messages(&messages);
assert_eq!(history.len(), 1);
match &history[0] {
RigMessage::Assistant { content, .. } => {
// Should have both text and tool call
assert!(content.iter().count() >= 2);
}
other => panic!("Expected Assistant message, got: {:?}", other),
}
}
#[test]
fn test_convert_tools() {
let tools = vec![IronToolDefinition {
name: "search".to_string(),
description: "Search the web".to_string(),
parameters: serde_json::json!({
"type": "object",
"properties": {
"query": {"type": "string"}
}
}),
}];
let rig_tools = convert_tools(&tools);
assert_eq!(rig_tools.len(), 1);
assert_eq!(rig_tools[0].name, "search");
assert_eq!(rig_tools[0].description, "Search the web");
}
#[test]
fn test_convert_tool_choice() {
assert!(matches!(
convert_tool_choice(Some("auto")),
Some(RigToolChoice::Auto)
));
assert!(matches!(
convert_tool_choice(Some("required")),
Some(RigToolChoice::Required)
));
assert!(matches!(
convert_tool_choice(Some("none")),
Some(RigToolChoice::None)
));
assert!(matches!(
convert_tool_choice(Some("AUTO")),
Some(RigToolChoice::Auto)
));
assert!(convert_tool_choice(None).is_none());
assert!(convert_tool_choice(Some("unknown")).is_none());
}
#[test]
fn test_extract_response_text_only() {
let content = OneOrMany::one(AssistantContent::text("Hello world"));
let usage = RigUsage::new();
let (text, calls, finish) = extract_response(&content, &usage);
assert_eq!(text, Some("Hello world".to_string()));
assert!(calls.is_empty());
assert_eq!(finish, FinishReason::Stop);
}
#[test]
fn test_extract_response_tool_call() {
let tc = AssistantContent::tool_call("call_1", "search", serde_json::json!({"q": "test"}));
let content = OneOrMany::one(tc);
let usage = RigUsage::new();
let (text, calls, finish) = extract_response(&content, &usage);
assert!(text.is_none());
assert_eq!(calls.len(), 1);
assert_eq!(calls[0].name, "search");
assert_eq!(finish, FinishReason::ToolUse);
}
#[test]
fn test_saturate_u32() {
assert_eq!(saturate_u32(100), 100);
assert_eq!(saturate_u32(u64::MAX), u32::MAX);
assert_eq!(saturate_u32(u32::MAX as u64), u32::MAX);
}
}
+96 -2
View File
@@ -61,6 +61,10 @@ pub struct SessionManager {
token: RwLock<Option<SecretString>>,
/// Prevents thundering herd during concurrent 401s.
renewal_lock: Mutex<()>,
/// Optional database store for persisting session to the settings table.
store: RwLock<Option<Arc<dyn crate::db::Database>>>,
/// User ID for DB settings (default: "default").
user_id: RwLock<String>,
}
impl SessionManager {
@@ -74,6 +78,8 @@ impl SessionManager {
.unwrap_or_else(|_| Client::new()),
token: RwLock::new(None),
renewal_lock: Mutex::new(()),
store: RwLock::new(None),
user_id: RwLock::new("default".to_string()),
};
// Try to load existing session synchronously during construction
@@ -103,6 +109,8 @@ impl SessionManager {
.unwrap_or_else(|_| Client::new()),
token: RwLock::new(None),
renewal_lock: Mutex::new(()),
store: RwLock::new(None),
user_id: RwLock::new("default".to_string()),
};
if let Err(e) = manager.load_session().await {
@@ -112,6 +120,21 @@ impl SessionManager {
manager
}
/// Attach a database store for persisting session tokens.
///
/// When a store is attached, session tokens are saved to the `settings`
/// table (key: `nearai.session_token`) in addition to the disk file.
/// On load, DB is preferred over disk.
pub async fn attach_store(&self, store: Arc<dyn crate::db::Database>, user_id: &str) {
*self.store.write().await = Some(store);
*self.user_id.write().await = user_id.to_string();
// Try to load from DB (may have been saved by a previous run)
if let Err(e) = self.load_session_from_db().await {
tracing::debug!("No session in DB: {}", e);
}
}
/// Get the current session token, returning an error if not authenticated.
pub async fn get_token(&self) -> Result<SecretString, LlmError> {
let guard = self.token.read().await;
@@ -460,7 +483,7 @@ impl SessionManager {
Ok(())
}
/// Save session data to disk.
/// Save session data to disk and (if available) to the database.
async fn save_session(&self, token: &str, auth_provider: Option<&str>) -> Result<(), LlmError> {
let session = SessionData {
session_token: token.to_string(),
@@ -468,7 +491,7 @@ impl SessionManager {
auth_provider: auth_provider.map(String::from),
};
// Ensure parent directory exists
// Save to disk (always, as bootstrap fallback)
if let Some(parent) = self.config.session_path.parent() {
tokio::fs::create_dir_all(parent).await.map_err(|e| {
LlmError::Io(std::io::Error::new(
@@ -497,7 +520,78 @@ impl SessionManager {
))
})?;
// Restrictive permissions: session file contains a secret token
#[cfg(unix)]
{
use std::os::unix::fs::PermissionsExt;
let perms = std::fs::Permissions::from_mode(0o600);
tokio::fs::set_permissions(&self.config.session_path, perms)
.await
.map_err(|e| {
LlmError::Io(std::io::Error::new(
e.kind(),
format!(
"Failed to set permissions on {}: {}",
self.config.session_path.display(),
e
),
))
})?;
}
tracing::debug!("Session saved to {}", self.config.session_path.display());
// Also save to DB if a store is attached
if let Some(ref store) = *self.store.read().await {
let user_id = self.user_id.read().await.clone();
let session_json = serde_json::to_value(&session)
.unwrap_or(serde_json::Value::String(token.to_string()));
if let Err(e) = store
.set_setting(&user_id, "nearai.session_token", &session_json)
.await
{
tracing::warn!("Failed to save session to DB: {}", e);
} else {
tracing::debug!("Session also saved to DB settings");
}
}
Ok(())
}
/// Try to load session from the database.
async fn load_session_from_db(&self) -> Result<(), LlmError> {
let store_guard = self.store.read().await;
let store = store_guard
.as_ref()
.ok_or_else(|| LlmError::SessionRenewalFailed {
provider: "nearai".to_string(),
reason: "No DB store attached".to_string(),
})?;
let user_id = self.user_id.read().await.clone();
let value = store
.get_setting(&user_id, "nearai.session_token")
.await
.map_err(|e| LlmError::SessionRenewalFailed {
provider: "nearai".to_string(),
reason: format!("DB query failed: {}", e),
})?
.ok_or_else(|| LlmError::SessionRenewalFailed {
provider: "nearai".to_string(),
reason: "No session in DB".to_string(),
})?;
let session: SessionData =
serde_json::from_value(value).map_err(|e| LlmError::SessionRenewalFailed {
provider: "nearai".to_string(),
reason: format!("Failed to parse DB session: {}", e),
})?;
let mut guard = self.token.write().await;
*guard = Some(SecretString::from(session.session_token));
tracing::info!("Loaded session from DB settings");
Ok(())
}
+458 -59
View File
@@ -17,25 +17,35 @@ use ironclaw::{
web::log_layer::{LogBroadcaster, WebLogLayer},
},
cli::{
Cli, Command, run_mcp_command, run_memory_command, run_status_command, run_tool_command,
Cli, Command, run_mcp_command, run_pairing_command, run_status_command, run_tool_command,
},
config::Config,
context::ContextManager,
extensions::ExtensionManager,
history::Store,
llm::{SessionConfig, create_llm_provider, create_session_manager},
orchestrator::{
ContainerJobConfig, ContainerJobManager, OrchestratorApi, TokenStore,
api::OrchestratorState,
},
pairing::PairingStore,
safety::SafetyLayer,
secrets::{PostgresSecretsStore, SecretsCrypto, SecretsStore},
settings::Settings,
setup::{SetupConfig, SetupWizard},
secrets::SecretsStore,
tools::{
ToolRegistry,
mcp::{McpClient, McpSessionManager, config::load_mcp_servers, is_authenticated},
wasm::{WasmToolLoader, WasmToolRuntime},
mcp::{McpClient, McpSessionManager, config::load_mcp_servers_from_db, is_authenticated},
wasm::{WasmToolLoader, WasmToolRuntime, load_dev_tools},
},
workspace::{EmbeddingProvider, NearAiEmbeddings, OpenAiEmbeddings, Workspace},
};
#[cfg(feature = "libsql")]
use ironclaw::secrets::LibSqlSecretsStore;
#[cfg(feature = "postgres")]
use ironclaw::secrets::PostgresSecretsStore;
use ironclaw::secrets::SecretsCrypto;
#[cfg(any(feature = "postgres", feature = "libsql"))]
use ironclaw::setup::{SetupConfig, SetupWizard};
#[tokio::main]
async fn main() -> anyhow::Result<()> {
let cli = Cli::parse();
@@ -53,9 +63,14 @@ async fn main() -> anyhow::Result<()> {
return run_tool_command(tool_cmd.clone()).await;
}
Some(Command::Config(config_cmd)) => {
// Config commands don't need logging setup
return ironclaw::cli::run_config_command(config_cmd.clone())
.map_err(|e| anyhow::anyhow!("{}", e));
// Config commands need DB access for settings
tracing_subscriber::fmt()
.with_env_filter(
EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new("warn")),
)
.init();
return ironclaw::cli::run_config_command(config_cmd.clone()).await;
}
Some(Command::Mcp(mcp_cmd)) => {
// Simple logging for MCP commands
@@ -76,9 +91,9 @@ async fn main() -> anyhow::Result<()> {
// Memory commands need database (and optionally embeddings)
let _ = dotenvy::dotenv();
let config = Config::from_env().map_err(|e| anyhow::anyhow!("{}", e))?;
let store = ironclaw::history::Store::new(&config.database).await?;
store.run_migrations().await?;
let config = Config::from_env()
.await
.map_err(|e| anyhow::anyhow!("{}", e))?;
// Set up embeddings if available
let session = ironclaw::llm::create_session_manager(ironclaw::llm::SessionConfig {
@@ -118,7 +133,23 @@ async fn main() -> anyhow::Result<()> {
None
};
return run_memory_command(mem_cmd.clone(), store.pool(), embeddings).await;
// Create a Database-trait-backed workspace for the memory command
let db: Arc<dyn ironclaw::db::Database> =
ironclaw::db::connect_from_config(&config.database)
.await
.map_err(|e| anyhow::anyhow!("{}", e))?;
return ironclaw::cli::run_memory_command_with_db(mem_cmd.clone(), db, embeddings)
.await;
}
Some(Command::Pairing(pairing_cmd)) => {
tracing_subscriber::fmt()
.with_env_filter(
EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new("warn")),
)
.init();
return run_pairing_command(pairing_cmd.clone()).map_err(|e| anyhow::anyhow!("{}", e));
}
Some(Command::Status) => {
let _ = dotenvy::dotenv();
@@ -130,6 +161,84 @@ async fn main() -> anyhow::Result<()> {
return run_status_command().await;
}
Some(Command::Worker {
job_id,
orchestrator_url,
max_iterations,
}) => {
// Worker mode: runs inside a Docker container.
// Simple logging (no TUI, no DB, no channels).
tracing_subscriber::fmt()
.with_env_filter(
EnvFilter::try_from_default_env()
.unwrap_or_else(|_| EnvFilter::new("ironclaw=info")),
)
.init();
tracing::info!(
"Starting worker for job {} (orchestrator: {})",
job_id,
orchestrator_url
);
let config = ironclaw::worker::runtime::WorkerConfig {
job_id: *job_id,
orchestrator_url: orchestrator_url.clone(),
max_iterations: *max_iterations,
timeout: std::time::Duration::from_secs(600),
};
let runtime = ironclaw::worker::WorkerRuntime::new(config)
.map_err(|e| anyhow::anyhow!("Worker init failed: {}", e))?;
runtime
.run()
.await
.map_err(|e| anyhow::anyhow!("Worker failed: {}", e))?;
return Ok(());
}
Some(Command::ClaudeBridge {
job_id,
orchestrator_url,
max_turns,
model,
}) => {
// Claude Code bridge mode: runs inside a Docker container.
// Spawns the `claude` CLI and streams output to the orchestrator.
tracing_subscriber::fmt()
.with_env_filter(
EnvFilter::try_from_default_env()
.unwrap_or_else(|_| EnvFilter::new("ironclaw=info")),
)
.init();
tracing::info!(
"Starting Claude Code bridge for job {} (orchestrator: {}, model: {})",
job_id,
orchestrator_url,
model
);
let config = ironclaw::worker::claude_bridge::ClaudeBridgeConfig {
job_id: *job_id,
orchestrator_url: orchestrator_url.clone(),
max_turns: *max_turns,
model: model.clone(),
timeout: std::time::Duration::from_secs(1800),
allowed_tools: Vec::new(),
};
let runtime = ironclaw::worker::ClaudeBridgeRuntime::new(config)
.map_err(|e| anyhow::anyhow!("Claude bridge init failed: {}", e))?;
runtime
.run()
.await
.map_err(|e| anyhow::anyhow!("Claude bridge failed: {}", e))?;
return Ok(());
}
Some(Command::Onboard {
skip_auth,
channels_only,
@@ -137,12 +246,20 @@ async fn main() -> anyhow::Result<()> {
// Load .env before running onboarding wizard
let _ = dotenvy::dotenv();
let config = SetupConfig {
skip_auth: *skip_auth,
channels_only: *channels_only,
};
let mut wizard = SetupWizard::with_config(config);
wizard.run().await?;
#[cfg(any(feature = "postgres", feature = "libsql"))]
{
let config = SetupConfig {
skip_auth: *skip_auth,
channels_only: *channels_only,
};
let mut wizard = SetupWizard::with_config(config);
wizard.run().await?;
}
#[cfg(not(any(feature = "postgres", feature = "libsql")))]
{
let _ = (skip_auth, channels_only);
eprintln!("Onboarding wizard requires the 'postgres' or 'libsql' feature.");
}
return Ok(());
}
None | Some(Command::Run) => {
@@ -154,8 +271,9 @@ async fn main() -> anyhow::Result<()> {
let _ = dotenvy::dotenv();
// Enhanced first-run detection
#[cfg(any(feature = "postgres", feature = "libsql"))]
if !cli.no_onboard {
if let Some(reason) = check_onboard_needed() {
if let Some(reason) = check_onboard_needed().await {
println!("Onboarding needed: {}", reason);
println!();
let mut wizard = SetupWizard::new();
@@ -163,8 +281,11 @@ async fn main() -> anyhow::Result<()> {
}
}
// Load configuration (after potential setup)
let config = match Config::from_env() {
// Load bootstrap config (4 fields that must live on disk)
let bootstrap = ironclaw::bootstrap::BootstrapConfig::load();
// Load initial config from env + disk (before DB is available)
let mut config = match Config::from_env().await {
Ok(c) => c,
Err(ironclaw::error::ConfigError::MissingRequired { key, hint }) => {
eprintln!("Configuration error: Missing required setting '{}'", key);
@@ -186,8 +307,10 @@ async fn main() -> anyhow::Result<()> {
};
let session = create_session_manager(session_config).await;
// Ensure we're authenticated before proceeding (may trigger login flow)
session.ensure_authenticated().await?;
// Ensure we're authenticated before proceeding (only needed for NEAR AI backend)
if config.llm.backend == ironclaw::config::LlmBackend::NearAi {
session.ensure_authenticated().await?;
}
// Initialize tracing
let env_filter = EnvFilter::try_from_default_env()
@@ -214,19 +337,108 @@ async fn main() -> anyhow::Result<()> {
tracing::info!("Starting IronClaw...");
tracing::info!("Loaded configuration for agent: {}", config.agent.name);
tracing::info!("NEAR AI session authenticated");
tracing::info!("LLM backend: {}", config.llm.backend);
// Initialize database store (optional for testing)
let store = if cli.no_db {
// Initialize database backend.
//
// Creates an `Arc<dyn Database>` that all consumers share.
// Backend is selected by the `DATABASE_BACKEND` env var / config.
//
// NOTE: For simpler call sites (CLI commands, Memory handler) use the shared
// helper `ironclaw::db::connect_from_config()`. This block is kept inline
// because it also captures backend-specific handles (`pg_pool`, `libsql_db`)
// needed by the secrets store.
#[cfg(feature = "postgres")]
let mut pg_pool: Option<deadpool_postgres::Pool> = None;
#[cfg(feature = "libsql")]
let mut libsql_db: Option<std::sync::Arc<libsql::Database>> = None;
let db: Option<Arc<dyn ironclaw::db::Database>> = if cli.no_db {
tracing::warn!("Running without database connection");
None
} else {
let store = Store::new(&config.database).await?;
store.run_migrations().await?;
tracing::info!("Database connected and migrations applied");
Some(Arc::new(store))
match config.database.backend {
#[cfg(feature = "libsql")]
ironclaw::config::DatabaseBackend::LibSql => {
use ironclaw::db::Database as _;
use ironclaw::db::libsql_backend::LibSqlBackend;
use secrecy::ExposeSecret as _;
let default_path = ironclaw::config::default_libsql_path();
let db_path = config
.database
.libsql_path
.as_deref()
.unwrap_or(&default_path);
let backend = if let Some(ref url) = config.database.libsql_url {
let token = config.database.libsql_auth_token.as_ref().ok_or_else(|| {
anyhow::anyhow!("LIBSQL_AUTH_TOKEN is required when LIBSQL_URL is set")
})?;
LibSqlBackend::new_remote_replica(db_path, url, token.expose_secret()).await?
} else {
LibSqlBackend::new_local(db_path).await?
};
backend.run_migrations().await?;
tracing::info!("libSQL database connected and migrations applied");
// Capture the Database handle for SecretsStore (connection-per-op)
libsql_db = Some(backend.shared_db());
Some(Arc::new(backend) as Arc<dyn ironclaw::db::Database>)
}
#[cfg(feature = "postgres")]
_ => {
use ironclaw::db::Database as _;
let pg = ironclaw::db::postgres::PgBackend::new(&config.database)
.await
.map_err(|e| anyhow::anyhow!("{}", e))?;
pg.run_migrations()
.await
.map_err(|e| anyhow::anyhow!("{}", e))?;
tracing::info!("PostgreSQL database connected and migrations applied");
pg_pool = Some(pg.pool());
Some(Arc::new(pg) as Arc<dyn ironclaw::db::Database>)
}
#[cfg(not(feature = "postgres"))]
_ => {
anyhow::bail!(
"No database backend available. Enable 'postgres' or 'libsql' feature."
);
}
}
};
// Post-init operations using the database
if let Some(ref db) = db {
// One-time migration: move disk config files into the DB settings table.
if let Err(e) = ironclaw::bootstrap::migrate_disk_to_db(db.as_ref(), "default").await {
tracing::warn!("Disk-to-DB settings migration failed: {}", e);
}
// Reload config from DB now that we have a connection.
match Config::from_db(db.as_ref(), "default", &bootstrap).await {
Ok(db_config) => {
config = db_config;
tracing::info!("Configuration reloaded from database");
}
Err(e) => {
tracing::warn!(
"Failed to reload config from DB, keeping env-based config: {}",
e
);
}
}
// Attach DB to session manager so tokens save to DB too
session.attach_store(Arc::clone(db), "default").await;
// Mark any jobs left in "running" or "creating" state as "interrupted".
if let Err(e) = db.cleanup_stale_sandbox_jobs().await {
tracing::warn!("Failed to cleanup stale sandbox jobs: {}", e);
}
}
// Initialize LLM provider (clone session so we can reuse it for embeddings)
let llm = create_llm_provider(&config.llm, session.clone())?;
tracing::info!("LLM provider initialized: {}", llm.model_name());
@@ -280,8 +492,8 @@ async fn main() -> anyhow::Result<()> {
};
// Register memory tools if database is available
if let Some(ref store) = store {
let mut workspace = Workspace::new("default", store.pool());
if let Some(ref db) = db {
let mut workspace = Workspace::new_with_db("default", Arc::clone(db));
if let Some(ref emb) = embeddings {
workspace = workspace.with_embeddings(emb.clone());
}
@@ -289,8 +501,11 @@ async fn main() -> anyhow::Result<()> {
tools.register_memory_tools(workspace);
}
// Register builder tool if enabled
if config.builder.enabled {
// Register builder tool if enabled.
// When sandbox is enabled and allow_local_tools is false, skip builder registration
// because register_builder_tool also registers dev tools (shell, file ops) that would
// bypass the sandbox. The builder runs inside containers instead.
if config.builder.enabled && (config.agent.allow_local_tools || !config.sandbox.enabled) {
tools
.register_builder_tool(
llm.clone(),
@@ -301,20 +516,46 @@ async fn main() -> anyhow::Result<()> {
tracing::info!("Builder mode enabled");
}
// Create secrets store if master key is configured (needed for MCP auth and WASM channels)
// Create secrets store if master key is configured (needed for MCP auth and WASM channels).
//
// When both `postgres` and `libsql` features are compiled, the runtime-selected
// backend determines which store is created: whichever DB init branch ran will
// have set its handle (pg_pool or libsql_db), and the or_else chain picks it up.
let secrets_store: Option<Arc<dyn SecretsStore + Send + Sync>> =
if let (Some(store), Some(master_key)) = (&store, config.secrets.master_key()) {
if let Some(master_key) = config.secrets.master_key() {
match SecretsCrypto::new(master_key.clone()) {
Ok(crypto) => Some(Arc::new(PostgresSecretsStore::new(
store.pool(),
Arc::new(crypto),
))),
Ok(crypto) => {
let crypto = Arc::new(crypto);
let store: Option<Arc<dyn SecretsStore + Send + Sync>> = None;
#[cfg(feature = "libsql")]
let store = store.or_else(|| {
libsql_db.take().map(|db| {
Arc::new(LibSqlSecretsStore::new(db, Arc::clone(&crypto)))
as Arc<dyn SecretsStore + Send + Sync>
})
});
#[cfg(feature = "postgres")]
let store = store.or_else(|| {
pg_pool.as_ref().map(|pool| {
Arc::new(PostgresSecretsStore::new(pool.clone(), Arc::clone(&crypto)))
as Arc<dyn SecretsStore + Send + Sync>
})
});
store
}
Err(e) => {
tracing::warn!("Failed to initialize secrets crypto: {}", e);
#[cfg(feature = "libsql")]
let _ = libsql_db.take();
None
}
}
} else {
#[cfg(feature = "libsql")]
let _ = libsql_db.take();
None
};
@@ -339,6 +580,8 @@ async fn main() -> anyhow::Result<()> {
let wasm_tools_future = async {
if let Some(ref runtime) = wasm_tool_runtime {
let loader = WasmToolLoader::new(Arc::clone(runtime), Arc::clone(&tools));
// Load installed tools from ~/.ironclaw/tools/
match loader.load_from_dir(&config.wasm.tools_dir).await {
Ok(results) => {
if !results.loaded.is_empty() {
@@ -356,12 +599,32 @@ async fn main() -> anyhow::Result<()> {
tracing::warn!("Failed to scan WASM tools directory: {}", e);
}
}
// Load dev tools from build artifacts (overrides installed if newer)
match load_dev_tools(&loader, &config.wasm.tools_dir).await {
Ok(results) => {
if !results.loaded.is_empty() {
tracing::info!(
"Loaded {} dev WASM tools from build artifacts",
results.loaded.len()
);
}
}
Err(e) => {
tracing::debug!("No dev WASM tools found: {}", e);
}
}
}
};
let mcp_servers_future = async {
if let Some(ref secrets) = secrets_store {
match load_mcp_servers().await {
let servers_result = if let Some(ref d) = db {
load_mcp_servers_from_db(d.as_ref(), "default").await
} else {
ironclaw::tools::mcp::config::load_mcp_servers().await
};
match servers_result {
Ok(servers) => {
let enabled: Vec<_> = servers.enabled_servers().cloned().collect();
if !enabled.is_empty() {
@@ -470,6 +733,7 @@ async fn main() -> anyhow::Result<()> {
config.channels.wasm_channels_dir.clone(),
config.tunnel.public_url.clone(),
"default".to_string(),
db.clone(),
));
tools.register_extension_tools(Arc::clone(&manager));
tracing::info!("Extension manager initialized with in-chat discovery tools");
@@ -482,6 +746,78 @@ async fn main() -> anyhow::Result<()> {
None
};
// Set up orchestrator for sandboxed job execution
// When allow_local_tools is false (default), the LLM uses create_job for FS/shell work.
// When allow_local_tools is true, dev tools are also registered directly (current behavior).
if config.agent.allow_local_tools {
tools.register_dev_tools();
tracing::info!(
"Local tools enabled (allow_local_tools=true), dev tools registered directly"
);
}
// Shared state for job events (used by both orchestrator and web gateway)
let job_event_tx: Option<
tokio::sync::broadcast::Sender<(uuid::Uuid, ironclaw::channels::web::types::SseEvent)>,
> = if config.sandbox.enabled {
let (tx, _) = tokio::sync::broadcast::channel(256);
Some(tx)
} else {
None
};
let prompt_queue = Arc::new(tokio::sync::Mutex::new(std::collections::HashMap::<
uuid::Uuid,
std::collections::VecDeque<ironclaw::orchestrator::api::PendingPrompt>,
>::new()));
let container_job_manager: Option<Arc<ContainerJobManager>> = if config.sandbox.enabled {
let token_store = TokenStore::new();
let job_config = ContainerJobConfig {
image: config.sandbox.image.clone(),
memory_limit_mb: config.sandbox.memory_limit_mb,
cpu_shares: config.sandbox.cpu_shares,
orchestrator_port: 50051,
claude_config_dir: if config.claude_code.enabled {
Some(config.claude_code.config_dir.clone())
} else {
None
},
claude_code_model: config.claude_code.model.clone(),
claude_code_max_turns: config.claude_code.max_turns,
claude_code_memory_limit_mb: config.claude_code.memory_limit_mb,
claude_code_allowed_tools: config.claude_code.allowed_tools.clone(),
};
let jm = Arc::new(ContainerJobManager::new(job_config, token_store.clone()));
// Start the orchestrator internal API in the background
let orchestrator_state = OrchestratorState {
llm: llm.clone(),
job_manager: Arc::clone(&jm),
token_store,
job_event_tx: job_event_tx.clone(),
prompt_queue: Arc::clone(&prompt_queue),
store: db.clone(),
};
tokio::spawn(async move {
if let Err(e) = OrchestratorApi::start(orchestrator_state, 50051).await {
tracing::error!("Orchestrator API failed: {}", e);
}
});
tracing::info!("Orchestrator API started on :50051, sandbox delegation enabled");
if config.claude_code.enabled {
tracing::info!(
"Claude Code sandbox mode available (model: {}, max_turns: {})",
config.claude_code.model,
config.claude_code.max_turns
);
}
Some(jm)
} else {
None
};
tracing::info!(
"Tool registry initialized with {} total tools",
tools.count()
@@ -507,7 +843,8 @@ async fn main() -> anyhow::Result<()> {
match WasmChannelRuntime::new(WasmChannelRuntimeConfig::default()) {
Ok(runtime) => {
let runtime = Arc::new(runtime);
let loader = WasmChannelLoader::new(Arc::clone(&runtime));
let pairing_store = Arc::new(PairingStore::new());
let loader = WasmChannelLoader::new(Arc::clone(&runtime), pairing_store);
match loader
.load_from_dir(&config.channels.wasm_channels_dir)
@@ -563,6 +900,17 @@ async fn main() -> anyhow::Result<()> {
);
}
// Inject owner_id for Telegram so the bot only responds
// to the bound user account.
if channel_name == "telegram" {
if let Some(owner_id) = config.channels.telegram_owner_id {
config_updates.insert(
"owner_id".to_string(),
serde_json::json!(owner_id),
);
}
}
if !config_updates.is_empty() {
channel_arc.update_config(config_updates).await;
tracing::info!(
@@ -621,7 +969,7 @@ async fn main() -> anyhow::Result<()> {
channels.add(Box::new(SharedWasmChannel::new(channel_arc)));
}
if has_webhook_channels && config.tunnel.public_url.is_some() {
if has_webhook_channels {
webhook_routes.push(create_wasm_channel_router(
wasm_router,
extension_manager.as_ref().map(Arc::clone),
@@ -685,13 +1033,28 @@ async fn main() -> anyhow::Result<()> {
};
// Create workspace for agent (shared with memory tools)
let workspace = store.as_ref().map(|s| {
let mut ws = Workspace::new("default", s.pool());
let workspace = if let Some(ref db_ref) = db {
let mut ws = Workspace::new_with_db("default", Arc::clone(db_ref));
if let Some(ref emb) = embeddings {
ws = ws.with_embeddings(emb.clone());
}
Arc::new(ws)
});
Some(Arc::new(ws))
} else {
None
};
// Seed workspace with core identity files on first boot
if let Some(ref ws) = workspace {
match ws.seed_if_empty().await {
Ok(count) if count > 0 => {
tracing::info!("Workspace seeded with {} core files", count);
}
Ok(_) => {}
Err(e) => {
tracing::warn!("Failed to seed workspace: {}", e);
}
}
}
// Backfill embeddings if we just enabled the provider
if let (Some(ws), Some(_)) = (&workspace, &embeddings) {
@@ -712,8 +1075,12 @@ async fn main() -> anyhow::Result<()> {
// Create session manager (shared between agent and web gateway)
let session_manager = Arc::new(SessionManager::new());
// Register job tools
tools.register_job_tools(Arc::clone(&context_manager));
// Register job tools (sandbox deps auto-injected when container_job_manager is available)
tools.register_job_tools(
Arc::clone(&context_manager),
container_job_manager.clone(),
db.clone(),
);
// Add web gateway channel if configured
if let Some(ref gw_config) = config.channels.gateway {
@@ -721,26 +1088,51 @@ async fn main() -> anyhow::Result<()> {
if let Some(ref ws) = workspace {
gw = gw.with_workspace(Arc::clone(ws));
}
gw = gw.with_context_manager(Arc::clone(&context_manager));
gw = gw.with_session_manager(Arc::clone(&session_manager));
gw = gw.with_log_broadcaster(Arc::clone(&log_broadcaster));
gw = gw.with_tool_registry(Arc::clone(&tools));
if let Some(ref ext_mgr) = extension_manager {
gw = gw.with_extension_manager(Arc::clone(ext_mgr));
}
if let Some(ref d) = db {
gw = gw.with_store(Arc::clone(d));
}
if let Some(ref jm) = container_job_manager {
gw = gw.with_job_manager(Arc::clone(jm));
}
if config.sandbox.enabled {
gw = gw.with_prompt_queue(Arc::clone(&prompt_queue));
// Spawn a task to forward job events from the broadcast channel to SSE
if let Some(ref tx) = job_event_tx {
let mut rx = tx.subscribe();
let gw_state = Arc::clone(gw.state());
tokio::spawn(async move {
while let Ok((_job_id, event)) = rx.recv().await {
gw_state.sse.broadcast(event);
}
});
}
}
tracing::info!(
"Web gateway enabled on {}:{}",
gw_config.host,
gw_config.port
);
tracing::info!(
"Web UI: http://{}:{}/?token={}",
gw_config.host,
gw_config.port,
gw.auth_token()
);
channels.add(Box::new(gw));
}
// Create and run the agent
let deps = AgentDeps {
store,
store: db,
llm,
safety,
tools,
@@ -752,6 +1144,7 @@ async fn main() -> anyhow::Result<()> {
deps,
channels,
Some(config.heartbeat.clone()),
Some(config.routines.clone()),
Some(context_manager),
Some(session_manager),
);
@@ -773,18 +1166,24 @@ async fn main() -> anyhow::Result<()> {
/// Check if onboarding is needed and return the reason.
///
/// Returns `Some(reason)` if onboarding should be triggered, `None` otherwise.
fn check_onboard_needed() -> Option<&'static str> {
let settings = Settings::load();
#[cfg(any(feature = "postgres", feature = "libsql"))]
async fn check_onboard_needed() -> Option<&'static str> {
let bootstrap = ironclaw::bootstrap::BootstrapConfig::load();
// Database not configured (and not in env)
if settings.database_url.is_none() && std::env::var("DATABASE_URL").is_err() {
let has_db = bootstrap.database_url.is_some()
|| std::env::var("DATABASE_URL").is_ok()
|| std::env::var("LIBSQL_PATH").is_ok()
|| ironclaw::config::default_libsql_path().exists();
if !has_db {
return Some("Database not configured");
}
// Secrets not configured (and not in env)
if settings.secrets_master_key_source == ironclaw::settings::KeySource::None
if bootstrap.secrets_master_key_source == ironclaw::settings::KeySource::None
&& std::env::var("SECRETS_MASTER_KEY").is_err()
&& !ironclaw::secrets::keychain::has_master_key()
&& !ironclaw::secrets::keychain::has_master_key().await
{
// Only require secrets setup if user hasn't explicitly disabled it
// For now, we don't require it for first run
@@ -792,7 +1191,7 @@ fn check_onboard_needed() -> Option<&'static str> {
// First run (onboarding never completed and no session)
let session_path = ironclaw::llm::session::default_session_path();
if !settings.onboard_completed && !session_path.exists() {
if !bootstrap.onboard_completed && !session_path.exists() {
return Some("First run");
}
+511
View File
@@ -0,0 +1,511 @@
//! Internal HTTP API for worker-to-orchestrator communication.
//!
//! This runs on a separate port (default 50051) from the web gateway.
//! All endpoints are authenticated via per-job bearer tokens.
use std::collections::{HashMap, VecDeque};
use std::sync::Arc;
use axum::extract::{Path, State};
use axum::http::StatusCode;
use axum::routing::{get, post};
use axum::{Json, Router};
use serde::{Deserialize, Serialize};
use tokio::sync::{Mutex, broadcast};
use uuid::Uuid;
use crate::channels::web::types::SseEvent;
use crate::db::Database;
use crate::llm::{CompletionRequest, LlmProvider, ToolCompletionRequest};
use crate::orchestrator::auth::{TokenStore, worker_auth_middleware};
use crate::orchestrator::job_manager::ContainerJobManager;
use crate::worker::api::JobEventPayload;
use crate::worker::api::{
CompletionReport, JobDescription, ProxyCompletionRequest, ProxyCompletionResponse,
ProxyToolCompletionRequest, ProxyToolCompletionResponse, StatusUpdate,
};
/// A follow-up prompt queued for a Claude Code bridge.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct PendingPrompt {
pub content: String,
pub done: bool,
}
/// Shared state for the orchestrator API.
#[derive(Clone)]
pub struct OrchestratorState {
pub llm: Arc<dyn LlmProvider>,
pub job_manager: Arc<ContainerJobManager>,
pub token_store: TokenStore,
/// Broadcast channel for job events (consumed by the web gateway SSE).
pub job_event_tx: Option<broadcast::Sender<(Uuid, SseEvent)>>,
/// Buffered follow-up prompts for sandbox jobs, keyed by job_id.
pub prompt_queue: Arc<Mutex<HashMap<Uuid, VecDeque<PendingPrompt>>>>,
/// Database handle for persisting job events.
pub store: Option<Arc<dyn Database>>,
}
/// The orchestrator's internal API server.
pub struct OrchestratorApi;
impl OrchestratorApi {
/// Build the axum router for the internal API.
pub fn router(state: OrchestratorState) -> Router {
Router::new()
// Worker routes: authenticated via route_layer middleware.
.route("/worker/{job_id}/job", get(get_job))
.route("/worker/{job_id}/llm/complete", post(llm_complete))
.route(
"/worker/{job_id}/llm/complete_with_tools",
post(llm_complete_with_tools),
)
.route("/worker/{job_id}/status", post(report_status))
.route("/worker/{job_id}/complete", post(report_complete))
.route("/worker/{job_id}/event", post(job_event_handler))
.route("/worker/{job_id}/prompt", get(get_prompt_handler))
.route_layer(axum::middleware::from_fn_with_state(
state.token_store.clone(),
worker_auth_middleware,
))
// Unauthenticated routes (added after the layer).
.route("/health", get(health_check))
.with_state(state)
}
/// Start the internal API server on the given port.
///
/// On macOS/Windows (Docker Desktop), binds to loopback only because
/// Docker Desktop routes `host.docker.internal` through its VM to the
/// host's `127.0.0.1`.
///
/// On Linux, containers reach the host via the docker bridge gateway
/// (`172.17.0.1`), which is NOT loopback. Binding to `127.0.0.1`
/// would reject container traffic. We bind to all interfaces instead
/// and rely on `worker_auth_middleware` (applied as a route_layer on
/// every `/worker/` endpoint) to reject unauthenticated requests.
pub async fn start(
state: OrchestratorState,
port: u16,
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let router = Self::router(state);
let addr = if cfg!(target_os = "linux") {
std::net::SocketAddr::from(([0, 0, 0, 0], port))
} else {
std::net::SocketAddr::from(([127, 0, 0, 1], port))
};
tracing::info!("Orchestrator internal API listening on {}", addr);
let listener = tokio::net::TcpListener::bind(addr).await?;
axum::serve(listener, router).await?;
Ok(())
}
}
// -- Handlers --
//
// All /worker/ handlers below are behind the worker_auth_middleware route_layer,
// so they don't need to validate tokens themselves.
async fn health_check() -> &'static str {
"ok"
}
async fn get_job(
State(state): State<OrchestratorState>,
Path(job_id): Path<Uuid>,
) -> Result<Json<JobDescription>, StatusCode> {
let handle = state
.job_manager
.get_handle(job_id)
.await
.ok_or(StatusCode::NOT_FOUND)?;
Ok(Json(JobDescription {
title: format!("Job {}", job_id),
description: handle.task_description,
project_dir: handle.project_dir.map(|p| p.display().to_string()),
}))
}
async fn llm_complete(
State(state): State<OrchestratorState>,
Path(job_id): Path<Uuid>,
Json(req): Json<ProxyCompletionRequest>,
) -> Result<Json<ProxyCompletionResponse>, StatusCode> {
let completion_req = CompletionRequest {
messages: req.messages,
max_tokens: req.max_tokens,
temperature: req.temperature,
stop_sequences: req.stop_sequences,
metadata: std::collections::HashMap::new(),
};
let resp = state.llm.complete(completion_req).await.map_err(|e| {
tracing::error!("LLM completion failed for job {}: {}", job_id, e);
StatusCode::BAD_GATEWAY
})?;
Ok(Json(ProxyCompletionResponse {
content: resp.content,
input_tokens: resp.input_tokens,
output_tokens: resp.output_tokens,
finish_reason: format_finish_reason(resp.finish_reason),
}))
}
async fn llm_complete_with_tools(
State(state): State<OrchestratorState>,
Path(job_id): Path<Uuid>,
Json(req): Json<ProxyToolCompletionRequest>,
) -> Result<Json<ProxyToolCompletionResponse>, StatusCode> {
let tool_req = ToolCompletionRequest {
messages: req.messages,
tools: req.tools,
max_tokens: req.max_tokens,
temperature: req.temperature,
tool_choice: req.tool_choice,
metadata: std::collections::HashMap::new(),
};
let resp = state.llm.complete_with_tools(tool_req).await.map_err(|e| {
tracing::error!("LLM tool completion failed for job {}: {}", job_id, e);
StatusCode::BAD_GATEWAY
})?;
Ok(Json(ProxyToolCompletionResponse {
content: resp.content,
tool_calls: resp.tool_calls,
input_tokens: resp.input_tokens,
output_tokens: resp.output_tokens,
finish_reason: format_finish_reason(resp.finish_reason),
}))
}
async fn report_status(
Path(job_id): Path<Uuid>,
Json(update): Json<StatusUpdate>,
) -> Result<StatusCode, StatusCode> {
tracing::debug!(
job_id = %job_id,
state = %update.state,
iteration = update.iteration,
"Worker status update"
);
Ok(StatusCode::OK)
}
async fn report_complete(
State(state): State<OrchestratorState>,
Path(job_id): Path<Uuid>,
Json(report): Json<CompletionReport>,
) -> Result<StatusCode, StatusCode> {
if report.success {
tracing::info!(
job_id = %job_id,
"Worker reported job complete"
);
} else {
tracing::warn!(
job_id = %job_id,
message = ?report.message,
"Worker reported job failure"
);
}
// Store the result and clean up the container
let result = crate::orchestrator::job_manager::CompletionResult {
success: report.success,
message: report.message.clone(),
};
let _ = state.job_manager.complete_job(job_id, result).await;
Ok(StatusCode::OK)
}
// -- Sandbox job event handlers --
/// Receive a job event from a worker or Claude Code bridge and broadcast + persist it.
async fn job_event_handler(
State(state): State<OrchestratorState>,
Path(job_id): Path<Uuid>,
Json(payload): Json<JobEventPayload>,
) -> Result<StatusCode, StatusCode> {
tracing::debug!(
job_id = %job_id,
event_type = %payload.event_type,
"Job event received"
);
// Persist to DB (fire-and-forget)
if let Some(ref store) = state.store {
let store = Arc::clone(store);
let event_type = payload.event_type.clone();
let data = payload.data.clone();
tokio::spawn(async move {
if let Err(e) = store.save_job_event(job_id, &event_type, &data).await {
tracing::warn!(job_id = %job_id, "Failed to persist job event: {}", e);
}
});
}
// Convert to SSE event and broadcast
let job_id_str = job_id.to_string();
let sse_event = match payload.event_type.as_str() {
"message" => SseEvent::JobMessage {
job_id: job_id_str,
role: payload
.data
.get("role")
.and_then(|v| v.as_str())
.unwrap_or("assistant")
.to_string(),
content: payload
.data
.get("content")
.and_then(|v| v.as_str())
.unwrap_or("")
.to_string(),
},
"tool_use" => SseEvent::JobToolUse {
job_id: job_id_str,
tool_name: payload
.data
.get("tool_name")
.and_then(|v| v.as_str())
.unwrap_or("unknown")
.to_string(),
input: payload
.data
.get("input")
.cloned()
.unwrap_or(serde_json::Value::Null),
},
"tool_result" => SseEvent::JobToolResult {
job_id: job_id_str,
tool_name: payload
.data
.get("tool_name")
.and_then(|v| v.as_str())
.unwrap_or("unknown")
.to_string(),
output: payload
.data
.get("output")
.and_then(|v| v.as_str())
.unwrap_or("")
.to_string(),
},
"result" => SseEvent::JobResult {
job_id: job_id_str,
status: payload
.data
.get("status")
.and_then(|v| v.as_str())
.unwrap_or("unknown")
.to_string(),
session_id: payload
.data
.get("session_id")
.and_then(|v| v.as_str())
.map(|s| s.to_string()),
},
_ => SseEvent::JobStatus {
job_id: job_id_str,
message: payload
.data
.get("message")
.and_then(|v| v.as_str())
.unwrap_or("")
.to_string(),
},
};
// Broadcast via the channel (if configured)
if let Some(ref tx) = state.job_event_tx {
let _ = tx.send((job_id, sse_event));
}
Ok(StatusCode::OK)
}
/// Return the next queued follow-up prompt for a Claude Code bridge.
/// Returns 204 No Content if no prompt is available.
async fn get_prompt_handler(
State(state): State<OrchestratorState>,
Path(job_id): Path<Uuid>,
) -> Result<(StatusCode, Json<serde_json::Value>), StatusCode> {
let mut queue = state.prompt_queue.lock().await;
if let Some(prompts) = queue.get_mut(&job_id) {
if let Some(prompt) = prompts.pop_front() {
return Ok((
StatusCode::OK,
Json(serde_json::json!({
"content": prompt.content,
"done": prompt.done,
})),
));
}
}
// Return 204 with an empty body. The Json wrapper requires some value
// but the status code signals "nothing here".
Ok((StatusCode::NO_CONTENT, Json(serde_json::Value::Null)))
}
fn format_finish_reason(reason: crate::llm::FinishReason) -> String {
match reason {
crate::llm::FinishReason::Stop => "stop".to_string(),
crate::llm::FinishReason::Length => "length".to_string(),
crate::llm::FinishReason::ToolUse => "tool_use".to_string(),
crate::llm::FinishReason::ContentFilter => "content_filter".to_string(),
crate::llm::FinishReason::Unknown => "unknown".to_string(),
}
}
#[cfg(test)]
mod tests {
use std::collections::HashMap;
use axum::body::Body;
use axum::http::Request;
use tower::ServiceExt;
use uuid::Uuid;
use crate::error::LlmError;
use crate::llm::{
CompletionRequest, CompletionResponse, ToolCompletionRequest, ToolCompletionResponse,
};
use crate::orchestrator::auth::TokenStore;
use crate::orchestrator::job_manager::{ContainerJobConfig, ContainerJobManager};
use super::*;
/// Stub LLM provider that panics if called (tests only exercise routing/auth).
struct StubLlm;
#[async_trait::async_trait]
impl crate::llm::LlmProvider for StubLlm {
fn model_name(&self) -> &str {
"stub"
}
fn cost_per_token(&self) -> (rust_decimal::Decimal, rust_decimal::Decimal) {
(rust_decimal::Decimal::ZERO, rust_decimal::Decimal::ZERO)
}
async fn complete(&self, _req: CompletionRequest) -> Result<CompletionResponse, LlmError> {
Err(LlmError::RequestFailed {
provider: "stub".into(),
reason: "not implemented".into(),
})
}
async fn complete_with_tools(
&self,
_req: ToolCompletionRequest,
) -> Result<ToolCompletionResponse, LlmError> {
Err(LlmError::RequestFailed {
provider: "stub".into(),
reason: "not implemented".into(),
})
}
}
fn test_state() -> OrchestratorState {
let token_store = TokenStore::new();
let jm = ContainerJobManager::new(ContainerJobConfig::default(), token_store.clone());
OrchestratorState {
llm: Arc::new(StubLlm),
job_manager: Arc::new(jm),
token_store,
job_event_tx: None,
prompt_queue: Arc::new(Mutex::new(HashMap::new())),
store: None,
}
}
#[tokio::test]
async fn health_requires_no_auth() {
let state = test_state();
let router = OrchestratorApi::router(state);
let req = Request::builder()
.uri("/health")
.body(Body::empty())
.unwrap();
let resp = router.oneshot(req).await.unwrap();
assert_eq!(resp.status(), StatusCode::OK);
}
#[tokio::test]
async fn worker_route_rejects_missing_token() {
let state = test_state();
let router = OrchestratorApi::router(state);
let job_id = Uuid::new_v4();
let req = Request::builder()
.uri(format!("/worker/{}/job", job_id))
.body(Body::empty())
.unwrap();
let resp = router.oneshot(req).await.unwrap();
assert_eq!(resp.status(), StatusCode::UNAUTHORIZED);
}
#[tokio::test]
async fn worker_route_rejects_wrong_token() {
let state = test_state();
let router = OrchestratorApi::router(state);
let job_id = Uuid::new_v4();
let req = Request::builder()
.uri(format!("/worker/{}/job", job_id))
.header("Authorization", "Bearer totally-bogus")
.body(Body::empty())
.unwrap();
let resp = router.oneshot(req).await.unwrap();
assert_eq!(resp.status(), StatusCode::UNAUTHORIZED);
}
#[tokio::test]
async fn worker_route_accepts_valid_token() {
let state = test_state();
let job_id = Uuid::new_v4();
let token = state.token_store.create_token(job_id).await;
let router = OrchestratorApi::router(state);
let req = Request::builder()
.uri(format!("/worker/{}/job", job_id))
.header("Authorization", format!("Bearer {}", token))
.body(Body::empty())
.unwrap();
let resp = router.oneshot(req).await.unwrap();
// 404 because no container exists for this job_id, but NOT 401.
assert_eq!(resp.status(), StatusCode::NOT_FOUND);
}
#[tokio::test]
async fn token_for_job_a_rejected_on_job_b() {
let state = test_state();
let job_a = Uuid::new_v4();
let job_b = Uuid::new_v4();
let token_a = state.token_store.create_token(job_a).await;
let router = OrchestratorApi::router(state);
// Use job_a's token to hit job_b's endpoint
let req = Request::builder()
.uri(format!("/worker/{}/job", job_b))
.header("Authorization", format!("Bearer {}", token_a))
.body(Body::empty())
.unwrap();
let resp = router.oneshot(req).await.unwrap();
assert_eq!(resp.status(), StatusCode::UNAUTHORIZED);
}
}
+163
View File
@@ -0,0 +1,163 @@
//! Per-job bearer token authentication for worker-to-orchestrator communication.
//!
//! Security properties:
//! - Tokens are cryptographically random (32 bytes, hex-encoded)
//! - Tokens are scoped to a specific job_id
//! - Tokens are ephemeral (in-memory only, never persisted)
//! - A token for Job A cannot access endpoints for Job B
use std::collections::HashMap;
use std::sync::Arc;
use axum::extract::{Request, State};
use axum::http::StatusCode;
use axum::middleware::Next;
use axum::response::Response;
use rand::Rng;
use subtle::ConstantTimeEq;
use tokio::sync::RwLock;
use uuid::Uuid;
/// In-memory store for per-job authentication tokens.
#[derive(Clone)]
pub struct TokenStore {
/// Maps job_id -> bearer token. Never logged or persisted.
tokens: Arc<RwLock<HashMap<Uuid, String>>>,
}
impl TokenStore {
pub fn new() -> Self {
Self {
tokens: Arc::new(RwLock::new(HashMap::new())),
}
}
/// Generate and store a new token for a job.
pub async fn create_token(&self, job_id: Uuid) -> String {
let token = generate_token();
self.tokens.write().await.insert(job_id, token.clone());
token
}
/// Validate a token for a specific job (constant-time comparison).
pub async fn validate(&self, job_id: Uuid, token: &str) -> bool {
self.tokens
.read()
.await
.get(&job_id)
.map(|stored| stored.as_bytes().ct_eq(token.as_bytes()).into())
.unwrap_or(false)
}
/// Remove a token (on container cleanup).
pub async fn revoke(&self, job_id: Uuid) {
self.tokens.write().await.remove(&job_id);
}
/// Get the number of active tokens (for diagnostics).
pub async fn active_count(&self) -> usize {
self.tokens.read().await.len()
}
}
impl Default for TokenStore {
fn default() -> Self {
Self::new()
}
}
/// Generate a cryptographically random token (32 bytes, hex-encoded = 64 chars).
fn generate_token() -> String {
let mut bytes = [0u8; 32];
rand::thread_rng().fill(&mut bytes);
hex_encode(&bytes)
}
fn hex_encode(bytes: &[u8]) -> String {
bytes.iter().map(|b| format!("{:02x}", b)).collect()
}
/// Axum middleware that validates worker bearer tokens.
///
/// Extracts the job_id from the path (`/worker/{job_id}/...`) and validates
/// the `Authorization: Bearer <token>` header against the token store.
///
/// Wire up with `axum::middleware::from_fn_with_state(token_store, worker_auth_middleware)`.
pub async fn worker_auth_middleware(
State(token_store): State<TokenStore>,
request: Request,
next: Next,
) -> Result<Response, StatusCode> {
let path = request.uri().path().to_string();
let job_id = extract_job_id_from_path(&path).ok_or(StatusCode::BAD_REQUEST)?;
let token = request
.headers()
.get("authorization")
.and_then(|v| v.to_str().ok())
.and_then(|v| v.strip_prefix("Bearer "))
.ok_or(StatusCode::UNAUTHORIZED)?;
if !token_store.validate(job_id, token).await {
return Err(StatusCode::UNAUTHORIZED);
}
Ok(next.run(request).await)
}
/// Extract job UUID from a path like `/worker/{uuid}/...`
fn extract_job_id_from_path(path: &str) -> Option<Uuid> {
let parts: Vec<&str> = path.trim_start_matches('/').split('/').collect();
if parts.len() >= 2 && parts[0] == "worker" {
Uuid::parse_str(parts[1]).ok()
} else {
None
}
}
#[cfg(test)]
mod tests {
use super::*;
#[tokio::test]
async fn test_token_create_and_validate() {
let store = TokenStore::new();
let job_id = Uuid::new_v4();
let token = store.create_token(job_id).await;
assert_eq!(token.len(), 64); // 32 bytes hex = 64 chars
assert!(store.validate(job_id, &token).await);
assert!(!store.validate(job_id, "wrong-token").await);
assert!(!store.validate(Uuid::new_v4(), &token).await);
}
#[tokio::test]
async fn test_token_revoke() {
let store = TokenStore::new();
let job_id = Uuid::new_v4();
let token = store.create_token(job_id).await;
assert!(store.validate(job_id, &token).await);
store.revoke(job_id).await;
assert!(!store.validate(job_id, &token).await);
}
#[test]
fn test_extract_job_id() {
let id = Uuid::new_v4();
let path = format!("/worker/{}/llm/complete", id);
assert_eq!(extract_job_id_from_path(&path), Some(id));
assert_eq!(extract_job_id_from_path("/other/path"), None);
assert_eq!(extract_job_id_from_path("/worker/not-a-uuid/foo"), None);
}
#[test]
fn test_token_is_random() {
let t1 = generate_token();
let t2 = generate_token();
assert_ne!(t1, t2);
}
}
+520
View File
@@ -0,0 +1,520 @@
//! Container lifecycle management for sandboxed jobs.
//!
//! Extends the existing `SandboxManager` infrastructure to support persistent
//! containers with their own agent loops (as opposed to ephemeral per-command containers).
use std::collections::HashMap;
use std::path::PathBuf;
use std::sync::Arc;
use chrono::{DateTime, Utc};
use tokio::sync::RwLock;
use uuid::Uuid;
use crate::error::OrchestratorError;
use crate::orchestrator::auth::TokenStore;
use crate::sandbox::connect_docker;
/// Which mode a sandbox container runs in.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum JobMode {
/// Standard IronClaw worker with proxied LLM calls.
Worker,
/// Claude Code bridge that spawns the `claude` CLI directly.
ClaudeCode,
}
impl JobMode {
pub fn as_str(&self) -> &'static str {
match self {
Self::Worker => "worker",
Self::ClaudeCode => "claude_code",
}
}
}
impl std::fmt::Display for JobMode {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(f, "{}", self.as_str())
}
}
/// Configuration for the container job manager.
#[derive(Debug, Clone)]
pub struct ContainerJobConfig {
/// Docker image for worker containers.
pub image: String,
/// Default memory limit in MB.
pub memory_limit_mb: u64,
/// Default CPU shares.
pub cpu_shares: u32,
/// Port the orchestrator internal API listens on.
pub orchestrator_port: u16,
/// Host directory containing Claude auth config (mounted read-only for ClaudeCode mode).
pub claude_config_dir: Option<PathBuf>,
/// Claude model to use in ClaudeCode mode.
pub claude_code_model: String,
/// Maximum turns for Claude Code.
pub claude_code_max_turns: u32,
/// Memory limit in MB for Claude Code containers (heavier than workers).
pub claude_code_memory_limit_mb: u64,
/// Allowed tool patterns for Claude Code (passed as CLAUDE_CODE_ALLOWED_TOOLS env var).
pub claude_code_allowed_tools: Vec<String>,
}
impl Default for ContainerJobConfig {
fn default() -> Self {
Self {
image: "ironclaw-worker:latest".to_string(),
memory_limit_mb: 2048,
cpu_shares: 1024,
orchestrator_port: 50051,
claude_config_dir: None,
claude_code_model: "sonnet".to_string(),
claude_code_max_turns: 50,
claude_code_memory_limit_mb: 4096,
claude_code_allowed_tools: crate::config::ClaudeCodeConfig::default().allowed_tools,
}
}
}
/// State of a container.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum ContainerState {
Creating,
Running,
Stopped,
Failed,
}
impl std::fmt::Display for ContainerState {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
Self::Creating => write!(f, "creating"),
Self::Running => write!(f, "running"),
Self::Stopped => write!(f, "stopped"),
Self::Failed => write!(f, "failed"),
}
}
}
/// Handle to a running container job.
#[derive(Debug, Clone)]
pub struct ContainerHandle {
pub job_id: Uuid,
pub container_id: String,
pub state: ContainerState,
pub mode: JobMode,
pub created_at: DateTime<Utc>,
pub project_dir: Option<PathBuf>,
pub task_description: String,
/// Completion result from the worker (set when the worker reports done).
pub completion_result: Option<CompletionResult>,
// NOTE: auth_token is intentionally NOT in this struct.
// It lives only in the TokenStore (never logged, serialized, or persisted).
}
/// Result reported by a worker on completion.
#[derive(Debug, Clone)]
pub struct CompletionResult {
pub success: bool,
pub message: Option<String>,
}
/// Manages the lifecycle of Docker containers for sandboxed job execution.
pub struct ContainerJobManager {
config: ContainerJobConfig,
token_store: TokenStore,
containers: Arc<RwLock<HashMap<Uuid, ContainerHandle>>>,
}
impl ContainerJobManager {
pub fn new(config: ContainerJobConfig, token_store: TokenStore) -> Self {
Self {
config,
token_store,
containers: Arc::new(RwLock::new(HashMap::new())),
}
}
/// Create and start a new container for a job.
///
/// The caller provides the `job_id` so it can be persisted to the database
/// before the container is created. Returns the auth token for the worker.
pub async fn create_job(
&self,
job_id: Uuid,
task: &str,
project_dir: Option<PathBuf>,
mode: JobMode,
) -> Result<String, OrchestratorError> {
// Generate auth token (stored in TokenStore, never logged)
let token = self.token_store.create_token(job_id).await;
// Record the handle
let handle = ContainerHandle {
job_id,
container_id: String::new(), // set after container creation
state: ContainerState::Creating,
mode,
created_at: Utc::now(),
project_dir: project_dir.clone(),
task_description: task.to_string(),
completion_result: None,
};
self.containers.write().await.insert(job_id, handle);
// Run the actual container creation. On any failure, revoke the token
// and remove the handle so we don't leak resources.
match self
.create_job_inner(job_id, &token, project_dir, mode)
.await
{
Ok(()) => Ok(token),
Err(e) => {
self.token_store.revoke(job_id).await;
self.containers.write().await.remove(&job_id);
Err(e)
}
}
}
/// Inner implementation of container creation (separated for cleanup).
async fn create_job_inner(
&self,
job_id: Uuid,
token: &str,
project_dir: Option<PathBuf>,
mode: JobMode,
) -> Result<(), OrchestratorError> {
// Connect to Docker
let docker = connect_docker()
.await
.map_err(|e| OrchestratorError::Docker {
reason: e.to_string(),
})?;
// Build container configuration
let orchestrator_host = if cfg!(target_os = "linux") {
"172.17.0.1"
} else {
"host.docker.internal"
};
let orchestrator_url = format!(
"http://{}:{}",
orchestrator_host, self.config.orchestrator_port
);
let mut env_vec = vec![
format!("IRONCLAW_WORKER_TOKEN={}", token),
format!("IRONCLAW_JOB_ID={}", job_id),
format!("IRONCLAW_ORCHESTRATOR_URL={}", orchestrator_url),
];
// Build volume mounts (validate project_dir stays within ~/.ironclaw/projects/)
let mut binds = Vec::new();
if let Some(ref dir) = project_dir {
let canonical =
dir.canonicalize()
.map_err(|e| OrchestratorError::ContainerCreationFailed {
job_id,
reason: format!(
"failed to canonicalize project dir {}: {}",
dir.display(),
e
),
})?;
let projects_base = dirs::home_dir()
.unwrap_or_else(|| PathBuf::from("."))
.join(".ironclaw")
.join("projects");
if let Ok(canonical_base) = projects_base.canonicalize() {
if !canonical.starts_with(&canonical_base) {
return Err(OrchestratorError::ContainerCreationFailed {
job_id,
reason: format!(
"project directory {} is outside allowed base {}",
canonical.display(),
canonical_base.display()
),
});
}
}
binds.push(format!("{}:/workspace:rw", canonical.display()));
env_vec.push("IRONCLAW_WORKSPACE=/workspace".to_string());
}
// Claude Code mode: mount host ~/.claude read-only for auth,
// and pass the tool allowlist so the bridge can write settings.json.
if mode == JobMode::ClaudeCode {
if let Some(ref claude_dir) = self.config.claude_config_dir {
binds.push(format!("{}:/home/sandbox/.claude:ro", claude_dir.display()));
}
if !self.config.claude_code_allowed_tools.is_empty() {
env_vec.push(format!(
"CLAUDE_CODE_ALLOWED_TOOLS={}",
self.config.claude_code_allowed_tools.join(",")
));
}
}
// Memory limit: Claude Code gets more memory
let memory_mb = match mode {
JobMode::ClaudeCode => self.config.claude_code_memory_limit_mb,
JobMode::Worker => self.config.memory_limit_mb,
};
// Create the container
use bollard::container::{Config, CreateContainerOptions};
use bollard::models::HostConfig;
let host_config = HostConfig {
binds: if binds.is_empty() { None } else { Some(binds) },
memory: Some((memory_mb * 1024 * 1024) as i64),
cpu_shares: Some(self.config.cpu_shares as i64),
network_mode: Some("bridge".to_string()),
extra_hosts: Some(vec!["host.docker.internal:host-gateway".to_string()]),
cap_drop: Some(vec!["ALL".to_string()]),
cap_add: Some(vec!["CHOWN".to_string()]),
security_opt: Some(vec!["no-new-privileges:true".to_string()]),
tmpfs: Some(
[("/tmp".to_string(), "size=512M".to_string())]
.into_iter()
.collect(),
),
..Default::default()
};
// Build CMD based on mode
let cmd = match mode {
JobMode::Worker => vec![
"worker".to_string(),
"--job-id".to_string(),
job_id.to_string(),
"--orchestrator-url".to_string(),
orchestrator_url,
],
JobMode::ClaudeCode => vec![
"claude-bridge".to_string(),
"--job-id".to_string(),
job_id.to_string(),
"--orchestrator-url".to_string(),
orchestrator_url,
"--max-turns".to_string(),
self.config.claude_code_max_turns.to_string(),
"--model".to_string(),
self.config.claude_code_model.clone(),
],
};
let container_config = Config {
image: Some(self.config.image.clone()),
cmd: Some(cmd),
env: Some(env_vec),
host_config: Some(host_config),
user: Some("1000:1000".to_string()),
working_dir: Some("/workspace".to_string()),
..Default::default()
};
let container_name = match mode {
JobMode::Worker => format!("ironclaw-worker-{}", job_id),
JobMode::ClaudeCode => format!("ironclaw-claude-{}", job_id),
};
let options = CreateContainerOptions {
name: container_name,
..Default::default()
};
let response = docker
.create_container(Some(options), container_config)
.await
.map_err(|e| OrchestratorError::ContainerCreationFailed {
job_id,
reason: e.to_string(),
})?;
let container_id = response.id;
// Start the container
docker
.start_container::<String>(&container_id, None)
.await
.map_err(|e| OrchestratorError::ContainerCreationFailed {
job_id,
reason: format!("failed to start container: {}", e),
})?;
// Update handle with container ID
if let Some(handle) = self.containers.write().await.get_mut(&job_id) {
handle.container_id = container_id;
handle.state = ContainerState::Running;
}
tracing::info!(
job_id = %job_id,
"Created and started worker container"
);
Ok(())
}
/// Stop a running container job.
pub async fn stop_job(&self, job_id: Uuid) -> Result<(), OrchestratorError> {
let container_id = {
let containers = self.containers.read().await;
containers
.get(&job_id)
.map(|h| h.container_id.clone())
.ok_or(OrchestratorError::ContainerNotFound { job_id })?
};
if container_id.is_empty() {
return Err(OrchestratorError::InvalidContainerState {
job_id,
state: "creating (no container ID yet)".to_string(),
});
}
let docker = connect_docker()
.await
.map_err(|e| OrchestratorError::Docker {
reason: e.to_string(),
})?;
// Stop the container (10 second grace period)
if let Err(e) = docker
.stop_container(
&container_id,
Some(bollard::container::StopContainerOptions { t: 10 }),
)
.await
{
tracing::warn!(job_id = %job_id, error = %e, "Failed to stop container (may already be stopped)");
}
// Remove the container
if let Err(e) = docker
.remove_container(
&container_id,
Some(bollard::container::RemoveContainerOptions {
force: true,
..Default::default()
}),
)
.await
{
tracing::warn!(job_id = %job_id, error = %e, "Failed to remove container (may require manual cleanup)");
}
// Update state
if let Some(handle) = self.containers.write().await.get_mut(&job_id) {
handle.state = ContainerState::Stopped;
}
// Revoke the auth token
self.token_store.revoke(job_id).await;
tracing::info!(job_id = %job_id, "Stopped worker container");
Ok(())
}
/// Mark a job as complete with a result. The container is stopped but the
/// handle is kept so `CreateJobTool` can read the completion message.
pub async fn complete_job(
&self,
job_id: Uuid,
result: CompletionResult,
) -> Result<(), OrchestratorError> {
// Store the result before stopping
{
let mut containers = self.containers.write().await;
if let Some(handle) = containers.get_mut(&job_id) {
handle.completion_result = Some(result);
handle.state = ContainerState::Stopped;
}
}
// Stop container and revoke token (but keep handle in map)
let container_id = {
let containers = self.containers.read().await;
containers.get(&job_id).map(|h| h.container_id.clone())
};
if let Some(cid) = container_id {
if !cid.is_empty() {
match connect_docker().await {
Ok(docker) => {
if let Err(e) = docker
.stop_container(
&cid,
Some(bollard::container::StopContainerOptions { t: 5 }),
)
.await
{
tracing::warn!(job_id = %job_id, error = %e, "Failed to stop completed container");
}
if let Err(e) = docker
.remove_container(
&cid,
Some(bollard::container::RemoveContainerOptions {
force: true,
..Default::default()
}),
)
.await
{
tracing::warn!(job_id = %job_id, error = %e, "Failed to remove completed container");
}
}
Err(e) => {
tracing::warn!(job_id = %job_id, error = %e, "Failed to connect to Docker for container cleanup");
}
}
}
}
self.token_store.revoke(job_id).await;
tracing::info!(job_id = %job_id, "Completed worker container");
Ok(())
}
/// Remove a completed job handle from memory (called after result is read).
pub async fn cleanup_job(&self, job_id: Uuid) {
self.containers.write().await.remove(&job_id);
}
/// Get the handle for a job.
pub async fn get_handle(&self, job_id: Uuid) -> Option<ContainerHandle> {
self.containers.read().await.get(&job_id).cloned()
}
/// List all active container jobs.
pub async fn list_jobs(&self) -> Vec<ContainerHandle> {
self.containers.read().await.values().cloned().collect()
}
/// Get a reference to the token store.
pub fn token_store(&self) -> &TokenStore {
&self.token_store
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_container_job_config_default() {
let config = ContainerJobConfig::default();
assert_eq!(config.orchestrator_port, 50051);
assert_eq!(config.memory_limit_mb, 2048);
}
#[test]
fn test_container_state_display() {
assert_eq!(ContainerState::Running.to_string(), "running");
assert_eq!(ContainerState::Stopped.to_string(), "stopped");
}
}
+37
View File
@@ -0,0 +1,37 @@
//! Orchestrator for managing sandboxed worker containers.
//!
//! The orchestrator runs in the main agent process and provides:
//! - An internal HTTP API for worker communication (LLM proxy, status, secrets)
//! - Per-job bearer token authentication
//! - Container lifecycle management (create, monitor, stop)
//!
//! ```text
//! ┌───────────────────────────────────────────────┐
//! │ Orchestrator │
//! │ │
//! │ Internal API (:50051) │
//! │ POST /worker/{id}/llm/complete │
//! │ POST /worker/{id}/llm/complete_with_tools │
//! │ GET /worker/{id}/job │
//! │ POST /worker/{id}/status │
//! │ POST /worker/{id}/complete │
//! │ │
//! │ ContainerJobManager │
//! │ create_job() -> container + token │
//! │ stop_job() │
//! │ list_jobs() │
//! │ │
//! │ TokenStore │
//! │ per-job bearer tokens (in-memory only) │
//! └───────────────────────────────────────────────┘
//! ```
pub mod api;
pub mod auth;
pub mod job_manager;
pub use api::OrchestratorApi;
pub use auth::TokenStore;
pub use job_manager::{
CompletionResult, ContainerHandle, ContainerJobConfig, ContainerJobManager, JobMode,
};
+10
View File
@@ -0,0 +1,10 @@
//! DM pairing for channels.
//!
//! Gates DMs from unknown senders. Only approved senders can message the agent.
//! Unknown senders receive a pairing code and must be approved via `ironclaw pairing approve`.
//!
//! OpenClaw reference: src/pairing/pairing-store.ts
mod store;
pub use store::{PairingRequest, PairingStore, PairingStoreError};
+707
View File
@@ -0,0 +1,707 @@
//! Pairing store: pending requests and allowFrom list.
//!
//! Stored in ~/.ironclaw/{channel}-pairing.json and {channel}-allowFrom.json.
use std::collections::HashSet;
use std::fs;
use std::io::{Seek, SeekFrom, Write};
use std::path::{Path, PathBuf};
use std::time::{SystemTime, UNIX_EPOCH};
use fs4::FileExt;
use rand::Rng;
use serde::{Deserialize, Serialize};
const PAIRING_CODE_LENGTH: usize = 8;
const PAIRING_ALPHABET: &[u8] = b"ABCDEFGHJKLMNPQRSTUVWXYZ23456789";
/// TTL for pending pairing requests (minutes, not hours — reduces brute-force window).
const PAIRING_PENDING_TTL_SECS: u64 = 15 * 60;
const PAIRING_PENDING_MAX: usize = 3;
/// Max failed approve attempts per channel before rate limit kicks in.
const PAIRING_APPROVE_RATE_LIMIT: usize = 10;
/// Time window for rate limit (seconds).
const PAIRING_APPROVE_RATE_WINDOW_SECS: u64 = 5 * 60;
/// Error from pairing store operations.
#[derive(Debug, thiserror::Error)]
pub enum PairingStoreError {
#[error("Invalid channel: {0}")]
InvalidChannel(String),
#[error("IO error: {0}")]
Io(#[from] std::io::Error),
#[error("JSON error: {0}")]
Json(#[from] serde_json::Error),
#[error("Rate limit: too many failed approve attempts; try again later")]
ApproveRateLimited,
}
/// Result of upserting a pairing request.
#[derive(Debug)]
pub struct UpsertResult {
pub code: String,
pub created: bool,
}
/// A pending pairing request.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct PairingRequest {
pub id: String,
pub code: String,
pub created_at: String,
pub last_seen_at: String,
#[serde(skip_serializing_if = "Option::is_none")]
pub meta: Option<serde_json::Value>,
}
#[derive(Debug, Serialize, Deserialize)]
struct PairingStoreFile {
version: u8,
requests: Vec<PairingRequest>,
}
#[derive(Debug, Serialize, Deserialize)]
struct AllowFromStoreFile {
version: u8,
#[serde(rename = "allowFrom")]
allow_from: Vec<String>,
}
fn default_pairing_dir() -> PathBuf {
dirs::home_dir()
.unwrap_or_else(|| PathBuf::from("."))
.join(".ironclaw")
}
fn safe_channel_key(channel: &str) -> Result<String, PairingStoreError> {
let raw = channel.trim().to_lowercase();
if raw.is_empty() {
return Err(PairingStoreError::InvalidChannel("empty".to_string()));
}
let safe = raw
.chars()
.map(|c| match c {
'\\' | '/' | ':' | '*' | '?' | '"' | '<' | '>' | '|' => '_',
_ => c,
})
.collect::<String>()
.replace("..", "_");
if safe.is_empty() || safe == "_" {
return Err(PairingStoreError::InvalidChannel(channel.to_string()));
}
Ok(safe)
}
fn pairing_path(base_dir: &Path, channel: &str) -> Result<PathBuf, PairingStoreError> {
let key = safe_channel_key(channel)?;
Ok(base_dir.join(format!("{}-pairing.json", key)))
}
fn allow_from_path(base_dir: &Path, channel: &str) -> Result<PathBuf, PairingStoreError> {
let key = safe_channel_key(channel)?;
Ok(base_dir.join(format!("{}-allowFrom.json", key)))
}
fn approve_attempts_path(base_dir: &Path, channel: &str) -> Result<PathBuf, PairingStoreError> {
let key = safe_channel_key(channel)?;
Ok(base_dir.join(format!("{}-approve-attempts.json", key)))
}
#[derive(Debug, Default, Serialize, Deserialize)]
struct ApproveAttemptsFile {
failed_at: Vec<u64>,
}
fn now_iso() -> String {
let now = SystemTime::now()
.duration_since(UNIX_EPOCH)
.unwrap_or_default();
#[allow(clippy::cast_possible_wrap)]
chrono::DateTime::from_timestamp(now.as_secs() as i64, 0)
.map(|dt| dt.to_rfc3339())
.unwrap_or_else(|| now.as_secs().to_string())
}
fn now_secs() -> u64 {
SystemTime::now()
.duration_since(UNIX_EPOCH)
.map(|d| d.as_secs())
.unwrap_or(0)
}
fn parse_timestamp(value: &str) -> Option<u64> {
chrono::DateTime::parse_from_rfc3339(value)
.ok()
.map(|dt| dt.timestamp() as u64)
.or_else(|| value.parse::<u64>().ok())
}
fn is_expired(req: &PairingRequest, now_secs: u64) -> bool {
let created = parse_timestamp(&req.created_at).unwrap_or(0);
now_secs.saturating_sub(created) > PAIRING_PENDING_TTL_SECS
}
fn random_code() -> String {
let mut rng = rand::thread_rng();
(0..PAIRING_CODE_LENGTH)
.map(|_| {
let idx = rng.gen_range(0..PAIRING_ALPHABET.len());
PAIRING_ALPHABET[idx] as char
})
.collect()
}
fn generate_unique_code(existing: &HashSet<String>) -> String {
let mut rng = rand::thread_rng();
for _ in 0..500 {
let code = random_code();
if !existing.contains(&code) {
return code;
}
}
// Fallback: add suffix
format!("{}{:04}", random_code(), rng.gen_range(0..10000))
}
/// Pairing store for a channel.
#[derive(Debug, Clone)]
pub struct PairingStore {
base_dir: PathBuf,
}
impl PairingStore {
/// Create a new pairing store using default directory (~/.ironclaw).
pub fn new() -> Self {
Self {
base_dir: default_pairing_dir(),
}
}
/// Create a pairing store with a custom base directory (for testing).
pub fn with_base_dir(base_dir: PathBuf) -> Self {
Self { base_dir }
}
/// List pending pairing requests for a channel.
pub fn list_pending(&self, channel: &str) -> Result<Vec<PairingRequest>, PairingStoreError> {
let path = pairing_path(&self.base_dir, channel)?;
let content = match fs::read_to_string(&path) {
Ok(c) => c,
Err(e) if e.kind() == std::io::ErrorKind::NotFound => {
return Ok(Vec::new());
}
Err(e) => return Err(e.into()),
};
let file: PairingStoreFile = serde_json::from_str(&content).unwrap_or(PairingStoreFile {
version: 1,
requests: Vec::new(),
});
let now = now_secs();
let original_len = file.requests.len();
let mut requests: Vec<_> = file
.requests
.into_iter()
.filter(|r| !is_expired(r, now))
.collect();
if requests.len() != original_len {
self.write_pairing_file(channel, &requests)?;
}
requests.sort_by(|a, b| a.created_at.cmp(&b.created_at));
Ok(requests)
}
/// Upsert a pairing request. Returns (code, created).
pub fn upsert_request(
&self,
channel: &str,
id: &str,
meta: Option<serde_json::Value>,
) -> Result<UpsertResult, PairingStoreError> {
let path = pairing_path(&self.base_dir, channel)?;
fs::create_dir_all(path.parent().unwrap())?;
let mut file = fs::OpenOptions::new()
.read(true)
.write(true)
.create(true)
.truncate(false)
.open(&path)?;
file.lock_exclusive()?;
let content = fs::read_to_string(&path).unwrap_or_default();
let mut store: PairingStoreFile =
serde_json::from_str(&content).unwrap_or(PairingStoreFile {
version: 1,
requests: Vec::new(),
});
let now = now_iso();
let now_secs = now_secs();
let id = id.trim().to_string();
if id.is_empty() {
fs4::FileExt::unlock(&file)?;
return Err(PairingStoreError::InvalidChannel("empty id".to_string()));
}
store.requests.retain(|r| !is_expired(r, now_secs));
let existing_codes: HashSet<String> = store
.requests
.iter()
.map(|r| r.code.to_uppercase())
.collect();
if let Some(idx) = store.requests.iter().position(|r| r.id == id) {
let req = &mut store.requests[idx];
let code = if req.code.is_empty() {
generate_unique_code(&existing_codes)
} else {
req.code.clone()
};
req.last_seen_at = now.clone();
req.code = code.clone();
if let Some(m) = meta {
req.meta = Some(m);
}
self.write_pairing_file_locked(&mut file, channel, &store.requests)?;
fs4::FileExt::unlock(&file)?;
return Ok(UpsertResult {
code,
created: false,
});
}
if store.requests.len() >= PAIRING_PENDING_MAX {
fs4::FileExt::unlock(&file)?;
return Ok(UpsertResult {
code: String::new(),
created: false,
});
}
let code = generate_unique_code(&existing_codes);
store.requests.push(PairingRequest {
id: id.clone(),
code: code.clone(),
created_at: now.clone(),
last_seen_at: now,
meta,
});
self.write_pairing_file_locked(&mut file, channel, &store.requests)?;
fs4::FileExt::unlock(&file)?;
Ok(UpsertResult {
code,
created: true,
})
}
fn is_approve_rate_limited(&self, channel: &str) -> Result<bool, PairingStoreError> {
let path = approve_attempts_path(&self.base_dir, channel)?;
let content = match fs::read_to_string(&path) {
Ok(c) => c,
Err(e) if e.kind() == std::io::ErrorKind::NotFound => return Ok(false),
Err(e) => return Err(e.into()),
};
let mut data: ApproveAttemptsFile = serde_json::from_str(&content).unwrap_or_default();
let now = now_secs();
let cutoff = now.saturating_sub(PAIRING_APPROVE_RATE_WINDOW_SECS);
data.failed_at.retain(|&t| t >= cutoff);
Ok(data.failed_at.len() >= PAIRING_APPROVE_RATE_LIMIT)
}
fn record_failed_approve(&self, channel: &str) -> Result<(), PairingStoreError> {
let path = approve_attempts_path(&self.base_dir, channel)?;
fs::create_dir_all(path.parent().unwrap())?;
// Open (or create) and lock before reading so concurrent callers
// don't clobber each other's writes.
let file = fs::OpenOptions::new()
.read(true)
.write(true)
.create(true)
.truncate(false)
.open(&path)?;
file.lock_exclusive()?;
let mut data: ApproveAttemptsFile = fs::read_to_string(&path)
.ok()
.and_then(|c| serde_json::from_str(&c).ok())
.unwrap_or_default();
let now = now_secs();
data.failed_at.push(now);
let cutoff = now.saturating_sub(PAIRING_APPROVE_RATE_WINDOW_SECS);
data.failed_at.retain(|&t| t >= cutoff);
let json = serde_json::to_string_pretty(&data)?;
fs::write(&path, json)?;
fs4::FileExt::unlock(&file)?;
Ok(())
}
/// Approve a pairing code and add the sender to allowFrom.
pub fn approve(
&self,
channel: &str,
code: &str,
) -> Result<Option<PairingRequest>, PairingStoreError> {
let code = code.trim().to_uppercase();
if code.is_empty() {
return Ok(None);
}
if self.is_approve_rate_limited(channel)? {
return Err(PairingStoreError::ApproveRateLimited);
}
let path = pairing_path(&self.base_dir, channel)?;
let mut file = fs::OpenOptions::new()
.read(true)
.write(true)
.create(false)
.open(&path)
.map_err(|e| {
if e.kind() == std::io::ErrorKind::NotFound {
PairingStoreError::InvalidChannel("no pairing file".to_string())
} else {
PairingStoreError::Io(e)
}
})?;
file.lock_exclusive()?;
let content = fs::read_to_string(&path).unwrap_or_default();
let mut store: PairingStoreFile =
serde_json::from_str(&content).unwrap_or(PairingStoreFile {
version: 1,
requests: Vec::new(),
});
let now_secs = now_secs();
store.requests.retain(|r| !is_expired(r, now_secs));
let idx = store
.requests
.iter()
.position(|r| r.code.to_uppercase() == code);
let entry = match idx {
Some(i) => store.requests.remove(i),
None => {
fs4::FileExt::unlock(&file)?;
self.record_failed_approve(channel)?;
return Ok(None);
}
};
self.write_pairing_file_locked(&mut file, channel, &store.requests)?;
fs4::FileExt::unlock(&file)?;
self.add_allow_from(channel, &entry.id)?;
Ok(Some(entry))
}
/// Read the allowFrom list for a channel.
pub fn read_allow_from(&self, channel: &str) -> Result<Vec<String>, PairingStoreError> {
let path = allow_from_path(&self.base_dir, channel)?;
let content = match fs::read_to_string(&path) {
Ok(c) => c,
Err(e) if e.kind() == std::io::ErrorKind::NotFound => {
return Ok(Vec::new());
}
Err(e) => return Err(e.into()),
};
let file: AllowFromStoreFile =
serde_json::from_str(&content).unwrap_or(AllowFromStoreFile {
version: 1,
allow_from: Vec::new(),
});
Ok(file.allow_from)
}
/// Check if a sender is allowed (by id or username).
pub fn is_sender_allowed(
&self,
channel: &str,
id: &str,
username: Option<&str>,
) -> Result<bool, PairingStoreError> {
let allow = self.read_allow_from(channel)?;
let id = id.trim();
let id_ok = allow.iter().any(|e| e.trim() == id);
if id_ok {
return Ok(true);
}
if let Some(u) = username {
let u = u.trim().to_lowercase();
let u_norm = u.strip_prefix('@').unwrap_or(&u);
if allow.iter().any(|e| {
e.trim().to_lowercase() == u || e.trim().to_lowercase() == format!("@{}", u_norm)
}) {
return Ok(true);
}
}
Ok(false)
}
fn add_allow_from(&self, channel: &str, entry: &str) -> Result<(), PairingStoreError> {
let entry = entry.trim().to_string();
if entry.is_empty() {
return Ok(());
}
let path = allow_from_path(&self.base_dir, channel)?;
fs::create_dir_all(path.parent().unwrap())?;
let file = fs::OpenOptions::new()
.read(true)
.write(true)
.create(true)
.truncate(true)
.open(&path)?;
file.lock_exclusive()?;
let content = fs::read_to_string(&path).unwrap_or_default();
let mut store: AllowFromStoreFile =
serde_json::from_str(&content).unwrap_or(AllowFromStoreFile {
version: 1,
allow_from: Vec::new(),
});
let normalized = entry.to_lowercase();
if store
.allow_from
.iter()
.any(|e| e.to_lowercase() == normalized)
{
fs4::FileExt::unlock(&file)?;
return Ok(());
}
store.allow_from.push(entry);
let json = serde_json::to_string_pretty(&store)?;
fs::write(&path, json)?;
fs4::FileExt::unlock(&file)?;
Ok(())
}
fn write_pairing_file(
&self,
channel: &str,
requests: &[PairingRequest],
) -> Result<(), PairingStoreError> {
let path = pairing_path(&self.base_dir, channel)?;
let mut file = fs::OpenOptions::new()
.write(true)
.create(true)
.truncate(true)
.open(&path)?;
file.lock_exclusive()?;
self.write_pairing_file_locked(&mut file, channel, requests)?;
fs4::FileExt::unlock(&file)?;
Ok(())
}
fn write_pairing_file_locked(
&self,
file: &mut fs::File,
_channel: &str,
requests: &[PairingRequest],
) -> Result<(), PairingStoreError> {
let store = PairingStoreFile {
version: 1,
requests: requests.to_vec(),
};
let json = serde_json::to_string_pretty(&store)?;
file.set_len(0)?;
file.seek(SeekFrom::Start(0))?;
file.write_all(json.as_bytes())?;
file.sync_all()?;
Ok(())
}
}
impl Default for PairingStore {
fn default() -> Self {
Self::new()
}
}
#[cfg(test)]
mod tests {
use super::*;
use tempfile::TempDir;
#[test]
fn test_safe_channel_key() {
assert_eq!(safe_channel_key("telegram").unwrap(), "telegram");
assert_eq!(safe_channel_key("Telegram").unwrap(), "telegram");
safe_channel_key("").unwrap_err();
}
#[test]
fn test_random_code() {
let c = random_code();
assert_eq!(c.len(), PAIRING_CODE_LENGTH);
assert!(c.chars().all(|c| PAIRING_ALPHABET.contains(&(c as u8))));
}
fn test_store() -> (PairingStore, TempDir) {
let dir = TempDir::new().unwrap();
let store = PairingStore::with_base_dir(dir.path().to_path_buf());
(store, dir)
}
#[test]
fn test_list_pending_empty() {
let (store, _) = test_store();
let requests = store.list_pending("telegram").unwrap();
assert!(requests.is_empty());
}
#[test]
fn test_upsert_request_creates_new() {
let (store, _) = test_store();
let result = store
.upsert_request(
"telegram",
"user123",
Some(serde_json::json!({"chat_id": 456})),
)
.unwrap();
assert!(result.created);
assert_eq!(result.code.len(), PAIRING_CODE_LENGTH);
assert!(
result
.code
.chars()
.all(|c| PAIRING_ALPHABET.contains(&(c as u8)))
);
}
#[test]
fn test_upsert_request_updates_existing() {
let (store, _) = test_store();
let r1 = store.upsert_request("telegram", "user123", None).unwrap();
assert!(r1.created);
let r2 = store
.upsert_request("telegram", "user123", Some(serde_json::json!({"x": 1})))
.unwrap();
assert!(!r2.created);
assert_eq!(r1.code, r2.code);
let pending = store.list_pending("telegram").unwrap();
assert_eq!(pending.len(), 1);
assert_eq!(pending[0].id, "user123");
assert_eq!(pending[0].meta, Some(serde_json::json!({"x": 1})));
}
#[test]
fn test_approve_adds_to_allow_from() {
let (store, _) = test_store();
let r = store.upsert_request("telegram", "user456", None).unwrap();
assert!(r.created);
let approved = store.approve("telegram", &r.code).unwrap();
assert!(approved.is_some());
assert_eq!(approved.unwrap().id, "user456");
let allow = store.read_allow_from("telegram").unwrap();
assert_eq!(allow, vec!["user456"]);
}
#[test]
fn test_approve_case_insensitive_code() {
let (store, _) = test_store();
let r = store.upsert_request("telegram", "user789", None).unwrap();
let code_lower = r.code.to_lowercase();
let approved = store.approve("telegram", &code_lower).unwrap();
assert!(approved.is_some());
}
#[test]
fn test_approve_invalid_code_returns_none() {
let (store, _) = test_store();
store.upsert_request("telegram", "user123", None).unwrap();
let approved = store.approve("telegram", "BADCODE1").unwrap();
assert!(approved.is_none());
}
#[test]
fn test_approve_rate_limited_after_many_failures() {
let (store, _) = test_store();
store.upsert_request("telegram", "user123", None).unwrap();
for _ in 0..PAIRING_APPROVE_RATE_LIMIT {
let _ = store.approve("telegram", "WRONG01");
}
let err = store.approve("telegram", "WRONG02").unwrap_err();
assert!(matches!(err, PairingStoreError::ApproveRateLimited));
}
#[test]
fn test_is_sender_allowed_by_id() {
let (store, _) = test_store();
let r = store.upsert_request("telegram", "user999", None).unwrap();
store.approve("telegram", &r.code).unwrap();
assert!(
store
.is_sender_allowed("telegram", "user999", None)
.unwrap()
);
assert!(!store.is_sender_allowed("telegram", "other", None).unwrap());
}
#[test]
fn test_is_sender_allowed_by_username() {
let (store, _) = test_store();
store
.upsert_request(
"telegram",
"alice",
Some(serde_json::json!({"username": "alice"})),
)
.unwrap();
let pending = store.list_pending("telegram").unwrap();
store.approve("telegram", &pending[0].code).unwrap();
// approve adds id to allow_from. For username we need to add it manually.
// Actually approve adds entry.id which is "alice". So is_sender_allowed("telegram", "alice", None) would work.
assert!(store.is_sender_allowed("telegram", "alice", None).unwrap());
assert!(
store
.is_sender_allowed("telegram", "alice", Some("alice"))
.unwrap()
);
}
#[test]
fn test_channel_normalization() {
let (store, _) = test_store();
store.upsert_request("Telegram", "u1", None).unwrap();
let pending = store.list_pending("telegram").unwrap();
assert_eq!(pending.len(), 1);
assert_eq!(pending[0].id, "u1");
}
#[test]
fn test_invalid_channel_rejected() {
let (store, _) = test_store();
store.upsert_request("telegram", "u1", None).unwrap();
store.list_pending("").unwrap_err();
store.upsert_request("", "u1", None).unwrap_err();
}
}
+17 -5
View File
@@ -306,12 +306,11 @@ impl LeakDetector {
})?;
}
// Scan body if present and valid UTF-8
// Scan body if present. Use lossy UTF-8 conversion so a leading
// non-UTF8 byte can't be used to skip scanning entirely.
if let Some(body_bytes) = body {
if let Ok(body_str) = std::str::from_utf8(body_bytes) {
self.scan_and_clean(body_str)?;
}
// Binary bodies are not scanned (could add hex pattern detection later)
let body_str = String::from_utf8_lossy(body_bytes);
self.scan_and_clean(&body_str)?;
}
Ok(())
@@ -705,4 +704,17 @@ mod tests {
let result = detector.scan_http_request("https://api.example.com/webhook", &[], Some(body));
assert!(result.is_err());
}
#[test]
fn test_scan_http_request_blocks_secret_in_binary_body() {
let detector = LeakDetector::new();
// Attacker prepends a non-UTF8 byte to bypass strict from_utf8 check.
// The lossy conversion should still detect the secret.
let mut body = vec![0xFF]; // invalid UTF-8 leading byte
body.extend_from_slice(b"sk-proj-test1234567890abcdefghij");
let result = detector.scan_http_request("https://api.example.com/exfil", &[], Some(&body));
assert!(result.is_err(), "binary body should still be scanned");
}
}
+21 -5
View File
@@ -98,15 +98,15 @@ impl SafetyLayer {
was_modified: true,
};
}
if violations
let force_sanitize = violations
.iter()
.any(|rule| rule.action == crate::safety::PolicyAction::Sanitize)
{
.any(|rule| rule.action == crate::safety::PolicyAction::Sanitize);
if force_sanitize {
was_modified = true;
}
// Run sanitization if enabled
if self.config.injection_check_enabled {
// Run sanitization once: if injection_check is enabled OR policy requires it
if self.config.injection_check_enabled || force_sanitize {
let mut sanitized = self.sanitizer.sanitize(&content);
sanitized.was_modified = sanitized.was_modified || was_modified;
sanitized
@@ -190,4 +190,20 @@ mod tests {
assert!(wrapped.contains("sanitized=\"true\""));
assert!(wrapped.contains("Hello &lt;world&gt;"));
}
#[test]
fn test_sanitize_action_forces_sanitization_when_injection_check_disabled() {
let config = SafetyConfig {
max_output_length: 100_000,
injection_check_enabled: false,
};
let safety = SafetyLayer::new(&config);
// Content with an injection-like pattern that a policy might flag
let output = safety.sanitize_tool_output("test", "normal text");
// With injection_check disabled and no policy violations, content
// should pass through unmodified
assert_eq!(output.content, "normal text");
assert!(!output.was_modified);
}
}
+30 -7
View File
@@ -279,11 +279,7 @@ impl ContainerRunner {
network_mode: Some("bridge".to_string()),
// Security: drop all capabilities and add back only what's needed
cap_drop: Some(vec!["ALL".to_string()]),
cap_add: Some(vec![
"CHOWN".to_string(),
"SETUID".to_string(),
"SETGID".to_string(),
]),
cap_add: Some(vec!["CHOWN".to_string()]),
// Prevent privilege escalation
security_opt: Some(vec!["no-new-privileges:true".to_string()]),
// Read-only root filesystem (workspace is still writable if policy allows)
@@ -491,9 +487,36 @@ impl ContainerRunner {
}
/// Connect to the Docker daemon.
///
/// Tries these locations in order:
/// 1. `DOCKER_HOST` env var (bollard default)
/// 2. `/var/run/docker.sock` (Linux default)
/// 3. `~/.docker/run/docker.sock` (Docker Desktop on macOS)
pub async fn connect_docker() -> Result<Docker> {
Docker::connect_with_local_defaults().map_err(|e| SandboxError::DockerNotAvailable {
reason: e.to_string(),
// First try bollard defaults (checks DOCKER_HOST, then /var/run/docker.sock)
if let Ok(docker) = Docker::connect_with_local_defaults() {
if docker.ping().await.is_ok() {
return Ok(docker);
}
}
// Try Docker Desktop socket (macOS)
if let Some(home) = std::env::var_os("HOME") {
let desktop_sock = std::path::Path::new(&home).join(".docker/run/docker.sock");
if desktop_sock.exists() {
let sock_str = desktop_sock.to_string_lossy();
if let Ok(docker) =
Docker::connect_with_socket(&sock_str, 120, bollard::API_DEFAULT_VERSION)
{
if docker.ping().await.is_ok() {
return Ok(docker);
}
}
}
}
Err(SandboxError::DockerNotAvailable {
reason: "Socket not found: /var/run/docker.sock".to_string(),
})
}

Some files were not shown because too many files have changed in this diff Show More