Compare commits

..
Author SHA1 Message Date
github-actions[bot]GitHubgithub-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
5e44185e48 chore: release v0.3.0 (#117)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-02-17 05:39:12 +00:00
72623c9e5b feat: direct api key and cheap model (#116)
* feat: Support direct API key auth and cheap model routing

Allow using IronClaw with any OpenAI-compatible API provider (e.g.
Anthropic Claude) via API key, without requiring NEAR AI session auth.

Changes:
- Skip session authentication in chat_completions mode (API key auth)
- Skip first-run onboard check when NEARAI_API_KEY is configured
- Add `cheap_model` config field (NEARAI_CHEAP_MODEL env var) for a
  secondary lightweight model used for heartbeat, routing, evaluation
- Add `create_cheap_llm_provider()` factory in llm module
- Add `cheap_llm` to AgentDeps with fallback to main model
- Route heartbeat through cheap model to reduce costs
- Fix wizard compilation for new config field

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: address PR #20 review feedback

- Check API key presence (not api_mode) for auth skip (ilblackdragon)
- Add Settings::load() call in check_onboard_needed (ilblackdragon)
- Warn and ignore cheap_model for non-NearAi backends (ilblackdragon)
- Add unit tests for create_cheap_llm_provider (ilblackdragon)
- Minor formatting cleanup in cheap provider match arm

Co-Authored-By: Claude Opus 4.6 <[email protected]>

---------

Co-authored-by: Samuel Barbosa <[email protected]>
Co-authored-by: Claude Opus 4.6 <[email protected]>
2026-02-17 01:24:27 +00:00
github-actions[bot]GitHubgithub-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
6895adbcc9 chore: release v0.2.0 (#60)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-02-16 22:28:14 +00:00
Vlad Frolov f1480f471b ci: Explicitly enable cargo-dist caching for binary artifacts building 2026-02-16 21:34:49 +01:00
Vlad Frolov 9db949746f ci: Skip building binary artifacts on every PR 2026-02-16 21:29:03 +01:00
61a123a746 Add GitHub tool and Discord channel (#34)
* Add GitHub tool for IronClaw - manage repos, issues, PRs, and workflows

* Add Discord channel for IronClaw - slash commands and button interactions

* Security fixes: URL encoding, secret validation, Discord button handler

- Add URL encoding for all path segments and query parameters (P1)
- Add path segment validation to prevent path traversal
- Add secret_exists check for better error messages (P2)
- Fix http_request signature to use 5 args (P2)
- Fix Discord button handler to check member field (P2)
- Fix typo in Discord slash command format (P2)
- Add github.capabilities.json and discord.capabilities.json (Blocker)
- Add Cargo.toml for Discord channel (Blocker)
- Add limit caps (max 100) for all list operations (P3)
- Remove debug logging

* Apply Copilot review fixes

Security & Code Quality:
- Use secret_get instead of workspace_read for GitHub token
- Remove manual Authorization header (host injects via capabilities)
- Add validation for file paths (reject path traversal)
- Add validation for workflow_id and git refs
- Fix url_encode_query comment
- Add release profile optimizations to Cargo.toml files
- Fix package names to match conventions (github-tool, discord-channel)
- Add metadata fields to Cargo.toml
- Fix rate limits to be consistent (60/min, 3600/hr)
- Fix Discord user_name to filter empty global_name
- Fix Discord metadata serialization error handling
- Update Discord README to clarify which secrets are used by host vs WASM
- Better formatting for Discord command option values

* applied all PR change requests and comments

* cleaned up workspace

* Adding validation for empty path segments and event enum in GitHub tool

* addedvalidation for events and vaidation to reject empty file path in github tools and implemented safe UTF-8 trunacating

* added codegen units and updated truncating logic also update capabilities.json as requested by copilot review

* added codegen units and updated truncating logic also update capabilities.json as requested by copilot review

* fixed message trucating and remove url_encode alias, also appled all requested changes from last PR comment

---------

Co-authored-by: root <root@cafx>
Co-authored-by: Peni <[email protected]>
Co-authored-by: Illia Polosukhin <[email protected]>
Co-authored-by: firat.sertgoz <[email protected]>
2026-02-16 15:58:13 +04:00
0e981429ee feat: mark Ollama + OpenAI-compatible as implemented (#102)
Co-authored-by: BroccoliFin <[email protected]>
2026-02-16 03:38:06 +00:00
Illia PolosukhinandClaude Opus 4.6 1b38a64e15 docs: add module specification rules to CLAUDE.md
Any agent working on a module with a README.md spec must read it first,
keep code and spec in sync, and treat the spec as the tiebreaker when
they disagree.

Co-Authored-By: Claude Opus 4.6 <[email protected]>
2026-02-15 00:33:41 -08:00
Illia PolosukhinandClaude Opus 4.6 2e5f8b60d5 docs: add setup/onboarding specification (src/setup/README.md)
Authoritative specification for the 7-step onboarding wizard. Documents
the full flow, settings persistence (two-layer architecture), platform
caveats (macOS keychain dialogs, URL passwords), secrets context, and
a modification checklist for future contributors.

Co-Authored-By: Claude Opus 4.6 <[email protected]>
2026-02-15 00:29:47 -08:00
f0a0642e7d feat: multi-provider inference + libSQL onboarding selection (#92)
* feat: add interactive database backend selection during onboarding

Previously the onboarding wizard silently defaulted to PostgreSQL because
libsql wasn't in the default feature set. Now both backends ship by default
and the wizard presents a selection prompt when both are available.

DATABASE_BACKEND env var still bypasses the prompt for headless/CI use.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: resolve libSQL onboarding crash, keychain double-prompt, and setup audit findings

Three bugs fixed:

1. libSQL onboarding crash ("Missing required setting 'database_url'"):
   DatabaseConfig::resolve() only checked DATABASE_BACKEND env var, falling
   back to Postgres default. Now reads settings.database_backend, plus
   settings.libsql_path and settings.libsql_url as fallbacks.

2. OS keychain prompts twice during startup: Config::from_env() and
   Config::from_db() both called get_master_key(). Now caches the key in
   SECRETS_MASTER_KEY env var after first read so from_db() skips keychain.

3. "Path not found: nearai.session" warning: from_db_map() tried to apply
   app-specific DB keys (nearai.session_token) to the Settings struct.
   Now skips keys that don't map to known Settings fields. Also fixed
   bootstrap migration key mismatch (nearai.session -> nearai.session_token).

Setup module audit fixes (14 findings):
- Replace unreachable!() with proper error in provider match
- Extract setup_api_key_provider() to deduplicate setup_anthropic/setup_openai
- Add SAFETY comments to all unsafe std::env::set_var blocks
- Fix .unwrap() calls with proper error handling
- Remove incorrect #[allow(dead_code)] on used TelegramUpdate::update_id
- Log warnings instead of silently discarding HTTP errors in Telegram binding
- Guard select_many against empty options, fix mask_api_key for non-ASCII
- Update stale doc comment in mod.rs, rename misleading variable
- Add 7 new tests (model fetcher fallbacks, channel discovery, secret gen)

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: address PR review feedback (set_var safety, parse warnings, db_map efficiency)

1. Replace unsafe set_var keychain caching with OnceLock<String> in
   SecretsConfig::resolve(). Eliminates the env var write from main.rs
   entirely, using a process-wide OnceLock cache instead.

2. Log tracing::warn when database_backend or llm_backend settings
   fail to parse, instead of silently falling back to defaults.

3. Remove O(K*S) get() pre-check in from_db_map(). Instead, let set()
   run and match on "Path not found" errors to skip unknown keys,
   avoiding full Settings serialization per key.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: address critical/high audit findings across WASM sub-crates

- Telegram: remove .unwrap() panic on workspace_read (owner_id check)
- WhatsApp: use configured api_version instead of hardcoded v18.0
- WhatsApp: log config parse errors before falling back to defaults
- Slack: log serialization errors in emit_message and json_response
- Google Docs: safe array access for batch update replies
- Google Sheets: safe array access for add_sheet replies
- Google Calendar: fix doc comment secret name mismatch
- Gmail: avoid unnecessary String allocation in UNREAD check

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: address second-round PR review feedback

- Validate custom model ID is non-empty (loop until valid input)
- Warn on unknown DATABASE_BACKEND env var before defaulting to Postgres
- Force re-selection when llm_backend contains unknown provider value
- Use ok_or_else for proper String error type in google-sheets

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: harden setup module error handling and secret safety

- Introduce ChannelSetupError typed enum replacing raw String errors
  across all channel setup functions (setup_telegram, setup_http,
  setup_tunnel, setup_wasm_channel, validate_telegram_token)
- Add From<ChannelSetupError> for SetupError to simplify call sites
- Convert setup_telegram retry from recursion to loop (unbounded stack)
- Stop printing HTTP webhook secret plaintext to terminal
- Use secret_input() for Turso auth token (was visible input())
- Replace dirs::home_dir().unwrap_or_default() with proper error
- Fix UTF-8 panic in model name truncation (byte-index to chars-based)
- Log warning in secret_exists() instead of silently swallowing errors
- Deduplicate generate_webhook_secret() to delegate to shared helper

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: replace unreachable!() with error return in setup wizard

The provider match in step_inference_provider was guarded by
is_known but used unreachable!() as the catch-all. If a new
provider is added to the is_known check without a corresponding
match arm, this would panic at runtime. Return a typed error
instead.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: remove unsafe set_var, use thread-safe overlay for injected secrets

Address PR #92 review comments:
- Replace all 5 unsafe `std::env::set_var()` calls with safe alternatives
- Add INJECTED_VARS OnceLock<HashMap> overlay in config.rs, checked by
  optional_env() before falling back to std::env::var()
- Cache wizard API key in SetupWizard.llm_api_key field instead of env
- Pass explicit key param to fetch_anthropic_models/fetch_openai_models
- Persist env-provided API keys to secrets store during onboarding

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: address remaining PR review comments (clippy, TODO, secrets backend ordering)

- Fix empty line after doc comment (clippy: empty_line_after_doc_comments)
- Collapse nested if in optional_env overlay check (clippy: collapsible_if)
- Remove dangling TODO(#XX) placeholder issue ref in channels.rs
- Fix init_secrets_context to respect selected database_backend when both
  postgres and libsql features are compiled, preventing wrong-backend
  secrets storage when DATABASE_URL is set but libsql was chosen

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: address latest PR review comments (SecretString, empty env, docs, embeddings)

- Change wizard llm_api_key from String to SecretString to prevent
  accidental logging of API keys
- Fix inject_llm_keys_from_secrets skipping when env var is set but
  empty, matching optional_env's treatment of empty as unset
- Fix inverted doc comment on INJECTED_VARS (env checked first, overlay
  is the fallback, not the other way around)
- Update stale "env vars" comments in main.rs to reflect overlay pattern
- Fix step_embeddings not seeing cached OpenAI key from wizard session

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: OAuth callback listener binds IPv4 first to match redirect URLs

The listener was binding to [::1] (IPv6) first, but NEAR AI and other
OAuth flows redirect to http://127.0.0.1:9876/... (IPv4 explicit).
On macOS and most systems, [::1] and 127.0.0.1 are separate addresses,
so the browser's connection to 127.0.0.1 was refused when the listener
was on [::1]. Reversed the bind order: try 127.0.0.1 first, fall back
to [::1] if IPv4 is unavailable.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: cache keychain key eagerly to avoid redundant macOS password dialogs

Replace has_master_key() with get_master_key() in step_security() and
immediately build SecretsCrypto from the result. This eliminates redundant
keychain accesses later in init_secrets_context(), each of which triggers
macOS system dialogs (keychain unlock + app authorization).

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: persist DATABASE_BACKEND to ~/.ironclaw/.env for libSQL startup

The wizard saved database_backend only to the database, but
Config::from_env() needs it BEFORE connecting to any database (to
decide which backend to use). Without it, the backend defaults to
Postgres and then fails with "Missing required setting database_url".

Now save all database bootstrap vars (DATABASE_BACKEND, DATABASE_URL,
LIBSQL_PATH, LIBSQL_URL) to ~/.ironclaw/.env via save_bootstrap_env().

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: status command shows libSQL backend and skips keychain probe

The status command only checked DATABASE_URL (postgres), showing
"not configured" for libSQL users. Now detects the DATABASE_BACKEND
env var and reports libSQL path and Turso sync status.

Also remove the keychain probe from status. get_generic_password()
triggers macOS unlock+authorization dialogs which is terrible UX
for a read-only diagnostic command.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* style: fix rustfmt formatting in bootstrap test

Co-Authored-By: Claude Opus 4.6 <[email protected]>

---------

Co-authored-by: Claude Opus 4.6 <[email protected]>
2026-02-15 08:24:51 +00:00
ca8d5c6b5e refactor: deduplicate tool code and remove dead stubs (#98)
* refactor: deduplicate tool parameter extraction and remove dead stub tools

Delete 4 never-registered stub tools (marketplace, restaurant, ecommerce,
taskrabbit) removing ~625 lines of dead code. Add require_str/require_param
helpers to tool.rs and refactor ~30 call sites across 10 tool files from
4-6 line inline extractions to single-line calls. Consolidate worker HTTP
client with get_json/post_json helpers, reducing boilerplate in 4 methods.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: return JSON from orchestrator /complete endpoint

The report_complete handler returned bare StatusCode::OK (no body),
which broke the post_json helper that expects a JSON response.
Return {"status": "ok"} for consistency with other worker endpoints.

Addresses review feedback on PR #98.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

---------

Co-authored-by: Claude Opus 4.6 <[email protected]>
2026-02-15 05:39:52 +00:00
9fed8453c7 fix: shell destructive-command check bypassed by Value::Object arguments (#72)
Co-authored-by: Yi LIU <[email protected]>
Co-authored-by: Illia Polosukhin <[email protected]>
2026-02-14 21:54:13 +00:00
eaef335db6 fix: propagate real tool_call_id instead of hardcoded placeholder (#73)
The worker (both agent/worker.rs and worker/runtime.rs) was passing the
literal string "tool_call_id" to ChatMessage::tool_result instead of
the actual tool call ID from the LLM response. This breaks
OpenAI-compatible providers that match tool results to their
corresponding calls by ID.

- Add tool_call_id field to ToolSelection struct
- Propagate ToolCall.id through select_tools() into ToolSelection
- Replace all hardcoded "tool_call_id" usages with selection.tool_call_id
- Generate unique IDs for plan-based synthetic selections
- Add test verifying tool_call_id is preserved

Co-authored-by: Yi LIU <[email protected]>
Co-authored-by: Illia Polosukhin <[email protected]>
2026-02-14 21:39:25 +00:00
Eric WinerandGitHub 225af29db2 Reformat architecture diagram in README (#64) 2026-02-14 21:22:58 +00:00
a53b2c10b5 fix: Fix wasm tool schemas and runtime (#42)
* feat: Move debug log truncation from agent loop to REPL channel

Full tool output now flows through StatusUpdate so the web gateway
gets untruncated content. The REPL channel truncates at display time
(200 chars for tool results, thinking, and status messages).

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: Flatten WASM tool schemas and fix host HTTP runtime contention

LLMs can't reliably follow oneOf + const discriminator patterns in JSON
Schema, causing tools like Google Calendar to receive malformed params
(e.g., {"operation":"list_events","data":{"calendarId":"primary"}} instead
of {"action":"list_events","calendar_id":"primary"}). Replace all 9 WASM
tool schemas with flat action enum + top-level properties. The serde
#[serde(tag = "action")] deserialization works identically.

Also fixes WASM host HTTP requests (channels and tools) stalling during
startup by replacing Handle::current().block_on() with a dedicated
single-threaded runtime per request, avoiding I/O driver contention.

Reduces verbose LLM debug logging (full request/response payloads) and
changes tower_http default from debug to warn.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* feat: Built-in OAuth credentials and combined Google scopes

Add infrastructure for shipping default OAuth credentials with the binary,
similar to how gcloud/rclone bake in their client_id. Credentials are set
at compile time via IRONCLAW_GOOGLE_CLIENT_ID / IRONCLAW_GOOGLE_CLIENT_SECRET
env vars, or can be hardcoded in src/cli/oauth_defaults.rs.

The fallback chain is: capabilities file > runtime env var > built-in defaults.

Also, when authing any Google tool, scopes from ALL installed Google tools
are now combined into a single OAuth request (they all share the same
google_oauth_token secret). One login covers Gmail, Calendar, Drive, etc.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* feat: Ship default Google OAuth credentials for zero-config auth

Google Desktop App credentials are not secret (per Google's own docs).
Hardcode them so `ironclaw tool auth <google-tool>` works out of the box
without requiring users to register their own OAuth app.

Credentials can still be overridden at compile time
(IRONCLAW_GOOGLE_CLIENT_ID) or runtime (GOOGLE_OAUTH_CLIENT_ID).

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: Consistent OAuth callback port and polished landing page

- Use fixed port 9876 instead of scanning 9876-9886 (one redirect URI
  to register in provider OAuth apps, deterministic behavior)
- Replace broken unicode checkmark with SVG icons (charset was missing,
  rendered as mojibake)
- Dark themed landing page with proper card layout for both success
  and error states
- Add charset=utf-8 to Content-Type headers

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* refactor: Unify OAuth callback server across all auth flows

All three OAuth flows (WASM tool auth, MCP server auth, NEAR AI login)
now share the same code from cli::oauth_defaults:

- Fixed port 9876 (one redirect URI to register per provider)
- Shared landing page HTML (dark card with SVG icons, proper charset)
- Parameterized wait_for_callback(listener, path, param, display_name)

Removes ~120 lines of duplicated callback/HTML code.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* Support for oauth token refresh

* refactor: Replace bootstrap.json with ~/.ironclaw/.env for DATABASE_URL

Kill the 4-field BootstrapConfig JSON file. Only DATABASE_URL actually
needs disk persistence (chicken-and-egg before DB connect). The other
three fields are now derived: pool_size defaults to 10 via env var,
secrets master key is auto-detected (env then keychain probe), and
onboard_completed is inferred from DATABASE_URL presence.

The new format is a standard .env file loaded via dotenvy early in
main, so DATABASE_URL is available as a regular env var everywhere.

Handles three upgrade paths:
- Clean start: wizard writes .env, reload after wizard completes
- Returning user: .env loaded at startup, business as usual
- Legacy upgrade: bootstrap.json auto-migrated to .env on first run

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: Address PR review findings

- Fix UTF-8 panic in truncate_for_preview (byte-slice on char boundary)
- Cap WASM guest timeout_ms at 5 minutes to prevent resource exhaustion
- Fix localhost detection in requires_auth() to avoid substring matches
  (e.g. "notlocalhost.com" no longer matches)
- Fix query param injection to insert before URL fragment
- Fix extract_host_from_url for IPv6 bracket notation
- Remove misleading schema defaults: Slack limit, Slides insertion_index,
  Docs index (per-action defaults documented in descriptions instead)

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* style: Fix cargo fmt formatting

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: IPv6 loopback support for OAuth listener and localhost detection

- bind_callback_listener: try [::1] first, fall back to 127.0.0.1,
  so OAuth redirects work on systems where localhost resolves to ::1
- is_localhost_url: replace manual string parsing with url::Url for
  correct handling of IPv6 brackets, ports, userinfo, etc.
- Add url crate as direct dependency (already a transitive dep)

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: Address PR review feedback on runtime reuse, onboard check, and OAuth binding

- Remove session file check from check_onboard_needed(); DATABASE_URL is sufficient
- Detect AddrInUse on IPv6 bind and fail immediately instead of falling through to IPv4
- Reuse dedicated tokio runtime across HTTP calls in both tool and channel WASM wrappers

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: HTML-escape provider name in OAuth landing page, simplify Slack limit description

- Add html_escape() to prevent XSS in landing_html() where provider_name
  was interpolated directly into HTML (defense-in-depth, source is trusted
  but escaping costs nothing)
- Remove per-action default numbers from Slack limit field description to
  avoid confusing LLMs with conflicting defaults

Addresses review feedback from zmanian on PR #42.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: Save all bootstrap fields from wizard, fix config module comment

- Wizard now saves secrets_master_key_source and database_pool_size to
  bootstrap.json (was only saving database_url and onboard_completed,
  which broke secrets after fresh onboard since SecretsConfig::resolve
  reads key source from bootstrap)
- Update config.rs module doc to reflect bootstrap.json priority chain
  instead of the removed ~/.ironclaw/.env approach

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* refactor: Replace BootstrapConfig with .env-based bootstrap

DATABASE_URL is the only setting that needs disk persistence before
the database is available. Instead of a custom bootstrap.json with 4
fields, use a standard ~/.ironclaw/.env file loaded via dotenvy.

- Remove BootstrapConfig struct entirely
- Restore ironclaw_env_path(), load_ironclaw_env(), save_database_url()
- SecretsConfig::resolve() now auto-detects (env var then keychain probe)
  instead of reading a saved source from bootstrap.json
- DatabaseConfig::resolve() reads DATABASE_URL from env only (dotenvy
  loads ~/.ironclaw/.env into the environment early in startup)
- check_onboard_needed() is now sync (just checks env vars)
- Wizard save_and_summarize() works for both postgres and libsql backends
- One-time migration from bootstrap.json to .env preserved

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: Ensure load_ironclaw_env() runs in all Config paths, fix .env priority

- Config::from_env() and Config::from_db() now call load_ironclaw_env()
  internally (after dotenvy::dotenv()), so CLI commands like `memory`
  and `config` correctly load DATABASE_URL from ~/.ironclaw/.env
- Fix load order: standard ./.env first (higher priority), then
  ~/.ironclaw/.env, matching the documented priority chain
- Collapse nested if/if-let into let-chains (clippy::collapsible_if)
  in oauth_defaults.rs, tool.rs, and secrets/store.rs
- Fix rename_to_migrated to take &Path instead of &PathBuf

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: Address PR review comments (quoting, SSRF, error mapping)

- Quote DATABASE_URL in .env writes so `#` in passwords isn't treated
  as a dotenv comment (e.g., `DATABASE_URL="postgres://..."`)
- Add SSRF defenses to refresh_oauth_token(): require HTTPS, reject
  private/loopback IPs (with DNS resolution), disable redirects.
  token_url comes from tool capabilities JSON, so a malicious tool
  could otherwise exfiltrate refresh tokens.
- Fix IPv4 bind error mapping: only map AddrInUse to PortInUse,
  use generic Io variant for other bind failures

Co-Authored-By: Claude Opus 4.6 <[email protected]>

---------

Co-authored-by: Claude Opus 4.6 <[email protected]>
2026-02-14 21:21:22 +00:00
408ae8a29a feat: add multi-provider LLM failover with retry backoff (#28)
* feat: add multi-provider LLM failover

Add FailoverProvider that wraps multiple LlmProvider instances and
tries each in sequence on transient failures. Non-retryable errors
(auth, context length, model not available) propagate immediately.

- New `FailoverProvider` with generic `try_providers` helper
- `is_retryable()` classifies transient errors (request failed,
  rate limited, invalid response, session renewal, HTTP, IO)
- Configurable via `NEARAI_FALLBACK_MODEL` env var
- Returns `Result` from constructor (no panics in production)
- Updates FEATURE_PARITY.md: failover chains , cooldown 

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: track last-used provider for accurate cost/model reporting

After failover, model_name() and cost_per_token() now reflect the
provider that actually handled the request, not always the primary.
Also corrects is_retryable() docs to list ModelNotAvailable as retryable.

Addresses PR #28 review comments.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* feat: add retry with exponential backoff for LLM providers

Add retry logic with exponential backoff and jitter to both NearAiProvider
and NearAiChatProvider for transient errors (HTTP 429, 500, 502, 503, 504).

Extract shared retry helpers (is_retryable_status, retry_backoff_delay)
into src/llm/retry.rs so both providers reuse the same logic.

Configurable via NEARAI_MAX_RETRIES env var (default: 3).

* docs: clarify max_retries means N retries, not N total attempts

* warn when fallback model equals primary model

* fix: saturating_mul in backoff delay, dedupe to_lowercase allocation

---------

Co-authored-by: Claude Opus 4.6 <[email protected]>
2026-02-14 15:43:38 +04:00
Zaki ManianGitHubClaude Opus 4.6Illia Polosukhingemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
d9ff86d7e0 docs: Add review discipline guidelines to CLAUDE.md (#68)
* docs: Add review discipline guidelines to CLAUDE.md

Codifies lessons learned from Illia's review fixes on the libSQL
backend PR -- patterns we missed that should be caught systematically
going forward.

- Ban .expect() alongside .unwrap() in production code
- Add mechanical grep checks before committing
- New "Review & Fix Discipline" section covering:
  - Fix all instances of a pattern, not just the one flagged
  - Propagate architectural changes to satellite types
  - Schema translation must include indexes and seed data
  - Feature flag testing with each feature in isolation

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* Apply suggestions from code review

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

---------

Co-authored-by: Claude Opus 4.6 <[email protected]>
Co-authored-by: Illia Polosukhin <[email protected]>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2026-02-14 04:25:53 +00:00
e843c18141 feat: add libSQL/Turso embedded database backend (#47)
* feat: add libSQL/Turso database backend with full feature parity

Introduce a Database trait abstraction (~60 async methods) enabling
compile-time backend selection between PostgreSQL and libSQL/Turso.
Convert all modules from concrete Store to Arc<dyn Database>, add
LibSqlSecretsStore and LibSqlWasmToolStore implementations, wire
libsql stores throughout CLI and main entry points, and make the
setup wizard backend-agnostic.

Key changes:
- src/db/: Database trait, PostgresDatabase adapter, LibSqlBackend
  with native SQLite-dialect SQL, and idempotent migration system
- src/secrets/store.rs: LibSqlSecretsStore (all 8 trait methods)
- src/tools/wasm/storage.rs: LibSqlWasmToolStore (all 7 trait methods)
- src/main.rs, cli/tool.rs, cli/mcp.rs: backend-conditional wiring
- src/setup/channels.rs: SecretsContext uses Arc<dyn SecretsStore>
- Feature-gate postgres-only tests and examples

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* feat: enable onboarding wizard for libSQL builds

Refactor the setup wizard to work with both postgres and libsql feature
flags. Previously the wizard was gated behind #[cfg(feature = "postgres")]
only, so libsql-only builds would print an error on `ironclaw onboard`.

- Add libsql fields to Settings (database_backend, libsql_path, libsql_url)
- Split wizard database/migration/secrets methods into feature-gated variants
- Add step_database_libsql() with local path and Turso remote replica prompts
- Update setup/mod.rs and main.rs feature gates to any(postgres, libsql)
- Extend check_onboard_needed() to detect libsql database presence

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: address PR review feedback for libSQL backend

- P0: Switch libsql_backend to connection-per-operation pattern to fix
  shared Connection concurrency issue across tokio tasks
- P0: Wrap secrets store INSERT+SELECT in transaction to fix TOCTOU race
- P0: Document encryption-at-rest limitations and json_patch divergence
- P1: Fix get_opt_text removing .filter(|s| !s.is_empty()) that conflated
  empty strings with NULL
- P1: Replace datetime('now') with fmt_ts(&Utc::now()) for consistent
  RFC 3339 timestamps across all queries
- P2: Use explicit _rowid column in FTS5 triggers and joins for stability
  across VACUUM operations
- P2: Add tracing::warn when embedding provided but vector search disabled
  in hybrid_search
- Extract shared connect_from_config() helper to deduplicate DB connection
  logic across main.rs, cli/config.rs, and cli/mcp.rs

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: add missing JobContext fields and resolve fmt/clippy warnings

Add total_tokens_used and max_tokens fields to JobContext in
libsql_backend.rs, apply cargo fmt, and fix clippy warnings.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: review fixes for libSQL backend (shared connections, panics, indexes)

- Replace .expect() with proper error propagation in 3 call sites
- Share Arc<Database> between backend and stores instead of single Connection
- Add connect-per-operation pattern to LibSqlSecretsStore and LibSqlWasmToolStore
- Wrap store() INSERT + SELECT-back in a transaction
- Add ~22 missing indexes for parity with PostgreSQL schema
- Add 18 leak_detection_patterns seed rows matching PostgreSQL V2 migration
- Fix super:: import to use crate:: style
- Gate mask_password_in_url behind #[cfg(feature = "postgres")]
- Rewrite secrets store init with or_else chain for runtime backend selection

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: Resolve clippy lints (collapsible_if, too_many_arguments)

Collapse nested if blocks into let_chains to satisfy clippy's
collapsible_if lint (CI uses -D warnings). Suppress too_many_arguments
on libsql_row_to_tool_at since refactoring the positional index
pattern would be a larger change.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

---------

Co-authored-by: Claude Opus 4.6 <[email protected]>
Co-authored-by: Illia Polosukhin <[email protected]>
2026-02-14 02:05:05 +00:00
54e9206f0b feat: Move debug log truncation from agent loop to REPL channel (#65)
* feat: Move debug log truncation from agent loop to REPL channel

Full tool output now flows through StatusUpdate so the web gateway
gets untruncated content. The REPL channel truncates at display time
(200 chars for tool results, thinking, and status messages).

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* feat: truncating fmt layer for terminal, full logs for web gateway

Instead of truncating debug output at each LLM call site (fragile),
use a custom MakeWriter on the fmt layer that caps each tracing event
at 500 bytes before flushing to stderr. The web gateway WebLogLayer
still receives full untruncated content for /api/logs/events SSE.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: UTF-8 safe truncation in truncate_for_preview, remove double truncation

- Use char_indices() instead of byte-based slicing to find the cut
  point, preventing panics on multi-byte characters (emoji, CJK, etc.)
- Remove redundant truncation in REPL channel (agent loop already
  truncates ToolResult previews to 200 chars)
- Add 9 unit tests covering edge cases: empty, exact length, multi-byte
  UTF-8 (emoji, CJK), mixed scripts, newline collapsing, whitespace

Addresses PR #65 review comments.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

---------

Co-authored-by: Claude Opus 4.6 <[email protected]>
2026-02-13 23:24:07 +00:00
5df0d13b59 Bump MSRV to 1.92, add GCP deployment files (#40)
* Bump MSRV to 1.92 and add GCP deployment files

rig-core 0.30 uses let_chains (stabilized post-1.87), which breaks
builds on Rust 1.85. Bump rust-version in Cargo.toml and both
Dockerfiles to 1.92 (verified working).

Add cloud deployment scaffolding:
- Dockerfile: multi-stage build for the main agent container
- deploy/cloud-sql-proxy.service: systemd unit for Cloud SQL Auth Proxy
- deploy/ironclaw.service: systemd unit for the IronClaw container
- deploy/setup.sh: VM bootstrap script (Docker, proxy, services)
- deploy/env.example: reference environment configuration

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* Address review feedback: harden deploy scaffolding

- Add comment explaining GATEWAY_HOST=0.0.0.0 and when to use 127.0.0.1
- Document /opt/ironclaw ownership model (root-owned, Docker reads as root)
- Switch cloud-sql-proxy service from User=root to DynamicUser=yes

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: Resolve clippy lints (Rust 1.93) and fix CI test workflow

- Fix 97 collapsible_if warnings using let-chains syntax (auto-fixed)
- Fix ptr_arg: change &PathBuf to &Path in pairing store functions
- Fix suspicious_open_options: add .truncate(false) to OpenOptions
- Fix too_many_arguments: add clippy allow on execute_status
- Fix unnecessary_unwrap: use if-let in repository.rs hybrid_search
- Gate unused EchoTool with #[cfg(test)]
- Add PairingStore argument to ChannelStoreData::new() test call sites
- Add skip guard for bundled channel test when WASM artifacts unavailable
- Split CI test workflow to exclude PostgreSQL-dependent integration tests

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: Address review feedback from ilblackdragon

- Add root check to setup.sh (exits with error if not root)
- Add warning comment to env.example about placeholder passwords
- Dockerfile.worker already uses rust:1.92 (no change needed)
- PR #41 overlap noted; will rebase after #41 merges

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: resolve 47 collapsible_if clippy warnings

Collapse nested if statements across the codebase to satisfy
clippy::collapsible_if on Rust 1.93.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

---------

Co-authored-by: Claude Opus 4.6 <[email protected]>
2026-02-13 22:21:50 +04:00
121 changed files with 9284 additions and 4559 deletions
-1
View File
@@ -39,7 +39,6 @@ permissions:
# If there's a prerelease-style suffix to the version, then the release(s)
# will be marked as a prerelease.
on:
pull_request:
push:
tags:
- '**[0-9]+.[0-9]+.[0-9]+*'
+2
View File
@@ -1,6 +1,8 @@
.env
.env.local
.env.*
!.env.example
target/
+36
View File
@@ -7,6 +7,42 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
## [0.3.0](https://github.com/nearai/ironclaw/compare/v0.2.0...v0.3.0) - 2026-02-17
### Added
- direct api key and cheap model ([#116](https://github.com/nearai/ironclaw/pull/116))
## [0.2.0](https://github.com/nearai/ironclaw/compare/v0.1.3...v0.2.0) - 2026-02-16
### Added
- mark Ollama + OpenAI-compatible as implemented ([#102](https://github.com/nearai/ironclaw/pull/102))
- multi-provider inference + libSQL onboarding selection ([#92](https://github.com/nearai/ironclaw/pull/92))
- add multi-provider LLM failover with retry backoff ([#28](https://github.com/nearai/ironclaw/pull/28))
- add libSQL/Turso embedded database backend ([#47](https://github.com/nearai/ironclaw/pull/47))
- Move debug log truncation from agent loop to REPL channel ([#65](https://github.com/nearai/ironclaw/pull/65))
### Fixed
- shell destructive-command check bypassed by Value::Object arguments ([#72](https://github.com/nearai/ironclaw/pull/72))
- propagate real tool_call_id instead of hardcoded placeholder ([#73](https://github.com/nearai/ironclaw/pull/73))
- Fix wasm tool schemas and runtime ([#42](https://github.com/nearai/ironclaw/pull/42))
- flatten tool messages for NEAR AI cloud-api compatibility ([#41](https://github.com/nearai/ironclaw/pull/41))
- security hardening across all layers ([#35](https://github.com/nearai/ironclaw/pull/35))
### Other
- Explicitly enable cargo-dist caching for binary artifacts building
- Skip building binary artifacts on every PR
- add module specification rules to CLAUDE.md
- add setup/onboarding specification (src/setup/README.md)
- deduplicate tool code and remove dead stubs ([#98](https://github.com/nearai/ironclaw/pull/98))
- Reformat architecture diagram in README ([#64](https://github.com/nearai/ironclaw/pull/64))
- Add review discipline guidelines to CLAUDE.md ([#68](https://github.com/nearai/ironclaw/pull/68))
- Bump MSRV to 1.92, add GCP deployment files ([#40](https://github.com/nearai/ironclaw/pull/40))
- Add OpenAI-compatible HTTP API (/v1/chat/completions, /v1/models) ([#31](https://github.com/nearai/ironclaw/pull/31))
## [0.1.3](https://github.com/nearai/ironclaw/compare/v0.1.2...v0.1.3) - 2026-02-12
### Other
+49 -1
View File
@@ -198,8 +198,9 @@ When designing new features or systems, always prefer generic/extensible archite
### Error Handling
- Use `thiserror` for error types in `error.rs`
- Never use `.unwrap()` in production code (tests are fine)
- Never use `.unwrap()` or `.expect()` in production code (tests are fine)
- Map errors with context: `.map_err(|e| SomeError::Variant { reason: e.to_string() })?`
- Before committing, grep for `.unwrap()` and `.expect(` in changed files to catch violations mechanically
### Async
- All I/O is async with tokio
@@ -629,6 +630,22 @@ RUST_LOG=ironclaw::agent=debug cargo run
RUST_LOG=ironclaw=debug,tower_http=debug cargo run
```
## Module Specifications
Some modules have a `README.md` that serves as the authoritative specification
for that module's behavior. When modifying code in a module that has a spec:
1. **Read the spec first** before making changes
2. **Code follows spec**: if the spec says X, the code must do X
3. **Update both sides**: if you change behavior, update the spec to match;
if you're implementing a spec change, update the code to match
4. **Spec is the tiebreaker**: when code and spec disagree, the spec is correct
(unless the spec is clearly outdated, in which case fix the spec first)
| Module | Spec File |
|--------|-----------|
| `src/setup/` | `src/setup/README.md` |
## Code Style
- Use `crate::` imports, not `super::`
@@ -637,6 +654,37 @@ RUST_LOG=ironclaw=debug,tower_http=debug cargo run
- Keep functions focused, extract helpers when logic is reused
- Comments for non-obvious logic only
## Review & Fix Discipline
Hard-won lessons from code review -- follow these when fixing bugs or addressing review feedback.
### Fix the pattern, not just the instance
When a reviewer flags a bug (e.g., TOCTOU race in INSERT + SELECT-back), search the entire codebase for all instances of that same pattern. A fix in `SecretsStore::create()` that doesn't also fix `WasmToolStore::store()` is half a fix.
### Propagate architectural fixes to satellite types
If a core type changes its concurrency model (e.g., `LibSqlBackend` switches to connection-per-operation), every type that was handed a resource from the old model (e.g., `LibSqlSecretsStore`, `LibSqlWasmToolStore` holding a single `Connection`) must also be updated. Grep for the old type across the codebase.
### Schema translation is more than DDL
When translating a database schema between backends (PostgreSQL to libSQL, etc.), check for:
- **Indexes** -- diff `CREATE INDEX` statements between the two schemas
- **Seed data** -- check for `INSERT INTO` in migrations (e.g., `leak_detection_patterns`)
- **Semantic differences** -- document where SQL functions behave differently (e.g., `json_patch` vs `jsonb_set`)
### Feature flag testing
When adding feature-gated code, test compilation with each feature in isolation:
```bash
cargo check # default features
cargo check --no-default-features --features libsql # libsql only
cargo check --all-features # all features
```
Dead code behind the wrong `#[cfg]` gate will only show up when building with a single feature.
### Mechanical verification before committing
Run these checks on changed files before committing:
- `grep -rnE '\.unwrap\(|\.expect\(' <files>` -- no panics in production
- `grep -rn 'super::' <files>` -- use `crate::` imports
- If you fixed a pattern bug, `grep` for other instances of that pattern across `src/`
## Workspace & Memory System
Inspired by [OpenClaw](https://github.com/openclaw/openclaw), the workspace provides persistent memory for agents with a flexible filesystem-like structure.
Generated
+4 -12
View File
@@ -2210,11 +2210,11 @@ dependencies = [
"hyper 1.8.1",
"hyper-util",
"rustls",
"rustls-native-certs",
"rustls-pki-types",
"tokio",
"tokio-rustls",
"tower-service",
"webpki-roots",
]
[[package]]
@@ -2490,7 +2490,7 @@ dependencies = [
[[package]]
name = "ironclaw"
version = "0.1.3"
version = "0.3.0"
dependencies = [
"aes-gcm",
"aho-corasick",
@@ -2548,6 +2548,7 @@ dependencies = [
"tower-http 0.6.8",
"tracing",
"tracing-subscriber",
"url",
"urlencoding",
"uuid",
"wasmparser 0.220.1",
@@ -4033,6 +4034,7 @@ dependencies = [
"pin-project-lite",
"quinn",
"rustls",
"rustls-native-certs",
"rustls-pki-types",
"serde",
"serde_json",
@@ -4050,7 +4052,6 @@ dependencies = [
"wasm-bindgen-futures",
"wasm-streams",
"web-sys",
"webpki-roots",
]
[[package]]
@@ -6237,15 +6238,6 @@ dependencies = [
"wasm-bindgen",
]
[[package]]
name = "webpki-roots"
version = "1.0.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "12bed680863276c63889429bfd6cab3b99943659923822de1c8a39c49e4d722c"
dependencies = [
"rustls-pki-types",
]
[[package]]
name = "which"
version = "4.4.2"
+9 -6
View File
@@ -1,8 +1,8 @@
[package]
name = "ironclaw"
version = "0.1.3"
version = "0.3.0"
edition = "2024"
rust-version = "1.85"
rust-version = "1.92"
description = "Secure personal AI assistant that protects your data and expands its capabilities on the fly"
authors = ["NEAR AI <[email protected]>"]
license = "MIT OR Apache-2.0"
@@ -22,7 +22,7 @@ tokio-stream = { version = "0.1", features = ["sync"] }
futures = "0.3"
# HTTP client
reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls", "stream"] }
reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls-native-roots", "stream"] }
# Serialization
serde = { version = "1", features = ["derive"] }
@@ -84,7 +84,8 @@ fs4 = "0.6"
# Secrecy for sensitive values
secrecy = { version = "0.10", features = ["serde"] }
# URL encoding for OAuth flow
# URL parsing and encoding
url = "2"
urlencoding = "2"
# Open URLs in browser
@@ -138,7 +139,7 @@ pretty_assertions = "1"
tempfile = "3"
[features]
default = ["postgres"]
default = ["postgres", "libsql"]
postgres = [
"dep:deadpool-postgres",
"dep:tokio-postgres",
@@ -182,11 +183,13 @@ windows-archive = ".tar.gz"
# The archive format to use for non-windows builds (defaults .tar.xz)
unix-archive = ".tar.gz"
# Which actions to run on pull requests
pr-run-mode = "upload"
pr-run-mode = "skip"
# Path that installers should place binaries in
install-path = "CARGO_HOME"
# Whether to install an updater program
install-updater = true
# Cache intermediate build artifacts to speed up the release pipelines
cache-builds = true
[workspace.metadata.dist.github-custom-runners]
aarch64-unknown-linux-gnu = "ubuntu-24.04-arm"
+46
View File
@@ -0,0 +1,46 @@
# Multi-stage Dockerfile for the IronClaw agent (cloud deployment).
#
# Build:
# docker build --platform linux/amd64 -t ironclaw:latest .
#
# Run:
# docker run --env-file .env -p 3000:3000 ironclaw:latest
# Stage 1: Build
FROM rust:1.92-slim-bookworm AS builder
RUN apt-get update && apt-get install -y --no-install-recommends \
pkg-config libssl-dev cmake gcc g++ \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /app
# Copy manifests first for layer caching
COPY Cargo.toml Cargo.lock ./
# Copy source and build artifacts
COPY src/ src/
COPY migrations/ migrations/
COPY wit/ wit/
RUN cargo build --release --bin ironclaw
# Stage 2: Runtime
FROM debian:bookworm-slim
RUN apt-get update && apt-get install -y --no-install-recommends \
ca-certificates libssl3 \
&& rm -rf /var/lib/apt/lists/*
COPY --from=builder /app/target/release/ironclaw /usr/local/bin/ironclaw
COPY --from=builder /app/migrations /app/migrations
# Non-root user
RUN useradd -m -u 1000 -s /bin/bash ironclaw
USER ironclaw
EXPOSE 3000
ENV RUST_LOG=ironclaw=info
ENTRYPOINT ["ironclaw"]
+2 -2
View File
@@ -9,7 +9,7 @@
# The image includes common development tools so workers can build software,
# run tests, and execute shell commands.
FROM rust:1.85-bookworm AS builder
FROM rust:1.92-bookworm AS builder
WORKDIR /build
COPY . .
@@ -40,7 +40,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
ENV RUSTUP_HOME=/usr/local/rustup \
CARGO_HOME=/usr/local/cargo \
PATH=/usr/local/cargo/bin:$PATH
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --default-toolchain 1.85.0 \
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --default-toolchain 1.92.0 \
&& chmod -R a+r /usr/local/rustup /usr/local/cargo
# Install Claude Code CLI (for claude-bridge mode)
+4 -4
View File
@@ -133,7 +133,7 @@ This document tracks feature parity between IronClaw (Rust implementation) and O
|---------|----------|----------|-------|
| Pi agent runtime | ✅ | | IronClaw uses custom runtime |
| RPC-based execution | ✅ | ✅ | Orchestrator/worker pattern |
| Multi-provider failover | ✅ | | Provider fallback chains |
| Multi-provider failover | ✅ | | `FailoverProvider` tries providers sequentially on retryable errors |
| Per-sender sessions | ✅ | ✅ | |
| Global sessions | ✅ | ❌ | Optional shared context |
| Session pruning | ✅ | ❌ | Auto cleanup old sessions |
@@ -164,7 +164,7 @@ This document tracks feature parity between IronClaw (Rust implementation) and O
| AWS Bedrock | ✅ | ❌ | P3 | |
| Google Gemini | ✅ | ❌ | P3 | |
| OpenRouter | ✅ | ❌ | P3 | |
| Ollama (local) | ✅ | | P2 | Local models |
| Ollama (local) | ✅ | | - | via `rig::providers::ollama` (full support) |
| node-llama-cpp | ✅ | | - | N/A for Rust |
| llama.cpp (native) | ❌ | 🔮 | P3 | Rust bindings |
@@ -173,7 +173,7 @@ This document tracks feature parity between IronClaw (Rust implementation) and O
| Feature | OpenClaw | IronClaw | Notes |
|---------|----------|----------|-------|
| Auto-discovery | ✅ | ❌ | |
| Failover chains | ✅ | | Provider fallback |
| Failover chains | ✅ | | `FailoverProvider` with configurable `fallback_model` |
| Cooldown management | ✅ | ❌ | Skip failed providers |
| Per-session model override | ✅ | ✅ | Model selector in TUI |
| Model selection UI | ✅ | ✅ | TUI keyboard shortcut |
@@ -419,7 +419,7 @@ This document tracks feature parity between IronClaw (Rust implementation) and O
- ❌ Slack channel (real implementation)
- ✅ Telegram channel (WASM, DM pairing, caption, /start)
- ❌ WhatsApp channel
- Multi-provider failover
- Multi-provider failover (`FailoverProvider` with retryable error classification)
- ❌ Hooks system (beforeInbound, beforeToolCall, etc.)
### P2 - Medium Priority
+36 -36
View File
@@ -181,42 +181,42 @@ External content passes through multiple security layers:
## Architecture
```
┌────────────────────────────────────────────────────────────────────
│ Channels
│ ┌──────┐ ┌──────┐ ┌─────────────┐ ┌─────────────┐
│ │ REPL │ │ HTTP │ │WASM Channels│ │ Web Gateway │
│ └──┬───┘ └──┬───┘ └──────┬──────┘ │ (SSE + WS) │
│ │ │ │ └──────┬──────┘
│ └─────────┴──────────────┴────────────────┘
│ │
│ ┌─────────▼─────────┐
│ │ Agent Loop │ Intent routing
│ └────┬─────────────┘
│ │ │
│ ┌──────────▼───┐ ┌──▼──────────────┐
│ │ Scheduler │ │ Routines Engine │
│ │(parallel jobs)│ │(cron, event, wh) │
│ └──────┬───────┘ └────────┬─────────┘
│ │ │
│ ┌─────────────┼───────────────────┘
│ │ │
│ ┌───▼────┐ ┌────▼────────────────┐
│ │ Local │ │ Orchestrator │
│ │Workers │ │ ┌───────────────┐ │
│ │(in-proc)│ │ │ Docker Sandbox│ │
│ └───┬────┘ │ │ Containers │ │
│ │ │ │ ┌───────────┐ │ │
│ │ │ │ │Worker / CC│ │ │
│ │ │ │ └───────────┘ │ │
│ │ │ └───────────────┘ │
│ │ └─────────┬───────────┘
│ └──────────────────┤
│ │
│ ┌───────────▼──────────┐
│ │ Tool Registry │
│ │ Built-in, MCP, WASM │
│ └──────────────────────┘
└────────────────────────────────────────────────────────────────────
┌────────────────────────────────────────────────────────────────┐
│ Channels │
│ ┌──────┐ ┌──────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ REPL │ │ HTTP │ │WASM Channels│ │ Web Gateway │ │
│ └──┬───┘ └──┬───┘ └──────┬──────┘ │ (SSE + WS) │ │
│ │ │ │ └──────┬──────┘ │
│ └─────────┴──────────────┴────────────────┘ │
│ │ │
│ ┌─────────▼─────────┐ │
│ │ Agent Loop │ Intent routing │
│ └────┬─────────────┘ │
│ │ │ │
│ ┌──────────▼───┐ ┌──▼──────────────┐ │
│ │ Scheduler │ │ Routines Engine │ │
│ │(parallel jobs)│ │(cron, event, wh) │ │
│ └──────┬───────┘ └────────┬─────────┘ │
│ │ │ │
│ ┌─────────────┼───────────────────┘ │
│ │ │ │
│ ┌───▼────┐ ┌────▼────────────────┐ │
│ │ Local │ │ Orchestrator │ │
│ │Workers │ │ ┌───────────────┐ │ │
│ │(in-proc)│ │ │ Docker Sandbox│ │ │
│ └───┬────┘ │ │ Containers │ │ │
│ │ │ │ ┌───────────┐ │ │ │
│ │ │ │ │Worker / CC│ │ │ │
│ │ │ │ └───────────┘ │ │ │
│ │ │ └───────────────┘ │ │
│ │ └─────────┬───────────┘ │
│ └──────────────────┤ │
│ │ │
│ ┌───────────▼──────────┐ │
│ │ Tool Registry │ │
│ │ Built-in, MCP, WASM │ │
│ └──────────────────────┘ │
└────────────────────────────────────────────────────────────────┘
```
### Core Components
+23
View File
@@ -0,0 +1,23 @@
[package]
name = "discord-channel"
version = "0.1.0"
edition = "2021"
description = "Discord channel for IronClaw"
license = "MIT OR Apache-2.0"
publish = false
[dependencies]
serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0"
wit-bindgen = "0.41.0"
[lib]
crate-type = ["cdylib"]
[profile.release]
strip = true
opt-level = "s"
lto = true
codegen-units = 1
+121
View File
@@ -0,0 +1,121 @@
# Discord Channel for IronClaw
WASM channel for Discord integration - handle slash commands and button interactions via webhooks.
## Features
- **Slash Commands** - Process Discord slash commands
- **Button Interactions** - Handle button clicks
- **Thread Support** - Respond in threads
- **DM Support** - Handle direct messages
## Setup
1. Create a Discord Application at <https://discord.com/developers/applications>
2. Create a Bot and get the token
3. Set up Interactions URL to point to your IronClaw instance
4. Copy the Application ID and Public Key
5. Store in IronClaw secrets:
```bash
ironclaw secret set discord_bot_token YOUR_BOT_TOKEN
```
**Note:** The `discord_bot_token` secret is the only value read directly by this
Discord channel WASM component. The `discord_app_id` and `discord_public_key`
secrets are used by the IronClaw host (for example, to verify Discord
interaction signatures and manage slash command registration) and are not
accessed from the WASM module itself.
## Discord Configuration
### Register Slash Commands
```bash
curl -X POST \
-H "Authorization: Bot YOUR_BOT_TOKEN" \
-H "Content-Type: application/json" \
https://discord.com/api/v10/applications/YOUR_APP_ID/commands \
-d '{
"name": "ask",
"description": "Ask the AI agent",
"options": [{
"name": "question",
"description": "Your question",
"type": 3,
"required": true
}]
}'
```
### Set Interactions Endpoint
In your Discord app settings, set:
- Interactions Endpoint URL: `https://your-ironclaw.com/webhook/discord`
## Usage Examples
### Slash Command
User types: `/ask question: What is the weather?`
The agent receives:
```text
User: @username
Content: /ask question: What is the weather?
```
### Button Click
When a user clicks a button in a message, the agent receives:
```text
User: @username
Content: [Button clicked] Original message content
```
## Error Handling
If an internal error occurs (e.g., metadata serialization failure), the tool attempts to send an ephemeral message to the user:
```text
❌ Internal Error: Failed to process command metadata.
```
Check the host logs for detailed error information.
## Advanced Usage
### Embeds
To send embeds, include an `embeds` array in the `metadata_json` field of the agent's response. The structure should match the Discord API `embed` object.
## Troubleshooting
### "Invalid Signature"
- Check that `discord_public_key` is set correctly in IronClaw secrets.
- This validation happens on the host before reaching the WASM.
### "401 Unauthorized"
- Check that `discord_bot_token` is set correctly in IronClaw secrets.
- Ensure the bot is added to the server.
### "Interaction Failed"
- The interaction might have timed out (Discord requires a response within 3 seconds).
- The `interactions_endpoint_url` might be unreachable.
## Building
```bash
cd channels-src/discord
cargo build --target wasm32-wasi --release
```
## License
MIT/Apache-2.0
@@ -0,0 +1,39 @@
{
"type": "channel",
"name": "discord",
"description": "Discord Gateway/Webhook channel for handling slash commands, buttons, and messages",
"capabilities": {
"http": {
"allowlist": [
{ "host": "discord.com", "path_prefix": "/api/v10" }
],
"credentials": {
"discord_bot_token": {
"secret_name": "discord_bot_token",
"location": { "type": "header", "header_name": "Authorization", "prefix": "Bot " },
"host_patterns": ["discord.com"]
}
},
"rate_limit": {
"requests_per_minute": 60,
"requests_per_hour": 3600
}
},
"secrets": {
"allowed_names": ["discord_bot_token", "discord_*"]
},
"channel": {
"allowed_paths": ["/webhook/discord"],
"allow_polling": false,
"callback_timeout_secs": 45,
"workspace_prefix": "channels/discord/",
"emit_rate_limit": {
"messages_per_minute": 100,
"messages_per_hour": 5000
}
}
},
"config": {
"require_signature_verification": true
}
}
+476
View File
@@ -0,0 +1,476 @@
//! Discord Gateway/Webhook channel for IronClaw.
//!
//! This WASM component implements the channel interface for handling Discord
//! interactions via webhooks and sending messages back to Discord.
//!
//! # Features
//!
//! - URL verification for Discord interactions
//! - Slash command handling
//! - Message event parsing (@mentions, DMs)
//! - Thread support for conversations
//! - Response posting via Discord Web API
//! - Automatic message truncation (> 2000 chars)
//!
//! # Security
//!
//! - Signature validation is handled by the host (webhook secrets)
//! - Bot token is injected by host during HTTP requests
//! - WASM never sees raw credentials
wit_bindgen::generate!({
world: "sandboxed-channel",
path: "../../wit/channel.wit",
});
use serde::{Deserialize, Serialize};
use exports::near::agent::channel::{
AgentResponse, ChannelConfig, Guest, HttpEndpointConfig, IncomingHttpRequest,
OutgoingHttpResponse, StatusUpdate,
};
use near::agent::channel_host::{self, EmittedMessage};
/// Discord interaction wrapper.
#[derive(Debug, Deserialize)]
struct DiscordInteraction {
/// Interaction type (1=Ping, 2=ApplicationCommand, 3=MessageComponent)
#[serde(rename = "type")]
interaction_type: u8,
/// Interaction ID
id: String,
/// Application ID
application_id: String,
/// Guild ID (if in server)
#[allow(dead_code)] // Part of API payload, currently unused
guild_id: Option<String>,
/// Channel ID
channel_id: Option<String>,
/// Member info (if in server)
member: Option<DiscordMember>,
/// User info (if DM)
user: Option<DiscordUser>,
/// Command data (for slash commands)
data: Option<DiscordCommandData>,
/// Message (for component interactions)
message: Option<DiscordMessage>,
/// Token for responding
token: String,
}
#[derive(Debug, Deserialize, Clone)]
struct DiscordMember {
user: DiscordUser,
#[allow(dead_code)] // Part of API payload, currently unused
nick: Option<String>,
}
#[derive(Debug, Deserialize, Clone)]
struct DiscordUser {
id: String,
username: String,
global_name: Option<String>,
}
#[derive(Debug, Deserialize, Clone)]
struct DiscordCommandData {
#[allow(dead_code)] // Part of API payload, currently unused
id: String,
name: String,
options: Option<Vec<DiscordCommandOption>>,
}
#[derive(Debug, Deserialize, Clone)]
struct DiscordCommandOption {
name: String,
value: serde_json::Value,
}
#[derive(Debug, Deserialize, Clone)]
struct DiscordMessage {
#[allow(dead_code)] // Part of API payload, currently unused
id: String,
content: String,
channel_id: String,
#[allow(dead_code)] // Part of API payload, currently unused
author: DiscordUser,
}
/// Metadata stored with emitted messages for response routing.
#[derive(Debug, Serialize, Deserialize)]
struct DiscordMessageMetadata {
/// Discord channel ID
channel_id: String,
/// Interaction ID for followups
interaction_id: String,
/// Interaction token for responding
token: String,
/// Application ID
application_id: String,
/// Thread ID (for forum threads)
thread_id: Option<String>,
}
struct DiscordChannel;
impl Guest for DiscordChannel {
fn on_start(_config_json: String) -> Result<ChannelConfig, String> {
channel_host::log(channel_host::LogLevel::Info, "Discord channel starting");
Ok(ChannelConfig {
display_name: "Discord".to_string(),
http_endpoints: vec![HttpEndpointConfig {
path: "/webhook/discord".to_string(),
methods: vec!["POST".to_string()],
require_secret: true,
}],
poll: None,
})
}
fn on_http_request(req: IncomingHttpRequest) -> OutgoingHttpResponse {
let body_str = match std::str::from_utf8(&req.body) {
Ok(s) => s,
Err(_) => {
return json_response(400, serde_json::json!({"error": "Invalid UTF-8 body"}));
}
};
let interaction: DiscordInteraction = match serde_json::from_str(body_str) {
Ok(i) => i,
Err(e) => {
channel_host::log(
channel_host::LogLevel::Error,
&format!("Failed to parse Discord interaction: {}", e),
);
return json_response(400, serde_json::json!({"error": "Invalid interaction"}));
}
};
match interaction.interaction_type {
// Ping - Discord verification
1 => {
channel_host::log(channel_host::LogLevel::Info, "Responding to Discord ping");
json_response(200, serde_json::json!({"type": 1}))
}
// Application Command (slash command)
2 => {
handle_slash_command(&interaction);
json_response(
200,
serde_json::json!({
"type": 5,
"data": {
"content": "🤔 Thinking..."
}
}),
)
}
// Message Component (buttons, selects)
3 => {
if let Some(ref message) = interaction.message {
handle_message_component(&interaction, message);
}
json_response(200, serde_json::json!({"type": 6}))
}
_ => {
channel_host::log(
channel_host::LogLevel::Warn,
&format!(
"Unknown Discord interaction type: {}",
interaction.interaction_type
),
);
json_response(200, serde_json::json!({"type": 6}))
}
}
}
fn on_poll() {}
fn on_respond(response: AgentResponse) -> Result<(), String> {
let metadata: DiscordMessageMetadata = serde_json::from_str(&response.metadata_json)
.map_err(|e| format!("Failed to parse metadata: {}", e))?;
// Use webhook endpoint for followup
let url = format!(
"https://discord.com/api/v10/webhooks/{}/{}",
metadata.application_id, metadata.token
);
// Truncate content to 2000 characters to comply with Discord limits
let content = truncate_message(&response.content);
let mut payload = serde_json::json!({
"content": content,
});
// Check for embeds in metadata
if let Ok(meta_json) = serde_json::from_str::<serde_json::Value>(&response.metadata_json) {
if let Some(embeds) = meta_json.get("embeds") {
payload["embeds"] = embeds.clone();
}
}
let payload_bytes =
serde_json::to_vec(&payload).map_err(|e| format!("Failed to serialize: {}", e))?;
let headers = serde_json::json!({
"Content-Type": "application/json"
});
let result = channel_host::http_request(
"POST",
&url,
&headers.to_string(),
Some(&payload_bytes),
None,
);
match result {
Ok(http_response) => {
if http_response.status >= 200 && http_response.status < 300 {
channel_host::log(channel_host::LogLevel::Debug, "Posted followup to Discord");
Ok(())
} else {
let body_str = String::from_utf8_lossy(&http_response.body);
Err(format!(
"Discord API error: {} - {}",
http_response.status, body_str
))
}
}
Err(e) => Err(format!("HTTP request failed: {}", e)),
}
}
fn on_status(_update: StatusUpdate) {}
fn on_shutdown() {
channel_host::log(
channel_host::LogLevel::Info,
"Discord channel shutting down",
);
}
}
fn handle_slash_command(interaction: &DiscordInteraction) {
let user = interaction
.member
.as_ref()
.map(|m| &m.user)
.or(interaction.user.as_ref());
let user_id = user.map(|u| u.id.clone()).unwrap_or_default();
let user_name = user
.map(|u| {
u.global_name
.as_ref()
.filter(|s| !s.is_empty())
.unwrap_or(&u.username)
.clone()
})
.unwrap_or_default();
let channel_id = interaction.channel_id.clone().unwrap_or_default();
let command_name = interaction
.data
.as_ref()
.map(|d| d.name.clone())
.unwrap_or_default();
let options = interaction.data.as_ref().and_then(|d| d.options.clone());
let content = if let Some(opts) = options {
let opt_str = opts
.iter()
.map(|o| format!("{}: {}", o.name, o.value))
.collect::<Vec<_>>()
.join(", ");
format!("/{} {}", command_name, opt_str)
} else {
format!("/{}", command_name)
};
let metadata = DiscordMessageMetadata {
channel_id: channel_id.clone(),
interaction_id: interaction.id.clone(),
token: interaction.token.clone(),
application_id: interaction.application_id.clone(),
thread_id: None,
};
let metadata_json = match serde_json::to_string(&metadata) {
Ok(json) => json,
Err(e) => {
channel_host::log(
channel_host::LogLevel::Error,
&format!("Failed to serialize metadata: {}", e),
);
// Attempt to notify user of internal error
let url = format!(
"https://discord.com/api/v10/webhooks/{}/{}",
interaction.application_id, interaction.token
);
let payload = serde_json::json!({
"content": "❌ Internal Error: Failed to process command metadata.",
"flags": 64 // Ephemeral
});
let _ = channel_host::http_request(
"POST",
&url,
&serde_json::json!({"Content-Type": "application/json"}).to_string(),
Some(&serde_json::to_vec(&payload).unwrap_or_default()),
None,
);
return;
}
};
channel_host::emit_message(&EmittedMessage {
user_id,
user_name: Some(user_name),
content,
thread_id: None,
metadata_json,
});
}
fn handle_message_component(interaction: &DiscordInteraction, message: &DiscordMessage) {
// Check member first (for server contexts), then user (for DMs)
let user = interaction
.member
.as_ref()
.map(|m| &m.user)
.or(interaction.user.as_ref());
let user_id = user.map(|u| u.id.clone()).unwrap_or_default();
let user_name = user
.map(|u| {
u.global_name
.as_ref()
.filter(|s| !s.is_empty())
.unwrap_or(&u.username)
.clone()
})
.unwrap_or_default();
let channel_id = message.channel_id.clone();
let metadata = DiscordMessageMetadata {
channel_id: channel_id.clone(),
interaction_id: interaction.id.clone(),
token: interaction.token.clone(),
application_id: interaction.application_id.clone(),
thread_id: None,
};
let metadata_json = match serde_json::to_string(&metadata) {
Ok(json) => json,
Err(e) => {
channel_host::log(
channel_host::LogLevel::Error,
&format!("Failed to serialize metadata: {}", e),
);
return; // Don't emit message if metadata can't be serialized
}
};
channel_host::emit_message(&EmittedMessage {
user_id,
user_name: Some(user_name),
content: format!("[Button clicked] {}", message.content),
thread_id: None,
metadata_json,
});
}
fn json_response(status: u16, value: serde_json::Value) -> OutgoingHttpResponse {
let body = serde_json::to_vec(&value).unwrap_or_default();
let headers = serde_json::json!({"Content-Type": "application/json"});
OutgoingHttpResponse {
status,
headers_json: headers.to_string(),
body,
}
}
export!(DiscordChannel);
fn truncate_message(content: &str) -> String {
if content.len() <= 2000 {
content.to_string()
} else {
let max_bytes = 1990;
let cutoff = content
.char_indices()
.map(|(i, c)| i + c.len_utf8())
.take_while(|&end| end <= max_bytes)
.last()
.unwrap_or(0);
let mut truncated = content[..cutoff].to_string();
truncated.push_str("\n... (truncated)");
truncated
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_truncate_message() {
let short = "Hello world";
assert_eq!(truncate_message(short), short);
let long = "a".repeat(2005);
let truncated = truncate_message(&long);
assert_eq!(truncated.len(), 2006); // 1990 + 16 chars suffix
assert!(truncated.ends_with("\n... (truncated)"));
// Test with multibyte characters (Euro sign is 3 bytes)
// 1000 chars * 3 bytes = 3000 bytes
let multi = "".repeat(1000);
let truncated_multi = truncate_message(&multi);
// 1990 bytes limit. 1990 / 3 = 663 with remainder 1.
// Should truncate at 663 chars (1989 bytes).
// Suffix is 16 bytes. Total: 1989 + 16 = 2005 bytes.
assert!(truncated_multi.len() <= 2006);
assert!(truncated_multi.len() >= 2006 - 4); // Allow for max utf8 char width variance
assert!(truncated_multi.ends_with("\n... (truncated)"));
let content_part = &truncated_multi[..truncated_multi.len() - 16];
assert!(content_part.chars().all(|c| c == '€'));
}
#[test]
fn test_metadata_serialization() {
let metadata = DiscordMessageMetadata {
channel_id: "123".into(),
interaction_id: "456".into(),
token: "abc".into(),
application_id: "789".into(),
thread_id: None,
};
let json = serde_json::to_string(&metadata).unwrap();
let parsed: DiscordMessageMetadata = serde_json::from_str(&json).unwrap();
assert_eq!(parsed.channel_id, "123");
assert_eq!(parsed.interaction_id, "456");
}
}
+14 -2
View File
@@ -338,7 +338,13 @@ fn emit_message(
team_id,
};
let metadata_json = serde_json::to_string(&metadata).unwrap_or_else(|_| "{}".to_string());
let metadata_json = serde_json::to_string(&metadata).unwrap_or_else(|e| {
channel_host::log(
channel_host::LogLevel::Error,
&format!("Failed to serialize Slack metadata: {}", e),
);
"{}".to_string()
});
// Strip @ mentions of the bot from the text for cleaner messages
let cleaned_text = strip_bot_mention(&text);
@@ -366,7 +372,13 @@ fn strip_bot_mention(text: &str) -> String {
/// Create a JSON HTTP response.
fn json_response(status: u16, value: serde_json::Value) -> OutgoingHttpResponse {
let body = serde_json::to_vec(&value).unwrap_or_default();
let body = serde_json::to_vec(&value).unwrap_or_else(|e| {
channel_host::log(
channel_host::LogLevel::Error,
&format!("Failed to serialize JSON response: {}", e),
);
Vec::new()
});
let headers = serde_json::json!({"Content-Type": "application/json"});
OutgoingHttpResponse {
+9 -19
View File
@@ -285,11 +285,7 @@ impl Guest for TelegramChannel {
}
// Persist dm_policy and allow_from for DM pairing in handle_message
let dm_policy = config
.dm_policy
.as_deref()
.unwrap_or("pairing")
.to_string();
let dm_policy = config.dm_policy.as_deref().unwrap_or("pairing").to_string();
let _ = channel_host::workspace_write(DM_POLICY_PATH, &dm_policy);
let allow_from_json = serde_json::to_string(&config.allow_from.unwrap_or_default())
@@ -844,8 +840,8 @@ fn send_pairing_reply(chat_id: i64, code: &str) -> Result<(), String> {
"parse_mode": "Markdown",
});
let payload_bytes = serde_json::to_vec(&payload)
.map_err(|e| format!("Failed to serialize payload: {}", e))?;
let payload_bytes =
serde_json::to_vec(&payload).map_err(|e| format!("Failed to serialize payload: {}", e))?;
let headers = serde_json::json!({
"Content-Type": "application/json"
@@ -915,15 +911,10 @@ fn handle_message(message: TelegramMessage) {
let is_private = message.chat.chat_type == "private";
// Owner validation: when owner_id is set, only that user can message
let owner_configured = channel_host::workspace_read(OWNER_ID_PATH)
.map(|s| !s.is_empty())
.unwrap_or(false);
let owner_id_str = channel_host::workspace_read(OWNER_ID_PATH).filter(|s| !s.is_empty());
if owner_configured {
if let Ok(owner_id) = channel_host::workspace_read(OWNER_ID_PATH)
.unwrap()
.parse::<i64>()
{
if let Some(ref id_str) = owner_id_str {
if let Ok(owner_id) = id_str.parse::<i64>() {
if from.id != owner_id {
channel_host::log(
channel_host::LogLevel::Debug,
@@ -937,8 +928,8 @@ fn handle_message(message: TelegramMessage) {
}
} else if is_private {
// No owner_id: apply dm_policy for private chats
let dm_policy = channel_host::workspace_read(DM_POLICY_PATH)
.unwrap_or_else(|| "pairing".to_string());
let dm_policy =
channel_host::workspace_read(DM_POLICY_PATH).unwrap_or_else(|| "pairing".to_string());
if dm_policy != "open" {
// Build effective allow list: config allow_from + pairing store
@@ -1001,8 +992,7 @@ fn handle_message(message: TelegramMessage) {
if !respond_to_all {
let has_command = content.starts_with('/');
let bot_username = channel_host::workspace_read(BOT_USERNAME_PATH)
.unwrap_or_default();
let bot_username = channel_host::workspace_read(BOT_USERNAME_PATH).unwrap_or_default();
let has_bot_mention = if bot_username.is_empty() {
content.contains('@')
} else {
+23 -6
View File
@@ -254,10 +254,19 @@ struct WhatsAppChannel;
impl Guest for WhatsAppChannel {
fn on_start(config_json: String) -> Result<ChannelConfig, String> {
let config: WhatsAppConfig = serde_json::from_str(&config_json).unwrap_or(WhatsAppConfig {
api_version: default_api_version(),
reply_to_message: default_reply_to_message(),
});
let config: WhatsAppConfig = match serde_json::from_str(&config_json) {
Ok(c) => c,
Err(e) => {
channel_host::log(
channel_host::LogLevel::Warn,
&format!("Failed to parse WhatsApp config, using defaults: {}", e),
);
WhatsAppConfig {
api_version: default_api_version(),
reply_to_message: default_reply_to_message(),
}
}
};
channel_host::log(
channel_host::LogLevel::Info,
@@ -267,6 +276,9 @@ impl Guest for WhatsAppChannel {
),
);
// Persist api_version in workspace so on_respond() can read it
let _ = channel_host::workspace_write("channels/whatsapp/api_version", &config.api_version);
// WhatsApp Cloud API is webhook-only, no polling available
Ok(ChannelConfig {
display_name: "WhatsApp".to_string(),
@@ -327,11 +339,16 @@ impl Guest for WhatsAppChannel {
let metadata: WhatsAppMessageMetadata = serde_json::from_str(&response.metadata_json)
.map_err(|e| format!("Failed to parse metadata: {}", e))?;
// Read api_version from workspace (set during on_start), fallback to default
let api_version = channel_host::workspace_read("channels/whatsapp/api_version")
.filter(|s| !s.is_empty())
.unwrap_or_else(|| "v18.0".to_string());
// Build WhatsApp API URL with token placeholder
// Host will replace {WHATSAPP_ACCESS_TOKEN} with actual token in Authorization header
let api_url = format!(
"https://graph.facebook.com/v18.0/{}/messages",
metadata.phone_number_id
"https://graph.facebook.com/{}/{}/messages",
api_version, metadata.phone_number_id
);
// Build sendMessage payload
+13
View File
@@ -0,0 +1,13 @@
[Unit]
Description=Cloud SQL Auth Proxy
After=network.target
[Service]
Type=simple
DynamicUser=yes
ExecStart=/usr/local/bin/cloud-sql-proxy ironclaw-prod:us-central1:ironclaw-db --port=5432
Restart=always
RestartSec=5
[Install]
WantedBy=multi-user.target
+27
View File
@@ -0,0 +1,27 @@
# WARNING: Replace all CHANGE_ME values before deploying.
# Do not use placeholder passwords in production.
DATABASE_URL=postgres://ironclaw:CHANGE_ME@localhost:5432/ironclaw
# NEAR AI
NEARAI_SESSION_TOKEN=CHANGE_ME
NEARAI_MODEL=claude-3-5-sonnet-20241022
NEARAI_BASE_URL=https://cloud-api.near.ai
NEARAI_AUTH_URL=https://private.near.ai
NEARAI_API_MODE=chat_completions
# Agent
AGENT_NAME=ironclaw
CLI_ENABLED=false
# Web Gateway
GATEWAY_ENABLED=true
# 0.0.0.0 binds to all interfaces (required for Docker --network=host).
# Use 127.0.0.1 if running outside Docker or for local-only access.
GATEWAY_HOST=0.0.0.0
GATEWAY_PORT=3000
GATEWAY_AUTH_TOKEN=CHANGE_ME
# Disabled for initial deploy
SANDBOX_ENABLED=false
HEARTBEAT_ENABLED=false
EMBEDDING_ENABLED=false
+20
View File
@@ -0,0 +1,20 @@
[Unit]
Description=IronClaw AI Assistant
After=cloud-sql-proxy.service docker.service
Requires=cloud-sql-proxy.service
[Service]
Type=simple
ExecStartPre=/usr/bin/docker pull us-central1-docker.pkg.dev/ironclaw-prod/ironclaw/agent:latest
ExecStart=/usr/bin/docker run --rm \
--name ironclaw \
--env-file /opt/ironclaw/.env \
--network=host \
us-central1-docker.pkg.dev/ironclaw-prod/ironclaw/agent:latest \
--no-onboard
ExecStop=/usr/bin/docker stop ironclaw
Restart=always
RestartSec=10
[Install]
WantedBy=multi-user.target
+68
View File
@@ -0,0 +1,68 @@
#!/usr/bin/env bash
# VM bootstrap script for IronClaw on GCP Compute Engine.
#
# Run on a fresh Debian 12 VM after SSH:
# sudo bash setup.sh
#
# Prerequisites:
# - VM has the ironclaw-vm service account attached
# - Cloud SQL Auth Proxy accessible via IAM
# - Artifact Registry image pushed
set -euo pipefail
# Must run as root
if [ "$(id -u)" -ne 0 ]; then
echo "ERROR: This script must be run as root (sudo bash setup.sh)"
exit 1
fi
echo "==> Installing Docker"
apt-get update
apt-get install -y docker.io
systemctl enable docker
systemctl start docker
echo "==> Installing Cloud SQL Auth Proxy"
curl -fsSL -o /usr/local/bin/cloud-sql-proxy \
https://storage.googleapis.com/cloud-sql-connectors/cloud-sql-proxy/v2.14.3/cloud-sql-proxy.linux.amd64
chmod +x /usr/local/bin/cloud-sql-proxy
echo "==> Installing systemd services"
cp /tmp/deploy/cloud-sql-proxy.service /etc/systemd/system/
cp /tmp/deploy/ironclaw.service /etc/systemd/system/
systemctl daemon-reload
echo "==> Starting Cloud SQL Auth Proxy"
systemctl enable cloud-sql-proxy
systemctl start cloud-sql-proxy
echo "==> Configuring Docker registry auth"
# The VM service account provides Artifact Registry access
gcloud auth configure-docker us-central1-docker.pkg.dev --quiet
echo "==> Creating config directory"
# Owned by root, readable only by root. Docker reads --env-file as root
# before dropping to uid 1000 (ironclaw) inside the container.
mkdir -p /opt/ironclaw
chmod 700 /opt/ironclaw
if [ ! -f /opt/ironclaw/.env ]; then
echo "WARNING: /opt/ironclaw/.env does not exist."
echo "Create it with your configuration before starting IronClaw."
echo "See deploy/env.example for the required variables."
echo ""
echo "Then run: systemctl enable ironclaw && systemctl start ironclaw"
else
chmod 600 /opt/ironclaw/.env
echo "==> Starting IronClaw"
systemctl enable ironclaw
systemctl start ironclaw
fi
echo "==> Setup complete"
echo ""
echo "Verify with:"
echo " systemctl status cloud-sql-proxy"
echo " systemctl status ironclaw"
echo " docker logs ironclaw"
-1
View File
@@ -81,7 +81,6 @@ async fn main() -> anyhow::Result<()> {
let session = create_session_manager(SessionConfig {
auth_base_url: config.llm.nearai.auth_base_url.clone(),
session_path: config.llm.nearai.session_path.clone(),
..Default::default()
})
.await;
let llm = create_llm_provider(&config.llm, session)?;
+228 -149
View File
@@ -28,7 +28,7 @@ use crate::tools::ToolRegistry;
use crate::workspace::Workspace;
/// Collapse a tool output string into a single-line preview for display.
fn truncate_for_preview(output: &str, max_chars: usize) -> String {
pub(crate) fn truncate_for_preview(output: &str, max_chars: usize) -> String {
let collapsed: String = output
.chars()
.take(max_chars + 50)
@@ -37,8 +37,14 @@ fn truncate_for_preview(output: &str, max_chars: usize) -> String {
.split_whitespace()
.collect::<Vec<_>>()
.join(" ");
if collapsed.len() > max_chars {
format!("{}...", &collapsed[..max_chars])
// char_indices gives us byte offsets at char boundaries, so the slice is always valid UTF-8.
if collapsed.chars().count() > max_chars {
let byte_offset = collapsed
.char_indices()
.nth(max_chars)
.map(|(i, _)| i)
.unwrap_or(collapsed.len());
format!("{}...", &collapsed[..byte_offset])
} else {
collapsed
}
@@ -61,6 +67,9 @@ enum AgenticLoopResult {
pub struct AgentDeps {
pub store: Option<Arc<dyn Database>>,
pub llm: Arc<dyn LlmProvider>,
/// Cheap/fast LLM for lightweight tasks (heartbeat, routing, evaluation).
/// Falls back to the main `llm` if None.
pub cheap_llm: Option<Arc<dyn LlmProvider>>,
pub safety: Arc<SafetyLayer>,
pub tools: Arc<ToolRegistry>,
pub workspace: Option<Arc<Workspace>>,
@@ -132,6 +141,11 @@ impl Agent {
&self.deps.llm
}
/// Get the cheap/fast LLM provider, falling back to the main one.
fn cheap_llm(&self) -> &Arc<dyn LlmProvider> {
self.deps.cheap_llm.as_ref().unwrap_or(&self.deps.llm)
}
fn safety(&self) -> &Arc<SafetyLayer> {
&self.deps.safety
}
@@ -295,7 +309,7 @@ impl Agent {
Some(spawn_heartbeat(
config,
workspace.clone(),
self.llm().clone(),
self.cheap_llm().clone(),
Some(notify_tx),
))
} else {
@@ -654,19 +668,17 @@ impl Agent {
}
// Restore response chain from conversation metadata
if let Some(store) = self.store() {
if let Ok(Some(metadata)) = store.get_conversation_metadata(thread_uuid).await {
if let Some(rid) = metadata
.get("last_response_id")
.and_then(|v| v.as_str())
.map(String::from)
{
thread.last_response_id = Some(rid.clone());
self.llm()
.seed_response_chain(&thread_uuid.to_string(), rid);
tracing::debug!("Restored response chain for thread {}", thread_uuid);
}
}
if let Some(store) = self.store()
&& let Ok(Some(metadata)) = store.get_conversation_metadata(thread_uuid).await
&& let Some(rid) = metadata
.get("last_response_id")
.and_then(|v| v.as_str())
.map(String::from)
{
thread.last_response_id = Some(rid.clone());
self.llm()
.seed_response_chain(&thread_uuid.to_string(), rid);
tracing::debug!("Restored response chain for thread {}", thread_uuid);
}
// Insert into session and register with session manager
@@ -954,13 +966,12 @@ impl Agent {
return;
}
if let Some(ref resp) = response {
if let Err(e) = store
if let Some(ref resp) = response
&& let Err(e) = store
.add_conversation_message(thread_id, "assistant", resp)
.await
{
tracing::warn!("Failed to persist assistant message: {}", e);
}
{
tracing::warn!("Failed to persist assistant message: {}", e);
}
});
}
@@ -1058,14 +1069,14 @@ impl Agent {
// Check if interrupted
{
let sess = session.lock().await;
if let Some(thread) = sess.threads.get(&thread_id) {
if thread.state == ThreadState::Interrupted {
return Err(crate::error::JobError::ContextError {
id: thread_id,
reason: "Interrupted".to_string(),
}
.into());
if let Some(thread) = sess.threads.get(&thread_id)
&& thread.state == ThreadState::Interrupted
{
return Err(crate::error::JobError::ContextError {
id: thread_id,
reason: "Interrupted".to_string(),
}
.into());
}
}
@@ -1140,11 +1151,11 @@ impl Agent {
// Record tool calls in the thread
{
let mut sess = session.lock().await;
if let Some(thread) = sess.threads.get_mut(&thread_id) {
if let Some(turn) = thread.last_turn_mut() {
for tc in &tool_calls {
turn.record_tool_call(&tc.name, tc.arguments.clone());
}
if let Some(thread) = sess.threads.get_mut(&thread_id)
&& let Some(turn) = thread.last_turn_mut()
{
for tc in &tool_calls {
turn.record_tool_call(&tc.name, tc.arguments.clone());
}
}
}
@@ -1152,54 +1163,56 @@ impl Agent {
// Execute each tool (with approval checking)
for tc in tool_calls {
// Check if tool requires approval
if let Some(tool) = self.tools().get(&tc.name).await {
if tool.requires_approval() {
// Check if auto-approved for this session
let mut is_auto_approved = {
let sess = session.lock().await;
sess.is_tool_auto_approved(&tc.name)
if let Some(tool) = self.tools().get(&tc.name).await
&& tool.requires_approval()
{
// Check if auto-approved for this session
let mut is_auto_approved = {
let sess = session.lock().await;
sess.is_tool_auto_approved(&tc.name)
};
// For shell commands, override auto-approval for
// destructive patterns that should always require
// explicit per-invocation approval.
if is_auto_approved
&& tc.name == "shell"
&& let Some(cmd) = tc
.arguments
.get("command")
.and_then(|c| c.as_str().map(String::from))
.or_else(|| {
tc.arguments
.as_str()
.and_then(|s| {
serde_json::from_str::<serde_json::Value>(s).ok()
})
.and_then(|v| {
v.get("command")
.and_then(|c| c.as_str().map(String::from))
})
})
&& crate::tools::builtin::shell::requires_explicit_approval(&cmd)
{
tracing::info!(
"Shell command '{}' requires explicit approval despite auto-approve",
cmd.chars().take(80).collect::<String>()
);
is_auto_approved = false;
}
if !is_auto_approved {
// Need approval - store pending request and return
let pending = PendingApproval {
request_id: Uuid::new_v4(),
tool_name: tc.name.clone(),
parameters: tc.arguments.clone(),
description: tool.description().to_string(),
tool_call_id: tc.id.clone(),
context_messages: context_messages.clone(),
};
// For shell commands, override auto-approval for
// destructive patterns that should always require
// explicit per-invocation approval.
if is_auto_approved && tc.name == "shell" {
if let Some(cmd) = tc
.arguments
.as_str()
.and_then(|s| {
serde_json::from_str::<serde_json::Value>(s).ok()
})
.and_then(|v| {
v.get("command")
.and_then(|c| c.as_str().map(String::from))
})
{
if crate::tools::builtin::shell::requires_explicit_approval(
&cmd,
) {
tracing::info!(
"Shell command '{}' requires explicit approval despite auto-approve",
cmd.chars().take(80).collect::<String>()
);
is_auto_approved = false;
}
}
}
if !is_auto_approved {
// Need approval - store pending request and return
let pending = PendingApproval {
request_id: Uuid::new_v4(),
tool_name: tc.name.clone(),
parameters: tc.arguments.clone(),
description: tool.description().to_string(),
tool_call_id: tc.id.clone(),
context_messages: context_messages.clone(),
};
return Ok(AgenticLoopResult::NeedApproval { pending });
}
return Ok(AgenticLoopResult::NeedApproval { pending });
}
}
@@ -1230,34 +1243,34 @@ impl Agent {
)
.await;
if let Ok(ref output) = tool_result {
if !output.is_empty() {
let _ = self
.channels
.send_status(
&message.channel,
StatusUpdate::ToolResult {
name: tc.name.clone(),
preview: truncate_for_preview(output, 200),
},
&message.metadata,
)
.await;
}
if let Ok(ref output) = tool_result
&& !output.is_empty()
{
let _ = self
.channels
.send_status(
&message.channel,
StatusUpdate::ToolResult {
name: tc.name.clone(),
preview: output.clone(),
},
&message.metadata,
)
.await;
}
// Record result in thread
{
let mut sess = session.lock().await;
if let Some(thread) = sess.threads.get_mut(&thread_id) {
if let Some(turn) = thread.last_turn_mut() {
match &tool_result {
Ok(output) => {
turn.record_tool_result(serde_json::json!(output));
}
Err(e) => {
turn.record_tool_error(e.to_string());
}
if let Some(thread) = sess.threads.get_mut(&thread_id)
&& let Some(turn) = thread.last_turn_mut()
{
match &tool_result {
Ok(output) => {
turn.record_tool_result(serde_json::json!(output));
}
Err(e) => {
turn.record_tool_error(e.to_string());
}
}
}
@@ -1640,17 +1653,17 @@ impl Agent {
};
// Verify request ID if provided
if let Some(req_id) = request_id {
if req_id != pending.request_id {
// Put it back and return error
let mut sess = session.lock().await;
if let Some(thread) = sess.threads.get_mut(&thread_id) {
thread.await_approval(pending);
}
return Ok(SubmissionResult::error(
"Request ID mismatch. Use the correct request ID.",
));
if let Some(req_id) = request_id
&& req_id != pending.request_id
{
// Put it back and return error
let mut sess = session.lock().await;
if let Some(thread) = sess.threads.get_mut(&thread_id) {
thread.await_approval(pending);
}
return Ok(SubmissionResult::error(
"Request ID mismatch. Use the correct request ID.",
));
}
if approved {
@@ -1704,20 +1717,20 @@ impl Agent {
)
.await;
if let Ok(ref output) = tool_result {
if !output.is_empty() {
let _ = self
.channels
.send_status(
&message.channel,
StatusUpdate::ToolResult {
name: pending.tool_name.clone(),
preview: truncate_for_preview(output, 200),
},
&message.metadata,
)
.await;
}
if let Ok(ref output) = tool_result
&& !output.is_empty()
{
let _ = self
.channels
.send_status(
&message.channel,
StatusUpdate::ToolResult {
name: pending.tool_name.clone(),
preview: output.clone(),
},
&message.metadata,
)
.await;
}
// Build context including the tool result
@@ -1726,15 +1739,15 @@ impl Agent {
// Record result in thread
{
let mut sess = session.lock().await;
if let Some(thread) = sess.threads.get_mut(&thread_id) {
if let Some(turn) = thread.last_turn_mut() {
match &tool_result {
Ok(output) => {
turn.record_tool_result(serde_json::json!(output));
}
Err(e) => {
turn.record_tool_error(e.to_string());
}
if let Some(thread) = sess.threads.get_mut(&thread_id)
&& let Some(turn) = thread.last_turn_mut()
{
match &tool_result {
Ok(output) => {
turn.record_tool_result(serde_json::json!(output));
}
Err(e) => {
turn.record_tool_error(e.to_string());
}
}
}
@@ -2094,15 +2107,15 @@ impl Agent {
}
// Persist new job to database (fire-and-forget)
if let Some(store) = self.store() {
if let Ok(ctx) = self.context_manager.get_context(job_id).await {
let store = store.clone();
tokio::spawn(async move {
if let Err(e) = store.save_job(&ctx).await {
tracing::warn!("Failed to persist new job {}: {}", job_id, e);
}
});
}
if let Some(store) = self.store()
&& let Ok(ctx) = self.context_manager.get_context(job_id).await
{
let store = store.clone();
tokio::spawn(async move {
if let Err(e) = store.save_job(&ctx).await {
tracing::warn!("Failed to persist new job {}: {}", job_id, e);
}
});
}
// Schedule for execution
@@ -2182,10 +2195,10 @@ impl Agent {
let mut output = String::from("Jobs:\n");
for job_id in jobs {
if let Ok(ctx) = self.context_manager.get_context(job_id).await {
if ctx.user_id == user_id {
output.push_str(&format!(" {} - {} ({:?})\n", job_id, ctx.title, ctx.state));
}
if let Ok(ctx) = self.context_manager.get_context(job_id).await
&& ctx.user_id == user_id
{
output.push_str(&format!(" {} - {} ({:?})\n", job_id, ctx.title, ctx.state));
}
}
@@ -2636,4 +2649,70 @@ mod tests {
assert!(detect_auth_awaiting("tool_activate", &result).is_none());
}
// --- truncate_for_preview tests ---
use super::truncate_for_preview;
#[test]
fn test_truncate_short_input() {
assert_eq!(truncate_for_preview("hello", 10), "hello");
}
#[test]
fn test_truncate_empty_input() {
assert_eq!(truncate_for_preview("", 10), "");
}
#[test]
fn test_truncate_exact_length() {
assert_eq!(truncate_for_preview("hello", 5), "hello");
}
#[test]
fn test_truncate_over_limit() {
let result = truncate_for_preview("hello world, this is long", 10);
assert!(result.ends_with("..."));
// "hello worl" = 10 chars + "..."
assert_eq!(result, "hello worl...");
}
#[test]
fn test_truncate_collapses_newlines() {
let result = truncate_for_preview("line1\nline2\nline3", 100);
assert!(!result.contains('\n'));
assert_eq!(result, "line1 line2 line3");
}
#[test]
fn test_truncate_collapses_whitespace() {
let result = truncate_for_preview("hello world", 100);
assert_eq!(result, "hello world");
}
#[test]
fn test_truncate_multibyte_utf8() {
// Each emoji is 4 bytes. Truncating at char boundary must not panic.
let input = "😀😁😂🤣😃😄😅😆😉😊";
let result = truncate_for_preview(input, 5);
assert!(result.ends_with("..."));
// First 5 chars = 5 emoji
assert_eq!(result, "😀😁😂🤣😃...");
}
#[test]
fn test_truncate_cjk_characters() {
// CJK chars are 3 bytes each in UTF-8.
let input = "你好世界测试数据很长的字符串";
let result = truncate_for_preview(input, 4);
assert_eq!(result, "你好世界...");
}
#[test]
fn test_truncate_mixed_multibyte_and_ascii() {
let input = "hello 世界 foo";
let result = truncate_for_preview(input, 8);
// 'h','e','l','l','o',' ','世','界' = 8 chars
assert_eq!(result, "hello 世界...");
}
}
+1
View File
@@ -26,6 +26,7 @@ pub mod task;
pub mod undo;
pub mod worker;
pub(crate) use agent_loop::truncate_for_preview;
pub use agent_loop::{Agent, AgentDeps};
pub use compaction::{CompactionResult, ContextCompactor};
pub use context_monitor::{CompactionStrategy, ContextBreakdown, ContextMonitor};
+2 -3
View File
@@ -103,10 +103,9 @@ impl RoutineEngine {
if let Trigger::Event {
channel: Some(ch), ..
} = &routine.trigger
&& ch != &message.channel
{
if ch != &message.channel {
continue;
}
continue;
}
// Regex match
+18 -18
View File
@@ -119,25 +119,25 @@ impl SelfRepair for DefaultSelfRepair {
let mut stuck_jobs = Vec::new();
for job_id in stuck_ids {
if let Ok(ctx) = self.context_manager.get_context(job_id).await {
if ctx.state == JobState::Stuck {
let stuck_duration = ctx
.started_at
.map(|start| {
let now = Utc::now();
let duration = now.signed_duration_since(start);
Duration::from_secs(duration.num_seconds().max(0) as u64)
})
.unwrap_or_default();
if let Ok(ctx) = self.context_manager.get_context(job_id).await
&& ctx.state == JobState::Stuck
{
let stuck_duration = ctx
.started_at
.map(|start| {
let now = Utc::now();
let duration = now.signed_duration_since(start);
Duration::from_secs(duration.num_seconds().max(0) as u64)
})
.unwrap_or_default();
stuck_jobs.push(StuckJob {
job_id,
last_activity: ctx.started_at.unwrap_or(ctx.created_at),
stuck_duration,
last_error: None,
repair_attempts: ctx.repair_attempts,
});
}
stuck_jobs.push(StuckJob {
job_id,
last_activity: ctx.started_at.unwrap_or(ctx.created_at),
stuck_duration,
last_error: None,
repair_attempts: ctx.repair_attempts,
});
}
}
+5 -5
View File
@@ -346,11 +346,11 @@ impl Thread {
let mut turn = Turn::new(turn_number, &msg.content);
// Check if next is assistant response
if let Some(next) = iter.peek() {
if next.role == crate::llm::Role::Assistant {
let response = iter.next().expect("peeked");
turn.complete(&response.content);
}
if let Some(next) = iter.peek()
&& next.role == crate::llm::Role::Assistant
{
let response = iter.next().expect("peeked");
turn.complete(&response.content);
}
self.turns.push(turn);
+4 -4
View File
@@ -199,10 +199,10 @@ impl SessionManager {
{
let sessions = self.sessions.read().await;
for user_id in &stale_users {
if let Some(session) = sessions.get(user_id) {
if let Ok(sess) = session.try_lock() {
stale_thread_ids.extend(sess.threads.keys());
}
if let Some(session) = sessions.get(user_id)
&& let Ok(sess) = session.try_lock()
{
stale_thread_ids.extend(sess.threads.keys());
}
}
}
+13 -14
View File
@@ -93,27 +93,26 @@ impl SubmissionParser {
// /thread <uuid> - switch thread
if let Some(rest) = lower.strip_prefix("/thread ") {
let rest = rest.trim();
if rest != "new" {
if let Ok(id) = Uuid::parse_str(rest) {
return Submission::SwitchThread { thread_id: id };
}
if rest != "new"
&& let Ok(id) = Uuid::parse_str(rest)
{
return Submission::SwitchThread { thread_id: id };
}
}
// /resume <uuid> - resume from checkpoint
if let Some(rest) = lower.strip_prefix("/resume ") {
if let Ok(id) = Uuid::parse_str(rest.trim()) {
return Submission::Resume { checkpoint_id: id };
}
if let Some(rest) = lower.strip_prefix("/resume ")
&& let Ok(id) = Uuid::parse_str(rest.trim())
{
return Submission::Resume { checkpoint_id: id };
}
// Try structured JSON approval (from web gateway's /api/chat/approval endpoint)
if trimmed.starts_with('{') {
if let Ok(submission) = serde_json::from_str::<Submission>(trimmed) {
if matches!(submission, Submission::ExecApproval { .. }) {
return submission;
}
}
if trimmed.starts_with('{')
&& let Ok(submission) = serde_json::from_str::<Submission>(trimmed)
&& matches!(submission, Submission::ExecApproval { .. })
{
return submission;
}
// Approval responses (simple yes/no/always for pending approvals)
+30 -8
View File
@@ -227,11 +227,11 @@ Report when the job is complete or if you encounter issues you cannot resolve."#
}
// Check for cancellation
if let Ok(ctx) = self.context_manager().get_context(self.job_id).await {
if ctx.state == JobState::Cancelled {
tracing::info!("Worker for job {} detected cancellation", self.job_id);
return Ok(());
}
if let Ok(ctx) = self.context_manager().get_context(self.job_id).await
&& ctx.state == JobState::Cancelled
{
tracing::info!("Worker for job {} detected cancellation", self.job_id);
return Ok(());
}
iteration += 1;
@@ -299,6 +299,7 @@ Report when the job is complete or if you encounter issues you cannot resolve."#
parameters: tc.arguments.clone(),
reasoning: String::new(),
alternatives: vec![],
tool_call_id: tc.id.clone(),
};
self.process_tool_result(reason_ctx, &selection, result)
@@ -565,7 +566,7 @@ Report when the job is complete or if you encounter issues you cannot resolve."#
);
reason_ctx.messages.push(ChatMessage::tool_result(
"tool_call_id",
&selection.tool_call_id,
&selection.tool_name,
wrapped,
));
@@ -597,7 +598,7 @@ Report when the job is complete or if you encounter issues you cannot resolve."#
}
reason_ctx.messages.push(ChatMessage::tool_result(
"tool_call_id",
&selection.tool_call_id,
&selection.tool_name,
format!("Error: {}", e),
));
@@ -647,12 +648,15 @@ Report when the job is complete or if you encounter issues you cannot resolve."#
.execute_tool(&action.tool_name, &action.parameters)
.await;
// Create a synthetic ToolSelection for process_tool_result
// Create a synthetic ToolSelection for process_tool_result.
// Plan actions don't originate from an LLM tool_call response so
// there is no real tool_call_id; generate a unique one.
let selection = ToolSelection {
tool_name: action.tool_name.clone(),
parameters: action.parameters.clone(),
reasoning: action.reasoning.clone(),
alternatives: vec![],
tool_call_id: format!("plan_{}_{}", self.job_id, i),
};
// Process the result
@@ -774,8 +778,26 @@ impl From<TaskOutput> for Result<String, Error> {
#[cfg(test)]
mod tests {
use crate::llm::ToolSelection;
use crate::util::llm_signals_completion;
#[test]
fn test_tool_selection_preserves_call_id() {
let selection = ToolSelection {
tool_name: "memory_search".to_string(),
parameters: serde_json::json!({"query": "test"}),
reasoning: "Need to search memory".to_string(),
alternatives: vec![],
tool_call_id: "call_abc123".to_string(),
};
assert_eq!(selection.tool_call_id, "call_abc123");
assert_ne!(
selection.tool_call_id, "tool_call_id",
"tool_call_id must not be the hardcoded placeholder string"
);
}
#[test]
fn test_completion_positive_signals() {
assert!(llm_signals_completion("The job is complete."));
+305 -166
View File
@@ -1,147 +1,145 @@
//! Bootstrap configuration for IronClaw.
//! Bootstrap helpers for IronClaw.
//!
//! These are the only settings that MUST live on disk because they're needed
//! before the database connection is established. Everything else lives in the
//! `settings` table in PostgreSQL.
//! The only setting that truly needs disk persistence before the database is
//! available is `DATABASE_URL` (chicken-and-egg: can't connect to DB without
//! it). Everything else is auto-detected or read from env vars.
//!
//! File: `~/.ironclaw/bootstrap.json`
//! File: `~/.ironclaw/.env` (standard dotenvy format)
use std::path::PathBuf;
use serde::{Deserialize, Serialize};
use crate::settings::KeySource;
/// Minimal config needed to connect to the database and decrypt secrets.
///
/// This is the only JSON file IronClaw reads from disk at startup.
/// All other configuration lives in the `settings` table in PostgreSQL.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct BootstrapConfig {
/// Database connection URL (postgres://...).
#[serde(default)]
pub database_url: Option<String>,
/// Database connection pool size.
#[serde(default)]
pub database_pool_size: Option<usize>,
/// Source for the secrets master key.
#[serde(default)]
pub secrets_master_key_source: KeySource,
/// Whether onboarding wizard has been completed.
#[serde(default)]
pub onboard_completed: bool,
/// Path to the IronClaw-specific `.env` file: `~/.ironclaw/.env`.
pub fn ironclaw_env_path() -> PathBuf {
dirs::home_dir()
.unwrap_or_else(|| PathBuf::from("."))
.join(".ironclaw")
.join(".env")
}
impl Default for BootstrapConfig {
fn default() -> Self {
Self {
database_url: None,
database_pool_size: None,
secrets_master_key_source: KeySource::None,
onboard_completed: false,
}
/// Load env vars from `~/.ironclaw/.env` (in addition to the standard `.env`).
///
/// Call this **after** `dotenvy::dotenv()` so that the standard `./.env`
/// takes priority over `~/.ironclaw/.env`. dotenvy never overwrites
/// existing env vars, so the effective priority is:
///
/// explicit env vars > `./.env` > `~/.ironclaw/.env`
///
/// If `~/.ironclaw/.env` doesn't exist but the legacy `bootstrap.json` does,
/// extracts `DATABASE_URL` from it and writes the `.env` file (one-time
/// upgrade from the old config format).
pub fn load_ironclaw_env() {
let path = ironclaw_env_path();
if !path.exists() {
// One-time upgrade: extract DATABASE_URL from legacy bootstrap.json
migrate_bootstrap_json_to_env(&path);
}
if path.exists() {
let _ = dotenvy::from_path(&path);
}
}
impl BootstrapConfig {
/// Default bootstrap file path: `~/.ironclaw/bootstrap.json`.
pub fn default_path() -> PathBuf {
dirs::home_dir()
.unwrap_or_else(|| PathBuf::from("."))
.join(".ironclaw")
.join("bootstrap.json")
/// If `bootstrap.json` exists, pull `database_url` out of it and write `.env`.
fn migrate_bootstrap_json_to_env(env_path: &std::path::Path) {
let ironclaw_dir = env_path
.parent()
.unwrap_or_else(|| std::path::Path::new("."));
let bootstrap_path = ironclaw_dir.join("bootstrap.json");
if !bootstrap_path.exists() {
return;
}
/// Legacy settings.json path (for migration detection).
pub fn legacy_settings_path() -> PathBuf {
dirs::home_dir()
.unwrap_or_else(|| PathBuf::from("."))
.join(".ironclaw")
.join("settings.json")
}
let content = match std::fs::read_to_string(&bootstrap_path) {
Ok(c) => c,
Err(_) => return,
};
/// Load from the default path, falling back to legacy settings.json,
/// then to defaults if neither exists.
pub fn load() -> Self {
let bootstrap_path = Self::default_path();
if bootstrap_path.exists() {
return Self::load_from(&bootstrap_path);
// Minimal parse: just grab database_url from the JSON
let parsed: serde_json::Value = match serde_json::from_str(&content) {
Ok(v) => v,
Err(_) => return,
};
if let Some(url) = parsed.get("database_url").and_then(|v| v.as_str()) {
if let Some(parent) = env_path.parent()
&& let Err(e) = std::fs::create_dir_all(parent)
{
eprintln!("Warning: failed to create {}: {}", parent.display(), e);
return;
}
// Fall back to legacy settings.json (extract just the 4 bootstrap fields)
let legacy_path = Self::legacy_settings_path();
if legacy_path.exists() {
return Self::load_from_legacy(&legacy_path);
if let Err(e) = std::fs::write(env_path, format!("DATABASE_URL=\"{}\"\n", url)) {
eprintln!("Warning: failed to migrate bootstrap.json to .env: {}", e);
return;
}
Self::default()
}
/// Load from a specific path.
pub fn load_from(path: &PathBuf) -> Self {
match std::fs::read_to_string(path) {
Ok(data) => serde_json::from_str(&data).unwrap_or_default(),
Err(_) => Self::default(),
}
}
/// Extract bootstrap fields from a legacy settings.json.
fn load_from_legacy(path: &PathBuf) -> Self {
match std::fs::read_to_string(path) {
Ok(data) => {
// The legacy Settings struct is a superset; serde will ignore extra fields.
serde_json::from_str(&data).unwrap_or_default()
}
Err(_) => Self::default(),
}
}
/// Save to the default path.
pub fn save(&self) -> std::io::Result<()> {
self.save_to(&Self::default_path())
}
/// Save to a specific path.
pub fn save_to(&self, path: &PathBuf) -> std::io::Result<()> {
if let Some(parent) = path.parent() {
std::fs::create_dir_all(parent)?;
}
let json = serde_json::to_string_pretty(self)
.map_err(|e| std::io::Error::new(std::io::ErrorKind::InvalidData, e.to_string()))?;
std::fs::write(path, json)
rename_to_migrated(&bootstrap_path);
eprintln!(
"Migrated DATABASE_URL from bootstrap.json to {}",
env_path.display()
);
}
}
/// One-time migration from disk config files to the database settings table.
/// Write database bootstrap vars to `~/.ironclaw/.env`.
///
/// On first boot after upgrade, checks if:
/// 1. `~/.ironclaw/settings.json` exists
/// 2. The DB settings table is empty for this user
/// These settings form the chicken-and-egg layer: they must be available
/// from the filesystem (env vars) BEFORE any database connection, because
/// they determine which database to connect to. Everything else is stored
/// in the database itself.
///
/// If both conditions hold, migrates settings, MCP servers, and session data
/// to the database, writes `bootstrap.json`, and renames old files to `.migrated`.
/// Creates the parent directory if it doesn't exist.
/// Values are double-quoted so that `#` (common in URL-encoded passwords)
/// and other shell-special characters are preserved by dotenvy.
pub fn save_bootstrap_env(vars: &[(&str, &str)]) -> std::io::Result<()> {
let path = ironclaw_env_path();
if let Some(parent) = path.parent() {
std::fs::create_dir_all(parent)?;
}
let mut content = String::new();
for (key, value) in vars {
content.push_str(&format!("{}=\"{}\"\n", key, value));
}
std::fs::write(&path, content)
}
/// Write `DATABASE_URL` to `~/.ironclaw/.env`.
///
/// Convenience wrapper around `save_bootstrap_env` for single-value migration
/// paths. Prefer `save_bootstrap_env` for new code.
pub fn save_database_url(url: &str) -> std::io::Result<()> {
save_bootstrap_env(&[("DATABASE_URL", url)])
}
/// One-time migration of legacy `~/.ironclaw/settings.json` into the database.
///
/// Only runs when a `settings.json` exists on disk AND the DB has no settings
/// yet. After the wizard writes directly to the DB, this path is only hit by
/// users upgrading from the old disk-only configuration.
///
/// After syncing, renames `settings.json` to `.migrated` so it won't trigger again.
pub async fn migrate_disk_to_db(
store: &dyn crate::db::Database,
user_id: &str,
) -> Result<(), MigrationError> {
let legacy_settings_path = BootstrapConfig::legacy_settings_path();
let ironclaw_dir = dirs::home_dir()
.unwrap_or_else(|| PathBuf::from("."))
.join(".ironclaw");
let legacy_settings_path = ironclaw_dir.join("settings.json");
if !legacy_settings_path.exists() {
tracing::debug!("No legacy settings.json found, skipping disk-to-DB migration");
return Ok(());
}
// Only migrate if DB is empty for this user
// If DB already has settings, this is not a first boot, the wizard already
// wrote directly to the DB. Just clean up the stale file.
let has_settings = store.has_settings(user_id).await.map_err(|e| {
MigrationError::Database(format!("Failed to check existing settings: {}", e))
})?;
if has_settings {
tracing::debug!(
"DB already has settings for user '{}', skipping migration",
user_id
);
tracing::info!("DB already has settings, renaming stale settings.json");
rename_to_migrated(&legacy_settings_path);
return Ok(());
}
@@ -160,22 +158,14 @@ pub async fn migrate_disk_to_db(
tracing::info!("Migrated {} settings to database", db_map.len());
}
// 2. Write bootstrap.json with the 4 essential fields
let bootstrap = BootstrapConfig {
database_url: settings.database_url.clone(),
database_pool_size: settings.database_pool_size,
secrets_master_key_source: settings.secrets_master_key_source,
onboard_completed: settings.onboard_completed,
};
bootstrap
.save()
.map_err(|e| MigrationError::Io(format!("Failed to write bootstrap.json: {}", e)))?;
tracing::info!("Wrote bootstrap.json");
// 2. Write DATABASE_URL to ~/.ironclaw/.env
if let Some(ref url) = settings.database_url {
save_database_url(url)
.map_err(|e| MigrationError::Io(format!("Failed to write .env: {}", e)))?;
tracing::info!("Wrote DATABASE_URL to {}", ironclaw_env_path().display());
}
// 3. Migrate mcp-servers.json if it exists
let ironclaw_dir = dirs::home_dir()
.unwrap_or_else(|| PathBuf::from("."))
.join(".ironclaw");
let mcp_path = ironclaw_dir.join("mcp-servers.json");
if mcp_path.exists() {
match std::fs::read_to_string(&mcp_path) {
@@ -211,7 +201,7 @@ pub async fn migrate_disk_to_db(
Ok(content) => match serde_json::from_str::<serde_json::Value>(&content) {
Ok(value) => {
store
.set_setting(user_id, "nearai.session", &value)
.set_setting(user_id, "nearai.session_token", &value)
.await
.map_err(|e| {
MigrationError::Database(format!(
@@ -236,12 +226,19 @@ pub async fn migrate_disk_to_db(
// 5. Rename settings.json to .migrated (don't delete, safety net)
rename_to_migrated(&legacy_settings_path);
// 6. Clean up old bootstrap.json if it exists (superseded by .env)
let old_bootstrap = ironclaw_dir.join("bootstrap.json");
if old_bootstrap.exists() {
rename_to_migrated(&old_bootstrap);
tracing::info!("Renamed old bootstrap.json to .migrated");
}
tracing::info!("Disk-to-DB migration complete");
Ok(())
}
/// Rename a file to `<name>.migrated` as a safety net.
fn rename_to_migrated(path: &PathBuf) {
fn rename_to_migrated(path: &std::path::Path) {
let mut migrated = path.as_os_str().to_owned();
migrated.push(".migrated");
if let Err(e) = std::fs::rename(path, &migrated) {
@@ -264,62 +261,204 @@ mod tests {
use tempfile::tempdir;
#[test]
fn test_bootstrap_save_load() {
fn test_save_and_load_database_url() {
let dir = tempdir().unwrap();
let path = dir.path().join("bootstrap.json");
let env_path = dir.path().join(".env");
let config = BootstrapConfig {
database_url: Some("postgres://localhost/test".to_string()),
database_pool_size: Some(5),
secrets_master_key_source: KeySource::Keychain,
onboard_completed: true,
};
// Write in the quoted format that save_database_url uses
let url = "postgres://localhost:5432/ironclaw_test";
std::fs::write(&env_path, format!("DATABASE_URL=\"{}\"\n", url)).unwrap();
config.save_to(&path).unwrap();
let loaded = BootstrapConfig::load_from(&path);
// Verify the content is a valid dotenv line (quoted)
let content = std::fs::read_to_string(&env_path).unwrap();
assert_eq!(
loaded.database_url,
Some("postgres://localhost/test".to_string())
content,
"DATABASE_URL=\"postgres://localhost:5432/ironclaw_test\"\n"
);
assert_eq!(loaded.database_pool_size, Some(5));
assert_eq!(loaded.secrets_master_key_source, KeySource::Keychain);
assert!(loaded.onboard_completed);
// Verify dotenvy can parse it (strips quotes automatically)
let parsed: Vec<(String, String)> = dotenvy::from_path_iter(&env_path)
.unwrap()
.filter_map(|r| r.ok())
.collect();
assert_eq!(parsed.len(), 1);
assert_eq!(parsed[0].0, "DATABASE_URL");
assert_eq!(parsed[0].1, url);
}
#[test]
fn test_bootstrap_from_legacy_settings() {
fn test_save_database_url_with_hash_in_password() {
let dir = tempdir().unwrap();
let path = dir.path().join("settings.json");
let env_path = dir.path().join(".env");
// Write a legacy settings.json with many extra fields
let legacy = serde_json::json!({
"database_url": "postgres://localhost/ironclaw",
"database_pool_size": 10,
// URLs with # in the password are common (URL-encoded special chars).
// Without quoting, dotenvy treats # as a comment delimiter.
let url = "postgres://user:p%23ss@localhost:5432/ironclaw";
std::fs::write(&env_path, format!("DATABASE_URL=\"{}\"\n", url)).unwrap();
let parsed: Vec<(String, String)> = dotenvy::from_path_iter(&env_path)
.unwrap()
.filter_map(|r| r.ok())
.collect();
assert_eq!(parsed.len(), 1);
assert_eq!(parsed[0].0, "DATABASE_URL");
assert_eq!(parsed[0].1, url);
}
#[test]
fn test_save_database_url_creates_parent_dirs() {
let dir = tempdir().unwrap();
let nested = dir.path().join("deep").join("nested");
let env_path = nested.join(".env");
// Parent doesn't exist yet
assert!(!nested.exists());
// The global function uses a fixed path, so we test the logic directly
std::fs::create_dir_all(&nested).unwrap();
std::fs::write(&env_path, "DATABASE_URL=postgres://test\n").unwrap();
assert!(env_path.exists());
let content = std::fs::read_to_string(&env_path).unwrap();
assert!(content.contains("DATABASE_URL=postgres://test"));
}
#[test]
fn test_ironclaw_env_path() {
let path = ironclaw_env_path();
assert!(path.ends_with(".ironclaw/.env"));
}
#[test]
fn test_migrate_bootstrap_json_to_env() {
let dir = tempdir().unwrap();
let env_path = dir.path().join(".env");
let bootstrap_path = dir.path().join("bootstrap.json");
// Write a legacy bootstrap.json
let bootstrap_json = serde_json::json!({
"database_url": "postgres://localhost/ironclaw_upgrade",
"database_pool_size": 5,
"secrets_master_key_source": "keychain",
"onboard_completed": true,
"selected_model": "claude-3-5-sonnet",
"agent": { "name": "testbot", "max_parallel_jobs": 3 },
"heartbeat": { "enabled": true }
"onboard_completed": true
});
std::fs::write(&path, serde_json::to_string_pretty(&legacy).unwrap()).unwrap();
std::fs::write(
&bootstrap_path,
serde_json::to_string_pretty(&bootstrap_json).unwrap(),
)
.unwrap();
let config = BootstrapConfig::load_from_legacy(&path);
assert!(!env_path.exists());
assert!(bootstrap_path.exists());
// Run the migration
migrate_bootstrap_json_to_env(&env_path);
// .env should now exist with DATABASE_URL
assert!(env_path.exists());
let content = std::fs::read_to_string(&env_path).unwrap();
assert_eq!(
config.database_url,
Some("postgres://localhost/ironclaw".to_string())
content,
"DATABASE_URL=\"postgres://localhost/ironclaw_upgrade\"\n"
);
assert_eq!(config.database_pool_size, Some(10));
assert_eq!(config.secrets_master_key_source, KeySource::Keychain);
assert!(config.onboard_completed);
// bootstrap.json should be renamed to .migrated
assert!(!bootstrap_path.exists());
assert!(dir.path().join("bootstrap.json.migrated").exists());
}
#[test]
fn test_bootstrap_defaults() {
let config = BootstrapConfig::default();
assert!(config.database_url.is_none());
assert!(config.database_pool_size.is_none());
assert_eq!(config.secrets_master_key_source, KeySource::None);
assert!(!config.onboard_completed);
fn test_migrate_bootstrap_json_no_database_url() {
let dir = tempdir().unwrap();
let env_path = dir.path().join(".env");
let bootstrap_path = dir.path().join("bootstrap.json");
// bootstrap.json with no database_url
let bootstrap_json = serde_json::json!({
"onboard_completed": false
});
std::fs::write(
&bootstrap_path,
serde_json::to_string_pretty(&bootstrap_json).unwrap(),
)
.unwrap();
migrate_bootstrap_json_to_env(&env_path);
// .env should NOT be created
assert!(!env_path.exists());
// bootstrap.json should remain (no migration happened)
assert!(bootstrap_path.exists());
}
#[test]
fn test_migrate_bootstrap_json_missing() {
let dir = tempdir().unwrap();
let env_path = dir.path().join(".env");
// No bootstrap.json at all
migrate_bootstrap_json_to_env(&env_path);
// Nothing should happen
assert!(!env_path.exists());
}
#[test]
fn test_save_bootstrap_env_multiple_vars() {
let dir = tempdir().unwrap();
let env_path = dir.path().join("nested").join(".env");
std::fs::create_dir_all(env_path.parent().unwrap()).unwrap();
let vars = [
("DATABASE_BACKEND", "libsql"),
("LIBSQL_PATH", "/home/user/.ironclaw/ironclaw.db"),
];
// Write manually to the temp path (save_bootstrap_env uses the global path)
let mut content = String::new();
for (key, value) in &vars {
content.push_str(&format!("{}=\"{}\"\n", key, value));
}
std::fs::write(&env_path, &content).unwrap();
// Verify dotenvy can parse all entries
let parsed: Vec<(String, String)> = dotenvy::from_path_iter(&env_path)
.unwrap()
.filter_map(|r| r.ok())
.collect();
assert_eq!(parsed.len(), 2);
assert_eq!(
parsed[0],
("DATABASE_BACKEND".to_string(), "libsql".to_string())
);
assert_eq!(
parsed[1],
(
"LIBSQL_PATH".to_string(),
"/home/user/.ironclaw/ironclaw.db".to_string()
)
);
}
#[test]
fn test_save_bootstrap_env_overwrites_previous() {
let dir = tempdir().unwrap();
let env_path = dir.path().join(".env");
// Write initial content
std::fs::write(&env_path, "DATABASE_URL=\"postgres://old\"\n").unwrap();
// Overwrite with new vars (simulating save_bootstrap_env behavior)
let content = "DATABASE_BACKEND=\"libsql\"\nLIBSQL_PATH=\"/new/path.db\"\n";
std::fs::write(&env_path, content).unwrap();
let parsed: Vec<(String, String)> = dotenvy::from_path_iter(&env_path)
.unwrap()
.filter_map(|r| r.ok())
.collect();
// Old DATABASE_URL should be gone
assert_eq!(parsed.len(), 2);
assert!(parsed.iter().all(|(k, _)| k != "DATABASE_URL"));
}
}
+17 -7
View File
@@ -33,9 +33,16 @@ use termimad::MadSkin;
use tokio::sync::mpsc;
use tokio_stream::wrappers::ReceiverStream;
use crate::agent::truncate_for_preview;
use crate::channels::{Channel, IncomingMessage, MessageStream, OutgoingResponse, StatusUpdate};
use crate::error::ChannelError;
/// Max characters for tool result previews in the terminal.
const CLI_TOOL_RESULT_MAX: usize = 200;
/// Max characters for thinking/status messages in the terminal.
const CLI_STATUS_MAX: usize = 200;
/// Slash commands available in the REPL.
const SLASH_COMMANDS: &[&str] = &[
"/help",
@@ -261,7 +268,7 @@ impl Channel for ReplChannel {
std::thread::spawn(move || {
// Single message mode: send it and return
if let Some(msg) = single_message {
let incoming = IncomingMessage::new("repl", "user", &msg);
let incoming = IncomingMessage::new("repl", "default", &msg);
let _ = tx.blocking_send(incoming);
return;
}
@@ -329,21 +336,21 @@ impl Channel for ReplChannel {
_ => {}
}
let msg = IncomingMessage::new("repl", "user", line);
let msg = IncomingMessage::new("repl", "default", line);
if tx.blocking_send(msg).is_err() {
break;
}
}
Err(ReadlineError::Interrupted) => {
// Ctrl+C: send /interrupt
let msg = IncomingMessage::new("repl", "user", "/interrupt");
let msg = IncomingMessage::new("repl", "default", "/interrupt");
if tx.blocking_send(msg).is_err() {
break;
}
}
Err(ReadlineError::Eof) => {
// Ctrl+D: send /quit so the agent loop runs graceful shutdown
let msg = IncomingMessage::new("repl", "user", "/quit");
let msg = IncomingMessage::new("repl", "default", "/quit");
let _ = tx.blocking_send(msg);
break;
}
@@ -400,7 +407,8 @@ impl Channel for ReplChannel {
match status {
StatusUpdate::Thinking(msg) => {
eprintln!(" \x1b[90m\u{25CB} {msg}\x1b[0m");
let display = truncate_for_preview(&msg, CLI_STATUS_MAX);
eprintln!(" \x1b[90m\u{25CB} {display}\x1b[0m");
}
StatusUpdate::ToolStarted { name } => {
eprintln!(" \x1b[33m\u{25CB} {name}\x1b[0m");
@@ -413,7 +421,8 @@ impl Channel for ReplChannel {
}
}
StatusUpdate::ToolResult { name: _, preview } => {
eprintln!(" \x1b[90m{preview}\x1b[0m");
let display = truncate_for_preview(&preview, CLI_TOOL_RESULT_MAX);
eprintln!(" \x1b[90m{display}\x1b[0m");
}
StatusUpdate::StreamChunk(chunk) => {
// Print separator on the false-to-true transition
@@ -438,7 +447,8 @@ impl Channel for ReplChannel {
}
StatusUpdate::Status(msg) => {
if debug || msg.contains("approval") || msg.contains("Approval") {
eprintln!(" \x1b[90m{msg}\x1b[0m");
let display = truncate_for_preview(&msg, CLI_STATUS_MAX);
eprintln!(" \x1b[90m{display}\x1b[0m");
}
}
StatusUpdate::ApprovalNeeded {
+112 -43
View File
@@ -76,6 +76,9 @@ struct ChannelStoreData {
credentials: HashMap<String, String>,
/// Pairing store for DM pairing (guest access control).
pairing_store: Arc<PairingStore>,
/// Dedicated tokio runtime for HTTP requests, lazily initialized.
/// Reused across multiple `http_request` calls within one execution.
http_runtime: Option<tokio::runtime::Runtime>,
}
impl ChannelStoreData {
@@ -96,6 +99,7 @@ impl ChannelStoreData {
table: ResourceTable::new(),
credentials,
pairing_store,
http_runtime: None,
}
}
@@ -134,13 +138,13 @@ impl ChannelStoreData {
if result.contains('{') && result.contains('}') {
// Only warn if it looks like an unresolved placeholder (not JSON braces)
let brace_pattern = regex::Regex::new(r"\{[A-Z_]+\}").ok();
if let Some(re) = brace_pattern {
if re.is_match(&result) {
tracing::warn!(
context = %context,
"String may contain unresolved credential placeholders"
);
}
if let Some(re) = brace_pattern
&& re.is_match(&result)
{
tracing::warn!(
context = %context,
"String may contain unresolved credential placeholders"
);
}
}
@@ -283,10 +287,25 @@ impl near::agent::channel_host::Host for ChannelStoreData {
.map(|h| h.max_response_bytes)
.unwrap_or(10 * 1024 * 1024);
// Make the HTTP request using blocking I/O
// We're already in a spawn_blocking context, so we can use block_on
let result = tokio::runtime::Handle::current().block_on(async {
let client = reqwest::Client::new();
// Make the HTTP request using a dedicated single-threaded runtime.
// We're inside spawn_blocking, so we can't rely on the main runtime's
// I/O driver (it may be busy with WASM compilation or other startup work).
// A dedicated runtime gives us our own I/O driver and avoids contention.
// The runtime is lazily created and reused across calls within one execution.
if self.http_runtime.is_none() {
self.http_runtime = Some(
tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.map_err(|e| format!("Failed to create HTTP runtime: {e}"))?,
);
}
let rt = self.http_runtime.as_ref().expect("just initialized");
let result = rt.block_on(async {
let client = reqwest::Client::builder()
.connect_timeout(std::time::Duration::from_secs(10))
.build()
.map_err(|e| format!("Failed to build HTTP client: {e}"))?;
let mut request = match method.to_uppercase().as_str() {
"GET" => client.get(&url),
@@ -308,9 +327,9 @@ impl near::agent::channel_host::Host for ChannelStoreData {
request = request.body(body_bytes);
}
// Send request with caller-specified timeout (default 30s).
// Cap at callback_timeout to prevent outliving the host wrapper.
let timeout = std::time::Duration::from_millis(timeout_ms.unwrap_or(30_000) as u64);
// Send request with caller-specified timeout (default 30s, max 5min).
let timeout_ms = timeout_ms.unwrap_or(30_000).min(300_000) as u64;
let timeout = std::time::Duration::from_millis(timeout_ms);
let response = request.timeout(timeout).send().await.map_err(|e| {
// Walk the full error chain so we get the actual root cause
// (DNS, TLS, connection refused, etc.) instead of just
@@ -338,13 +357,13 @@ impl near::agent::channel_host::Host for ChannelStoreData {
// Enforce max response body size to prevent memory exhaustion.
let max_response = max_response_bytes;
if let Some(cl) = response.content_length() {
if cl as usize > max_response {
return Err(format!(
"Response body too large: {} bytes exceeds limit of {} bytes",
cl, max_response
));
}
if let Some(cl) = response.content_length()
&& cl as usize > max_response
{
return Err(format!(
"Response body too large: {} bytes exceeds limit of {} bytes",
cl, max_response
));
}
let body = response
.bytes()
@@ -795,7 +814,21 @@ impl WasmChannel {
.await;
match result {
Ok(Ok((config, _host_state))) => {
Ok(Ok((config, mut host_state))) => {
// Surface WASM guest logs (errors/warnings from webhook setup, etc.)
for entry in host_state.take_logs() {
match entry.level {
crate::tools::wasm::LogLevel::Error => {
tracing::error!(channel = %self.name, "{}", entry.message);
}
crate::tools::wasm::LogLevel::Warn => {
tracing::warn!(channel = %self.name, "{}", entry.message);
}
_ => {
tracing::debug!(channel = %self.name, "{}", entry.message);
}
}
}
tracing::info!(
channel = %self.name,
display_name = %config.display_name,
@@ -1495,8 +1528,8 @@ impl WasmChannel {
match result {
Ok(emitted_messages) => {
// Process any emitted messages
if !emitted_messages.is_empty() {
if let Err(e) = Self::dispatch_emitted_messages(
if !emitted_messages.is_empty()
&& let Err(e) = Self::dispatch_emitted_messages(
&channel_name,
emitted_messages,
&message_tx,
@@ -1508,7 +1541,6 @@ impl WasmChannel {
"Failed to dispatch emitted messages from poll"
);
}
}
}
Err(e) => {
tracing::warn!(
@@ -1738,22 +1770,22 @@ impl Channel for WasmChannel {
*self.endpoints.write().await = endpoints;
// Start polling if configured
if let Some(poll_config) = &config.poll {
if poll_config.enabled {
let interval = self
.capabilities
.validate_poll_interval(poll_config.interval_ms)
.map_err(|e| ChannelError::StartupFailed {
name: self.name.clone(),
reason: e,
})?;
if let Some(poll_config) = &config.poll
&& poll_config.enabled
{
let interval = self
.capabilities
.validate_poll_interval(poll_config.interval_ms)
.map_err(|e| ChannelError::StartupFailed {
name: self.name.clone(),
reason: e,
})?;
// Create shutdown channel for polling and store the sender to keep it alive
let (poll_shutdown_tx, poll_shutdown_rx) = oneshot::channel();
*self.poll_shutdown_tx.write().await = Some(poll_shutdown_tx);
// Create shutdown channel for polling and store the sender to keep it alive
let (poll_shutdown_tx, poll_shutdown_rx) = oneshot::channel();
*self.poll_shutdown_tx.write().await = Some(poll_shutdown_tx);
self.start_polling(Duration::from_millis(interval as u64), poll_shutdown_rx);
}
self.start_polling(Duration::from_millis(interval as u64), poll_shutdown_rx);
}
tracing::info!(
@@ -2616,15 +2648,52 @@ mod tests {
assert_eq!(store.redact_credentials(input), input);
}
/// Verify that the block_on-inside-spawn_blocking pattern used by the WASM
/// channel HTTP host function doesn't deadlock or panic.
/// Verify that WASM HTTP host functions work using a dedicated
/// current-thread runtime inside spawn_blocking.
#[tokio::test]
async fn test_block_on_inside_spawn_blocking_does_not_deadlock() {
async fn test_dedicated_runtime_inside_spawn_blocking() {
let result = tokio::task::spawn_blocking(|| {
tokio::runtime::Handle::current().block_on(async { 42 })
let rt = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.expect("failed to build runtime");
rt.block_on(async { 42 })
})
.await
.expect("spawn_blocking panicked");
assert_eq!(result, 42);
}
/// Verify a real HTTP request works using the dedicated-runtime pattern.
/// This catches DNS, TLS, and I/O driver issues that trivial tests miss.
#[tokio::test]
#[ignore] // requires network
async fn test_dedicated_runtime_real_http() {
let result = tokio::task::spawn_blocking(|| {
let rt = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.expect("failed to build runtime");
rt.block_on(async {
let client = reqwest::Client::builder()
.connect_timeout(std::time::Duration::from_secs(10))
.build()
.expect("failed to build client");
let resp = client
.get("https://api.telegram.org/bot000/getMe")
.timeout(std::time::Duration::from_secs(10))
.send()
.await;
match resp {
Ok(r) => r.status().as_u16(),
Err(e) if e.is_timeout() => panic!("request timed out: {e}"),
Err(e) => panic!("unexpected error: {e}"),
}
})
})
.await
.expect("spawn_blocking panicked");
// 404 because "000" is not a valid bot token
assert_eq!(result, 404);
}
}
+10 -12
View File
@@ -25,23 +25,21 @@ pub async fn auth_middleware(
next: Next,
) -> Response {
// Try Authorization header first (constant-time comparison)
if let Some(auth_header) = headers.get("authorization") {
if let Ok(value) = auth_header.to_str() {
if let Some(token) = value.strip_prefix("Bearer ") {
if bool::from(token.as_bytes().ct_eq(auth.token.as_bytes())) {
return next.run(request).await;
}
}
}
if let Some(auth_header) = headers.get("authorization")
&& let Ok(value) = auth_header.to_str()
&& let Some(token) = value.strip_prefix("Bearer ")
&& bool::from(token.as_bytes().ct_eq(auth.token.as_bytes()))
{
return next.run(request).await;
}
// Fall back to query parameter for SSE EventSource (constant-time comparison)
if let Some(query) = request.uri().query() {
for pair in query.split('&') {
if let Some(token) = pair.strip_prefix("token=") {
if bool::from(token.as_bytes().ct_eq(auth.token.as_bytes())) {
return next.run(request).await;
}
if let Some(token) = pair.strip_prefix("token=")
&& bool::from(token.as_bytes().ct_eq(auth.token.as_bytes()))
{
return next.run(request).await;
}
}
}
+8 -8
View File
@@ -473,10 +473,10 @@ pub async fn chat_completions_handler(
if let Some(mt) = req.max_tokens {
tool_req = tool_req.with_max_tokens(mt);
}
if let Some(ref tc) = req.tool_choice {
if let Some(choice) = normalize_tool_choice(tc) {
tool_req = tool_req.with_tool_choice(choice);
}
if let Some(ref tc) = req.tool_choice
&& let Some(choice) = normalize_tool_choice(tc)
{
tool_req = tool_req.with_tool_choice(choice);
}
let resp = llm
@@ -591,10 +591,10 @@ async fn handle_streaming(
if let Some(mt) = req.max_tokens {
tool_req = tool_req.with_max_tokens(mt);
}
if let Some(ref tc) = req.tool_choice {
if let Some(choice) = normalize_tool_choice(tc) {
tool_req = tool_req.with_tool_choice(choice);
}
if let Some(ref tc) = req.tool_choice
&& let Some(choice) = normalize_tool_choice(tc)
{
tool_req = tool_req.with_tool_choice(choice);
}
LlmResult::WithTools(
llm.complete_with_tools(tool_req)
+154 -155
View File
@@ -525,10 +525,10 @@ pub async fn clear_auth_mode(state: &GatewayState) {
if let Some(ref sm) = state.session_manager {
let session = sm.get_or_create_session(&state.user_id).await;
let mut sess = session.lock().await;
if let Some(thread_id) = sess.active_thread {
if let Some(thread) = sess.threads.get_mut(&thread_id) {
thread.pending_auth = None;
}
if let Some(thread_id) = sess.active_thread
&& let Some(thread) = sess.threads.get_mut(&thread_id)
{
thread.pending_auth = None;
}
}
}
@@ -626,69 +626,69 @@ async fn chat_history_handler(
// Verify the thread belongs to the authenticated user before returning any data.
// In-memory threads are already scoped by user via session_manager, but DB
// lookups could expose another user's conversation if the UUID is guessed.
if query.thread_id.is_some() {
if let Some(ref store) = state.store {
let owned = store
.conversation_belongs_to_user(thread_id, &state.user_id)
.await
.unwrap_or(false);
if !owned && !sess.threads.contains_key(&thread_id) {
return Err((StatusCode::NOT_FOUND, "Thread not found".to_string()));
}
if query.thread_id.is_some()
&& let Some(ref store) = state.store
{
let owned = store
.conversation_belongs_to_user(thread_id, &state.user_id)
.await
.unwrap_or(false);
if !owned && !sess.threads.contains_key(&thread_id) {
return Err((StatusCode::NOT_FOUND, "Thread not found".to_string()));
}
}
// For paginated requests (before cursor set), always go to DB
if before_cursor.is_some() {
if let Some(ref store) = state.store {
let (messages, has_more) = store
.list_conversation_messages_paginated(thread_id, before_cursor, limit as i64)
.await
.map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, e.to_string()))?;
if before_cursor.is_some()
&& let Some(ref store) = state.store
{
let (messages, has_more) = store
.list_conversation_messages_paginated(thread_id, before_cursor, limit as i64)
.await
.map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, e.to_string()))?;
let oldest_timestamp = messages.first().map(|m| m.created_at.to_rfc3339());
let turns = build_turns_from_db_messages(&messages);
return Ok(Json(HistoryResponse {
thread_id,
turns,
has_more,
oldest_timestamp,
}));
}
let oldest_timestamp = messages.first().map(|m| m.created_at.to_rfc3339());
let turns = build_turns_from_db_messages(&messages);
return Ok(Json(HistoryResponse {
thread_id,
turns,
has_more,
oldest_timestamp,
}));
}
// Try in-memory first (freshest data for active threads)
if let Some(thread) = sess.threads.get(&thread_id) {
if !thread.turns.is_empty() {
let turns: Vec<TurnInfo> = thread
.turns
.iter()
.map(|t| TurnInfo {
turn_number: t.turn_number,
user_input: t.user_input.clone(),
response: t.response.clone(),
state: format!("{:?}", t.state),
started_at: t.started_at.to_rfc3339(),
completed_at: t.completed_at.map(|dt| dt.to_rfc3339()),
tool_calls: t
.tool_calls
.iter()
.map(|tc| ToolCallInfo {
name: tc.name.clone(),
has_result: tc.result.is_some(),
has_error: tc.error.is_some(),
})
.collect(),
})
.collect();
if let Some(thread) = sess.threads.get(&thread_id)
&& !thread.turns.is_empty()
{
let turns: Vec<TurnInfo> = thread
.turns
.iter()
.map(|t| TurnInfo {
turn_number: t.turn_number,
user_input: t.user_input.clone(),
response: t.response.clone(),
state: format!("{:?}", t.state),
started_at: t.started_at.to_rfc3339(),
completed_at: t.completed_at.map(|dt| dt.to_rfc3339()),
tool_calls: t
.tool_calls
.iter()
.map(|tc| ToolCallInfo {
name: tc.name.clone(),
has_result: tc.result.is_some(),
has_error: tc.error.is_some(),
})
.collect(),
})
.collect();
return Ok(Json(HistoryResponse {
thread_id,
turns,
has_more: false,
oldest_timestamp: None,
}));
}
return Ok(Json(HistoryResponse {
thread_id,
turns,
has_more: false,
oldest_timestamp: None,
}));
}
// Fall back to DB for historical threads not in memory (paginated)
@@ -738,12 +738,12 @@ fn build_turns_from_db_messages(messages: &[crate::history::ConversationMessage]
};
// Check if next message is an assistant response
if let Some(next) = iter.peek() {
if next.role == "assistant" {
let assistant_msg = iter.next().expect("peeked");
turn.response = Some(assistant_msg.content.clone());
turn.completed_at = Some(assistant_msg.created_at.to_rfc3339());
}
if let Some(next) = iter.peek()
&& next.role == "assistant"
{
let assistant_msg = iter.next().expect("peeked");
turn.response = Some(assistant_msg.content.clone());
turn.completed_at = Some(assistant_msg.created_at.to_rfc3339());
}
// Incomplete turn (user message without response)
@@ -1126,65 +1126,65 @@ async fn jobs_detail_handler(
.map_err(|_| (StatusCode::BAD_REQUEST, "Invalid job ID".to_string()))?;
// Try sandbox job from DB first, scoped to the authenticated user.
if let Some(ref store) = state.store {
if let Ok(Some(job)) = store.get_sandbox_job(job_id).await {
if job.user_id != state.user_id {
return Err((StatusCode::NOT_FOUND, "Job not found".to_string()));
}
let browse_id = std::path::Path::new(&job.project_dir)
.file_name()
.map(|n| n.to_string_lossy().to_string())
.unwrap_or_else(|| job.id.to_string());
let ui_state = match job.status.as_str() {
"creating" => "pending",
"running" => "in_progress",
s => s,
};
let elapsed_secs = job.started_at.map(|start| {
let end = job.completed_at.unwrap_or_else(chrono::Utc::now);
(end - start).num_seconds().max(0) as u64
});
// Synthesize transitions from timestamps.
let mut transitions = Vec::new();
if let Some(started) = job.started_at {
transitions.push(TransitionInfo {
from: "creating".to_string(),
to: "running".to_string(),
timestamp: started.to_rfc3339(),
reason: None,
});
}
if let Some(completed) = job.completed_at {
transitions.push(TransitionInfo {
from: "running".to_string(),
to: job.status.clone(),
timestamp: completed.to_rfc3339(),
reason: job.failure_reason.clone(),
});
}
return Ok(Json(JobDetailResponse {
id: job.id,
title: job.task.clone(),
description: String::new(),
state: ui_state.to_string(),
user_id: job.user_id.clone(),
created_at: job.created_at.to_rfc3339(),
started_at: job.started_at.map(|dt| dt.to_rfc3339()),
completed_at: job.completed_at.map(|dt| dt.to_rfc3339()),
elapsed_secs,
project_dir: Some(job.project_dir.clone()),
browse_url: Some(format!("/projects/{}/", browse_id)),
job_mode: {
let mode = store.get_sandbox_job_mode(job.id).await.ok().flatten();
mode.filter(|m| m != "worker")
},
transitions,
}));
if let Some(ref store) = state.store
&& let Ok(Some(job)) = store.get_sandbox_job(job_id).await
{
if job.user_id != state.user_id {
return Err((StatusCode::NOT_FOUND, "Job not found".to_string()));
}
let browse_id = std::path::Path::new(&job.project_dir)
.file_name()
.map(|n| n.to_string_lossy().to_string())
.unwrap_or_else(|| job.id.to_string());
let ui_state = match job.status.as_str() {
"creating" => "pending",
"running" => "in_progress",
s => s,
};
let elapsed_secs = job.started_at.map(|start| {
let end = job.completed_at.unwrap_or_else(chrono::Utc::now);
(end - start).num_seconds().max(0) as u64
});
// Synthesize transitions from timestamps.
let mut transitions = Vec::new();
if let Some(started) = job.started_at {
transitions.push(TransitionInfo {
from: "creating".to_string(),
to: "running".to_string(),
timestamp: started.to_rfc3339(),
reason: None,
});
}
if let Some(completed) = job.completed_at {
transitions.push(TransitionInfo {
from: "running".to_string(),
to: job.status.clone(),
timestamp: completed.to_rfc3339(),
reason: job.failure_reason.clone(),
});
}
return Ok(Json(JobDetailResponse {
id: job.id,
title: job.task.clone(),
description: String::new(),
state: ui_state.to_string(),
user_id: job.user_id.clone(),
created_at: job.created_at.to_rfc3339(),
started_at: job.started_at.map(|dt| dt.to_rfc3339()),
completed_at: job.completed_at.map(|dt| dt.to_rfc3339()),
elapsed_secs,
project_dir: Some(job.project_dir.clone()),
browse_url: Some(format!("/projects/{}/", browse_id)),
job_mode: {
let mode = store.get_sandbox_job_mode(job.id).await.ok().flatten();
mode.filter(|m| m != "worker")
},
transitions,
}));
}
Err((StatusCode::NOT_FOUND, "Job not found".to_string()))
@@ -1198,35 +1198,35 @@ async fn jobs_cancel_handler(
.map_err(|_| (StatusCode::BAD_REQUEST, "Invalid job ID".to_string()))?;
// Try sandbox job cancellation, scoped to the authenticated user.
if let Some(ref store) = state.store {
if let Ok(Some(job)) = store.get_sandbox_job(job_id).await {
if job.user_id != state.user_id {
return Err((StatusCode::NOT_FOUND, "Job not found".to_string()));
}
if job.status == "running" || job.status == "creating" {
// Stop the container if we have a job manager.
if let Some(ref jm) = state.job_manager {
if let Err(e) = jm.stop_job(job_id).await {
tracing::warn!(job_id = %job_id, error = %e, "Failed to stop container during cancellation");
}
}
store
.update_sandbox_job_status(
job_id,
"failed",
Some(false),
Some("Cancelled by user"),
None,
Some(chrono::Utc::now()),
)
.await
.map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, e.to_string()))?;
}
return Ok(Json(serde_json::json!({
"status": "cancelled",
"job_id": job_id,
})));
if let Some(ref store) = state.store
&& let Ok(Some(job)) = store.get_sandbox_job(job_id).await
{
if job.user_id != state.user_id {
return Err((StatusCode::NOT_FOUND, "Job not found".to_string()));
}
if job.status == "running" || job.status == "creating" {
// Stop the container if we have a job manager.
if let Some(ref jm) = state.job_manager
&& let Err(e) = jm.stop_job(job_id).await
{
tracing::warn!(job_id = %job_id, error = %e, "Failed to stop container during cancellation");
}
store
.update_sandbox_job_status(
job_id,
"failed",
Some(false),
Some("Cancelled by user"),
None,
Some(chrono::Utc::now()),
)
.await
.map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, e.to_string()))?;
}
return Ok(Json(serde_json::json!({
"status": "cancelled",
"job_id": job_id,
})));
}
Err((StatusCode::NOT_FOUND, "Job not found".to_string()))
@@ -1334,14 +1334,13 @@ async fn jobs_prompt_handler(
.map_err(|_| (StatusCode::BAD_REQUEST, "Invalid job ID".to_string()))?;
// Verify user owns this job.
if let Some(ref store) = state.store {
if !store
if let Some(ref store) = state.store
&& !store
.sandbox_job_belongs_to_user(job_id, &state.user_id)
.await
.unwrap_or(false)
{
return Err((StatusCode::NOT_FOUND, "Job not found".to_string()));
}
{
return Err((StatusCode::NOT_FOUND, "Job not found".to_string()));
}
let content = body
+28 -60
View File
@@ -48,8 +48,6 @@ pub enum ConfigCommand {
/// Connects to the database to read/write settings. Falls back to disk
/// if the database is not available.
pub async fn run_config_command(cmd: ConfigCommand) -> anyhow::Result<()> {
let _ = dotenvy::dotenv();
// Try to connect to the DB for settings access
let db: Option<Arc<dyn crate::db::Database>> = match connect_db().await {
Ok(d) => Some(d),
@@ -92,7 +90,7 @@ async fn load_settings(store: Option<&dyn crate::db::Database>) -> Settings {
_ => {}
}
}
Settings::load()
Settings::default()
}
/// List all settings.
@@ -110,10 +108,10 @@ async fn list_settings(
println!();
for (key, value) in all {
if let Some(ref f) = filter {
if !key.starts_with(f) {
continue;
}
if let Some(ref f) = filter
&& !key.starts_with(f)
{
continue;
}
let display_value = if value.len() > 60 {
@@ -155,19 +153,17 @@ async fn set_setting(
.set(path, value)
.map_err(|e| anyhow::anyhow!("{}", e))?;
// Save to DB if available, otherwise disk
if let Some(store) = store {
let json_value = match serde_json::from_str::<serde_json::Value>(value) {
Ok(v) => v,
Err(_) => serde_json::Value::String(value.to_string()),
};
store
.set_setting(DEFAULT_USER_ID, path, &json_value)
.await
.map_err(|e| anyhow::anyhow!("Failed to save to database: {}", e))?;
} else {
settings.save()?;
}
let store = store.ok_or_else(|| {
anyhow::anyhow!("Database connection required to save settings. Check DATABASE_URL.")
})?;
let json_value = match serde_json::from_str::<serde_json::Value>(value) {
Ok(v) => v,
Err(_) => serde_json::Value::String(value.to_string()),
};
store
.set_setting(DEFAULT_USER_ID, path, &json_value)
.await
.map_err(|e| anyhow::anyhow!("Failed to save to database: {}", e))?;
println!("Set {} = {}", path, value);
Ok(())
@@ -180,17 +176,13 @@ async fn reset_setting(store: Option<&dyn crate::db::Database>, path: &str) -> a
.get(path)
.ok_or_else(|| anyhow::anyhow!("Unknown setting: {}", path))?;
// Delete from DB (falling back to default) or reset on disk
if let Some(store) = store {
store
.delete_setting(DEFAULT_USER_ID, path)
.await
.map_err(|e| anyhow::anyhow!("Failed to delete setting from database: {}", e))?;
} else {
let mut settings = Settings::load();
settings.reset(path).map_err(|e| anyhow::anyhow!("{}", e))?;
settings.save()?;
}
let store = store.ok_or_else(|| {
anyhow::anyhow!("Database connection required to reset settings. Check DATABASE_URL.")
})?;
store
.delete_setting(DEFAULT_USER_ID, path)
.await
.map_err(|e| anyhow::anyhow!("Failed to delete setting from database: {}", e))?;
println!("Reset {} to default: {}", path, default_value);
Ok(())
@@ -200,37 +192,13 @@ async fn reset_setting(store: Option<&dyn crate::db::Database>, path: &str) -> a
fn show_path(has_db: bool) -> anyhow::Result<()> {
if has_db {
println!("Settings stored in: database (settings table)");
println!(
"Bootstrap config: {}",
crate::bootstrap::BootstrapConfig::default_path().display()
);
} else {
let path = Settings::default_path();
println!("Settings stored in: {} (disk fallback)", path.display());
if path.exists() {
let metadata = std::fs::metadata(&path)?;
println!(" Size: {} bytes", metadata.len());
if let Ok(modified) = metadata.modified() {
use std::time::SystemTime;
let duration = SystemTime::now()
.duration_since(modified)
.unwrap_or_default();
let secs = duration.as_secs();
if secs < 60 {
println!(" Modified: {} seconds ago", secs);
} else if secs < 3600 {
println!(" Modified: {} minutes ago", secs / 60);
} else if secs < 86400 {
println!(" Modified: {} hours ago", secs / 3600);
} else {
println!(" Modified: {} days ago", secs / 86400);
}
}
} else {
println!(" (does not exist, using defaults)");
}
println!("Settings stored in: PostgreSQL (not connected, using defaults)");
}
println!(
"Env config: {}",
crate::bootstrap::ironclaw_env_path().display()
);
Ok(())
}
+1
View File
@@ -12,6 +12,7 @@
mod config;
mod mcp;
pub mod memory;
pub mod oauth_defaults;
mod pairing;
pub mod status;
mod tool;
+342
View File
@@ -0,0 +1,342 @@
//! Shared OAuth infrastructure: built-in credentials, callback server, landing pages.
//!
//! Every OAuth flow in the codebase (WASM tool auth, MCP server auth, NEAR AI login)
//! uses the same callback port, landing page, and listener logic from this module.
//!
//! # Built-in Credentials
//!
//! Many CLI tools (gcloud, rclone, gdrive) ship with default OAuth credentials
//! so users don't need to register their own OAuth app. Google explicitly
//! documents that client_secret for "Desktop App" / "Installed App" types
//! is NOT actually secret.
//!
//! Default credentials are hardcoded below. They can be overridden at:
//!
//! - **Compile time**: Set IRONCLAW_GOOGLE_CLIENT_ID / IRONCLAW_GOOGLE_CLIENT_SECRET
//! env vars before building to replace the hardcoded defaults.
//! - **Runtime**: Users can set GOOGLE_OAUTH_CLIENT_ID / GOOGLE_OAUTH_CLIENT_SECRET
//! env vars, which take priority over built-in defaults.
use std::time::Duration;
use tokio::io::{AsyncBufReadExt, AsyncWriteExt, BufReader};
use tokio::net::TcpListener;
// ── Built-in credentials ────────────────────────────────────────────────
pub struct OAuthCredentials {
pub client_id: &'static str,
pub client_secret: &'static str,
}
/// Google OAuth "Desktop App" credentials, shared across all Google tools.
/// Compile-time env vars override the hardcoded defaults below.
const GOOGLE_CLIENT_ID: &str = match option_env!("IRONCLAW_GOOGLE_CLIENT_ID") {
Some(v) => v,
None => "564604149681-efo25d43rs85v0tibdepsmdv5dsrhhr0.apps.googleusercontent.com",
};
const GOOGLE_CLIENT_SECRET: &str = match option_env!("IRONCLAW_GOOGLE_CLIENT_SECRET") {
Some(v) => v,
None => "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",
};
/// Returns built-in OAuth credentials for a provider, keyed by secret_name.
///
/// The secret_name comes from the tool's capabilities.json `auth.secret_name` field.
/// Returns `None` if no built-in credentials are configured for that provider.
pub fn builtin_credentials(secret_name: &str) -> Option<OAuthCredentials> {
match secret_name {
"google_oauth_token" => Some(OAuthCredentials {
client_id: GOOGLE_CLIENT_ID,
client_secret: GOOGLE_CLIENT_SECRET,
}),
_ => None,
}
}
// ── Shared callback server ──────────────────────────────────────────────
/// Fixed port for all OAuth callbacks.
///
/// Every redirect URI registered with providers must use this port:
/// `http://localhost:9876/callback` (or `/auth/callback` for NEAR AI).
pub const OAUTH_CALLBACK_PORT: u16 = 9876;
/// Error from the OAuth callback listener.
#[derive(Debug, thiserror::Error)]
pub enum OAuthCallbackError {
#[error("Port {0} is in use (another auth flow running?): {1}")]
PortInUse(u16, String),
#[error("Authorization denied by user")]
Denied,
#[error("Timed out waiting for authorization")]
Timeout,
#[error("IO error: {0}")]
Io(String),
}
/// Bind the OAuth callback listener on the fixed port.
///
/// Binds to IPv4 `127.0.0.1` first because callback URLs use `127.0.0.1`
/// explicitly (e.g., NEAR AI redirects to `http://127.0.0.1:9876/auth/callback`).
/// Falls back to IPv6 `[::1]` only if IPv4 binding fails for a reason other
/// than `AddrInUse`. If the port is already occupied, fails immediately.
pub async fn bind_callback_listener() -> Result<TcpListener, OAuthCallbackError> {
let ipv4_addr = format!("127.0.0.1:{}", OAUTH_CALLBACK_PORT);
match TcpListener::bind(&ipv4_addr).await {
Ok(listener) => return Ok(listener),
Err(e) if e.kind() == std::io::ErrorKind::AddrInUse => {
return Err(OAuthCallbackError::PortInUse(
OAUTH_CALLBACK_PORT,
e.to_string(),
));
}
Err(_) => {
// IPv4 not available, fall back to IPv6
}
}
TcpListener::bind(format!("[::1]:{}", OAUTH_CALLBACK_PORT))
.await
.map_err(|e| {
if e.kind() == std::io::ErrorKind::AddrInUse {
OAuthCallbackError::PortInUse(OAUTH_CALLBACK_PORT, e.to_string())
} else {
OAuthCallbackError::Io(e.to_string())
}
})
}
/// Wait for an OAuth callback and extract a query parameter value.
///
/// Listens for a GET request matching `path_prefix` (e.g., "/callback" or "/auth/callback"),
/// extracts the value of `param_name` (e.g., "code" or "token"), and shows a branded
/// landing page using `display_name` (e.g., "Google", "Notion", "NEAR AI").
///
/// Times out after 5 minutes.
pub async fn wait_for_callback(
listener: TcpListener,
path_prefix: &str,
param_name: &str,
display_name: &str,
) -> Result<String, OAuthCallbackError> {
let path_prefix = path_prefix.to_string();
let param_name = param_name.to_string();
let display_name = display_name.to_string();
tokio::time::timeout(Duration::from_secs(300), async move {
loop {
let (mut socket, _) = listener
.accept()
.await
.map_err(|e| OAuthCallbackError::Io(e.to_string()))?;
let mut reader = BufReader::new(&mut socket);
let mut request_line = String::new();
reader
.read_line(&mut request_line)
.await
.map_err(|e| OAuthCallbackError::Io(e.to_string()))?;
if let Some(path) = request_line.split_whitespace().nth(1)
&& path.starts_with(&path_prefix)
&& let Some(query) = path.split('?').nth(1)
{
// Check for error first
if query.contains("error=") {
let html = landing_html(&display_name, false);
let response = format!(
"HTTP/1.1 400 Bad Request\r\n\
Content-Type: text/html; charset=utf-8\r\n\
Connection: close\r\n\
\r\n\
{}",
html
);
let _ = socket.write_all(response.as_bytes()).await;
return Err(OAuthCallbackError::Denied);
}
// Look for the target parameter
for param in query.split('&') {
let parts: Vec<&str> = param.splitn(2, '=').collect();
if parts.len() == 2 && parts[0] == param_name {
let value = urlencoding::decode(parts[1])
.unwrap_or_else(|_| parts[1].into())
.into_owned();
let html = landing_html(&display_name, true);
let response = format!(
"HTTP/1.1 200 OK\r\n\
Content-Type: text/html; charset=utf-8\r\n\
Connection: close\r\n\
\r\n\
{}",
html
);
let _ = socket.write_all(response.as_bytes()).await;
let _ = socket.shutdown().await;
return Ok(value);
}
}
}
// Not the callback we're looking for
let response = "HTTP/1.1 404 Not Found\r\nConnection: close\r\n\r\n";
let _ = socket.write_all(response.as_bytes()).await;
}
})
.await
.map_err(|_| OAuthCallbackError::Timeout)?
}
/// Escape a string for safe interpolation into HTML content.
fn html_escape(s: &str) -> String {
let mut out = String::with_capacity(s.len());
for c in s.chars() {
match c {
'&' => out.push_str("&amp;"),
'<' => out.push_str("&lt;"),
'>' => out.push_str("&gt;"),
'"' => out.push_str("&quot;"),
'\'' => out.push_str("&#x27;"),
_ => out.push(c),
}
}
out
}
/// HTML landing page shown in the browser after an OAuth redirect.
pub fn landing_html(provider_name: &str, success: bool) -> String {
let safe_name = html_escape(provider_name);
let (icon, heading, subtitle, accent) = if success {
(
r##"<div style="width:64px;height:64px;border-radius:50%;background:#22c55e;display:flex;align-items:center;justify-content:center;margin:0 auto 24px">
<svg width="32" height="32" viewBox="0 0 24 24" fill="none" stroke="#fff" stroke-width="3" stroke-linecap="round" stroke-linejoin="round"><polyline points="20 6 9 17 4 12"/></svg>
</div>"##,
format!("{} Connected", safe_name),
"You can close this window and return to your terminal.",
"#22c55e",
)
} else {
(
r##"<div style="width:64px;height:64px;border-radius:50%;background:#ef4444;display:flex;align-items:center;justify-content:center;margin:0 auto 24px">
<svg width="32" height="32" viewBox="0 0 24 24" fill="none" stroke="#fff" stroke-width="3" stroke-linecap="round" stroke-linejoin="round"><line x1="18" y1="6" x2="6" y2="18"/><line x1="6" y1="6" x2="18" y2="18"/></svg>
</div>"##,
"Authorization Failed".to_string(),
"The request was denied. You can close this window and try again.",
"#ef4444",
)
};
format!(
r#"<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width,initial-scale=1">
<title>IronClaw - {heading}</title>
<style>
* {{ margin:0; padding:0; box-sizing:border-box }}
body {{
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif;
background: #0a0a0a;
color: #e5e5e5;
display: flex;
justify-content: center;
align-items: center;
min-height: 100vh;
}}
.card {{
text-align: center;
padding: 48px 40px;
max-width: 420px;
border: 1px solid #262626;
border-radius: 16px;
background: #141414;
}}
h1 {{
font-size: 22px;
font-weight: 600;
margin-bottom: 8px;
color: #fafafa;
}}
p {{
font-size: 14px;
color: #a3a3a3;
line-height: 1.5;
}}
.accent {{ color: {accent}; }}
.brand {{
margin-top: 32px;
font-size: 12px;
color: #525252;
letter-spacing: 0.5px;
text-transform: uppercase;
}}
</style>
</head>
<body>
<div class="card">
{icon}
<h1>{heading}</h1>
<p>{subtitle}</p>
<div class="brand">IronClaw</div>
</div>
</body>
</html>"#,
heading = heading,
icon = icon,
subtitle = subtitle,
accent = accent,
)
}
#[cfg(test)]
mod tests {
use crate::cli::oauth_defaults::{builtin_credentials, landing_html};
#[test]
fn test_unknown_provider_returns_none() {
assert!(builtin_credentials("unknown_token").is_none());
}
#[test]
fn test_google_returns_based_on_compile_env() {
let creds = builtin_credentials("google_oauth_token");
assert!(creds.is_some());
let creds = creds.unwrap();
assert!(!creds.client_id.is_empty());
assert!(!creds.client_secret.is_empty());
}
#[test]
fn test_landing_html_success_contains_key_elements() {
let html = landing_html("Google", true);
assert!(html.contains("Google Connected"));
assert!(html.contains("charset"));
assert!(html.contains("IronClaw"));
assert!(html.contains("#22c55e")); // green accent
assert!(!html.contains("Failed"));
}
#[test]
fn test_landing_html_escapes_provider_name() {
let html = landing_html("<script>alert(1)</script>", true);
assert!(!html.contains("<script>"));
assert!(html.contains("&lt;script&gt;"));
}
#[test]
fn test_landing_html_error_contains_key_elements() {
let html = landing_html("Notion", false);
assert!(html.contains("Authorization Failed"));
assert!(html.contains("charset"));
assert!(html.contains("IronClaw"));
assert!(html.contains("#ef4444")); // red accent
assert!(!html.contains("Connected"));
}
}
+44 -24
View File
@@ -9,7 +9,7 @@ use crate::settings::Settings;
/// Run the status command, printing system health info.
pub async fn run_status_command() -> anyhow::Result<()> {
let settings = Settings::load();
let settings = Settings::default();
println!("IronClaw Status");
println!("===============\n");
@@ -22,16 +22,36 @@ pub async fn run_status_command() -> anyhow::Result<()> {
);
// Database
let db_url_set = settings.database_url.is_some() || std::env::var("DATABASE_URL").is_ok();
print!(" Database: ");
if db_url_set {
// Try to connect
match check_database().await {
Ok(()) => println!("connected"),
Err(e) => println!("error ({})", e),
let db_backend = std::env::var("DATABASE_BACKEND")
.ok()
.unwrap_or_else(|| "postgres".to_string());
match db_backend.as_str() {
"libsql" | "turso" | "sqlite" => {
let path = std::env::var("LIBSQL_PATH")
.map(std::path::PathBuf::from)
.unwrap_or_else(|_| crate::config::default_libsql_path());
if path.exists() {
let turso = if std::env::var("LIBSQL_URL").is_ok() {
" + Turso sync"
} else {
""
};
println!("libSQL ({}{})", path.display(), turso);
} else {
println!("libSQL (file missing: {})", path.display());
}
}
_ => {
if std::env::var("DATABASE_URL").is_ok() {
match check_database().await {
Ok(()) => println!("connected (PostgreSQL)"),
Err(e) => println!("error ({})", e),
}
} else {
println!("not configured");
}
}
} else {
println!("not configured");
}
// Session / Auth
@@ -43,15 +63,17 @@ pub async fn run_status_command() -> anyhow::Result<()> {
println!("not found (run `ironclaw onboard`)");
}
// Secrets
// Secrets (auto-detect from env only; skip keychain probe to avoid
// triggering macOS system password dialogs on a simple status check)
print!(" Secrets: ");
let secrets_configured = settings.secrets_master_key_source != crate::settings::KeySource::None
|| std::env::var("SECRETS_MASTER_KEY").is_ok()
|| crate::secrets::keychain::has_master_key().await;
if secrets_configured {
println!("configured ({:?})", settings.secrets_master_key_source);
if std::env::var("SECRETS_MASTER_KEY").is_ok() {
println!("configured (env)");
} else {
println!("not configured");
// We don't probe the keychain here because get_generic_password()
// triggers macOS unlock+authorization dialogs, which is bad UX for
// a read-only status command. If onboarding completed with keychain
// storage, the key is there; we just can't cheaply verify it.
println!("env not set (keychain may be configured)");
}
// Embeddings
@@ -129,20 +151,18 @@ pub async fn run_status_command() -> anyhow::Result<()> {
Err(_) => println!("none configured"),
}
// Settings path
println!("\n Settings: {}", Settings::default_path().display());
// Config path
println!(
"\n Config: {}",
crate::bootstrap::ironclaw_env_path().display()
);
Ok(())
}
#[cfg(feature = "postgres")]
async fn check_database() -> anyhow::Result<()> {
let _ = dotenvy::dotenv();
let settings = Settings::load();
let url = std::env::var("DATABASE_URL")
.ok()
.or(settings.database_url)
.ok_or_else(|| anyhow::anyhow!("no URL"))?;
let url = std::env::var("DATABASE_URL").map_err(|_| anyhow::anyhow!("DATABASE_URL not set"))?;
let config: deadpool_postgres::Config = deadpool_postgres::Config {
url: Some(url),
+175 -153
View File
@@ -423,11 +423,11 @@ async fn extract_crate_name(cargo_toml: &Path) -> anyhow::Result<String> {
// Simple TOML parsing for [package] name
for line in content.lines() {
let line = line.trim();
if line.starts_with("name") {
if let Some((_, value)) = line.split_once('=') {
let name = value.trim().trim_matches('"').trim_matches('\'');
return Ok(name.to_string());
}
if line.starts_with("name")
&& let Some((_, value)) = line.split_once('=')
{
let name = value.trim().trim_matches('"').trim_matches('\'');
return Ok(name.to_string());
}
}
@@ -491,10 +491,10 @@ async fn list_tools(dir: Option<PathBuf>, verbose: bool) -> anyhow::Result<()> {
if has_caps {
let caps_path = path.with_extension("capabilities.json");
if let Ok(content) = fs::read_to_string(&caps_path).await {
if let Ok(caps) = CapabilitiesFile::from_json(&content) {
print_capabilities_summary(&caps);
}
if let Ok(content) = fs::read_to_string(&caps_path).await
&& let Ok(caps) = CapabilitiesFile::from_json(&content)
{
print_capabilities_summary(&caps);
}
}
println!();
@@ -607,16 +607,16 @@ fn print_capabilities_summary(caps: &CapabilitiesFile) {
}
}
if let Some(ref secrets) = caps.secrets {
if !secrets.allowed_names.is_empty() {
parts.push(format!("secrets: {}", secrets.allowed_names.len()));
}
if let Some(ref secrets) = caps.secrets
&& !secrets.allowed_names.is_empty()
{
parts.push(format!("secrets: {}", secrets.allowed_names.len()));
}
if let Some(ref ws) = caps.workspace {
if !ws.allowed_prefixes.is_empty() {
parts.push("workspace: read".to_string());
}
if let Some(ref ws) = caps.workspace
&& !ws.allowed_prefixes.is_empty()
{
parts.push("workspace: read".to_string());
}
if !parts.is_empty() {
@@ -653,30 +653,30 @@ fn print_capabilities_detail(caps: &CapabilitiesFile) {
}
}
if let Some(ref secrets) = caps.secrets {
if !secrets.allowed_names.is_empty() {
println!(" Secrets (existence check only):");
for name in &secrets.allowed_names {
println!(" {}", name);
}
if let Some(ref secrets) = caps.secrets
&& !secrets.allowed_names.is_empty()
{
println!(" Secrets (existence check only):");
for name in &secrets.allowed_names {
println!(" {}", name);
}
}
if let Some(ref tool_invoke) = caps.tool_invoke {
if !tool_invoke.aliases.is_empty() {
println!(" Tool aliases:");
for (alias, real_name) in &tool_invoke.aliases {
println!(" {} -> {}", alias, real_name);
}
if let Some(ref tool_invoke) = caps.tool_invoke
&& !tool_invoke.aliases.is_empty()
{
println!(" Tool aliases:");
for (alias, real_name) in &tool_invoke.aliases {
println!(" {} -> {}", alias, real_name);
}
}
if let Some(ref ws) = caps.workspace {
if !ws.allowed_prefixes.is_empty() {
println!(" Workspace read prefixes:");
for prefix in &ws.allowed_prefixes {
println!(" {}", prefix);
}
if let Some(ref ws) = caps.workspace
&& !ws.allowed_prefixes.is_empty()
{
println!(" Workspace read prefixes:");
for prefix in &ws.allowed_prefixes {
println!(" {}", prefix);
}
}
}
@@ -802,48 +802,100 @@ async fn auth_tool(name: String, dir: Option<PathBuf>, user_id: String) -> anyho
}
// Check for environment variable
if let Some(ref env_var) = auth.env_var {
if let Ok(token) = std::env::var(env_var) {
if !token.is_empty() {
println!(" Found {} in environment.", env_var);
println!();
if let Some(ref env_var) = auth.env_var
&& let Ok(token) = std::env::var(env_var)
&& !token.is_empty()
{
println!(" Found {} in environment.", env_var);
println!();
// Validate if endpoint is provided
if let Some(ref validation) = auth.validation_endpoint {
print!(" Validating token...");
std::io::stdout().flush()?;
// Validate if endpoint is provided
if let Some(ref validation) = auth.validation_endpoint {
print!(" Validating token...");
std::io::stdout().flush()?;
match validate_token(&token, validation, &auth.secret_name).await {
Ok(()) => {
println!("");
}
Err(e) => {
println!("");
println!(" Validation failed: {}", e);
println!();
println!(" Falling back to manual entry...");
return auth_tool_manual(secrets_store.as_ref(), &user_id, &auth).await;
}
}
match validate_token(&token, validation, &auth.secret_name).await {
Ok(()) => {
println!("");
}
Err(e) => {
println!("");
println!(" Validation failed: {}", e);
println!();
println!(" Falling back to manual entry...");
return auth_tool_manual(secrets_store.as_ref(), &user_id, &auth).await;
}
// Save the token
save_token(secrets_store.as_ref(), &user_id, &auth, &token).await?;
print_success(display_name);
return Ok(());
}
}
// Save the token
save_token(secrets_store.as_ref(), &user_id, &auth, &token, None, None).await?;
print_success(display_name);
return Ok(());
}
// Check for OAuth configuration
if let Some(ref oauth) = auth.oauth {
return auth_tool_oauth(secrets_store.as_ref(), &user_id, &auth, oauth).await;
// For providers with shared tokens (e.g., all Google tools share google_oauth_token),
// combine scopes from all installed tools so one auth covers everything.
let combined = combine_provider_scopes(&tools_dir, &auth.secret_name, oauth).await;
if combined.scopes.len() > oauth.scopes.len() {
let extra = combined.scopes.len() - oauth.scopes.len();
println!(
" Including scopes from {} other installed tool(s) sharing this credential.",
extra
);
println!();
}
return auth_tool_oauth(secrets_store.as_ref(), &user_id, &auth, &combined).await;
}
// Fall back to manual entry
auth_tool_manual(secrets_store.as_ref(), &user_id, &auth).await
}
/// Scan the tools directory for all capabilities files sharing the same secret_name
/// and combine their OAuth scopes. This way, authing any Google tool requests scopes
/// for ALL installed Google tools, so one login covers everything.
async fn combine_provider_scopes(
tools_dir: &Path,
secret_name: &str,
base_oauth: &crate::tools::wasm::OAuthConfigSchema,
) -> crate::tools::wasm::OAuthConfigSchema {
let mut all_scopes: std::collections::HashSet<String> =
base_oauth.scopes.iter().cloned().collect();
if let Ok(mut entries) = tokio::fs::read_dir(tools_dir).await {
while let Ok(Some(entry)) = entries.next_entry().await {
let path = entry.path();
if path.extension().and_then(|e| e.to_str()) != Some("json") {
continue;
}
let name = path
.file_name()
.and_then(|n| n.to_str())
.unwrap_or_default();
if !name.ends_with(".capabilities.json") {
continue;
}
if let Ok(content) = tokio::fs::read_to_string(&path).await
&& let Ok(caps) = CapabilitiesFile::from_json(&content)
&& let Some(auth) = &caps.auth
&& auth.secret_name == secret_name
&& let Some(oauth) = &auth.oauth
{
all_scopes.extend(oauth.scopes.iter().cloned());
}
}
}
let mut combined = base_oauth.clone();
combined.scopes = all_scopes.into_iter().collect();
combined.scopes.sort(); // deterministic ordering
combined
}
/// OAuth browser-based login flow.
async fn auth_tool_oauth(
store: &(dyn SecretsStore + Send + Sync),
@@ -854,12 +906,14 @@ async fn auth_tool_oauth(
use base64::{Engine, engine::general_purpose::URL_SAFE_NO_PAD};
use rand::RngCore;
use sha2::{Digest, Sha256};
use tokio::io::{AsyncBufReadExt, AsyncWriteExt, BufReader};
use tokio::net::TcpListener;
use crate::cli::oauth_defaults::{self, OAUTH_CALLBACK_PORT};
let display_name = auth.display_name.as_deref().unwrap_or(&auth.secret_name);
// Get client_id from config or env
// Get client_id: capabilities file > runtime env var > built-in defaults
let builtin = oauth_defaults::builtin_credentials(&auth.secret_name);
let client_id = oauth
.client_id
.clone()
@@ -869,41 +923,32 @@ async fn auth_tool_oauth(
.as_ref()
.and_then(|env| std::env::var(env).ok())
})
.or_else(|| builtin.as_ref().map(|c| c.client_id.to_string()))
.ok_or_else(|| {
anyhow::anyhow!(
"OAuth client_id not configured.\n\
Set it in the capabilities file or via environment variable."
Set {} env var, or build with IRONCLAW_GOOGLE_CLIENT_ID.",
oauth.client_id_env.as_deref().unwrap_or("the client_id")
)
})?;
// Get client_secret if provided
let client_secret = oauth.client_secret.clone().or_else(|| {
oauth
.client_secret_env
.as_ref()
.and_then(|env| std::env::var(env).ok())
});
// Get client_secret: capabilities file > runtime env var > built-in defaults
let client_secret = oauth
.client_secret
.clone()
.or_else(|| {
oauth
.client_secret_env
.as_ref()
.and_then(|env| std::env::var(env).ok())
})
.or_else(|| builtin.as_ref().map(|c| c.client_secret.to_string()));
println!(" Starting OAuth authentication...");
println!();
// Find an available port for the callback
let mut listener = None;
let mut port = 0;
for p in 9876..=9886 {
match TcpListener::bind(format!("127.0.0.1:{}", p)).await {
Ok(l) => {
listener = Some(l);
port = p;
break;
}
Err(_) => continue,
}
}
let listener = listener.ok_or_else(|| anyhow::anyhow!("Could not find available port"))?;
let redirect_uri = format!("http://localhost:{}/callback", port);
let listener = oauth_defaults::bind_callback_listener().await?;
let redirect_uri = format!("http://localhost:{}/callback", OAUTH_CALLBACK_PORT);
// Generate PKCE verifier and challenge
let (code_verifier, code_challenge) = if oauth.use_pkce {
@@ -962,65 +1007,8 @@ async fn auth_tool_oauth(
println!(" Waiting for authorization...");
// Wait for callback with timeout
let timeout = std::time::Duration::from_secs(300);
let code = tokio::time::timeout(timeout, async {
loop {
let (mut socket, _) = listener.accept().await?;
let mut reader = BufReader::new(&mut socket);
let mut request_line = String::new();
reader.read_line(&mut request_line).await?;
// Parse GET /callback?code=xxx HTTP/1.1
if let Some(path) = request_line.split_whitespace().nth(1) {
if path.starts_with("/callback") {
if let Some(query) = path.split('?').nth(1) {
for param in query.split('&') {
let parts: Vec<&str> = param.splitn(2, '=').collect();
if parts.len() == 2 && parts[0] == "code" {
let code = urlencoding::decode(parts[1])
.unwrap_or_else(|_| parts[1].into())
.into_owned();
// Send success response
let response = format!(
"HTTP/1.1 200 OK\r\n\
Content-Type: text/html\r\n\
\r\n\
<!DOCTYPE html><html><body style=\"font-family: sans-serif; \
display: flex; justify-content: center; align-items: center; \
height: 100vh; margin: 0; background: #191919; color: white;\">\
<div style=\"text-align: center;\">\
<h1>✓ {} Connected!</h1>\
<p>You can close this window.</p>\
</div></body></html>",
display_name
);
let _ = socket.write_all(response.as_bytes()).await;
let _ = socket.shutdown().await;
return Ok::<_, anyhow::Error>(code);
}
}
// Check for error
if query.contains("error=") {
let response =
"HTTP/1.1 400 Bad Request\r\n\r\nAuthorization denied";
let _ = socket.write_all(response.as_bytes()).await;
return Err(anyhow::anyhow!("Authorization denied by user"));
}
}
}
}
let response = "HTTP/1.1 404 Not Found\r\n\r\n";
let _ = socket.write_all(response.as_bytes()).await;
}
})
.await
.map_err(|_| anyhow::anyhow!("Timed out waiting for authorization"))??;
let code =
oauth_defaults::wait_for_callback(listener, "/callback", "code", display_name).await?;
println!();
println!(" Exchanging code for token...");
@@ -1071,8 +1059,19 @@ async fn auth_tool_oauth(
)
})?;
// Save the token
save_token(store, user_id, auth, access_token).await?;
let refresh_token = token_data.get("refresh_token").and_then(|v| v.as_str());
let expires_in = token_data.get("expires_in").and_then(|v| v.as_u64());
// Save the token (with refresh token and expiry if provided)
save_token(
store,
user_id,
auth,
access_token,
refresh_token,
expires_in,
)
.await?;
// Extract any additional info for display
let workspace_name = token_data
@@ -1174,8 +1173,8 @@ async fn auth_tool_manual(
}
}
// Save the token
save_token(store, user_id, auth, &token).await?;
// Save the token (manual path: no refresh token or expiry)
save_token(store, user_id, auth, &token, None, None).await?;
print_success(display_name);
Ok(())
}
@@ -1266,11 +1265,16 @@ async fn validate_token(
}
/// Save token to secrets store.
///
/// Optionally stores a refresh token (as `{secret_name}_refresh_token`) and
/// sets `expires_at` on the access token so the runtime can auto-refresh.
async fn save_token(
store: &(dyn SecretsStore + Send + Sync),
user_id: &str,
auth: &crate::tools::wasm::AuthCapabilitySchema,
token: &str,
refresh_token: Option<&str>,
expires_in: Option<u64>,
) -> anyhow::Result<()> {
let mut params = CreateSecretParams::new(&auth.secret_name, token);
@@ -1278,11 +1282,29 @@ async fn save_token(
params = params.with_provider(provider);
}
if let Some(secs) = expires_in {
let expires_at = chrono::Utc::now() + chrono::Duration::seconds(secs as i64);
params = params.with_expiry(expires_at);
}
store
.create(user_id, params)
.await
.map_err(|e| anyhow::anyhow!("Failed to save token: {}", e))?;
// Store refresh token separately (no expiry, it's long-lived)
if let Some(rt) = refresh_token {
let refresh_name = format!("{}_refresh_token", auth.secret_name);
let mut refresh_params = CreateSecretParams::new(&refresh_name, rt);
if let Some(ref provider) = auth.provider {
refresh_params = refresh_params.with_provider(provider);
}
store
.create(user_id, refresh_params)
.await
.map_err(|e| anyhow::anyhow!("Failed to save refresh token: {}", e))?;
}
Ok(())
}
+141 -75
View File
@@ -1,11 +1,13 @@
//! Configuration for IronClaw.
//!
//! Settings are loaded with priority: env var > database > default.
//! The database replaces the old `settings.json` file for all settings
//! except the 4 bootstrap fields (database_url, pool_size, secrets key
//! source, onboard_completed) which live in `~/.ironclaw/bootstrap.json`.
//! `DATABASE_URL` lives in `~/.ironclaw/.env` (loaded via dotenvy early
//! in startup). Everything else comes from env vars, the DB settings
//! table, or auto-detection.
use std::collections::HashMap;
use std::path::PathBuf;
use std::sync::OnceLock;
use std::time::Duration;
use secrecy::{ExposeSecret, SecretString};
@@ -13,6 +15,13 @@ use secrecy::{ExposeSecret, SecretString};
use crate::error::ConfigError;
use crate::settings::Settings;
/// Thread-safe overlay for injected env vars (secrets loaded from DB).
///
/// Used by `inject_llm_keys_from_secrets()` to make API keys available to
/// `optional_env()` without unsafe `set_var` calls. `optional_env()` checks
/// real env vars first, then falls back to this overlay.
static INJECTED_VARS: OnceLock<HashMap<String, String>> = OnceLock::new();
/// Main configuration for the agent.
#[derive(Debug, Clone)]
pub struct Config {
@@ -40,9 +49,9 @@ impl Config {
pub async fn from_db(
store: &dyn crate::db::Database,
user_id: &str,
bootstrap: &crate::bootstrap::BootstrapConfig,
) -> Result<Self, ConfigError> {
let _ = dotenvy::dotenv();
crate::bootstrap::load_ironclaw_env();
// Load all settings from DB into a Settings struct
let db_settings = match store.get_all_settings(user_id).await {
@@ -53,7 +62,7 @@ impl Config {
}
};
Self::build(bootstrap, &db_settings).await
Self::build(&db_settings).await
}
/// Load configuration from environment variables only (no database).
@@ -61,20 +70,20 @@ impl Config {
/// Used during early startup before the database is connected,
/// and by CLI commands that don't have DB access.
/// Falls back to legacy `settings.json` on disk if present.
///
/// Loads both `./.env` (standard, higher priority) and `~/.ironclaw/.env`
/// (lower priority) via dotenvy, which never overwrites existing vars.
pub async fn from_env() -> Result<Self, ConfigError> {
let _ = dotenvy::dotenv();
let bootstrap = crate::bootstrap::BootstrapConfig::load();
crate::bootstrap::load_ironclaw_env();
let settings = Settings::load();
Self::build(&bootstrap, &settings).await
Self::build(&settings).await
}
/// Build config from bootstrap + settings (shared by from_env and from_db).
async fn build(
bootstrap: &crate::bootstrap::BootstrapConfig,
settings: &Settings,
) -> Result<Self, ConfigError> {
/// Build config from settings (shared by from_env and from_db).
async fn build(settings: &Settings) -> Result<Self, ConfigError> {
Ok(Self {
database: DatabaseConfig::resolve(bootstrap)?,
database: DatabaseConfig::resolve()?,
llm: LlmConfig::resolve(settings)?,
embeddings: EmbeddingsConfig::resolve(settings)?,
tunnel: TunnelConfig::resolve(settings)?,
@@ -82,7 +91,7 @@ impl Config {
agent: AgentConfig::resolve(settings)?,
safety: SafetyConfig::resolve()?,
wasm: WasmConfig::resolve()?,
secrets: SecretsConfig::resolve(bootstrap).await?,
secrets: SecretsConfig::resolve().await?,
builder: BuilderModeConfig::resolve()?,
heartbeat: HeartbeatConfig::resolve(settings)?,
routines: RoutineConfig::resolve()?,
@@ -107,13 +116,13 @@ impl TunnelConfig {
let public_url = optional_env("TUNNEL_URL")?
.or_else(|| settings.tunnel.public_url.clone().filter(|s| !s.is_empty()));
if let Some(ref url) = public_url {
if !url.starts_with("https://") {
return Err(ConfigError::InvalidValue {
key: "TUNNEL_URL".to_string(),
message: "must start with https:// (webhooks require HTTPS)".to_string(),
});
}
if let Some(ref url) = public_url
&& !url.starts_with("https://")
{
return Err(ConfigError::InvalidValue {
key: "TUNNEL_URL".to_string(),
message: "must start with https:// (webhooks require HTTPS)".to_string(),
});
}
Ok(Self { public_url })
@@ -179,7 +188,7 @@ pub struct DatabaseConfig {
}
impl DatabaseConfig {
fn resolve(bootstrap: &crate::bootstrap::BootstrapConfig) -> Result<Self, ConfigError> {
fn resolve() -> Result<Self, ConfigError> {
let backend: DatabaseBackend = if let Some(b) = optional_env("DATABASE_BACKEND")? {
b.parse().map_err(|e| ConfigError::InvalidValue {
key: "DATABASE_BACKEND".to_string(),
@@ -191,8 +200,8 @@ impl DatabaseConfig {
// PostgreSQL URL is required only when using the postgres backend.
// For libsql backend, default to an empty placeholder.
// DATABASE_URL is loaded from ~/.ironclaw/.env via dotenvy early in startup.
let url = optional_env("DATABASE_URL")?
.or_else(|| bootstrap.database_url.clone())
.or_else(|| {
if backend == DatabaseBackend::LibSql {
Some("unused://libsql".to_string())
@@ -205,15 +214,7 @@ impl DatabaseConfig {
hint: "Run 'ironclaw onboard' or set DATABASE_URL environment variable".to_string(),
})?;
let pool_size = optional_env("DATABASE_POOL_SIZE")?
.map(|s| s.parse())
.transpose()
.map_err(|e| ConfigError::InvalidValue {
key: "DATABASE_POOL_SIZE".to_string(),
message: format!("must be a positive integer: {e}"),
})?
.or(bootstrap.database_pool_size)
.unwrap_or(10);
let pool_size = parse_optional_env("DATABASE_POOL_SIZE", 10)?;
let libsql_path = optional_env("LIBSQL_PATH")?.map(PathBuf::from).or_else(|| {
if backend == DatabaseBackend::LibSql {
@@ -387,6 +388,9 @@ impl std::str::FromStr for NearAiApiMode {
pub struct NearAiConfig {
/// Model to use (e.g., "claude-3-5-sonnet-20241022", "gpt-4o")
pub model: String,
/// Cheap/fast model for lightweight tasks (heartbeat, routing, evaluation).
/// Falls back to the main model if not set.
pub cheap_model: Option<String>,
/// Base URL for the NEAR AI API (default: https://api.near.ai)
pub base_url: String,
/// Base URL for auth/refresh endpoints (default: https://private.near.ai)
@@ -397,16 +401,37 @@ pub struct NearAiConfig {
pub api_mode: NearAiApiMode,
/// API key for cloud-api (required for chat_completions mode)
pub api_key: Option<SecretString>,
/// Optional fallback model for failover (default: None).
/// When set, a secondary provider is created with this model and wrapped
/// in a `FailoverProvider` so transient errors on the primary model
/// automatically fall through to the fallback.
pub fallback_model: Option<String>,
/// Maximum number of retries for transient errors (default: 3).
/// With the default of 3, the provider makes up to 4 total attempts
/// (1 initial + 3 retries) before giving up.
pub max_retries: u32,
}
impl LlmConfig {
fn resolve(settings: &Settings) -> Result<Self, ConfigError> {
// Determine backend (default: NearAi)
// Determine backend: env var > settings > default (NearAi)
let backend: LlmBackend = if let Some(b) = optional_env("LLM_BACKEND")? {
b.parse().map_err(|e| ConfigError::InvalidValue {
key: "LLM_BACKEND".to_string(),
message: e,
})?
} else if let Some(ref b) = settings.llm_backend {
match b.parse() {
Ok(backend) => backend,
Err(e) => {
tracing::warn!(
"Invalid llm_backend '{}' in settings: {}. Using default NearAi.",
b,
e
);
LlmBackend::NearAi
}
}
} else {
LlmBackend::NearAi
};
@@ -432,6 +457,7 @@ impl LlmConfig {
"fireworks::accounts/fireworks/models/llama4-maverick-instruct-basic"
.to_string()
}),
cheap_model: optional_env("NEARAI_CHEAP_MODEL")?,
base_url: optional_env("NEARAI_BASE_URL")?
.unwrap_or_else(|| "https://cloud-api.near.ai".to_string()),
auth_base_url: optional_env("NEARAI_AUTH_URL")?
@@ -441,6 +467,8 @@ impl LlmConfig {
.unwrap_or_else(default_session_path),
api_mode,
api_key: nearai_api_key,
fallback_model: optional_env("NEARAI_FALLBACK_MODEL")?,
max_retries: parse_optional_env("NEARAI_MAX_RETRIES", 3)?,
};
// Resolve provider-specific configs based on backend
@@ -473,6 +501,7 @@ impl LlmConfig {
let ollama = if backend == LlmBackend::Ollama {
let base_url = optional_env("OLLAMA_BASE_URL")?
.or_else(|| settings.ollama_base_url.clone())
.unwrap_or_else(|| "http://localhost:11434".to_string());
let model = optional_env("OLLAMA_MODEL")?.unwrap_or_else(|| "llama3".to_string());
Some(OllamaConfig { base_url, model })
@@ -481,8 +510,9 @@ impl LlmConfig {
};
let openai_compatible = if backend == LlmBackend::OpenAiCompatible {
let base_url =
optional_env("LLM_BASE_URL")?.ok_or_else(|| ConfigError::MissingRequired {
let base_url = optional_env("LLM_BASE_URL")?
.or_else(|| settings.openai_compatible_base_url.clone())
.ok_or_else(|| ConfigError::MissingRequired {
key: "LLM_BASE_URL".to_string(),
hint: "Set LLM_BASE_URL when LLM_BACKEND=openai_compatible".to_string(),
})?;
@@ -852,52 +882,41 @@ impl std::fmt::Debug for SecretsConfig {
}
}
/// Process-wide cache for the keychain master key.
///
/// Avoids re-prompting the OS keychain on every `SecretsConfig::resolve()` call
/// (e.g. `Config::from_env()` then `Config::from_db()`). Thread-safe alternative
/// to caching in a process env var.
impl SecretsConfig {
async fn resolve(bootstrap: &crate::bootstrap::BootstrapConfig) -> Result<Self, ConfigError> {
/// Auto-detect secrets master key from env var, then OS keychain.
///
/// Sequential probe: SECRETS_MASTER_KEY env var first, then OS keychain.
/// No saved "source" needed; just try each source in order.
async fn resolve() -> Result<Self, ConfigError> {
use crate::settings::KeySource;
let (master_key, source) = if let Some(env_key) = optional_env("SECRETS_MASTER_KEY")? {
(Some(SecretString::from(env_key)), KeySource::Env)
} else {
match bootstrap.secrets_master_key_source {
KeySource::Keychain => {
// Try to load from OS keychain (async on Linux)
match crate::secrets::keychain::get_master_key().await {
Ok(key_bytes) => {
let key_hex: String =
key_bytes.iter().map(|b| format!("{:02x}", b)).collect();
(Some(SecretString::from(key_hex)), KeySource::Keychain)
}
Err(_) => {
// Keychain configured but key not found
// This might happen if keychain was cleared
tracing::warn!(
"Secrets configured for keychain but key not found. \
Run 'ironclaw onboard' to reconfigure."
);
(None, KeySource::None)
}
}
// Probe the OS keychain; if a key is stored, use it
match crate::secrets::keychain::get_master_key().await {
Ok(key_bytes) => {
let key_hex: String = key_bytes.iter().map(|b| format!("{:02x}", b)).collect();
(Some(SecretString::from(key_hex)), KeySource::Keychain)
}
KeySource::Env => {
tracing::warn!(
"Secrets configured for env var but SECRETS_MASTER_KEY not set."
);
(None, KeySource::None)
}
KeySource::None => (None, KeySource::None),
Err(_) => (None, KeySource::None),
}
};
let enabled = master_key.is_some();
if let Some(ref key) = master_key {
if key.expose_secret().len() < 32 {
return Err(ConfigError::InvalidValue {
key: "SECRETS_MASTER_KEY".to_string(),
message: "must be at least 32 bytes for AES-256-GCM".to_string(),
});
}
if let Some(ref key) = master_key
&& key.expose_secret().len() < 32
{
return Err(ConfigError::InvalidValue {
key: "SECRETS_MASTER_KEY".to_string(),
message: "must be at least 32 bytes for AES-256-GCM".to_string(),
});
}
Ok(Self {
@@ -1351,17 +1370,64 @@ impl ClaudeCodeConfig {
}
}
/// Load API keys from the encrypted secrets store into a thread-safe overlay.
///
/// This bridges the gap between secrets stored during onboarding and the
/// env-var-first resolution in `LlmConfig::resolve()`. Keys in the overlay
/// are read by `optional_env()` before falling back to `std::env::var()`,
/// so explicit env vars always win.
pub async fn inject_llm_keys_from_secrets(
secrets: &dyn crate::secrets::SecretsStore,
user_id: &str,
) {
let mappings = [
("llm_openai_api_key", "OPENAI_API_KEY"),
("llm_anthropic_api_key", "ANTHROPIC_API_KEY"),
("llm_compatible_api_key", "LLM_API_KEY"),
];
let mut injected = HashMap::new();
for (secret_name, env_var) in mappings {
match std::env::var(env_var) {
Ok(val) if !val.is_empty() => continue,
_ => {}
}
match secrets.get_decrypted(user_id, secret_name).await {
Ok(decrypted) => {
injected.insert(env_var.to_string(), decrypted.expose().to_string());
tracing::debug!("Loaded secret '{}' for env var '{}'", secret_name, env_var);
}
Err(_) => {
// Secret doesn't exist, that's fine
}
}
}
let _ = INJECTED_VARS.set(injected);
}
// Helper functions
fn optional_env(key: &str) -> Result<Option<String>, ConfigError> {
// Check real env vars first (always win over injected secrets)
match std::env::var(key) {
Ok(val) if val.is_empty() => Ok(None),
Ok(val) => Ok(Some(val)),
Err(std::env::VarError::NotPresent) => Ok(None),
Err(e) => Err(ConfigError::ParseError(format!(
"failed to read {key}: {e}"
))),
Ok(val) if val.is_empty() => {}
Ok(val) => return Ok(Some(val)),
Err(std::env::VarError::NotPresent) => {}
Err(e) => {
return Err(ConfigError::ParseError(format!(
"failed to read {key}: {e}"
)));
}
}
// Fall back to thread-safe overlay (secrets injected from DB)
if let Some(val) = INJECTED_VARS.get().and_then(|map| map.get(key)) {
return Ok(Some(val.clone()));
}
Ok(None)
}
fn parse_optional_env<T>(key: &str, default: T) -> Result<T, ConfigError>
+8 -8
View File
@@ -772,10 +772,10 @@ impl Database for LibSqlBackend {
.await
.map_err(|e| DatabaseError::Query(e.to_string()))?
{
if let Ok(id_str) = row.get::<String>(0) {
if let Ok(id) = id_str.parse() {
ids.push(id);
}
if let Ok(id_str) = row.get::<String>(0)
&& let Ok(id) = id_str.parse()
{
ids.push(id);
}
}
Ok(ids)
@@ -2199,10 +2199,10 @@ impl Database for LibSqlBackend {
e.content_preview = None;
}
// Update to latest timestamp
if let (Some(existing), Some(new)) = (&e.updated_at, &updated_at) {
if new > existing {
e.updated_at = Some(*new);
}
if let (Some(existing), Some(new)) = (&e.updated_at, &updated_at)
&& new > existing
{
e.updated_at = Some(*new);
}
})
.or_insert(WorkspaceEntry {
+1 -5
View File
@@ -20,10 +20,6 @@ impl CostEstimator {
// Default tool costs (in USD or equivalent)
tool_costs.insert("http".to_string(), dec!(0.0001)); // API call
tool_costs.insert("marketplace".to_string(), dec!(0.01)); // Gas costs
tool_costs.insert("ecommerce".to_string(), dec!(0.001)); // API call
tool_costs.insert("taskrabbit".to_string(), dec!(0.0)); // Cost comes from task itself
tool_costs.insert("restaurant".to_string(), dec!(0.001)); // API call
tool_costs.insert("echo".to_string(), dec!(0.0)); // Free
tool_costs.insert("time".to_string(), dec!(0.0)); // Free
tool_costs.insert("json".to_string(), dec!(0.0)); // Free
@@ -74,7 +70,7 @@ mod tests {
let estimator = CostEstimator::new();
assert_eq!(estimator.estimate_tool("echo"), dec!(0.0));
assert_eq!(estimator.estimate_tool("marketplace"), dec!(0.01));
assert_eq!(estimator.estimate_tool("http"), dec!(0.0001));
assert!(estimator.estimate_tool("unknown") > dec!(0.0));
}
-4
View File
@@ -16,10 +16,6 @@ impl TimeEstimator {
// Default tool durations
tool_durations.insert("http".to_string(), Duration::from_secs(5));
tool_durations.insert("marketplace".to_string(), Duration::from_secs(10));
tool_durations.insert("ecommerce".to_string(), Duration::from_secs(8));
tool_durations.insert("taskrabbit".to_string(), Duration::from_secs(30)); // Just API, not task itself
tool_durations.insert("restaurant".to_string(), Duration::from_secs(5));
tool_durations.insert("echo".to_string(), Duration::from_millis(10));
tool_durations.insert("time".to_string(), Duration::from_millis(1));
tool_durations.insert("json".to_string(), Duration::from_millis(5));
+5 -6
View File
@@ -144,12 +144,11 @@ impl SuccessEvaluator for RuleBasedEvaluator {
// Check for critical errors
for action in actions.iter().filter(|a| !a.success) {
if let Some(ref error) = action.error {
if error.to_lowercase().contains("critical")
|| error.to_lowercase().contains("fatal")
{
issues.push(format!("Critical error in {}: {}", action.tool_name, error));
}
if let Some(ref error) = action.error
&& (error.to_lowercase().contains("critical")
|| error.to_lowercase().contains("fatal"))
{
issues.push(format!("Critical error in {}: {}", action.tool_name, error));
}
}
+27 -27
View File
@@ -492,13 +492,13 @@ impl ExtensionManager {
}
// Check Content-Length header before downloading the full body
if let Some(len) = response.content_length() {
if len as usize > MAX_WASM_SIZE {
return Err(ExtensionError::InstallFailed(format!(
"WASM binary too large ({} bytes, max {} bytes)",
len, MAX_WASM_SIZE
)));
}
if let Some(len) = response.content_length()
&& len as usize > MAX_WASM_SIZE
{
return Err(ExtensionError::InstallFailed(format!(
"WASM binary too large ({} bytes, max {} bytes)",
len, MAX_WASM_SIZE
)));
}
let bytes = response
@@ -768,27 +768,27 @@ impl ExtensionManager {
};
// Check env var first
if let Some(ref env_var) = auth.env_var {
if let Ok(value) = std::env::var(env_var) {
// Store the env var value as a secret
let params = CreateSecretParams::new(&auth.secret_name, &value)
.with_provider(name.to_string());
self.secrets
.create(&self.user_id, params)
.await
.map_err(|e| ExtensionError::AuthFailed(e.to_string()))?;
if let Some(ref env_var) = auth.env_var
&& let Ok(value) = std::env::var(env_var)
{
// Store the env var value as a secret
let params =
CreateSecretParams::new(&auth.secret_name, &value).with_provider(name.to_string());
self.secrets
.create(&self.user_id, params)
.await
.map_err(|e| ExtensionError::AuthFailed(e.to_string()))?;
return Ok(AuthResult {
name: name.to_string(),
kind: ExtensionKind::WasmTool,
auth_url: None,
callback_type: None,
instructions: None,
setup_url: None,
awaiting_token: false,
status: "authenticated".to_string(),
});
}
return Ok(AuthResult {
name: name.to_string(),
kind: ExtensionKind::WasmTool,
auth_url: None,
callback_type: None,
instructions: None,
setup_url: None,
awaiting_token: false,
status: "authenticated".to_string(),
});
}
// Check if already authenticated
+5
View File
@@ -36,6 +36,11 @@ pub struct Store {
#[cfg(feature = "postgres")]
impl Store {
/// Wrap an existing pool (useful when the caller already has a connection).
pub fn from_pool(pool: Pool) -> Self {
Self { pool }
}
/// Create a new store and connect to the database.
pub async fn new(config: &DatabaseConfig) -> Result<Self, DatabaseError> {
let mut cfg = Config::new();
+1
View File
@@ -59,6 +59,7 @@ pub mod secrets;
pub mod settings;
pub mod setup;
pub mod tools;
pub mod tracing_fmt;
pub mod util;
pub mod worker;
pub mod workspace;
+483
View File
@@ -0,0 +1,483 @@
//! Multi-provider LLM failover.
//!
//! Wraps multiple LlmProvider instances and tries each in sequence
//! until one succeeds. Transparent to callers --- same LlmProvider trait.
use std::future::Future;
use std::sync::Arc;
use std::sync::atomic::{AtomicUsize, Ordering};
use async_trait::async_trait;
use rust_decimal::Decimal;
use crate::error::LlmError;
use crate::llm::provider::{
CompletionRequest, CompletionResponse, LlmProvider, ToolCompletionRequest,
ToolCompletionResponse,
};
/// Returns `true` if the error is transient and the request should be retried
/// on the next provider in the failover chain.
///
/// Retryable: `RequestFailed`, `RateLimited`, `InvalidResponse`,
/// `SessionRenewalFailed`, `ModelNotAvailable`, `Http`, `Io`.
///
/// `ModelNotAvailable` is retryable because the next provider in the chain may
/// offer a different model, so it's worth trying.
///
/// Non-retryable errors (`AuthFailed`, `SessionExpired`, `ContextLengthExceeded`)
/// propagate immediately because a different provider won't fix them.
fn is_retryable(err: &LlmError) -> bool {
matches!(
err,
LlmError::RequestFailed { .. }
| LlmError::RateLimited { .. }
| LlmError::InvalidResponse { .. }
| LlmError::SessionRenewalFailed { .. }
// ModelNotAvailable is retryable: the next provider may offer a different model.
| LlmError::ModelNotAvailable { .. }
| LlmError::Http(_)
| LlmError::Io(_)
)
}
/// An LLM provider that wraps multiple providers and tries each in sequence
/// on transient failures.
///
/// The first provider in the list is the primary. If it fails with a retryable
/// error, the next provider is tried, and so on. Non-retryable errors
/// (e.g. `AuthFailed`, `ContextLengthExceeded`) propagate immediately.
pub struct FailoverProvider {
providers: Vec<Arc<dyn LlmProvider>>,
/// Index of the provider that last handled a request successfully.
/// Used by `model_name()` and `cost_per_token()` so downstream cost
/// tracking reflects the provider that actually served the request.
last_used: AtomicUsize,
}
impl FailoverProvider {
/// Create a new failover provider.
///
/// Returns an error if `providers` is empty.
pub fn new(providers: Vec<Arc<dyn LlmProvider>>) -> Result<Self, LlmError> {
if providers.is_empty() {
return Err(LlmError::RequestFailed {
provider: "failover".to_string(),
reason: "FailoverProvider requires at least one provider".to_string(),
});
}
Ok(Self {
providers,
last_used: AtomicUsize::new(0),
})
}
/// Try each provider in sequence until one succeeds or all fail.
async fn try_providers<T, F, Fut>(&self, mut call: F) -> Result<T, LlmError>
where
F: FnMut(Arc<dyn LlmProvider>) -> Fut,
Fut: Future<Output = Result<T, LlmError>>,
{
let mut last_error: Option<LlmError> = None;
for (i, provider) in self.providers.iter().enumerate() {
let result = call(Arc::clone(provider)).await;
match result {
Ok(response) => {
self.last_used.store(i, Ordering::Relaxed);
return Ok(response);
}
Err(err) => {
if !is_retryable(&err) {
return Err(err);
}
if i + 1 < self.providers.len() {
tracing::warn!(
provider = %provider.model_name(),
error = %err,
next_provider = %self.providers[i + 1].model_name(),
"Provider failed with retryable error, trying next provider"
);
}
last_error = Some(err);
}
}
}
// SAFETY: providers is non-empty (checked in `new`), so at least one
// iteration ran and `last_error` is `Some`.
Err(last_error.expect("providers list is non-empty"))
}
}
#[async_trait]
impl LlmProvider for FailoverProvider {
fn model_name(&self) -> &str {
self.providers[self.last_used.load(Ordering::Relaxed)].model_name()
}
fn cost_per_token(&self) -> (Decimal, Decimal) {
self.providers[self.last_used.load(Ordering::Relaxed)].cost_per_token()
}
async fn complete(&self, request: CompletionRequest) -> Result<CompletionResponse, LlmError> {
self.try_providers(|provider| {
let req = request.clone();
async move { provider.complete(req).await }
})
.await
}
async fn complete_with_tools(
&self,
request: ToolCompletionRequest,
) -> Result<ToolCompletionResponse, LlmError> {
self.try_providers(|provider| {
let req = request.clone();
async move { provider.complete_with_tools(req).await }
})
.await
}
async fn list_models(&self) -> Result<Vec<String>, LlmError> {
let mut all_models = Vec::new();
for provider in &self.providers {
match provider.list_models().await {
Ok(models) => all_models.extend(models),
Err(err) => {
tracing::warn!(
provider = %provider.model_name(),
error = %err,
"Failed to list models from provider, skipping"
);
}
}
}
all_models.sort();
all_models.dedup();
Ok(all_models)
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::sync::Mutex;
use std::time::Duration;
use crate::llm::provider::{CompletionResponse, FinishReason, ToolCompletionResponse};
/// A mock LLM provider that returns a predetermined result.
struct MockProvider {
name: String,
input_cost: Decimal,
output_cost: Decimal,
complete_result: Mutex<Option<Result<CompletionResponse, LlmError>>>,
tool_complete_result: Mutex<Option<Result<ToolCompletionResponse, LlmError>>>,
}
impl MockProvider {
fn succeeding(name: &str, content: &str) -> Self {
Self {
name: name.to_string(),
input_cost: Decimal::ZERO,
output_cost: Decimal::ZERO,
complete_result: Mutex::new(Some(Ok(CompletionResponse {
content: content.to_string(),
input_tokens: 10,
output_tokens: 5,
finish_reason: FinishReason::Stop,
response_id: None,
}))),
tool_complete_result: Mutex::new(Some(Ok(ToolCompletionResponse {
content: Some(content.to_string()),
tool_calls: vec![],
input_tokens: 10,
output_tokens: 5,
finish_reason: FinishReason::Stop,
response_id: None,
}))),
}
}
fn succeeding_with_cost(
name: &str,
content: &str,
input_cost: Decimal,
output_cost: Decimal,
) -> Self {
Self {
input_cost,
output_cost,
..Self::succeeding(name, content)
}
}
fn failing_retryable(name: &str) -> Self {
Self {
name: name.to_string(),
input_cost: Decimal::ZERO,
output_cost: Decimal::ZERO,
complete_result: Mutex::new(Some(Err(LlmError::RequestFailed {
provider: name.to_string(),
reason: "server error".to_string(),
}))),
tool_complete_result: Mutex::new(Some(Err(LlmError::RequestFailed {
provider: name.to_string(),
reason: "server error".to_string(),
}))),
}
}
fn failing_non_retryable(name: &str) -> Self {
Self {
name: name.to_string(),
input_cost: Decimal::ZERO,
output_cost: Decimal::ZERO,
complete_result: Mutex::new(Some(Err(LlmError::AuthFailed {
provider: name.to_string(),
}))),
tool_complete_result: Mutex::new(Some(Err(LlmError::AuthFailed {
provider: name.to_string(),
}))),
}
}
fn failing_rate_limited(name: &str) -> Self {
Self {
name: name.to_string(),
input_cost: Decimal::ZERO,
output_cost: Decimal::ZERO,
complete_result: Mutex::new(Some(Err(LlmError::RateLimited {
provider: name.to_string(),
retry_after: Some(Duration::from_secs(30)),
}))),
tool_complete_result: Mutex::new(Some(Err(LlmError::RateLimited {
provider: name.to_string(),
retry_after: Some(Duration::from_secs(30)),
}))),
}
}
}
#[async_trait]
impl LlmProvider for MockProvider {
fn model_name(&self) -> &str {
&self.name
}
fn cost_per_token(&self) -> (Decimal, Decimal) {
(self.input_cost, self.output_cost)
}
async fn complete(
&self,
_request: CompletionRequest,
) -> Result<CompletionResponse, LlmError> {
self.complete_result
.lock()
.unwrap()
.take()
.expect("MockProvider::complete called more than once")
}
async fn complete_with_tools(
&self,
_request: ToolCompletionRequest,
) -> Result<ToolCompletionResponse, LlmError> {
self.tool_complete_result
.lock()
.unwrap()
.take()
.expect("MockProvider::complete_with_tools called more than once")
}
async fn list_models(&self) -> Result<Vec<String>, LlmError> {
Ok(vec![self.name.clone()])
}
}
fn make_request() -> CompletionRequest {
CompletionRequest::new(vec![crate::llm::ChatMessage::user("hello")])
}
fn make_tool_request() -> ToolCompletionRequest {
ToolCompletionRequest::new(vec![crate::llm::ChatMessage::user("hello")], vec![])
}
// Test 1: Primary succeeds, no failover occurs.
#[tokio::test]
async fn primary_succeeds_no_failover() {
let primary = Arc::new(MockProvider::succeeding("primary", "primary response"));
let fallback = Arc::new(MockProvider::succeeding("fallback", "fallback response"));
let failover = FailoverProvider::new(vec![primary, fallback]).unwrap();
let response = failover.complete(make_request()).await.unwrap();
assert_eq!(response.content, "primary response");
}
// Test 2: Primary fails with retryable error, fallback succeeds.
#[tokio::test]
async fn primary_fails_retryable_fallback_succeeds() {
let primary = Arc::new(MockProvider::failing_retryable("primary"));
let fallback = Arc::new(MockProvider::succeeding("fallback", "fallback response"));
let failover = FailoverProvider::new(vec![primary, fallback]).unwrap();
let response = failover.complete(make_request()).await.unwrap();
assert_eq!(response.content, "fallback response");
}
// Test 3: All providers fail, returns last error.
#[tokio::test]
async fn all_providers_fail_returns_last_error() {
let primary = Arc::new(MockProvider::failing_retryable("primary"));
let fallback = Arc::new(MockProvider::failing_retryable("fallback"));
let failover = FailoverProvider::new(vec![primary, fallback]).unwrap();
let err = failover.complete(make_request()).await.unwrap_err();
match err {
LlmError::RequestFailed { provider, .. } => {
assert_eq!(provider, "fallback");
}
other => panic!("expected RequestFailed, got: {other:?}"),
}
}
// Test 4: Non-retryable error fails immediately, no failover.
#[tokio::test]
async fn non_retryable_error_fails_immediately() {
let primary = Arc::new(MockProvider::failing_non_retryable("primary"));
let fallback = Arc::new(MockProvider::succeeding("fallback", "fallback response"));
let failover = FailoverProvider::new(vec![primary, fallback]).unwrap();
let err = failover.complete(make_request()).await.unwrap_err();
match err {
LlmError::AuthFailed { provider } => {
assert_eq!(provider, "primary");
}
other => panic!("expected AuthFailed, got: {other:?}"),
}
}
// Test 5: Three providers, first two fail (retryable), third succeeds.
#[tokio::test]
async fn three_providers_first_two_fail_third_succeeds() {
let p1 = Arc::new(MockProvider::failing_retryable("provider-1"));
let p2 = Arc::new(MockProvider::failing_rate_limited("provider-2"));
let p3 = Arc::new(MockProvider::succeeding("provider-3", "third time lucky"));
let failover = FailoverProvider::new(vec![p1, p2, p3]).unwrap();
let response = failover.complete(make_request()).await.unwrap();
assert_eq!(response.content, "third time lucky");
}
// Test: complete_with_tools follows same failover logic.
#[tokio::test]
async fn complete_with_tools_failover() {
let primary = Arc::new(MockProvider::failing_retryable("primary"));
let fallback = Arc::new(MockProvider::succeeding("fallback", "tools fallback"));
let failover = FailoverProvider::new(vec![primary, fallback]).unwrap();
let response = failover
.complete_with_tools(make_tool_request())
.await
.unwrap();
assert_eq!(response.content.as_deref(), Some("tools fallback"));
}
// Test: model_name and cost_per_token reflect the last-used provider.
#[tokio::test]
async fn model_name_and_cost_track_last_used_provider() {
let fallback_cost = Decimal::new(15, 6); // 0.000015
let primary = Arc::new(MockProvider::failing_retryable("primary-model"));
let fallback = Arc::new(MockProvider::succeeding_with_cost(
"fallback-model",
"ok",
fallback_cost,
fallback_cost,
));
let failover = FailoverProvider::new(vec![primary, fallback]).unwrap();
// Before any call, defaults to primary (index 0).
assert_eq!(failover.model_name(), "primary-model");
assert_eq!(failover.cost_per_token(), (Decimal::ZERO, Decimal::ZERO));
// After failover, should reflect the fallback provider.
let _ = failover.complete(make_request()).await.unwrap();
assert_eq!(failover.model_name(), "fallback-model");
assert_eq!(failover.cost_per_token(), (fallback_cost, fallback_cost));
}
// Test: list_models aggregates from all providers.
#[tokio::test]
async fn list_models_aggregates_all() {
let p1 = Arc::new(MockProvider::succeeding("model-a", "ok"));
let p2 = Arc::new(MockProvider::succeeding("model-b", "ok"));
let failover = FailoverProvider::new(vec![p1, p2]).unwrap();
let models = failover.list_models().await.unwrap();
assert!(models.contains(&"model-a".to_string()));
assert!(models.contains(&"model-b".to_string()));
}
// Test: is_retryable correctly classifies errors.
#[test]
fn retryable_classification() {
// Retryable
assert!(is_retryable(&LlmError::RequestFailed {
provider: "p".into(),
reason: "err".into(),
}));
assert!(is_retryable(&LlmError::RateLimited {
provider: "p".into(),
retry_after: None,
}));
assert!(is_retryable(&LlmError::InvalidResponse {
provider: "p".into(),
reason: "bad json".into(),
}));
assert!(is_retryable(&LlmError::SessionRenewalFailed {
provider: "p".into(),
reason: "timeout".into(),
}));
assert!(is_retryable(&LlmError::Io(std::io::Error::new(
std::io::ErrorKind::ConnectionReset,
"reset"
))));
assert!(is_retryable(&LlmError::ModelNotAvailable {
provider: "p".into(),
model: "m".into(),
}));
// Non-retryable
assert!(!is_retryable(&LlmError::AuthFailed {
provider: "p".into(),
}));
assert!(!is_retryable(&LlmError::SessionExpired {
provider: "p".into(),
}));
assert!(!is_retryable(&LlmError::ContextLengthExceeded {
used: 100_000,
limit: 50_000,
}));
}
// Test: empty providers list returns error (not panic).
#[test]
fn empty_providers_returns_error() {
let result = FailoverProvider::new(vec![]);
assert!(result.is_err());
}
}
+125 -12
View File
@@ -8,13 +8,16 @@
//! - **OpenAI-compatible**: Any endpoint that speaks the OpenAI API
mod costs;
pub mod failover;
mod nearai;
mod nearai_chat;
mod provider;
mod reasoning;
mod retry;
mod rig_adapter;
pub mod session;
pub use failover::FailoverProvider;
pub use nearai::{ModelInfo, NearAiProvider};
pub use nearai_chat::NearAiChatProvider;
pub use provider::{
@@ -33,7 +36,7 @@ use std::sync::Arc;
use rig::client::CompletionClient;
use secrecy::ExposeSecret;
use crate::config::{LlmBackend, LlmConfig, NearAiApiMode};
use crate::config::{LlmBackend, LlmConfig, NearAiApiMode, NearAiConfig};
use crate::error::LlmError;
/// Create an LLM provider based on configuration.
@@ -46,7 +49,7 @@ pub fn create_llm_provider(
session: Arc<SessionManager>,
) -> Result<Arc<dyn LlmProvider>, LlmError> {
match config.backend {
LlmBackend::NearAi => create_nearai_provider(config, session),
LlmBackend::NearAi => create_llm_provider_with_config(&config.nearai, session),
LlmBackend::OpenAi => create_openai_provider(config),
LlmBackend::Anthropic => create_anthropic_provider(config),
LlmBackend::Ollama => create_ollama_provider(config),
@@ -54,21 +57,28 @@ pub fn create_llm_provider(
}
}
fn create_nearai_provider(
config: &LlmConfig,
/// Create an LLM provider from a `NearAiConfig` directly.
///
/// This is useful when constructing additional providers for failover,
/// where only the model name differs from the primary config.
pub fn create_llm_provider_with_config(
config: &NearAiConfig,
session: Arc<SessionManager>,
) -> Result<Arc<dyn LlmProvider>, LlmError> {
match config.nearai.api_mode {
match config.api_mode {
NearAiApiMode::Responses => {
tracing::info!("Using NEAR AI Responses API (chat-api) with session auth");
Ok(Arc::new(NearAiProvider::new(
config.nearai.clone(),
session,
)))
tracing::info!(
model = %config.model,
"Using Responses API (chat-api) with session auth"
);
Ok(Arc::new(NearAiProvider::new(config.clone(), session)))
}
NearAiApiMode::ChatCompletions => {
tracing::info!("Using NEAR AI Chat Completions API (cloud-api) with API key auth");
Ok(Arc::new(NearAiChatProvider::new(config.nearai.clone())?))
tracing::info!(
model = %config.model,
"Using Chat Completions API (cloud-api) with API key auth"
);
Ok(Arc::new(NearAiChatProvider::new(config.clone())?))
}
}
}
@@ -173,3 +183,106 @@ fn create_openai_compatible_provider(config: &LlmConfig) -> Result<Arc<dyn LlmPr
);
Ok(Arc::new(RigAdapter::new(model, &compat.model)))
}
/// Create a cheap/fast LLM provider for lightweight tasks (heartbeat, routing, evaluation).
///
/// Uses `NEARAI_CHEAP_MODEL` if set, otherwise falls back to the main provider.
/// Currently only supports NEAR AI backends (Responses and ChatCompletions modes).
pub fn create_cheap_llm_provider(
config: &LlmConfig,
session: Arc<SessionManager>,
) -> Result<Option<Arc<dyn LlmProvider>>, LlmError> {
let Some(ref cheap_model) = config.nearai.cheap_model else {
return Ok(None);
};
if config.backend != LlmBackend::NearAi {
tracing::warn!(
"NEARAI_CHEAP_MODEL is set but LLM_BACKEND is {:?}, not NearAi. \
Cheap model setting will be ignored.",
config.backend
);
return Ok(None);
}
let mut cheap_config = config.nearai.clone();
cheap_config.model = cheap_model.clone();
tracing::info!("Cheap LLM provider: {}", cheap_model);
match cheap_config.api_mode {
NearAiApiMode::Responses => Ok(Some(Arc::new(NearAiProvider::new(cheap_config, session)))),
NearAiApiMode::ChatCompletions => {
Ok(Some(Arc::new(NearAiChatProvider::new(cheap_config)?)))
}
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::config::{LlmBackend, NearAiApiMode, NearAiConfig};
use std::path::PathBuf;
fn test_nearai_config() -> NearAiConfig {
NearAiConfig {
model: "test-model".to_string(),
cheap_model: None,
base_url: "https://api.near.ai".to_string(),
auth_base_url: "https://private.near.ai".to_string(),
session_path: PathBuf::from("/tmp/test-session.json"),
api_mode: NearAiApiMode::Responses,
api_key: None,
fallback_model: None,
max_retries: 3,
}
}
fn test_llm_config() -> LlmConfig {
LlmConfig {
backend: LlmBackend::NearAi,
nearai: test_nearai_config(),
openai: None,
anthropic: None,
ollama: None,
openai_compatible: None,
}
}
#[test]
fn test_create_cheap_llm_provider_returns_none_when_not_configured() {
let config = test_llm_config();
let session = Arc::new(SessionManager::new(SessionConfig::default()));
let result = create_cheap_llm_provider(&config, session);
assert!(result.is_ok());
assert!(result.unwrap().is_none());
}
#[test]
fn test_create_cheap_llm_provider_creates_provider_when_configured() {
let mut config = test_llm_config();
config.nearai.cheap_model = Some("cheap-test-model".to_string());
let session = Arc::new(SessionManager::new(SessionConfig::default()));
let result = create_cheap_llm_provider(&config, session);
assert!(result.is_ok());
let provider = result.unwrap();
assert!(provider.is_some());
assert_eq!(provider.unwrap().model_name(), "cheap-test-model");
}
#[test]
fn test_create_cheap_llm_provider_ignored_for_non_nearai_backend() {
let mut config = test_llm_config();
config.backend = LlmBackend::OpenAi;
config.nearai.cheap_model = Some("cheap-test-model".to_string());
let session = Arc::new(SessionManager::new(SessionConfig::default()));
let result = create_cheap_llm_provider(&config, session);
assert!(result.is_ok());
assert!(result.unwrap().is_none());
}
}
+135 -93
View File
@@ -19,6 +19,7 @@ use crate::llm::provider::{
ChatMessage, CompletionRequest, CompletionResponse, FinishReason, LlmProvider, Role, ToolCall,
ToolCompletionRequest, ToolCompletionResponse,
};
use crate::llm::retry::{is_retryable_status, retry_backoff_delay};
use crate::llm::session::SessionManager;
/// Information about an available model from NEAR AI API.
@@ -209,20 +210,20 @@ impl NearAiProvider {
data: Option<Vec<ModelEntry>>,
}
if let Ok(resp) = serde_json::from_str::<ModelsResponse>(&response_text) {
if let Some(entries) = resp.models.or(resp.data) {
let models: Vec<ModelInfo> = entries
.into_iter()
.filter_map(|e| {
e.get_name().map(|name| ModelInfo {
name,
provider: None,
})
if let Ok(resp) = serde_json::from_str::<ModelsResponse>(&response_text)
&& let Some(entries) = resp.models.or(resp.data)
{
let models: Vec<ModelInfo> = entries
.into_iter()
.filter_map(|e| {
e.get_name().map(|name| ModelInfo {
name,
provider: None,
})
.collect();
if !models.is_empty() {
return Ok(models);
}
})
.collect();
if !models.is_empty() {
return Ok(models);
}
}
@@ -270,88 +271,139 @@ impl NearAiProvider {
}
}
/// Inner request implementation without retry logic.
/// Inner request implementation with retry logic for transient errors.
///
/// Retries on HTTP 429, 500, 502, 503, 504 with exponential backoff.
/// Does not retry on client errors (400, 401, 403, 404) or parse errors.
async fn send_request_inner<T: Serialize + std::fmt::Debug, R: for<'de> Deserialize<'de>>(
&self,
path: &str,
body: &T,
) -> Result<R, LlmError> {
let url = self.api_url(path);
let token = self.session.get_token().await?;
let max_retries = self.config.max_retries;
tracing::debug!("Sending request to NEAR AI: {}", url);
tracing::debug!("Request body: {:?}", body);
for attempt in 0..=max_retries {
let token = self.session.get_token().await?;
let response = self
.client
.post(&url)
.header("Authorization", format!("Bearer {}", token.expose_secret()))
.header("Content-Type", "application/json")
.json(body)
.send()
.await
.map_err(|e| {
tracing::error!("NEAR AI request failed: {}", e);
e
})?;
tracing::debug!(
"Sending request to NEAR AI: {} (attempt {})",
url,
attempt + 1
);
tracing::debug!("Request body: {:?}", body);
let status = response.status();
let response_text = response.text().await.unwrap_or_default();
let response = self
.client
.post(&url)
.header("Authorization", format!("Bearer {}", token.expose_secret()))
.header("Content-Type", "application/json")
.json(body)
.send()
.await;
tracing::debug!("NEAR AI response status: {}", status);
tracing::debug!("NEAR AI response body: {}", response_text);
let response = match response {
Ok(r) => r,
Err(e) => {
tracing::error!("NEAR AI request failed: {}", e);
// Network errors (timeout, connection refused) are transient
if attempt < max_retries {
let delay = retry_backoff_delay(attempt);
tracing::warn!(
"NEAR AI request error (attempt {}/{}), retrying in {:?}: {}",
attempt + 1,
max_retries + 1,
delay,
e,
);
tokio::time::sleep(delay).await;
continue;
}
return Err(e.into());
}
};
if !status.is_success() {
// Check for session expiration (401 with specific message patterns)
if status.as_u16() == 401 {
let is_session_expired = response_text.to_lowercase().contains("session")
&& (response_text.to_lowercase().contains("expired")
|| response_text.to_lowercase().contains("invalid"));
let status = response.status();
let response_text = response.text().await.unwrap_or_default();
if is_session_expired {
return Err(LlmError::SessionExpired {
tracing::debug!("NEAR AI response status: {}", status);
tracing::debug!("NEAR AI response body: {}", response_text);
if !status.is_success() {
let status_code = status.as_u16();
// Check for session expiration (401 with specific message patterns)
if status_code == 401 {
let lower = response_text.to_lowercase();
let is_session_expired = lower.contains("session")
&& (lower.contains("expired") || lower.contains("invalid"));
if is_session_expired {
return Err(LlmError::SessionExpired {
provider: "nearai".to_string(),
});
}
// Generic 401 -- not retryable
return Err(LlmError::AuthFailed {
provider: "nearai".to_string(),
});
}
// Generic 401 without session expiration indication
return Err(LlmError::AuthFailed {
provider: "nearai".to_string(),
});
}
// Check if this is a transient error worth retrying
if is_retryable_status(status_code) && attempt < max_retries {
let delay = retry_backoff_delay(attempt);
tracing::warn!(
"NEAR AI returned HTTP {} (attempt {}/{}), retrying in {:?}",
status_code,
attempt + 1,
max_retries + 1,
delay,
);
tokio::time::sleep(delay).await;
continue;
}
// Try to parse as JSON error
if let Ok(error) = serde_json::from_str::<NearAiErrorResponse>(&response_text) {
if status.as_u16() == 429 {
return Err(LlmError::RateLimited {
// Non-retryable error or exhausted retries
if let Ok(error) = serde_json::from_str::<NearAiErrorResponse>(&response_text) {
if status_code == 429 {
return Err(LlmError::RateLimited {
provider: "nearai".to_string(),
retry_after: None,
});
}
return Err(LlmError::RequestFailed {
provider: "nearai".to_string(),
retry_after: None,
reason: error.error,
});
}
return Err(LlmError::RequestFailed {
provider: "nearai".to_string(),
reason: error.error,
reason: format!("HTTP {}: {}", status, response_text),
});
}
return Err(LlmError::RequestFailed {
provider: "nearai".to_string(),
reason: format!("HTTP {}: {}", status, response_text),
});
// Success -- parse the response
return match serde_json::from_str::<R>(&response_text) {
Ok(parsed) => Ok(parsed),
Err(e) => {
tracing::debug!("Response is not expected JSON format: {}", e);
tracing::debug!("Will try alternative parsing in caller");
Err(LlmError::InvalidResponse {
provider: "nearai".to_string(),
reason: format!("Parse error: {}. Raw: {}", e, response_text),
})
}
};
}
// Try to parse as our expected type
match serde_json::from_str::<R>(&response_text) {
Ok(parsed) => Ok(parsed),
Err(e) => {
tracing::debug!("Response is not expected JSON format: {}", e);
tracing::debug!("Will try alternative parsing in caller");
Err(LlmError::InvalidResponse {
provider: "nearai".to_string(),
reason: format!("Parse error: {}. Raw: {}", e, response_text),
})
}
}
// This is unreachable because the loop always returns, but the compiler
// cannot prove that. Return a generic error as a safety net.
Err(LlmError::RequestFailed {
provider: "nearai".to_string(),
reason: "retry loop exited unexpectedly".to_string(),
})
}
}
@@ -456,7 +508,7 @@ impl LlmProvider for NearAiProvider {
Err(e) => return Err(e),
};
tracing::debug!("NEAR AI response: {:?}", response);
tracing::debug!("NEAR AI response: output_items={}", response.output.len());
// Extract text from response output
// Try multiple formats since API response shape may vary
@@ -464,11 +516,6 @@ impl LlmProvider for NearAiProvider {
.output
.iter()
.filter_map(|item| {
tracing::debug!(
"Processing output item: type={}, text={:?}",
item.item_type,
item.text
);
if item.item_type == "message" {
// First check for direct text field on item
if let Some(ref text) = item.text {
@@ -479,11 +526,6 @@ impl LlmProvider for NearAiProvider {
contents
.iter()
.filter_map(|c| {
tracing::debug!(
"Content item: type={}, text={:?}",
c.content_type,
c.text
);
// Accept various content types that might contain text
match c.content_type.as_str() {
"output_text" | "text" => c.text.clone(),
@@ -694,21 +736,21 @@ impl LlmProvider for NearAiProvider {
}
}
}
} else if item.item_type == "function_call" {
if let (Some(name), Some(call_id)) = (&item.name, &item.call_id) {
// Parse arguments JSON string into Value
let arguments = item
.arguments
.as_ref()
.and_then(|s| serde_json::from_str(s).ok())
.unwrap_or(serde_json::Value::Object(Default::default()));
} else if item.item_type == "function_call"
&& let (Some(name), Some(call_id)) = (&item.name, &item.call_id)
{
// Parse arguments JSON string into Value
let arguments = item
.arguments
.as_ref()
.and_then(|s| serde_json::from_str(s).ok())
.unwrap_or(serde_json::Value::Object(Default::default()));
tool_calls.push(ToolCall {
id: call_id.clone(),
name: name.clone(),
arguments,
});
}
tool_calls.push(ToolCall {
id: call_id.clone(),
name: name.clone(),
arguments,
});
}
}
+95 -41
View File
@@ -16,6 +16,7 @@ use crate::llm::provider::{
ChatMessage, CompletionRequest, CompletionResponse, FinishReason, LlmProvider, ModelMetadata,
Role, ToolCall, ToolCompletionRequest, ToolCompletionResponse,
};
use crate::llm::retry::{is_retryable_status, retry_backoff_delay};
/// NEAR AI Chat Completions API provider.
pub struct NearAiChatProvider {
@@ -62,63 +63,116 @@ impl NearAiChatProvider {
.unwrap_or_default()
}
/// Send a request to the chat completions API.
/// Send a request to the chat completions API with retry on transient errors.
///
/// Retries on HTTP 429, 500, 502, 503, 504 with exponential backoff.
/// Does not retry on client errors (400, 401, 403, 404) or parse errors.
async fn send_request<T: Serialize, R: for<'de> Deserialize<'de>>(
&self,
body: &T,
) -> Result<R, LlmError> {
let url = self.api_url("chat/completions");
let max_retries = self.config.max_retries;
tracing::debug!("Sending request to NEAR AI Chat: {}", url);
for attempt in 0..=max_retries {
tracing::debug!(
"Sending request to NEAR AI Chat: {} (attempt {})",
url,
attempt + 1,
);
// Log the request body for debugging tool call issues
if let Ok(json) = serde_json::to_string(body) {
tracing::debug!("NEAR AI Chat request body: {}", json);
}
if tracing::enabled!(tracing::Level::DEBUG)
&& let Ok(json) = serde_json::to_string(body)
{
tracing::debug!("NEAR AI Chat request body: {}", json);
}
let response = self
.client
.post(&url)
.header("Authorization", format!("Bearer {}", self.api_key()))
.header("Content-Type", "application/json")
.json(body)
.send()
.await
.map_err(|e| {
tracing::error!("NEAR AI Chat request failed: {}", e);
LlmError::RequestFailed {
provider: "nearai_chat".to_string(),
reason: e.to_string(),
let response = self
.client
.post(&url)
.header("Authorization", format!("Bearer {}", self.api_key()))
.header("Content-Type", "application/json")
.json(body)
.send()
.await;
let response = match response {
Ok(r) => r,
Err(e) => {
tracing::error!("NEAR AI Chat request failed: {}", e);
if attempt < max_retries {
let delay = retry_backoff_delay(attempt);
tracing::warn!(
"NEAR AI Chat request error (attempt {}/{}), retrying in {:?}: {}",
attempt + 1,
max_retries + 1,
delay,
e,
);
tokio::time::sleep(delay).await;
continue;
}
return Err(LlmError::RequestFailed {
provider: "nearai_chat".to_string(),
reason: e.to_string(),
});
}
})?;
};
let status = response.status();
let response_text = response.text().await.unwrap_or_default();
let status = response.status();
let response_text = response.text().await.unwrap_or_default();
tracing::debug!("NEAR AI Chat response status: {}", status);
tracing::debug!("NEAR AI Chat response body: {}", response_text);
tracing::debug!("NEAR AI Chat response status: {}", status);
tracing::debug!("NEAR AI Chat response body: {}", response_text);
if !status.is_success() {
if status.as_u16() == 401 {
return Err(LlmError::AuthFailed {
if !status.is_success() {
let status_code = status.as_u16();
// Auth errors are not retryable
if status_code == 401 {
return Err(LlmError::AuthFailed {
provider: "nearai_chat".to_string(),
});
}
// Transient errors: retry with backoff
if is_retryable_status(status_code) && attempt < max_retries {
let delay = retry_backoff_delay(attempt);
tracing::warn!(
"NEAR AI Chat returned HTTP {} (attempt {}/{}), retrying in {:?}",
status_code,
attempt + 1,
max_retries + 1,
delay,
);
tokio::time::sleep(delay).await;
continue;
}
// Non-retryable or exhausted retries
if status_code == 429 {
return Err(LlmError::RateLimited {
provider: "nearai_chat".to_string(),
retry_after: None,
});
}
return Err(LlmError::RequestFailed {
provider: "nearai_chat".to_string(),
reason: format!("HTTP {}: {}", status, response_text),
});
}
if status.as_u16() == 429 {
return Err(LlmError::RateLimited {
provider: "nearai_chat".to_string(),
retry_after: None,
});
}
return Err(LlmError::RequestFailed {
// Success — parse the response
return serde_json::from_str(&response_text).map_err(|e| LlmError::InvalidResponse {
provider: "nearai_chat".to_string(),
reason: format!("HTTP {}: {}", status, response_text),
reason: format!("JSON parse error: {}. Raw: {}", e, response_text),
});
}
serde_json::from_str(&response_text).map_err(|e| LlmError::InvalidResponse {
// Safety net: unreachable because the loop always returns
Err(LlmError::RequestFailed {
provider: "nearai_chat".to_string(),
reason: format!("JSON parse error: {}. Raw: {}", e, response_text),
reason: "retry loop exited unexpectedly".to_string(),
})
}
@@ -395,10 +449,10 @@ fn flatten_tool_messages(messages: Vec<ChatCompletionMessage>) -> Vec<ChatComple
if let (true, Some(calls)) = (msg.role == "assistant", &msg.tool_calls) {
// Convert assistant tool_calls into descriptive text
let mut parts: Vec<String> = Vec::new();
if let Some(ref text) = msg.content {
if !text.is_empty() {
parts.push(text.clone());
}
if let Some(ref text) = msg.content
&& !text.is_empty()
{
parts.push(text.clone());
}
for tc in calls {
parts.push(format!(
+21 -15
View File
@@ -113,6 +113,12 @@ pub struct ToolSelection {
pub reasoning: String,
/// Alternative tools considered.
pub alternatives: Vec<String>,
/// The tool call ID from the LLM response.
///
/// OpenAI-compatible providers assign each tool call a unique ID that must
/// be echoed back in the corresponding tool result message. Without this,
/// the provider cannot match results to their originating calls.
pub tool_call_id: String,
}
/// Token usage from a single LLM call.
@@ -244,6 +250,7 @@ impl Reasoning {
parameters: tool_call.arguments,
reasoning: reasoning.clone(),
alternatives: vec![],
tool_call_id: tool_call.id,
})
.collect();
@@ -581,21 +588,20 @@ fn recover_tool_calls_from_content(
}
// Try JSON first: {"name":"x","arguments":{}}
if let Ok(parsed) = serde_json::from_str::<serde_json::Value>(inner) {
if let Some(name) = parsed.get("name").and_then(|v| v.as_str()) {
if tool_names.contains(name) {
let arguments = parsed
.get("arguments")
.cloned()
.unwrap_or(serde_json::Value::Object(Default::default()));
calls.push(ToolCall {
id: format!("recovered_{}", calls.len()),
name: name.to_string(),
arguments,
});
continue;
}
}
if let Ok(parsed) = serde_json::from_str::<serde_json::Value>(inner)
&& let Some(name) = parsed.get("name").and_then(|v| v.as_str())
&& tool_names.contains(name)
{
let arguments = parsed
.get("arguments")
.cloned()
.unwrap_or(serde_json::Value::Object(Default::default()));
calls.push(ToolCall {
id: format!("recovered_{}", calls.len()),
name: name.to_string(),
arguments,
});
continue;
}
// Bare tool name (e.g. "<tool_call>tool_list</tool_call>")
+96
View File
@@ -0,0 +1,96 @@
//! Shared retry helpers for LLM providers.
//!
//! Provides exponential backoff with jitter and retryable status classification
//! used by both `NearAiProvider` and `NearAiChatProvider`.
use std::time::Duration;
use rand::Rng;
/// Returns `true` if the HTTP status code is transient and worth retrying.
pub(crate) fn is_retryable_status(status: u16) -> bool {
matches!(status, 429 | 500 | 502 | 503 | 504)
}
/// Calculate exponential backoff delay with random jitter.
///
/// Base delay is 1 second, doubled each attempt, with +/-25% jitter.
/// - attempt 0: ~1s (0.75s - 1.25s)
/// - attempt 1: ~2s (1.5s - 2.5s)
/// - attempt 2: ~4s (3.0s - 5.0s)
pub(crate) fn retry_backoff_delay(attempt: u32) -> Duration {
let base_ms: u64 = 1000u64.saturating_mul(2u64.saturating_pow(attempt));
let jitter_range = base_ms / 4; // 25%
let jitter = if jitter_range > 0 {
let offset = rand::thread_rng().gen_range(0..=jitter_range * 2);
offset as i64 - jitter_range as i64
} else {
0
};
let delay_ms = (base_ms as i64 + jitter).max(100) as u64;
Duration::from_millis(delay_ms)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_is_retryable_status() {
// Transient errors should be retryable
assert!(is_retryable_status(429));
assert!(is_retryable_status(500));
assert!(is_retryable_status(502));
assert!(is_retryable_status(503));
assert!(is_retryable_status(504));
// Client errors should not be retryable
assert!(!is_retryable_status(400));
assert!(!is_retryable_status(401));
assert!(!is_retryable_status(403));
assert!(!is_retryable_status(404));
assert!(!is_retryable_status(422));
// Success codes should not be retryable
assert!(!is_retryable_status(200));
assert!(!is_retryable_status(201));
}
#[test]
fn test_retry_backoff_delay_exponential_growth() {
// Run multiple samples to verify the range, accounting for jitter
for _ in 0..20 {
let d0 = retry_backoff_delay(0);
let d1 = retry_backoff_delay(1);
let d2 = retry_backoff_delay(2);
// Attempt 0: base 1000ms, jitter +/-250ms -> [750, 1250]
assert!(d0.as_millis() >= 750, "attempt 0 too low: {:?}", d0);
assert!(d0.as_millis() <= 1250, "attempt 0 too high: {:?}", d0);
// Attempt 1: base 2000ms, jitter +/-500ms -> [1500, 2500]
assert!(d1.as_millis() >= 1500, "attempt 1 too low: {:?}", d1);
assert!(d1.as_millis() <= 2500, "attempt 1 too high: {:?}", d1);
// Attempt 2: base 4000ms, jitter +/-1000ms -> [3000, 5000]
assert!(d2.as_millis() >= 3000, "attempt 2 too low: {:?}", d2);
assert!(d2.as_millis() <= 5000, "attempt 2 too high: {:?}", d2);
}
}
#[test]
fn test_retry_backoff_delay_minimum() {
// Even at attempt 0, delay should be at least 100ms (the minimum floor)
for _ in 0..20 {
let delay = retry_backoff_delay(0);
assert!(delay.as_millis() >= 100);
}
}
#[test]
fn test_retry_backoff_delay_no_overflow() {
// Very high attempt numbers should not panic from overflow
let delay = retry_backoff_delay(30);
assert!(delay.as_millis() >= 100);
}
}
+35 -180
View File
@@ -31,8 +31,6 @@ pub struct SessionConfig {
pub auth_base_url: String,
/// Path to session file (e.g., ~/.ironclaw/session.json).
pub session_path: PathBuf,
/// Port range for OAuth callback server.
pub callback_port_range: (u16, u16),
}
impl Default for SessionConfig {
@@ -40,7 +38,6 @@ impl Default for SessionConfig {
Self {
auth_base_url: "https://private.near.ai".to_string(),
session_path: default_session_path(),
callback_port_range: (9876, 9886),
}
}
}
@@ -83,16 +80,16 @@ impl SessionManager {
};
// Try to load existing session synchronously during construction
if let Ok(data) = std::fs::read_to_string(&manager.config.session_path) {
if let Ok(session) = serde_json::from_str::<SessionData>(&data) {
// We can't await here, so we use try_write
if let Ok(mut guard) = manager.token.try_write() {
*guard = Some(SecretString::from(session.session_token));
tracing::info!(
"Loaded session token from {}",
manager.config.session_path.display()
);
}
if let Ok(data) = std::fs::read_to_string(&manager.config.session_path)
&& let Ok(session) = serde_json::from_str::<SessionData>(&data)
{
// We can't await here, so we use try_write
if let Ok(mut guard) = manager.token.try_write() {
*guard = Some(SecretString::from(session.session_token));
tracing::info!(
"Loaded session token from {}",
manager.config.session_path.display()
);
}
}
@@ -222,38 +219,21 @@ impl SessionManager {
/// Start the OAuth login flow.
///
/// 1. Find an available port for the callback server
/// 1. Bind the fixed callback port
/// 2. Print the auth URL and attempt to open browser
/// 3. Wait for OAuth callback with session token
/// 4. Save and return the token
async fn initiate_login(&self) -> Result<(), LlmError> {
use tokio::io::{AsyncBufReadExt, AsyncWriteExt, BufReader};
use tokio::net::TcpListener;
use crate::cli::oauth_defaults::{self, OAUTH_CALLBACK_PORT};
// Find an available port
let mut listener = None;
let mut port = 0;
let listener = oauth_defaults::bind_callback_listener()
.await
.map_err(|e| LlmError::SessionRenewalFailed {
provider: "nearai".to_string(),
reason: e.to_string(),
})?;
for p in self.config.callback_port_range.0..=self.config.callback_port_range.1 {
match TcpListener::bind(format!("127.0.0.1:{}", p)).await {
Ok(l) => {
listener = Some(l);
port = p;
break;
}
Err(_) => continue,
}
}
let listener = listener.ok_or_else(|| LlmError::SessionRenewalFailed {
provider: "nearai".to_string(),
reason: format!(
"Could not find available port in range {}-{}",
self.config.callback_port_range.0, self.config.callback_port_range.1
),
})?;
let callback_url = format!("http://127.0.0.1:{}", port);
let callback_url = format!("http://127.0.0.1:{}", OAUTH_CALLBACK_PORT);
// Show auth provider menu
println!();
@@ -333,138 +313,16 @@ impl SessionManager {
println!();
println!("Waiting for authentication...");
// Wait for callback with timeout
// The API redirects to: {frontend_callback}/auth/callback?token=X&session_id=X&expires_at=X&is_new_user=X
let timeout = std::time::Duration::from_secs(300); // 5 minutes
let selected_provider = auth_provider.to_string();
let (session_token, auth_provider) = tokio::time::timeout(timeout, async move {
loop {
let (mut socket, _) = listener.accept().await.map_err(|e| {
LlmError::SessionRenewalFailed {
provider: "nearai".to_string(),
reason: format!("Failed to accept connection: {}", e),
}
// The NEAR AI API redirects to: {frontend_callback}/auth/callback?token=X&...
let session_token =
oauth_defaults::wait_for_callback(listener, "/auth/callback", "token", "NEAR AI")
.await
.map_err(|e| LlmError::SessionRenewalFailed {
provider: "nearai".to_string(),
reason: e.to_string(),
})?;
let mut reader = BufReader::new(&mut socket);
let mut request_line = String::new();
reader.read_line(&mut request_line).await.map_err(|e| {
LlmError::SessionRenewalFailed {
provider: "nearai".to_string(),
reason: format!("Failed to read request: {}", e),
}
})?;
// Parse GET /auth/callback?token=xxx&session_id=xxx&expires_at=xxx&is_new_user=xxx HTTP/1.1
if let Some(path) = request_line.split_whitespace().nth(1) {
if path.starts_with("/auth/callback") {
// Parse query parameters
if let Some(query) = path.split('?').nth(1) {
let mut token = None;
for param in query.split('&') {
let parts: Vec<&str> = param.splitn(2, '=').collect();
if parts.len() == 2 && parts[0] == "token" {
token = Some(
urlencoding::decode(parts[1])
.unwrap_or_else(|_| parts[1].into())
.into_owned(),
);
}
}
if let Some(token) = token {
// Send success response with nice styling
let response = concat!(
"HTTP/1.1 200 OK\r\n",
"Content-Type: text/html; charset=utf-8\r\n",
"Connection: close\r\n",
"\r\n",
"<!DOCTYPE html>\n",
"<html>\n",
"<head>\n",
" <meta charset=\"utf-8\">\n",
" <title>NEAR AI - Authentication Successful</title>\n",
" <style>\n",
" * { margin: 0; padding: 0; box-sizing: border-box; }\n",
" body {\n",
" font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;\n",
" background: linear-gradient(135deg, #1a1a2e 0%, #16213e 100%);\n",
" min-height: 100vh;\n",
" display: flex;\n",
" align-items: center;\n",
" justify-content: center;\n",
" color: #fff;\n",
" }\n",
" .container {\n",
" text-align: center;\n",
" padding: 3rem;\n",
" background: rgba(255,255,255,0.05);\n",
" border-radius: 16px;\n",
" backdrop-filter: blur(10px);\n",
" border: 1px solid rgba(255,255,255,0.1);\n",
" max-width: 400px;\n",
" }\n",
" .checkmark {\n",
" width: 80px;\n",
" height: 80px;\n",
" background: linear-gradient(135deg, #00d9a5 0%, #00b386 100%);\n",
" border-radius: 50%;\n",
" display: flex;\n",
" align-items: center;\n",
" justify-content: center;\n",
" margin: 0 auto 1.5rem;\n",
" font-size: 40px;\n",
" }\n",
" h1 {\n",
" font-size: 1.5rem;\n",
" font-weight: 600;\n",
" margin-bottom: 0.75rem;\n",
" }\n",
" p {\n",
" color: rgba(255,255,255,0.7);\n",
" font-size: 0.95rem;\n",
" line-height: 1.5;\n",
" }\n",
" .brand {\n",
" margin-top: 2rem;\n",
" padding-top: 1.5rem;\n",
" border-top: 1px solid rgba(255,255,255,0.1);\n",
" font-size: 0.8rem;\n",
" color: rgba(255,255,255,0.4);\n",
" }\n",
" </style>\n",
"</head>\n",
"<body>\n",
" <div class=\"container\">\n",
" <div class=\"checkmark\">&#10003;</div>\n",
" <h1>Authentication Successful</h1>\n",
" <p>You can close this window and return to the terminal.</p>\n",
" <div class=\"brand\">NEAR AI Agent</div>\n",
" </div>\n",
"</body>\n",
"</html>"
);
let _ = socket.write_all(response.as_bytes()).await;
let _ = socket.shutdown().await;
return Ok::<_, LlmError>((token, Some(selected_provider.clone())));
}
}
}
}
// Not the callback we're looking for, send 404
let response = "HTTP/1.1 404 Not Found\r\nConnection: close\r\n\r\n";
let _ = socket.write_all(response.as_bytes()).await;
}
})
.await
.map_err(|_| LlmError::SessionRenewalFailed {
provider: "nearai".to_string(),
reason: "Authentication timed out after 5 minutes".to_string(),
})??;
let auth_provider = Some(auth_provider.to_string());
// Save the token
self.save_session(&session_token, auth_provider.as_deref())
@@ -642,15 +500,14 @@ pub async fn create_session_manager(config: SessionConfig) -> Arc<SessionManager
let manager = SessionManager::new_async(config).await;
// Check for legacy env var and migrate if present and no file token
if !manager.has_token().await {
if let Ok(token) = std::env::var("NEARAI_SESSION_TOKEN") {
if !token.is_empty() {
tracing::info!("Migrating session token from NEARAI_SESSION_TOKEN env var to file");
manager.set_token(SecretString::from(token.clone())).await;
if let Err(e) = manager.save_session(&token, None).await {
tracing::warn!("Failed to save migrated session: {}", e);
}
}
if !manager.has_token().await
&& let Ok(token) = std::env::var("NEARAI_SESSION_TOKEN")
&& !token.is_empty()
{
tracing::info!("Migrating session token from NEARAI_SESSION_TOKEN env var to file");
manager.set_token(SecretString::from(token.clone())).await;
if let Err(e) = manager.save_session(&token, None).await {
tracing::warn!("Failed to save migrated session: {}", e);
}
}
@@ -671,7 +528,6 @@ mod tests {
let config = SessionConfig {
auth_base_url: "https://example.com".to_string(),
session_path: session_path.clone(),
callback_port_range: (9900, 9910),
};
let manager = SessionManager::new_async(config.clone()).await;
@@ -712,7 +568,6 @@ mod tests {
let config = SessionConfig {
auth_base_url: "https://example.com".to_string(),
session_path: dir.path().join("nonexistent.json"),
callback_port_range: (9900, 9910),
};
let manager = SessionManager::new_async(config).await;
+166 -110
View File
@@ -22,7 +22,10 @@ use ironclaw::{
config::Config,
context::ContextManager,
extensions::ExtensionManager,
llm::{SessionConfig, create_llm_provider, create_session_manager},
llm::{
FailoverProvider, LlmProvider, SessionConfig, create_cheap_llm_provider,
create_llm_provider, create_llm_provider_with_config, create_session_manager,
},
orchestrator::{
ContainerJobConfig, ContainerJobManager, OrchestratorApi, TokenStore,
api::OrchestratorState,
@@ -45,7 +48,6 @@ use ironclaw::secrets::PostgresSecretsStore;
use ironclaw::secrets::SecretsCrypto;
#[cfg(any(feature = "postgres", feature = "libsql"))]
use ironclaw::setup::{SetupConfig, SetupWizard};
#[tokio::main]
async fn main() -> anyhow::Result<()> {
let cli = Cli::parse();
@@ -90,7 +92,6 @@ async fn main() -> anyhow::Result<()> {
.init();
// Memory commands need database (and optionally embeddings)
let _ = dotenvy::dotenv();
let config = Config::from_env()
.await
.map_err(|e| anyhow::anyhow!("{}", e))?;
@@ -99,7 +100,6 @@ async fn main() -> anyhow::Result<()> {
let session = ironclaw::llm::create_session_manager(ironclaw::llm::SessionConfig {
auth_base_url: config.llm.nearai.auth_base_url.clone(),
session_path: config.llm.nearai.session_path.clone(),
..Default::default()
})
.await;
@@ -152,7 +152,6 @@ async fn main() -> anyhow::Result<()> {
return run_pairing_command(pairing_cmd.clone()).map_err(|e| anyhow::anyhow!("{}", e));
}
Some(Command::Status) => {
let _ = dotenvy::dotenv();
tracing_subscriber::fmt()
.with_env_filter(
EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new("warn")),
@@ -243,8 +242,10 @@ async fn main() -> anyhow::Result<()> {
skip_auth,
channels_only,
}) => {
// Load .env before running onboarding wizard
// Load .env files before running onboarding wizard.
// Standard ./.env first (higher priority), then ~/.ironclaw/.env.
let _ = dotenvy::dotenv();
ironclaw::bootstrap::load_ironclaw_env();
#[cfg(any(feature = "postgres", feature = "libsql"))]
{
@@ -267,23 +268,23 @@ async fn main() -> anyhow::Result<()> {
}
}
// Load .env if present
// Load .env files early so DATABASE_URL (and any other vars) are
// available to all subsequent env-based config resolution.
// Standard ./.env first (higher priority), then ~/.ironclaw/.env.
let _ = dotenvy::dotenv();
ironclaw::bootstrap::load_ironclaw_env();
// Enhanced first-run detection
#[cfg(any(feature = "postgres", feature = "libsql"))]
if !cli.no_onboard {
if let Some(reason) = check_onboard_needed().await {
println!("Onboarding needed: {}", reason);
println!();
let mut wizard = SetupWizard::new();
wizard.run().await?;
}
if !cli.no_onboard
&& let Some(reason) = check_onboard_needed()
{
println!("Onboarding needed: {}", reason);
println!();
let mut wizard = SetupWizard::new();
wizard.run().await?;
}
// Load bootstrap config (4 fields that must live on disk)
let bootstrap = ironclaw::bootstrap::BootstrapConfig::load();
// Load initial config from env + disk (before DB is available)
let mut config = match Config::from_env().await {
Ok(c) => c,
@@ -303,18 +304,20 @@ async fn main() -> anyhow::Result<()> {
let session_config = SessionConfig {
auth_base_url: config.llm.nearai.auth_base_url.clone(),
session_path: config.llm.nearai.session_path.clone(),
..Default::default()
};
let session = create_session_manager(session_config).await;
// Ensure we're authenticated before proceeding (only needed for NEAR AI backend)
if config.llm.backend == ironclaw::config::LlmBackend::NearAi {
// Session-based auth is only needed for NEAR AI backend without an API key.
// ChatCompletions mode with an API key skips session auth entirely.
if config.llm.backend == ironclaw::config::LlmBackend::NearAi
&& config.llm.nearai.api_key.is_none()
{
session.ensure_authenticated().await?;
}
// Initialize tracing
let env_filter = EnvFilter::try_from_default_env()
.unwrap_or_else(|_| EnvFilter::new("ironclaw=info,tower_http=debug"));
.unwrap_or_else(|_| EnvFilter::new("ironclaw=info,tower_http=warn"));
// Create log broadcaster before tracing init so the WebLogLayer can capture all events.
// This gets wired to the gateway's /api/logs/events SSE endpoint later.
@@ -322,7 +325,11 @@ async fn main() -> anyhow::Result<()> {
tracing_subscriber::registry()
.with(env_filter)
.with(tracing_subscriber::fmt::layer().with_target(false))
.with(
tracing_subscriber::fmt::layer()
.with_target(false)
.with_writer(ironclaw::tracing_fmt::TruncatingStderr::default()),
)
.with(WebLogLayer::new(Arc::clone(&log_broadcaster)))
.init();
@@ -418,7 +425,7 @@ async fn main() -> anyhow::Result<()> {
}
// Reload config from DB now that we have a connection.
match Config::from_db(db.as_ref(), "default", &bootstrap).await {
match Config::from_db(db.as_ref(), "default").await {
Ok(db_config) => {
config = db_config;
tracing::info!("Configuration reloaded from database");
@@ -439,10 +446,103 @@ async fn main() -> anyhow::Result<()> {
tracing::warn!("Failed to cleanup stale sandbox jobs: {}", e);
}
}
// Create secrets store early: needed for injecting LLM API keys from encrypted
// storage before creating the LLM provider, and later for MCP auth + WASM channels.
//
// When both `postgres` and `libsql` features are compiled, the runtime-selected
// backend determines which store is created: whichever DB init branch ran will
// have set its handle (pg_pool or libsql_db), and the or_else chain picks it up.
let secrets_store: Option<Arc<dyn SecretsStore + Send + Sync>> =
if let Some(master_key) = config.secrets.master_key() {
match SecretsCrypto::new(master_key.clone()) {
Ok(crypto) => {
let crypto = Arc::new(crypto);
let store: Option<Arc<dyn SecretsStore + Send + Sync>> = None;
#[cfg(feature = "libsql")]
let store = store.or_else(|| {
libsql_db.take().map(|db| {
Arc::new(LibSqlSecretsStore::new(db, Arc::clone(&crypto)))
as Arc<dyn SecretsStore + Send + Sync>
})
});
#[cfg(feature = "postgres")]
let store = store.or_else(|| {
pg_pool.as_ref().map(|pool| {
Arc::new(PostgresSecretsStore::new(pool.clone(), Arc::clone(&crypto)))
as Arc<dyn SecretsStore + Send + Sync>
})
});
store
}
Err(e) => {
tracing::warn!("Failed to initialize secrets crypto: {}", e);
#[cfg(feature = "libsql")]
let _ = libsql_db.take();
None
}
}
} else {
#[cfg(feature = "libsql")]
let _ = libsql_db.take();
None
};
// Inject LLM API keys from the encrypted secrets store into a thread-safe
// overlay so that optional_env() (used by LlmConfig::resolve()) picks them
// up. Then re-resolve LlmConfig with the newly available keys (backend may
// have been set during onboarding but the API key is in the secrets store).
if let Some(ref secrets) = secrets_store {
ironclaw::config::inject_llm_keys_from_secrets(secrets.as_ref(), "default").await;
// Re-resolve LlmConfig now that secrets overlay has been populated
if let Some(ref db_ref) = db {
match Config::from_db(db_ref.as_ref(), "default").await {
Ok(refreshed) => {
config = refreshed;
tracing::debug!("LlmConfig re-resolved after secret injection");
}
Err(e) => {
tracing::warn!("Failed to re-resolve config after secret injection: {}", e);
}
}
}
}
// Initialize LLM provider (clone session so we can reuse it for embeddings)
let llm = create_llm_provider(&config.llm, session.clone())?;
tracing::info!("LLM provider initialized: {}", llm.model_name());
// Wrap in failover if a fallback model is configured
let llm: Arc<dyn LlmProvider> =
if let Some(fallback_model) = config.llm.nearai.fallback_model.as_ref() {
if fallback_model == &config.llm.nearai.model {
tracing::warn!(
"fallback_model is the same as primary model, failover may not be effective"
);
}
let mut fallback_config = config.llm.nearai.clone();
fallback_config.model = fallback_model.clone();
let fallback = create_llm_provider_with_config(&fallback_config, session.clone())?;
tracing::info!(
primary = %llm.model_name(),
fallback = %fallback.model_name(),
"LLM failover enabled"
);
Arc::new(FailoverProvider::new(vec![llm, fallback])?)
} else {
llm
};
// Initialize cheap LLM provider for lightweight tasks (heartbeat, evaluation)
let cheap_llm = create_cheap_llm_provider(&config.llm, session.clone())?;
if let Some(ref cheap) = cheap_llm {
tracing::info!("Cheap LLM provider initialized: {}", cheap.model_name());
}
// Initialize safety layer
let safety = Arc::new(SafetyLayer::new(&config.safety));
tracing::info!("Safety layer initialized");
@@ -516,49 +616,6 @@ async fn main() -> anyhow::Result<()> {
tracing::info!("Builder mode enabled");
}
// Create secrets store if master key is configured (needed for MCP auth and WASM channels).
//
// When both `postgres` and `libsql` features are compiled, the runtime-selected
// backend determines which store is created: whichever DB init branch ran will
// have set its handle (pg_pool or libsql_db), and the or_else chain picks it up.
let secrets_store: Option<Arc<dyn SecretsStore + Send + Sync>> =
if let Some(master_key) = config.secrets.master_key() {
match SecretsCrypto::new(master_key.clone()) {
Ok(crypto) => {
let crypto = Arc::new(crypto);
let store: Option<Arc<dyn SecretsStore + Send + Sync>> = None;
#[cfg(feature = "libsql")]
let store = store.or_else(|| {
libsql_db.take().map(|db| {
Arc::new(LibSqlSecretsStore::new(db, Arc::clone(&crypto)))
as Arc<dyn SecretsStore + Send + Sync>
})
});
#[cfg(feature = "postgres")]
let store = store.or_else(|| {
pg_pool.as_ref().map(|pool| {
Arc::new(PostgresSecretsStore::new(pool.clone(), Arc::clone(&crypto)))
as Arc<dyn SecretsStore + Send + Sync>
})
});
store
}
Err(e) => {
tracing::warn!("Failed to initialize secrets crypto: {}", e);
#[cfg(feature = "libsql")]
let _ = libsql_db.take();
None
}
}
} else {
#[cfg(feature = "libsql")]
let _ = libsql_db.take();
None
};
let mcp_session_manager = Arc::new(McpSessionManager::new());
// Create WASM tool runtime (sync, just builds the wasmtime engine)
@@ -579,7 +636,10 @@ async fn main() -> anyhow::Result<()> {
// Both register into the shared ToolRegistry (RwLock-based) so concurrent writes are safe.
let wasm_tools_future = async {
if let Some(ref runtime) = wasm_tool_runtime {
let loader = WasmToolLoader::new(Arc::clone(runtime), Arc::clone(&tools));
let mut loader = WasmToolLoader::new(Arc::clone(runtime), Arc::clone(&tools));
if let Some(ref secrets) = secrets_store {
loader = loader.with_secrets_store(Arc::clone(secrets));
}
// Load installed tools from ~/.ironclaw/tools/
match loader.load_from_dir(&config.wasm.tools_dir).await {
@@ -902,13 +962,13 @@ async fn main() -> anyhow::Result<()> {
// Inject owner_id for Telegram so the bot only responds
// to the bound user account.
if channel_name == "telegram" {
if let Some(owner_id) = config.channels.telegram_owner_id {
config_updates.insert(
"owner_id".to_string(),
serde_json::json!(owner_id),
);
}
if channel_name == "telegram"
&& let Some(owner_id) = config.channels.telegram_owner_id
{
config_updates.insert(
"owner_id".to_string(),
serde_json::json!(owner_id),
);
}
if !config_updates.is_empty() {
@@ -999,23 +1059,23 @@ async fn main() -> anyhow::Result<()> {
// Extract its routes for the unified server; the channel itself just
// provides the mpsc stream.
let mut webhook_server_addr: Option<std::net::SocketAddr> = None;
if !cli.cli_only {
if let Some(ref http_config) = config.channels.http {
let http_channel = HttpChannel::new(http_config.clone());
webhook_routes.push(http_channel.routes());
let (host, port) = http_channel.addr();
webhook_server_addr = Some(
format!("{}:{}", host, port)
.parse()
.expect("HttpConfig host:port must be a valid SocketAddr"),
);
channels.add(Box::new(http_channel));
tracing::info!(
"HTTP channel enabled on {}:{}",
http_config.host,
http_config.port
);
}
if !cli.cli_only
&& let Some(ref http_config) = config.channels.http
{
let http_channel = HttpChannel::new(http_config.clone());
webhook_routes.push(http_channel.routes());
let (host, port) = http_channel.addr();
webhook_server_addr = Some(
format!("{}:{}", host, port)
.parse()
.expect("HttpConfig host:port must be a valid SocketAddr"),
);
channels.add(Box::new(http_channel));
tracing::info!(
"HTTP channel enabled on {}:{}",
http_config.host,
http_config.port
);
}
// Start the unified webhook server if any routes were registered.
@@ -1134,6 +1194,7 @@ async fn main() -> anyhow::Result<()> {
let deps = AgentDeps {
store: db,
llm,
cheap_llm,
safety,
tools,
workspace,
@@ -1166,13 +1227,11 @@ async fn main() -> anyhow::Result<()> {
/// Check if onboarding is needed and return the reason.
///
/// Returns `Some(reason)` if onboarding should be triggered, `None` otherwise.
/// Called after `load_ironclaw_env()`, so DATABASE_URL from `~/.ironclaw/.env`
/// is already in the environment.
#[cfg(any(feature = "postgres", feature = "libsql"))]
async fn check_onboard_needed() -> Option<&'static str> {
let bootstrap = ironclaw::bootstrap::BootstrapConfig::load();
// Database not configured (and not in env)
let has_db = bootstrap.database_url.is_some()
|| std::env::var("DATABASE_URL").is_ok()
fn check_onboard_needed() -> Option<&'static str> {
let has_db = std::env::var("DATABASE_URL").is_ok()
|| std::env::var("LIBSQL_PATH").is_ok()
|| ironclaw::config::default_libsql_path().exists();
@@ -1180,19 +1239,16 @@ async fn check_onboard_needed() -> Option<&'static str> {
return Some("Database not configured");
}
// Secrets not configured (and not in env)
if bootstrap.secrets_master_key_source == ironclaw::settings::KeySource::None
&& std::env::var("SECRETS_MASTER_KEY").is_err()
&& !ironclaw::secrets::keychain::has_master_key().await
{
// Only require secrets setup if user hasn't explicitly disabled it
// For now, we don't require it for first run
}
// First run (onboarding never completed and no session)
let session_path = ironclaw::llm::session::default_session_path();
if !bootstrap.onboard_completed && !session_path.exists() {
return Some("First run");
// First run (onboarding never completed and no session).
// Reads NEARAI_API_KEY env var directly because this function runs
// before Config is loaded -- Config::from_env() may fail without a
// database URL, which is what triggers onboarding in the first place.
if std::env::var("NEARAI_API_KEY").is_err() {
let settings = ironclaw::settings::Settings::load();
let session_path = ironclaw::llm::session::default_session_path();
if !settings.onboard_completed && !session_path.exists() {
return Some("First run");
}
}
None
+12 -12
View File
@@ -202,7 +202,7 @@ async fn report_complete(
State(state): State<OrchestratorState>,
Path(job_id): Path<Uuid>,
Json(report): Json<CompletionReport>,
) -> Result<StatusCode, StatusCode> {
) -> Result<Json<serde_json::Value>, StatusCode> {
if report.success {
tracing::info!(
job_id = %job_id,
@@ -223,7 +223,7 @@ async fn report_complete(
};
let _ = state.job_manager.complete_job(job_id, result).await;
Ok(StatusCode::OK)
Ok(Json(serde_json::json!({"status": "ok"})))
}
// -- Sandbox job event handlers --
@@ -339,16 +339,16 @@ async fn get_prompt_handler(
Path(job_id): Path<Uuid>,
) -> Result<(StatusCode, Json<serde_json::Value>), StatusCode> {
let mut queue = state.prompt_queue.lock().await;
if let Some(prompts) = queue.get_mut(&job_id) {
if let Some(prompt) = prompts.pop_front() {
return Ok((
StatusCode::OK,
Json(serde_json::json!({
"content": prompt.content,
"done": prompt.done,
})),
));
}
if let Some(prompts) = queue.get_mut(&job_id)
&& let Some(prompt) = prompts.pop_front()
{
return Ok((
StatusCode::OK,
Json(serde_json::json!({
"content": prompt.content,
"done": prompt.done,
})),
));
}
// Return 204 with an empty body. The Json wrapper requires some value
+38 -38
View File
@@ -229,17 +229,17 @@ impl ContainerJobManager {
.unwrap_or_else(|| PathBuf::from("."))
.join(".ironclaw")
.join("projects");
if let Ok(canonical_base) = projects_base.canonicalize() {
if !canonical.starts_with(&canonical_base) {
return Err(OrchestratorError::ContainerCreationFailed {
job_id,
reason: format!(
"project directory {} is outside allowed base {}",
canonical.display(),
canonical_base.display()
),
});
}
if let Ok(canonical_base) = projects_base.canonicalize()
&& !canonical.starts_with(&canonical_base)
{
return Err(OrchestratorError::ContainerCreationFailed {
job_id,
reason: format!(
"project directory {} is outside allowed base {}",
canonical.display(),
canonical_base.display()
),
});
}
binds.push(format!("{}:/workspace:rw", canonical.display()));
env_vec.push("IRONCLAW_WORKSPACE=/workspace".to_string());
@@ -442,36 +442,36 @@ impl ContainerJobManager {
let containers = self.containers.read().await;
containers.get(&job_id).map(|h| h.container_id.clone())
};
if let Some(cid) = container_id {
if !cid.is_empty() {
match connect_docker().await {
Ok(docker) => {
if let Err(e) = docker
.stop_container(
&cid,
Some(bollard::container::StopContainerOptions { t: 5 }),
)
.await
{
tracing::warn!(job_id = %job_id, error = %e, "Failed to stop completed container");
}
if let Err(e) = docker
.remove_container(
&cid,
Some(bollard::container::RemoveContainerOptions {
force: true,
..Default::default()
}),
)
.await
{
tracing::warn!(job_id = %job_id, error = %e, "Failed to remove completed container");
}
if let Some(cid) = container_id
&& !cid.is_empty()
{
match connect_docker().await {
Ok(docker) => {
if let Err(e) = docker
.stop_container(
&cid,
Some(bollard::container::StopContainerOptions { t: 5 }),
)
.await
{
tracing::warn!(job_id = %job_id, error = %e, "Failed to stop completed container");
}
Err(e) => {
tracing::warn!(job_id = %job_id, error = %e, "Failed to connect to Docker for container cleanup");
if let Err(e) = docker
.remove_container(
&cid,
Some(bollard::container::RemoveContainerOptions {
force: true,
..Default::default()
}),
)
.await
{
tracing::warn!(job_id = %job_id, error = %e, "Failed to remove completed container");
}
}
Err(e) => {
tracing::warn!(job_id = %job_id, error = %e, "Failed to connect to Docker for container cleanup");
}
}
}
self.token_store.revoke(job_id).await;
+4 -4
View File
@@ -147,10 +147,10 @@ impl LeakDetector {
// Build prefix matcher for patterns that start with a known prefix
let mut prefixes = Vec::new();
for (idx, pattern) in patterns.iter().enumerate() {
if let Some(prefix) = extract_literal_prefix(pattern.regex.as_str()) {
if prefix.len() >= 3 {
prefixes.push((prefix, idx));
}
if let Some(prefix) = extract_literal_prefix(pattern.regex.as_str())
&& prefix.len() >= 3
{
prefixes.push((prefix, idx));
}
}
+6 -7
View File
@@ -494,10 +494,10 @@ impl ContainerRunner {
/// 3. `~/.docker/run/docker.sock` (Docker Desktop on macOS)
pub async fn connect_docker() -> Result<Docker> {
// First try bollard defaults (checks DOCKER_HOST, then /var/run/docker.sock)
if let Ok(docker) = Docker::connect_with_local_defaults() {
if docker.ping().await.is_ok() {
return Ok(docker);
}
if let Ok(docker) = Docker::connect_with_local_defaults()
&& docker.ping().await.is_ok()
{
return Ok(docker);
}
// Try Docker Desktop socket (macOS)
@@ -507,10 +507,9 @@ pub async fn connect_docker() -> Result<Docker> {
let sock_str = desktop_sock.to_string_lossy();
if let Ok(docker) =
Docker::connect_with_socket(&sock_str, 120, bollard::API_DEFAULT_VERSION)
&& docker.ping().await.is_ok()
{
if docker.ping().await.is_ok() {
return Ok(docker);
}
return Ok(docker);
}
}
}
+9 -9
View File
@@ -259,11 +259,11 @@ async fn handle_connect(
let decision = state.decider.decide(&network_req).await;
if !decision.is_allowed() {
if let NetworkDecision::Deny { reason } = decision {
tracing::info!("Proxy: blocked CONNECT {} - {}", host, reason);
return error_response(StatusCode::FORBIDDEN, reason);
}
if !decision.is_allowed()
&& let NetworkDecision::Deny { reason } = decision
{
tracing::info!("Proxy: blocked CONNECT {} - {}", host, reason);
return error_response(StatusCode::FORBIDDEN, reason);
}
tracing::debug!("Proxy: allowing CONNECT to {}", host);
@@ -294,10 +294,10 @@ async fn forward_request(
// Copy headers (except hop-by-hop headers)
for (name, value) in req.headers() {
if !is_hop_by_hop_header(name.as_str()) {
if let Ok(v) = value.to_str() {
builder = builder.header(name.as_str(), v);
}
if !is_hop_by_hop_header(name.as_str())
&& let Ok(v) = value.to_str()
{
builder = builder.header(name.as_str(), v);
}
}
+4 -5
View File
@@ -109,12 +109,11 @@ impl NetworkPolicyDecider for DefaultPolicyDecider {
async fn decide(&self, request: &NetworkRequest) -> NetworkDecision {
// First check if the domain is allowed
let validation = self.allowlist.is_allowed(&request.host);
if !validation.is_allowed() {
if let crate::sandbox::proxy::allowlist::DomainValidationResult::Denied(reason) =
if !validation.is_allowed()
&& let crate::sandbox::proxy::allowlist::DomainValidationResult::Denied(reason) =
validation
{
return NetworkDecision::Deny { reason };
}
{
return NetworkDecision::Deny { reason };
}
// Check if we need to inject credentials
+1 -1
View File
@@ -261,7 +261,7 @@ pub use platform::{delete_master_key, get_master_key, has_master_key, store_mast
/// Parse a hex string to bytes.
fn hex_to_bytes(hex: &str) -> Result<Vec<u8>, SecretError> {
if hex.len() % 2 != 0 {
if !hex.len().is_multiple_of(2) {
return Err(SecretError::KeychainError(
"Invalid hex string length".to_string(),
));
+59 -22
View File
@@ -153,10 +153,10 @@ impl SecretsStore for PostgresSecretsStore {
let secret = row_to_secret(&r);
// Check expiration
if let Some(expires_at) = secret.expires_at {
if expires_at < Utc::now() {
return Err(SecretError::Expired);
}
if let Some(expires_at) = secret.expires_at
&& expires_at < Utc::now()
{
return Err(SecretError::Expired);
}
Ok(secret)
@@ -276,10 +276,10 @@ impl SecretsStore for PostgresSecretsStore {
}
// Simple glob: * matches any suffix
if let Some(prefix) = pattern.strip_suffix('*') {
if secret_name.starts_with(prefix) {
return Ok(true);
}
if let Some(prefix) = pattern.strip_suffix('*')
&& secret_name.starts_with(prefix)
{
return Ok(true);
}
}
@@ -432,10 +432,10 @@ impl SecretsStore for LibSqlSecretsStore {
Some(row) => {
let secret = libsql_row_to_secret(&row)?;
if let Some(expires_at) = secret.expires_at {
if expires_at < Utc::now() {
return Err(SecretError::Expired);
}
if let Some(expires_at) = secret.expires_at
&& expires_at < Utc::now()
{
return Err(SecretError::Expired);
}
Ok(secret)
@@ -541,10 +541,10 @@ impl SecretsStore for LibSqlSecretsStore {
return Ok(true);
}
if let Some(prefix) = pattern.strip_suffix('*') {
if secret_name.starts_with(prefix) {
return Ok(true);
}
if let Some(prefix) = pattern.strip_suffix('*')
&& secret_name.starts_with(prefix)
{
return Ok(true);
}
}
@@ -695,12 +695,21 @@ pub mod testing {
}
async fn get(&self, user_id: &str, name: &str) -> Result<Secret, SecretError> {
self.secrets
let secret = self
.secrets
.read()
.await
.get(&(user_id.to_string(), name.to_string()))
.cloned()
.ok_or_else(|| SecretError::NotFound(name.to_string()))
.ok_or_else(|| SecretError::NotFound(name.to_string()))?;
if let Some(expires_at) = secret.expires_at
&& expires_at < Utc::now()
{
return Err(SecretError::Expired);
}
Ok(secret)
}
async fn get_decrypted(
@@ -761,10 +770,10 @@ pub mod testing {
if pattern == secret_name {
return Ok(true);
}
if let Some(prefix) = pattern.strip_suffix('*') {
if secret_name.starts_with(prefix) {
return Ok(true);
}
if let Some(prefix) = pattern.strip_suffix('*')
&& secret_name.starts_with(prefix)
{
return Ok(true);
}
}
Ok(false)
@@ -889,6 +898,34 @@ mod tests {
);
}
#[tokio::test]
async fn test_expired_secret_returns_error() {
let store = test_store();
let expires_at = chrono::Utc::now() - chrono::Duration::hours(1);
let params = CreateSecretParams::new("expired_key", "value").with_expiry(expires_at);
store.create("user1", params).await.unwrap();
let result = store.get("user1", "expired_key").await;
assert!(result.is_err());
assert!(matches!(
result.unwrap_err(),
crate::secrets::SecretError::Expired
));
}
#[tokio::test]
async fn test_non_expired_secret_succeeds() {
let store = test_store();
let expires_at = chrono::Utc::now() + chrono::Duration::hours(1);
let params = CreateSecretParams::new("fresh_key", "value").with_expiry(expires_at);
store.create("user1", params).await.unwrap();
let result = store.get("user1", "fresh_key").await;
assert!(result.is_ok());
}
#[tokio::test]
async fn test_user_isolation() {
let store = test_store();
+75 -83
View File
@@ -40,8 +40,18 @@ pub struct Settings {
#[serde(default)]
pub secrets_master_key_source: KeySource,
// === Step 3: NEAR AI Auth ===
// Session stored separately in session.json
// === Step 3: Inference Provider ===
/// LLM backend: "nearai", "anthropic", "openai", "ollama", "openai_compatible".
#[serde(default)]
pub llm_backend: Option<String>,
/// Ollama base URL (when llm_backend = "ollama").
#[serde(default)]
pub ollama_base_url: Option<String>,
/// OpenAI-compatible endpoint base URL (when llm_backend = "openai_compatible").
#[serde(default)]
pub openai_compatible_base_url: Option<String>,
// === Step 4: Model Selection ===
/// Currently selected model.
@@ -499,20 +509,16 @@ impl Default for BuilderSettings {
}
impl Settings {
/// Get the default settings file path (~/.ironclaw/settings.json).
pub fn default_path() -> PathBuf {
dirs::home_dir()
.unwrap_or_else(|| PathBuf::from("."))
.join(".ironclaw")
.join("settings.json")
}
/// Reconstruct Settings from a flat key-value map (as stored in the DB).
///
/// Each key is a dotted path (e.g., "agent.name"), value is a JSONB value.
/// Missing keys get their default value.
pub fn from_db_map(map: &std::collections::HashMap<String, serde_json::Value>) -> Self {
// Start with defaults, then overlay each DB setting
// Start with defaults, then overlay each DB setting.
//
// The settings table stores both Settings struct fields and app-specific
// data (e.g. nearai.session_token). Skip keys that don't correspond to
// a known Settings path.
let mut settings = Self::default();
for (key, value) in map {
@@ -521,17 +527,23 @@ impl Settings {
serde_json::Value::String(s) => s.clone(),
serde_json::Value::Bool(b) => b.to_string(),
serde_json::Value::Number(n) => n.to_string(),
serde_json::Value::Null => "null".to_string(),
serde_json::Value::Null => continue, // null means default, skip
other => other.to_string(),
};
if let Err(e) = settings.set(key, &value_str) {
tracing::warn!(
"Failed to apply DB setting '{}' = '{}': {}",
key,
value_str,
e
);
match settings.set(key, &value_str) {
Ok(()) => {}
// The settings table stores both Settings fields and app-specific
// data (e.g. nearai.session_token). Silently skip unknown paths.
Err(e) if e.starts_with("Path not found") => {}
Err(e) => {
tracing::warn!(
"Failed to apply DB setting '{}' = '{}': {}",
key,
value_str,
e
);
}
}
}
@@ -552,50 +564,27 @@ impl Settings {
map
}
/// Get the default settings file path (~/.ironclaw/settings.json).
pub fn default_path() -> std::path::PathBuf {
dirs::home_dir()
.unwrap_or_else(|| std::path::PathBuf::from("."))
.join(".ironclaw")
.join("settings.json")
}
/// Load settings from disk, returning default if not found.
pub fn load() -> Self {
Self::load_from(&Self::default_path())
}
/// Load settings from a specific path.
pub fn load_from(path: &PathBuf) -> Self {
/// Load settings from a specific path (used by bootstrap legacy migration).
pub fn load_from(path: &std::path::Path) -> Self {
match std::fs::read_to_string(path) {
Ok(data) => serde_json::from_str(&data).unwrap_or_default(),
Err(_) => Self::default(),
}
}
/// Save settings to disk.
pub fn save(&self) -> std::io::Result<()> {
self.save_to(&Self::default_path())
}
/// Save settings to a specific path.
pub fn save_to(&self, path: &PathBuf) -> std::io::Result<()> {
// Ensure parent directory exists
if let Some(parent) = path.parent() {
std::fs::create_dir_all(parent)?;
}
let json = serde_json::to_string_pretty(self)
.map_err(|e| std::io::Error::new(std::io::ErrorKind::InvalidData, e.to_string()))?;
std::fs::write(path, json)
}
/// Get the selected model, falling back to the provided default.
pub fn model_or(&self, default: &str) -> String {
self.selected_model
.clone()
.unwrap_or_else(|| default.to_string())
}
/// Set the selected model and save.
pub fn set_model(&mut self, model: &str) -> std::io::Result<()> {
self.selected_model = Some(model.to_string());
self.save()
}
/// Get a setting value by dotted path (e.g., "agent.max_parallel_jobs").
pub fn get(&self, path: &str) -> Option<String> {
let json = serde_json::to_value(self).ok()?;
@@ -780,42 +769,22 @@ fn collect_settings(
#[cfg(test)]
mod tests {
use super::*;
use tempfile::tempdir;
#[test]
fn test_settings_save_load() {
let dir = tempdir().unwrap();
let path = dir.path().join("settings.json");
fn test_db_map_round_trip() {
let settings = Settings {
selected_model: Some("claude-3-5-sonnet-20241022".to_string()),
..Default::default()
};
settings.save_to(&path).unwrap();
let loaded = Settings::load_from(&path);
let map = settings.to_db_map();
let restored = Settings::from_db_map(&map);
assert_eq!(
loaded.selected_model,
restored.selected_model,
Some("claude-3-5-sonnet-20241022".to_string())
);
}
#[test]
fn test_model_or_default() {
let settings = Settings::default();
assert_eq!(
settings.model_or("default-model"),
"default-model".to_string()
);
let settings = Settings {
selected_model: Some("my-model".to_string()),
..Default::default()
};
assert_eq!(settings.model_or("default-model"), "my-model".to_string());
}
#[test]
fn test_get_setting() {
let settings = Settings::default();
@@ -886,16 +855,13 @@ mod tests {
}
#[test]
fn test_telegram_owner_id_round_trip() {
let dir = tempdir().unwrap();
let path = dir.path().join("settings.json");
fn test_telegram_owner_id_db_round_trip() {
let mut settings = Settings::default();
settings.channels.telegram_owner_id = Some(123456789);
settings.save_to(&path).unwrap();
let loaded = Settings::load_from(&path);
assert_eq!(loaded.channels.telegram_owner_id, Some(123456789));
let map = settings.to_db_map();
let restored = Settings::from_db_map(&map);
assert_eq!(restored.channels.telegram_owner_id, Some(123456789));
}
#[test]
@@ -912,4 +878,30 @@ mod tests {
.unwrap();
assert_eq!(settings.channels.telegram_owner_id, Some(987654321));
}
#[test]
fn test_llm_backend_round_trip() {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("settings.json");
let settings = Settings {
llm_backend: Some("anthropic".to_string()),
ollama_base_url: Some("http://localhost:11434".to_string()),
openai_compatible_base_url: Some("http://my-vllm:8000/v1".to_string()),
..Default::default()
};
let json = serde_json::to_string_pretty(&settings).unwrap();
std::fs::write(&path, json).unwrap();
let loaded = Settings::load_from(&path);
assert_eq!(loaded.llm_backend, Some("anthropic".to_string()));
assert_eq!(
loaded.ollama_base_url,
Some("http://localhost:11434".to_string())
);
assert_eq!(
loaded.openai_compatible_base_url,
Some("http://my-vllm:8000/v1".to_string())
);
}
}
+539
View File
@@ -0,0 +1,539 @@
# Setup / Onboarding Specification
This document is the authoritative specification for IronClaw's onboarding
wizard. Any code change to `src/setup/` **must** keep this document in sync.
If a future contributor or coding agent modifies setup behavior, update this
file first, then adjust the code to match.
---
## Entry Points
```
ironclaw onboard [--skip-auth] [--channels-only]
```
Explicit invocation. Loads `.env` files, runs the wizard, exits.
```
ironclaw (first run, no database configured)
```
Auto-detection via `check_onboard_needed()` in `main.rs`. Triggers when
none of these are true:
- `DATABASE_URL` env var is set
- `LIBSQL_PATH` env var is set
- `~/.ironclaw/ironclaw.db` exists on disk
The `--no-onboard` CLI flag suppresses auto-detection.
---
## Startup Sequence (main.rs)
```
1. Parse CLI args
2. If Command::Onboard → load .env, run wizard, exit
3. If Command::Run or no command:
a. Load .env files (dotenvy::dotenv() then load_ironclaw_env())
b. check_onboard_needed() → run wizard if needed
c. Config::from_env() → build config from env vars
d. Create SessionManager → load session token
e. ensure_authenticated() → validate session (NEAR AI only)
f. ... rest of agent startup
```
**Critical ordering:** `.env` files must be loaded (step 3a) before
`Config::from_env()` (step 3c) because bootstrap vars like
`DATABASE_BACKEND` live in `~/.ironclaw/.env`.
---
## The 7-Step Wizard
### Overview
```
Step 1: Database Connection
Step 2: Security (master key)
Step 3: Inference Provider ← skipped if --skip-auth
Step 4: Model Selection
Step 5: Embeddings
Step 6: Channel Configuration
Step 7: Background Tasks (heartbeat)
save_and_summarize()
```
`--channels-only` mode runs only Step 6, skipping everything else.
---
### Step 1: Database Connection
**Module:** `wizard.rs``step_database()`
**Goal:** Select backend, establish connection, run migrations.
**Decision tree:**
```
Both features compiled?
├─ Yes → DATABASE_BACKEND env var set?
│ ├─ Yes → use that backend
│ └─ No → interactive selection (PostgreSQL vs libSQL)
├─ Only postgres feature → step_database_postgres()
└─ Only libsql feature → step_database_libsql()
```
**PostgreSQL path** (`step_database_postgres`):
1. Check `DATABASE_URL` from env or settings
2. Test connection (creates `deadpool_postgres::Pool`)
3. Optionally run refinery migrations
4. Store pool in `self.db_pool`
**libSQL path** (`step_database_libsql`):
1. Offer local path (default: `~/.ironclaw/ironclaw.db`)
2. Optional Turso cloud sync (URL + auth token)
3. Test connection (creates `LibSqlBackend`)
4. Always run migrations (idempotent CREATE IF NOT EXISTS)
5. Store backend in `self.db_backend`
**Invariant:** After Step 1, exactly one of `self.db_pool` or
`self.db_backend` is `Some`. This is required for settings persistence
in `save_and_summarize()`.
---
### Step 2: Security (Master Key)
**Module:** `wizard.rs``step_security()`
**Goal:** Configure encryption for API tokens and secrets.
**Decision tree:**
```
SECRETS_MASTER_KEY env var set?
├─ Yes → use env var, done
└─ No → try get_master_key() from OS keychain
├─ Ok(bytes) → cache in self.secrets_crypto, ask "use existing?"
│ ├─ Yes → done (keychain)
│ └─ No → clear cache, fall through to options
└─ Err → fall through to options
├─ OS Keychain: generate + store + build SecretsCrypto
├─ Env variable: generate + print export command
└─ Skip: disable secrets features
```
**CRITICAL CAVEAT: macOS Keychain Dialogs**
On macOS, `security_framework::get_generic_password()` can trigger TWO
system dialogs:
1. "Enter your password to unlock the keychain" (keychain locked)
2. "Allow ironclaw to access this keychain item" (per-app authorization)
This is OS-level behavior we cannot prevent. To minimize pain:
- **Use `get_master_key()` not `has_master_key()`** in step 2. Both call
the same underlying API, but `get_master_key()` returns the key bytes
so we can cache them. `has_master_key()` throws them away, forcing a
second keychain access later.
- **Build `SecretsCrypto` eagerly.** When the keychain key is retrieved,
immediately construct `SecretsCrypto` and store in `self.secrets_crypto`.
Later calls to `init_secrets_context()` check this field first, avoiding
redundant keychain probes.
- **Never probe the keychain in read-only commands** (e.g., `ironclaw status`).
The status command reports "env not set (keychain may be configured)"
rather than triggering system dialogs.
**Invariant:** After Step 2, `self.secrets_crypto` is `Some` if the user
chose Keychain or generated a new key. It may be `None` if the user chose
env-var mode or skipped secrets.
---
### Step 3: Inference Provider
**Module:** `wizard.rs``step_inference_provider()`
**Goal:** Choose LLM backend and authenticate.
**Providers:**
| Provider | Auth Method | Secret Name | Env Var |
|----------|-------------|-------------|---------|
| NEAR AI | Browser OAuth | (session token) | `NEARAI_SESSION_TOKEN` |
| Anthropic | API key | `anthropic_api_key` | `ANTHROPIC_API_KEY` |
| OpenAI | API key | `openai_api_key` | `OPENAI_API_KEY` |
| Ollama | None | - | - |
| OpenAI-compatible | Optional API key | `llm_compatible_api_key` | `LLM_API_KEY` |
**API-key providers** (`setup_api_key_provider`):
1. Check env var → if set, ask to reuse, persist to secrets store
2. Otherwise prompt for key entry via `secret_input()`
3. Store encrypted in secrets via `init_secrets_context()`
4. **Cache key in `self.llm_api_key`** for model fetching in Step 4
**NEAR AI** (`setup_nearai`):
- Calls `session_manager.ensure_authenticated()` which opens browser
- Session token saved to `~/.ironclaw/session.json`
**`self.llm_api_key` caching:** The wizard caches the API key as
`Option<SecretString>` so that Step 4 (model fetching) and Step 5
(embeddings) can use it without re-reading from the secrets store or
mutating environment variables.
---
### Step 4: Model Selection
**Module:** `wizard.rs``step_model_selection()`
**Goal:** Choose which model to use.
**Flow:**
1. If model already set → offer to keep it
2. Fetch models from provider API (5-second timeout)
3. On timeout or error → use static fallback list
4. Present list + "Custom model ID" escape hatch
5. Store in `self.settings.selected_model`
**Model fetchers pass the cached API key explicitly:**
```rust
let cached = self.llm_api_key.as_ref().map(|k| k.expose_secret().to_string());
let models = fetch_anthropic_models(cached.as_deref()).await;
```
This avoids mutating environment variables. The fetcher checks the explicit
key first, then falls back to the standard env var.
---
### Step 5: Embeddings
**Module:** `wizard.rs``step_embeddings()`
**Goal:** Configure semantic search for workspace memory.
**Flow:**
1. Ask "Enable semantic search?" (default: yes)
2. Detect available providers:
- NEAR AI: if backend is `nearai` OR valid session exists
- OpenAI: if `OPENAI_API_KEY` in env OR (backend is `openai` AND cached key)
3. If both available → let user choose
4. If only one → use it
5. If neither → disable embeddings
**Default model:** `text-embedding-3-small` (for both providers)
---
### Step 6: Channel Configuration
**Module:** `wizard.rs``step_channels()`, delegating to `channels.rs`
**Goal:** Enable input channels (TUI, HTTP, Telegram, etc.).
**Sub-steps:**
```
6a. Tunnel setup (if webhook channels needed)
6b. Discover WASM channels from ~/.ironclaw/channels/
6c. Multi-select: CLI/TUI, HTTP, discovered channels, bundled channels
6d. Install missing bundled channels (copy WASM binaries)
6e. Initialize SecretsContext (for token storage)
6f. Setup HTTP webhook (if selected)
6g. Setup each WASM channel (secrets, owner binding)
```
**Tunnel setup** (`setup_tunnel`):
- Options: ngrok, Cloudflare Tunnel, localtunnel, custom URL
- Validates HTTPS requirement
- Stored in `self.settings.tunnel.public_url`
**WASM channel setup** (`setup_wasm_channel`):
- Reads `capabilities.json` for `setup.required_secrets`
- For each secret: check existing, prompt or auto-generate, validate regex
- Save each secret via `SecretsContext`
**Telegram special case** (`setup_telegram`):
- Validates bot token via Telegram `getMe` API
- Owner binding: polls `getUpdates` for 120s to capture sender's user ID
- Optional webhook secret generation
**SecretsContext creation** (`init_secrets_context`):
1. Check `self.secrets_crypto` (set in Step 2) → use if available
2. Else try `SECRETS_MASTER_KEY` env var
3. Else try `get_master_key()` from keychain (only in `channels_only` mode)
4. Create backend-appropriate secrets store (respects selected database backend)
---
### Step 7: Heartbeat
**Module:** `wizard.rs``step_heartbeat()`
**Goal:** Configure periodic background execution.
**Flow:**
1. Ask "Enable heartbeat?" (default: no)
2. If yes: interval in minutes (default: 30), notification channel
3. Store in `self.settings.heartbeat`
---
## Settings Persistence
### Two-Layer Architecture
Settings are persisted in two places:
**Layer 1: `~/.ironclaw/.env`** (bootstrap vars)
Contains only the settings needed BEFORE database connection. Written by
`save_bootstrap_env()` in `bootstrap.rs`.
```env
DATABASE_BACKEND="libsql"
LIBSQL_PATH="/Users/name/.ironclaw/ironclaw.db"
```
Or for PostgreSQL:
```env
DATABASE_BACKEND="postgres"
DATABASE_URL="postgres://user:pass@localhost/ironclaw"
```
**Why separate?** Chicken-and-egg: you need `DATABASE_BACKEND` to know
which database to connect to, so it can't be stored in the database.
**Layer 2: Database settings table** (everything else)
All other settings are stored as key-value pairs in the `settings` table,
keyed by `(user_id, key)`. Written by `set_all_settings()`.
Settings are serialized via `Settings::to_db_map()` as dotted paths:
```
database_backend = "libsql"
llm_backend = "nearai"
selected_model = "anthropic/claude-sonnet-4-5"
embeddings.enabled = "true"
embeddings.provider = "nearai"
channels.http_enabled = "true"
heartbeat.enabled = "true"
heartbeat.interval_secs = "300"
```
### save_and_summarize()
Final step of the wizard:
```
1. Mark onboard_completed = true
2. Write ALL settings to database (try postgres pool, then libSQL backend)
3. Write bootstrap vars to ~/.ironclaw/.env:
- DATABASE_BACKEND (always)
- DATABASE_URL (if postgres)
- LIBSQL_PATH (if libsql)
- LIBSQL_URL (if turso sync)
4. Print configuration summary
```
**Invariant:** Both Layer 1 and Layer 2 must be written. If the database
write fails, the wizard returns an error and the `.env` file is not written.
### Legacy Migration
`bootstrap.rs` handles one-time upgrades from older config formats:
- `bootstrap.json` → extracts `DATABASE_URL`, writes `.env`, renames to `.migrated`
- `settings.json` → migrated to database via `migrate_disk_to_db()`
---
## Settings Struct
**Module:** `settings.rs`
```rust
pub struct Settings {
// Meta
pub onboard_completed: bool,
// Step 1: Database
pub database_backend: Option<String>, // "postgres" | "libsql"
pub database_url: Option<String>,
pub libsql_path: Option<String>,
pub libsql_url: Option<String>,
// Step 2: Security
pub secrets_master_key_source: KeySource, // Keychain | Env | None
// Step 3: Inference
pub llm_backend: Option<String>, // "nearai" | "anthropic" | "openai" | "ollama" | "openai_compatible"
pub ollama_base_url: Option<String>,
pub openai_compatible_base_url: Option<String>,
// Step 4: Model
pub selected_model: Option<String>,
// Step 5: Embeddings
pub embeddings: EmbeddingsSettings, // enabled, provider, model
// Step 6: Channels
pub tunnel: TunnelSettings, // provider, public_url
pub channels: ChannelSettings, // http config, telegram owner, etc.
// Step 7: Heartbeat
pub heartbeat: HeartbeatSettings, // enabled, interval, notify
// Advanced (not in wizard, set via `ironclaw config set`)
pub agent: AgentSettings,
pub wasm: WasmSettings,
pub sandbox: SandboxSettings,
pub safety: SafetySettings,
pub builder: BuilderSettings,
}
```
**KeySource enum:** `Keychain | Env | None`
---
## Secrets Flow
### SecretsContext
Thin wrapper for setup-time secret operations:
```rust
pub struct SecretsContext {
store: Arc<dyn SecretsStore>,
user_id: String,
}
```
Created by `init_secrets_context()` which:
1. Gets `SecretsCrypto` from `self.secrets_crypto` or loads from keychain/env
2. Creates the appropriate backend store:
- If both features compiled: respects `self.settings.database_backend`
- Tries selected backend first, falls back to the other
3. Returns `SecretsContext` wrapping the store
### Secret Storage
Secrets are encrypted with AES-256-GCM using the master key, then stored
in the database `secrets` table. The wizard writes secrets like:
```
telegram_bot_token → encrypted bot token
telegram_webhook_secret → encrypted webhook HMAC secret
anthropic_api_key → encrypted API key
```
---
## Prompt Utilities
**Module:** `prompts.rs`
| Function | Description |
|----------|-------------|
| `select_one(label, options)` | Numbered single-choice menu |
| `select_many(label, options, defaults)` | Checkbox multi-select (raw terminal mode) |
| `input(label)` | Single line text input |
| `optional_input(label, hint)` | Text input that can be empty |
| `secret_input(label)` | Hidden input (shows `*` per char), returns `SecretString` |
| `confirm(label, default)` | `[Y/n]` or `[y/N]` prompt |
| `print_header(text)` | Bold section header with underline |
| `print_step(n, total, text)` | `[1/7] Step Name` |
| `print_success(text)` | Green checkmark prefix |
| `print_error(text)` | Red X prefix |
| `print_info(text)` | Blue info prefix |
`select_many` uses `crossterm` raw mode for arrow key navigation.
Must properly restore terminal state on all exit paths.
---
## Platform Caveats
### macOS Keychain
- `get_generic_password()` triggers system dialogs (unlock + authorize)
- Two dialogs per call is normal, not a bug
- Cache the result after first access to avoid repeat prompts
- Never probe keychain in read-only commands (`status`, `--help`)
- Service name: `"ironclaw"`, account: `"master_key"`
### Linux Secret Service
- Uses GNOME Keyring or KWallet via `secret-service` crate
- May need `gnome-keyring` daemon running
- Collection unlock may prompt for password
### URL Passwords
- `#` is common in URL-encoded passwords (`%23` decoded)
- `.env` values must be double-quoted to preserve `#`
- Display masked: `postgres://user:****@host/db`
### Telegram API
- Bot token format: `123456:ABC-DEF...`
- Token goes in URL path: `https://api.telegram.org/bot{TOKEN}/method`
- Webhook secret header: `X-Telegram-Bot-Api-Secret-Token`
- Owner binding polls `getUpdates` (must delete webhook first)
---
## Testing
Tests live in `mod tests {}` at the bottom of each file.
**What to test when modifying setup:**
- Settings round-trip: `to_db_map()` then `from_db_map()` preserves values
- Bootstrap `.env`: dotenvy can parse what `save_bootstrap_env()` writes
- Model fetchers: static fallback works when API is unreachable
- Channel discovery: handles missing dir, invalid JSON, deduplication
- Prompt functions: not tested (interactive I/O), but ensure error paths
don't panic
**Run setup tests:**
```bash
cargo test --lib -- setup
cargo test --lib -- bootstrap
```
---
## Modification Checklist
When changing the onboarding flow:
1. Update this README first with the intended behavior change
2. If adding a new wizard step:
- Add to the step enum in `run()`, adjust `total_steps`
- Add corresponding settings fields to `Settings`
- Add `to_db_map` / `from_db_map` serialization
- If the setting is needed before DB connection, add to `save_bootstrap_env()`
3. If adding a new provider or channel:
- Add to the selection menu in the appropriate step
- Add authentication flow (API key or OAuth)
- Add model fetcher with static fallback + 5s timeout
4. If touching keychain:
- Cache the result, never call `get_master_key()` twice
- Test on macOS (dialog behavior differs from Linux)
5. If touching secrets:
- Ensure `init_secrets_context()` respects the selected database backend
- Test with both postgres and libsql features
6. Run the full shipping checklist:
```bash
cargo fmt
cargo clippy --all --benches --tests --examples --all-features -- -D warnings
cargo test --lib -- setup bootstrap
```
7. Test a fresh onboarding: `rm -rf ~/.ironclaw && cargo run`
+184 -145
View File
@@ -15,11 +15,27 @@ use serde::Deserialize;
#[cfg(feature = "postgres")]
use crate::secrets::SecretsCrypto;
use crate::secrets::{CreateSecretParams, SecretsStore};
use crate::settings::Settings;
use crate::settings::{Settings, TunnelSettings};
use crate::setup::prompts::{
confirm, input, optional_input, print_error, print_info, print_success, secret_input,
};
/// Typed errors for channel setup flows.
#[derive(Debug, thiserror::Error)]
pub enum ChannelSetupError {
#[error("I/O error: {0}")]
Io(#[from] std::io::Error),
#[error("{0}")]
Network(String),
#[error("{0}")]
Secrets(String),
#[error("{0}")]
Validation(String),
}
/// Context for saving secrets during setup.
pub struct SecretsContext {
store: Arc<dyn SecretsStore>,
@@ -45,32 +61,39 @@ impl SecretsContext {
}
/// Save a secret to the database.
pub async fn save_secret(&self, name: &str, value: &SecretString) -> Result<(), String> {
pub async fn save_secret(
&self,
name: &str,
value: &SecretString,
) -> Result<(), ChannelSetupError> {
let params = CreateSecretParams::new(name, value.expose_secret());
self.store
.create(&self.user_id, params)
.await
.map_err(|e| format!("Failed to save secret: {}", e))?;
.map_err(|e| ChannelSetupError::Secrets(format!("Failed to save secret: {}", e)))?;
Ok(())
}
/// Check if a secret exists.
pub async fn secret_exists(&self, name: &str) -> bool {
self.store
.exists(&self.user_id, name)
.await
.unwrap_or(false)
match self.store.exists(&self.user_id, name).await {
Ok(exists) => exists,
Err(e) => {
tracing::warn!(secret = name, error = %e, "Failed to check if secret exists, assuming absent");
false
}
}
}
/// Read a secret from the database (decrypted).
pub async fn get_secret(&self, name: &str) -> Result<SecretString, String> {
pub async fn get_secret(&self, name: &str) -> Result<SecretString, ChannelSetupError> {
let decrypted = self
.store
.get_decrypted(&self.user_id, name)
.await
.map_err(|e| format!("Failed to read secret: {}", e))?;
.map_err(|e| ChannelSetupError::Secrets(format!("Failed to read secret: {}", e)))?;
Ok(SecretString::from(decrypted.expose().to_string()))
}
}
@@ -107,7 +130,6 @@ struct TelegramGetUpdatesResponse {
#[derive(Debug, Deserialize)]
struct TelegramUpdate {
#[allow(dead_code)]
update_id: i64,
message: Option<TelegramUpdateMessage>,
}
@@ -131,7 +153,10 @@ struct TelegramUpdateUser {
/// 2. Entering the bot token
/// 3. Validating the token
/// 4. Saving the token to the database
pub async fn setup_telegram(secrets: &SecretsContext) -> Result<TelegramSetupResult, String> {
pub async fn setup_telegram(
secrets: &SecretsContext,
settings: &Settings,
) -> Result<TelegramSetupResult, ChannelSetupError> {
println!("Telegram Setup:");
println!();
print_info("To create a Telegram bot:");
@@ -143,10 +168,10 @@ pub async fn setup_telegram(secrets: &SecretsContext) -> Result<TelegramSetupRes
// Check if token already exists
if secrets.secret_exists("telegram_bot_token").await {
print_info("Existing Telegram token found in database.");
if !confirm("Replace existing token?", false).map_err(|e| e.to_string())? {
if !confirm("Replace existing token?", false)? {
// Still offer to configure webhook secret and owner binding
let webhook_secret = setup_telegram_webhook_secret(secrets).await?;
let owner_id = bind_telegram_owner_flow(secrets).await?;
let webhook_secret = setup_telegram_webhook_secret(secrets, &settings.tunnel).await?;
let owner_id = bind_telegram_owner_flow(secrets, settings).await?;
return Ok(TelegramSetupResult {
enabled: true,
bot_username: None,
@@ -156,47 +181,48 @@ pub async fn setup_telegram(secrets: &SecretsContext) -> Result<TelegramSetupRes
}
}
let token = secret_input("Bot token (from @BotFather)").map_err(|e| e.to_string())?;
loop {
let token = secret_input("Bot token (from @BotFather)")?;
// Validate the token
print_info("Validating bot token...");
// Validate the token
print_info("Validating bot token...");
match validate_telegram_token(&token).await {
Ok(username) => {
print_success(&format!(
"Bot validated: @{}",
username.as_deref().unwrap_or("unknown")
));
match validate_telegram_token(&token).await {
Ok(username) => {
print_success(&format!(
"Bot validated: @{}",
username.as_deref().unwrap_or("unknown")
));
// Save to database
secrets.save_secret("telegram_bot_token", &token).await?;
print_success("Token saved to database");
// Save to database
secrets.save_secret("telegram_bot_token", &token).await?;
print_success("Token saved to database");
// Bind bot to owner's Telegram account
let owner_id = bind_telegram_owner(&token).await?;
// Bind bot to owner's Telegram account
let owner_id = bind_telegram_owner(&token).await?;
// Offer webhook secret configuration
let webhook_secret = setup_telegram_webhook_secret(secrets).await?;
// Offer webhook secret configuration
let webhook_secret =
setup_telegram_webhook_secret(secrets, &settings.tunnel).await?;
Ok(TelegramSetupResult {
enabled: true,
bot_username: username,
webhook_secret,
owner_id,
})
}
Err(e) => {
print_error(&format!("Token validation failed: {}", e));
return Ok(TelegramSetupResult {
enabled: true,
bot_username: username,
webhook_secret,
owner_id,
});
}
Err(e) => {
print_error(&format!("Token validation failed: {}", e));
if confirm("Try again?", true).map_err(|e| e.to_string())? {
Box::pin(setup_telegram(secrets)).await
} else {
Ok(TelegramSetupResult {
enabled: false,
bot_username: None,
webhook_secret: None,
owner_id: None,
})
if !confirm("Try again?", true)? {
return Ok(TelegramSetupResult {
enabled: false,
bot_username: None,
webhook_secret: None,
owner_id: None,
});
}
}
}
}
@@ -206,14 +232,14 @@ pub async fn setup_telegram(secrets: &SecretsContext) -> Result<TelegramSetupRes
///
/// Polls `getUpdates` until a message arrives, then captures the sender's user ID.
/// Returns `None` if the user declines or the flow times out.
async fn bind_telegram_owner(token: &SecretString) -> Result<Option<i64>, String> {
async fn bind_telegram_owner(token: &SecretString) -> Result<Option<i64>, ChannelSetupError> {
println!();
print_info("Account Binding (recommended):");
print_info("Binding restricts the bot so only YOU can use it.");
print_info("Without this, anyone who finds your bot can send it messages.");
println!();
if !confirm("Bind bot to your Telegram account?", true).map_err(|e| e.to_string())? {
if !confirm("Bind bot to your Telegram account?", true)? {
print_info("Skipping account binding. Bot will accept messages from all users.");
return Ok(None);
}
@@ -224,14 +250,16 @@ async fn bind_telegram_owner(token: &SecretString) -> Result<Option<i64>, String
let client = Client::builder()
.timeout(std::time::Duration::from_secs(35))
.build()
.map_err(|e| format!("Failed to create HTTP client: {}", e))?;
.map_err(|e| ChannelSetupError::Network(format!("Failed to create HTTP client: {}", e)))?;
// Clear any existing webhook so getUpdates works
let delete_url = format!(
"https://api.telegram.org/bot{}/deleteWebhook",
token.expose_secret()
);
let _ = client.post(&delete_url).send().await;
if let Err(e) = client.post(&delete_url).send().await {
tracing::warn!("Failed to delete webhook (getUpdates may not work): {e}");
}
let updates_url = format!(
"https://api.telegram.org/bot{}/getUpdates",
@@ -246,49 +274,56 @@ async fn bind_telegram_owner(token: &SecretString) -> Result<Option<i64>, String
.query(&[("timeout", "30"), ("allowed_updates", "[\"message\"]")])
.send()
.await
.map_err(|e| format!("getUpdates request failed: {}", e))?;
.map_err(|e| ChannelSetupError::Network(format!("getUpdates request failed: {}", e)))?;
if !response.status().is_success() {
return Err(format!("getUpdates returned status {}", response.status()));
return Err(ChannelSetupError::Network(format!(
"getUpdates returned status {}",
response.status()
)));
}
let body: TelegramGetUpdatesResponse = response
.json()
.await
.map_err(|e| format!("Failed to parse getUpdates response: {}", e))?;
let body: TelegramGetUpdatesResponse = response.json().await.map_err(|e| {
ChannelSetupError::Network(format!("Failed to parse getUpdates response: {}", e))
})?;
if !body.ok {
return Err("Telegram API returned error for getUpdates".to_string());
return Err(ChannelSetupError::Network(
"Telegram API returned error for getUpdates".to_string(),
));
}
// Find the first message with a sender
for update in &body.result {
if let Some(ref msg) = update.message {
if let Some(ref from) = msg.from {
let display_name = from
.username
.as_ref()
.map(|u| format!("@{}", u))
.unwrap_or_else(|| from.first_name.clone());
if let Some(ref msg) = update.message
&& let Some(ref from) = msg.from
{
let display_name = from
.username
.as_ref()
.map(|u| format!("@{}", u))
.unwrap_or_else(|| from.first_name.clone());
print_success(&format!(
"Received message from {} (ID: {})",
display_name, from.id
));
print_success(&format!(
"Received message from {} (ID: {})",
display_name, from.id
));
// Acknowledge the update so it doesn't pile up
let ack_url = format!(
"https://api.telegram.org/bot{}/getUpdates",
token.expose_secret()
);
let _ = client
.get(&ack_url)
.query(&[("offset", &(update.update_id + 1).to_string())])
.send()
.await;
return Ok(Some(from.id));
// Acknowledge the update so it doesn't pile up
let ack_url = format!(
"https://api.telegram.org/bot{}/getUpdates",
token.expose_secret()
);
if let Err(e) = client
.get(&ack_url)
.query(&[("offset", &(update.update_id + 1).to_string())])
.send()
.await
{
tracing::warn!("Failed to acknowledge Telegram update: {e}");
}
return Ok(Some(from.id));
}
}
}
@@ -301,12 +336,13 @@ async fn bind_telegram_owner(token: &SecretString) -> Result<Option<i64>, String
/// Bind flow when the token already exists (reads from secrets store).
///
/// Retrieves the saved bot token and delegates to `bind_telegram_owner`.
async fn bind_telegram_owner_flow(secrets: &SecretsContext) -> Result<Option<i64>, String> {
// Check current settings first
let settings = Settings::load();
async fn bind_telegram_owner_flow(
secrets: &SecretsContext,
settings: &Settings,
) -> Result<Option<i64>, ChannelSetupError> {
if settings.channels.telegram_owner_id.is_some() {
print_info("Bot is already bound to a Telegram account.");
if !confirm("Re-bind to a different account?", false).map_err(|e| e.to_string())? {
if !confirm("Re-bind to a different account?", false)? {
return Ok(settings.channels.telegram_owner_id);
}
}
@@ -321,12 +357,10 @@ async fn bind_telegram_owner_flow(secrets: &SecretsContext) -> Result<Option<i64
///
/// This is shared across all channels that need webhook endpoints.
/// Returns the tunnel URL if configured.
pub fn setup_tunnel() -> Result<Option<String>, String> {
// Check if already configured
let settings = Settings::load();
pub fn setup_tunnel(settings: &Settings) -> Result<Option<String>, ChannelSetupError> {
if let Some(ref url) = settings.tunnel.public_url {
print_info(&format!("Existing tunnel configured: {}", url));
if !confirm("Change tunnel configuration?", false).map_err(|e| e.to_string())? {
if !confirm("Change tunnel configuration?", false)? {
return Ok(Some(url.clone()));
}
}
@@ -346,30 +380,24 @@ pub fn setup_tunnel() -> Result<Option<String>, String> {
print_info("Security comes from provider-specific secrets (e.g., Telegram webhook secret).");
println!();
if !confirm("Configure a tunnel?", false).map_err(|e| e.to_string())? {
if !confirm("Configure a tunnel?", false)? {
return Ok(None);
}
let tunnel_url =
input("Tunnel URL (e.g., https://abc123.ngrok.io)").map_err(|e| e.to_string())?;
let tunnel_url = input("Tunnel URL (e.g., https://abc123.ngrok.io)")?;
// Validate URL format
if !tunnel_url.starts_with("https://") {
print_error("URL must start with https:// (webhooks require HTTPS)");
return Err("Invalid tunnel URL: must use HTTPS".to_string());
return Err(ChannelSetupError::Validation(
"Invalid tunnel URL: must use HTTPS".to_string(),
));
}
// Remove trailing slash if present
let tunnel_url = tunnel_url.trim_end_matches('/').to_string();
// Save to settings
let mut settings = Settings::load();
settings.tunnel.public_url = Some(tunnel_url.clone());
settings
.save()
.map_err(|e| format!("Failed to save settings: {}", e))?;
print_success(&format!("Tunnel URL saved: {}", tunnel_url));
print_success(&format!("Tunnel URL configured: {}", tunnel_url));
print_info("");
print_info("Make sure your tunnel is running before starting the agent.");
print_info("You can also set TUNNEL_URL environment variable to override.");
@@ -380,10 +408,11 @@ pub fn setup_tunnel() -> Result<Option<String>, String> {
/// Set up Telegram webhook secret for signature validation.
///
/// Returns the webhook secret if configured.
async fn setup_telegram_webhook_secret(secrets: &SecretsContext) -> Result<Option<String>, String> {
// Check if tunnel is configured
let settings = Settings::load();
if settings.tunnel.public_url.is_none() {
async fn setup_telegram_webhook_secret(
secrets: &SecretsContext,
tunnel: &TunnelSettings,
) -> Result<Option<String>, ChannelSetupError> {
if tunnel.public_url.is_none() {
print_info("");
print_info("No tunnel configured. Telegram will use polling mode (30s+ delay).");
print_info("Run setup again to configure a tunnel for instant delivery.");
@@ -395,7 +424,7 @@ async fn setup_telegram_webhook_secret(secrets: &SecretsContext) -> Result<Optio
print_info("A webhook secret adds an extra layer of security by validating");
print_info("that requests actually come from Telegram's servers.");
if !confirm("Generate a webhook secret?", true).map_err(|e| e.to_string())? {
if !confirm("Generate a webhook secret?", true)? {
return Ok(None);
}
@@ -414,11 +443,13 @@ async fn setup_telegram_webhook_secret(secrets: &SecretsContext) -> Result<Optio
/// Validate a Telegram bot token by calling the getMe API.
///
/// Returns the bot's username if valid.
pub async fn validate_telegram_token(token: &SecretString) -> Result<Option<String>, String> {
pub async fn validate_telegram_token(
token: &SecretString,
) -> Result<Option<String>, ChannelSetupError> {
let client = Client::builder()
.timeout(std::time::Duration::from_secs(10))
.build()
.map_err(|e| format!("Failed to create HTTP client: {}", e))?;
.map_err(|e| ChannelSetupError::Network(format!("Failed to create HTTP client: {}", e)))?;
let url = format!(
"https://api.telegram.org/bot{}/getMe",
@@ -429,21 +460,26 @@ pub async fn validate_telegram_token(token: &SecretString) -> Result<Option<Stri
.get(&url)
.send()
.await
.map_err(|e| format!("Request failed: {}", e))?;
.map_err(|e| ChannelSetupError::Network(format!("Request failed: {}", e)))?;
if !response.status().is_success() {
return Err(format!("API returned status {}", response.status()));
return Err(ChannelSetupError::Network(format!(
"API returned status {}",
response.status()
)));
}
let body: TelegramGetMeResponse = response
.json()
.await
.map_err(|e| format!("Failed to parse response: {}", e))?;
.map_err(|e| ChannelSetupError::Network(format!("Failed to parse response: {}", e)))?;
if body.ok {
Ok(body.result.and_then(|u| u.username))
} else {
Err("Telegram API returned error".to_string())
Err(ChannelSetupError::Network(
"Telegram API returned error".to_string(),
))
}
}
@@ -456,38 +492,34 @@ pub struct HttpSetupResult {
}
/// Set up HTTP webhook channel.
pub async fn setup_http(secrets: &SecretsContext) -> Result<HttpSetupResult, String> {
pub async fn setup_http(secrets: &SecretsContext) -> Result<HttpSetupResult, ChannelSetupError> {
println!("HTTP Webhook Setup:");
println!();
print_info("The HTTP webhook allows external services to send messages to the agent.");
println!();
let port_str = optional_input("Port", Some("default: 8080")).map_err(|e| e.to_string())?;
let port_str = optional_input("Port", Some("default: 8080"))?;
let port: u16 = port_str
.as_deref()
.unwrap_or("8080")
.parse()
.map_err(|e| format!("Invalid port: {}", e))?;
.map_err(|e| ChannelSetupError::Validation(format!("Invalid port: {}", e)))?;
if port < 1024 {
print_info("Note: Ports below 1024 may require root privileges");
}
let host = optional_input("Host", Some("default: 0.0.0.0"))
.map_err(|e| e.to_string())?
.unwrap_or_else(|| "0.0.0.0".to_string());
let host =
optional_input("Host", Some("default: 0.0.0.0"))?.unwrap_or_else(|| "0.0.0.0".to_string());
// Generate a webhook secret
if confirm("Generate a webhook secret for authentication?", true).map_err(|e| e.to_string())? {
if confirm("Generate a webhook secret for authentication?", true)? {
let secret = generate_webhook_secret();
secrets
.save_secret("http_webhook_secret", &SecretString::from(secret.clone()))
.save_secret("http_webhook_secret", &SecretString::from(secret))
.await?;
print_success("Webhook secret generated and saved to database");
print_info(&format!(
"Secret: {} (store this for your webhook clients)",
secret
));
print_info("Retrieve it later with: ironclaw secret get http_webhook_secret");
}
print_success(&format!("HTTP webhook will listen on {}:{}", host, port));
@@ -501,11 +533,7 @@ pub async fn setup_http(secrets: &SecretsContext) -> Result<HttpSetupResult, Str
/// Generate a random webhook secret.
pub fn generate_webhook_secret() -> String {
use rand::RngCore;
let mut rng = rand::thread_rng();
let mut bytes = [0u8; 32];
rng.fill_bytes(&mut bytes);
bytes.iter().map(|b| format!("{:02x}", b)).collect()
generate_secret_with_length(32)
}
/// Result of WASM channel setup.
@@ -523,7 +551,7 @@ pub async fn setup_wasm_channel(
secrets: &SecretsContext,
channel_name: &str,
setup: &crate::channels::wasm::SetupSchema,
) -> Result<WasmChannelSetupResult, String> {
) -> Result<WasmChannelSetupResult, ChannelSetupError> {
println!("{} Setup:", channel_name);
println!();
@@ -534,7 +562,7 @@ pub async fn setup_wasm_channel(
"Existing {} found in database.",
secret_config.name
));
if !confirm("Replace existing value?", false).map_err(|e| e.to_string())? {
if !confirm("Replace existing value?", false)? {
continue;
}
}
@@ -542,8 +570,7 @@ pub async fn setup_wasm_channel(
// Get the value from user or auto-generate
let value = if secret_config.optional {
let input_value =
optional_input(&secret_config.prompt, Some("leave empty to auto-generate"))
.map_err(|e| e.to_string())?;
optional_input(&secret_config.prompt, Some("leave empty to auto-generate"))?;
if let Some(v) = input_value {
if !v.is_empty() {
@@ -570,18 +597,21 @@ pub async fn setup_wasm_channel(
}
} else {
// Required secret
let input_value = secret_input(&secret_config.prompt).map_err(|e| e.to_string())?;
let input_value = secret_input(&secret_config.prompt)?;
// Validate if pattern is provided
if let Some(ref pattern) = secret_config.validation {
let re = regex::Regex::new(pattern)
.map_err(|e| format!("Invalid validation pattern: {}", e))?;
let re = regex::Regex::new(pattern).map_err(|e| {
ChannelSetupError::Validation(format!("Invalid validation pattern: {}", e))
})?;
if !re.is_match(input_value.expose_secret()) {
print_error(&format!(
"Value does not match expected format: {}",
pattern
));
return Err("Validation failed".to_string());
return Err(ChannelSetupError::Validation(
"Validation failed".to_string(),
));
}
}
@@ -593,14 +623,11 @@ pub async fn setup_wasm_channel(
print_success(&format!("{} saved to database", secret_config.name));
}
// Optionally validate the configuration
// TODO: Substitute secrets into the validation URL and make a
// GET request to verify the configured credentials actually work.
if let Some(ref validation_endpoint) = setup.validation_endpoint {
print_info("Validating configuration...");
// The validation endpoint may contain placeholders like {telegram_bot_token}
// For now, we skip validation since we'd need to substitute secrets
// A full implementation would fetch secrets and substitute them
print_info(&format!(
"Validation endpoint configured: {} (validation skipped)",
"Validation endpoint configured: {} (validation not yet implemented)",
validation_endpoint
));
}
@@ -624,11 +651,23 @@ fn generate_secret_with_length(length: usize) -> String {
#[cfg(test)]
mod tests {
use super::*;
use crate::setup::channels::generate_webhook_secret;
#[test]
fn test_generate_webhook_secret() {
let secret = generate_webhook_secret();
assert_eq!(secret.len(), 64); // 32 bytes = 64 hex chars
}
#[test]
fn test_generate_secret_with_length() {
use super::generate_secret_with_length;
let s = generate_secret_with_length(16);
assert_eq!(s.len(), 32); // 16 bytes = 32 hex chars
assert!(s.chars().all(|c| c.is_ascii_hexdigit()));
let s2 = generate_secret_with_length(1);
assert_eq!(s2.len(), 2);
}
}
+3 -2
View File
@@ -3,7 +3,7 @@
//! Provides a guided setup experience for:
//! 1. Database connection
//! 2. Security (secrets master key)
//! 3. NEAR AI authentication
//! 3. Inference provider selection
//! 4. Model selection
//! 5. Embeddings
//! 6. Channel configuration (HTTP, Telegram, etc.)
@@ -24,7 +24,8 @@ mod prompts;
mod wizard;
pub use channels::{
SecretsContext, setup_http, setup_telegram, setup_tunnel, validate_telegram_token,
ChannelSetupError, SecretsContext, setup_http, setup_telegram, setup_tunnel,
validate_telegram_token,
};
pub use prompts::{
confirm, input, optional_input, print_error, print_header, print_info, print_step,
+10 -4
View File
@@ -21,6 +21,7 @@ use secrecy::SecretString;
/// Display a numbered menu and get user selection.
///
/// Returns the index (0-based) of the selected option.
/// Pressing Enter without input selects the first option (index 0).
///
/// # Example
///
@@ -54,10 +55,11 @@ pub fn select_one(prompt: &str, options: &[&str]) -> io::Result<usize> {
}
// Parse number
if let Ok(num) = input.parse::<usize>() {
if num >= 1 && num <= options.len() {
return Ok(num - 1);
}
if let Ok(num) = input.parse::<usize>()
&& num >= 1
&& num <= options.len()
{
return Ok(num - 1);
}
writeln!(
@@ -83,6 +85,10 @@ pub fn select_one(prompt: &str, options: &[&str]) -> io::Result<usize> {
/// ])?;
/// ```
pub fn select_many(prompt: &str, options: &[(&str, bool)]) -> io::Result<Vec<usize>> {
if options.is_empty() {
return Ok(vec![]);
}
let mut stdout = io::stdout();
let mut selected: Vec<bool> = options.iter().map(|(_, s)| *s).collect();
let mut cursor_pos = 0;
+941 -126
View File
File diff suppressed because it is too large Load Diff
+27 -27
View File
@@ -326,20 +326,20 @@ impl TestHarness {
}
// Verify expected output
if let Some(ref expected) = test.expected_output {
if &actual != expected {
return TestResult {
name: test.name.clone(),
passed: false,
duration,
error: Some(format!(
"Output mismatch:\nExpected: {}\nActual: {}",
serde_json::to_string_pretty(expected).unwrap_or_default(),
serde_json::to_string_pretty(&actual).unwrap_or_default()
)),
actual_output: Some(actual),
};
}
if let Some(ref expected) = test.expected_output
&& &actual != expected
{
return TestResult {
name: test.name.clone(),
passed: false,
duration,
error: Some(format!(
"Output mismatch:\nExpected: {}\nActual: {}",
serde_json::to_string_pretty(expected).unwrap_or_default(),
serde_json::to_string_pretty(&actual).unwrap_or_default()
)),
actual_output: Some(actual),
};
}
// Verify expected fields
@@ -357,19 +357,19 @@ impl TestHarness {
};
}
if let Some(ref expected_value) = field.value {
if field_value != Some(expected_value) {
return TestResult {
name: test.name.clone(),
passed: false,
duration,
error: Some(format!(
"Field '{}' mismatch: expected {:?}, got {:?}",
field.path, expected_value, field_value
)),
actual_output: Some(actual),
};
}
if let Some(ref expected_value) = field.value
&& field_value != Some(expected_value)
{
return TestResult {
name: test.name.clone(),
passed: false,
duration,
error: Some(format!(
"Field '{}' mismatch: expected {:?}, got {:?}",
field.path, expected_value, field_value
)),
actual_output: Some(actual),
};
}
}
}
+2 -7
View File
@@ -3,7 +3,7 @@
use async_trait::async_trait;
use crate::context::JobContext;
use crate::tools::tool::{Tool, ToolError, ToolOutput};
use crate::tools::tool::{Tool, ToolError, ToolOutput, require_str};
/// Simple echo tool for testing.
pub struct EchoTool;
@@ -38,12 +38,7 @@ impl Tool for EchoTool {
) -> Result<ToolOutput, ToolError> {
let start = std::time::Instant::now();
let message = params
.get("message")
.and_then(|v| v.as_str())
.ok_or_else(|| {
ToolError::InvalidParameters("missing 'message' parameter".to_string())
})?;
let message = require_str(&params, "message")?;
Ok(ToolOutput::text(message, start.elapsed()))
}
-136
View File
@@ -1,136 +0,0 @@
//! E-commerce tool for shopping and price comparison.
use async_trait::async_trait;
use crate::context::JobContext;
use crate::tools::tool::{Tool, ToolError, ToolOutput};
/// Tool for e-commerce operations (Amazon, price comparison, etc.).
pub struct EcommerceTool {
// TODO: Add API clients
}
impl EcommerceTool {
/// Create a new e-commerce tool.
pub fn new() -> Self {
Self {}
}
}
impl Default for EcommerceTool {
fn default() -> Self {
Self::new()
}
}
#[async_trait]
impl Tool for EcommerceTool {
fn name(&self) -> &str {
"ecommerce"
}
fn description(&self) -> &str {
"Search products, compare prices, and find deals across e-commerce platforms."
}
fn parameters_schema(&self) -> serde_json::Value {
serde_json::json!({
"type": "object",
"properties": {
"action": {
"type": "string",
"enum": ["search", "get_product", "compare_prices", "track_price"],
"description": "The e-commerce action to perform"
},
"query": {
"type": "string",
"description": "Search query (for search action)"
},
"product_id": {
"type": "string",
"description": "Product ID or ASIN (for get_product, compare_prices)"
},
"platform": {
"type": "string",
"enum": ["amazon", "ebay", "walmart", "all"],
"description": "E-commerce platform to search"
},
"max_price": {
"type": "number",
"description": "Maximum price filter"
},
"category": {
"type": "string",
"description": "Product category filter"
}
},
"required": ["action"]
})
}
async fn execute(
&self,
params: serde_json::Value,
_ctx: &JobContext,
) -> Result<ToolOutput, ToolError> {
let start = std::time::Instant::now();
let action = params
.get("action")
.and_then(|v| v.as_str())
.ok_or_else(|| {
ToolError::InvalidParameters("missing 'action' parameter".to_string())
})?;
// TODO: Implement actual e-commerce API integrations
let result = match action {
"search" => {
let query = params.get("query").and_then(|v| v.as_str()).unwrap_or("");
serde_json::json!({
"query": query,
"results": [],
"message": "E-commerce integration not yet implemented"
})
}
"get_product" => {
let product_id = params
.get("product_id")
.and_then(|v| v.as_str())
.ok_or_else(|| {
ToolError::InvalidParameters("missing 'product_id' parameter".to_string())
})?;
serde_json::json!({
"product_id": product_id,
"found": false,
"message": "E-commerce integration not yet implemented"
})
}
"compare_prices" => {
serde_json::json!({
"prices": [],
"message": "E-commerce integration not yet implemented"
})
}
"track_price" => {
serde_json::json!({
"tracking": false,
"message": "E-commerce integration not yet implemented"
})
}
_ => {
return Err(ToolError::InvalidParameters(format!(
"unknown action: {}",
action
)));
}
};
Ok(ToolOutput::success(result, start.elapsed()))
}
fn requires_sanitization(&self) -> bool {
true // External e-commerce data
}
}
+5 -17
View File
@@ -9,7 +9,7 @@ use async_trait::async_trait;
use crate::context::JobContext;
use crate::extensions::{ExtensionKind, ExtensionManager};
use crate::tools::tool::{Tool, ToolError, ToolOutput};
use crate::tools::tool::{Tool, ToolError, ToolOutput, require_str};
// ── tool_search ──────────────────────────────────────────────────────────
@@ -133,10 +133,7 @@ impl Tool for ToolInstallTool {
) -> Result<ToolOutput, ToolError> {
let start = std::time::Instant::now();
let name = params
.get("name")
.and_then(|v| v.as_str())
.ok_or_else(|| ToolError::InvalidParameters("name is required".to_string()))?;
let name = require_str(&params, "name")?;
let url = params.get("url").and_then(|v| v.as_str());
@@ -210,10 +207,7 @@ impl Tool for ToolAuthTool {
) -> Result<ToolOutput, ToolError> {
let start = std::time::Instant::now();
let name = params
.get("name")
.and_then(|v| v.as_str())
.ok_or_else(|| ToolError::InvalidParameters("name is required".to_string()))?;
let name = require_str(&params, "name")?;
let result = self
.manager
@@ -306,10 +300,7 @@ impl Tool for ToolActivateTool {
) -> Result<ToolOutput, ToolError> {
let start = std::time::Instant::now();
let name = params
.get("name")
.and_then(|v| v.as_str())
.ok_or_else(|| ToolError::InvalidParameters("name is required".to_string()))?;
let name = require_str(&params, "name")?;
match self.manager.activate(name).await {
Ok(result) => {
@@ -471,10 +462,7 @@ impl Tool for ToolRemoveTool {
) -> Result<ToolOutput, ToolError> {
let start = std::time::Instant::now();
let name = params
.get("name")
.and_then(|v| v.as_str())
.ok_or_else(|| ToolError::InvalidParameters("name is required".to_string()))?;
let name = require_str(&params, "name")?;
let message = self
.manager
+7 -25
View File
@@ -11,7 +11,7 @@ use async_trait::async_trait;
use tokio::fs;
use crate::context::JobContext;
use crate::tools::tool::{Tool, ToolDomain, ToolError, ToolOutput};
use crate::tools::tool::{Tool, ToolDomain, ToolError, ToolOutput, require_str};
use crate::workspace::paths as ws_paths;
/// Well-known workspace filenames that must go through memory_write, not write_file.
@@ -203,10 +203,7 @@ impl Tool for ReadFileTool {
params: serde_json::Value,
_ctx: &JobContext,
) -> Result<ToolOutput, ToolError> {
let path_str = params
.get("path")
.and_then(|v| v.as_str())
.ok_or_else(|| ToolError::InvalidParameters("missing 'path' parameter".into()))?;
let path_str = require_str(&params, "path")?;
let offset = params.get("offset").and_then(|v| v.as_u64()).unwrap_or(0) as usize;
let limit = params.get("limit").and_then(|v| v.as_u64());
@@ -328,10 +325,7 @@ impl Tool for WriteFileTool {
params: serde_json::Value,
_ctx: &JobContext,
) -> Result<ToolOutput, ToolError> {
let path_str = params
.get("path")
.and_then(|v| v.as_str())
.ok_or_else(|| ToolError::InvalidParameters("missing 'path' parameter".into()))?;
let path_str = require_str(&params, "path")?;
// Reject workspace paths: these live in the database, not on disk.
if is_workspace_path(path_str) {
@@ -342,10 +336,7 @@ impl Tool for WriteFileTool {
)));
}
let content = params
.get("content")
.and_then(|v| v.as_str())
.ok_or_else(|| ToolError::InvalidParameters("missing 'content' parameter".into()))?;
let content = require_str(&params, "content")?;
let start = std::time::Instant::now();
@@ -650,20 +641,11 @@ impl Tool for ApplyPatchTool {
params: serde_json::Value,
_ctx: &JobContext,
) -> Result<ToolOutput, ToolError> {
let path_str = params
.get("path")
.and_then(|v| v.as_str())
.ok_or_else(|| ToolError::InvalidParameters("missing 'path' parameter".into()))?;
let path_str = require_str(&params, "path")?;
let old_string = params
.get("old_string")
.and_then(|v| v.as_str())
.ok_or_else(|| ToolError::InvalidParameters("missing 'old_string' parameter".into()))?;
let old_string = require_str(&params, "old_string")?;
let new_string = params
.get("new_string")
.and_then(|v| v.as_str())
.ok_or_else(|| ToolError::InvalidParameters("missing 'new_string' parameter".into()))?;
let new_string = require_str(&params, "new_string")?;
let replace_all = params
.get("replace_all")
+9 -17
View File
@@ -9,7 +9,7 @@ use reqwest::Client;
use crate::context::JobContext;
use crate::safety::LeakDetector;
use crate::tools::tool::{Tool, ToolError, ToolOutput};
use crate::tools::tool::{Tool, ToolError, ToolOutput, require_str};
/// Maximum response body size (5 MB). Prevents OOM from unbounded responses.
const MAX_RESPONSE_SIZE: usize = 5 * 1024 * 1024;
@@ -54,12 +54,12 @@ fn validate_url(url: &str) -> Result<reqwest::Url, ToolError> {
}
// Check literal IP addresses
if let Ok(ip) = host.parse::<IpAddr>() {
if is_disallowed_ip(&ip) {
return Err(ToolError::NotAuthorized(
"private or local IPs are not allowed".to_string(),
));
}
if let Ok(ip) = host.parse::<IpAddr>()
&& is_disallowed_ip(&ip)
{
return Err(ToolError::NotAuthorized(
"private or local IPs are not allowed".to_string(),
));
}
// Resolve hostname and check all resolved IPs against the blocklist.
@@ -154,17 +154,9 @@ impl Tool for HttpTool {
) -> Result<ToolOutput, ToolError> {
let start = std::time::Instant::now();
let method = params
.get("method")
.and_then(|v| v.as_str())
.ok_or_else(|| {
ToolError::InvalidParameters("missing 'method' parameter".to_string())
})?;
let method = require_str(&params, "method")?;
let url = params
.get("url")
.and_then(|v| v.as_str())
.ok_or_else(|| ToolError::InvalidParameters("missing 'url' parameter".to_string()))?;
let url = require_str(&params, "url")?;
let parsed_url = validate_url(url)?;
// Parse headers
+17 -31
View File
@@ -18,7 +18,7 @@ use crate::context::{ContextManager, JobContext, JobState};
use crate::db::Database;
use crate::history::SandboxJobRecord;
use crate::orchestrator::job_manager::{ContainerJobManager, JobMode};
use crate::tools::tool::{Tool, ToolError, ToolOutput};
use crate::tools::tool::{Tool, ToolError, ToolOutput, require_str};
/// Tool for creating a new job.
///
@@ -158,18 +158,18 @@ impl CreateJobTool {
});
// Persist the job mode to DB
if mode == JobMode::ClaudeCode {
if let Some(store) = self.store.clone() {
let job_id_copy = job_id;
tokio::spawn(async move {
if let Err(e) = store
.update_sandbox_job_mode(job_id_copy, "claude_code")
.await
{
tracing::warn!(job_id = %job_id_copy, "Failed to set job mode: {}", e);
}
});
}
if mode == JobMode::ClaudeCode
&& let Some(store) = self.store.clone()
{
let job_id_copy = job_id;
tokio::spawn(async move {
if let Err(e) = store
.update_sandbox_job_mode(job_id_copy, "claude_code")
.await
{
tracing::warn!(job_id = %job_id_copy, "Failed to set job mode: {}", e);
}
});
}
// Create the container job with the pre-determined job_id.
@@ -467,17 +467,9 @@ impl Tool for CreateJobTool {
params: serde_json::Value,
ctx: &JobContext,
) -> Result<ToolOutput, ToolError> {
let title = params
.get("title")
.and_then(|v| v.as_str())
.ok_or_else(|| ToolError::InvalidParameters("missing 'title' parameter".into()))?;
let title = require_str(&params, "title")?;
let description = params
.get("description")
.and_then(|v| v.as_str())
.ok_or_else(|| {
ToolError::InvalidParameters("missing 'description' parameter".into())
})?;
let description = require_str(&params, "description")?;
if self.sandbox_enabled() {
let wait = params.get("wait").and_then(|v| v.as_bool()).unwrap_or(true);
@@ -635,10 +627,7 @@ impl Tool for JobStatusTool {
let start = std::time::Instant::now();
let requester_id = ctx.user_id.clone();
let job_id_str = params
.get("job_id")
.and_then(|v| v.as_str())
.ok_or_else(|| ToolError::InvalidParameters("missing 'job_id' parameter".into()))?;
let job_id_str = require_str(&params, "job_id")?;
let job_id = Uuid::parse_str(job_id_str).map_err(|_| {
ToolError::InvalidParameters(format!("invalid job ID format: {}", job_id_str))
@@ -720,10 +709,7 @@ impl Tool for CancelJobTool {
let start = std::time::Instant::now();
let requester_id = ctx.user_id.clone();
let job_id_str = params
.get("job_id")
.and_then(|v| v.as_str())
.ok_or_else(|| ToolError::InvalidParameters("missing 'job_id' parameter".into()))?;
let job_id_str = require_str(&params, "job_id")?;
let job_id = Uuid::parse_str(job_id_str).map_err(|_| {
ToolError::InvalidParameters(format!("invalid job ID format: {}", job_id_str))
+3 -10
View File
@@ -3,7 +3,7 @@
use async_trait::async_trait;
use crate::context::JobContext;
use crate::tools::tool::{Tool, ToolError, ToolOutput};
use crate::tools::tool::{Tool, ToolError, ToolOutput, require_param, require_str};
/// Tool for JSON manipulation (parse, query, transform).
pub struct JsonTool;
@@ -46,16 +46,9 @@ impl Tool for JsonTool {
) -> Result<ToolOutput, ToolError> {
let start = std::time::Instant::now();
let operation = params
.get("operation")
.and_then(|v| v.as_str())
.ok_or_else(|| {
ToolError::InvalidParameters("missing 'operation' parameter".to_string())
})?;
let operation = require_str(&params, "operation")?;
let data = params
.get("data")
.ok_or_else(|| ToolError::InvalidParameters("missing 'data' parameter".to_string()))?;
let data = require_param(&params, "data")?;
let result = match operation {
"parse" => {
-160
View File
@@ -1,160 +0,0 @@
//! NEAR AI Marketplace tool.
use async_trait::async_trait;
use rust_decimal::Decimal;
use crate::context::JobContext;
use crate::tools::tool::{Tool, ToolError, ToolOutput};
/// Tool for interacting with the NEAR AI marketplace.
pub struct MarketplaceTool {
// TODO: Add marketplace client
}
impl MarketplaceTool {
/// Create a new marketplace tool.
pub fn new() -> Self {
Self {}
}
}
impl Default for MarketplaceTool {
fn default() -> Self {
Self::new()
}
}
#[async_trait]
impl Tool for MarketplaceTool {
fn name(&self) -> &str {
"marketplace"
}
fn description(&self) -> &str {
"Interact with the NEAR AI marketplace: search jobs, submit bids, deliver work."
}
fn parameters_schema(&self) -> serde_json::Value {
serde_json::json!({
"type": "object",
"properties": {
"action": {
"type": "string",
"enum": ["search_jobs", "get_job", "submit_bid", "accept_job", "submit_work", "get_status"],
"description": "The marketplace action to perform"
},
"job_id": {
"type": "string",
"description": "Job ID (for get_job, submit_bid, accept_job, submit_work)"
},
"query": {
"type": "string",
"description": "Search query (for search_jobs)"
},
"category": {
"type": "string",
"description": "Job category filter (for search_jobs)"
},
"bid_amount": {
"type": "number",
"description": "Bid amount in NEAR (for submit_bid)"
},
"work_url": {
"type": "string",
"description": "URL to submitted work (for submit_work)"
},
"work_description": {
"type": "string",
"description": "Description of completed work (for submit_work)"
}
},
"required": ["action"]
})
}
async fn execute(
&self,
params: serde_json::Value,
_ctx: &JobContext,
) -> Result<ToolOutput, ToolError> {
let start = std::time::Instant::now();
let action = params
.get("action")
.and_then(|v| v.as_str())
.ok_or_else(|| {
ToolError::InvalidParameters("missing 'action' parameter".to_string())
})?;
// TODO: Implement actual marketplace integration
let result = match action {
"search_jobs" => {
// Placeholder response
serde_json::json!({
"jobs": [],
"total": 0,
"message": "Marketplace integration not yet implemented"
})
}
"get_job" => {
let job_id = params
.get("job_id")
.and_then(|v| v.as_str())
.ok_or_else(|| {
ToolError::InvalidParameters("missing 'job_id' parameter".to_string())
})?;
serde_json::json!({
"job_id": job_id,
"status": "not_found",
"message": "Marketplace integration not yet implemented"
})
}
"submit_bid" => {
serde_json::json!({
"success": false,
"message": "Marketplace integration not yet implemented"
})
}
"accept_job" => {
serde_json::json!({
"success": false,
"message": "Marketplace integration not yet implemented"
})
}
"submit_work" => {
serde_json::json!({
"success": false,
"message": "Marketplace integration not yet implemented"
})
}
"get_status" => {
serde_json::json!({
"connected": false,
"message": "Marketplace integration not yet implemented"
})
}
_ => {
return Err(ToolError::InvalidParameters(format!(
"unknown action: {}",
action
)));
}
};
Ok(ToolOutput::success(result, start.elapsed()))
}
fn estimated_cost(&self, params: &serde_json::Value) -> Option<Decimal> {
// Bidding has a cost
if params.get("action").and_then(|v| v.as_str()) == Some("submit_bid") {
Some(Decimal::new(1, 2)) // 0.01 NEAR gas cost
} else {
None
}
}
fn requires_sanitization(&self) -> bool {
true // External marketplace data
}
}
+4 -15
View File
@@ -17,7 +17,7 @@ use std::sync::Arc;
use async_trait::async_trait;
use crate::context::JobContext;
use crate::tools::tool::{Tool, ToolError, ToolOutput};
use crate::tools::tool::{Tool, ToolError, ToolOutput, require_str};
use crate::workspace::{Workspace, paths};
/// Identity files that the LLM must not overwrite via tool calls.
@@ -81,10 +81,7 @@ impl Tool for MemorySearchTool {
) -> Result<ToolOutput, ToolError> {
let start = std::time::Instant::now();
let query = params
.get("query")
.and_then(|v| v.as_str())
.ok_or_else(|| ToolError::InvalidParameters("missing 'query' parameter".to_string()))?;
let query = require_str(&params, "query")?;
let limit = params
.get("limit")
@@ -176,12 +173,7 @@ impl Tool for MemoryWriteTool {
) -> Result<ToolOutput, ToolError> {
let start = std::time::Instant::now();
let content = params
.get("content")
.and_then(|v| v.as_str())
.ok_or_else(|| {
ToolError::InvalidParameters("missing 'content' parameter".to_string())
})?;
let content = require_str(&params, "content")?;
if content.trim().is_empty() {
return Err(ToolError::InvalidParameters(
@@ -337,10 +329,7 @@ impl Tool for MemoryReadTool {
) -> Result<ToolOutput, ToolError> {
let start = std::time::Instant::now();
let path = params
.get("path")
.and_then(|v| v.as_str())
.ok_or_else(|| ToolError::InvalidParameters("missing 'path' parameter".to_string()))?;
let path = require_str(&params, "path")?;
let doc = self
.workspace
-8
View File
@@ -1,22 +1,17 @@
//! Built-in tools that come with the agent.
mod echo;
mod ecommerce;
pub mod extension_tools;
mod file;
mod http;
mod job;
mod json;
mod marketplace;
mod memory;
mod restaurant;
pub mod routine;
pub(crate) mod shell;
mod taskrabbit;
mod time;
pub use echo::EchoTool;
pub use ecommerce::EcommerceTool;
pub use extension_tools::{
ToolActivateTool, ToolAuthTool, ToolInstallTool, ToolListTool, ToolRemoveTool, ToolSearchTool,
};
@@ -24,12 +19,9 @@ pub use file::{ApplyPatchTool, ListDirTool, ReadFileTool, WriteFileTool};
pub use http::HttpTool;
pub use job::{CancelJobTool, CreateJobTool, JobStatusTool, ListJobsTool};
pub use json::JsonTool;
pub use marketplace::MarketplaceTool;
pub use memory::{MemoryReadTool, MemorySearchTool, MemoryTreeTool, MemoryWriteTool};
pub use restaurant::RestaurantTool;
pub use routine::{
RoutineCreateTool, RoutineDeleteTool, RoutineHistoryTool, RoutineListTool, RoutineUpdateTool,
};
pub use shell::ShellTool;
pub use taskrabbit::TaskRabbitTool;
pub use time::TimeTool;
-172
View File
@@ -1,172 +0,0 @@
//! Restaurant reservation tool.
use async_trait::async_trait;
use crate::context::JobContext;
use crate::tools::tool::{Tool, ToolError, ToolOutput};
/// Tool for restaurant reservations (OpenTable, Resy, etc.).
pub struct RestaurantTool {
// TODO: Add reservation API clients
}
impl RestaurantTool {
/// Create a new restaurant tool.
pub fn new() -> Self {
Self {}
}
}
impl Default for RestaurantTool {
fn default() -> Self {
Self::new()
}
}
#[async_trait]
impl Tool for RestaurantTool {
fn name(&self) -> &str {
"restaurant"
}
fn description(&self) -> &str {
"Search restaurants, check availability, and make reservations via OpenTable, Resy, etc."
}
fn parameters_schema(&self) -> serde_json::Value {
serde_json::json!({
"type": "object",
"properties": {
"action": {
"type": "string",
"enum": ["search", "check_availability", "make_reservation", "cancel_reservation", "get_reservation"],
"description": "The restaurant action to perform"
},
"query": {
"type": "string",
"description": "Search query (cuisine type, restaurant name, etc.)"
},
"location": {
"type": "object",
"properties": {
"city": { "type": "string" },
"neighborhood": { "type": "string" },
"latitude": { "type": "number" },
"longitude": { "type": "number" }
},
"description": "Location to search near"
},
"date": {
"type": "string",
"description": "Reservation date (YYYY-MM-DD)"
},
"time": {
"type": "string",
"description": "Preferred time (HH:MM)"
},
"party_size": {
"type": "integer",
"description": "Number of guests"
},
"restaurant_id": {
"type": "string",
"description": "Restaurant ID (for check_availability, make_reservation)"
},
"reservation_id": {
"type": "string",
"description": "Reservation ID (for cancel_reservation, get_reservation)"
},
"guest_name": {
"type": "string",
"description": "Name for the reservation"
},
"guest_phone": {
"type": "string",
"description": "Phone number for the reservation"
},
"guest_email": {
"type": "string",
"description": "Email for the reservation"
}
},
"required": ["action"]
})
}
async fn execute(
&self,
params: serde_json::Value,
_ctx: &JobContext,
) -> Result<ToolOutput, ToolError> {
let start = std::time::Instant::now();
let action = params
.get("action")
.and_then(|v| v.as_str())
.ok_or_else(|| {
ToolError::InvalidParameters("missing 'action' parameter".to_string())
})?;
// TODO: Implement actual restaurant reservation API integrations
let result = match action {
"search" => {
let query = params.get("query").and_then(|v| v.as_str()).unwrap_or("");
serde_json::json!({
"query": query,
"restaurants": [],
"message": "Restaurant integration not yet implemented"
})
}
"check_availability" => {
let restaurant_id = params
.get("restaurant_id")
.and_then(|v| v.as_str())
.ok_or_else(|| {
ToolError::InvalidParameters(
"missing 'restaurant_id' parameter".to_string(),
)
})?;
serde_json::json!({
"restaurant_id": restaurant_id,
"available_times": [],
"message": "Restaurant integration not yet implemented"
})
}
"make_reservation" => {
serde_json::json!({
"success": false,
"message": "Restaurant integration not yet implemented"
})
}
"cancel_reservation" => {
serde_json::json!({
"cancelled": false,
"message": "Restaurant integration not yet implemented"
})
}
"get_reservation" => {
let reservation_id = params.get("reservation_id").and_then(|v| v.as_str());
serde_json::json!({
"reservation_id": reservation_id,
"found": false,
"message": "Restaurant integration not yet implemented"
})
}
_ => {
return Err(ToolError::InvalidParameters(format!(
"unknown action: {}",
action
)));
}
};
Ok(ToolOutput::success(result, start.elapsed()))
}
fn requires_sanitization(&self) -> bool {
true // External restaurant data
}
}
+7 -25
View File
@@ -20,7 +20,7 @@ use crate::agent::routine::{
use crate::agent::routine_engine::RoutineEngine;
use crate::context::JobContext;
use crate::db::Database;
use crate::tools::tool::{Tool, ToolError, ToolOutput};
use crate::tools::tool::{Tool, ToolError, ToolOutput, require_str};
// ==================== routine_create ====================
@@ -106,25 +106,16 @@ impl Tool for RoutineCreateTool {
) -> Result<ToolOutput, ToolError> {
let start = std::time::Instant::now();
let name = params
.get("name")
.and_then(|v| v.as_str())
.ok_or_else(|| ToolError::InvalidParameters("missing 'name'".to_string()))?;
let name = require_str(&params, "name")?;
let description = params
.get("description")
.and_then(|v| v.as_str())
.unwrap_or("");
let trigger_type = params
.get("trigger_type")
.and_then(|v| v.as_str())
.ok_or_else(|| ToolError::InvalidParameters("missing 'trigger_type'".to_string()))?;
let trigger_type = require_str(&params, "trigger_type")?;
let prompt = params
.get("prompt")
.and_then(|v| v.as_str())
.ok_or_else(|| ToolError::InvalidParameters("missing 'prompt'".to_string()))?;
let prompt = require_str(&params, "prompt")?;
// Build trigger
let trigger = match trigger_type {
@@ -408,10 +399,7 @@ impl Tool for RoutineUpdateTool {
) -> Result<ToolOutput, ToolError> {
let start = std::time::Instant::now();
let name = params
.get("name")
.and_then(|v| v.as_str())
.ok_or_else(|| ToolError::InvalidParameters("missing 'name'".to_string()))?;
let name = require_str(&params, "name")?;
let mut routine = self
.store
@@ -514,10 +502,7 @@ impl Tool for RoutineDeleteTool {
) -> Result<ToolOutput, ToolError> {
let start = std::time::Instant::now();
let name = params
.get("name")
.and_then(|v| v.as_str())
.ok_or_else(|| ToolError::InvalidParameters("missing 'name'".to_string()))?;
let name = require_str(&params, "name")?;
let routine = self
.store
@@ -595,10 +580,7 @@ impl Tool for RoutineHistoryTool {
) -> Result<ToolOutput, ToolError> {
let start = std::time::Instant::now();
let name = params
.get("name")
.and_then(|v| v.as_str())
.ok_or_else(|| ToolError::InvalidParameters("missing 'name'".to_string()))?;
let name = require_str(&params, "name")?;
let limit = params
.get("limit")
+50 -11
View File
@@ -30,7 +30,7 @@ use tokio::process::Command;
use crate::context::JobContext;
use crate::sandbox::{SandboxManager, SandboxPolicy};
use crate::tools::tool::{Tool, ToolDomain, ToolError, ToolOutput};
use crate::tools::tool::{Tool, ToolDomain, ToolError, ToolOutput, require_str};
/// Maximum output size before truncation (64KB).
const MAX_OUTPUT_SIZE: usize = 64 * 1024;
@@ -343,12 +343,12 @@ impl ShellTool {
// Use sandbox if configured; fail-closed (never silently fall through
// to unsandboxed execution when sandbox was intended).
if let Some(ref sandbox) = self.sandbox {
if sandbox.is_initialized() || sandbox.config().enabled {
return self
.execute_sandboxed(sandbox, cmd, &cwd, timeout_duration)
.await;
}
if let Some(ref sandbox) = self.sandbox
&& (sandbox.is_initialized() || sandbox.config().enabled)
{
return self
.execute_sandboxed(sandbox, cmd, &cwd, timeout_duration)
.await;
}
// Only execute directly when no sandbox was configured at all.
@@ -401,10 +401,7 @@ impl Tool for ShellTool {
params: serde_json::Value,
_ctx: &JobContext,
) -> Result<ToolOutput, ToolError> {
let command = params
.get("command")
.and_then(|v| v.as_str())
.ok_or_else(|| ToolError::InvalidParameters("missing 'command' parameter".into()))?;
let command = require_str(&params, "command")?;
let workdir = params.get("workdir").and_then(|v| v.as_str());
let timeout = params.get("timeout").and_then(|v| v.as_u64());
@@ -527,6 +524,48 @@ mod tests {
));
}
/// Replicate the extraction logic from agent_loop.rs to prove it works
/// when `arguments` is a `serde_json::Value::Object` (the common case
/// that was previously broken because `Value::Object.as_str()` returns None).
#[test]
fn test_destructive_command_extraction_from_object_args() {
let arguments = serde_json::json!({"command": "rm -rf /tmp/stuff"});
let cmd = arguments
.get("command")
.and_then(|c| c.as_str().map(String::from))
.or_else(|| {
arguments
.as_str()
.and_then(|s| serde_json::from_str::<serde_json::Value>(s).ok())
.and_then(|v| v.get("command").and_then(|c| c.as_str().map(String::from)))
});
assert_eq!(cmd.as_deref(), Some("rm -rf /tmp/stuff"));
assert!(requires_explicit_approval(cmd.as_deref().unwrap()));
}
/// Verify extraction still works when `arguments` is a JSON string
/// (rare, but possible if the LLM provider returns string-encoded JSON).
#[test]
fn test_destructive_command_extraction_from_string_args() {
let arguments =
serde_json::Value::String(r#"{"command": "git push --force origin main"}"#.to_string());
let cmd = arguments
.get("command")
.and_then(|c| c.as_str().map(String::from))
.or_else(|| {
arguments
.as_str()
.and_then(|s| serde_json::from_str::<serde_json::Value>(s).ok())
.and_then(|v| v.get("command").and_then(|c| c.as_str().map(String::from)))
});
assert_eq!(cmd.as_deref(), Some("git push --force origin main"));
assert!(requires_explicit_approval(cmd.as_deref().unwrap()));
}
#[test]
fn test_sandbox_policy_builder() {
let tool = ShellTool::new()
-157
View File
@@ -1,157 +0,0 @@
//! TaskRabbit tool for real-world task delegation.
use async_trait::async_trait;
use rust_decimal::Decimal;
use crate::context::JobContext;
use crate::tools::tool::{Tool, ToolError, ToolOutput};
/// Tool for delegating real-world tasks via TaskRabbit.
pub struct TaskRabbitTool {
// TODO: Add TaskRabbit API client
}
impl TaskRabbitTool {
/// Create a new TaskRabbit tool.
pub fn new() -> Self {
Self {}
}
}
impl Default for TaskRabbitTool {
fn default() -> Self {
Self::new()
}
}
#[async_trait]
impl Tool for TaskRabbitTool {
fn name(&self) -> &str {
"taskrabbit"
}
fn description(&self) -> &str {
"Delegate real-world tasks to TaskRabbit taskers (delivery, assembly, cleaning, etc.)."
}
fn parameters_schema(&self) -> serde_json::Value {
serde_json::json!({
"type": "object",
"properties": {
"action": {
"type": "string",
"enum": ["search_taskers", "get_quote", "book_task", "get_status", "cancel_task"],
"description": "The TaskRabbit action to perform"
},
"task_type": {
"type": "string",
"enum": ["delivery", "assembly", "moving", "cleaning", "handyman", "other"],
"description": "Type of task"
},
"description": {
"type": "string",
"description": "Detailed description of the task"
},
"location": {
"type": "object",
"properties": {
"address": { "type": "string" },
"city": { "type": "string" },
"state": { "type": "string" },
"zip": { "type": "string" }
},
"description": "Location for the task"
},
"scheduled_time": {
"type": "string",
"description": "ISO 8601 datetime for when the task should be performed"
},
"budget": {
"type": "number",
"description": "Maximum budget for the task in USD"
},
"task_id": {
"type": "string",
"description": "Task ID (for get_status, cancel_task)"
}
},
"required": ["action"]
})
}
async fn execute(
&self,
params: serde_json::Value,
_ctx: &JobContext,
) -> Result<ToolOutput, ToolError> {
let start = std::time::Instant::now();
let action = params
.get("action")
.and_then(|v| v.as_str())
.ok_or_else(|| {
ToolError::InvalidParameters("missing 'action' parameter".to_string())
})?;
// TODO: Implement actual TaskRabbit API integration
let result = match action {
"search_taskers" => {
serde_json::json!({
"taskers": [],
"message": "TaskRabbit integration not yet implemented"
})
}
"get_quote" => {
serde_json::json!({
"quotes": [],
"message": "TaskRabbit integration not yet implemented"
})
}
"book_task" => {
serde_json::json!({
"booked": false,
"message": "TaskRabbit integration not yet implemented"
})
}
"get_status" => {
let task_id = params.get("task_id").and_then(|v| v.as_str());
serde_json::json!({
"task_id": task_id,
"status": "unknown",
"message": "TaskRabbit integration not yet implemented"
})
}
"cancel_task" => {
serde_json::json!({
"cancelled": false,
"message": "TaskRabbit integration not yet implemented"
})
}
_ => {
return Err(ToolError::InvalidParameters(format!(
"unknown action: {}",
action
)));
}
};
Ok(ToolOutput::success(result, start.elapsed()))
}
fn estimated_cost(&self, params: &serde_json::Value) -> Option<Decimal> {
// Booking a task has associated costs
if params.get("action").and_then(|v| v.as_str()) == Some("book_task") {
params
.get("budget")
.and_then(|v| v.as_f64())
.map(|b| Decimal::try_from(b).unwrap_or_default())
} else {
None
}
}
fn requires_sanitization(&self) -> bool {
true // External TaskRabbit data
}
}
+5 -25
View File
@@ -4,7 +4,7 @@ use async_trait::async_trait;
use chrono::{DateTime, Utc};
use crate::context::JobContext;
use crate::tools::tool::{Tool, ToolError, ToolOutput};
use crate::tools::tool::{Tool, ToolError, ToolOutput, require_str};
/// Tool for getting current time and date operations.
pub struct TimeTool;
@@ -52,12 +52,7 @@ impl Tool for TimeTool {
) -> Result<ToolOutput, ToolError> {
let start = std::time::Instant::now();
let operation = params
.get("operation")
.and_then(|v| v.as_str())
.ok_or_else(|| {
ToolError::InvalidParameters("missing 'operation' parameter".to_string())
})?;
let operation = require_str(&params, "operation")?;
let result = match operation {
"now" => {
@@ -69,12 +64,7 @@ impl Tool for TimeTool {
})
}
"parse" => {
let timestamp = params
.get("timestamp")
.and_then(|v| v.as_str())
.ok_or_else(|| {
ToolError::InvalidParameters("missing 'timestamp' parameter".to_string())
})?;
let timestamp = require_str(&params, "timestamp")?;
let dt: DateTime<Utc> = timestamp.parse().map_err(|e| {
ToolError::InvalidParameters(format!("invalid timestamp: {}", e))
@@ -87,19 +77,9 @@ impl Tool for TimeTool {
})
}
"diff" => {
let ts1 = params
.get("timestamp")
.and_then(|v| v.as_str())
.ok_or_else(|| {
ToolError::InvalidParameters("missing 'timestamp' parameter".to_string())
})?;
let ts1 = require_str(&params, "timestamp")?;
let ts2 = params
.get("timestamp2")
.and_then(|v| v.as_str())
.ok_or_else(|| {
ToolError::InvalidParameters("missing 'timestamp2' parameter".to_string())
})?;
let ts2 = require_str(&params, "timestamp2")?;
let dt1: DateTime<Utc> = ts1.parse().map_err(|e| {
ToolError::InvalidParameters(format!("invalid timestamp: {}", e))
+15 -72
View File
@@ -11,9 +11,9 @@ use base64::{Engine, engine::general_purpose::URL_SAFE_NO_PAD};
use rand::RngCore;
use serde::{Deserialize, Serialize};
use sha2::{Digest, Sha256};
use tokio::io::{AsyncBufReadExt, AsyncWriteExt, BufReader};
use tokio::net::TcpListener;
use crate::cli::oauth_defaults::{self, OAUTH_CALLBACK_PORT};
use crate::secrets::{CreateSecretParams, SecretsStore};
use crate::tools::mcp::config::McpServerConfig;
@@ -466,14 +466,12 @@ pub async fn authorize_mcp_server(
Ok(token)
}
/// Find an available port for the OAuth callback.
/// Bind the OAuth callback listener on the shared fixed port.
pub async fn find_available_port() -> Result<(TcpListener, u16), AuthError> {
for port in 9876..=9886 {
if let Ok(listener) = TcpListener::bind(format!("127.0.0.1:{}", port)).await {
return Ok((listener, port));
}
}
Err(AuthError::PortUnavailable)
let listener = oauth_defaults::bind_callback_listener()
.await
.map_err(|_| AuthError::PortUnavailable)?;
Ok((listener, OAUTH_CALLBACK_PORT))
}
/// Build the authorization URL with all required parameters.
@@ -522,71 +520,16 @@ pub async fn wait_for_authorization_callback(
listener: TcpListener,
server_name: &str,
) -> Result<String, AuthError> {
let timeout = Duration::from_secs(300);
tokio::time::timeout(timeout, async {
loop {
let (mut socket, _) = listener
.accept()
.await
.map_err(|e| AuthError::Http(e.to_string()))?;
let mut reader = BufReader::new(&mut socket);
let mut request_line = String::new();
reader
.read_line(&mut request_line)
.await
.map_err(|e| AuthError::Http(e.to_string()))?;
// Parse GET /callback?code=xxx HTTP/1.1
if let Some(path) = request_line.split_whitespace().nth(1) {
if path.starts_with("/callback") {
if let Some(query) = path.split('?').nth(1) {
// Check for error first
if query.contains("error=") {
let response = "HTTP/1.1 400 Bad Request\r\n\r\nAuthorization denied";
let _ = socket.write_all(response.as_bytes()).await;
return Err(AuthError::AuthorizationDenied);
}
// Look for code
for param in query.split('&') {
let parts: Vec<&str> = param.splitn(2, '=').collect();
if parts.len() == 2 && parts[0] == "code" {
let code = urlencoding::decode(parts[1])
.unwrap_or_else(|_| parts[1].into())
.into_owned();
// Send success response
let response = format!(
"HTTP/1.1 200 OK\r\n\
Content-Type: text/html\r\n\
\r\n\
<!DOCTYPE html><html><body style=\"font-family: sans-serif; \
display: flex; justify-content: center; align-items: center; \
height: 100vh; margin: 0; background: #191919; color: white;\">\
<div style=\"text-align: center;\">\
<h1> {} Connected!</h1>\
<p>You can close this window.</p>\
</div></body></html>",
server_name
);
let _ = socket.write_all(response.as_bytes()).await;
let _ = socket.shutdown().await;
return Ok(code);
}
}
}
}
oauth_defaults::wait_for_callback(listener, "/callback", "code", server_name)
.await
.map_err(|e| match e {
oauth_defaults::OAuthCallbackError::Denied => AuthError::AuthorizationDenied,
oauth_defaults::OAuthCallbackError::Timeout => AuthError::Timeout,
oauth_defaults::OAuthCallbackError::PortInUse(_, msg) => {
AuthError::Http(format!("Port error: {}", msg))
}
let response = "HTTP/1.1 404 Not Found\r\n\r\n";
let _ = socket.write_all(response.as_bytes()).await;
}
})
.await
.map_err(|_| AuthError::Timeout)?
oauth_defaults::OAuthCallbackError::Io(msg) => AuthError::Http(msg),
})
}
/// Exchange the authorization code for an access token.
+44 -43
View File
@@ -184,44 +184,46 @@ impl McpClient {
}
// Add Mcp-Session-Id header if we have a session
if let Some(ref session_manager) = self.session_manager {
if let Some(session_id) = session_manager.get_session_id(&self.server_name).await {
req_builder = req_builder.header("Mcp-Session-Id", session_id);
}
if let Some(ref session_manager) = self.session_manager
&& let Some(session_id) = session_manager.get_session_id(&self.server_name).await
{
req_builder = req_builder.header("Mcp-Session-Id", session_id);
}
let response = req_builder
.send()
.await
.map_err(|e| ToolError::ExternalService(format!("MCP request failed: {}", e)))?;
let response = req_builder.send().await.map_err(|e| {
let mut chain = format!("MCP request failed: {}", e);
let mut source = std::error::Error::source(&e);
while let Some(cause) = source {
chain.push_str(&format!(" -> {}", cause));
source = cause.source();
}
ToolError::ExternalService(chain)
})?;
// Check for 401 Unauthorized - try to refresh token on first attempt
if response.status() == reqwest::StatusCode::UNAUTHORIZED {
if attempt == 0 {
// Try to refresh the token
if let Some(ref secrets) = self.secrets {
if let Some(ref config) = self.server_config {
tracing::debug!(
"MCP token expired, attempting refresh for '{}'",
self.server_name
);
match refresh_access_token(config, secrets, &self.user_id).await {
Ok(_) => {
tracing::info!(
"MCP token refreshed for '{}'",
self.server_name
);
// Continue to next iteration to retry with new token
continue;
}
Err(e) => {
tracing::debug!(
"Token refresh failed for '{}': {}",
self.server_name,
e
);
// Fall through to return auth error
}
if let Some(ref secrets) = self.secrets
&& let Some(ref config) = self.server_config
{
tracing::debug!(
"MCP token expired, attempting refresh for '{}'",
self.server_name
);
match refresh_access_token(config, secrets, &self.user_id).await {
Ok(_) => {
tracing::info!("MCP token refreshed for '{}'", self.server_name);
// Continue to next iteration to retry with new token
continue;
}
Err(e) => {
tracing::debug!(
"Token refresh failed for '{}': {}",
self.server_name,
e
);
// Fall through to return auth error
}
}
}
@@ -245,16 +247,15 @@ impl McpClient {
/// Parse the HTTP response into an MCP response.
async fn parse_response(&self, response: reqwest::Response) -> Result<McpResponse, ToolError> {
// Extract session ID from response header
if let Some(ref session_manager) = self.session_manager {
if let Some(session_id) = response
if let Some(ref session_manager) = self.session_manager
&& let Some(session_id) = response
.headers()
.get("Mcp-Session-Id")
.and_then(|v| v.to_str().ok())
{
session_manager
.update_session_id(&self.server_name, Some(session_id.to_string()))
.await;
}
{
session_manager
.update_session_id(&self.server_name, Some(session_id.to_string()))
.await;
}
if !response.status().is_success() {
@@ -316,11 +317,11 @@ impl McpClient {
/// This should be called once per session to establish capabilities.
pub async fn initialize(&self) -> Result<InitializeResult, ToolError> {
// Check if already initialized
if let Some(ref session_manager) = self.session_manager {
if session_manager.is_initialized(&self.server_name).await {
// Return cached/default capabilities
return Ok(InitializeResult::default());
}
if let Some(ref session_manager) = self.session_manager
&& session_manager.is_initialized(&self.server_name).await
{
// Return cached/default capabilities
return Ok(InitializeResult::default());
}
// Ensure we have a session
+82 -1
View File
@@ -88,8 +88,18 @@ impl McpServerConfig {
}
/// Check if this server requires authentication.
///
/// Returns true if OAuth is pre-configured OR if this is a remote HTTPS server
/// (which likely supports Dynamic Client Registration even without pre-configured OAuth).
pub fn requires_auth(&self) -> bool {
self.oauth.is_some()
if self.oauth.is_some() {
return true;
}
// Remote HTTPS servers need auth handling (DCR, token refresh, 401 detection).
// Localhost/127.0.0.1 servers are assumed to be dev servers without auth.
let url_lower = self.url.to_lowercase();
let is_localhost = is_localhost_url(&url_lower);
url_lower.starts_with("https://") && !is_localhost
}
/// Get the secret name used to store the access token.
@@ -402,11 +412,43 @@ pub async fn remove_mcp_server_db(
Ok(())
}
/// Check if a URL points to a loopback address (localhost, 127.0.0.1, [::1]).
///
/// Uses `url::Url` for proper parsing so edge cases (IPv6, userinfo, ports)
/// are handled correctly without manual string splitting.
fn is_localhost_url(url: &str) -> bool {
let Ok(parsed) = url::Url::parse(url) else {
return false;
};
match parsed.host() {
Some(url::Host::Domain(d)) => d.eq_ignore_ascii_case("localhost"),
Some(url::Host::Ipv4(ip)) => ip.is_loopback(),
Some(url::Host::Ipv6(ip)) => ip.is_loopback(),
None => false,
}
}
#[cfg(test)]
mod tests {
use super::*;
use tempfile::tempdir;
#[test]
fn test_is_localhost_url() {
assert!(is_localhost_url("http://localhost:3000/path"));
assert!(is_localhost_url("https://localhost/path"));
assert!(is_localhost_url("http://127.0.0.1:8080"));
assert!(is_localhost_url("http://127.0.0.1"));
assert!(!is_localhost_url("https://notlocalhost.com/path"));
assert!(!is_localhost_url("https://example-localhost.io"));
assert!(!is_localhost_url("https://mcp.notion.com"));
assert!(is_localhost_url("http://user:pass@localhost:3000/path"));
// IPv6 loopback
assert!(is_localhost_url("http://[::1]:8080/path"));
assert!(is_localhost_url("http://[::1]/path"));
assert!(!is_localhost_url("http://[::2]:8080/path"));
}
#[test]
fn test_server_config_validation() {
// Valid HTTPS server
@@ -514,4 +556,43 @@ mod tests {
"mcp_notion_refresh_token"
);
}
#[test]
fn test_requires_auth_with_oauth() {
let config = McpServerConfig::new("notion", "https://mcp.notion.com")
.with_oauth(OAuthConfig::new("client-123"));
assert!(config.requires_auth());
}
#[test]
fn test_requires_auth_remote_https_without_oauth() {
// Remote HTTPS servers need auth even without pre-configured OAuth (DCR)
let config = McpServerConfig::new("github-copilot", "https://api.githubcopilot.com/mcp/");
assert!(config.requires_auth());
let config = McpServerConfig::new("notion", "https://mcp.notion.com");
assert!(config.requires_auth());
}
#[test]
fn test_requires_auth_localhost_no_auth() {
// Localhost servers are dev servers, no auth needed
let config = McpServerConfig::new("local", "http://localhost:8080");
assert!(!config.requires_auth());
let config = McpServerConfig::new("local", "http://127.0.0.1:3000/mcp");
assert!(!config.requires_auth());
// Even HTTPS localhost doesn't require auth
let config = McpServerConfig::new("local", "https://localhost:8443");
assert!(!config.requires_auth());
}
#[test]
fn test_requires_auth_http_remote_no_auth() {
// HTTP remote servers won't pass validation, but if they existed
// they wouldn't trigger HTTPS auth detection
let config = McpServerConfig::new("bad", "http://mcp.example.com");
assert!(!config.requires_auth());
}
}
+19 -6
View File
@@ -11,6 +11,7 @@ use crate::extensions::ExtensionManager;
use crate::llm::{LlmProvider, ToolDefinition};
use crate::orchestrator::job_manager::ContainerJobManager;
use crate::safety::SafetyLayer;
use crate::secrets::SecretsStore;
use crate::tools::builder::{BuildSoftwareTool, BuilderConfig, LlmSoftwareBuilder};
use crate::tools::builtin::{
ApplyPatchTool, CancelJobTool, CreateJobTool, EchoTool, HttpTool, JobStatusTool, JsonTool,
@@ -20,8 +21,8 @@ use crate::tools::builtin::{
};
use crate::tools::tool::{Tool, ToolDomain};
use crate::tools::wasm::{
Capabilities, ResourceLimits, WasmError, WasmStorageError, WasmToolRuntime, WasmToolStore,
WasmToolWrapper,
Capabilities, OAuthRefreshConfig, ResourceLimits, WasmError, WasmStorageError, WasmToolRuntime,
WasmToolStore, WasmToolWrapper,
};
use crate::workspace::Workspace;
@@ -96,10 +97,10 @@ impl ToolRegistry {
if let Ok(mut tools) = self.tools.try_write() {
tools.insert(name.clone(), tool);
// Mark as built-in so it can't be shadowed later
if PROTECTED_TOOL_NAMES.contains(&name.as_str()) {
if let Ok(mut builtins) = self.builtin_names.try_write() {
builtins.insert(name.clone());
}
if PROTECTED_TOOL_NAMES.contains(&name.as_str())
&& let Ok(mut builtins) = self.builtin_names.try_write()
{
builtins.insert(name.clone());
}
tracing::debug!("Registered tool: {}", name);
}
@@ -366,6 +367,12 @@ impl ToolRegistry {
if let Some(s) = reg.schema {
wrapper = wrapper.with_schema(s);
}
if let Some(store) = reg.secrets_store {
wrapper = wrapper.with_secrets_store(store);
}
if let Some(oauth) = reg.oauth_refresh {
wrapper = wrapper.with_oauth_refresh(oauth);
}
// Register the tool
self.register(Arc::new(wrapper)).await;
@@ -421,6 +428,8 @@ impl ToolRegistry {
limits: None,
description: Some(&tool_with_binary.tool.description),
schema: Some(tool_with_binary.tool.parameters_schema.clone()),
secrets_store: None,
oauth_refresh: None,
})
.await
.map_err(WasmRegistrationError::Wasm)?;
@@ -462,6 +471,10 @@ pub struct WasmToolRegistration<'a> {
pub description: Option<&'a str>,
/// Optional parameter schema override.
pub schema: Option<serde_json::Value>,
/// Secrets store for credential injection at request time.
pub secrets_store: Option<Arc<dyn SecretsStore + Send + Sync>>,
/// OAuth refresh configuration for auto-refreshing expired tokens.
pub oauth_refresh: Option<OAuthRefreshConfig>,
}
impl Default for ToolRegistry {
+59 -6
View File
@@ -199,6 +199,28 @@ pub trait Tool: Send + Sync {
}
}
/// Extract a required string parameter from a JSON object.
///
/// Returns `ToolError::InvalidParameters` if the key is missing or not a string.
pub fn require_str<'a>(params: &'a serde_json::Value, name: &str) -> Result<&'a str, ToolError> {
params
.get(name)
.and_then(|v| v.as_str())
.ok_or_else(|| ToolError::InvalidParameters(format!("missing '{}' parameter", name)))
}
/// Extract a required parameter of any type from a JSON object.
///
/// Returns `ToolError::InvalidParameters` if the key is missing.
pub fn require_param<'a>(
params: &'a serde_json::Value,
name: &str,
) -> Result<&'a serde_json::Value, ToolError> {
params
.get(name)
.ok_or_else(|| ToolError::InvalidParameters(format!("missing '{}' parameter", name)))
}
#[cfg(test)]
mod tests {
use super::*;
@@ -235,12 +257,7 @@ mod tests {
params: serde_json::Value,
_ctx: &JobContext,
) -> Result<ToolOutput, ToolError> {
let message = params
.get("message")
.and_then(|v| v.as_str())
.ok_or_else(|| {
ToolError::InvalidParameters("missing 'message' parameter".to_string())
})?;
let message = require_str(&params, "message")?;
Ok(ToolOutput::text(message, Duration::from_millis(1)))
}
@@ -277,4 +294,40 @@ mod tests {
let tool = EchoTool;
assert_eq!(tool.execution_timeout(), Duration::from_secs(60));
}
#[test]
fn test_require_str_present() {
let params = serde_json::json!({"name": "alice"});
assert_eq!(require_str(&params, "name").unwrap(), "alice");
}
#[test]
fn test_require_str_missing() {
let params = serde_json::json!({});
let err = require_str(&params, "name").unwrap_err();
assert!(err.to_string().contains("missing 'name'"));
}
#[test]
fn test_require_str_wrong_type() {
let params = serde_json::json!({"name": 42});
let err = require_str(&params, "name").unwrap_err();
assert!(err.to_string().contains("missing 'name'"));
}
#[test]
fn test_require_param_present() {
let params = serde_json::json!({"data": [1, 2, 3]});
assert_eq!(
require_param(&params, "data").unwrap(),
&serde_json::json!([1, 2, 3])
);
}
#[test]
fn test_require_param_missing() {
let params = serde_json::json!({});
let err = require_param(&params, "data").unwrap_err();
assert!(err.to_string().contains("missing 'data'"));
}
}
+16 -15
View File
@@ -209,10 +209,10 @@ impl EndpointPattern {
}
// Check path prefix
if let Some(ref prefix) = self.path_prefix {
if !url_path.starts_with(prefix) {
return false;
}
if let Some(ref prefix) = self.path_prefix
&& !url_path.starts_with(prefix)
{
return false;
}
// Check method
@@ -237,13 +237,14 @@ impl EndpointPattern {
}
// Support wildcard: *.example.com matches sub.example.com
if let Some(suffix) = self.host.strip_prefix("*.") {
if url_host.ends_with(suffix) && url_host.len() > suffix.len() {
// Ensure there's a dot before the suffix (or it's the whole thing)
let prefix = &url_host[..url_host.len() - suffix.len()];
if prefix.ends_with('.') || prefix.is_empty() {
return true;
}
if let Some(suffix) = self.host.strip_prefix("*.")
&& url_host.ends_with(suffix)
&& url_host.len() > suffix.len()
{
// Ensure there's a dot before the suffix (or it's the whole thing)
let prefix = &url_host[..url_host.len() - suffix.len()];
if prefix.ends_with('.') || prefix.is_empty() {
return true;
}
}
@@ -291,10 +292,10 @@ impl SecretsCapability {
if pattern == name {
return true;
}
if let Some(prefix) = pattern.strip_suffix('*') {
if name.starts_with(prefix) {
return true;
}
if let Some(prefix) = pattern.strip_suffix('*')
&& name.starts_with(prefix)
{
return true;
}
}
false
+13 -12
View File
@@ -158,10 +158,10 @@ impl CredentialInjector {
if pattern == name {
return true;
}
if let Some(prefix) = pattern.strip_suffix('*') {
if name.starts_with(prefix) {
return true;
}
if let Some(prefix) = pattern.strip_suffix('*')
&& name.starts_with(prefix)
{
return true;
}
}
false
@@ -169,7 +169,7 @@ impl CredentialInjector {
}
/// Inject a single credential into the result.
fn inject_credential(
pub(crate) fn inject_credential(
result: &mut InjectedCredentials,
location: &CredentialLocation,
secret: &DecryptedSecret,
@@ -208,18 +208,19 @@ fn inject_credential(
}
/// Check if a host matches a pattern (supports wildcards).
fn host_matches_pattern(host: &str, pattern: &str) -> bool {
pub(crate) fn host_matches_pattern(host: &str, pattern: &str) -> bool {
if pattern == host {
return true;
}
// Support wildcard: *.example.com matches sub.example.com
if let Some(suffix) = pattern.strip_prefix("*.") {
if host.ends_with(suffix) && host.len() > suffix.len() {
let prefix = &host[..host.len() - suffix.len()];
if prefix.ends_with('.') || prefix.is_empty() {
return true;
}
if let Some(suffix) = pattern.strip_prefix("*.")
&& host.ends_with(suffix)
&& host.len() > suffix.len()
{
let prefix = &host[..host.len() - suffix.len()];
if prefix.ends_with('.') || prefix.is_empty() {
return true;
}
}
+179 -7
View File
@@ -39,10 +39,11 @@ use std::sync::Arc;
use tokio::fs;
use crate::secrets::SecretsStore;
use crate::tools::registry::{ToolRegistry, WasmRegistrationError, WasmToolRegistration};
use crate::tools::wasm::capabilities_schema::CapabilitiesFile;
use crate::tools::wasm::{
Capabilities, WasmError, WasmStorageError, WasmToolRuntime, WasmToolStore,
Capabilities, OAuthRefreshConfig, WasmError, WasmStorageError, WasmToolRuntime, WasmToolStore,
};
/// Error during WASM tool loading.
@@ -77,12 +78,23 @@ pub enum WasmLoadError {
pub struct WasmToolLoader {
runtime: Arc<WasmToolRuntime>,
registry: Arc<ToolRegistry>,
secrets_store: Option<Arc<dyn SecretsStore + Send + Sync>>,
}
impl WasmToolLoader {
/// Create a new loader with the given runtime and registry.
pub fn new(runtime: Arc<WasmToolRuntime>, registry: Arc<ToolRegistry>) -> Self {
Self { runtime, registry }
Self {
runtime,
registry,
secrets_store: None,
}
}
/// Set the secrets store for credential injection in WASM tools.
pub fn with_secrets_store(mut self, store: Arc<dyn SecretsStore + Send + Sync>) -> Self {
self.secrets_store = Some(store);
self
}
/// Load a single WASM tool from a file pair.
@@ -108,22 +120,24 @@ impl WasmToolLoader {
}
let wasm_bytes = fs::read(wasm_path).await?;
// Read capabilities (optional)
let capabilities = if let Some(cap_path) = capabilities_path {
// Read capabilities (optional) and extract OAuth refresh config
let (capabilities, oauth_refresh) = if let Some(cap_path) = capabilities_path {
if cap_path.exists() {
let cap_bytes = fs::read(cap_path).await?;
let cap_file = CapabilitiesFile::from_bytes(&cap_bytes)
.map_err(|e| WasmLoadError::InvalidCapabilities(e.to_string()))?;
cap_file.to_capabilities()
let caps = cap_file.to_capabilities();
let oauth = resolve_oauth_refresh_config(&cap_file);
(caps, oauth)
} else {
tracing::warn!(
path = %cap_path.display(),
"Capabilities file not found, using default (no permissions)"
);
Capabilities::default()
(Capabilities::default(), None)
}
} else {
Capabilities::default()
(Capabilities::default(), None)
};
// Register the tool
@@ -136,6 +150,8 @@ impl WasmToolLoader {
limits: None,
description: None,
schema: None,
secrets_store: self.secrets_store.clone(),
oauth_refresh,
})
.await?;
@@ -293,6 +309,50 @@ impl WasmToolLoader {
}
}
/// Extract OAuth refresh configuration from a parsed capabilities file.
///
/// Returns `None` if there's no `auth.oauth` section or if the client_id
/// can't be resolved from any source (inline, env var, or built-in defaults).
///
/// Fallback chain for client_id:
/// `oauth.client_id` > env var (`oauth.client_id_env`) > `builtin_credentials()`
fn resolve_oauth_refresh_config(cap_file: &CapabilitiesFile) -> Option<OAuthRefreshConfig> {
let auth = cap_file.auth.as_ref()?;
let oauth = auth.oauth.as_ref()?;
let builtin = crate::cli::oauth_defaults::builtin_credentials(&auth.secret_name);
let client_id = oauth
.client_id
.clone()
.or_else(|| {
oauth
.client_id_env
.as_ref()
.and_then(|env| std::env::var(env).ok())
})
.or_else(|| builtin.as_ref().map(|c| c.client_id.to_string()))?;
let client_secret = oauth
.client_secret
.clone()
.or_else(|| {
oauth
.client_secret_env
.as_ref()
.and_then(|env| std::env::var(env).ok())
})
.or_else(|| builtin.as_ref().map(|c| c.client_secret.to_string()));
Some(OAuthRefreshConfig {
token_url: oauth.token_url.clone(),
client_id,
client_secret,
secret_name: auth.secret_name.clone(),
provider: auth.provider.clone(),
})
}
/// Results from loading multiple tools.
#[derive(Debug, Default)]
pub struct LoadResults {
@@ -618,4 +678,116 @@ mod tests {
);
}
}
#[test]
fn test_resolve_oauth_refresh_config_with_oauth() {
use crate::tools::wasm::capabilities_schema::{
AuthCapabilitySchema, CapabilitiesFile, OAuthConfigSchema,
};
let caps = CapabilitiesFile {
auth: Some(AuthCapabilitySchema {
secret_name: "google_oauth_token".to_string(),
provider: Some("google".to_string()),
oauth: Some(OAuthConfigSchema {
authorization_url: "https://accounts.google.com/o/oauth2/v2/auth".to_string(),
token_url: "https://oauth2.googleapis.com/token".to_string(),
client_id: Some("test-client-id".to_string()),
client_secret: Some("test-client-secret".to_string()),
..Default::default()
}),
..Default::default()
}),
..Default::default()
};
let config = super::resolve_oauth_refresh_config(&caps);
assert!(config.is_some());
let config = config.unwrap();
assert_eq!(config.token_url, "https://oauth2.googleapis.com/token");
assert_eq!(config.client_id, "test-client-id");
assert_eq!(config.client_secret, Some("test-client-secret".to_string()));
assert_eq!(config.secret_name, "google_oauth_token");
assert_eq!(config.provider, Some("google".to_string()));
}
#[test]
fn test_resolve_oauth_refresh_config_no_auth() {
use crate::tools::wasm::capabilities_schema::CapabilitiesFile;
let caps = CapabilitiesFile::default();
let config = super::resolve_oauth_refresh_config(&caps);
assert!(config.is_none());
}
#[test]
fn test_resolve_oauth_refresh_config_no_oauth() {
use crate::tools::wasm::capabilities_schema::{AuthCapabilitySchema, CapabilitiesFile};
let caps = CapabilitiesFile {
auth: Some(AuthCapabilitySchema {
secret_name: "manual_token".to_string(),
..Default::default()
}),
..Default::default()
};
let config = super::resolve_oauth_refresh_config(&caps);
assert!(config.is_none());
}
#[test]
fn test_resolve_oauth_refresh_config_no_client_id() {
use crate::tools::wasm::capabilities_schema::{
AuthCapabilitySchema, CapabilitiesFile, OAuthConfigSchema,
};
// A non-Google provider with no client_id anywhere should return None
let caps = CapabilitiesFile {
auth: Some(AuthCapabilitySchema {
secret_name: "unknown_provider_token".to_string(),
oauth: Some(OAuthConfigSchema {
authorization_url: "https://example.com/auth".to_string(),
token_url: "https://example.com/token".to_string(),
// No client_id, no client_id_env, no builtin
..Default::default()
}),
..Default::default()
}),
..Default::default()
};
let config = super::resolve_oauth_refresh_config(&caps);
assert!(config.is_none());
}
#[test]
fn test_resolve_oauth_refresh_config_builtin_google() {
use crate::tools::wasm::capabilities_schema::{
AuthCapabilitySchema, CapabilitiesFile, OAuthConfigSchema,
};
// google_oauth_token should fall back to built-in credentials
let caps = CapabilitiesFile {
auth: Some(AuthCapabilitySchema {
secret_name: "google_oauth_token".to_string(),
provider: Some("google".to_string()),
oauth: Some(OAuthConfigSchema {
authorization_url: "https://accounts.google.com/o/oauth2/v2/auth".to_string(),
token_url: "https://oauth2.googleapis.com/token".to_string(),
// No inline client_id, should fall back to builtin
..Default::default()
}),
..Default::default()
}),
..Default::default()
};
let config = super::resolve_oauth_refresh_config(&caps);
assert!(config.is_some());
let config = config.unwrap();
assert!(!config.client_id.is_empty());
assert!(config.client_secret.is_some());
}
}
+1 -1
View File
@@ -94,7 +94,7 @@ pub use limits::{
WasmResourceLimiter,
};
pub use runtime::{PreparedModule, WasmRuntimeConfig, WasmToolRuntime};
pub use wrapper::WasmToolWrapper;
pub use wrapper::{OAuthRefreshConfig, WasmToolWrapper};
// Capabilities (V2)
pub use capabilities::{
+1
View File
@@ -953,6 +953,7 @@ fn libsql_row_to_tool_with_offset(row: &libsql::Row) -> Result<StoredWasmTool, W
}
#[cfg(feature = "libsql")]
#[allow(clippy::too_many_arguments)]
fn libsql_row_to_tool_at(
row: &libsql::Row,
id_idx: i32,
+884 -21
View File
File diff suppressed because it is too large Load Diff
+319
View File
@@ -0,0 +1,319 @@
//! Truncating terminal writer for tracing.
//!
//! Tracing events from LLM providers can dump 10KB+ JSON bodies to stderr.
//! Rather than truncating at every call site (fragile, easy to miss), we
//! handle it at the writer level: the fmt layer gets a `TruncatingStderr`
//! that caps each event before flushing, while the web gateway `WebLogLayer`
//! still sees the full, untruncated content.
//!
//! ```text
//! tracing::debug!("body: {huge_json}")
//! |
//! v
//! tracing_subscriber::registry()
//! |
//! +-- fmt::layer().with_writer(TruncatingStderr) <-- caps at 500B
//! | \-- stderr (truncated)
//! |
//! \-- WebLogLayer (unchanged)
//! \-- SSE broadcast (full)
//! ```
use std::io::{self, Write};
use tracing_subscriber::fmt::MakeWriter;
/// Maximum bytes per tracing event written to the terminal.
const TERMINAL_MAX_EVENT_BYTES: usize = 500;
/// A `MakeWriter` that creates per-event buffers which truncate on flush.
///
/// Each call to `make_writer()` returns an `EventBuffer`. All `write()`
/// calls accumulate into the buffer. When the buffer drops (after the fmt
/// layer finishes writing one event), it flushes to stderr, truncating if
/// the total exceeds `TERMINAL_MAX_EVENT_BYTES`.
#[derive(Clone)]
pub struct TruncatingStderr {
max_bytes: usize,
}
impl Default for TruncatingStderr {
fn default() -> Self {
Self {
max_bytes: TERMINAL_MAX_EVENT_BYTES,
}
}
}
impl TruncatingStderr {
#[cfg(test)]
fn with_max_bytes(max_bytes: usize) -> Self {
Self { max_bytes }
}
}
impl<'a> MakeWriter<'a> for TruncatingStderr {
type Writer = EventBuffer;
fn make_writer(&'a self) -> Self::Writer {
EventBuffer {
buf: Vec::with_capacity(256),
max_bytes: self.max_bytes,
#[cfg(test)]
sink: None,
}
}
}
/// Per-event buffer that truncates on drop.
pub struct EventBuffer {
buf: Vec<u8>,
max_bytes: usize,
/// Test-only: capture output instead of writing to stderr.
#[cfg(test)]
sink: Option<std::sync::Arc<std::sync::Mutex<Vec<u8>>>>,
}
impl Write for EventBuffer {
fn write(&mut self, data: &[u8]) -> io::Result<usize> {
self.buf.extend_from_slice(data);
Ok(data.len())
}
fn flush(&mut self) -> io::Result<()> {
Ok(())
}
}
/// Find the last valid UTF-8 char boundary at or before `pos` in `bytes`.
///
/// Walks backwards from `pos` until we find a byte that isn't a UTF-8
/// continuation byte (0x80..0xBF). Returns 0 if the entire prefix is
/// somehow invalid (shouldn't happen with valid UTF-8 input from tracing).
fn utf8_floor(bytes: &[u8], pos: usize) -> usize {
let mut i = pos;
// UTF-8 continuation bytes have the form 10xxxxxx (0x80..0xBF).
// Walk backwards past them to find the start of the last character.
while i > 0 && bytes[i] & 0xC0 == 0x80 {
i -= 1;
}
i
}
impl Drop for EventBuffer {
fn drop(&mut self) {
if self.buf.is_empty() {
return;
}
let output = if self.buf.len() <= self.max_bytes {
&self.buf[..]
} else {
// Truncate at a UTF-8 safe boundary
let cut = utf8_floor(&self.buf, self.max_bytes);
let suffix = format!("...[{}B total]\n", self.buf.len());
let mut truncated = Vec::with_capacity(cut + suffix.len());
// Strip trailing newline from the cut portion (we add our own via suffix)
let cut_slice = &self.buf[..cut];
let trimmed = if cut_slice.last() == Some(&b'\n') {
&cut_slice[..cut_slice.len() - 1]
} else {
cut_slice
};
truncated.extend_from_slice(trimmed);
truncated.extend_from_slice(suffix.as_bytes());
#[cfg(test)]
if let Some(ref sink) = self.sink {
let mut s = sink.lock().expect("test sink lock poisoned");
s.extend_from_slice(&truncated);
return;
}
let _ = io::stderr().write_all(&truncated);
return;
};
#[cfg(test)]
if let Some(ref sink) = self.sink {
let mut s = sink.lock().expect("test sink lock poisoned");
s.extend_from_slice(output);
return;
}
let _ = io::stderr().write_all(output);
}
}
#[cfg(test)]
mod tests {
use std::sync::{Arc, Mutex};
use crate::tracing_fmt::{EventBuffer, TruncatingStderr, utf8_floor};
use std::io::Write;
/// Helper: create an EventBuffer that captures output to a shared Vec
/// instead of writing to stderr.
fn test_buffer(max_bytes: usize) -> (EventBuffer, Arc<Mutex<Vec<u8>>>) {
let sink = Arc::new(Mutex::new(Vec::new()));
let buf = EventBuffer {
buf: Vec::new(),
max_bytes,
sink: Some(Arc::clone(&sink)),
};
(buf, sink)
}
#[test]
fn test_short_event_not_truncated() {
let (mut buf, sink) = test_buffer(500);
buf.write_all(b"hello world\n").unwrap();
drop(buf);
let output = sink.lock().unwrap();
assert_eq!(&*output, b"hello world\n");
}
#[test]
fn test_long_event_truncated() {
let (mut buf, sink) = test_buffer(20);
let data = "abcdefghijklmnopqrstuvwxyz0123456789\n";
buf.write_all(data.as_bytes()).unwrap();
let total = data.len();
drop(buf);
let output = sink.lock().unwrap();
let output_str = String::from_utf8_lossy(&output);
// Should contain the suffix with total byte count
assert!(
output_str.contains(&format!("...[{}B total]", total)),
"expected truncation suffix, got: {}",
output_str
);
// Should be shorter than the original
assert!(output.len() < total);
}
#[test]
fn test_utf8_boundary_safe() {
// "Helloé" = [72, 101, 108, 108, 111, 195, 169]
// ^-- 2-byte UTF-8 char
// If we truncate at 6 bytes, we'd land in the middle of 'é'.
// utf8_floor should back up to byte 5 (start of 'é' = 195).
let (mut buf, sink) = test_buffer(6);
let data = "Helloé world";
buf.write_all(data.as_bytes()).unwrap();
drop(buf);
let output = sink.lock().unwrap();
let output_str = String::from_utf8(output.clone());
assert!(
output_str.is_ok(),
"output should be valid UTF-8, got bytes: {:?}",
&*output
);
let s = output_str.unwrap();
assert!(
s.contains("...["),
"should be truncated with suffix, got: {}",
s
);
// The truncated prefix must be valid UTF-8 up to the cut point.
// "Hello" (5 bytes) is the last valid cut before the 2-byte é.
assert!(
s.starts_with("Hello"),
"should start with 'Hello', got: {}",
s
);
}
#[test]
fn test_utf8_floor_basic() {
// ASCII: every byte is a valid boundary
assert_eq!(utf8_floor(b"hello", 3), 3);
// 2-byte UTF-8 char é = [0xC3, 0xA9]
// Landing on the continuation byte (0xA9) should back up to 0xC3
let bytes = "".as_bytes(); // [72, 0xC3, 0xA9]
assert_eq!(utf8_floor(bytes, 2), 1); // backs up to start of é
// 3-byte UTF-8 char (e.g. あ = [0xE3, 0x81, 0x82])
let bytes = "aあ".as_bytes(); // [97, 0xE3, 0x81, 0x82]
assert_eq!(utf8_floor(bytes, 2), 1); // backs up past continuation to 0xE3
assert_eq!(utf8_floor(bytes, 3), 1); // same: 0x82 is continuation, 0x81 is too
}
#[test]
fn test_multiple_writes_accumulated() {
let (mut buf, sink) = test_buffer(500);
buf.write_all(b"hello ").unwrap();
buf.write_all(b"world\n").unwrap();
drop(buf);
let output = sink.lock().unwrap();
assert_eq!(&*output, b"hello world\n");
}
#[test]
fn test_empty_buffer_no_output() {
let (_buf, sink) = test_buffer(500);
// drop without writing
drop(_buf);
let output = sink.lock().unwrap();
assert!(output.is_empty());
}
#[test]
fn test_default_max_bytes() {
let writer = TruncatingStderr::default();
assert_eq!(writer.max_bytes, 500);
}
#[test]
fn test_custom_max_bytes() {
let writer = TruncatingStderr::with_max_bytes(100);
assert_eq!(writer.max_bytes, 100);
}
#[test]
fn test_exactly_at_limit_not_truncated() {
let (mut buf, sink) = test_buffer(5);
buf.write_all(b"hello").unwrap();
drop(buf);
let output = sink.lock().unwrap();
assert_eq!(&*output, b"hello");
}
#[test]
fn test_one_over_limit_truncated() {
let (mut buf, sink) = test_buffer(5);
buf.write_all(b"hello!").unwrap();
drop(buf);
let output = sink.lock().unwrap();
let s = String::from_utf8_lossy(&output);
assert!(s.contains("...[6B total]"), "got: {}", s);
}
#[test]
fn test_4byte_utf8_boundary() {
// 4-byte UTF-8 char: 𝄞 (musical symbol) = [0xF0, 0x9D, 0x84, 0x9E]
let data = "AB𝄞CD";
// bytes: [65, 66, 0xF0, 0x9D, 0x84, 0x9E, 67, 68]
// Truncating at byte 4 lands in the middle of the 4-byte char
let (mut buf, sink) = test_buffer(4);
buf.write_all(data.as_bytes()).unwrap();
drop(buf);
let output = sink.lock().unwrap();
let s = String::from_utf8(output.clone());
assert!(s.is_ok(), "output must be valid UTF-8, got: {:?}", &*output);
let s = s.unwrap();
// Should back up to byte 2 (just "AB"), since bytes 2..5 are all part of 𝄞
assert!(s.starts_with("AB"), "expected 'AB', got: {}", s);
assert!(s.contains("...["), "should be truncated, got: {}", s);
}
}
+54 -70
View File
@@ -129,11 +129,15 @@ impl WorkerHttpClient {
format!("{}/worker/{}/{}", self.orchestrator_url, self.job_id, path)
}
/// Fetch the job description from the orchestrator.
pub async fn get_job(&self) -> Result<JobDescription, WorkerError> {
/// Send a GET request, check the status, and deserialize the JSON body.
async fn get_json<T: serde::de::DeserializeOwned>(
&self,
path: &str,
context: &str,
) -> Result<T, WorkerError> {
let resp = self
.client
.get(self.url("job"))
.get(self.url(path))
.bearer_auth(&self.token)
.send()
.await
@@ -145,15 +149,51 @@ impl WorkerHttpClient {
if !resp.status().is_success() {
return Err(WorkerError::OrchestratorRejected {
job_id: self.job_id,
reason: format!("GET /job returned {}", resp.status()),
reason: format!("{} returned {}", context, resp.status()),
});
}
resp.json().await.map_err(|e| WorkerError::LlmProxyFailed {
reason: format!("failed to parse job description: {}", e),
reason: format!("{}: failed to parse response: {}", context, e),
})
}
/// Send a POST request with a JSON body, check the status, and deserialize the response.
async fn post_json<B: Serialize, T: serde::de::DeserializeOwned>(
&self,
path: &str,
body: &B,
context: &str,
) -> Result<T, WorkerError> {
let resp = self
.client
.post(self.url(path))
.bearer_auth(&self.token)
.json(body)
.send()
.await
.map_err(|e| WorkerError::LlmProxyFailed {
reason: format!("{}: {}", context, e),
})?;
if !resp.status().is_success() {
let status = resp.status();
let body = resp.text().await.unwrap_or_default();
return Err(WorkerError::LlmProxyFailed {
reason: format!("{}: orchestrator returned {}: {}", context, status, body),
});
}
resp.json().await.map_err(|e| WorkerError::LlmProxyFailed {
reason: format!("{}: failed to parse response: {}", context, e),
})
}
/// Fetch the job description from the orchestrator.
pub async fn get_job(&self) -> Result<JobDescription, WorkerError> {
self.get_json("job", "GET /job").await
}
/// Proxy an LLM completion request through the orchestrator.
pub async fn llm_complete(
&self,
@@ -166,29 +206,9 @@ impl WorkerHttpClient {
stop_sequences: request.stop_sequences.clone(),
};
let resp = self
.client
.post(self.url("llm/complete"))
.bearer_auth(&self.token)
.json(&proxy_req)
.send()
.await
.map_err(|e| WorkerError::LlmProxyFailed {
reason: e.to_string(),
})?;
if !resp.status().is_success() {
let status = resp.status();
let body = resp.text().await.unwrap_or_default();
return Err(WorkerError::LlmProxyFailed {
reason: format!("orchestrator returned {}: {}", status, body),
});
}
let proxy_resp: ProxyCompletionResponse =
resp.json().await.map_err(|e| WorkerError::LlmProxyFailed {
reason: format!("failed to parse LLM response: {}", e),
})?;
let proxy_resp: ProxyCompletionResponse = self
.post_json("llm/complete", &proxy_req, "LLM complete")
.await?;
Ok(CompletionResponse {
content: proxy_resp.content,
@@ -212,29 +232,9 @@ impl WorkerHttpClient {
tool_choice: request.tool_choice.clone(),
};
let resp = self
.client
.post(self.url("llm/complete_with_tools"))
.bearer_auth(&self.token)
.json(&proxy_req)
.send()
.await
.map_err(|e| WorkerError::LlmProxyFailed {
reason: e.to_string(),
})?;
if !resp.status().is_success() {
let status = resp.status();
let body = resp.text().await.unwrap_or_default();
return Err(WorkerError::LlmProxyFailed {
reason: format!("orchestrator returned {}: {}", status, body),
});
}
let proxy_resp: ProxyToolCompletionResponse =
resp.json().await.map_err(|e| WorkerError::LlmProxyFailed {
reason: format!("failed to parse tool completion response: {}", e),
})?;
let proxy_resp: ProxyToolCompletionResponse = self
.post_json("llm/complete_with_tools", &proxy_req, "LLM tool complete")
.await?;
Ok(ToolCompletionResponse {
content: proxy_resp.content,
@@ -337,25 +337,9 @@ impl WorkerHttpClient {
/// Signal job completion to the orchestrator.
pub async fn report_complete(&self, report: &CompletionReport) -> Result<(), WorkerError> {
let resp = self
.client
.post(self.url("complete"))
.bearer_auth(&self.token)
.json(report)
.send()
.await
.map_err(|e| WorkerError::ConnectionFailed {
url: self.orchestrator_url.clone(),
reason: e.to_string(),
})?;
if !resp.status().is_success() {
return Err(WorkerError::OrchestratorRejected {
job_id: self.job_id,
reason: format!("completion report rejected: {}", resp.status()),
});
}
let _: serde_json::Value = self
.post_json("complete", report, "report complete")
.await?;
Ok(())
}
}

Some files were not shown because too many files have changed in this diff Show More