Compare commits

..
Author SHA1 Message Date
Henry ParkandClaude Opus 4.6 63e2e0f516 refactor: address PR review feedback on test credentials
- Fix TEST_CRYPTO_KEY doc comment ("32-byte hex" → "32-character key string")
- Rename confusing "real"/"fake" Anthropic constant names and values
- Change TEST_STRIPE_KEY from "sk-live" to "sk_test_fake123" to avoid scanners
- Use test_secrets_store() helper in orchestrator and http tool tests
- Clarify config_round_trip.rs doc comment about integration test visibility

Co-Authored-By: Claude Opus 4.6 <[email protected]>
2026-03-10 11:07:17 -07:00
Henry ParkandClaude Sonnet 4.6 3d4ccd884e refactor: replace real Telegram bot token with obviously fake test stub
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-03-09 22:19:57 -07:00
Henry ParkandClaude Opus 4.6 a268790b88 refactor: centralize test credential constants into testing::credentials
Scattered test credential strings (API keys, OAuth tokens, crypto keys,
Telegram tokens, session tokens) across ~25 files made security auditing
harder and created unnecessary duplication. Centralize all test-only fake
credentials into a new `src/testing/credentials.rs` module with named
constants and a shared `test_secrets_store()` helper.

- Convert `src/testing.rs` to directory module (`src/testing/mod.rs`)
- Add `src/testing/credentials.rs` with ~30 named constants
- Replace hardcoded literals in 24 source files
- Deduplicate `test_store()` helper (was copy-pasted in 3 files)
- Leave leak_detector/shell/signature tests as-is (inline values
  aid readability for pattern detection tests)

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
2026-03-09 22:16:36 -07:00
3a2989d009 feat(setup): quick onboarding, preserve .env vars, clean up boot logging (#821)
* feat(setup): quick onboarding, preserve .env vars, clean up boot logging (#751, #674)

- Add upsert_bootstrap_vars() to preserve user-added .env vars on re-onboarding
- Add --quick mode: auto-defaults DB + security, asks only LLM provider (2 steps)
- Auto-triggered onboarding uses quick mode for near-instant first run
- Fix NEAR AI model fetch to use cloud-api.near.ai when API key is set
- Handle missing WASM tools/channels directories gracefully
- Downgrade all boot/shutdown tracing::info! to debug (boot screen shows user output)

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix(setup): gate env_backend variable behind postgres feature flag [skip-regression-check]

Clippy lint fix — not a behavioral change, just moving a variable declaration
inside the cfg(feature = "postgres") block where it's used.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix(review): address PR review comments

- WASM loaders: use tokio::fs::metadata, only treat NotFound as empty,
  propagate other IO errors, handle TOCTOU in read_dir
- bootstrap: only ignore NotFound in read_to_string, propagate other errors
- wizard: restore print_info/print_success for migrations in interactive
  mode (gated by !config.quick), keep tracing::debug for diagnostics
- tests: use shared crate::config::helpers::ENV_MUTEX instead of separate
  NEARAI_ENV_MUTEX to prevent cross-test env var races
- README: fix quick mode description to mention model selection, clarify
  auto_setup_database may prompt when DATABASE_URL is set

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* feat(setup): skip prompts for DATABASE_URL in quick mode [skip-regression-check]

auto_setup_database() now uses DATABASE_URL directly without calling
step_database_postgres() (which prompts for confirmation). Quick mode
should be fully non-interactive when env vars are already set.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix(cli): update --quick help text to mention model selection [skip-regression-check]

Co-Authored-By: Claude Opus 4.6 <[email protected]>

---------

Co-authored-by: Claude Opus 4.6 <[email protected]>
2026-03-10 05:02:33 +00:00
94d101924e refactor: encapsulate leaked abstractions into owning modules (#778)
* refactor: encapsulate leaked abstractions from main.rs and app.rs into owning modules

Move module-specific initialization logic out of main.rs (1222→665 lines, -46%) and
app.rs (944→780 lines, -17%) into their respective owning modules as public factory
functions. This enforces separation of concerns so that adding a new DB backend, MCP
transport, or channel doesn't require editing main.rs/app.rs.

Key changes:
- Tracing init functions → src/tracing_fmt.rs
- DB connection factory (connect_with_handles + DatabaseHandles) → src/db/mod.rs
- Secrets store factory (create_secrets_store) → src/secrets/mod.rs
- MCP transport dispatch factory (create_client_from_config) → src/tools/mcp/factory.rs
- Orchestrator setup (setup_orchestrator + OrchestratorSetup) → src/orchestrator/mod.rs
- WASM channel setup (setup_wasm_channels) → src/channels/wasm/setup.rs
- Worker entry points (run_worker, run_claude_bridge) → src/worker/mod.rs
- Shared CLI secrets init (init_secrets_store) → src/cli/mod.rs
- Tunnel startup (start_managed_tunnel) → src/tunnel/mod.rs
- Onboard check (check_onboard_needed) → src/setup/mod.rs
- ExtensionManager unified MCP: uses create_client_from_config via McpProcessManager,
  enabling stdio/Unix transports for hot-activated MCP servers
- Deduplicated ~130 lines of secrets store init across cli/mcp.rs and cli/tool.rs
- CLAUDE.md updated with module-owned initialization guideline

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* refactor: address review feedback — deduplicate db factory, extract channel helper

- connect_from_config() now delegates to connect_with_handles() to eliminate
  duplicated backend-matching logic (Copilot review feedback)
- Extract register_channel() helper from setup_wasm_channels() loop body
  to improve readability (Gemini review feedback)

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* style: fix rustfmt line wrapping in setup_wasm_channels

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* test: add integration test for module-owned initialization factories

Exercises the full factory chain end-to-end to verify nothing was lost
when initialization logic was moved from main.rs/app.rs into owning modules:

- connect_with_handles returns Database + populated backend handles
- connect_from_config delegates correctly (produces working Database)
- secrets::create_secrets_store builds working store from DatabaseHandles
- db::create_secrets_store standalone factory round-trips secrets
- Both secrets factories produce compatible stores (cross-read works)
- ExtensionManager constructs with McpProcessManager and is functional
- DatabaseHandles default is empty

All tests run without external services using libsql in-memory/tempfile.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: wire cli/mcp.rs and cli/tool.rs to shared init_secrets_store()

Both files had inline implementations identical to cli::init_secrets_store().
Replace with delegation to complete the claimed deduplication.

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* style: fix rustfmt line wrapping in integration test

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix(review): remove unused Config import and deduplicate Error Handling section

- Remove `#[allow(unused_imports)]` and unused `use crate::config::Config`
  from cli/tool.rs (no longer needed after delegating to shared
  `cli::init_secrets_store()`)
- Remove duplicate Error Handling subsection from CLAUDE.md Key Patterns
  (all four bullets already exist in Code Style section and
  review-discipline.md)

Addresses Copilot review comments.

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix(review): address remaining Copilot review comments

- secrets/mod.rs: clarify docstring that None is a normal no-db condition
- app.rs: add comment explaining the empty_handles fallback path
- orchestrator/mod.rs: combine duplicated sandbox condition into single block
- setup/mod.rs: document env var reads and thread-safety caveat

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 <[email protected]>

---------

Co-authored-by: Claude Opus 4.6 <[email protected]>
Co-authored-by: Henry Park <[email protected]>
2026-03-10 04:39:51 +00:00
a868b14221 Fix/lightweight action tool (#785)
* feat: add tool execution support to lightweight routines

Lightweight routines now execute tools instead of outputting raw tool-call XML.

**Problem:** Lightweight routines had no tool execution loop, causing the LLM to generate
tool-call XML as text output (visible to users as garbage on Telegram). All 4 scheduled
routines were disabled and Emil saw the same issue in health-ping routine.

**Solution:** Implement a simplified agentic loop for lightweight routines that:
- Supports up to 3-5 tool iterations (configurable, capped at 5)
- Executes tools sequentially (not parallel, keeps overhead low)
- Auto-approves non-Always tools (lightweight routines are autonomous)
- Sanitizes and wraps tool outputs via SafetyLayer (same as dispatcher)
- Forces text-only response at iteration limit (guarantees termination)
- Maintains backward compatibility (disabled by default, toggled by config)

**Changes:**
1. **src/config/routines.rs:**
   - Added lightweight_tools_enabled (default: true)
   - Added lightweight_max_iterations (default: 3, capped at 5)
   - Added env var support: ROUTINES_LIGHTWEIGHT_TOOLS, ROUTINES_LIGHTWEIGHT_MAX_ITERATIONS

2. **src/agent/routine_engine.rs:**
   - Extended EngineContext with tools and safety fields
   - Split execute_lightweight into three functions:
     - execute_lightweight: router that dispatches to tool or no-tool version
     - execute_lightweight_no_tools: original single-call behavior
     - execute_lightweight_with_tools: new agentic loop with tool support
   - Added execute_routine_tool: isolated tool execution with validation and timeout
   - Uses ToolCompletionRequest/ToolCompletionResponse for tool-aware LLM calls
   - Integrates SafetyLayer for tool output sanitization

3. **src/agent/agent_loop.rs:**
   - Updated RoutineEngine::new call to pass tools and safety

**Tool Execution Loop:**
1. Build initial messages (system + user prompt)
2. Get tool definitions (empty at iteration limit)
3. Call LLM with ToolCompletionRequest
4. If text response: check for ROUTINE_OK sentinel, return result
5. If tool calls: execute sequentially, sanitize, wrap, add to context, loop
6. Safety ceiling at 5 iterations prevents runaway execution

**Approval Handling:** Auto-approves UnlessAutoApproved and Never tools;
blocks Always tools with error message (routines are autonomous by design).

**Testing:** All 2756 tests pass. Zero clippy warnings.

Co-Authored-By: Claude Haiku 4.5 <[email protected]>

* test: add comprehensive unit tests for lightweight routine tool execution

Added 9 new unit tests covering:
- Configuration defaults (lightweight_tools_enabled, lightweight_max_iterations)
- Max iterations capped at 5 (safety ceiling)
- Routine name sanitization (special chars, alphanumeric preservation)
- Sentinel detection for ROUTINE_OK (exact match, contains, whitespace handling)
- Iteration limit safety ceiling enforcement
- Approval requirement pattern matching (Never, UnlessAutoApproved, Always)
- Empty response handling (finish_reason detection)

All 2765 tests pass (11 routine_engine tests, +9 new).

The tests cover the core logic paths of:
- Configuration validation
- Response parsing and sentinel detection
- Name sanitization for workspace paths
- Approval requirement logic
- Iteration limits and safety ceilings

Note: These are unit tests for core logic. Full integration tests with mock LLM
and tool registry would require more complex test infrastructure and are a future enhancement.

Co-Authored-By: Claude Haiku 4.5 <[email protected]>

* style: format routine_engine.rs per cargo fmt

Apply consistent formatting to match Rust style guidelines:
- Break long import lines
- Reformat method chains for readability
- Format multi-line return tuples

No functional changes.

Co-Authored-By: Claude Haiku 4.5 <[email protected]>

* fix: address security and code quality issues in lightweight routine tool execution

**Security Fixes:**

1. Sanitize tool error messages (medium severity)
   - Tool error messages were sent directly to LLM without sanitization
   - Now wrapped through SafetyLayer like successful outputs
   - Prevents leakage of API keys, internal paths, or PII from errors

2. Use unique job_id for each routine run (medium severity)
   - Previously reused routine.id across all executions
   - Caused state collisions and race conditions
   - Now generates unique run_id (Uuid::new_v4()) for each execution
   - Matches behavior of full_job routines

**Code Quality Fixes:**

3. Remove unreachable code
   - Deleted dead if iteration > 5 check
   - max_iterations is capped at 5 via .min(5), so check was impossible
   - Improves code clarity

4. Extract duplicated response handling logic
   - Created handle_text_response() helper function
   - Eliminated 20+ lines of duplicated ROUTINE_OK sentinel detection
   - Reduces maintenance burden and risk of inconsistencies

5. Fix test duplication
   - Tests now call actual super::sanitize_routine_name()
   - Removes duplicate implementation in tests
   - Ensures tests detect changes to original function

**Testing:**
- All 2765 tests pass (no regressions)
- Zero clippy warnings
- Test coverage maintained

Co-Authored-By: Claude Haiku 4.5 <[email protected]>

* fix: address security issue and improve code quality in lightweight routine tool execution

**SECURITY FIX (High Severity):**

1. Block UnlessAutoApproved tools in lightweight routines
   - Previously auto-approved UnlessAutoApproved tools, creating prompt injection vulnerability
   - Lightweight routines can be triggered by external events (channel messages, webhooks)
   - If susceptible to prompt injection, attacker could trick LLM into calling sensitive tools
   - Now blocks both UnlessAutoApproved and Always tools (only Never tools allowed)
   - Only safe approach without requiring tool_permissions allowlist in routine data model
   - Prevents unauthorized file access, network requests, and other sensitive operations

**Code Quality Improvements:**

2. Use ToolError::Timeout for consistent error handling (medium)
   - Changed from std::io::Error to proper ToolError::Timeout variant
   - More idiomatic and consistent with tool execution error handling
   - Makes errors easier to debug and handle uniformly

3. Fix misleading test names and remove tautological tests (medium)
   - Renamed test_routine_config_lightweight_max_iterations_capped_at_five to
     test_routine_config_can_hold_uncapped_max_iterations
   - Clarified comments to explain where capping actually occurs
   - Removed test_iteration_limit_safety_ceiling (tautological: asserts x.min(5) <= 5)
   - Improves test clarity and prevents false sense of coverage

**Testing:**
- 2764 tests passing (1 test removed, no regressions)
- Zero clippy warnings
- Security vulnerability eliminated

Co-Authored-By: Claude Haiku 4.5 <[email protected]>

* style: format routine_engine.rs per cargo fmt

Apply consistent formatting:
- Fix method chain indentation for LLM completion calls
- Reformat error handling closures for readability
- Break long method calls (wrap_for_llm) across multiple lines

No functional changes.

Co-Authored-By: Claude Haiku 4.5 <[email protected]>

* style: apply cargo fmt formatting fixes to routine_engine.rs

Align formatting with project standards:
- Break long method chains across multiple lines for readability
- Reformat error return statements for consistency
- Split long assert/assert_eq statements across multiple lines

No logic changes; purely cosmetic formatting.

Co-Authored-By: Claude Haiku 4.5 <[email protected]>

* test: update routine engine tests for tool/safety layer parameters

Update test code to pass newly required ToolRegistry and SafetyLayer
parameters to RoutineEngine::new(). Also add missing lightweight_tools_enabled
and lightweight_max_iterations fields to RoutineConfig initializers in tests.

Tests affected:
- tests/support/test_rig.rs: Added tools and safety layer to RoutineEngine::new()
- tests/e2e_routine_heartbeat.rs: Added three instances of tools and safety layer construction

All tests pass (2764 tests).

Co-Authored-By: Claude Haiku 4.5 <[email protected]>

---------

Co-authored-by: Claude Haiku 4.5 <[email protected]>
Co-authored-by: Henry Park <[email protected]>
2026-03-09 20:22:10 -07:00
Illia PolosukhinandGitHub a95f5ebb05 Updating feature parity 03/09 (#808) 2026-03-10 02:59:20 +00:00
83950d11a4 fix: job token budget, iteration cap → Failed, web cancel stops worker (#788)
* feat: persist user_id in save_job and expose job_id on routine runs (#709)

* feat: persist worker events to DB and fix activity tab rendering

In-process Worker (used by Scheduler::dispatch_job) now persists events
via save_job_event at key execution points: plan creation, LLM
responses, tool_use, tool_result, and job completion/failure/stuck.
Event data shapes match the container worker format so the gateway
activity tab renders them correctly.

Frontend: tool_result errors now show a red X icon with danger styling
instead of a silent empty output. The result event falls back to the
error field when message is absent.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* feat: wire RoutineEngine into gateway for direct manual trigger firing

Replace the message-channel hack in routines_trigger_handler with a
direct call to RoutineEngine::fire_manual(), ensuring FullJob routines
dispatch correctly when triggered from the web UI. Inject the engine
into GatewayState from Agent::run after construction.

Also persists user_id in save_job for both PG and libSQL backends,
removes the source='sandbox' filter so all jobs are visible, and
exposes job_id on RoutineRunInfo for the frontend job link.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: remove stale gateway_state argument from Agent::new test call sites

The gateway_state parameter was removed from Agent::new during rebase
(replaced by post-construction set_routine_engine_slot), but three test
call sites still passed the extra None argument.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: address PR review — restore sandbox source filter, remove blank lines

- Revert removal of `source = 'sandbox'` filter in all SandboxStore
  queries (8 sites across PG and libSQL). Sandbox-specific APIs should
  stay scoped to sandbox jobs; unified job listing for the Jobs tab
  should use a separate query path.
- Remove extra blank lines in agent_loop.rs and worker.rs that caused
  formatting CI failure.

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: address review — regenerate Cargo.lock, add user_id regression test

- Regenerate Cargo.lock from main's lockfile to eliminate dependency
  version downgrades (anyhow, syn, etc.) that were churn from rebase.
- Add regression test verifying user_id round-trips through save_job
  and get_job in the libSQL backend.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* style: remove trailing blank line in libsql jobs.rs

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* test: add Postgres-side regression test for user_id persistence in save_job

Mirrors the existing libSQL test (test_save_job_persists_user_id) for the
Postgres backend. Gated behind #[cfg(feature = "postgres")] + #[ignore]
since it requires a running PostgreSQL instance (integration tier).

Co-Authored-By: Claude Opus 4.6 <[email protected]>

---------

Co-authored-by: Claude Opus 4.6 <[email protected]>

* fix: add job token budget, change iteration cap to Failed, fix web cancel (#698)

Jobs could enter infinite retry loops because: (1) no token budget was
enforced, (2) iteration cap marked jobs as Stuck (allowing self-repair to
restart them), and (3) the web UI cancel button only updated the DB without
stopping the running worker.

- Add `max_tokens_per_job` config (settings.json + AGENT_MAX_TOKENS_PER_JOB
  env var, default 0 = unlimited) with per-job metadata override
- Track token usage after respond_with_tools() and fail the job on budget
  exceeded
- Change iteration cap and persistent rate limiting from mark_stuck to
  mark_failed, preventing self-repair restart loops
- Fix web cancel handler to call scheduler.stop() which updates in-memory
  state AND aborts the worker task, falling back to DB-only update

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: address PR review — always persist cancel to DB, simplify token check

- Cancel handler now always persists Cancelled to DB regardless of whether
  scheduler.stop() ran, fixing the edge case where stop() returns Ok(())
  for jobs not in the scheduler map
- Collapse nested ifs per clippy (let-chains)
- Add NOTE comment about select_tools() not exposing TokenUsage

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: rustfmt formatting in wizard.rs (pre-existing)

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 <[email protected]>

---------

Co-authored-by: Claude Opus 4.6 <[email protected]>
2026-03-10 02:19:56 +00:00
Nick PismenkovandGitHub 764be8547f fix: fmt (#805) 2026-03-09 19:14:36 -07:00
7de639e782 fix(ci): cherry-pick fmt + clippy for staging PRs [skip-regression-check] (#803)
Cherry-pick of #802: run fmt + clippy on staging PRs, skip Windows clippy,
simplify claude-review trigger to labeled-only.

Co-authored-by: Claude Opus 4.6 <[email protected]>
2026-03-09 18:44:57 -07:00
a5f88b32fd fix(setup): pass NEARAI_API_KEY to provider in model selection step (#799)
When users authenticate via NEAR AI Cloud API key (option 4) during
onboarding, the key is stored as an env var but fetch_nearai_models()
was hardcoding api_key: None. This caused resolve_bearer_token() to
re-trigger the interactive auth prompt at step 4 (model selection).

Co-authored-by: Claude Opus 4.6 <[email protected]>
2026-03-10 01:26:12 +00:00
Nick PismenkovandGitHub 7d8576a464 fix: destructive actions from ambiguous user prompts (#782)
* fix: destructive actions from ambiguous user prompts

* review fixes

* review fixes
2026-03-09 18:03:39 -07:00
f4b7309523 fix(ci): cherry-pick CI cleanup onto staging [skip-regression-check] (#798)
Cherry-pick of #794: remove continue-on-error hack, skip redundant checks
on staging PRs, allow ironclaw-ci[bot] in Claude Code review.

Co-authored-by: Claude Opus 4.6 <[email protected]>
2026-03-09 17:59:59 -07:00
Henry ParkandClaude Sonnet 4.6 577e26eff4 fix(ci): secrets can't be used in step if conditions [skip-regression-check]
GitHub Actions step-level `if:` doesn't have access to `secrets` context.
Replace `if: secrets.X != ''` with `continue-on-error: true` and let
the Set token step handle the fallback.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-03-09 16:41:43 -07:00
bcbdc273a5 Restructure CLAUDE.md into modular rules + add pr-shepherd command (#750)
* refactor: restructure CLAUDE.md into modular rules and add pr-shepherd command

Trim CLAUDE.md from 710 lines to 92 by moving detailed guidance into
path-scoped `.claude/rules/` files that load on demand. Add a new
`/pr-shepherd` command that consolidates the full PR lifecycle
(review, fix, quality gate, CI fix loop, merge) into one workflow.

Changes:
- CLAUDE.md: keep only essentials (build commands, code style, architecture,
  module specs, config reference, debugging)
- .claude/rules/review-discipline.md: 15+ review rules, scoped to src/**/*.rs
- .claude/rules/database.md: dual-backend rules with SQL dialect translation
  table, scoped to src/db/** and migrations/**
- .claude/rules/safety-and-sandbox.md: safety layer and sandbox rules, scoped
  to src/safety/**, src/sandbox/**, src/secrets/**
- .claude/rules/testing.md: test tiers and patterns, scoped to src/** and tests/**
- .claude/rules/tools.md: tool architecture and implementation pattern, scoped
  to src/tools/** and tools-src/**
- .claude/commands/pr-shepherd.md: 7-phase PR lifecycle command that subsumes
  review-pr, respond-pr, ship, and manual CI fix loops

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: address review feedback on CLAUDE.md restructure

- Restore project structure tree in CLAUDE.md (zmanian blocking)
- Create .claude/rules/skills.md with trust model, SKILL.md format,
  selection pipeline, and skill tools (zmanian blocking)
- Restore configuration section with key env vars (zmanian medium)
- Restore "Adding a New Channel" guide (zmanian medium)
- Add heartbeat mention to Workspace & Memory section (zmanian low)
- Fix pr-shepherd: replace `git add -A` with specific file staging (zmanian)
- Fix pr-shepherd: ask user for merge strategy instead of hardcoding --squash (zmanian)
- Fix pr-shepherd: replace `--watch` with polling + 10min timeout (zmanian)
- Fix testing.md: "skipped if DB is unreachable" not "expected to fail" (Copilot)

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: address review comments on PR #750

- Narrow `crate::` import rule: `super::` is fine in tests and intra-module refs
- Fix capabilities file naming: `<name>.capabilities.json` sidecar, not bare `capabilities.json`
- Update mechanical verification checklist to match narrowed import rule

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* refactor: move Bedrock docs from CLAUDE.md to src/llm/CLAUDE.md

Bedrock provider details (auth, config, feature flag) belong in the
LLM module spec, not the top-level guide. Added file map entry,
provider table row, and dedicated section in src/llm/CLAUDE.md.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* refactor: move env var config block out of CLAUDE.md

Replace 20-line config block with one-liner pointing to .env.example
and src/llm/CLAUDE.md. Config details are only needed during deployment,
not everyday coding.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: use gh pr checkout for fork-safe PR checkout in pr-shepherd

Replaces git fetch/checkout with gh pr checkout {number} which
handles both same-repo and fork-based PRs automatically.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: address Copilot review round 5 on PR #750

- Add gh pr list and gh pr checkout to pr-shepherd allowed-tools
- Align crate:: import rule in pr-shepherd with updated CLAUDE.md guidance
- Fix vector type in database.md: BLOB (flexible dims), not F32_BLOB(1536)
- Update MCP limitation: stdio/HTTP/Unix transports exist, no streaming

Co-Authored-By: Claude Opus 4.6 <[email protected]>

---------

Co-authored-by: Claude Opus 4.6 <[email protected]>
2026-03-09 23:19:25 +00:00
Henry ParkandGitHub c541220ea4 feat(ci): chained promotion PRs with multi-agent Claude review (#776)
* feat(ci): chained promotion PRs with multi-agent Claude review [skip-regression-check]

Staging CI workflow with batched promotion PRs:
- Creates staging-promote/<sha> branches per batch
- Chains PRs onto previous promotion branch (incremental diffs)
- Claude Code reviews only the incremental changes per batch
- Blocked PRs stay open as records of findings
- staging-tested tag advances regardless of gate outcome
- Runs every 60 min on cron + manual dispatch

Multi-agent Claude review (Sonnet orchestrator + Haiku agents):
- 4 parallel Sonnet review agents (security, architecture, bugs, performance)
- Haiku agents for severity/confidence scoring
- [SEVERITY:CONFIDENCE] output format
- Severity/confidence matrix for issue creation and gate blocking:
  CRITICAL: always create issue, block if confidence >=80
  HIGH: create issue if confidence >=50
  MEDIUM/LOW: create issue if confidence >=80
2026-03-09 16:17:20 -07:00
14aadd3063 refactor: make src/llm/ self-contained for crate extraction (#767)
* refactor: make src/llm/ self-contained for crate extraction

Move LlmError, LLM config types, and OAuth callback helpers into
src/llm/ so the module has zero `use crate::` imports outside of
crate::llm. This prepares the module for extraction into a standalone
workspace crate.

- Move LlmError enum from src/error.rs to src/llm/error.rs
- Move LlmConfig, NearAiConfig, RegistryProviderConfig, BedrockConfig,
  CacheRetention, OAUTH_PLACEHOLDER from src/config/llm.rs to
  src/llm/config.rs
- Move OAuth callback utilities (callback_url, bind_callback_listener,
  wait_for_callback, landing_html, etc.) from src/cli/oauth_defaults.rs
  to src/llm/oauth_helpers.rs
- Remove session.rs dependency on crate::bootstrap (inline default path)
- Add cache_retention field to RegistryProviderConfig, resolve from env
  in config/llm.rs instead of reading env var in llm/mod.rs
- Add Check 6 to scripts/check-boundaries.sh enforcing LLM isolation
- All original locations re-export for backward compatibility

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* style: fix formatting

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: address PR #767 review — session path bug and boundary check

1. Fix SessionConfig::default() usage in setup wizard: the fallback at
   wizard.rs:995 now constructs SessionConfig with the real
   default_session_path() instead of a relative "session.json", which
   would write auth tokens to the CWD instead of ~/.ironclaw/.

2. Widen check-boundaries.sh Check 6 to catch all `crate::` references
   (not just `use crate::` imports). Pre-existing inline references
   (16 occurrences) are reported as warnings; only new `use crate::`
   imports are hard violations.

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: address PR #767 review and audit findings in src/llm/

PR review fixes:
- Reject wildcard addresses (0.0.0.0, ::) in OAuth callback listener
  to prevent session token exposure on all interfaces
- Fix boundary check comment-stripping that could hide real violations
  (use sed to strip inline comments before matching)

Audit fixes:
- Fix UTF-8 byte-index slicing panic in recording.rs hint extraction
- Add effective_model_name() delegation to RetryProvider and
  SmartRoutingProvider for consistency with other wrappers
- Add calculate_cost() delegation to CachedProvider and RecordingLlm
- Deduplicate retry loop logic in RetryProvider via generic helper
- Replace hardcoded /tmp path in recording tests with tempfile

Co-Authored-By: Claude Opus 4.6 <[email protected]>

---------

Co-authored-by: Claude Opus 4.6 <[email protected]>
2026-03-09 22:31:17 +00:00
45923ef360 feat: add background sandbox reaper for orphaned Docker containers (#634)
* feat: add background sandbox reaper for orphaned Docker containers

* add tests

* review fixes

* linter fix

* review fixes

* style: format test assertion in reaper

Apply rustfmt to improve code formatting consistency.

Co-Authored-By: Claude Haiku 4.5 <[email protected]>

* fix: revert assertion to single-line format for CI compatibility

The assertion should remain on a single line to match CI's
rustfmt expectations.

Co-Authored-By: Claude Haiku 4.5 <[email protected]>

* fix: format assertion to multi-line for CI rustfmt

Use multi-line format for the assert macro to comply with
CI's rustfmt line length limit (100 chars).

Co-Authored-By: Claude Haiku 4.5 <[email protected]>

---------

Co-authored-by: Claude Haiku 4.5 <[email protected]>
2026-03-09 12:46:04 -07:00
fcb152e408 feat(wasm): lazy schema injection on WASM tool errors (#638)
* feat(wasm): lazy schema injection on WASM tool errors

When a WASM tool returns an error (ToolReturnedError), call the module's
description() and schema() WIT exports and append them as a hint in the
error message. This lets the LLM retry with correct parameters without
us including large schemas in every request's tools array.

- Change ToolReturnedError from tuple to struct variant with hint field
- Add build_tool_hint() that calls WASM description()/schema() exports
- Cap description at 500 chars, schema at 3000 chars to limit context
- Hint flows automatically through Display → ToolError → ChatMessage

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* fix: use floor_char_boundary for UTF-8 safe truncation in tool hints

Use existing crate::util::floor_char_boundary() to avoid panicking
when truncation lands mid-multibyte character. Addresses review
feedback on PR #638.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>
2026-03-09 11:27:46 -07:00
e86b372fa6 fix: prevent irreversible context loss when compaction archive write fails (#754)
* fix(compaction): preserve turns when archival write fails

* style: cargo fmt

Co-Authored-By: Claude Opus 4.6 <[email protected]>

---------

Co-authored-by: Zaki <[email protected]>
Co-authored-by: Claude Opus 4.6 <[email protected]>
2026-03-09 10:13:13 -07:00
Nick PismenkovandGitHub 63f140d391 fix: button styles (#637) 2026-03-09 09:58:33 -07:00
ab0a2e05de fix(mcp): JSON-RPC spec compliance — flexible id, correct notification format (#685)
* fix(mcp): JSON-RPC spec compliance — flexible id, correct notification format

- McpRequest.id is now Option<u64> with skip_serializing_if, so
  notifications omit the id field as required by JSON-RPC 2.0 spec.
  Previously sent id: 0 which violates the spec.

- McpResponse.id uses flexible deserialization that accepts number,
  string, or null — fixes interop with non-standard MCP servers that
  return string ids or missing id fields on error responses.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* Fix review feedback: remove serde(default) from McpResponse.id, fix test assertions

- Remove #[serde(default)] from McpResponse.id so notifications (no id field)
  don't incorrectly parse as responses — prevents DoS/spoofing via SSE
- Update test assertions to use Some(value) after id became Option<u64>

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: update new transport files for Option<u64> id after rebase

Upstream #721 added stdio/unix/transport modules that use McpRequest.id
and McpResponse.id as u64. After our rebase (which changes id to
Option<u64>), these need .unwrap_or(0) for HashMap keys and Some()
wrapping in tests.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* test: add regression tests for JSON-RPC spec compliance

Tests for notification serialization without id field,
flexible id deserialization (string, null, non-numeric).

Co-Authored-By: Claude Opus 4.6 <[email protected]>

---------

Co-authored-by: Claude Opus 4.6 <[email protected]>
2026-03-09 08:37:42 -07:00
ReidandGitHub 290d925c7f fix: preserve tool-call history across thread hydration (#568) (#670)
Prevent model re-attempts and data inconsistencies when rebuilding
  conversation context from persisted tool-call records.

  - Remove raw tool parameters from persisted tool_calls JSON to prevent
    unredacted sensitive data from being stored in the database. The LLM
    context rebuild only needs call_id + name + result.
  - Make record_tool_error/record_tool_result mutually exclusive in all
    three execution paths (dispatcher, approval, deferred). Previously
    error cases called both methods, violating the TurnToolCall invariant
    and sending contradictory outcomes to the LLM.
  - Unify call_id format to turn{N}_{i} between live sessions and
    persisted hydration to eliminate ID mismatch in the LLM context.
  - Auto-close </tool_output> XML tags after truncate_preview truncation
    to prevent malformed tool output reaching the LLM.

  [skip-regression-check]
2026-03-09 08:36:07 -07:00
d73e35cfb0 feat: add AWS Bedrock LLM provider via native Converse API (#713)
* feat: add AWS Bedrock LLM provider via native Converse API

* fix: use JSON parsing for tool result error detection instead of brittle substring matching

* refactor: extract duplicated inference config builder into helper function

* fix: address review feedback — safe casts, input validation, and tests

- Safe u32→i32 cast for max_tokens using try_from with clamp
- Remove brittle string-based error detection fallback for tool results
- Validate BEDROCK_CROSS_REGION against allowed values (us/eu/apac/global)
- Validate message list is non-empty before Converse API call
- Log when using default us-east-1 region
- Update llm_backend doc comment to list all backends
- Add tests for build_inference_config and empty message handling

* fix: persist AWS_PROFILE for Bedrock named profile auth

The wizard collected the profile name but only printed a hint to set
it manually. Now it saves to settings and writes AWS_PROFILE to the
bootstrap .env, consistent with how BEDROCK_REGION and other Bedrock
settings are persisted.

* feat: gate AWS Bedrock behind optional `bedrock` feature flag

The AWS SDK dependencies (aws-config, aws-sdk-bedrockruntime,
aws-smithy-types) require cmake and a C compiler to build aws-lc-sys.
Gate them behind an opt-in `bedrock` feature flag so default builds
are unaffected.

Build with: cargo build --features bedrock
All config, settings, and wizard code stays unconditional (no AWS deps)
so users can configure Bedrock even without the feature compiled — they
get a clear error at startup directing them to rebuild.

* fix: address review feedback and adapt Bedrock provider to registry architecture (takeover #345)

- Resolve merge conflicts with main's registry-based provider system
- Add missing cache_creation_input_tokens/cache_read_input_tokens fields
- Add missing content_parts field in test ChatMessage
- Fix string literal type mismatches in wizard env_vars (.to_string())
- Remove non-functional bearer token auth (AWS_BEARER_TOKEN_BEDROCK) from
  wizard and documentation per reviewer feedback from @zmanian and @serrrfirat
- Remove stale BEDROCK_ACCESS_KEY proxy entry from provider table
- Update Bedrock provider to use is_bedrock string check (LlmBackend enum removed)
- Add bedrock_profile fallback from settings in config resolution

[skip-regression-check]

Co-Authored-By: cgorski <[email protected]>
Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: use main's Cargo.lock as base to preserve dependency versions

Regenerating Cargo.lock from scratch caused transitive dependency version
drift that broke the html_to_markdown fixture test in CI.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: bedrock config bugs — spurious warning, alias normalization, profile fallback

- Move is_bedrock check before unknown-backend warning to prevent
  spurious "unknown backend" log for bedrock users
- Normalize backend aliases ("aws", "aws_bedrock") to "bedrock" so
  the provider factory matches correctly
- Add settings.bedrock_profile fallback for AWS_PROFILE, consistent
  with region and cross_region resolution

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: address Copilot review feedback — bearer token cleanup, stop_sequences, model dedup

- Remove stale bearer token refs from setup README and CHANGELOG
- Remove dead bedrock_api_key secret injection mapping
- Pass stop_sequences through to Bedrock InferenceConfiguration
- Remove "API key" from wizard menu description (bearer token removed)
- Skip duplicate LLM_MODEL write for bedrock backend in wizard
- Fix cargo fmt formatting

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: address review feedback — async new(), remove LiteLLM entry, wizard fixes

- Remove dead LiteLLM-based bedrock entry from providers.json (native
  Converse API intercepts before registry lookup)
- Make BedrockProvider::new() async to avoid block_in_place panic in
  current_thread runtimes; propagate async to create_llm_provider,
  build_provider_chain, and init_llm
- Document CMake build prerequisite in docs/LLM_PROVIDERS.md
- Clear bedrock_profile when user selects "default credentials" in wizard
- Fix selected_model clearing to match established pattern (conditional
  on provider switch, not unconditional)
- Add regression tests for bedrock model preservation and profile clearing

Addresses review feedback from @zmanian on PR #713.
Streaming support tracked in #741.

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: address remaining review comments — CLAUDE.md backends, wizard UX

- Add `bedrock` to CLAUDE.md inline backend list (#10)
- Skip full setup re-run when keeping existing Bedrock config (#11)
- Clear stale bedrock_profile on empty named-profile input (#12)
- Add regression test for empty profile clearing

Co-Authored-By: Claude Opus 4.6 <[email protected]>

---------

Co-authored-by: Chris Gorski <[email protected]>
Co-authored-by: cgorski <[email protected]>
Co-authored-by: Claude Opus 4.6 <[email protected]>
2026-03-09 07:10:25 +00:00
30d81fcdee docs: add simplified Chinese (zh-CN) README translation (#488)
Add README.zh-CN.md with full simplified Chinese translation of the
README, and add language switcher links to the original README.

Co-authored-by: smartchoice <[email protected]>
Co-authored-by: Claude Opus 4.6 <[email protected]>
2026-03-09 07:06:14 +00:00
d8dcc34319 fix: CLI commands ignore runtime DATABASE_BACKEND when both features compiled (#740)
* fix: CLI commands ignore runtime DATABASE_BACKEND when both features compiled

`tool auth` and `mcp` CLI subcommands used compile-time `#[cfg]` gates
to select the database backend for secrets storage. When the binary is
compiled with both `postgres` and `libsql` features, the `#[cfg(feature
= "postgres")]` block always wins regardless of the runtime
`DATABASE_BACKEND` setting. This causes `tool auth` and `mcp auth` to
fail with a connection error for users running the libsql backend.

Switch both functions to `match config.database.backend { ... }` with
inner `#[cfg]` guards on each arm, matching the pattern already used in
`main.rs` and `app.rs`.

Also adds a top-level `auth` section to the Telegram channel
capabilities file so `ironclaw tool auth telegram` works for channels
(previously only tools had this section).

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: extract create_secrets_store factory into src/db, bump telegram version

- Move duplicated DB backend selection logic from cli/tool.rs and
  cli/mcp.rs into a shared db::create_secrets_store() factory, following
  the existing db::connect_from_config() pattern.
- Bump telegram channel version 0.2.0 → 0.2.1 to fix CI Version Bump Check.
- Add regression test for create_secrets_store with libsql backend.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: address review feedback — wizard.rs pattern, formatting, version bump

- Convert setup/wizard.rs secrets store creation from tri-branch #[cfg]
  to runtime match on selected_backend (same pattern as CLI fix).
- Fix formatting (assert! line wrapping caught by CI).
- Bump telegram version to 0.2.2 (main already has 0.2.1).
- Merge latest main.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* chore: fix regression test doc comment formatting

Co-Authored-By: Claude Opus 4.6 <[email protected]>

[skip-regression-check]

* fix: address Copilot review — wizard default backend, error chain preservation

- Fix wizard.rs: default selected_backend to "libsql" in libsql-only
  builds so create_libsql_secrets_store is not skipped.
- Preserve error chain: replace .map_err(|e| anyhow!("{}", e)) with ?
  in cli/tool.rs and cli/mcp.rs since DatabaseError implements
  std::error::Error.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

---------

Co-authored-by: Tiny Tim <[email protected]>
Co-authored-by: Claude Opus 4.6 <[email protected]>
Co-authored-by: firat.sertgoz <[email protected]>
2026-03-09 07:01:22 +00:00
652f30a826 fix(web): prevent fetch error when hostname is an IP address in TEE check (#672)
Currently, teeApiBase() splits the hostname by '.' and incorrectly parses IP addresses like 127.0.0.1 or localhost into invalid URLs (e.g., http://api.0.0.1/), which causes the fetch API to throw a 'Failed to construct Request' TypeError and crashes the web UI.

This fix:
- Skips TEE checks if the hostname is an IP address or localhost.
- Wraps checkTeeStatus() and fetchTeeReport() with try...catch to gracefully handle any unforeseen fetch errors without bubbling up to the global scope.

Co-authored-by: lighterEB <[email protected]>
2026-03-09 03:50:07 +00:00
Protocol ZeroandGitHub 98e9a40762 test(job): cover job tool validation and state transitions (#681)
Add focused coverage for create/list/status/cancel job tools so validation errors, summary formatting, and cancellation behavior stay stable. This locks in the current user-facing responses for running and completed jobs without changing production code.

Made-with: Cursor
2026-03-09 03:49:58 +00:00
553c306c52 feat: full image support across all channels (#725)
* feat: full image support across all channels

End-to-end image handling: upload, generation, analysis, editing, and
rendering across web gateway, HTTP webhook, WASM (Telegram/Slack), and
REPL channels. Builds on the attachment infrastructure from #596 and
draws inspiration from PR #641's image pipeline approach — credit to
that PR's author for the sentinel JSON pattern and base64-in-JSON
upload design.

Key changes:
- Image upload in web UI (file picker, paste, preview strip)
- Image generation tool (FLUX/DALL-E via /v1/images/generations)
- Image edit tool (multipart /v1/images/edits with fallback)
- Image analysis tool (vision model for workspace images)
- Model detection utilities (image_models.rs, vision_models.rs)
- Sentinel JSON detection in dispatcher for generated image rendering
- StatusUpdate::ImageGenerated → SSE/WS/REPL/WASM broadcast
- HTTP webhook attachment support (base64, 5MB/file, 10MB total)
- WASM channel image download (Telegram via file API, Slack via host HTTP)
- Tool registration wiring in app.rs

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: address PR #725 review comments (16 issues)

- SecretString for API keys in all image tools (image_gen, image_edit, image_analyze)
- Binary image read via tokio::fs::read instead of DB-backed workspace.read()
- Replace Arc<Workspace> with Option<PathBuf> base_dir (workspace has no filesystem API)
- ApprovalRequirement::UnlessAutoApproved for cost-sensitive image tools
- Scope sentinel detection to image_generate/image_edit tool names only
- Skip ToolResult preview broadcast for image sentinels (avoids multi-MB base64 in SSE)
- Extract shared media_type_from_path() to builtin/mod.rs
- Rename fallback_chat_edit → fallback_generate with tracing::warn
- Increase gateway body limit from 1MB to 10MB for image uploads
- Increase webhook body limit to 15MB (base64 overhead)
- Log warning on invalid base64 in images_to_attachments
- Client-side image size limits (5MB/file, 5 images max) in app.js
- aria-label on attach button for accessibility
- Update body_too_large test for new 10MB limit

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: add Slack file size check before download (PR review item #15)

Skip downloading files larger than 20 MB in the Slack WASM channel to
avoid excessive memory use and slow downloads in the WASM runtime.
Logs a warning when a file is skipped. Also bumps channel versions
for Slack and Telegram (prior branch changes).

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* style: cargo fmt

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix(security): add path validation and approval requirement to image tools

Add sandbox path validation via validate_path() to both ImageAnalyzeTool
and ImageEditTool to prevent path traversal attacks that could exfiltrate
arbitrary files through external vision/edit APIs. Also fix
ImageAnalyzeTool::requires_approval to return UnlessAutoApproved,
consistent with ImageEditTool and ImageGenerateTool.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: post-download size guards and empty data_url sentinel check

- Slack: add post-download size check on actual bytes when metadata
  size_bytes is absent, preventing bypass of the 20MB limit
- Telegram: add 20MB download size limit (matching Slack) enforced
  in download_telegram_file() after receiving response bytes
- Dispatcher: skip broadcasting ImageGenerated SSE event when
  data_url is empty from unwrap_or_default(), log warning instead

Closes correctness issues #3, #4, #5 from PR #725 review.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: use mime_guess for media type detection, add alt attrs and media_type validation

- Replace hardcoded media type mapping with mime_guess crate (already in deps)
- Add alt attributes to img elements in web UI for accessibility
- Validate media_type starts with "image/" in images_to_attachments()
- Update bmp test assertion to match mime_guess behavior

Co-Authored-By: Claude Opus 4.6 <[email protected]>

---------

Co-authored-by: Claude Opus 4.6 <[email protected]>
Co-authored-by: Zaki <[email protected]>
2026-03-09 03:41:27 +00:00
7fb2f47999 feat(skills): exclude_keywords veto in skill activation scoring (#688)
* feat(skills): exclude_keywords veto in skill activation scoring

Add exclude_keywords field to ActivationCriteria. If any exclude
keyword is present in the user message, the skill scores 0 regardless
of keyword or pattern matches — prevents cross-skill interference.

Behaviour: exclude_keywords is a hard veto. Even an exact skill name
match gets vetoed if an exclude keyword is also present. This is
intentional; partial exclusion (score reduction) would create
unpredictable interference behaviour.

Example use case: a writing skill with keywords ["write", "draft"]
and exclude_keywords ["route", "redirect"] will not activate on
messages like "don't route this to the writing agent".

Changes:
- ActivationCriteria: new exclude_keywords field (serde default)
- LoadedSkill: new lowercased_exclude_keywords (preprocessed at load)
- selector.rs: early-return 0 in score_skill() on veto match
- registry.rs: populate lowercased_exclude_keywords during loading
- Test helpers updated across mod.rs, selector.rs, attenuation.rs

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* Fix review feedback: enforce limits on exclude_keywords, extract helper, use any()

- Add exclude_keywords to enforce_limits() with same min-length and cap
  rules as keywords — prevents empty string always-match and unbounded lists
- Extract to_lowercase_vec() helper to deduplicate three identical blocks
- Use idiomatic any() iterator instead of for loop in score_skill veto check

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* test(skills): add exclude_keywords veto tests

Adds 4 tests for the exclude_keywords veto behavior as requested in review:

1. test_exclude_keyword_vetos_match — skill scores 0 when exclude keyword present
2. test_exclude_keyword_absent_does_not_block — skill activates normally without it
3. test_exclude_keyword_veto_wins_over_positive_match — veto wins even with multiple keyword hits
4. test_exclude_keyword_case_insensitive — veto fires regardless of message case

Also adds make_skill_with_excludes() test helper to avoid repeating the
LoadedSkill construction boilerplate in each test.

Note on substring matching: exclude_keywords uses message_lower.contains(excl)
(substring match), consistent with the existing positive keyword scoring path.
This means "red" would veto "redirect". This is documented behaviour — if
word-boundary semantics are needed, that's a follow-up change.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

* style: run cargo fmt on selector.rs

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

---------

Co-authored-by: Claude Opus 4.6 <[email protected]>
2026-03-08 20:23:41 -07:00
02f85a8ad5 feat(mcp): transport abstraction, stdio/UDS transports, and OAuth fixes (#721)
* feat(mcp): transport abstraction, stdio/UDS transports, and OAuth fixes

Extract McpTransport trait from HTTP-coupled McpClient, enabling pluggable
transport backends. Implements stdio and Unix domain socket transports for
local MCP server integration, fixes OAuth discovery per RFC 9728, and adds
SSRF protection.

Transport abstraction (Step 2):
- McpTransport trait with send(), shutdown(), supports_http_features()
- HttpMcpTransport extracted from McpClient with SSE parsing, session tracking
- Shared JSON-RPC framing helpers (write_jsonrpc_line, spawn_jsonrpc_reader)
- McpClient refactored to hold Arc<dyn McpTransport>

Stdio transport (#652, Step 4):
- StdioMcpTransport spawns child process, communicates via stdin/stdout
- McpProcessManager for lifecycle management with exponential backoff restart
- Background stderr drain task for debug logging

Unix domain socket transport (#134, Step 5):
- UnixMcpTransport connects to existing Unix sockets
- Reuses shared JSON-RPC framing from transport.rs

HTML error body sanitization (#263, Step 1):
- sanitize_error_body() detects HTML, strips control chars, truncates to 500

Custom headers (#639, Step 3):
- headers field on McpServerConfig, merged into every HTTP request
- --header CLI arg for `mcp add`

Config and CLI updates (Step 6):
- McpTransportConfig tagged enum (Http/Stdio/Unix) with serde support
- EffectiveTransport for zero-copy config dispatch
- CLI: --transport, --command, --arg, --env, --socket flags for `mcp add`
- `mcp list` shows transport type

OAuth fixes (#299, Step 8):
- Multi-strategy discovery (401-based, RFC 9728, direct)
- RFC 8707 resource parameter in auth and refresh flows
- SSRF protection with IPv4-mapped IPv6 bypass detection
- Well-known URI construction per RFC 8414

Closes #652, #134, #639, #263, #299

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix(mcp): address audit findings from crate review

- Fix SSRF bypass: make validate_url_safe async with DNS resolution to
  block hostnames that resolve to private/link-local IPs
- Fix UTF-8 truncation: use char-based truncation in sanitize_error_body
  to avoid panicking on multi-byte characters
- Fix SSE parser: process only complete lines to handle chunks split
  across boundaries, add 10MB buffer size limit
- Add debug_assert for transport type mismatch in new_with_config
- Propagate custom headers in new_with_transport constructor
- Deduplicate effective_transport() calls in CLI list command
- Gate test-only accessors with #[cfg(test)] to eliminate dead_code warnings
- Document JSON-RPC notification id:0 limitation in protocol.rs
- Document total backoff wait time (31s) in process.rs
- Add regression test for multi-byte UTF-8 truncation

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix(mcp): address PR review findings from Copilot, Gemini, and zmanian

Moderate/High fixes:
- Plumb custom headers through new_authenticated constructor
- Restrict HTTP to localhost only in validate_url_safe (prevent
  plaintext credential leaks over non-localhost HTTP)
- Add mcp_process_manager.shutdown_all() to app shutdown path to
  prevent orphaning stdio child processes
- Validate discovered authorization_url before opening browser
  (prevent malicious MCP server redirecting to phishing page)

Medium fixes:
- Upgrade debug_assert to assert in new_with_config (fires in release)
- Remove pending map entry on Ok(Err(_)) in stdio/unix send() to avoid
  stale entries and unnecessary 30s waits
- Shut down old transport in try_restart() before spawning replacement
- Redact env var values in mcp list --verbose (may contain secrets)
- Drain pending requests on shutdown to wake waiters immediately
- Add IPv6 link-local, site-local, unique-local, and documentation
  ranges to is_dangerous_ip SSRF protection

Low fixes:
- Truncate logged JSON parse error lines to 200 chars (prevent
  sensitive data in logs)
- Remove misleading shutdown comment in unix_transport
- Use tempfile::tempdir() instead of hardcoded /tmp/ path in test
- Adopt main's improved sanitize_error_body (HTML tag stripping,
  200-char truncation with char_indices)

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix(mcp): gate unix_transport with #[cfg(unix)] for Windows compat

- Add #[cfg(unix)] to unix_transport module declaration
- Add #[cfg(unix)]/#[cfg(not(unix))] branches in app.rs for Unix
  socket MCP server setup
- Remove unused sanitize_error_body import in client.rs tests

[skip-regression-check]

---------

Co-authored-by: Claude Opus 4.6 <[email protected]>
2026-03-09 02:47:42 +00:00
FrankandGitHub 9401ab0d58 fix: add timezone conversion support to time tool (#687) 2026-03-08 21:17:07 +00:00
7d1461fc74 fix: standardize libSQL timestamps as RFC 3339 UTC (#683)
* fix: standardize libsql timestamps

* style: fix formatting in libsql/mod.rs

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 <[email protected]>

---------

Co-authored-by: Zaki <[email protected]>
Co-authored-by: Claude Opus 4.6 <[email protected]>
2026-03-08 21:16:46 +00:00
605a4ba46e fix(docker): bind postgres to localhost only (#686)
5432:5432 → 127.0.0.1:5432:5432 — the default docker-compose.yml
exposed postgres on all interfaces, making it reachable from the
local network in any docker compose deployment.

Co-authored-by: Claude Opus 4.6 <[email protected]>
Co-authored-by: Zaki Manian <[email protected]>
2026-03-08 20:55:57 +00:00
fe91ba2ab4 fix(repl): skip /quit on EOF when stdin is not a TTY (#724)
When running as a launchd/systemd daemon, stdin is /dev/null.
rustyline reads EOF immediately and the REPL thread was sending
a /quit message, causing the agent to shut down right after
startup — making service mode non-functional on both macOS and Linux.

Fix: check std::io::stdin().is_terminal() before sending /quit on
EOF. In daemon mode (no TTY) the REPL thread exits silently, leaving
other channels (gateway, telegram, …) running as expected.

Fixes #723

Co-authored-by: Claude Sonnet 4.6 <[email protected]>
Co-authored-by: Zaki Manian <[email protected]>
2026-03-08 20:40:56 +00:00
da2569bb77 fix(web): prevent Enter key from sending message during IME composition (#715)
Co-authored-by: Zaki Manian <[email protected]>
2026-03-08 20:40:31 +00:00
732b3ecfeb test(agent): wire TestRig job tools through the scheduler (#716)
Align TestRig with the production agent wiring so create_job exercises the real scheduler path instead of silently falling back to an unscheduled context-only job. Tighten the e2e assertion to lock in the in-progress scheduler behavior for future refactors.

Made-with: Cursor

Co-authored-by: Zaki Manian <[email protected]>
2026-03-08 20:40:03 +00:00
461d7712e8 fix(config): init_secrets no longer overwrites entire config (#726)
* fix(config): init_secrets no longer overwrites entire config

init_secrets() was calling Config::from_db_with_toml() to re-resolve
config after injecting credentials. This rebuilt the entire config from
env/DB/defaults, nuking all other config fields (agent, safety, tools,
etc.) even though only LlmConfig depends on injected credentials.

This caused 5 CI test failures: the test rig's carefully chosen config
values (max_tool_iterations, allow_local_tools, etc.) were silently
overwritten with production defaults after secret injection.

Fix: add Config::re_resolve_llm() that re-resolves only the LLM config
after credential injection, leaving all other config fields untouched.
Also fix TraceLlm::complete() to skip ToolCalls steps when called in
force_text mode (iteration limit).

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix(test): update test to match TraceLlm::complete() skip-tool-calls behavior [skip-regression-check]

TraceLlm::complete() now skips ToolCalls steps (force_text mode) instead
of erroring. Update the test to verify it skips past a ToolCalls step and
returns the subsequent Text step.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

---------

Co-authored-by: Claude Opus 4.6 <[email protected]>
Co-authored-by: Zaki <[email protected]>
2026-03-08 13:32:42 -07:00
ReidandGitHub 1c5117eded feat: add PID-based gateway lock to prevent multiple instances (#717) 2026-03-08 13:17:46 -07:00
ReidandGitHub 33b02eabb7 fix(cli): status command ignores config.toml and settings.json (#354) (#734) 2026-03-08 13:17:43 -07:00
ReidandGitHub 068ad2d4b7 Fix single-message mode to exit after one turn when background channels are enabled (#719) 2026-03-08 12:54:18 -07:00
56b7218897 fix(setup): preserve model name when re-running onboarding with same provider (#600) (#694)
Each provider setup function unconditionally cleared selected_model,
so re-running the wizard with "Keep current provider? Yes" would lose
the model name, forcing the user to re-select it every time.

Now only clears selected_model when the backend actually changes
(old model may be invalid for the new provider). When keeping the
same provider, the model is preserved and Step 4 shows the
"Keep current model" prompt.

Co-authored-by: Claude Opus 4.6 <[email protected]>
2026-03-08 08:32:02 +00:00
200aed16cd feat: configurable LLM request timeout via LLM_REQUEST_TIMEOUT_SECS (#615) (#630)
Add LLM_REQUEST_TIMEOUT_SECS env var (default: 120) to configure the
HTTP request timeout for LLM API calls. Primarily useful for local
models (Ollama, vLLM, LM Studio) that need more time for prompt
evaluation on consumer hardware.

The timeout is applied to the NearAI provider's HTTP client. Other
providers (Anthropic, OpenAI) use rig-core's default client.

- Add request_timeout_secs field to LlmConfig
- Thread timeout through create_llm_provider -> NearAiChatProvider
- Add NearAiChatProvider::new_with_timeout constructor
- Add .env.example documentation
- 2 regression tests for default and custom timeout values

Co-authored-by: Claude Opus 4.6 <[email protected]>
2026-03-08 08:30:52 +00:00
4c0275bcdc fix(setup): initialize secrets crypto for env-var security option (#666) (#706)
The "Environment variable" option in the setup wizard's security step
generated a master key but never initialized `secrets_crypto`, causing
subsequent API key saves to fail silently. Fix by:

1. Creating SecretsCrypto from the generated key (matching keychain path)
2. Storing the key hex in settings for write_bootstrap_env to persist
3. Auto-writing SECRETS_MASTER_KEY to ~/.ironclaw/.env
4. Using inject_single_var for thread-safe env overlay
5. Fixing misleading message (shell profiles don't work, only .env)

Co-authored-by: Claude Opus 4.6 <[email protected]>
2026-03-08 08:30:02 +00:00
272d31797e chore: remove dead code (#648) (#703)
* chore: remove dead code (LlmEvaluator, chunk_by_paragraphs, bundled channel installer, Reasoning::safety)

Delete unused code flagged in #648:
- evaluation/success.rs: delete LlmEvaluator struct/impl, remove #[allow(dead_code)] from RuleBasedEvaluator methods
- workspace/chunker.rs: delete chunk_by_paragraphs() and its tests (zero production callers)
- extensions/manager.rs: delete install_bundled_channel_from_artifacts() (hot-activation never shipped)
- llm/reasoning.rs: remove unused safety field from Reasoning struct; cascade removal through ContextCompactor, HeartbeatRunner, LlmSoftwareBuilder, and all callers

Closes #648

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: move RuleBasedEvaluator into test module to fix dead_code warning

RuleBasedEvaluator has no production callers -- it was only used in
tests of itself. Moving it into #[cfg(test)] eliminates the clippy
dead_code error that broke CI.

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 <[email protected]>

---------

Co-authored-by: Claude Opus 4.6 <[email protected]>
2026-03-08 08:26:04 +00:00
edff54b0b1 fix: persist /model selection across restarts (#707)
* fix: persist /model selection across restarts

The /model command called set_model() on the LLM provider but never
saved the choice to settings, so the model reverted on restart. Now
persists to both the DB settings store and config.toml.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: address CI clippy lint and use spawn_blocking for TOML I/O

- Use struct init syntax instead of field reassignment in test (clippy)
- Wrap sync filesystem operations in spawn_blocking to avoid blocking
  the tokio executor

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* style: fix rustfmt formatting

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: address review feedback — handle JoinError, remove exists() guard

- Log warning if spawn_blocking task panics/is cancelled (JoinError)
- Remove toml_path.exists() guard; load_toml already returns Ok(None)
  for missing files, so permission errors are no longer silently skipped

Co-Authored-By: Claude Opus 4.6 <[email protected]>

---------

Co-authored-by: Claude Opus 4.6 <[email protected]>
2026-03-08 08:10:46 +00:00
4d61d3eedf fix(routines): resolve message tool channel/target from per-job metadata (#708)
* fix(routines): resolve message tool channel/target from per-job metadata

When a routine's notify.channel is None, the message tool had no way to
resolve channel/target for full-job workers, causing "No target specified"
errors. The previous approach mutated shared global state via
set_message_tool_context(), which also raced with concurrent jobs.

Now the routine's notify config (channel + user) is carried in the job's
metadata JSON, and MessageTool::execute falls back to ctx.metadata when
neither explicit params nor conversation defaults are available. This
eliminates both the None-channel bug and the concurrent-job race.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* style: apply cargo fmt

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix(message): broadcast to all channels when notify.channel is None

Address review feedback:
- Fix stale "see above" comment → "populated below"
- When notify.channel is None, use broadcast_all instead of erroring
  with "No channel specified". This matches NotifyConfig semantics
  where channel=None means "broadcast to all channels"
- Channel resolution is now Option<String>: param → default → metadata → None
- When None, MessageTool uses ChannelManager::broadcast_all(target, response)
  and reports which channels succeeded/failed
- Add regression test for broadcast-all behavior

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: use failed channels in error message, remove redundant comment

Address review feedback:
- Use `failed` vec in error message instead of re-querying channel_names
- Remove redundant orphaned comment block in routine_engine.rs

Co-Authored-By: Claude Opus 4.6 <[email protected]>

---------

Co-authored-by: Claude Opus 4.6 <[email protected]>
2026-03-08 08:04:19 +00:00
df3635d6be feat(timezone): add timezone-aware session context (#671)
* feat(timezone): add timezone-aware session context (#661)

All timestamps were UTC-only, causing daily logs to split at UTC midnight,
cron schedules to fire in UTC, and no quiet hours for heartbeat. This adds
timezone as a per-session property flowing from the client.

Key changes:
- New `src/timezone.rs` module with resolution chain, parsing, and detection
- `IncomingMessage` carries optional timezone from client
- `JobContext.user_timezone` flows timezone to tools
- `next_cron_fire()` accepts timezone for schedule evaluation
- `Trigger::Cron` stores optional timezone (backward-compatible)
- Workspace gains `_tz` variants for daily logs and system prompt
- Heartbeat supports quiet hours (`HEARTBEAT_QUIET_START/END`)
- Web frontend sends `Intl.DateTimeFormat().resolvedOptions().timeZone`
- REPL auto-detects system timezone
- `DEFAULT_TIMEZONE` env var / settings for server-wide default

Storage stays UTC. Conversion happens at display boundaries.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix(timezone): address review feedback on timezone-aware sessions

- Validate quiet hours values (0-23) in HeartbeatConfig::resolve()
- Fall back to settings values when env vars are unset for quiet hours
- Validate IANA timezone strings in routine_create/update with parse_timezone
- Add timezone field to routine_create tool schema
- Allow standalone timezone update on cron routines without changing schedule
- Return path from append_daily_log_tz to avoid TOCTOU race at midnight
- Delegate append_daily_log to append_daily_log_tz(entry, UTC) to avoid drift
- Preserve timezone through approval flow via PendingApproval.user_timezone
- Improve test_today_in_tz to not depend on hardcoded year
- Add 3 regression tests for quiet hours config validation

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* style: fix formatting in routine.rs

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix(timezone): address second round of review feedback

- Remove .claude/scheduled_tasks.lock from repo and add to .gitignore
- Store resolved timezone (not raw message.timezone) in PendingApproval
- Carry forward user_timezone through chained approvals in thread_ops
- Wire quiet_hours_start/end from config to HeartbeatRunner
- Support X-Timezone header as fallback in chat_send_handler

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix(timezone): include user's local time in time tool response

The time tool's "now" operation now returns local_iso and timezone
fields based on ctx.user_timezone, so the LLM can report time in
the user's timezone instead of always UTC.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* style: fix formatting in time.rs

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix(timezone): address Copilot review round 3 — validation, deterministic tests, schema fixes

- Validate DEFAULT_TIMEZONE and HEARTBEAT_TIMEZONE at config load time
- Add timezone field to HeartbeatSettings and config::HeartbeatConfig
- Wire heartbeat timezone from config through agent_loop to HeartbeatRunner
- Add timezone to routine_update tool schema (was accepted but not advertised)
- Error on schedule/timezone update for non-cron routines
- Validate timezone in Trigger::from_db (coerce invalid to None with warning)
- Validate timezone in approval path (thread_ops.rs) before overwriting
- Time tool always includes timezone/local_iso fields (fallback to UTC)
- Make quiet hours tests deterministic using current UTC hour
- Add regression tests for config validation

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 <[email protected]>

---------

Co-authored-by: Claude Opus 4.6 <[email protected]>
2026-03-08 08:01:56 +00:00
ReidandGitHub a20e19ab16 fix: sanitize HTML error bodies from MCP servers to prevent web UI white screen (#263) (#656)
* fix: sanitize HTML error bodies from MCP servers to prevent web UI white screen (#263)

* style: fix cargo fmt formatting in sanitize_error_body tests
2026-03-08 02:53:02 +00:00
3b57d5bec9 chore: add reviewer-feedback guardrails (CLAUDE.md, pre-commit hook, skill) (#665)
* chore: add reviewer-feedback guardrails (CLAUDE.md, pre-commit hook, skill)

Analysis of ~50 PRs from the past week identified 10 recurring themes
in Copilot and Gemini code review comments. This change addresses them
at development time through three layers:

1. CLAUDE.md additions (7 new rules):
   - Transaction safety for multi-step DB operations
   - UTF-8 string safety (no byte-index slicing)
   - Case-insensitive comparisons for paths/media types
   - Decorator/wrapper trait method delegation
   - Sensitive data redaction in logs/SSE
   - tempfile crate for test temporary files
   - Trust boundaries for worker container data

2. Pre-commit hook (scripts/pre-commit-safety.sh):
   Mechanical checks for unsafe byte slicing, case-sensitive
   extension comparisons, hardcoded /tmp paths, unredacted
   tool parameter logging, and non-transactional DB operations.
   Installed via dev-setup.sh alongside existing commit-msg hook.

3. Review checklist skill (skills/review-checklist/SKILL.md):
   Activates on "review"/"merge" keywords. Covers the judgment-based
   items that can't be linted: transaction safety, SSRF validation,
   approval checks, decorator delegation, test quality, and doc accuracy.

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: address PR review feedback on pre-commit-safety.sh

- Cache diff output in variable to avoid ~10 redundant git diff calls (Gemini)
- Add early exit when no .rs files are changed (Gemini)
- Fix header comment: list all 5 checks, not just 4 (Copilot)
- Fix check 2 comment: only mentions file extensions, not media types (Copilot)
- Add resolve_base_ref() with fallback candidates instead of hardcoded
  origin/main for standalone mode (Copilot)
- TX check: use -W (function context) to reduce false positives, honor
  // safety: suppression, print triggering lines (Copilot)

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 <[email protected]>

---------

Co-authored-by: Claude Opus 4.6 <[email protected]>
2026-03-07 21:20:37 +00:00
11c5e25422 feat(setup): Anthropic OAuth onboarding with setup-token support (#384)
* feat(setup): add Anthropic OAuth and Codex OAuth onboarding flows

Add OAuth token authentication as an alternative to API keys during
onboarding for both Anthropic (via `claude login`) and OpenAI/Codex
(via `~/.codex/auth.json`).

Key changes:
- New `AnthropicOAuthProvider` using `Authorization: Bearer` header
  (rig-core hardcodes `x-api-key` which rejects OAuth tokens)
- Wizard auth method selector: "Direct API Key" vs "OAuth Token"
  for both Anthropic and OpenAI providers
- Codex token extraction from `$CODEX_HOME/auth.json` / `~/.codex/auth.json`
- Claude Code sandbox sub-step in Docker setup (checks for credentials)
- Secret injection mappings for `ANTHROPIC_OAUTH_TOKEN` and `CODEX_OAUTH_TOKEN`
- `CODEX_OAUTH_TOKEN` falls back to `OPENAI_API_KEY` (same Bearer auth)

Supersedes #143 which had a broken auth flow (OAuth token sent as
x-api-key → 401). Credit to @bigguybobby for the original approach.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: persist OAuth tokens in bootstrap .env and re-extract at startup

OAuth tokens stored only in the secrets DB were invisible to
Config::from_env() which runs before the DB connects (chicken-and-egg).

Two fixes:
1. write_bootstrap_env() now persists ANTHROPIC_OAUTH_TOKEN and
   CODEX_OAUTH_TOKEN to ~/.ironclaw/.env (same pattern as NEARAI_API_KEY)
2. main.rs re-extracts a fresh token from the OS credential store
   (macOS Keychain / ~/.claude/.credentials.json) before config resolution,
   handling token expiry (8-12h) gracefully

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: persist all LLM credentials in bootstrap .env, not just NEAR AI

All providers had the same chicken-and-egg issue: API keys stored in the
secrets DB were invisible to Config::from_env() which runs before DB
connects. Only NEARAI_API_KEY was written to bootstrap .env.

Now write_bootstrap_env() persists all credential env vars:
NEARAI_API_KEY, ANTHROPIC_API_KEY, ANTHROPIC_OAUTH_TOKEN, OPENAI_API_KEY,
CODEX_OAUTH_TOKEN, LLM_API_KEY, TINFOIL_API_KEY.

Also: setup_api_key_provider() now sets the env var during the wizard
session so write_bootstrap_env() can pick it up.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: address security review findings for OAuth onboarding

- Extract "oauth-placeholder" to named OAUTH_PLACEHOLDER constant shared
  across config and wizard to prevent silent drift
- Document plaintext credential tradeoff in write_bootstrap_env (API keys
  stored with 0o600 permissions, recommend full-disk encryption)
- Add blocking "Press Enter" wait in Anthropic OAuth retry flow so user
  has time to run `claude login` in another terminal
- Add escape hatch from manual OAuth paste back to API key flow (empty
  input switches to setup_api_key_provider)
- Fix Retry-After header: parse u64 seconds into Duration before passing
  to LlmError::RateLimited
- Make config::llm module pub(crate) for constant visibility
- Use .bearer_auth() instead of manual format!("Bearer {}")
- Remove response body from debug log (may contain PII)
- Update Anthropic API version to 2024-10-22

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* security: remove plaintext credentials from bootstrap .env

Credentials (API keys, OAuth tokens) were being written in plaintext to
~/.ironclaw/.env to work around a chicken-and-egg problem: Config::from_env()
runs before the encrypted secrets DB is connected.

Instead of storing secrets on disk, LlmConfig::resolve() now defers
gracefully when credentials are missing — it returns None for the provider
config instead of hard-erroring with MissingRequired. After the DB connects,
AppBuilder::build_all() loads secrets from encrypted storage via
inject_llm_keys_from_secrets() and re-resolves the config.

For Anthropic OAuth tokens (which expire in 8-12h), the secret injection
step also tries the OS credential store (macOS Keychain / Linux
credentials.json) for a fresh token, overriding the potentially stale
copy in the DB.

Changes:
- LlmConfig::resolve(): OpenAI, Anthropic, OpenAI-compatible, and Tinfoil
  all return None instead of MissingRequired when credentials are absent
- write_bootstrap_env(): no longer writes any credential env vars
- inject_llm_keys_from_secrets(): refreshes Anthropic OAuth from OS
  credential store before overlay is finalized
- main.rs: removed OAuth re-extraction hack (no longer needed)

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: load OS credential store tokens even without secrets DB

The OAuth token extraction from macOS Keychain / Linux credentials files
was only running inside inject_llm_keys_from_secrets(), which requires
the encrypted secrets DB. When no master key is configured, init_secrets()
returned early — skipping both DB secret loading AND OS credential store
extraction, leaving the Anthropic OAuth token unavailable.

Split into two paths:
- inject_llm_keys_from_secrets(): loads from encrypted DB + OS stores
- inject_os_credentials(): loads from OS stores only (no DB needed)

init_secrets() now calls inject_os_credentials() and re-resolves config
even in the no-master-key early-return path, so `claude login` tokens
are always available regardless of secrets DB state.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: add anthropic-beta header required for OAuth authentication

Anthropic's api.anthropic.com requires the `anthropic-beta: oauth-2025-04-20`
header to accept OAuth Bearer tokens. Without it, the API returns 401
"OAuth authentication is currently not supported."

Also reverts API version to 2023-06-01 since the OAuth beta flag does
not support the 2024-10-22 version (returns 400 "not a valid version").

This was the same bug that caused PR #143's 401 errors — the beta header
was missing entirely.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: Anthropic and OpenAI model resolution respects selected_model

The Anthropic and OpenAI config resolution ignored settings.selected_model
entirely, only checking the provider-specific env var (ANTHROPIC_MODEL,
OPENAI_MODEL) and falling back to a hardcoded default. This meant the
model chosen during onboarding wizard was silently overridden.

Now follows the same pattern as NearAI and OpenAI-compatible:
env var > settings.selected_model > hardcoded default.

Also deduplicated the Anthropic config construction (two identical
branches for API key vs OAuth now share model/base_url resolution).

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* test: add provider resolution tests for all LLM backends

Covers deferred resolution (no credentials → None instead of error),
credential presence, model selection fallback chain, and OAuth token
routing for Anthropic, OpenAI, Tinfoil, Ollama, and NearAI.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: handle nested tokens.access_token format in Codex auth.json

Codex CLI stores OAuth tokens in a nested format under
tokens.access_token (ChatGPT OAuth flow), not at the top level.
Also adds ENV_MUTEX to Codex token tests for thread safety.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* refactor: remove Codex OAuth onboarding (incompatible with OpenAI API)

Codex CLI OAuth tokens use a different endpoint
(chatgpt.com/backend-api/codex) and the Responses API wire format,
not api.openai.com with Chat Completions. The tokens lack the
model.request scope needed for the platform API, so they can't be
used as drop-in OPENAI_API_KEY replacements.

Removes: extract_codex_oauth_token(), wizard Codex OAuth flow,
CODEX_OAUTH_TOKEN env var support, and related tests.

OpenAI onboarding now uses direct API key only.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* style: fix formatting for CI (cargo fmt)

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: address Gemini review feedback

- Use ? operator for ANTHROPIC_MODEL/BASE_URL env resolution instead of
  .ok().flatten() to propagate ConfigErrors consistently
- Skip Tool messages without tool_call_id with a warning instead of
  using unwrap_or_default() which would send empty string to Anthropic
- Extract credential check into closure to reduce duplication in
  Claude Code sandbox setup

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* refactor(review): address PR review feedback for OAuth onboarding

- Gate ANTHROPIC_OAUTH_TOKEN resolution to Anthropic provider only
  (was needlessly checked for all registry providers)
- Add 3 regression tests for OAuth config resolution:
  - oauth_token sets placeholder api_key
  - real api_key takes priority over oauth
  - non-Anthropic providers don't pick up oauth_token
- Validate OAuth token prefix (sk-ant-oat) in wizard to catch
  accidentally pasted API keys
- Improve error body read handling in AnthropicOAuthProvider
  (was silently swallowing read errors with unwrap_or_default)
- Remove extra blank line in write_bootstrap_env
- Remove stale blank line in RegistryProviderConfig doc comment

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: address PR #384 review comments

Blocker:
- Replace OnceLock<HashMap> with LazyLock<Mutex<HashMap>> for INJECTED_VARS
  so both inject_os_credentials() and inject_llm_keys_from_secrets() merge
  data instead of the second caller silently dropping its entries.

High:
- Add 401 retry with OS credential store re-extraction in
  AnthropicOAuthProvider, recovering from expired OAuth tokens (~8-12h)
  without manual intervention.
- Fix comment in app.rs: ~/.codex/auth.json → ~/.claude/.credentials.json.

Medium:
- Remove unsafe { std::env::set_var } from wizard; use thread-safe
  inject_single_var() overlay instead (safe on multi-threaded Tokio).
- Add post-init validation in AppBuilder: fail early with clear error when
  LLM_BACKEND is set but no credentials were resolved after secret injection.
- Add sk-ant-oat prefix validation in parse_oauth_access_token().
- Only route to AnthropicOAuthProvider when api_key is missing or equals
  OAUTH_PLACEHOLDER (API key takes priority over OAuth token).
- Teach fetch_anthropic_models() to use Bearer auth when only OAuth token
  is available (model listing no longer fails for OAuth-only users).

Low:
- Use optional_env() in wizard credential checks to read from injected
  overlay, not just raw env vars.

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* style: cargo fmt

Co-Authored-By: Claude Opus 4.6 <[email protected]>

---------

Co-authored-by: Claude Opus 4.6 <[email protected]>
Co-authored-by: [email protected] <[email protected]>
2026-03-07 20:59:17 +00:00
ArtemandGitHub 12ba79ffc3 feat(llm): add Google Gemini, AWS Bedrock, io.net, Mistral, Yandex, and Cloudflare WS AI providers (#676)
* feat(llm): add Google Gemini and AWS Bedrock providers

* feat(llm): add io.net, Mistral, Yandex, and Cloudflare WS AI providers
2026-03-07 20:49:26 +00:00
github-actions[bot]GitHubgithub-actions[bot] <github-actions[bot]@users.noreply.github.com>
d3cf637d4a chore: update WASM artifact SHA256 checksums [skip ci] (#631)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2026-03-07 20:06:00 +00:00
b6cf2a6b73 fix: prevent Instant duration overflow on Windows (#657) (#664)
* fix: use checked_sub to prevent Instant duration overflow on Windows (#657)

On Windows, Instant starts from system boot time. Subtracting a duration
longer than uptime (e.g., 1 hour on a freshly booted system) panics with
"overflow when subtracting duration from instant", crashing the tokio
worker thread.

Replace `Instant::now() - Duration` with `Instant::now().checked_sub()`
in cost_guard.rs (production), server.rs and session.rs (tests).

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: use expect() instead of unwrap_or() in test code

Address PR review: unwrap_or(Instant::now()) silently breaks test
semantics when checked_sub returns None. Using expect() ensures tests
fail explicitly with a clear message about insufficient system uptime.

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 <[email protected]>

---------

Co-authored-by: Claude Opus 4.6 <[email protected]>
2026-03-07 20:00:40 +00:00
9851f2a6ae docs: add explanatory comments to coverage workflow (#610)
Add comprehensive documentation at the top of the coverage workflow file
to help developers understand:
- What the coverage workflow does
- How to view coverage reports (Codecov links)
- What coverage files are generated
- Configuration options and requirements

This improves developer experience by making the CI/CD pipeline more
transparent and easier to understand for contributors.

Co-authored-by: enihsago <[email protected]>
2026-03-07 19:56:07 +00:00
Eric ElizesandGitHub 8dc4ca5a98 fix: enable libsql remote + tls features for Turso cloud sync (#587)
The onboard wizard offers Turso cloud sync, but the libsql dependency
is compiled without the `remote` and `tls` features, causing a panic
at runtime when LIBSQL_URL is set:

  "The `tls` feature is disabled, you must provide your own http connector"

This adds the missing features to the libsql dependency.
2026-03-07 19:55:11 +00:00
9f71bd0d44 feat: unified thread model for web gateway (#607)
* feat: unified thread model for web gateway

Every piece of activity (user chat, routine run, heartbeat alert, external
channel message) now lives in its own thread, properly isolated, with
meaningful titles and visual distinction.

Key changes:
- Add `channel` field to ConversationSummary and ThreadInfo so the gateway
  can distinguish thread origins (gateway, telegram, routine, heartbeat).
- Add `list_conversations_all_channels` to Database trait (both postgres
  and libsql) so chat_threads_handler shows cross-channel threads.
- Routine runs get a persistent conversation per routine via
  `get_or_create_routine_conversation`; notifications carry thread_id.
- Heartbeat gets a persistent conversation via
  `get_or_create_heartbeat_conversation`; HeartbeatRunner accepts an
  optional Database store and binds notifications to the thread.
- Fix broadcast() in web gateway to propagate response.thread_id instead
  of hardcoding empty string.
- Fix isCurrentThread(null) returning true (the core notification leak
  bug) — now returns false so events without a thread_id don't leak into
  the active thread.
- Rewrite frontend thread sidebar: meaningful titles with channel-specific
  fallbacks, relative timestamps instead of turn counts, channel badges
  for non-gateway threads, unread notification dots, read-only indicator
  for external channel threads.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: address PR review — TOCTOU races, stale comment, debounce, broadcast warning

- Fix TOCTOU race in get_or_create_routine_conversation (postgres):
  use INSERT ON CONFLICT on new uq_conv_routine unique index + SELECT-back.
- Fix TOCTOU race in get_or_create_heartbeat_conversation (postgres):
  use INSERT ON CONFLICT on new uq_conv_heartbeat unique index + SELECT-back.
- Fix TOCTOU race in get_or_create_routine_conversation (libsql):
  use BEGIN IMMEDIATE transaction to serialize concurrent writers.
- Fix TOCTOU race in get_or_create_heartbeat_conversation (libsql):
  use BEGIN IMMEDIATE transaction to serialize concurrent writers.
- Add V11 migration with partial unique indexes for postgres.
- Add matching unique indexes to libsql schema.
- Update stale comment on isCurrentThread (said "always shown" but logic
  now returns false for missing thread_id).
- Debounce loadThreads() on off-thread SSE events to prevent request storms.
- Log warning in broadcast() when thread_id is None (clients will drop it).

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: sort in-memory thread fallback by updated_at descending

The in-memory thread list fallback (when no DB is available) used
HashMap::values() which has no guaranteed ordering. Sort by
updated_at descending to match the SQL query ordering.

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: retry libsql connect() on transient "unable to open database file"

The cron ticker's background task occasionally fails with "unable to
open database file" when creating a new SQLite connection concurrently
with the main thread. Add retry with exponential backoff (50ms, 100ms,
200ms) to handle transient VFS/locking issues in libsql's local mode.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: use ON CONFLICT with index expressions instead of named constraints

PostgreSQL ON CONFLICT ON CONSTRAINT requires a named table constraint,
but V11 migration creates unique indexes. Switch to the expression form
(ON CONFLICT (columns) WHERE condition) which works with unique indexes.

Also fix dead code in threadTitle() where thread.title was already
checked on the previous line.

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* style: fix rustfmt chain collapse in heartbeat.rs

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: skip broadcast when thread_id is None instead of sending empty

Clients drop SSE events with empty thread_id anyway, so avoid the
unnecessary network traffic by returning early.

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* test: add libsql routine/heartbeat conversation idempotency tests

Add tests proving get_or_create_routine_conversation returns the same
conversation ID across multiple invocations with the same routine_id.
Add debug logging to routine engine to track conversation resolution.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* feat: show "New chat" title for empty threads

- threadTitle() returns "New chat" when turn_count is 0
- Assistant thread label updates dynamically from API data
- Default HTML label changed from "Assistant" to "New chat"
- New threads naturally sort to top via last_activity DESC

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: thread sorting, routine isolation, and UI polish

- Fix libsql timestamp format mismatch causing broken thread sort order.
  SQLite defaults used `datetime('now')` (space-separated) while Rust code
  used RFC3339 (T-separated), breaking string-based ORDER BY. All INSERTs
  now use RFC3339, and queries use `datetime()` to normalize comparison.
- Route manual routine triggers through RoutineEngine.fire_manual() instead
  of injecting as regular chat messages, so routines always run in their
  dedicated conversation thread.
- Add RoutineEngineSlot to GatewayState for gateway<->engine communication.
- Derive routine thread titles from conversation metadata (routine_name)
  instead of showing truncated UUID hashes.
- Make chat_new_thread_handler persist to DB synchronously so loadThreads()
  sees newly created threads immediately.
- Fix enableChatInput() no-op and wrong element ID in disableChatInputReadOnly().
- Fix handlers/chat.rs stale gateway-only query (use list_conversations_all_channels).
- Sort in-memory threads by DateTime before converting to RFC3339 strings.
- Trigger debouncedLoadThreads() on thinking/status SSE events for non-current
  threads so routine/heartbeat threads appear in sidebar promptly.
- Remove "Threads" text from sidebar header.

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: routine history display, orphaned tool_results, duplicate system messages

Three independent fixes with regression tests:

1. Routine conversations now display in the web UI. build_turns_from_db_messages()
   handles standalone assistant messages (no preceding user message) by creating
   turns with empty user_input. Frontend skips empty user bubbles.

2. Worker select_tools and execute_plan paths now push an
   assistant_with_tool_calls message before tool execution, preventing
   sanitize_tool_messages from rewriting tool_results as orphaned user messages.

3. Reasoning::plan() and respond_with_tools() merge system messages from
   context into a single system prompt instead of creating [system, system, ...]
   sequences that strict LLM providers (Qwen) reject.

Also: sidebar padding/spacing improvements, wider thread panel (240px).

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: address PR #607 review — RwLock held across await, missing ownership check, heartbeat config

- Clone Arc<RoutineEngine> out of RwLock before .await in trigger handler
- Add user_id ownership check to fire_manual() with NotAuthorized error
- Wire heartbeat notify_user/notify_channel from config to AgentHeartbeatConfig

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* chore: gitignore trace_*.json files and remove stale traces

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* chore: remove trace JSON files from repo

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: proper HTTP status codes for routine errors, read-only input guard, respond thread_id

- Map RoutineError::NotFound → 404, NotAuthorized → 403, Disabled → 409
- Guard enableChatInput() against re-enabling on read-only threads
- Skip respond() when thread_id is None (matches broadcast() behavior)

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 <[email protected]>

---------

Co-authored-by: Claude Opus 4.6 <[email protected]>
2026-03-07 19:53:43 +00:00
205 changed files with 20292 additions and 4164 deletions
+303
View File
@@ -0,0 +1,303 @@
---
description: Full PR lifecycle — review, fix findings, address comments, quality gate, push, CI fix loop, merge
disable-model-invocation: true
allowed-tools: Bash(gh pr view:*), Bash(gh pr diff:*), Bash(gh pr comment:*), Bash(gh pr merge:*), Bash(gh pr checks:*), Bash(gh pr edit:*), Bash(gh pr list:*), Bash(gh pr checkout:*), Bash(gh api:*), Bash(gh repo view:*), Bash(gh run view:*), Bash(gh run watch:*), Bash(git diff:*), Bash(git log:*), Bash(git fetch:*), Bash(git checkout:*), Bash(git status:*), Bash(git branch:*), Bash(git add:*), Bash(git commit:*), Bash(git push:*), Bash(git merge:*), Bash(git rebase:*), Bash(cargo fmt:*), Bash(cargo clippy:*), Bash(cargo test:*), Bash(cargo check:*), Read, Edit, Write, Grep, Glob, Agent
argument-hint: "<pr-number or url> [--fix] [--merge] [--review-only]"
---
# PR Shepherd
Full PR lifecycle: review → fix → quality gate → push → CI → merge.
Parse `$ARGUMENTS`:
- Extract PR number from bare number or `https://github.com/owner/repo/pull/123` URL.
- Flags: `--fix` (auto-fix without asking), `--merge` (merge when CI green), `--review-only` (stop after review, don't fix).
- If no PR number, detect from current branch: `gh pr list --head $(git branch --show-current) --json number --jq '.[0].number'`
- If still nothing, stop and ask the user.
---
## Phase 1: Situational Awareness
Gather everything in parallel:
**PR metadata:**
```
gh pr view {number} --json number,title,body,author,baseRefName,headRefName,headRefOid,state,isDraft,mergeable,mergeStateStatus,files,additions,deletions,labels,reviewRequests
```
**Diff:**
```
gh pr diff {number}
gh pr diff {number} --name-only
```
**CI status:**
```
gh pr checks {number} --json name,status,conclusion,detailsUrl
```
**Review comments (human + bot):**
```
gh api --paginate repos/{owner}/{repo}/pulls/{number}/comments
gh api --paginate repos/{owner}/{repo}/pulls/{number}/reviews
```
Resolve `{owner}/{repo}`:
```
gh repo view --json owner,name --jq '"\(.owner.login)/\(.name)"'
```
Save `headRefOid` — needed for posting line comments later.
**Assess the situation and print a status card:**
```
PR #{number}: {title}
Author: {author} Base: {base} ← {head}
Size: +{additions} -{deletions} across {file_count} files
CI: {PASS|FAIL|PENDING|NONE} Mergeable: {yes|no|conflict}
Reviews: {N approved, N changes_requested, N comments-only, N bot-only}
Unresolved comments: {N}
Draft: {yes|no}
```
**Decide the mode** based on situation:
- **Has unresolved review comments** → Phase 2a (address comments first, then review remaining)
- **No reviews yet / bot-only reviews** → Phase 2b (full deep review)
- **CI failing, no review issues** → Phase 4 (jump to CI fix)
- **Everything green + approved** → Phase 6 (ready to merge)
---
## Phase 2a: Address Existing Review Comments
For each unresolved review comment or review with CHANGES_REQUESTED:
1. **Read the referenced code** at the file and line mentioned. Never assess without reading.
2. **Classify each comment:**
-**Valid & unresolved** — needs a code fix
-**Already fixed** — a later commit addressed it
-**False positive** — explain why the code is correct
- 🔧 **Nit** — optional improvement, not blocking
3. **Deduplicate** — bots (Copilot, Gemini) often post the same finding. Group by actual issue.
Present a table:
| # | Source | File:Line | Issue | Status | Planned Fix |
|---|--------|-----------|-------|--------|-------------|
Wait for user confirmation (unless `--fix` flag set), then proceed to Phase 3.
---
## Phase 2b: Deep Review (6 Lenses)
Read EVERY changed file in full (not just diff hunks). For PRs touching >20 files, prioritize: service logic > handlers > types > tests > docs. Batch reads in parallel via Agent tool.
### IronClaw-specific checks (always)
- No `.unwrap()` or `.expect()` in production code
- Prefer `crate::` for cross-module imports (`super::` OK in tests/intra-module)
- Error types use `thiserror`
- If persistence touched, both backends updated (postgres.rs AND libsql/)
- New tools implement `Tool` trait correctly and registered
- External tool output passes through safety layer
- Tool parameters redacted before logging/SSE
- No byte-index slicing on external strings
- Case-insensitive comparisons where needed
### Correctness
Off-by-one, wrong operators, inverted conditions, unreachable code, type confusion, error propagation, broken invariants, TOCTOU races.
### Edge cases & failure handling
Empty/None/zero-length input, external service failures, integer boundaries, malformed/adversarial input, partial failure handling.
### Security (assume adversarial actors)
Auth/authz bypass, IDOR, injection (SQL/command/log/header), data leakage in logs/errors/API responses, resource exhaustion, replay/race conditions.
### Test coverage
New public functions tested? Error paths tested? Edge cases covered? Existing tests still valid?
### Architecture
Follows existing patterns? Unnecessary abstractions? Duplicated logic? Clean module dependencies?
**Present findings as a table:**
| # | Severity | Category | File:Line | Finding | Suggested Fix |
|---|----------|----------|-----------|---------|---------------|
Severity: Critical > High > Medium > Low > Nit
If `--review-only` flag is set, post findings as GitHub comments (see Phase 2c) and STOP.
Otherwise, ask which findings to fix (default: all Critical + High + Medium). Then proceed to Phase 3.
---
## Phase 2c: Post Review Comments on GitHub
For each finding the user approved (or all Critical/High/Medium if `--fix`):
**Line-specific findings** — post as PR review comments:
```
gh api repos/{owner}/{repo}/pulls/{number}/comments \
-f body="**{Severity}**: {finding}\n\n{explanation}\n\n**Suggested fix:** {suggestion}" \
-f path="{file}" \
-f commit_id="{headRefOid}" \
-F line={line} \
-f side="RIGHT"
```
**Cross-cutting/architectural findings** — post as regular PR comment:
```
gh pr comment {number} --body "..."
```
---
## Phase 3: Fix
Checkout the PR branch if not already on it (handles fork PRs automatically):
```
gh pr checkout {number}
```
**Implement fixes** for:
1. All approved review comment fixes (from Phase 2a)
2. All approved review findings (from Phase 2b)
Follow IronClaw conventions:
- `thiserror` for errors
- `crate::` imports
- No `.unwrap()` in production
- Both DB backends if persistence touched
- Regression test for every bug fix (enforced by commit-msg hook; bypass only with `[skip-regression-check]` if genuinely not feasible)
After all fixes implemented, proceed to Phase 4.
---
## Phase 4: Quality Gate
Run the full IronClaw shipping checklist:
```bash
cargo fmt
```
```bash
cargo clippy --all --benches --tests --examples --all-features
```
```bash
cargo test --lib
```
If persistence changes are present, also verify feature isolation:
```bash
cargo check --no-default-features --features libsql
cargo check --all-features
```
**If any step fails:** fix the issue and re-run. Do NOT proceed past a failing step. Loop up to 3 times per step. If still failing after 3 attempts, report the failure and stop.
---
## Phase 5: Commit & Push
Stage changed files by name (never `git add -A` — it can include unintended files):
```bash
git add path/to/changed/file1 path/to/changed/file2
git commit -m "{message}"
```
Commit message format:
- For review fixes: `fix: address review findings on PR #{number}`
- For comment responses: `fix: address review comments on PR #{number}`
- For CI fixes: `fix: resolve CI failures on PR #{number}`
- Include specifics in the body (which findings/comments were addressed)
Push:
```bash
git push origin {headRefName}
```
**Reply to addressed review comments on GitHub.** For each comment that was fixed, reply with the commit SHA and a brief description of what was done. For false positives, reply explaining why no change was needed.
---
## Phase 6: CI Monitor & Fix Loop
Wait briefly for CI to start, then poll (do NOT use `--watch` as it can hang indefinitely):
```
gh pr checks {number} --json name,status,conclusion
```
Re-check every 30 seconds, up to 10 minutes. If still pending after 10 minutes, report status and ask the user whether to keep waiting.
**If CI passes** → proceed to Phase 7.
**If CI fails** (up to 3 fix attempts):
1. Identify the failing check:
```
gh run view {run_id} --log-failed
```
If `--log-failed` shows nothing useful:
```
gh run view {run_id} --log | tail -100
```
2. Diagnose and fix the failure.
3. Re-run Phase 4 (quality gate).
4. Commit and push (Phase 5).
5. Go back to top of Phase 6.
**After 3 failed CI fix attempts:** Report what's failing and why, then stop. Don't keep looping.
---
## Phase 7: Merge Decision
Print final status:
```
PR #{number}: {title}
CI: ✅ PASS
Reviews: {summary}
Findings fixed: {N}
Comments addressed: {N}
Commits added: {N}
```
**Auto-merge conditions** (if `--merge` flag or user confirms):
- CI is passing
- No unresolved CHANGES_REQUESTED reviews
- PR is not draft
- PR is mergeable (no conflicts)
If all conditions met, ask the user for merge strategy:
"CI is green. Merge this PR? [squash/rebase/merge/no]"
Then execute:
```
gh pr merge {number} --{strategy} --delete-branch
```
If any condition NOT met, report what's blocking and let the user decide.
---
## Rules
- **Read before judging.** Never comment on code you haven't read in full. Verify line numbers.
- **Be specific.** "Line 42 returns 404 but should return 400 because X" not "this might have issues."
- **Fix the pattern, not just the instance.** When fixing a bug, grep for the same pattern across `src/`.
- **Respect the commit-msg hook.** Bug fixes need regression tests. Use `[skip-regression-check]` only if genuinely not feasible.
- **Don't over-fix.** Only change what was flagged. Don't refactor surrounding code or add improvements beyond the review scope.
- **Credit original authors.** If taking over someone else's PR, credit them in commits and comments.
- **No secrets in comments.** Never include customer data, credentials, or PII in GitHub comments.
- **Distinguish certainty.** "This IS a bug" vs "This COULD be a bug if X." Be honest.
- **Round up severity when uncertain.** Cheaper to dismiss a false alarm than miss a real bug.
- **Parallel where possible.** Use Agent tool for parallel file reads on large PRs. Batch `gh api` calls.
+63
View File
@@ -0,0 +1,63 @@
---
paths:
- "src/db/**"
- "src/history/**"
- "migrations/**"
---
# Database Rules
Dual-backend persistence: PostgreSQL + libSQL/Turso. **All new persistence features must support both backends.**
See `src/db/CLAUDE.md` for full schema, dialect differences, and libSQL limitations.
## Adding a New Operation
1. Decide which sub-trait it belongs to (`ConversationStore`, `JobStore`, `SandboxStore`, `RoutineStore`, `ToolFailureStore`, `SettingsStore`, `WorkspaceStore`) or create a new one
2. Add the async method signature to that sub-trait in `src/db/mod.rs`
3. Implement in `src/db/postgres.rs` (delegate to `Store`/`Repository`)
4. Implement in `src/db/libsql/<module>.rs` (use `self.connect().await?` per operation)
5. Add migration if needed:
- PostgreSQL: new `migrations/VN__description.sql`
- libSQL: add `CREATE TABLE IF NOT EXISTS` to `libsql_migrations.rs`
6. Test feature isolation:
```bash
cargo check # postgres (default)
cargo check --no-default-features --features libsql # libsql only
cargo check --all-features # both
```
## SQL Dialect Translation Checklist
When writing SQL for both backends, translate these types:
| PostgreSQL | libSQL |
|-----------|--------|
| `UUID` | `TEXT` |
| `TIMESTAMPTZ` | `TEXT` (ISO-8601, write with `fmt_ts()`, read with `get_ts()`) |
| `JSONB` | `TEXT` (JSON string) |
| `BOOLEAN` | `INTEGER` (0/1 -- use `get_i64(row, idx) != 0` to read) |
| `NUMERIC` | `TEXT` (preserves `rust_decimal` precision) |
| `TEXT[]` | `TEXT` (JSON-encoded array) |
| `VECTOR` | `BLOB` (flexible dimensions; vector index dropped, brute-force search fallback) |
| `jsonb_set(col, '{key}', val)` | `json_patch(col, '{"key": val}')` -- replaces top-level keys entirely, cannot do partial nested updates |
| `DEFAULT NOW()` | `DEFAULT (datetime('now'))` |
| `tsvector` + `ts_rank_cd` | FTS5 virtual table + sync triggers |
## Schema Translation Beyond DDL
Don't just translate `CREATE TABLE`. Also check:
- **Indexes** -- diff `CREATE INDEX` statements between backends
- **Seed data** -- check for `INSERT INTO` in migrations (e.g., `leak_detection_patterns`)
- **Triggers** -- PostgreSQL functions vs SQLite triggers (no stored procs in SQLite)
## Transaction Safety
Multi-step operations (INSERT+INSERT, UPDATE+DELETE, read-modify-write) MUST be wrapped in a transaction. Ask: "If this crashes between step N and N+1, is the database consistent?" If not, wrap in a transaction. Applies to both backends.
## libSQL Connection Model
`LibSqlBackend::connect()` creates a fresh connection per operation with `PRAGMA busy_timeout = 5000`. This is intentional -- no pool exists. Never hold connections open across `await` points. Satellite stores (`LibSqlSecretsStore`, `LibSqlWasmToolStore`) receive `Arc<LibSqlDatabase>` via `shared_db()` and call `.connect()` themselves -- never pass a live `Connection`.
## Fix the Pattern, Not the Instance
When fixing a bug in one backend's SQL, always grep for the same pattern in the other. A fix to `postgres.rs` that doesn't also fix `libsql/jobs.rs` is half a fix. Same applies to satellite stores.
+48
View File
@@ -0,0 +1,48 @@
---
paths:
- "src/**/*.rs"
---
# Review & Fix Discipline
Hard-won lessons from code review -- follow these when fixing bugs or addressing review feedback.
**Fix the pattern, not just the instance:** When a reviewer flags a bug (e.g., TOCTOU race in INSERT + SELECT-back), search the entire codebase for all instances of that same pattern. A fix in `SecretsStore::create()` that doesn't also fix `WasmToolStore::store()` is half a fix.
**Propagate architectural fixes to satellite types:** If a core type changes its concurrency model (e.g., `LibSqlBackend` switches to connection-per-operation), every type that was handed a resource from the old model must also be updated. Grep for the old type across the codebase.
**Schema translation is more than DDL:** When translating a database schema between backends (PostgreSQL to libSQL, etc.), check for:
- **Indexes** -- diff `CREATE INDEX` statements between the two schemas
- **Seed data** -- check for `INSERT INTO` in migrations (e.g., `leak_detection_patterns`)
- **Semantic differences** -- document where SQL functions behave differently (e.g., `json_patch` vs `jsonb_set`)
**Feature flag testing:** When adding feature-gated code, test compilation with each feature in isolation:
```bash
cargo check # default features
cargo check --no-default-features --features libsql # libsql only
cargo check --all-features # all features
```
**Regression test with every fix:** Every bug fix must include a test that would have caught the bug. Add a `#[test]` or `#[tokio::test]` that reproduces the original failure. Exempt: changes limited to `src/channels/web/static/` or `.md` files. Use `[skip-regression-check]` in commit message or PR label if genuinely not feasible. The `commit-msg` hook and CI workflow enforce this automatically.
**Zero clippy warnings policy:** Fix ALL clippy warnings before committing, including pre-existing ones in files you didn't change. Never leave warnings behind.
**Transaction safety:** Multi-step database operations (INSERT+INSERT, UPDATE+DELETE, read-then-write) MUST be wrapped in a transaction. Never assume sequential calls are atomic. This applies to both postgres and libsql backends.
**UTF-8 string safety:** Never use byte-index slicing (`&s[..n]`) on user-supplied or external strings -- it panics on multi-byte characters. Use `is_char_boundary()` or `char_indices()`. Grep for `[..` in changed files.
**Case-insensitive comparisons:** When comparing user-supplied strings (file paths, media types, extension names), normalize to lowercase with `.to_ascii_lowercase()`. Path comparisons must be case-insensitive on macOS/Windows.
**Decorator/wrapper trait delegation:** When adding a new method to `LlmProvider` (or any trait with decorator wrappers), update ALL wrapper types to delegate. Grep for `impl LlmProvider for` to find all implementations. Test through the full provider chain.
**Sensitive data in logs & events:** Tool parameters and outputs MUST be redacted before logging or broadcasting via SSE/WebSocket. Use `redact_params()` before any `tracing::info!`, `JobEvent`, or SSE emission that includes tool call data.
**Test temporary files:** Use the `tempfile` crate. Never hardcode `/tmp/...` paths.
**Trust boundaries in multi-process architecture:** Data from worker containers is untrusted. The orchestrator MUST validate: tool domain, nesting depth (server-side tracking), and parameter sensitivity.
**Mechanical verification before committing:**
- `cargo clippy --all --benches --tests --examples --all-features` -- zero warnings
- `grep -rnE '\.unwrap\(|\.expect\(' <files>` -- no panics in production
- `grep -rn 'super::' <files>` -- prefer `crate::` for cross-module imports (`super::` OK in tests/intra-module)
- If you fixed a pattern bug, `grep` for other instances across `src/`
- Run `scripts/pre-commit-safety.sh` to catch UTF-8, case-sensitivity, hardcoded /tmp, and logging issues
+34
View File
@@ -0,0 +1,34 @@
---
paths:
- "src/safety/**"
- "src/sandbox/**"
- "src/secrets/**"
- "src/tools/wasm/**"
---
# Safety Layer & Sandbox Rules
## Safety Layer
All external tool output passes through `SafetyLayer`:
1. **Sanitizer** - Detects injection patterns, escapes dangerous content
2. **Validator** - Checks length, encoding, forbidden patterns
3. **Policy** - Rules with severity (Critical/High/Medium/Low) and actions (Block/Warn/Review/Sanitize)
4. **Leak Detector** - Scans for 15+ secret patterns at two points: tool output before LLM, and LLM responses before user
Tool outputs are wrapped in `<tool_output>` XML before reaching the LLM.
## Shell Environment Scrubbing
The shell tool scrubs sensitive env vars before executing commands. The sanitizer detects command injection patterns (chained commands, subshells, path traversal).
## Sandbox Policies
| Policy | Filesystem | Network |
|--------|-----------|---------|
| ReadOnly | Read-only workspace | Allowlisted domains |
| WorkspaceWrite | Read-write workspace | Allowlisted domains |
| FullAccess | Full filesystem | Unrestricted |
## Zero-Exposure Credential Model
Secrets are stored encrypted on the host and injected into HTTP requests by the proxy at transit time. Container processes never see raw credential values.
+56
View File
@@ -0,0 +1,56 @@
---
paths:
- "src/skills/**"
- "skills/**"
---
# Skills System
SKILL.md files extend the agent's prompt with domain-specific instructions. Each skill is a YAML frontmatter block (metadata, activation criteria, required tools) followed by a markdown body injected into the LLM context.
## Trust Model
| Trust Level | Source | Tool Access |
|-------------|--------|-------------|
| **Trusted** | User-placed in `~/.ironclaw/skills/` or workspace `skills/` | All tools available to the agent |
| **Installed** | Downloaded from ClawHub registry (`~/.ironclaw/installed_skills/`) | Read-only tools only (no shell, file write, HTTP) |
## SKILL.md Format
```yaml
---
name: my-skill
version: 0.1.0
description: Does something useful
activation:
patterns:
- "deploy to.*production"
keywords:
- "deployment"
exclude_keywords:
- "rollback"
tags:
- "devops"
max_context_tokens: 2000
metadata:
openclaw:
requires:
bins: [docker, kubectl]
env: [KUBECONFIG]
---
# Skill instructions here...
```
## Selection Pipeline
1. **Gating** -- Check binary/env/config requirements; skip skills whose prerequisites are missing
2. **Scoring** -- Deterministic scoring: keywords (10/5 pts, cap 30) + patterns (20 pts, cap 40) + tags (3 pts, cap 15). `exclude_keywords` veto (score = 0 if any present)
3. **Budget** -- Select top-scoring skills within `SKILLS_MAX_TOKENS` prompt budget
4. **Attenuation** -- Minimum trust across active skills determines tool ceiling; installed skills lose dangerous tools
## Skill Tools
- `skill_list` -- List all discovered skills with trust level and status
- `skill_search` -- Search ClawHub registry for available skills
- `skill_install` -- Download and install a skill from ClawHub
- `skill_remove` -- Remove an installed skill
+25
View File
@@ -0,0 +1,25 @@
---
paths:
- "src/**/*.rs"
- "tests/**"
---
# Testing Rules
## Test Tiers
| Tier | Command | External deps |
|------|---------|---------------|
| Unit | `cargo test` | None |
| Integration | `cargo test --features integration` | Running PostgreSQL |
| Live | `cargo test --features integration -- --ignored` | PostgreSQL + LLM API keys |
Run `bash scripts/check-boundaries.sh` to verify test tier gating.
## Key Patterns
- Unit tests in `mod tests {}` at the bottom of each file
- Async tests with `#[tokio::test]`
- No mocks, prefer real implementations or stubs
- Use `tempfile` crate for test directories, never hardcode `/tmp/`
- Regression test with every bug fix (enforced by commit-msg hook)
- Integration tests (`--test workspace_integration`) require PostgreSQL; skipped if DB is unreachable
+39
View File
@@ -0,0 +1,39 @@
---
paths:
- "src/tools/**"
- "tools-src/**"
---
# Tool Architecture
**Keep tool-specific logic out of the main agent codebase.** The main agent provides generic infrastructure; tools are self-contained units that declare requirements through `<name>.capabilities.json` sidecar files (in dev mode: `tools-src/<name>/<name>-tool.capabilities.json`).
Tools can be WASM (sandboxed, credential-injected, single binary) or MCP servers (ecosystem, any language, no sandbox). Both are first-class via `ironclaw tool install`.
See `src/tools/README.md` for full architecture, adding new tools, auth JSON examples, and WASM vs MCP decision guide.
## Tool Implementation Pattern
```rust
#[async_trait]
impl Tool for MyTool {
fn name(&self) -> &str { "my_tool" }
fn description(&self) -> &str { "Does something useful" }
fn parameters_schema(&self) -> serde_json::Value {
serde_json::json!({
"type": "object",
"properties": {
"param": { "type": "string", "description": "A parameter" }
},
"required": ["param"]
})
}
async fn execute(&self, params: serde_json::Value, ctx: &JobContext)
-> Result<ToolOutput, ToolError>
{
let start = std::time::Instant::now();
// ... do work ...
Ok(ToolOutput::text("result", start.elapsed()))
}
fn requires_sanitization(&self) -> bool { true } // External data
}
```
+15
View File
@@ -5,6 +5,19 @@ DATABASE_POOL_SIZE=10
# LLM Provider
# LLM_BACKEND=nearai # default
# Possible values: nearai, ollama, openai_compatible, openai, anthropic, tinfoil
# LLM_REQUEST_TIMEOUT_SECS=120 # Increase for local LLMs (Ollama, vLLM, LM Studio)
# === Anthropic Direct ===
# Two auth modes:
# 1. API key: Set ANTHROPIC_API_KEY (from console.anthropic.com/settings/keys)
# 2. OAuth token: Set ANTHROPIC_OAUTH_TOKEN (from `claude login`)
# OAuth tokens use Authorization: Bearer instead of x-api-key header.
# ANTHROPIC_API_KEY=sk-ant-...
# ANTHROPIC_OAUTH_TOKEN=sk-ant-oat01-... # from `claude login` credentials
# ANTHROPIC_MODEL=claude-sonnet-4-20250514
# === OpenAI Direct ===
# OPENAI_API_KEY=sk-...
# === NEAR AI (Chat Completions API) ===
# Two auth modes:
@@ -102,6 +115,8 @@ AGENT_NAME=ironclaw
AGENT_MAX_PARALLEL_JOBS=5
AGENT_JOB_TIMEOUT_SECS=3600
AGENT_STUCK_THRESHOLD_SECS=300
# Maximum tokens per job (0 = unlimited, also settable via settings.json agent.max_tokens_per_job)
# AGENT_MAX_TOKENS_PER_JOB=0
# Enable planning phase before tool execution (default: true)
AGENT_USE_PLANNING=true
+100
View File
@@ -0,0 +1,100 @@
name: Claude Code Review
on:
pull_request:
types: [labeled]
permissions:
contents: read
pull-requests: write
issues: write
id-token: write
concurrency:
group: claude-review-${{ github.event.pull_request.number || github.run_id }}
cancel-in-progress: true
jobs:
review:
name: Claude Code Review
if: contains(github.event.pull_request.labels.*.name, 'staging-promotion')
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
with:
fetch-depth: 0
- name: Run Claude Code review
uses: anthropics/claude-code-action@v1
with:
anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
allowed_bots: "ironclaw-ci[bot]"
claude_args: "--max-turns 50 --model claude-haiku-4-5-20251001 --allowedTools 'Bash(gh pr comment:*),Bash(gh pr diff:*),Bash(gh pr view:*),Bash(gh pr list:*),Bash(gh issue view:*),Bash(gh issue list:*),Bash(gh search:*),Bash(git blame:*),Bash(git log:*),Bash(git diff:*)'"
prompt: |
Code review this pull request. Follow these steps precisely:
1. Use a Haiku agent to find relevant CLAUDE.md files: the root CLAUDE.md
and any CLAUDE.md files in directories whose files this PR modifies.
2. Use a Haiku agent to summarize the PR change (use `gh pr diff`).
3. Launch 4 parallel agents to review the change independently. Each agent should
read the PR diff with `gh pr diff` and the full source files for changed
code, then return a list of issues found:
Agent 1 — Security & Safety
Check for: command injection, path traversal, SSRF, XSS, auth bypass,
secrets in logs, .unwrap()/.expect() in production code (not tests),
race conditions, TOCTOU, unsafe blocks, panics in async, unbounded allocations.
Agent 2 — Architecture & Patterns
Check for: extensible design (traits/enums over nested conditionals),
clean abstractions, proper error types (thiserror), CLAUDE.md compliance,
type-driven design over stringly-typed code, DRY violations.
Agent 3 — Bug Scan
Shallow diff-only scan for obvious bugs: logic errors, off-by-one,
missing error handling, division by zero, incorrect return values.
Ignore nitpicks and likely false positives. Do NOT read extra context
beyond the diff — focus only on the changes.
Agent 4 — Performance & Production
Check for: blocking in async, N+1 queries, unbounded loops, missing
timeouts, resource leaks (file handles, connections), large allocations
in hot paths.
4. For each issue found, launch a parallel Haiku agent to:
a. Assign a severity:
- CRITICAL: security vulns, panics in prod (.unwrap/.expect), data exfiltration, race conditions
- HIGH: logic bugs, missing error handling, breaking API/schema changes
- MEDIUM: missing tests, unnecessary complexity, performance issues
- LOW: documentation gaps, naming suggestions
b. Score confidence 0-100 (give this rubric verbatim):
0: False positive, doesn't stand up to scrutiny, or pre-existing issue.
25: Might be real, but may be false positive. Stylistic issues not in CLAUDE.md.
50: Real issue but nitpick or rare in practice. Not very important.
75: Verified real issue, will be hit in practice. Directly impacts functionality
or explicitly mentioned in CLAUDE.md.
100: Certain, confirmed, will happen frequently. Evidence directly confirms.
5. Post a single comment on the PR using `gh pr comment` with this format.
If no issues were found, post "No issues found." instead:
### Code review
Found N issues:
1. [SEVERITY:CONFIDENCE] <brief description>
<permalink to file:line using full SHA, eg https://github.com/owner/repo/blob/abc123def/src/file.rs#L10-L15>
Example: [CRITICAL:92] `.unwrap()` can panic in production when config is missing
You MUST use the full git SHA in links (not HEAD or branch name).
Provide 1 line of context before and after each linked range.
Notes:
- Use `gh` for all GitHub interactions, not web fetch
- Do NOT check build signal or attempt to build/test the code
- Ignore pre-existing issues not introduced by this PR
- Ignore issues a linter/compiler would catch (formatting, imports, types)
+7 -1
View File
@@ -44,6 +44,7 @@ jobs:
clippy-windows:
name: Clippy Windows (${{ matrix.name }})
if: github.base_ref == 'main'
runs-on: windows-latest
strategy:
fail-fast: false
@@ -76,7 +77,12 @@ jobs:
needs: [format, clippy, clippy-windows]
steps:
- run: |
if [[ "${{ needs.format.result }}" != "success" || "${{ needs.clippy.result }}" != "success" || "${{ needs.clippy-windows.result }}" != "success" ]]; then
if [[ "${{ needs.format.result }}" != "success" || "${{ needs.clippy.result }}" != "success" ]]; then
echo "One or more jobs failed"
exit 1
fi
# clippy-windows only runs on main PRs, so skip/success are both acceptable
if [[ "${{ needs.clippy-windows.result }}" == "failure" ]]; then
echo "Windows clippy failed"
exit 1
fi
+28
View File
@@ -1,3 +1,31 @@
# Code Coverage Workflow
#
# This workflow runs test coverage analysis and uploads reports to Codecov.
# Coverage reports help identify untested code paths and maintain code quality.
#
# What it does:
# - Runs unit and integration tests with coverage instrumentation
# - Runs E2E tests with coverage instrumentation
# - Uploads coverage reports to Codecov (https://codecov.io/gh/nearai/ironclaw)
#
# Viewing coverage reports:
# - PRs automatically get coverage comments showing changes in coverage
# - Visit https://codecov.io/gh/nearai/ironclaw for detailed coverage reports
# - Coverage reports are generated for three configurations:
# 1. all-features: Full feature set
# 2. default: Default features
# 3. libsql-only: Minimal libSQL-only configuration
# - E2E coverage tracks end-to-end test coverage separately
#
# Coverage files:
# - Unit/integration: lcov.info (uploaded to Codecov with "unit" flag)
# - E2E: e2e-coverage.info (uploaded to Codecov with "e2e" flag)
#
# Requirements:
# - Uses cargo-llvm-cov for coverage instrumentation
# - Requires PostgreSQL for integration tests (pgvector/pgvector:pg16)
# - E2E tests require Python 3.12 and Playwright
name: Code Coverage
on:
push:
+1
View File
@@ -1,5 +1,6 @@
name: E2E Tests
on:
workflow_call:
schedule:
- cron: "0 6 * * 1" # Weekly Monday 6 AM UTC
workflow_dispatch:
+471
View File
@@ -0,0 +1,471 @@
name: Staging CI (Batched)
on:
schedule:
- cron: "0 * * * *" # Every 60 minutes
workflow_dispatch:
inputs:
force:
description: "Force run even if no new commits"
type: boolean
default: false
skip_claude_gate:
description: "Skip Claude review gate (bypass blocking findings)"
type: boolean
default: false
permissions:
contents: write
issues: write
pull-requests: write
checks: read
concurrency:
group: staging-ci
cancel-in-progress: false # Let running suites finish
jobs:
# ── Check for new commits ──────────────────────────────────────
check-changes:
name: Check for new commits
runs-on: ubuntu-latest
outputs:
has_changes: ${{ steps.check.outputs.has_changes }}
current_head: ${{ steps.check.outputs.current_head }}
diff_range: ${{ steps.check.outputs.diff_range }}
steps:
- uses: actions/checkout@v6
with:
ref: staging
fetch-depth: 0
fetch-tags: true
- name: Check for changes since last tested
id: check
env:
FORCE_RUN: ${{ inputs.force }}
run: |
CURRENT_HEAD=$(git rev-parse HEAD)
echo "current_head=${CURRENT_HEAD}" >> "$GITHUB_OUTPUT"
if git rev-parse staging-tested >/dev/null 2>&1; then
LAST_TESTED=$(git rev-parse staging-tested)
else
LAST_TESTED=""
fi
DIFF_RANGE=""
if [ -n "$LAST_TESTED" ] && [ "$LAST_TESTED" = "$CURRENT_HEAD" ]; then
echo "No new commits since last tested (${CURRENT_HEAD})"
HAS_CHANGES=false
else
HAS_CHANGES=true
if [ -n "$LAST_TESTED" ]; then
COMMIT_COUNT=$(git rev-list --count "${LAST_TESTED}..HEAD")
echo "Found ${COMMIT_COUNT} new commit(s) since last tested"
DIFF_RANGE="${LAST_TESTED}..${CURRENT_HEAD}"
else
git fetch origin main
MERGE_BASE=$(git merge-base origin/main HEAD)
echo "First run -- reviewing from merge-base ${MERGE_BASE}"
DIFF_RANGE="${MERGE_BASE}..${CURRENT_HEAD}"
fi
fi
# Force override from workflow_dispatch
if [ "$FORCE_RUN" = "true" ]; then
echo "Force run requested"
HAS_CHANGES=true
if [ -z "$DIFF_RANGE" ]; then
DIFF_RANGE="${CURRENT_HEAD}..${CURRENT_HEAD}"
fi
fi
echo "has_changes=${HAS_CHANGES}" >> "$GITHUB_OUTPUT"
echo "diff_range=${DIFF_RANGE}" >> "$GITHUB_OUTPUT"
# ── Run full test suite ──────────────────────────────────────────
tests:
name: Test Suite
needs: check-changes
if: needs.check-changes.outputs.has_changes == 'true'
uses: ./.github/workflows/test.yml
# ── Run E2E browser tests ────────────────────────────────────────
e2e:
name: E2E Browser Tests
needs: check-changes
if: needs.check-changes.outputs.has_changes == 'true'
uses: ./.github/workflows/e2e.yml
# ── Create promotion PR (triggers claude-review.yml on the PR) ──
create-promotion-pr:
name: Create Promotion PR
needs: check-changes
if: needs.check-changes.outputs.has_changes == 'true'
runs-on: ubuntu-latest
outputs:
pr_number: ${{ steps.create-pr.outputs.pr_number }}
promotion_branch: ${{ steps.branch.outputs.branch }}
steps:
- uses: actions/checkout@v6
with:
ref: staging
fetch-depth: 0
- name: Generate GitHub App token
id: app-token
uses: actions/create-github-app-token@v2
with:
app-id: ${{ secrets.GH_RELEASES_MANAGER_APP_ID }}
private-key: ${{ secrets.GH_RELEASES_MANAGER_APP_PRIVATE_KEY }}
- name: Set token
id: token
run: |
if [ -n "${{ steps.app-token.outputs.token }}" ]; then
echo "token=${{ steps.app-token.outputs.token }}" >> "$GITHUB_OUTPUT"
else
echo "token=${{ github.token }}" >> "$GITHUB_OUTPUT"
fi
- name: Check if staging is ahead of main
id: ahead-check
env:
GH_TOKEN: ${{ steps.token.outputs.token }}
run: |
git fetch origin main
AHEAD=$(git rev-list --count origin/main..origin/staging)
echo "commits_ahead=${AHEAD}" >> "$GITHUB_OUTPUT"
if [ "$AHEAD" -eq 0 ]; then
echo "Staging is not ahead of main. Nothing to promote."
else
echo "Staging is ${AHEAD} commits ahead of main."
fi
- name: Create promotion branch
id: branch
if: steps.ahead-check.outputs.commits_ahead != '0'
run: |
SHORT_SHA=$(echo "${{ needs.check-changes.outputs.current_head }}" | cut -c1-8)
BRANCH="staging-promote/${SHORT_SHA}-${{ github.run_id }}"
git checkout -b "$BRANCH"
git push origin "$BRANCH"
echo "branch=${BRANCH}" >> "$GITHUB_OUTPUT"
echo "Created promotion branch: ${BRANCH}"
- name: Find base branch
id: find-base
if: steps.ahead-check.outputs.commits_ahead != '0'
env:
GH_TOKEN: ${{ steps.token.outputs.token }}
run: |
# Find the newest open promotion PR with a staging-promote/* head branch
LATEST=$(gh pr list --label staging-promotion --state open \
--json headRefName,createdAt \
--jq '[.[] | select(.headRefName | startswith("staging-promote/"))] | sort_by(.createdAt) | last | .headRefName // empty')
if [ -n "$LATEST" ]; then
echo "base=${LATEST}" >> "$GITHUB_OUTPUT"
echo "Chaining onto existing promotion branch: ${LATEST}"
else
echo "base=main" >> "$GITHUB_OUTPUT"
echo "No existing promotion PR — targeting main"
fi
- name: Create promotion PR
id: create-pr
if: steps.ahead-check.outputs.commits_ahead != '0'
env:
GH_TOKEN: ${{ steps.token.outputs.token }}
run: |
RANGE="${{ needs.check-changes.outputs.diff_range }}"
TIMESTAMP=$(date -u +"%Y-%m-%d %H:%M UTC")
BRANCH="${{ steps.branch.outputs.branch }}"
BASE="${{ steps.find-base.outputs.base }}"
PR_URL=$(gh pr create \
--base "$BASE" \
--head "$BRANCH" \
--title "chore: promote staging to main (${TIMESTAMP})" \
--body "## Auto-promotion from staging CI
**Batch range:** \`${RANGE}\`
**Promotion branch:** \`${BRANCH}\`
**Base:** \`${BASE}\`
**Triggered by:** Staging CI batch at ${TIMESTAMP}
Waiting for gates:
- Tests: pending
- E2E: pending
- Claude Code review: pending (will post comments on this PR)
---
*Auto-created by staging-ci workflow*" \
--label "staging-promotion")
PR_NUM=$(echo "$PR_URL" | grep -oE '[0-9]+$')
echo "pr_number=${PR_NUM}" >> "$GITHUB_OUTPUT"
echo "Created promotion PR #${PR_NUM}"
# ── Gate: wait for review, process findings, merge or block ─────
gate:
name: Staging Gate
needs: [check-changes, tests, e2e, create-promotion-pr]
if: >
always() &&
needs.check-changes.outputs.has_changes == 'true' &&
needs.tests.result == 'success' &&
needs.e2e.result == 'success' &&
needs.create-promotion-pr.result == 'success'
runs-on: ubuntu-latest
timeout-minutes: 25
outputs:
gate_passed: ${{ steps.evaluate.outputs.passed }}
steps:
- uses: actions/checkout@v6
with:
ref: staging
fetch-depth: 1
- name: Generate GitHub App token
id: app-token
uses: actions/create-github-app-token@v2
with:
app-id: ${{ secrets.GH_RELEASES_MANAGER_APP_ID }}
private-key: ${{ secrets.GH_RELEASES_MANAGER_APP_PRIVATE_KEY }}
- name: Set token
id: token
run: |
if [ -n "${{ steps.app-token.outputs.token }}" ]; then
echo "token=${{ steps.app-token.outputs.token }}" >> "$GITHUB_OUTPUT"
else
echo "token=${{ github.token }}" >> "$GITHUB_OUTPUT"
fi
- name: Wait for Claude review job
env:
GH_TOKEN: ${{ steps.token.outputs.token }}
PR_NUMBER: ${{ needs.create-promotion-pr.outputs.pr_number }}
REPO: ${{ github.repository }}
run: |
if [ -z "$PR_NUMBER" ]; then
echo "No PR number — skipping wait"
exit 0
fi
PR_SHA=$(gh pr view "$PR_NUMBER" --json headRefOid --jq '.headRefOid' || echo "")
if [ -z "$PR_SHA" ]; then
echo "::warning::Could not get PR head SHA"
exit 0
fi
echo "Polling for Claude Code Review job on PR #${PR_NUMBER} (SHA: ${PR_SHA})..."
TIMEOUT=1200 # 20 minutes
ELAPSED=0
INTERVAL=30
while [ "$ELAPSED" -lt "$TIMEOUT" ]; do
STATUS=$(gh api "repos/${REPO}/commits/${PR_SHA}/check-runs" \
--jq '[.check_runs[] | select(.name == "Claude Code Review") | .conclusion // .status] | first // "pending"' 2>/dev/null || echo "pending")
if [ "$STATUS" = "success" ] || [ "$STATUS" = "failure" ] || [ "$STATUS" = "cancelled" ]; then
echo "Claude review job completed with status: ${STATUS} (${ELAPSED}s)"
exit 0
fi
echo "Claude review status: ${STATUS} (${ELAPSED}s elapsed)"
sleep "$INTERVAL"
ELAPSED=$((ELAPSED + INTERVAL))
done
echo "::warning::Claude review job not completed after ${TIMEOUT}s"
- name: Process Claude review comments and create issues
id: process-findings
env:
GH_TOKEN: ${{ steps.token.outputs.token }}
PR_NUMBER: ${{ needs.create-promotion-pr.outputs.pr_number }}
REPO: ${{ github.repository }}
run: |
HAS_BLOCKING=false
ISSUES_CREATED=0
if [ -z "$PR_NUMBER" ]; then
echo "No PR — skipping finding processing"
echo "has_blocking=false" >> "$GITHUB_OUTPUT"
exit 0
fi
# Check for "No issues found" first (clean pass)
NO_ISSUES=$(gh api "repos/${REPO}/issues/${PR_NUMBER}/comments" \
--jq '[.[] | select(.user.login == "claude[bot]") | select(.body | test("No issues found"))] | length' 2>/dev/null || echo "0")
if [ "$NO_ISSUES" -gt 0 ]; then
echo "Claude review found no issues — gate passes"
echo "has_blocking=false" >> "$GITHUB_OUTPUT"
exit 0
fi
# Get the last Claude comment that contains findings
JQ_FILTER='[.[] | select(.user.login == "claude[bot]") | select(.body | test("Found [0-9]+ issue"))] | last'
BODY=$(gh api "repos/${REPO}/issues/${PR_NUMBER}/comments" \
--jq "${JQ_FILTER} | .body // empty" 2>/dev/null || echo "")
COMMENT_URL=$(gh api "repos/${REPO}/issues/${PR_NUMBER}/comments" \
--jq "${JQ_FILTER} | .html_url // empty" 2>/dev/null || echo "")
if [ -z "$BODY" ]; then
echo "::warning::No Claude review comment found for PR #${PR_NUMBER} — treating as blocking"
echo "has_blocking=true" >> "$GITHUB_OUTPUT"
exit 0
fi
# Parse [SEVERITY:CONFIDENCE] tags from each numbered finding
# Matrix: CRITICAL always→issue, ≥80→block. HIGH ≥50→issue. MEDIUM ≥80→issue. LOW ≥80→issue.
# Use process substitution so variables propagate to parent shell
while read -r line; do
TAG=$(echo "$line" | grep -oE '^\[(CRITICAL|HIGH|MEDIUM|LOW):[0-9]+\]')
SEVERITY=$(echo "$TAG" | sed 's/\[\(.*\):\(.*\)\]/\1/')
CONFIDENCE=$(echo "$TAG" | sed 's/\[\(.*\):\(.*\)\]/\2/')
DESC=$(echo "$line" | sed "s/\[${SEVERITY}:${CONFIDENCE}\] *//" | head -1)
echo "Found: [${SEVERITY}:${CONFIDENCE}] ${DESC}"
# Check if blocking (CRITICAL ≥80)
if [ "$SEVERITY" = "CRITICAL" ] && [ "$CONFIDENCE" -ge 80 ]; then
HAS_BLOCKING=true
fi
# Determine if this should create an issue
CREATE_ISSUE=false
case "$SEVERITY" in
CRITICAL) CREATE_ISSUE=true ;;
HIGH) [ "$CONFIDENCE" -ge 50 ] && CREATE_ISSUE=true ;;
MEDIUM) [ "$CONFIDENCE" -ge 80 ] && CREATE_ISSUE=true ;;
LOW) [ "$CONFIDENCE" -ge 80 ] && CREATE_ISSUE=true ;;
esac
if [ "$CREATE_ISSUE" = "true" ]; then
case "$SEVERITY" in
CRITICAL) LABELS="bug,risk: high,staging-ci-review" ;;
HIGH) LABELS="bug,risk: medium,staging-ci-review" ;;
MEDIUM) LABELS="risk: medium,staging-ci-review" ;;
LOW) LABELS="risk: low,staging-ci-review" ;;
esac
TITLE=$(echo "$DESC" | cut -c1-80)
{
echo "## [${SEVERITY}:${CONFIDENCE}] Issue Found by Staging CI Review"
echo ""
echo "**Severity:** ${SEVERITY}"
echo "**Confidence:** ${CONFIDENCE}/100"
echo "**PR comment:** ${COMMENT_URL}"
echo ""
echo "### Description"
echo "$DESC"
echo ""
echo "---"
echo "*Auto-created by staging-ci Claude Code review*"
} > /tmp/issue-body.md
if gh issue create \
--title "[${SEVERITY}] ${TITLE}" \
--body-file /tmp/issue-body.md \
--label "${LABELS}"; then
ISSUES_CREATED=$((ISSUES_CREATED + 1))
else
echo "::warning::Failed to create issue for ${SEVERITY} finding"
fi
fi
done < <(echo "$BODY" | grep -oE '\[(CRITICAL|HIGH|MEDIUM|LOW):[0-9]+\].*')
echo "Created ${ISSUES_CREATED} issues"
echo "has_blocking=${HAS_BLOCKING}" >> "$GITHUB_OUTPUT"
- name: Evaluate gate
id: evaluate
env:
PR_NUMBER: ${{ needs.create-promotion-pr.outputs.pr_number }}
SKIP_GATE: ${{ inputs.skip_claude_gate }}
HAS_BLOCKING: ${{ steps.process-findings.outputs.has_blocking }}
run: |
SKIP_INPUT="$SKIP_GATE"
if [ "$HAS_BLOCKING" = "true" ]; then
echo "::warning::Claude review found blocking issues (CRITICAL ≥80 confidence)"
if [ "$SKIP_INPUT" = "true" ]; then
echo "::warning::Gate overridden by skip_claude_gate workflow input"
echo "passed=true" >> "$GITHUB_OUTPUT"
else
echo "::error::Blocking promotion due to CRITICAL findings (≥80 confidence)"
echo "::error::PR #${PR_NUMBER} left open with review comments"
echo "passed=false" >> "$GITHUB_OUTPUT"
exit 1
fi
else
echo "No blocking findings. Gate passed."
echo "passed=true" >> "$GITHUB_OUTPUT"
fi
- name: Merge promotion PR
id: merge
if: steps.evaluate.outputs.passed == 'true'
env:
GH_TOKEN: ${{ steps.token.outputs.token }}
PR_NUMBER: ${{ needs.create-promotion-pr.outputs.pr_number }}
run: |
if [ -n "$PR_NUMBER" ]; then
echo "Merging promotion PR #${PR_NUMBER}"
# Do NOT use --delete-branch: deleting a promotion branch closes
# any chained PRs that use it as their base (verified in ironclaw-ci-test).
# Stale promotion branches are cleaned up separately.
gh pr merge "$PR_NUMBER" --merge
echo "merged=true" >> "$GITHUB_OUTPUT"
fi
# ── Update tested tag (always, so next batch covers only new commits) ──
update-tag:
name: Update staging-tested tag
needs: [check-changes, tests, e2e, create-promotion-pr, gate]
if: >
always() &&
needs.check-changes.outputs.has_changes == 'true' &&
needs.tests.result == 'success' &&
needs.e2e.result == 'success' &&
needs.create-promotion-pr.result == 'success'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
with:
ref: staging
fetch-depth: 1
- name: Update staging-tested tag
run: |
git tag -f staging-tested "${{ needs.check-changes.outputs.current_head }}"
git push origin staging-tested --force
echo "Updated staging-tested tag to ${{ needs.check-changes.outputs.current_head }}"
# ── Report ───────────────────────────────────────────────────────
report:
name: Staging CI Summary
needs: [check-changes, tests, e2e, create-promotion-pr, gate, update-tag]
if: always() && needs.check-changes.outputs.has_changes == 'true'
runs-on: ubuntu-latest
steps:
- name: Summary
run: |
echo "## Staging CI Batch Results" >> "$GITHUB_STEP_SUMMARY"
echo "" >> "$GITHUB_STEP_SUMMARY"
echo "| Check | Result |" >> "$GITHUB_STEP_SUMMARY"
echo "|-------|--------|" >> "$GITHUB_STEP_SUMMARY"
echo "| Tests | ${{ needs.tests.result }} |" >> "$GITHUB_STEP_SUMMARY"
echo "| E2E | ${{ needs.e2e.result }} |" >> "$GITHUB_STEP_SUMMARY"
echo "| Promotion PR | ${{ needs.create-promotion-pr.result }} |" >> "$GITHUB_STEP_SUMMARY"
echo "| Gate | ${{ needs.gate.result }} |" >> "$GITHUB_STEP_SUMMARY"
echo "| Tag Updated | ${{ needs.update-tag.result }} |" >> "$GITHUB_STEP_SUMMARY"
echo "" >> "$GITHUB_STEP_SUMMARY"
echo "Range: ${{ needs.check-changes.outputs.diff_range }}" >> "$GITHUB_STEP_SUMMARY"
PR_NUM="${{ needs.create-promotion-pr.outputs.pr_number }}"
if [ -n "$PR_NUM" ]; then
echo "Promotion PR: #${PR_NUM}" >> "$GITHUB_STEP_SUMMARY"
fi
+32 -7
View File
@@ -1,6 +1,9 @@
name: Run Tests
on:
workflow_call:
pull_request:
branches:
- main
push:
branches:
- main
@@ -38,6 +41,9 @@ jobs:
telegram-tests:
name: Telegram Channel Tests
if: >
github.event_name == 'push' ||
(github.event_name == 'pull_request' && github.base_ref != 'staging')
runs-on: ubuntu-latest
steps:
- name: Checkout repository
@@ -50,6 +56,9 @@ jobs:
windows-build:
name: Windows Build (${{ matrix.name }})
if: >
github.event_name == 'push' ||
(github.event_name == 'pull_request' && github.base_ref != 'staging')
runs-on: windows-latest
strategy:
fail-fast: false
@@ -74,6 +83,9 @@ jobs:
wasm-wit-compat:
name: WASM WIT Compatibility
if: >
github.event_name == 'push' ||
(github.event_name == 'pull_request' && github.base_ref != 'staging')
runs-on: ubuntu-latest
steps:
- name: Checkout repository
@@ -94,6 +106,9 @@ jobs:
docker-build:
name: Docker Build
if: >
github.event_name == 'push' ||
(github.event_name == 'pull_request' && github.base_ref != 'staging')
runs-on: ubuntu-latest
steps:
- name: Checkout repository
@@ -123,12 +138,22 @@ jobs:
needs: [tests, telegram-tests, wasm-wit-compat, docker-build, windows-build, version-check]
steps:
- run: |
if [[ "${{ needs.tests.result }}" != "success" || "${{ needs.telegram-tests.result }}" != "success" || "${{ needs.wasm-wit-compat.result }}" != "success" || "${{ needs.docker-build.result }}" != "success" || "${{ needs.windows-build.result }}" != "success" ]]; then
echo "One or more jobs failed"
exit 1
fi
# version-check only runs on PRs, so skip/success are both acceptable
if [[ "${{ needs.version-check.result }}" == "failure" ]]; then
echo "Version bump check failed"
# Unit tests must always pass
if [[ "${{ needs.tests.result }}" != "success" ]]; then
echo "Unit tests failed"
exit 1
fi
# Gated jobs: must pass on promotion PRs / push, skipped on developer PRs
for job in telegram-tests wasm-wit-compat docker-build windows-build version-check; do
case "$job" in
telegram-tests) result="${{ needs.telegram-tests.result }}" ;;
wasm-wit-compat) result="${{ needs.wasm-wit-compat.result }}" ;;
docker-build) result="${{ needs.docker-build.result }}" ;;
windows-build) result="${{ needs.windows-build.result }}" ;;
version-check) result="${{ needs.version-check.result }}" ;;
esac
if [[ "$result" == "failure" || "$result" == "cancelled" ]]; then
echo "$job failed"
exit 1
fi
done
+7 -1
View File
@@ -4,8 +4,9 @@
.env.*
!.env.example
# Claude Code worktrees
# Claude Code worktrees and lock files
.claude/worktrees/
.claude/scheduled_tasks.lock
# Sidecar tool data
.sidecar/
@@ -22,3 +23,8 @@ bench-results/
# WASM build artifacts (loaded from disk, not bundled)
*.wasm
# Traces
trace_*.json
# Local Claude Code settings (machine-specific, should not be committed)
.claude/settings.local.json
+4
View File
@@ -7,6 +7,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Added
- AWS Bedrock LLM provider via native Converse API with IAM and SSO auth support (feature-gated: `--features bedrock`)
## [0.16.1](https://github.com/nearai/ironclaw/compare/v0.16.0...v0.16.1) - 2026-03-06
### Fixed
+82 -537
View File
@@ -1,73 +1,37 @@
# IronClaw Development Guide
## Project Overview
**IronClaw** is a secure personal AI assistant that protects your data and expands its capabilities on the fly.
### Core Philosophy
- **User-first security** - Your data stays yours, encrypted and local
- **Self-expanding** - Build new tools dynamically without vendor dependency
- **Defense in depth** - Multiple security layers against prompt injection and data exfiltration
- **Always available** - Multi-channel access with proactive background execution
### Features
- **Multi-channel input**: TUI (Ratatui), HTTP webhooks, WASM channels (Telegram, Slack), web gateway
- **Parallel job execution** with state machine and self-repair for stuck jobs
- **Sandbox execution**: Docker container isolation with network proxy and credential injection
- **Claude Code mode**: Delegate jobs to Claude CLI inside containers
- **Skills system**: SKILL.md prompt extensions with trust model, tool attenuation, and ClawHub registry
- **Routines**: Scheduled (cron) and reactive (event, webhook) task execution
- **Web gateway**: Browser UI with SSE/WebSocket real-time streaming
- **Extension management**: Install, auth, activate MCP/WASM extensions
- **Extensible tools**: Built-in tools, WASM sandbox, MCP client, dynamic builder
- **Persistent memory**: Workspace with hybrid search (FTS + vector via RRF)
- **Prompt injection defense**: Sanitizer, validator, policy rules, leak detection, shell env scrubbing
- **Multi-provider LLM**: NEAR AI, OpenAI, Anthropic, Ollama, OpenAI-compatible, Tinfoil private inference
- **Setup wizard**: 7-step interactive onboarding for first-run configuration
- **Heartbeat system**: Proactive periodic execution with checklist
**IronClaw** is a secure personal AI assistant — user-first security, self-expanding tools, defense in depth, multi-channel access with proactive background execution.
## Build & Test
```bash
# Format code
cargo fmt
# Lint (fix ALL warnings before committing, including pre-existing ones)
cargo clippy --all --benches --tests --examples --all-features
# Run all tests
cargo test
# Run specific test
cargo test test_name
# Run with logging
RUST_LOG=ironclaw=debug cargo run
# Run integration tests (may require running services/DB)
cargo test --test workspace_integration
cargo test --test ws_gateway_integration
cargo test --test heartbeat_integration
# Run E2E tests (Python/Playwright — requires a running ironclaw instance)
# See tests/e2e/CLAUDE.md for full setup instructions
cd tests/e2e
python -m venv .venv && source .venv/bin/activate # On Windows: .venv\Scripts\activate
pip install -e .
playwright install chromium
pytest scenarios/ # all scenarios
pytest scenarios/test_chat.py # specific scenario
cargo fmt # format
cargo clippy --all --benches --tests --examples --all-features # lint (zero warnings)
cargo test # unit tests
cargo test --features integration # + PostgreSQL tests
RUST_LOG=ironclaw=debug cargo run # run with logging
```
### Test Tiers
E2E tests: see `tests/e2e/CLAUDE.md`.
| Tier | Command | What runs | External deps |
|------|---------|-----------|---------------|
| Unit | `cargo test` | All `mod tests` + self-contained integration tests | None |
| Integration | `cargo test --features integration` | + PostgreSQL-dependent tests | Running PostgreSQL |
| Live | `cargo test --features integration -- --ignored` | + LLM-dependent tests | PostgreSQL + LLM API keys |
## Code Style
Run `bash scripts/check-boundaries.sh` to verify test tier gating and other architecture rules.
- Prefer `crate::` for cross-module imports; `super::` is fine in tests and intra-module refs
- No `pub use` re-exports unless exposing to downstream consumers
- No `.unwrap()` or `.expect()` in production code (tests are fine)
- Use `thiserror` for error types in `error.rs`
- Map errors with context: `.map_err(|e| SomeError::Variant { reason: e.to_string() })?`
- Prefer strong types over strings (enums, newtypes)
- Keep functions focused, extract helpers when logic is reused
- Comments for non-obvious logic only
## Architecture
Prefer generic/extensible architectures over hardcoding specific integrations. Ask clarifying questions about the desired abstraction level before implementing.
Key traits for extensibility: `Database`, `Channel`, `Tool`, `LlmProvider`, `SuccessEvaluator`, `EmbeddingProvider`, `NetworkPolicyDecider`, `Hook`, `Observer`, `Tunnel`.
All I/O is async with tokio. Use `Arc<T>` for shared state, `RwLock` for concurrent access.
## Project Structure
@@ -95,12 +59,6 @@ src/
│ ├── channel.rs # Channel trait, IncomingMessage, OutgoingResponse
│ ├── manager.rs # ChannelManager merges streams
│ ├── cli/ # Full TUI with Ratatui
│ │ ├── mod.rs # TuiChannel implementation
│ │ ├── app.rs # Application state
│ │ ├── render.rs # UI rendering
│ │ ├── events.rs # Input handling
│ │ ├── overlay.rs # Approval overlays
│ │ └── composer.rs # Message composition
│ ├── http.rs # HTTP webhook (axum) with secret validation
│ ├── webhook_server.rs # Unified HTTP server composing all webhook routes
│ ├── repl.rs # Simple REPL (for testing)
@@ -111,62 +69,38 @@ src/
│ ├── capabilities.rs # Channel-specific capabilities (HTTP endpoint, emit rate)
│ ├── error.rs # WASM channel error types
│ ├── runtime.rs # WASM channel execution runtime
│ ├── setup.rs # WasmChannelSetup, setup_wasm_channels(), inject_channel_credentials()
│ └── wrapper.rs # Channel trait wrapper for WASM modules
├── cli/ # CLI subcommands (clap)
│ ├── mod.rs # Cli struct, Command enum (run/onboard/config/tool/registry/mcp/memory/pairing/service/doctor/status/completion)
── config.rs # config list/get/set subcommands
│ ├── tool.rs # tool install/list/remove subcommands
│ ├── registry.rs # registry list/install subcommands
│ ├── mcp.rs # mcp add/auth/list/test subcommands
│ ├── memory.rs # memory search/read/write subcommands
│ ├── pairing.rs # pairing list/approve subcommands
│ ├── service.rs # service install/start/stop subcommands
│ ├── doctor.rs # Active health diagnostics
│ ├── status.rs # System health/status display
│ ├── completion.rs # Shell completion script generation
│ └── oauth_defaults.rs # Default OAuth redirect URIs
── config.rs, tool.rs, registry.rs, mcp.rs, memory.rs, pairing.rs, service.rs, doctor.rs, status.rs, completion.rs
├── registry/ # Extension registry catalog
│ ├── mod.rs # Public API; re-exports RegistryCatalog, RegistryInstaller, manifest types
│ ├── manifest.rs # ExtensionManifest, ArtifactSpec, BundleDefinition types
│ ├── catalog.rs # RegistryCatalog: load from filesystem and embedded JSON
── installer.rs # RegistryInstaller: download, verify, install WASM artifacts
│ ├── artifacts.rs # Artifact download and caching
│ └── embedded.rs # Catalog compiled into binary at build time (via build.rs)
── installer.rs # RegistryInstaller: download, verify, install WASM artifacts
├── hooks/ # Lifecycle hooks for intercepting agent operations
│ ├── mod.rs # 6 HookPoints: BeforeInbound, BeforeToolCall, BeforeOutbound, OnSessionStart, OnSessionEnd, TransformResponse
│ ├── hook.rs # Hook trait, HookContext, HookEvent, HookOutcome, HookFailureMode
│ ├── registry.rs # HookRegistry: register, prioritize, execute hooks
│ └── bundled.rs # Built-in hooks: rule-based filters, webhook forwarders, HookBundleConfig
├── hooks/ # Lifecycle hooks (6 points: BeforeInbound, BeforeToolCall, BeforeOutbound, OnSessionStart, OnSessionEnd, TransformResponse)
├── tunnel/ # Tunnel abstraction for public internet exposure
│ ├── mod.rs # Tunnel trait, TunnelProviderConfig, create_tunnel() factory
│ ├── mod.rs # Tunnel trait, TunnelProviderConfig, create_tunnel(), start_managed_tunnel()
│ ├── cloudflare.rs # CloudflareTunnel (cloudflared binary)
│ ├── ngrok.rs # NgrokTunnel
│ ├── tailscale.rs # TailscaleTunnel (serve/funnel modes)
│ ├── custom.rs # CustomTunnel (arbitrary command with {host}/{port})
│ └── none.rs # NoneTunnel (local-only, no exposure)
├── observability/ # Pluggable event/metric recording
│ ├── mod.rs # create_observer() factory, ObservabilityConfig
│ ├── traits.rs # Observer trait, ObserverEvent, ObserverMetric
│ ├── noop.rs # NoopObserver (zero overhead, default)
│ ├── log.rs # LogObserver (tracing-based)
│ └── multi.rs # MultiObserver (fan-out to multiple backends)
├── observability/ # Pluggable event/metric recording (noop, log, multi)
├── orchestrator/ # Internal HTTP API for sandbox containers
│ ├── mod.rs
│ ├── api.rs # Axum endpoints (LLM proxy, events, prompts)
│ ├── auth.rs # Per-job bearer token store
│ └── job_manager.rs # Container lifecycle (create, stop, cleanup)
├── worker/ # Runs inside Docker containers
│ ├── mod.rs
│ ├── runtime.rs # Worker execution loop (tool calls, LLM)
│ ├── claude_bridge.rs # Claude Code bridge (spawns claude CLI)
│ ├── api.rs # HTTP client to orchestrator
│ └── proxy_llm.rs # LlmProvider that proxies through orchestrator
├── safety/ # Prompt injection defense
@@ -174,30 +108,15 @@ src/
│ ├── validator.rs # Input validation (length, encoding, patterns)
│ ├── policy.rs # PolicyRule system with severity/actions
│ ├── leak_detector.rs # Secret detection (API keys, tokens, etc.)
│ └── credential_detect.rs # HTTP request credential detection (headers, URL params)
│ └── credential_detect.rs # HTTP request credential detection
├── llm/ # Multi-provider LLM integration — see src/llm/CLAUDE.md
├── tools/ # Extensible tool system
│ ├── tool.rs # Tool trait, ToolOutput, ToolError
│ ├── registry.rs # ToolRegistry for discovery
│ ├── sandbox.rs # Process-based sandbox (stub, superseded by wasm/)
│ ├── rate_limiter.rs # Shared sliding-window rate limiter for built-in and WASM tools
│ ├── builtin/ # Built-in tools
│ │ ├── echo.rs, time.rs, json.rs, http.rs
│ │ ├── web_fetch.rs # GET URL → clean Markdown (readability + html-to-md conversion)
│ │ ├── file.rs # ReadFile, WriteFile, ListDir, ApplyPatch
│ │ ├── shell.rs # Shell command execution
│ │ ├── memory.rs # Memory tools (search, write, read, tree)
│ │ ├── message.rs # MessageTool: agent proactively messages users on any channel
│ │ ├── job.rs # CreateJob, ListJobs, JobStatus, CancelJob
│ │ ├── routine.rs # routine_create/list/update/delete/history
│ │ ├── extension_tools.rs # Extension install/auth/activate/remove
│ │ ├── skill_tools.rs # skill_list/search/install/remove tools
│ │ ├── secrets_tools.rs # secret_list/secret_delete (zero-exposure: no values exposed)
│ │ ├── html_converter.rs # HTML→Markdown via readability + html-to-markdown-rs
│ │ ├── path_utils.rs # Shared path validation/canonicalization helpers
│ │ └── marketplace.rs, ecommerce.rs, taskrabbit.rs, restaurant.rs (stubs)
│ ├── rate_limiter.rs # Shared sliding-window rate limiter
│ ├── builtin/ # Built-in tools (echo, time, json, http, web_fetch, file, shell, memory, message, job, routine, extension_tools, skill_tools, secrets_tools)
│ ├── builder/ # Dynamic tool building
│ │ ├── core.rs # BuildRequirement, SoftwareType, Language
│ │ ├── templates.rs # Project scaffolding
@@ -205,6 +124,7 @@ src/
│ │ └── validation.rs # WASM validation
│ ├── mcp/ # Model Context Protocol
│ │ ├── client.rs # MCP client over HTTP
│ │ ├── factory.rs # create_client_from_config() — transport dispatch factory
│ │ ├── protocol.rs # JSON-RPC types
│ │ └── session.rs # MCP session management (Mcp-Session-Id header, per-server state)
│ └── wasm/ # Full WASM sandbox (wasmtime)
@@ -221,132 +141,55 @@ src/
├── db/ # Dual-backend persistence (PostgreSQL + libSQL) — see src/db/CLAUDE.md
├── workspace/ # Persistent memory system (OpenClaw-inspired)
│ ├── mod.rs # Workspace struct, memory operations
│ ├── document.rs # MemoryDocument, MemoryChunk, WorkspaceEntry
│ ├── chunker.rs # Document chunking (800 tokens, 15% overlap)
│ ├── embeddings.rs # EmbeddingProvider trait, OpenAI implementation
│ ├── search.rs # Hybrid search with RRF algorithm
│ └── repository.rs # PostgreSQL CRUD and search operations
├── workspace/ # Persistent memory system — see src/workspace/README.md
├── context/ # Job context isolation
├── state.rs # JobState enum, JobContext, state machine
│ ├── memory.rs # ActionRecord, ConversationMemory
│ └── manager.rs # ContextManager for concurrent jobs
├── estimation/ # Cost/time/value estimation
│ ├── cost.rs # CostEstimator
│ ├── time.rs # TimeEstimator
│ ├── value.rs # ValueEstimator (profit margins)
│ └── learner.rs # Exponential moving average learning
├── evaluation/ # Success evaluation
│ ├── success.rs # SuccessEvaluator trait, RuleBasedEvaluator, LlmEvaluator
│ └── metrics.rs # MetricsCollector, QualityMetrics
├── context/ # Job context isolation (JobState, JobContext, ContextManager)
├── estimation/ # Cost/time/value estimation with EMA learning
├── evaluation/ # Success evaluation (rule-based, LLM-based)
├── sandbox/ # Docker execution sandbox
│ ├── mod.rs # Public API, default allowlist
│ ├── config.rs # SandboxConfig, SandboxPolicy enum
│ ├── config.rs # SandboxConfig, SandboxPolicy enum (ReadOnly/WorkspaceWrite/FullAccess)
│ ├── manager.rs # SandboxManager orchestration
│ ├── container.rs # ContainerRunner, Docker lifecycle
── error.rs # SandboxError types
│ └── proxy/ # Network proxy for containers
│ ├── mod.rs # NetworkProxyBuilder
│ ├── http.rs # HttpProxy, CredentialResolver trait
│ ├── policy.rs # NetworkPolicyDecider trait
│ └── allowlist.rs # DomainAllowlist validation
── proxy/ # Network proxy: domain allowlist, credential injection, CONNECT tunnel
├── secrets/ # Secrets management
│ ├── mod.rs # SecretsStore trait, public API
│ ├── types.rs # Core types (Secret, SecretRef, SecretMetadata)
│ ├── crypto.rs # AES-256-GCM encryption
│ ├── keychain.rs # OS keychain integration (macOS Keychain, GNOME Keyring) for master key
│ └── store.rs # Encrypted secret storage
├── secrets/ # Secrets management (AES-256-GCM, OS keychain for master key)
├── setup/ # Onboarding wizard (spec: src/setup/README.md)
│ ├── mod.rs # Entry point, check_onboard_needed()
│ ├── wizard.rs # 7-step interactive wizard
│ ├── channels.rs # Channel setup helpers
│ └── prompts.rs # Terminal prompts (select, confirm, secret)
├── setup/ # 7-step onboarding wizard — see src/setup/README.md
├── skills/ # SKILL.md prompt extension system
│ ├── mod.rs # Core types (SkillTrust, LoadedSkill)
│ ├── registry.rs # SkillRegistry: discover, install, remove
│ ├── selector.rs # Deterministic scoring prefilter
│ ├── attenuation.rs # Trust-based tool ceiling
│ ├── gating.rs # Requirement checks (bins, env, config)
│ ├── parser.rs # SKILL.md frontmatter + markdown parser
│ └── catalog.rs # ClawHub registry client
├── skills/ # SKILL.md prompt extension system — see .claude/rules/skills.md
└── history/ # Persistence
├── store.rs # PostgreSQL repositories
└── analytics.rs # Aggregation queries (JobStats, ToolStats)
└── history/ # Persistence (PostgreSQL repositories, analytics)
tests/
├── *.rs # Integration tests (workspace, heartbeat, WS gateway, pairing, etc.)
├── test-pages/ # HTML→Markdown conversion fixtures (CNN, Medium, Yahoo)
├── test-pages/ # HTML→Markdown conversion fixtures
└── e2e/ # Python/Playwright E2E scenarios (see tests/e2e/CLAUDE.md)
```
## Key Patterns
## Database
### Architecture
Dual-backend: PostgreSQL + libSQL/Turso. **All new persistence features must support both backends.** See `src/db/CLAUDE.md` and `.claude/rules/database.md`.
When designing new features or systems, always prefer generic/extensible architectures over hardcoding specific integrations. Ask clarifying questions about the desired abstraction level before implementing.
## Module Specs
### Error Handling
- Use `thiserror` for error types in `error.rs`
- Never use `.unwrap()` or `.expect()` in production code (tests are fine)
- Map errors with context: `.map_err(|e| SomeError::Variant { reason: e.to_string() })?`
- Before committing, grep for `.unwrap()` and `.expect(` in changed files to catch violations mechanically
When modifying a module with a spec, read the spec first. Code follows spec; spec is the tiebreaker.
### Async
- All I/O is async with tokio
- Use `Arc<T>` for shared state across tasks
- Use `RwLock` for concurrent read/write access
**Module-owned initialization:** Module-specific initialization logic (database connection, transport creation, channel setup) must live in the owning module as a public factory function — not in `main.rs` or `app.rs`. These entry-point files orchestrate calls to module factories. Feature-flag branching (`#[cfg(feature = ...)]`) must be confined to the module that owns the abstraction.
### Traits for Extensibility
- `Database` - Add new database backends (must implement all ~78 methods)
- `Channel` - Add new input sources
- `Tool` - Add new capabilities
- `LlmProvider` - Add new LLM backends
- `SuccessEvaluator` - Custom evaluation logic
- `EmbeddingProvider` - Add embedding backends (workspace search)
- `NetworkPolicyDecider` - Custom network access policies for sandbox containers
- `Hook` - Lifecycle hook at 6 interception points (BeforeInbound, BeforeToolCall, BeforeOutbound, OnSessionStart, OnSessionEnd, TransformResponse)
- `Observer` - Observability backend (noop/log/multi; future: OpenTelemetry, Prometheus)
- `Tunnel` - Tunnel provider for public internet exposure
| Module | Spec |
|--------|------|
| `src/agent/` | `src/agent/CLAUDE.md` |
| `src/channels/web/` | `src/channels/web/CLAUDE.md` |
| `src/db/` | `src/db/CLAUDE.md` |
| `src/llm/` | `src/llm/CLAUDE.md` |
| `src/setup/` | `src/setup/README.md` |
| `src/tools/` | `src/tools/README.md` |
| `src/workspace/` | `src/workspace/README.md` |
| `tests/e2e/` | `tests/e2e/CLAUDE.md` |
### Tool Implementation
```rust
#[async_trait]
impl Tool for MyTool {
fn name(&self) -> &str { "my_tool" }
fn description(&self) -> &str { "Does something useful" }
fn parameters_schema(&self) -> serde_json::Value {
serde_json::json!({
"type": "object",
"properties": {
"param": { "type": "string", "description": "A parameter" }
},
"required": ["param"]
})
}
## Job State Machine
async fn execute(&self, params: serde_json::Value, ctx: &JobContext)
-> Result<ToolOutput, ToolError>
{
let start = std::time::Instant::now();
// ... do work ...
Ok(ToolOutput::text("result", start.elapsed()))
}
fn requires_sanitization(&self) -> bool { true } // External data
}
```
### State Transitions
Job states follow a defined state machine in `context/state.rs`:
```
Pending -> InProgress -> Completed -> Submitted -> Accepted
\-> Failed
@@ -354,291 +197,17 @@ Pending -> InProgress -> Completed -> Submitted -> Accepted
\-> Failed
```
### Code Style
## Skills System
- Use `crate::` imports, not `super::`
- No `pub use` re-exports unless exposing to downstream consumers
- Prefer strong types over strings (enums, newtypes)
- Keep functions focused, extract helpers when logic is reused
- Comments for non-obvious logic only
SKILL.md files extend the agent's prompt with domain-specific instructions. See `.claude/rules/skills.md` for full details.
### Review & Fix Discipline
Hard-won lessons from code review -- follow these when fixing bugs or addressing review feedback.
**Fix the pattern, not just the instance:** When a reviewer flags a bug (e.g., TOCTOU race in INSERT + SELECT-back), search the entire codebase for all instances of that same pattern. A fix in `SecretsStore::create()` that doesn't also fix `WasmToolStore::store()` is half a fix.
**Propagate architectural fixes to satellite types:** If a core type changes its concurrency model (e.g., `LibSqlBackend` switches to connection-per-operation), every type that was handed a resource from the old model (e.g., `LibSqlSecretsStore`, `LibSqlWasmToolStore` holding a single `Connection`) must also be updated. Grep for the old type across the codebase.
**Schema translation is more than DDL:** When translating a database schema between backends (PostgreSQL to libSQL, etc.), check for:
- **Indexes** -- diff `CREATE INDEX` statements between the two schemas
- **Seed data** -- check for `INSERT INTO` in migrations (e.g., `leak_detection_patterns`)
- **Semantic differences** -- document where SQL functions behave differently (e.g., `json_patch` vs `jsonb_set`)
**Feature flag testing:** When adding feature-gated code, test compilation with each feature in isolation:
```bash
cargo check # default features
cargo check --no-default-features --features libsql # libsql only
cargo check --all-features # all features
```
Dead code behind the wrong `#[cfg]` gate will only show up when building with a single feature.
**Regression test with every fix:** Every bug fix must include a test that would have caught the bug. Add a `#[test]` or `#[tokio::test]` that reproduces the original failure. Exempt: changes limited to `src/channels/web/static/` or `.md` files. Use `[skip-regression-check]` in commit message or PR label if genuinely not feasible. The `commit-msg` hook and CI workflow enforce this automatically.
**Zero clippy warnings policy:** Fix ALL clippy warnings before committing, including pre-existing ones in files you didn't change. Never leave warnings behind — treat `cargo clippy` output as a zero-tolerance gate.
**Mechanical verification before committing:** Run these checks on changed files before committing:
- `cargo clippy --all --benches --tests --examples --all-features` -- zero warnings
- `grep -rnE '\.unwrap\(|\.expect\(' <files>` -- no panics in production
- `grep -rn 'super::' <files>` -- use `crate::` imports
- If you fixed a pattern bug, `grep` for other instances of that pattern across `src/`
- Fix commits must include regression tests (enforced by `commit-msg` hook; bypass with `[skip-regression-check]`)
- **Trust model**: Trusted (user-placed in `~/.ironclaw/skills/` or workspace `skills/`, full tool access) vs Installed (registry, read-only tools)
- **Selection pipeline**: gating (check bin/env/config requirements) -> scoring (keywords/patterns/tags) -> budget (fit within `SKILLS_MAX_TOKENS`) -> attenuation (trust-based tool ceiling)
- **Skill tools**: `skill_list`, `skill_search`, `skill_install`, `skill_remove`
## Configuration
Environment variables (see `.env.example`):
```bash
# Database backend (default: postgres)
DATABASE_BACKEND=postgres # or "libsql" / "turso"
DATABASE_URL=postgres://user:pass@localhost/ironclaw
LIBSQL_PATH=~/.ironclaw/ironclaw.db # libSQL local path (default)
# LIBSQL_URL=libsql://xxx.turso.io # Turso cloud (optional)
# LIBSQL_AUTH_TOKEN=xxx # Required with LIBSQL_URL
# NEAR AI (when LLM_BACKEND=nearai, the default)
# Two auth modes: session token (default) or API key
# Session token auth (default): uses browser OAuth on first run
NEARAI_SESSION_TOKEN=sess_... # hosting providers: set this
NEARAI_BASE_URL=https://private.near.ai
# API key auth: set NEARAI_API_KEY, base URL defaults to cloud-api.near.ai
# NEARAI_API_KEY=... # API key from cloud.near.ai
NEARAI_MODEL=claude-3-5-sonnet-20241022
# Agent settings
AGENT_NAME=ironclaw
MAX_PARALLEL_JOBS=5
# Embeddings (for semantic memory search)
OPENAI_API_KEY=sk-... # For OpenAI embeddings
# Or use NEAR AI embeddings:
# EMBEDDING_PROVIDER=nearai
# EMBEDDING_ENABLED=true
EMBEDDING_MODEL=text-embedding-3-small # or text-embedding-3-large
# Heartbeat (proactive periodic execution)
HEARTBEAT_ENABLED=true
HEARTBEAT_INTERVAL_SECS=1800 # 30 minutes
HEARTBEAT_NOTIFY_CHANNEL=tui
HEARTBEAT_NOTIFY_USER=default
# Web gateway
GATEWAY_ENABLED=true
GATEWAY_HOST=127.0.0.1
GATEWAY_PORT=3001
GATEWAY_AUTH_TOKEN=changeme # Required for API access
GATEWAY_USER_ID=default
# Docker sandbox
SANDBOX_ENABLED=true
SANDBOX_IMAGE=ironclaw-worker:latest
SANDBOX_MEMORY_LIMIT_MB=512
SANDBOX_TIMEOUT_SECS=1800
SANDBOX_CPU_LIMIT=1.0 # CPU cores per container
SANDBOX_NETWORK_PROXY=true # Enable network proxy for containers
SANDBOX_PROXY_PORT=8080 # Proxy listener port
SANDBOX_DEFAULT_POLICY=workspace_write # ReadOnly, WorkspaceWrite, FullAccess
# Claude Code mode (runs inside sandbox containers)
CLAUDE_CODE_ENABLED=false
CLAUDE_CODE_MODEL=claude-sonnet-4-20250514
CLAUDE_CODE_MAX_TURNS=50
CLAUDE_CODE_CONFIG_DIR=/home/worker/.claude
# Routines (scheduled/reactive execution)
ROUTINES_ENABLED=true
ROUTINES_CRON_INTERVAL=60 # Tick interval in seconds
ROUTINES_MAX_CONCURRENT=3
# Skills system
SKILLS_ENABLED=true
SKILLS_MAX_TOKENS=4000 # Max prompt budget per turn
SKILLS_CATALOG_URL=https://clawhub.dev # ClawHub registry URL
SKILLS_AUTO_DISCOVER=true # Scan skill directories on startup
# Tinfoil private inference
TINFOIL_API_KEY=... # Required when LLM_BACKEND=tinfoil
TINFOIL_MODEL=kimi-k2-5 # Default model
# Tunnel (public internet exposure for webhooks)
TUNNEL_URL=https://abc123.ngrok.io # Static public URL (manual tunnel)
# Or use a managed tunnel provider:
TUNNEL_PROVIDER=none # none (default), cloudflare, tailscale, ngrok, custom
TUNNEL_CF_TOKEN=... # Required for TUNNEL_PROVIDER=cloudflare
TUNNEL_NGROK_TOKEN=... # Required for TUNNEL_PROVIDER=ngrok
# TUNNEL_NGROK_DOMAIN=... # Custom domain (paid ngrok plan)
# TUNNEL_TS_FUNNEL=true # Use tailscale funnel (public) vs serve (tailnet)
TUNNEL_CUSTOM_COMMAND=... # Command with {host}/{port} for custom providers
# Observability backend
OBSERVABILITY_BACKEND=none # none/noop (default) or log
```
### LLM Providers
Backends: `nearai` (default), `openai`, `anthropic`, `ollama`, `openai_compatible`, `tinfoil` — set via `LLM_BACKEND`. See [src/llm/CLAUDE.md](src/llm/CLAUDE.md) for per-provider auth and configuration details.
## Database
Dual-backend persistence (PostgreSQL + libSQL/Turso). **All new persistence features must support both backends** — see [src/db/CLAUDE.md](src/db/CLAUDE.md) for schema, SQL dialect differences, adding operations, and libSQL limitations.
Implement every new operation in both `src/db/postgres.rs` and `src/db/libsql/mod.rs`. Test in isolation:
```bash
cargo check # postgres (default)
cargo check --no-default-features --features libsql # libsql only
cargo check --all-features # both
```
Database configuration: see Configuration section above.
## Safety Layer
All external tool output passes through `SafetyLayer`:
1. **Sanitizer** - Detects injection patterns, escapes dangerous content
2. **Validator** - Checks length, encoding, forbidden patterns
3. **Policy** - Rules with severity (Critical/High/Medium/Low) and actions (Block/Warn/Review/Sanitize)
4. **Leak Detector** - Scans for 15+ secret patterns (API keys, tokens, private keys, connection strings) at two points: tool output before it reaches the LLM, and LLM responses before they reach the user. Actions per pattern: Block (reject entirely), Redact (mask the secret), or Warn (flag but allow)
Tool outputs are wrapped before reaching LLM:
```xml
<tool_output name="search" sanitized="true">
[escaped content]
</tool_output>
```
### Shell Environment Scrubbing
The shell tool (`src/tools/builtin/shell.rs`) scrubs sensitive environment variables before executing commands, preventing secrets from leaking through `env`, `printenv`, or `$VAR` expansion. The sanitizer (`src/safety/sanitizer.rs`) also detects command injection patterns (chained commands, subshells, path traversal) and blocks or escapes them based on policy rules.
## Skills System
Skills are SKILL.md files that extend the agent's prompt with domain-specific instructions. Each skill is a YAML frontmatter block (metadata, activation criteria, required tools) followed by a markdown body that gets injected into the LLM context when the skill activates.
### Trust Model
| Trust Level | Source | Tool Access |
|-------------|--------|-------------|
| **Trusted** | User-placed in `~/.ironclaw/skills/` or workspace `skills/` | All tools available to the agent |
| **Installed** | Downloaded from ClawHub registry | Read-only tools only (no shell, file write, HTTP) |
### SKILL.md Format
```yaml
---
name: my-skill
version: 0.1.0
description: Does something useful
activation:
patterns:
- "deploy to.*production"
keywords:
- "deployment"
max_context_tokens: 2000
metadata:
openclaw:
requires:
bins: [docker, kubectl]
env: [KUBECONFIG]
---
# Deployment Skill
Instructions for the agent when this skill activates...
```
### Selection Pipeline
1. **Gating** -- Check binary/env/config requirements; skip skills whose prerequisites are missing
2. **Scoring** -- Deterministic scoring against message content using keywords, tags, and regex patterns
3. **Budget** -- Select top-scoring skills that fit within `SKILLS_MAX_TOKENS` prompt budget
4. **Attenuation** -- Apply trust-based tool ceiling; installed skills lose access to dangerous tools
### Skill Tools
Four built-in tools for managing skills at runtime:
- **`skill_list`** -- List all discovered skills with trust level and status
- **`skill_search`** -- Search ClawHub registry for available skills
- **`skill_install`** -- Download and install a skill from ClawHub
- **`skill_remove`** -- Remove an installed skill
### Skill Directories
- `~/.ironclaw/skills/` -- User's global skills (trusted)
- `<workspace>/skills/` -- Per-workspace skills (trusted)
- `~/.ironclaw/installed_skills/` -- Registry-installed skills (installed trust)
### Testing Skills
- `skills/web-ui-test/` -- Manual test checklist for the web gateway UI via Claude for Chrome extension. Covers connection, chat, skills search/install/remove, and other tabs.
Skills configuration: see Configuration section above.
## Docker Sandbox
The `src/sandbox/` module provides Docker-based isolation for job execution with a network proxy that controls outbound access and injects credentials.
### Sandbox Policies
| Policy | Filesystem | Network | Use Case |
|--------|-----------|---------|----------|
| **ReadOnly** | Read-only workspace mount | Allowlisted domains only | Analysis, code review |
| **WorkspaceWrite** | Read-write workspace mount | Allowlisted domains only | Code generation, file edits |
| **FullAccess** | Full filesystem | Unrestricted | Trusted admin tasks |
### Network Proxy
Containers route all HTTP/HTTPS traffic through a host-side proxy (`src/sandbox/proxy/`):
- **Domain allowlist** -- Only allowlisted domains are reachable (default: package registries, docs sites, GitHub, common APIs)
- **Credential injection** -- The `CredentialResolver` trait injects auth headers into proxied requests so secrets never enter the container environment
- **CONNECT tunnel** -- HTTPS traffic uses CONNECT method; the proxy validates the target domain against the allowlist before establishing the tunnel
- **Policy decisions** -- The `NetworkPolicyDecider` trait allows custom logic for allow/deny/inject decisions per request
### Zero-Exposure Credential Model
Secrets (API keys, tokens) are stored encrypted on the host and injected into HTTP requests by the proxy at transit time. Container processes never have access to raw credential values, preventing exfiltration even if container code is compromised.
Sandbox configuration: see Configuration section above.
## Testing
Tests are in `mod tests {}` blocks at the bottom of each file. Run specific module tests:
```bash
cargo test safety::sanitizer::tests
cargo test tools::registry::tests
```
Key test patterns:
- Unit tests for pure functions
- Async tests with `#[tokio::test]`
- No mocks, prefer real implementations or stubs
## Current Limitations / TODOs
1. **Domain-specific tools** - `marketplace.rs`, `restaurant.rs`, `taskrabbit.rs`, `ecommerce.rs` return placeholder responses; need real API integrations
2. **Integration tests** - Need testcontainers setup for PostgreSQL
3. **MCP stdio transport** - Only HTTP transport implemented
4. **WIT bindgen integration** - Auto-extract tool description/schema from WASM modules (stubbed)
5. **Capability granting after tool build** - Built tools get empty capabilities; need UX for granting HTTP/secrets access
6. **Tool versioning workflow** - No version tracking or rollback for dynamically built tools
7. **Full channel status view** - Gateway status widget exists, but no per-channel connection dashboard
8. **Observability backends** - Only `log` and `noop` implemented; OpenTelemetry/Prometheus not yet supported
## Tool Architecture
**Keep tool-specific logic out of the main agent codebase.** The main agent provides generic infrastructure; tools are self-contained units that declare their requirements through `capabilities.json` files (API endpoints, credentials, rate limits, auth setup). Service-specific auth flows, CLI commands, and configuration do not belong in the main agent.
Tools can be built as **WASM** (sandboxed, credential-injected, single binary) or **MCP servers** (ecosystem of pre-built servers, any language, but no sandbox). Both are first-class via `ironclaw tool install`. Auth is declared in capabilities files with OAuth and manual token entry support.
See `src/tools/README.md` for full tool architecture, adding new tools (built-in Rust and WASM), auth JSON examples, and WASM vs MCP decision guide.
See `.env.example` for all environment variables. LLM backends (`nearai`, `openai`, `anthropic`, `ollama`, `openai_compatible`, `tinfoil`, `bedrock`) documented in `src/llm/CLAUDE.md`.
## Adding a New Channel
@@ -647,48 +216,24 @@ See `src/tools/README.md` for full tool architecture, adding new tools (built-in
3. Add config in `src/config/channels.rs`
4. Wire up in `src/app.rs` channel setup section
## Workspace & Memory
Persistent memory with hybrid search (FTS + vector via RRF). Four tools: `memory_search`, `memory_write`, `memory_read`, `memory_tree`. Identity files (AGENTS.md, SOUL.md, USER.md, IDENTITY.md) injected into system prompt. Heartbeat system runs proactive periodic execution (default: 30 minutes), reading `HEARTBEAT.md` and notifying via channel if findings. See `src/workspace/README.md`.
## Debugging
```bash
# Verbose logging
RUST_LOG=ironclaw=trace cargo run
# Just the agent module
RUST_LOG=ironclaw::agent=debug cargo run
# With HTTP request logging
RUST_LOG=ironclaw=debug,tower_http=debug cargo run
RUST_LOG=ironclaw=trace cargo run # verbose
RUST_LOG=ironclaw::agent=debug cargo run # agent module only
RUST_LOG=ironclaw=debug,tower_http=debug cargo run # + HTTP request logging
```
## Module Specifications
## Current Limitations
Some modules have a `README.md` that serves as the authoritative specification
for that module's behavior. When modifying code in a module that has a spec:
1. **Read the spec first** before making changes
2. **Code follows spec**: if the spec says X, the code must do X
3. **Update both sides**: if you change behavior, update the spec to match;
if you're implementing a spec change, update the code to match
4. **Spec is the tiebreaker**: when code and spec disagree, the spec is correct
(unless the spec is clearly outdated, in which case fix the spec first)
| Module | Spec File |
|--------|-----------|
| `src/setup/` | `src/setup/README.md` |
| `src/workspace/` | `src/workspace/README.md` |
| `src/tools/` | `src/tools/README.md` |
| `src/agent/` | `src/agent/CLAUDE.md` |
| `src/channels/web/` | `src/channels/web/CLAUDE.md` |
| `src/db/` | `src/db/CLAUDE.md` |
| `src/llm/` | `src/llm/CLAUDE.md` |
| `tests/e2e/` | `tests/e2e/CLAUDE.md` |
## Workspace & Memory System
OpenClaw-inspired persistent memory with a flexible filesystem-like structure. Principle: "Memory is database, not RAM" -- if you want to remember something, write it explicitly. Uses hybrid search combining FTS (keyword) + vector (semantic) via Reciprocal Rank Fusion.
Four memory tools for LLM use: `memory_search` (hybrid search -- call before answering questions about prior work), `memory_write`, `memory_read`, `memory_tree`. Identity files (AGENTS.md, SOUL.md, USER.md, IDENTITY.md) are injected into the LLM system prompt.
The heartbeat system runs proactive periodic execution (default: 30 minutes), reading `HEARTBEAT.md` and notifying via channel if findings are detected.
See `src/workspace/README.md` for full API documentation, filesystem structure, hybrid search details, chunking strategy, and heartbeat system.
1. Domain-specific tools (`marketplace.rs`, `restaurant.rs`, etc.) are stubs
2. Integration tests need testcontainers for PostgreSQL
3. MCP: no streaming support; stdio/HTTP/Unix transports all use request-response
4. WIT bindgen: auto-extract tool schema from WASM is stubbed
5. Built tools get empty capabilities; need UX for granting access
6. No tool versioning or rollback
7. Observability: only `log` and `noop` backends (no OpenTelemetry)
Generated
+707 -23
View File
File diff suppressed because it is too large Load Diff
+9 -1
View File
@@ -56,7 +56,7 @@ rustls = { version = "0.23", optional = true, default-features = false }
rustls-native-certs = { version = "0.8", optional = true }
# Database - libSQL/Turso (optional embedded database)
libsql = { version = "0.6", optional = true, default-features = false, features = ["core", "replication"] }
libsql = { version = "0.6", optional = true, default-features = false, features = ["core", "replication", "remote", "tls"] }
# Error handling
thiserror = "2"
@@ -73,6 +73,8 @@ toml = "0.8"
# Core types
uuid = { version = "1", features = ["v4", "v5", "serde"] }
chrono = { version = "0.4", features = ["serde"] }
chrono-tz = "0.10"
iana-time-zone = "0.1"
rust_decimal = { version = "1", features = ["serde", "serde-with-str", "maths"] }
rust_decimal_macros = "1"
@@ -140,6 +142,11 @@ subtle = "2" # Constant-time comparisons for token validation
# Multi-provider LLM support
rig-core = "0.30"
# AWS Bedrock (native Converse API, opt-in via --features bedrock)
aws-config = { version = "1", features = ["behavior-version-latest"], optional = true }
aws-sdk-bedrockruntime = { version = "1", optional = true }
aws-smithy-types = { version = "1", optional = true }
# Docker sandbox
bollard = "0.18"
@@ -201,6 +208,7 @@ postgres = [
libsql = ["dep:libsql"]
integration = []
html-to-markdown = ["dep:html-to-markdown-rs", "dep:readabilityrs"]
bedrock = ["dep:aws-config", "dep:aws-sdk-bedrockruntime", "dep:aws-smithy-types"]
[[test]]
name = "html_to_markdown"
+47 -36
View File
@@ -10,6 +10,8 @@ This document tracks feature parity between IronClaw (Rust implementation) and O
- 🚫 Out of scope (intentionally skipped)
- N/A (not applicable to Rust implementation)
**Last reviewed against OpenClaw PRs:** 2026-03-10 (merged 2026-02-24 through 2026-03-10)
---
## 1. Architecture
@@ -43,7 +45,7 @@ This document tracks feature parity between IronClaw (Rust implementation) and O
| launchd/systemd integration | ✅ | ❌ | |
| Bonjour/mDNS discovery | ✅ | ❌ | |
| Tailscale integration | ✅ | ❌ | |
| Health check endpoints | ✅ | ✅ | /api/health + /api/gateway/status |
| Health check endpoints | ✅ | ✅ | /api/health + /api/gateway/status + /healthz + /readyz, with channel-backed readiness probes |
| `doctor` diagnostics | ✅ | ❌ | |
| Agent event broadcast | ✅ | 🚧 | SSE broadcast manager exists (SseManager) but tool/job-state events not fully wired |
| Channel health monitor | ✅ | ❌ | Auto-restart with configurable interval |
@@ -66,17 +68,17 @@ This document tracks feature parity between IronClaw (Rust implementation) and O
| REPL (simple) | ✅ | ✅ | - | For testing |
| WASM channels | ❌ | ✅ | - | IronClaw innovation |
| WhatsApp | ✅ | ❌ | P1 | Baileys (Web), same-phone mode with echo detection |
| Telegram | ✅ | ✅ | - | WASM channel(MTProto), DM pairing, caption, /start, bot_username |
| Telegram | ✅ | ✅ | - | WASM channel(MTProto), DM pairing, caption, /start, bot_username, DM topics |
| Discord | ✅ | ❌ | P2 | discord.js, thread parent binding inheritance |
| Signal | ✅ | ✅ | P2 | signal-cli daemonPC, SSE listener HTTP/JSON-R, user/group allowlists, DM pairing |
| Slack | ✅ | ✅ | - | WASM tool |
| iMessage | ✅ | ❌ | P3 | BlueBubbles or Linq recommended |
| Linq | ✅ | ❌ | P3 | Real iMessage via API, no Mac required |
| Feishu/Lark | ✅ | ❌ | P3 | Bitable create app/field tools |
| Feishu/Lark | ✅ | ❌ | P3 | Bitable create app/field tools, Docx table/image/file actions, rich-text media extraction |
| LINE | ✅ | ❌ | P3 | |
| WebChat | ✅ | ✅ | - | Web gateway chat |
| Matrix | ✅ | ❌ | P3 | E2EE support |
| Mattermost | ✅ | ❌ | P3 | Emoji reactions |
| Mattermost | ✅ | ❌ | P3 | Emoji reactions, interactive buttons, model picker |
| Google Chat | ✅ | ❌ | P3 | |
| MS Teams | ✅ | ❌ | P3 | |
| Twitch | ✅ | ❌ | P3 | |
@@ -92,6 +94,8 @@ This document tracks feature parity between IronClaw (Rust implementation) and O
| User message reactions | ✅ | ❌ | Surface inbound reactions |
| sendPoll | ✅ | ❌ | Poll creation via agent |
| Cron/heartbeat topic targeting | ✅ | ❌ | Messages land in correct topic |
| DM topics support | ✅ | ❌ | Agent/topic bindings in DMs and agent-scoped SessionKeys |
| Persistent ACP topic binding | ✅ | ❌ | ACP harness sessions can pin to Telegram forum or DM topics |
### Discord-Specific Features (since Feb 2025)
@@ -107,21 +111,36 @@ This document tracks feature parity between IronClaw (Rust implementation) and O
|---------|----------|----------|-------|
| Streaming draft replies | ✅ | ❌ | Partial replies via draft message updates |
| Configurable stream modes | ✅ | ❌ | Per-channel stream behavior |
| Thread ownership | ✅ | ❌ | Thread-level ownership tracking |
| Thread ownership | ✅ | ❌ | Thread-level ownership tracking plus reply participation memory |
| Download-file action | ✅ | ❌ | On-demand attachment downloads via message actions |
### Mattermost-Specific Features (since Mar 2026)
| Feature | OpenClaw | IronClaw | Notes |
|---------|----------|----------|-------|
| Interactive buttons | ✅ | ❌ | Clickable message buttons with signed callback flow |
| Interactive model picker | ✅ | ❌ | In-channel provider/model chooser |
### Feishu/Lark-Specific Features (since Mar 2026)
| Feature | OpenClaw | IronClaw | Notes |
|---------|----------|----------|-------|
| Doc/table actions | ✅ | ❌ | `feishu_doc` supports tables, positional insert, color_text, image upload, and file upload |
| Rich-text embedded media extraction | ✅ | ❌ | Pull video/media attachments from post messages |
### Channel Features
| Feature | OpenClaw | IronClaw | Notes |
|---------|----------|----------|-------|
| DM pairing codes | ✅ | ✅ | `ironclaw pairing list/approve`, host APIs |
| Allowlist/blocklist | ✅ | 🚧 | allow_from + pairing store |
| Allowlist/blocklist | ✅ | 🚧 | `allow_from` + pairing store + hardened command/group allowlists |
| Self-message bypass | ✅ | ❌ | Own messages skip pairing |
| Mention-based activation | ✅ | ✅ | bot_username + respond_to_all_group_messages |
| Per-group tool policies | ✅ | ❌ | Allow/deny specific tools |
| Thread isolation | ✅ | ✅ | Separate sessions per thread |
| Per-channel media limits | ✅ | | Attachment type in WIT; max 10 per msg, 20MB total, MIME allowlist |
| Typing indicators | ✅ | 🚧 | TUI + Telegram typing/actionable status prompts; richer parity pending |
| Per-channel ackReaction config | ✅ | ❌ | Customizable acknowledgement reactions |
| Thread isolation | ✅ | ✅ | Separate sessions per thread/topic |
| Per-channel media limits | ✅ | 🚧 | Caption support plus `mediaMaxMb` enforcement for WhatsApp, Telegram, and Discord |
| Typing indicators | ✅ | 🚧 | TUI + channel typing, with configurable silence timeout; richer parity pending |
| Per-channel ackReaction config | ✅ | ❌ | Customizable acknowledgement reactions/scopes |
| Group session priming | ✅ | ❌ | Member roster injected for context |
| Sender_id in trusted metadata | ✅ | ❌ | Exposed in system metadata |
@@ -138,7 +157,8 @@ This document tracks feature parity between IronClaw (Rust implementation) and O
| `gateway start/stop` | ✅ | ❌ | P2 | |
| `onboard` (wizard) | ✅ | ✅ | - | Interactive setup |
| `tui` | ✅ | ✅ | - | Ratatui TUI |
| `config` | ✅ | ✅ | - | Read/write config |
| `config` | ✅ | ✅ | - | Read/write config plus validate/path helpers |
| `backup` | ✅ | ❌ | P3 | Create/verify local backup archives |
| `channels` | ✅ | ❌ | P2 | Channel management |
| `models` | ✅ | 🚧 | - | Model selector in TUI |
| `status` | ✅ | ✅ | - | System status (enriched session details) |
@@ -177,14 +197,15 @@ This document tracks feature parity between IronClaw (Rust implementation) and O
| Global sessions | ✅ | ❌ | Optional shared context |
| Session pruning | ✅ | ❌ | Auto cleanup old sessions |
| Context compaction | ✅ | ✅ | Auto summarization |
| Compaction model override | ✅ | ❌ | Use a dedicated provider/model for summarization only |
| Post-compaction read audit | ✅ | ❌ | Layer 3: workspace rules appended to summaries |
| Post-compaction context injection | ✅ | ❌ | Workspace context as system event |
| Custom system prompts | ✅ | ✅ | Template variables, safety guardrails |
| Skills (modular capabilities) | ✅ | ✅ | Prompt-based skills with trust gating, attenuation, activation criteria, catalog, selector |
| Skill routing blocks | ✅ | 🚧 | ActivationCriteria (keywords, patterns, tags) but no "Use when / Don't use when" blocks |
| Skill path compaction | ✅ | ❌ | ~ prefix to reduce prompt tokens |
| Thinking modes (low/med/high) | ✅ | ❌ | Configurable reasoning depth |
| Per-model thinkingDefault override | ✅ | ❌ | Override thinking level per model |
| Thinking modes (off/minimal/low/medium/high/xhigh/adaptive) | ✅ | ❌ | Configurable reasoning depth |
| Per-model thinkingDefault override | ✅ | ❌ | Override thinking level per model; Anthropic Claude 4.6 defaults to adaptive |
| Block-level streaming | ✅ | ❌ | |
| Tool-level streaming | ✅ | ❌ | |
| Z.AI tool_stream | ✅ | ❌ | Real-time tool call streaming |
@@ -213,8 +234,8 @@ This document tracks feature parity between IronClaw (Rust implementation) and O
| Provider | OpenClaw | IronClaw | Priority | Notes |
|----------|----------|----------|----------|-------|
| NEAR AI | ✅ | ✅ | - | Primary provider |
| Anthropic (Claude) | ✅ | 🚧 | - | Via NEAR AI proxy; Opus 4.5, Sonnet 4, Sonnet 4.6 |
| OpenAI | ✅ | 🚧 | - | Via NEAR AI proxy |
| Anthropic (Claude) | ✅ | 🚧 | - | Via NEAR AI proxy; Opus 4.5, Sonnet 4, Sonnet 4.6, adaptive thinking default |
| OpenAI | ✅ | 🚧 | - | Via NEAR AI proxy; GPT-5.4 + Codex OAuth |
| AWS Bedrock | ✅ | ❌ | P3 | |
| Google Gemini | ✅ | ❌ | P3 | |
| NVIDIA API | ✅ | ❌ | P3 | New provider |
@@ -238,7 +259,7 @@ This document tracks feature parity between IronClaw (Rust implementation) and O
| Per-session model override | ✅ | ✅ | Model selector in TUI |
| Model selection UI | ✅ | ✅ | TUI keyboard shortcut |
| Per-model thinkingDefault | ✅ | ❌ | Override thinking level per model in config |
| 1M context beta header | ✅ | ❌ | Anthropic extended context support |
| 1M context support | ✅ | ❌ | Anthropic extended context beta + OpenAI Codex GPT-5.4 1M context |
### Owner: _Unassigned_
@@ -248,32 +269,20 @@ This document tracks feature parity between IronClaw (Rust implementation) and O
| Feature | OpenClaw | IronClaw | Priority | Notes |
|---------|----------|----------|----------|-------|
| WIT inbound-attachment type | N/A | ✅ | P1 | `inbound-attachment` record in channel-host (id, mime_type, filename, size_bytes, source_url, storage_key, extracted_text) |
| WIT outbound attachment type | N/A | ✅ | P1 | `attachment` record in channel (filename, mime_type, data) on `agent-response` |
| WIT on-broadcast export | N/A | ✅ | P1 | Proactive message sending without prior incoming message |
| IncomingMessage attachments | N/A | ✅ | P1 | `IncomingAttachment` struct on `IncomingMessage`, populated from WASM channels |
| OutgoingResponse attachments | N/A | ✅ | P1 | File paths on `OutgoingResponse`, read from disk and sent as WIT attachments |
| Attachment security (size/MIME) | N/A | ✅ | P1 | Inbound: max 10, 20MB total, MIME allowlist. Outbound: 50MB total |
| Telegram media parsing | ✅ | ✅ | P1 | Photo, document, audio, video, voice, sticker parsed and emitted as attachments |
| Telegram media sending | ✅ | ✅ | P1 | sendPhoto/sendDocument multipart upload, auto photo→document fallback >10MB |
| Slack file parsing | ✅ | ✅ | P1 | `files` array from Events API parsed into attachments |
| WhatsApp media parsing | ✅ | ✅ | P1 | Image, audio, video, document parsed with caption as extracted_text |
| Discord attachment parsing | ✅ | ❌ | P2 | Discord interaction payloads don't include file attachments (needs message events) |
| HTTP tool save_to | N/A | ✅ | P1 | Download binary files to /tmp/ for attachment sending (50MB limit, path traversal protection) |
| Credential env var fallback | N/A | ✅ | P2 | Channels can use env vars (e.g., TELEGRAM_BOT_TOKEN) when secrets store not configured |
| Image processing (Sharp) | ✅ | ❌ | P2 | Resize, format convert |
| Configurable image resize dims | ✅ | ❌ | P2 | Per-agent dimension config |
| Multiple images per tool call | ✅ | ❌ | P2 | Single tool invocation, multiple images |
| Audio transcription | ✅ | ❌ | P2 | |
| Video support | ✅ | ❌ | P3 | |
| PDF parsing | ✅ | ❌ | P2 | pdfjs-dist |
| MIME detection | ✅ | | P2 | MIME allowlist in host validates attachment types |
| PDF analysis tool | ✅ | ❌ | P2 | Native Anthropic/Gemini path with text/image extraction fallback |
| PDF parsing | ✅ | | P2 | `pdfjs-dist` fallback path |
| MIME detection | ✅ | ❌ | P2 | |
| Media caching | ✅ | ❌ | P3 | |
| Vision model integration | ✅ | ❌ | P2 | Image understanding |
| TTS (Edge TTS) | ✅ | ❌ | P3 | Text-to-speech |
| TTS (OpenAI) | ✅ | ❌ | P3 | |
| Incremental TTS playback | ✅ | ❌ | P3 | iOS progressive playback |
| Sticker-to-image | ✅ | | P3 | Telegram stickers emitted as image/webp attachments |
| Sticker-to-image | ✅ | | P3 | Telegram stickers |
### Owner: _Unassigned_
@@ -289,7 +298,8 @@ This document tracks feature parity between IronClaw (Rust implementation) and O
| Workspace-relative install | ✅ | ✅ | ~/.ironclaw/tools/ |
| Channel plugins | ✅ | ✅ | WASM channels |
| Auth plugins | ✅ | ❌ | |
| Memory plugins | ✅ | ❌ | Custom backends |
| Memory plugins | ✅ | ❌ | Custom backends + selectable memory slot |
| Context-engine plugins | ✅ | ❌ | Custom context management + subagent/context hooks |
| Tool plugins | ✅ | ✅ | WASM tools |
| Hook plugins | ✅ | ✅ | Declarative hooks from extension capabilities |
| Provider plugins | ✅ | ❌ | |
@@ -311,7 +321,7 @@ This document tracks feature parity between IronClaw (Rust implementation) and O
| JSON5 support | ✅ | ❌ | Comments, trailing commas |
| YAML alternative | ✅ | ❌ | |
| Environment variable interpolation | ✅ | ✅ | `${VAR}` |
| Config validation/schema | ✅ | ✅ | Type-safe Config struct |
| Config validation/schema | ✅ | ✅ | Type-safe Config struct + `openclaw config validate` |
| Hot-reload | ✅ | ❌ | |
| Legacy migration | ✅ | | |
| State directory | ✅ `~/.openclaw-state/` | ✅ `~/.ironclaw/` | |
@@ -418,6 +428,7 @@ This document tracks feature parity between IronClaw (Rust implementation) and O
| Feature | OpenClaw | IronClaw | Priority | Notes |
|---------|----------|----------|----------|-------|
| Cron jobs | ✅ | ✅ | - | Routines with cron trigger |
| Per-job model fallback override | ✅ | ❌ | P2 | `payload.fallbacks` overrides agent-level fallbacks |
| Cron stagger controls | ✅ | ❌ | P3 | Default stagger for scheduled jobs |
| Cron finished-run webhook | ✅ | ❌ | P3 | Webhook on job completion |
| Timezone support | ✅ | ✅ | - | Via cron expressions |
@@ -471,10 +482,10 @@ This document tracks feature parity between IronClaw (Rust implementation) and O
| Elevated mode | ✅ | ❌ | |
| Safe bins allowlist | ✅ | ❌ | Hardened path trust |
| LD*/DYLD* validation | ✅ | ❌ | |
| Path traversal prevention | ✅ | ✅ | Including config includes (OC-06) |
| Path traversal prevention | ✅ | ✅ | Including config includes (OC-06) + workspace-only tool mounts |
| Credential theft via env injection | ✅ | 🚧 | Shell env scrubbing + command injection detection; no full OC-09 defense |
| Session file permissions (0o600) | ✅ | ✅ | Session token file set to 0o600 in llm/session.rs |
| Skill download path restriction | ✅ | ❌ | Prevent arbitrary write targets |
| Skill download path restriction | ✅ | ❌ | Validated download roots prevent arbitrary write targets |
| Webhook signature verification | ✅ | ✅ | |
| Media URL validation | ✅ | ❌ | |
| Prompt injection defense | ✅ | ✅ | Pattern detection, sanitization |
+5
View File
@@ -14,6 +14,11 @@
<a href="https://www.reddit.com/r/ironclawAI/"><img src="https://img.shields.io/badge/Reddit-r%2FironclawAI-FF4500?style=flat&logo=reddit&logoColor=white" alt="Reddit: r/ironclawAI" /></a>
</p>
<p align="center">
<a href="README.md">English</a> |
<a href="README.zh-CN.md">简体中文</a>
</p>
<p align="center">
<a href="#philosophy">Philosophy</a> •
<a href="#features">Features</a> •
+319
View File
@@ -0,0 +1,319 @@
<p align="center">
<img src="ironclaw.png?v=2" alt="IronClaw" width="200"/>
</p>
<h1 align="center">IronClaw</h1>
<p align="center">
<strong>安全可靠的个人 AI 助手,始终站在你这边</strong>
</p>
<p align="center">
<a href="#license"><img src="https://img.shields.io/badge/license-MIT%20OR%20Apache%202.0-blue.svg" alt="License: MIT OR Apache-2.0" /></a>
<a href="https://t.me/ironclawAI"><img src="https://img.shields.io/badge/Telegram-%40ironclawAI-26A5E4?style=flat&logo=telegram&logoColor=white" alt="Telegram: @ironclawAI" /></a>
<a href="https://www.reddit.com/r/ironclawAI/"><img src="https://img.shields.io/badge/Reddit-r%2FironclawAI-FF4500?style=flat&logo=reddit&logoColor=white" alt="Reddit: r/ironclawAI" /></a>
</p>
<p align="center">
<a href="README.md">English</a> |
<a href="README.zh-CN.md">简体中文</a>
</p>
<p align="center">
<a href="#设计理念">设计理念</a> •
<a href="#功能特性">功能特性</a> •
<a href="#安装">安装</a> •
<a href="#配置">配置</a> •
<a href="#安全机制">安全机制</a> •
<a href="#系统架构">系统架构</a>
</p>
---
## 设计理念
IronClaw 基于一个简单的原则:**你的 AI 助手应该为你服务,而不是与你为敌。**
在 AI 系统对数据处理日益不透明、与企业利益捆绑的今天,IronClaw 选择了一条不同的路:
- **数据归你所有** — 所有信息存储在本地,加密保护,始终在你掌控之下
- **透明至上** — 完全开源,可审计,没有隐藏的遥测或数据收集
- **自主扩展** — 随时构建新工具,无需等待供应商更新
- **纵深防御** — 多层安全机制抵御提示注入和数据泄露
IronClaw 是一个你真正可以信赖的 AI 助手,无论是个人生活还是工作。
## 功能特性
### 安全优先
- **WASM 沙箱** — 不受信任的工具在隔离的 WebAssembly 容器中运行,采用基于能力的权限模型
- **凭据保护** — 密钥永远不会暴露给工具;在宿主边界注入并进行泄露检测
- **提示注入防御** — 模式检测、内容清理和策略执行
- **端点白名单** — HTTP 请求仅限于明确批准的主机和路径
### 随时可用
- **多渠道接入** — REPL、HTTP webhook、WASM 渠道(Telegram、Slack)和 Web 网关
- **Docker 沙箱** — 隔离的容器执行,支持每任务令牌和编排器/工作器模式
- **Web 网关** — 浏览器 UI,支持实时 SSE/WebSocket 流式传输
- **定时任务** — Cron 调度、事件触发器、Webhook 处理器,实现后台自动化
- **心跳系统** — 主动后台执行,用于监控和维护任务
- **并行任务** — 使用隔离上下文同时处理多个请求
- **自修复** — 自动检测并恢复卡住的操作
### 自主扩展
- **动态工具构建** — 描述你的需求,IronClaw 会将其构建为 WASM 工具
- **MCP 协议** — 连接模型上下文协议(Model Context Protocol)服务器以获取额外能力
- **插件架构** — 无需重启即可加载新的 WASM 工具和渠道
### 持久记忆
- **混合搜索** — 全文搜索 + 向量搜索,采用倒数排名融合(Reciprocal Rank Fusion
- **工作空间文件系统** — 灵活的基于路径的存储,用于笔记、日志和上下文
- **身份文件** — 跨会话保持一致的个性和偏好设置
## 安装
### 前置要求
- Rust 1.85+
- PostgreSQL 15+,需安装 [pgvector](https://github.com/pgvector/pgvector) 扩展
- NEAR AI 账户(通过设置向导进行身份验证)
## 下载或编译
访问 [Releases 页面](https://github.com/nearai/ironclaw/releases/) 查看最新版本。
<details>
<summary>通过 Windows 安装程序安装 (Windows)</summary>
下载 [Windows 安装程序](https://github.com/nearai/ironclaw/releases/latest/download/ironclaw-x86_64-pc-windows-msvc.msi) 并运行。
</details>
<details>
<summary>通过 PowerShell 脚本安装 (Windows)</summary>
```sh
irm https://github.com/nearai/ironclaw/releases/latest/download/ironclaw-installer.ps1 | iex
```
</details>
<details>
<summary>通过 Shell 脚本安装 (macOS、Linux、Windows/WSL)</summary>
```sh
curl --proto '=https' --tlsv1.2 -LsSf https://github.com/nearai/ironclaw/releases/latest/download/ironclaw-installer.sh | sh
```
</details>
<details>
<summary>通过 Homebrew 安装 (macOS/Linux)</summary>
```sh
brew install ironclaw
```
</details>
<details>
<summary>从源码编译 (Windows、Linux、macOS 上使用 Cargo)</summary>
确保你已安装 [Rust](https://rustup.rs)。
```bash
# 克隆仓库
git clone https://github.com/nearai/ironclaw.git
cd ironclaw
# 编译
cargo build --release
# 运行测试
cargo test
```
如需进行**完整发布构建**(修改了渠道源码后),先运行 `./scripts/build-all.sh` 重新编译渠道。
</details>
### 数据库设置
```bash
# 创建数据库
createdb ironclaw
# 启用 pgvector 扩展
psql ironclaw -c "CREATE EXTENSION IF NOT EXISTS vector;"
```
## 配置
运行设置向导来配置 IronClaw:
```bash
ironclaw onboard
```
向导将引导你完成数据库连接、NEAR AI 身份验证(通过浏览器 OAuth)和密钥加密(使用系统钥匙串)。设置会保存在数据库中;引导变量(如 `DATABASE_URL``LLM_BACKEND`)写入 `~/.ironclaw/.env`,以便在数据库连接前可用。
### 替代 LLM 提供商
IronClaw 默认使用 NEAR AI,但兼容任何 OpenAI 兼容的端点。
常用选项包括 **OpenRouter**300+ 模型)、**Together AI**、**Fireworks AI**、**Ollama**(本地部署)以及自托管服务器如 **vLLM****LiteLLM**
在向导中选择 *"OpenAI-compatible"*,或直接设置环境变量:
```env
LLM_BACKEND=openai_compatible
LLM_BASE_URL=https://openrouter.ai/api/v1
LLM_API_KEY=sk-or-...
LLM_MODEL=anthropic/claude-sonnet-4
```
详见 [docs/LLM_PROVIDERS.md](docs/LLM_PROVIDERS.md) 获取完整的提供商指南。
## 安全机制
IronClaw 实现了纵深防御策略来保护你的数据并防止滥用。
### WASM 沙箱
所有不受信任的工具都在隔离的 WebAssembly 容器中运行:
- **基于能力的权限** — 明确授权 HTTP、密钥、工具调用等能力
- **端点白名单** — HTTP 请求仅限已批准的主机和路径
- **凭据注入** — 密钥在宿主边界注入,永远不会暴露给 WASM 代码
- **泄露检测** — 扫描请求和响应以防止密钥外泄
- **速率限制** — 每个工具独立的请求限制,防止滥用
- **资源限制** — 内存、CPU 和执行时间约束
```
WASM ──► 白名单 ──► 泄露扫描 ──► 凭据 ──► 执行 ──► 泄露扫描 ──► WASM
验证器 (请求) 注入器 请求 (响应)
```
### 提示注入防御
外部内容需通过多个安全层:
- 基于模式的注入尝试检测
- 内容清理和转义
- 带严重级别的策略规则(阻止/警告/审核/清理)
- 工具输出包装,确保安全的 LLM 上下文注入
### 数据保护
- 所有数据存储在本地 PostgreSQL 数据库中
- 密钥使用 AES-256-GCM 加密
- 无遥测、无分析、无数据共享
- 所有工具执行的完整审计日志
## 系统架构
```
┌────────────────────────────────────────────────────────────────┐
│ 渠道 │
│ ┌──────┐ ┌──────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ REPL │ │ HTTP │ │ WASM 渠道 │ │ Web 网关 │ │
│ └──┬───┘ └──┬───┘ └──────┬──────┘ │ (SSE + WS) │ │
│ │ │ │ └──────┬──────┘ │
│ └─────────┴──────────────┴────────────────┘ │
│ │ │
│ ┌─────────▼─────────┐ │
│ │ 代理循环 │ 意图路由 │
│ └────┬──────────┬───┘ │
│ │ │ │
│ ┌──────────▼────┐ ┌──▼───────────────┐ │
│ │ 调度器 │ │ 定时任务引擎 │ │
│ │ (并行任务) │ │(cron, 事件, wh) │ │
│ └──────┬────────┘ └────────┬─────────┘ │
│ │ │ │
│ ┌─────────────┼────────────────────┘ │
│ │ │ │
│ ┌───▼─────┐ ┌────▼────────────────┐ │
│ │ 本地 │ │ 编排器 │ │
│ │ 工作器 │ │ ┌───────────────┐ │ │
│ │(进程内) │ │ │ Docker 沙箱 │ │ │
│ └───┬─────┘ │ │ 容器 │ │ │
│ │ │ │ ┌───────────┐ │ │ │
│ │ │ │ │工作器/CC │ │ │ │
│ │ │ │ └───────────┘ │ │ │
│ │ │ └───────────────┘ │ │
│ │ └─────────┬───────────┘ │
│ └──────────────────┤ │
│ │ │
│ ┌───────────▼──────────┐ │
│ │ 工具注册表 │ │
│ │ 内置、MCP、WASM │ │
│ └──────────────────────┘ │
└────────────────────────────────────────────────────────────────┘
```
### 核心组件
| 组件 | 用途 |
|------|------|
| **代理循环** | 主消息处理和任务协调 |
| **路由器** | 分类用户意图(命令、查询、任务) |
| **调度器** | 管理带优先级的并行任务执行 |
| **工作器** | 执行包含 LLM 推理和工具调用的任务 |
| **编排器** | 容器生命周期、LLM 代理、每任务认证 |
| **Web 网关** | 浏览器 UI,含聊天、记忆、任务、日志、扩展、定时任务 |
| **定时任务引擎** | 定时(cron)和响应式(事件、webhook)后台任务 |
| **工作空间** | 带混合搜索的持久记忆 |
| **安全层** | 提示注入防御和内容清理 |
## 使用方式
```bash
# 首次设置(配置数据库、认证等)
ironclaw onboard
# 启动交互式 REPL
cargo run
# 启用调试日志
RUST_LOG=ironclaw=debug cargo run
```
## 开发
```bash
# 格式化代码
cargo fmt
# 代码检查
cargo clippy --all --benches --tests --examples --all-features
# 运行测试
createdb ironclaw_test
cargo test
# 运行指定测试
cargo test test_name
```
- **Telegram 渠道**:参见 [docs/TELEGRAM_SETUP.md](docs/TELEGRAM_SETUP.md) 了解设置和私信配对。
- **修改渠道源码**:在 `cargo build` 之前运行 `./channels-src/telegram/build.sh` 以便打包更新后的 WASM。
## OpenClaw 传承
IronClaw 是受 [OpenClaw](https://github.com/openclaw/openclaw) 启发的 Rust 重新实现。参见 [FEATURE_PARITY.md](FEATURE_PARITY.md) 了解完整的功能追踪矩阵。
主要差异:
- **Rust vs TypeScript** — 原生性能、内存安全、单一二进制文件
- **WASM 沙箱 vs Docker** — 轻量级、基于能力的安全机制
- **PostgreSQL vs SQLite** — 生产级持久化存储
- **安全优先设计** — 多层防御、凭据保护
## 许可证
可选择以下任一许可证:
- Apache License, Version 2.0 ([LICENSE-APACHE](LICENSE-APACHE))
- MIT License ([LICENSE-MIT](LICENSE-MIT))
+1 -1
View File
@@ -267,7 +267,7 @@ dependencies = [
[[package]]
name = "slack-channel"
version = "0.1.0"
version = "0.2.1"
dependencies = [
"hex",
"hmac",
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "slack-channel"
version = "0.2.0"
version = "0.2.1"
edition = "2021"
description = "Slack Events API channel for IronClaw"
license = "MIT OR Apache-2.0"
+104
View File
@@ -357,10 +357,108 @@ fn extract_slack_attachments(files: &Option<Vec<SlackFile>>) -> Vec<InboundAttac
.collect()
}
/// Download a file from Slack using the url_private endpoint.
///
/// Slack file downloads require Bearer auth with the bot token, which is
/// injected by the host credential system via `channel_host::http_request`.
fn download_slack_file(url: &str) -> Result<Vec<u8>, String> {
let headers = serde_json::json!({});
let result = channel_host::http_request("GET", url, &headers.to_string(), None, None);
let response = result.map_err(|e| format!("Slack file download failed: {}", e))?;
if response.status != 200 {
let body_str = String::from_utf8_lossy(&response.body);
return Err(format!(
"Slack file download returned {}: {}",
response.status, body_str
));
}
Ok(response.body)
}
/// Download file bytes and store them via the host for processing.
///
/// Downloads all file types (images, documents, etc.) so the host-side
/// middleware can process them (vision pipeline for images, text extraction
/// for documents, transcription for audio, etc.).
/// Maximum file size to download (20 MB). Files larger than this are skipped
/// to avoid excessive memory use and slow downloads in the WASM runtime.
const MAX_DOWNLOAD_SIZE_BYTES: u64 = 20 * 1024 * 1024;
fn download_and_store_slack_files(attachments: &[InboundAttachment]) {
for att in attachments {
let Some(ref url) = att.source_url else {
continue;
};
// Skip files that exceed the size limit
if let Some(size) = att.size_bytes {
if size > MAX_DOWNLOAD_SIZE_BYTES {
channel_host::log(
channel_host::LogLevel::Warn,
&format!(
"Skipping Slack file download: {} bytes exceeds {} MB limit (id={})",
size,
MAX_DOWNLOAD_SIZE_BYTES / (1024 * 1024),
att.id
),
);
continue;
}
}
match download_slack_file(url) {
Ok(bytes) => {
// Post-download size guard: metadata size_bytes is optional,
// so a file with no size info could bypass the pre-download check.
if bytes.len() as u64 > MAX_DOWNLOAD_SIZE_BYTES {
channel_host::log(
channel_host::LogLevel::Warn,
&format!(
"Discarding Slack file after download: {} bytes exceeds {} MB limit (id={})",
bytes.len(),
MAX_DOWNLOAD_SIZE_BYTES / (1024 * 1024),
att.id
),
);
continue;
}
channel_host::log(
channel_host::LogLevel::Info,
&format!(
"Downloaded Slack file: {} bytes, mime={}",
bytes.len(),
att.mime_type
),
);
if let Err(e) = channel_host::store_attachment_data(&att.id, &bytes) {
channel_host::log(
channel_host::LogLevel::Error,
&format!("Failed to store Slack file data: {}", e),
);
}
}
Err(e) => {
channel_host::log(
channel_host::LogLevel::Error,
&format!("Failed to download Slack file: {}", e),
);
}
}
}
}
/// Handle a Slack event and emit message if applicable.
fn handle_slack_event(event: SlackEvent, team_id: Option<String>, _event_id: Option<String>) {
let attachments = extract_slack_attachments(&event.files);
// Download and store file attachments for host-side processing
download_and_store_slack_files(&attachments);
match event.event_type.as_str() {
// Direct mention of the bot (always in a channel, not a DM)
"app_mention" => {
@@ -722,4 +820,10 @@ mod tests {
let event: SlackEvent = serde_json::from_str(json).unwrap();
assert!(event.files.is_none());
}
#[test]
fn test_max_download_size_constant() {
// Verify the constant is 20 MB
assert_eq!(MAX_DOWNLOAD_SIZE_BYTES, 20 * 1024 * 1024);
}
}
+1 -1
View File
@@ -212,7 +212,7 @@ dependencies = [
[[package]]
name = "telegram-channel"
version = "0.2.0"
version = "0.2.1"
dependencies = [
"serde",
"serde_json",
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "telegram-channel"
version = "0.2.0"
version = "0.2.1"
edition = "2021"
description = "Telegram Bot API channel for IronClaw"
license = "MIT OR Apache-2.0"
+57 -5
View File
@@ -878,10 +878,6 @@ fn send_message(
// Voice File Download
// ============================================================================
/// Download a voice file from Telegram by file_id.
///
/// 1. Call getFile to get the file_path.
/// 2. Download the file bytes from /file/bot{TOKEN}/{file_path}.
/// Percent-encode a string for safe use as a URL query parameter value.
fn percent_encode(s: &str) -> String {
let mut out = String::with_capacity(s.len());
@@ -898,6 +894,10 @@ fn percent_encode(s: &str) -> String {
out
}
/// Maximum file size to download (20 MB). Files larger than this are discarded
/// to avoid excessive memory use and slow downloads in the WASM runtime.
const MAX_DOWNLOAD_SIZE_BYTES: u64 = 20 * 1024 * 1024;
fn download_telegram_file(file_id: &str) -> Result<Vec<u8>, String> {
// Reject file_id containing curly braces to prevent credential placeholder injection
if file_id.contains('{') || file_id.contains('}') {
@@ -965,6 +965,16 @@ fn download_telegram_file(file_id: &str) -> Result<Vec<u8>, String> {
));
}
// Post-download size guard: Telegram metadata file_size is optional,
// so enforce the limit on actual downloaded bytes.
if response.body.len() as u64 > MAX_DOWNLOAD_SIZE_BYTES {
return Err(format!(
"Downloaded file exceeds {} MB limit ({} bytes)",
MAX_DOWNLOAD_SIZE_BYTES / (1024 * 1024),
response.body.len()
));
}
Ok(response.body)
}
@@ -1535,6 +1545,39 @@ fn download_and_store_voice(attachments: &[InboundAttachment]) {
}
}
/// Download image file bytes and store them via the host for the vision pipeline.
///
/// Separated from `extract_attachments` so that function stays pure (no host
/// calls) and remains testable in native unit tests.
fn download_and_store_images(attachments: &[InboundAttachment]) {
for att in attachments {
if !att.mime_type.starts_with("image/") {
continue;
}
match download_telegram_file(&att.id) {
Ok(bytes) => {
channel_host::log(
channel_host::LogLevel::Info,
&format!("Downloaded image file: {} bytes", bytes.len()),
);
if let Err(e) = channel_host::store_attachment_data(&att.id, &bytes) {
channel_host::log(
channel_host::LogLevel::Error,
&format!("Failed to store image data: {}", e),
);
}
}
Err(e) => {
channel_host::log(
channel_host::LogLevel::Error,
&format!("Failed to download image file: {}", e),
);
}
}
}
}
/// Returns true if the attachment should be downloaded for document text extraction.
///
/// Excludes voice (handled by transcription), image (vision pipeline),
@@ -1608,6 +1651,9 @@ fn handle_message(message: TelegramMessage) {
// Download and store voice attachments for host-side transcription
download_and_store_voice(&attachments);
// Download and store image attachments for host-side vision pipeline
download_and_store_images(&attachments);
// Download and store document attachments for host-side text extraction
download_and_store_documents(&mut attachments);
@@ -1681,7 +1727,7 @@ fn handle_message(message: TelegramMessage) {
let username_opt = from.username.as_deref();
let is_allowed = allowed.contains(&"*".to_string())
|| allowed.contains(&id_str)
|| username_opt.map_or(false, |u| allowed.contains(&u.to_string()));
|| username_opt.is_some_and(|u| allowed.contains(&u.to_string()));
if !is_allowed {
if is_private && dm_policy == "pairing" {
@@ -2605,4 +2651,10 @@ mod tests {
assert!(!is_downloadable_document(&make("audio/mpeg", Some("song.mp3"))));
assert!(!is_downloadable_document(&make("video/mp4", Some("clip.mp4"))));
}
#[test]
fn test_max_download_size_constant() {
// Verify the constant is 20 MB, matching the Slack channel limit
assert_eq!(MAX_DOWNLOAD_SIZE_BYTES, 20 * 1024 * 1024);
}
}
@@ -1,9 +1,17 @@
{
"version": "0.2.0",
"version": "0.2.2",
"wit_version": "0.3.0",
"type": "channel",
"name": "telegram",
"description": "Telegram Bot API channel for receiving and responding to Telegram messages",
"auth": {
"secret_name": "telegram_bot_token",
"display_name": "Telegram",
"instructions": "Get your bot token from @BotFather on Telegram (https://t.me/BotFather). Send /newbot or /token to get it.",
"setup_url": "https://t.me/BotFather",
"token_hint": "Looks like 123456789:AABBccDDeeFFgg...",
"env_var": "TELEGRAM_BOT_TOKEN"
},
"setup": {
"required_secrets": [
{
+1 -1
View File
@@ -3,7 +3,7 @@ services:
postgres:
image: pgvector/pgvector:pg16
ports:
- "5432:5432"
- "127.0.0.1:5432:5432"
environment:
POSTGRES_DB: ironclaw
POSTGRES_USER: ironclaw
+55
View File
@@ -11,7 +11,13 @@ configurations.
| NEAR AI | `nearai` | OAuth (browser) | Default; multi-model |
| Anthropic | `anthropic` | `ANTHROPIC_API_KEY` | Claude models |
| OpenAI | `openai` | `OPENAI_API_KEY` | GPT models |
| Google Gemini | `gemini` | `GEMINI_API_KEY` | Gemini models |
| io.net | `ionet` | `IONET_API_KEY` | Intelligence API |
| Mistral | `mistral` | `MISTRAL_API_KEY` | Mistral models |
| Yandex AI Studio | `yandex` | `YANDEX_API_KEY` | YandexGPT models |
| Cloudflare Workers AI | `cloudflare` | `CLOUDFLARE_API_KEY` | Access to Workers AI |
| Ollama | `ollama` | No | Local inference |
| AWS Bedrock | `bedrock` | AWS credentials | Native Converse API |
| OpenRouter | `openai_compatible` | `LLM_API_KEY` | 300+ models |
| Together AI | `openai_compatible` | `LLM_API_KEY` | Fast inference |
| Fireworks AI | `openai_compatible` | `LLM_API_KEY` | Fast inference |
@@ -68,6 +74,55 @@ Pull a model first: `ollama pull llama3.2`
---
## AWS Bedrock (requires `--features bedrock`)
Uses the native AWS Converse API via `aws-sdk-bedrockruntime`. Supports standard AWS
authentication methods: IAM credentials, SSO profiles, and instance roles.
> **Build prerequisite:** The `aws-lc-sys` crate (transitive dependency via AWS SDK)
> requires **CMake** to compile. Install it before building with `--features bedrock`:
> - macOS: `brew install cmake`
> - Ubuntu/Debian: `sudo apt install cmake`
> - Fedora: `sudo dnf install cmake`
### With AWS credentials (IAM, SSO, instance roles)
```env
LLM_BACKEND=bedrock
BEDROCK_MODEL=anthropic.claude-opus-4-6-v1
BEDROCK_REGION=us-east-1
BEDROCK_CROSS_REGION=us
# AWS_PROFILE=my-sso-profile # optional, for named profiles
```
The AWS SDK credential chain automatically resolves credentials from environment
variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`), shared credentials file
(`~/.aws/credentials`), SSO profiles, and EC2/ECS instance roles.
### Cross-region inference
Set `BEDROCK_CROSS_REGION` to route requests across AWS regions for capacity:
| Prefix | Routing |
|---|---|
| `us` | US regions (us-east-1, us-east-2, us-west-2) |
| `eu` | European regions |
| `apac` | Asia-Pacific regions |
| `global` | All commercial AWS regions |
| _(unset)_ | Single-region only |
### Popular Bedrock model IDs
| Model | ID |
|---|---|
| Claude Opus 4.6 | `anthropic.claude-opus-4-6-v1` |
| Claude Sonnet 4.5 | `anthropic.claude-sonnet-4-5-20250929-v1:0` |
| Claude Haiku 4.5 | `anthropic.claude-haiku-4-5-20251001-v1:0` |
| Amazon Nova Pro | `amazon.nova-pro-v1:0` |
| Llama 4 Maverick | `meta.llama4-maverick-17b-instruct-v1:0` |
---
## OpenAI-Compatible Endpoints
All providers below use `LLM_BACKEND=openai_compatible`. Set `LLM_BASE_URL` to the
@@ -0,0 +1,13 @@
-- Partial unique indexes to prevent duplicate singleton conversations.
-- These guard against TOCTOU races in get_or_create_routine_conversation
-- and get_or_create_heartbeat_conversation.
-- One routine conversation per user per routine_id.
CREATE UNIQUE INDEX IF NOT EXISTS uq_conv_routine
ON conversations (user_id, (metadata->>'routine_id'))
WHERE metadata->>'routine_id' IS NOT NULL;
-- One heartbeat conversation per user.
CREATE UNIQUE INDEX IF NOT EXISTS uq_conv_heartbeat
ON conversations (user_id)
WHERE metadata->>'thread_type' = 'heartbeat';
+141 -11
View File
@@ -1,7 +1,9 @@
[
{
"id": "openai",
"aliases": ["open_ai"],
"aliases": [
"open_ai"
],
"protocol": "open_ai_completions",
"api_key_env": "OPENAI_API_KEY",
"api_key_required": true,
@@ -19,7 +21,9 @@
},
{
"id": "anthropic",
"aliases": ["claude"],
"aliases": [
"claude"
],
"protocol": "anthropic",
"api_key_env": "ANTHROPIC_API_KEY",
"api_key_required": true,
@@ -52,7 +56,10 @@
},
{
"id": "openai_compatible",
"aliases": ["openai-compatible", "compatible"],
"aliases": [
"openai-compatible",
"compatible"
],
"protocol": "open_ai_completions",
"base_url_env": "LLM_BASE_URL",
"base_url_required": true,
@@ -89,7 +96,9 @@
},
{
"id": "openrouter",
"aliases": ["open_router"],
"aliases": [
"open_router"
],
"protocol": "open_ai_completions",
"default_base_url": "https://openrouter.ai/api/v1",
"api_key_env": "OPENROUTER_API_KEY",
@@ -126,7 +135,10 @@
},
{
"id": "nvidia",
"aliases": ["nvidia_nim", "nim"],
"aliases": [
"nvidia_nim",
"nim"
],
"protocol": "open_ai_completions",
"default_base_url": "https://integrate.api.nvidia.com/v1",
"api_key_env": "NVIDIA_API_KEY",
@@ -144,7 +156,10 @@
},
{
"id": "venice",
"aliases": ["venice_ai", "veniceai"],
"aliases": [
"venice_ai",
"veniceai"
],
"protocol": "open_ai_completions",
"default_base_url": "https://api.venice.ai/api/v1",
"api_key_env": "VENICE_API_KEY",
@@ -162,7 +177,10 @@
},
{
"id": "together",
"aliases": ["together_ai", "togetherai"],
"aliases": [
"together_ai",
"togetherai"
],
"protocol": "open_ai_completions",
"default_base_url": "https://api.together.xyz/v1",
"api_key_env": "TOGETHER_API_KEY",
@@ -180,7 +198,9 @@
},
{
"id": "fireworks",
"aliases": ["fireworks_ai"],
"aliases": [
"fireworks_ai"
],
"protocol": "open_ai_completions",
"default_base_url": "https://api.fireworks.ai/inference/v1",
"api_key_env": "FIREWORKS_API_KEY",
@@ -198,7 +218,9 @@
},
{
"id": "deepseek",
"aliases": ["deep_seek"],
"aliases": [
"deep_seek"
],
"protocol": "open_ai_completions",
"default_base_url": "https://api.deepseek.com/v1",
"api_key_env": "DEEPSEEK_API_KEY",
@@ -234,7 +256,9 @@
},
{
"id": "sambanova",
"aliases": ["samba_nova"],
"aliases": [
"samba_nova"
],
"protocol": "open_ai_completions",
"default_base_url": "https://api.sambanova.ai/v1",
"api_key_env": "SAMBANOVA_API_KEY",
@@ -249,5 +273,111 @@
"display_name": "SambaNova",
"can_list_models": false
}
},
{
"id": "gemini",
"aliases": [
"google_gemini",
"google"
],
"protocol": "open_ai_completions",
"default_base_url": "https://generativelanguage.googleapis.com/v1beta/openai",
"api_key_env": "GEMINI_API_KEY",
"api_key_required": true,
"model_env": "GEMINI_MODEL",
"default_model": "gemini-2.5-flash",
"description": "Google Gemini (via OpenAI-compatible endpoint)",
"setup": {
"kind": "api_key",
"secret_name": "llm_gemini_api_key",
"key_url": "https://aistudio.google.com/app/apikey",
"display_name": "Google Gemini",
"can_list_models": true
}
},
{
"id": "ionet",
"aliases": [
"io_net",
"io.net"
],
"protocol": "open_ai_completions",
"default_base_url": "https://api.intelligence.io.solutions/api/v1",
"api_key_env": "IONET_API_KEY",
"api_key_required": true,
"model_env": "IONET_MODEL",
"default_model": "deepseek-coder-v2-instruct",
"description": "io.net Intelligence API",
"setup": {
"kind": "api_key",
"secret_name": "llm_ionet_api_key",
"key_url": "https://cloud.io.net/intelligence",
"display_name": "io.net",
"can_list_models": true
}
},
{
"id": "mistral",
"aliases": [
"mistral_ai",
"mistralai"
],
"protocol": "open_ai_completions",
"default_base_url": "https://api.mistral.ai/v1",
"api_key_env": "MISTRAL_API_KEY",
"api_key_required": true,
"model_env": "MISTRAL_MODEL",
"default_model": "mistral-large-latest",
"description": "Mistral AI API",
"setup": {
"kind": "api_key",
"secret_name": "llm_mistral_api_key",
"key_url": "https://console.mistral.ai/api-keys",
"display_name": "Mistral",
"can_list_models": true
}
},
{
"id": "yandex",
"aliases": [
"yandex_ai_studio",
"yandexgpt",
"yandex_gpt"
],
"protocol": "open_ai_completions",
"default_base_url": "https://ai.api.cloud.yandex.net/v1",
"api_key_env": "YANDEX_API_KEY",
"api_key_required": true,
"model_env": "YANDEX_MODEL",
"extra_headers_env": "YANDEX_EXTRA_HEADERS",
"default_model": "yandexgpt-lite",
"description": "Yandex AI Studio (YandexGPT)",
"setup": {
"kind": "api_key",
"secret_name": "llm_yandex_api_key",
"key_url": "https://aistudio.yandex.ru/platform/folders/",
"display_name": "Yandex AI Studio",
"can_list_models": true
}
},
{
"id": "cloudflare",
"aliases": [
"cloudflare_ai",
"cf_ai"
],
"protocol": "open_ai_completions",
"api_key_env": "CLOUDFLARE_API_KEY",
"api_key_required": true,
"base_url_env": "CLOUDFLARE_BASE_URL",
"model_env": "CLOUDFLARE_MODEL",
"default_model": "@cf/meta/llama-3.3-70b-instruct-fp8-fast",
"description": "Cloudflare Workers AI",
"setup": {
"kind": "open_ai_compatible",
"secret_name": "llm_cloudflare_api_key",
"display_name": "Cloudflare Workers AI",
"can_list_models": false
}
}
]
]
+1 -1
View File
@@ -19,7 +19,7 @@
"artifacts": {
"wasm32-wasip2": {
"url": "https://github.com/nearai/ironclaw/releases/latest/download/discord-wasm32-wasip2.tar.gz",
"sha256": null
"sha256": "030707431717bca3411a48f311c6ab5f92a45c747de26cafe4f6e3e23a8b3b2d"
}
},
"auth_summary": {
+2 -2
View File
@@ -2,7 +2,7 @@
"name": "slack",
"display_name": "Slack Channel",
"kind": "channel",
"version": "0.2.0",
"version": "0.2.1",
"wit_version": "0.3.0",
"description": "Talk to your agent in Slack",
"keywords": [
@@ -19,7 +19,7 @@
"artifacts": {
"wasm32-wasip2": {
"url": "https://github.com/nearai/ironclaw/releases/latest/download/slack-wasm32-wasip2.tar.gz",
"sha256": null
"sha256": "6ed36077b67ac70a041f06f760f93ba79b33269885413c3c3f2c8c87ee60807e"
}
},
"auth_summary": {
+2 -2
View File
@@ -2,7 +2,7 @@
"name": "telegram",
"display_name": "Telegram Channel",
"kind": "channel",
"version": "0.2.0",
"version": "0.2.2",
"wit_version": "0.3.0",
"description": "Talk to your agent through a Telegram bot",
"keywords": [
@@ -19,7 +19,7 @@
"artifacts": {
"wasm32-wasip2": {
"url": "https://github.com/nearai/ironclaw/releases/latest/download/telegram-wasm32-wasip2.tar.gz",
"sha256": null
"sha256": "98c86895a9c4b0a1e19fe8a47f1ccbfe7e972e112b05e584bc897130dc32283a"
}
},
"auth_summary": {
+1 -1
View File
@@ -19,7 +19,7 @@
"artifacts": {
"wasm32-wasip2": {
"url": "https://github.com/nearai/ironclaw/releases/latest/download/whatsapp-wasm32-wasip2.tar.gz",
"sha256": null
"sha256": "bd35cad18d87292ea8d2f52db9b514ed9f814a414de910f59073d475c26c4c14"
}
},
"auth_summary": {
+1 -1
View File
@@ -20,7 +20,7 @@
"artifacts": {
"wasm32-wasip2": {
"url": "https://github.com/nearai/ironclaw/releases/latest/download/github-wasm32-wasip2.tar.gz",
"sha256": null
"sha256": "6fcd32719a4ff15641a4b50fff8984686550f0c491dce60518f4126857d0c544"
}
},
"auth_summary": {
+1 -1
View File
@@ -19,7 +19,7 @@
"artifacts": {
"wasm32-wasip2": {
"url": "https://github.com/nearai/ironclaw/releases/latest/download/gmail-wasm32-wasip2.tar.gz",
"sha256": null
"sha256": "023da7000b17568bf0e64b2e5013c8a042b2f323c85f1632339231c73d500e39"
}
},
"auth_summary": {
+1 -1
View File
@@ -19,7 +19,7 @@
"artifacts": {
"wasm32-wasip2": {
"url": "https://github.com/nearai/ironclaw/releases/latest/download/google-calendar-wasm32-wasip2.tar.gz",
"sha256": null
"sha256": "fc42277b65881d6e9bcc5403dc54c7f5b3ddeaaaf04617fce2c5da05d76325f0"
}
},
"auth_summary": {
+1 -1
View File
@@ -19,7 +19,7 @@
"artifacts": {
"wasm32-wasip2": {
"url": "https://github.com/nearai/ironclaw/releases/latest/download/google-docs-wasm32-wasip2.tar.gz",
"sha256": null
"sha256": "385c04abd1e6b8011ccc330e1f4bd7ce58577e488959b51594aa04eb26cbe7cc"
}
},
"auth_summary": {
+1 -1
View File
@@ -19,7 +19,7 @@
"artifacts": {
"wasm32-wasip2": {
"url": "https://github.com/nearai/ironclaw/releases/latest/download/google-drive-wasm32-wasip2.tar.gz",
"sha256": null
"sha256": "1b107d575a5d52cc8c76d9a681802190f4373fb485f7f54f445533f097fa37c0"
}
},
"auth_summary": {
+1 -1
View File
@@ -19,7 +19,7 @@
"artifacts": {
"wasm32-wasip2": {
"url": "https://github.com/nearai/ironclaw/releases/latest/download/google-sheets-wasm32-wasip2.tar.gz",
"sha256": null
"sha256": "c4f6b1e8c5126ac2c8a4b98e4283a3afa32223d2488fc3c3a609758c0c9beb90"
}
},
"auth_summary": {
+1 -1
View File
@@ -18,7 +18,7 @@
"artifacts": {
"wasm32-wasip2": {
"url": "https://github.com/nearai/ironclaw/releases/latest/download/google-slides-wasm32-wasip2.tar.gz",
"sha256": null
"sha256": "7110b8565340c888e51f99e9c013bf4de8f8a7f7b33bace00eb8fc47831ff20b"
}
},
"auth_summary": {
+1 -1
View File
@@ -18,7 +18,7 @@
"artifacts": {
"wasm32-wasip2": {
"url": "https://github.com/nearai/ironclaw/releases/latest/download/slack-tool-wasm32-wasip2.tar.gz",
"sha256": null
"sha256": "6ed36077b67ac70a041f06f760f93ba79b33269885413c3c3f2c8c87ee60807e"
}
},
"auth_summary": {
+1 -1
View File
@@ -19,7 +19,7 @@
"artifacts": {
"wasm32-wasip2": {
"url": "https://github.com/nearai/ironclaw/releases/latest/download/telegram-mtproto-wasm32-wasip2.tar.gz",
"sha256": null
"sha256": "98c86895a9c4b0a1e19fe8a47f1ccbfe7e972e112b05e584bc897130dc32283a"
}
},
"auth_summary": {
+1 -1
View File
@@ -19,7 +19,7 @@
"artifacts": {
"wasm32-wasip2": {
"url": "https://github.com/nearai/ironclaw/releases/latest/download/web-search-wasm32-wasip2.tar.gz",
"sha256": null
"sha256": "66cb2b9b00652385e9f30f17c74902b9222c17c53e9d3bd1ef42f5cab705bcf6"
}
},
"auth_summary": {
+50
View File
@@ -209,6 +209,56 @@ else
fi
echo
# --------------------------------------------------------------------------
# Check 6: LLM module isolation — no imports from other crate modules
# --------------------------------------------------------------------------
# src/llm/ should only import from:
# - crate::llm (self-references)
# - external crates (no crate:: prefix)
# It must NOT import from crate::agent, crate::tools, crate::channels,
# crate::safety, crate::config, crate::bootstrap, crate::cli, crate::db,
# crate::workspace, crate::worker, crate::orchestrator, crate::skills,
# crate::hooks, crate::setup, crate::context, etc.
#
# Test-only imports (crate::testing) are excluded since they don't affect
# the runtime dependency graph and won't exist in the extracted crate.
# --------------------------------------------------------------------------
echo "--- Check 6: LLM module isolation ---"
# Match any `crate::` reference (use-imports AND inline paths) that isn't
# crate::llm or crate::testing. Filter out comments.
# We strip inline comments (everything after //) with sed before checking,
# so a line like `real_code(crate::foo); // crate::llm` is still caught.
results=$(grep -rn 'crate::' src/llm/ \
--include='*.rs' \
| grep -v '^\s*//' \
| sed 's|//.*||' \
| grep 'crate::' \
| grep -v 'crate::llm' \
| grep -v 'crate::testing' \
|| true)
if [ -n "$results" ]; then
count=$(echo "$results" | wc -l | tr -d ' ')
echo "WARNING: src/llm/ has $count reference(s) to modules outside crate::llm:"
echo "$results"
echo
echo "(These are pre-existing; fix them before extracting the crate.)"
echo "(New 'use crate::' imports are hard violations — see below.)"
echo
# Hard-fail only on new `use crate::` imports (easy to avoid in new code).
use_imports=$(echo "$results" | grep '^[^:]*:.*use crate::' || true)
if [ -n "$use_imports" ]; then
echo "HARD VIOLATION: new 'use crate::' imports in src/llm/:"
echo "$use_imports"
violations=$((violations + 1))
fi
else
echo "OK"
fi
echo
# --------------------------------------------------------------------------
# Summary
# --------------------------------------------------------------------------
+4 -2
View File
@@ -51,9 +51,11 @@ echo "[6/6] Installing git hooks..."
HOOKS_DIR=$(git rev-parse --git-path hooks 2>/dev/null) || true
if [ -n "$HOOKS_DIR" ]; then
mkdir -p "$HOOKS_DIR"
SCRIPT_ABS="$(cd "$(dirname "$0")" && pwd)/commit-msg-regression.sh"
ln -sf "$SCRIPT_ABS" "$HOOKS_DIR/commit-msg"
SCRIPTS_ABS="$(cd "$(dirname "$0")" && pwd)"
ln -sf "$SCRIPTS_ABS/commit-msg-regression.sh" "$HOOKS_DIR/commit-msg"
echo " commit-msg hook installed (regression test enforcement)"
ln -sf "$SCRIPTS_ABS/pre-commit-safety.sh" "$HOOKS_DIR/pre-commit"
echo " pre-commit hook installed (UTF-8, case-sensitivity, /tmp, redaction checks)"
else
echo " Skipped: not a git repository"
fi
+136
View File
@@ -0,0 +1,136 @@
#!/usr/bin/env bash
# Pre-commit safety checks for common issues caught by AI code reviewers.
#
# Can be run standalone: bash scripts/pre-commit-safety.sh
# Or installed as a git pre-commit hook via dev-setup.sh.
#
# Checks staged .rs files for:
# 1. Unsafe UTF-8 byte slicing (panics on multi-byte chars)
# 2. Case-sensitive file extension comparisons
# 3. Hardcoded /tmp paths in tests (flaky in parallel runs)
# 4. Tool parameters logged without redaction (secret leaks)
# 5. Multi-step DB operations without transaction wrapping
#
# Suppress individual lines with an inline "// safety: <reason>" comment.
set -euo pipefail
# Determine a suitable base ref for standalone diffs.
resolve_base_ref() {
local candidates=(
"@{upstream}"
"origin/HEAD"
"origin/main"
"origin/master"
"main"
"master"
)
for ref in "${candidates[@]}"; do
if git rev-parse --verify --quiet "$ref" >/dev/null 2>&1; then
echo "$ref"
return 0
fi
done
echo "pre-commit-safety: could not determine a base Git ref for diff (tried: ${candidates[*]})." >&2
echo "pre-commit-safety: ensure your repository has an upstream or a local main/master branch." >&2
exit 1
}
# Support both pre-commit hook (staged files) and standalone (all changed vs base)
if git diff --cached --quiet 2>/dev/null; then
# No staged changes -- compare working tree against a resolved base ref
BASE_REF="$(resolve_base_ref)"
DIFF_OUTPUT=$(git diff "$BASE_REF" -- '*.rs' 2>/dev/null || true)
else
DIFF_OUTPUT=$(git diff --cached -U0 -- '*.rs' 2>/dev/null || true)
fi
# Early exit if there are no relevant .rs changes
if [ -z "$DIFF_OUTPUT" ]; then
exit 0
fi
WARNINGS=0
warn() {
if [ "$WARNINGS" -eq 0 ]; then
echo ""
echo "=== Pre-commit Safety Checks ==="
echo ""
fi
WARNINGS=$((WARNINGS + 1))
echo " [$1] $2"
}
# 1. Unsafe UTF-8 byte slicing: &s[..N] or &s[..some_var] on strings
# Safe patterns: is_char_boundary, char_indices, // safety:
if echo "$DIFF_OUTPUT" | grep -nE '^\+' | grep -E '\[\.\..*\]' | grep -vE 'is_char_boundary|char_indices|// safety:|as_bytes|Vec<|&\[u8\]|\[u8\]|bytes\(\)|&bytes' | head -3 | grep -q .; then
warn "UTF8" "Possible unsafe byte-index string slicing. Use is_char_boundary() or char_indices()."
echo "$DIFF_OUTPUT" | grep -nE '^\+' | grep -E '\[\.\..*\]' | grep -vE 'is_char_boundary|char_indices|// safety:|as_bytes|Vec<|&\[u8\]|\[u8\]|bytes\(\)|&bytes' | head -3 | sed 's/^/ /'
fi
# 2. Case-sensitive file extension checks
# Match: .ends_with(".png") without prior to_lowercase
if echo "$DIFF_OUTPUT" | grep -nE '^\+.*ends_with\("\.([pP][nN][gG]|[jJ][pP][eE]?[gG]|[gG][iI][fF]|[wW][eE][bB][pP]|[mM][dD])"\)' | grep -vE 'to_lowercase|to_ascii_lowercase|// safety:' | head -3 | grep -q .; then
warn "CASE" "Case-sensitive file extension comparison. Normalize to lowercase first."
echo "$DIFF_OUTPUT" | grep -nE '^\+.*ends_with\("\.([pP][nN][gG]|[jJ][pP][eE]?[gG]|[gG][iI][fF]|[wW][eE][bB][pP]|[mM][dD])"\)' | grep -vE 'to_lowercase|to_ascii_lowercase|// safety:' | head -3 | sed 's/^/ /'
fi
# 3. Hardcoded /tmp paths in test files
if echo "$DIFF_OUTPUT" | grep -nE '^\+.*"/tmp/' | grep -vE 'tempfile|tempdir|// safety:' | head -3 | grep -q .; then
warn "TMPDIR" "Hardcoded /tmp path. Use tempfile::tempdir() for parallel-safe tests."
echo "$DIFF_OUTPUT" | grep -nE '^\+.*"/tmp/' | grep -vE 'tempfile|tempdir|// safety:' | head -3 | sed 's/^/ /'
fi
# 4. Logging tool parameters without redaction
if echo "$DIFF_OUTPUT" | grep -nE '^\+.*tracing::(info|debug|warn|error).*param' | grep -vE 'redact|// safety:' | head -3 | grep -q .; then
warn "REDACT" "Logging tool parameters without redaction. Use redact_params() first."
echo "$DIFF_OUTPUT" | grep -nE '^\+.*tracing::(info|debug|warn|error).*param' | grep -vE 'redact|// safety:' | head -3 | sed 's/^/ /'
fi
# 5. Multi-step DB operations without transaction
# Uses -W (function context) to reduce false positives from existing transactions.
# Suppressible with "// safety:" in the hunk.
DIFF_W_OUTPUT=$(git diff --cached -W -- '*.rs' 2>/dev/null || git diff "$(resolve_base_ref)" -W -- '*.rs' 2>/dev/null || true)
if [ -n "$DIFF_W_OUTPUT" ]; then
HUNK_COUNT=$(echo "$DIFF_W_OUTPUT" | awk '
/^@@/ {
if (count >= 2 && !has_tx && !has_safety) found++
count=0; has_tx=0; has_safety=0
}
/^\+.*\.(execute|query)\(/ { count++ }
/^\+.*(transaction|\.tx\.|\.begin\()/ { has_tx=1 }
/ .*(transaction|\.tx\.|\.begin\()/ { has_tx=1 }
/\/\/ safety:/ { has_safety=1 }
END {
if (count >= 2 && !has_tx && !has_safety) found++
print found+0
}
')
if [ "$HUNK_COUNT" -gt 0 ]; then
warn "TX" "Multiple DB operations in same function without transaction. Wrap in a transaction for atomicity."
echo "$DIFF_W_OUTPUT" | awk '
/^@@/ {
if (count >= 2 && !has_tx && !has_safety) { print buf }
buf=""; count=0; has_tx=0; has_safety=0
}
/^\+.*\.(execute|query)\(/ { count++ }
/^\+.*(transaction|\.tx\.|\.begin\()/ { has_tx=1 }
/ .*(transaction|\.tx\.|\.begin\()/ { has_tx=1 }
/\/\/ safety:/ { has_safety=1 }
{ buf = buf "\n" $0 }
END {
if (count >= 2 && !has_tx && !has_safety) { print buf }
}
' | grep -E '^\+.*\.(execute|query)\(' | head -4 | sed 's/^/ /'
fi
fi
if [ "$WARNINGS" -gt 0 ]; then
echo ""
echo "Found $WARNINGS potential issue(s). Fix them or add '// safety: <reason>' to suppress."
echo ""
exit 1
fi
+54
View File
@@ -0,0 +1,54 @@
---
name: review-checklist
version: 0.1.0
description: Pre-merge review checklist based on recurring AI reviewer feedback patterns
activation:
patterns:
- "review.*checklist"
- "ready to merge"
- "pre-merge check"
- "check.*before.*merge"
keywords:
- review
- checklist
- merge
- pre-merge
max_context_tokens: 1500
---
# Pre-Merge Review Checklist
Before merging, verify these items. They represent the most common issues caught by automated code reviewers (Copilot, Gemini) on IronClaw PRs.
## Database Operations
- [ ] Multi-step DB operations are wrapped in transactions (INSERT+INSERT, UPDATE+DELETE, read-modify-write)
- [ ] Both postgres AND libsql backends updated for any new Database trait methods
- [ ] Migrations are atomic (SQL execution + version recording in same transaction)
## Security & Data Safety
- [ ] Tool parameters are redacted via `redact_params()` before logging or SSE/WebSocket broadcast
- [ ] URL validation resolves DNS before checking for private/loopback IPs (anti-SSRF via DNS rebinding)
- [ ] Destructive tools have `requires_approval()` returning `Always` or `UnlessAutoApproved`
- [ ] Data from worker containers is treated as untrusted (tool domain checks, server-side nesting depth)
- [ ] No secrets or credentials in error messages, logs, or SSE events
## String Safety
- [ ] No byte-index slicing (`&s[..n]`) on external/user strings -- use `is_char_boundary()` or `char_indices()`
- [ ] File extension and media type comparisons are case-insensitive (`.to_ascii_lowercase()` before matching)
- [ ] Path comparisons are case-insensitive where needed (macOS/Windows filesystems)
## Trait Wrappers & Decorator Chain
- [ ] New `LlmProvider` trait methods are delegated in ALL wrapper types (grep `impl LlmProvider for`)
- [ ] New trait methods are tested through the full decorator/provider chain, not just the base impl
- [ ] Default trait method implementations are intentional -- wrappers that silently return defaults are bugs
## Tests
- [ ] Temporary files/dirs use `tempfile` crate, no hardcoded `/tmp/` paths
- [ ] Tests don't mutate global statics without synchronization (use per-test state or `serial_test`)
- [ ] Tests don't make real network requests (use mocks, stubs, or RFC 5737 TEST-NET IPs like 192.0.2.1)
- [ ] Test names and comments match actual test behavior and assertions
## Comments & Documentation
- [ ] Code comments match actual behavior (especially route paths, tool names, function semantics)
- [ ] Spec/README files updated if module behavior changed
- [ ] Error messages are clear and non-redundant (don't nest tool name inside tool error that already contains it)
+38 -8
View File
@@ -96,6 +96,9 @@ pub struct Agent {
pub(super) heartbeat_config: Option<HeartbeatConfig>,
pub(super) hygiene_config: Option<crate::config::HygieneConfig>,
pub(super) routine_config: Option<RoutineConfig>,
/// Optional slot to expose the routine engine to the gateway for manual triggering.
pub(super) routine_engine_slot:
Option<Arc<tokio::sync::RwLock<Option<Arc<crate::agent::routine_engine::RoutineEngine>>>>>,
}
impl Agent {
@@ -148,9 +151,18 @@ impl Agent {
heartbeat_config,
hygiene_config,
routine_config,
routine_engine_slot: None,
}
}
/// Set the routine engine slot for exposing the engine to the gateway.
pub fn set_routine_engine_slot(
&mut self,
slot: Arc<tokio::sync::RwLock<Option<Arc<crate::agent::routine_engine::RoutineEngine>>>>,
) {
self.routine_engine_slot = Some(slot);
}
// Convenience accessors
/// Get the scheduler (for external wiring, e.g. CreateJobTool).
@@ -342,8 +354,19 @@ impl Agent {
let heartbeat_handle = if let Some(ref hb_config) = self.heartbeat_config {
if hb_config.enabled {
if let Some(workspace) = self.workspace() {
let config = AgentHeartbeatConfig::default()
let mut config = AgentHeartbeatConfig::default()
.with_interval(std::time::Duration::from_secs(hb_config.interval_secs));
config.quiet_hours_start = hb_config.quiet_hours_start;
config.quiet_hours_end = hb_config.quiet_hours_end;
config.timezone = hb_config
.timezone
.clone()
.or_else(|| Some(self.config.default_timezone.clone()));
if let (Some(user), Some(channel)) =
(&hb_config.notify_user, &hb_config.notify_channel)
{
config = config.with_notify(user, channel);
}
// Set up notification channel
let (notify_tx, mut notify_rx) =
@@ -394,8 +417,8 @@ impl Agent {
hygiene,
workspace.clone(),
self.cheap_llm().clone(),
self.safety().clone(),
Some(notify_tx),
self.store().map(Arc::clone),
))
} else {
tracing::warn!("Heartbeat enabled but no workspace available");
@@ -423,6 +446,8 @@ impl Agent {
Arc::clone(workspace),
notify_tx,
Some(self.scheduler.clone()),
self.tools().clone(),
self.safety().clone(),
));
// Register routine tools
@@ -486,7 +511,12 @@ impl Agent {
// SAFETY: self is consumed by run(), we can smuggle the engine in
// via a local to use in the message loop below.
tracing::info!(
// Expose engine to gateway for manual triggering
if let Some(ref slot) = self.routine_engine_slot {
*slot.write().await = Some(Arc::clone(&engine));
}
tracing::debug!(
"Routines enabled: cron ticker every {}s, max {} concurrent",
rt_config.cron_check_interval_secs,
rt_config.max_concurrent_routines
@@ -508,20 +538,20 @@ impl Agent {
let routine_engine_for_loop = routine_handle.as_ref().map(|(_, e)| Arc::clone(e));
// Main message loop
tracing::info!("Agent {} ready and listening", self.config.name);
tracing::debug!("Agent {} ready and listening", self.config.name);
loop {
let message = tokio::select! {
biased;
_ = tokio::signal::ctrl_c() => {
tracing::info!("Ctrl+C received, shutting down...");
tracing::debug!("Ctrl+C received, shutting down...");
break;
}
msg = message_stream.next() => {
match msg {
Some(m) => m,
None => {
tracing::info!("All channel streams ended, shutting down...");
tracing::debug!("All channel streams ended, shutting down...");
break;
}
}
@@ -596,7 +626,7 @@ impl Agent {
}
Ok(None) => {
// Shutdown signal received (/quit, /exit, /shutdown)
tracing::info!("Shutdown command received, exiting...");
tracing::debug!("Shutdown command received, exiting...");
break;
}
Err(e) => {
@@ -625,7 +655,7 @@ impl Agent {
}
// Cleanup
tracing::info!("Agent shutting down...");
tracing::debug!("Agent shutting down...");
repair_handle.abort();
pruning_handle.abort();
if let Some(handle) = heartbeat_handle {
+48 -7
View File
@@ -345,7 +345,6 @@ impl Agent {
crate::workspace::hygiene::HygieneConfig::default(),
workspace.clone(),
self.llm().clone(),
self.safety().clone(),
);
match runner.check_heartbeat().await {
@@ -406,7 +405,7 @@ impl Agent {
.with_max_tokens(512)
.with_temperature(0.3);
let reasoning = Reasoning::new(self.llm().clone(), self.safety().clone());
let reasoning = Reasoning::new(self.llm().clone());
match reasoning.complete(request).await {
Ok((text, _usage)) => Ok(SubmissionResult::response(format!(
"Thread Summary:\n\n{}",
@@ -454,7 +453,7 @@ impl Agent {
.with_max_tokens(512)
.with_temperature(0.5);
let reasoning = Reasoning::new(self.llm().clone(), self.safety().clone());
let reasoning = Reasoning::new(self.llm().clone());
match reasoning.complete(request).await {
Ok((text, _usage)) => Ok(SubmissionResult::response(format!(
"Suggested Next Steps:\n\n{}",
@@ -663,10 +662,14 @@ impl Agent {
}
match self.llm().set_model(requested) {
Ok(()) => Ok(SubmissionResult::response(format!(
"Switched model to: {}",
requested
))),
Ok(()) => {
// Persist the model choice so it survives restarts.
self.persist_selected_model(requested).await;
Ok(SubmissionResult::response(format!(
"Switched model to: {}",
requested
)))
}
Err(e) => Ok(SubmissionResult::error(format!(
"Failed to switch model: {}",
e
@@ -822,4 +825,42 @@ impl Agent {
_ => Ok(None),
}
}
/// Persist the selected model to the settings store (DB and/or TOML config).
///
/// Best-effort: logs warnings on failure but does not propagate errors,
/// since the in-memory model switch already succeeded.
async fn persist_selected_model(&self, model: &str) {
// 1. Persist to DB if available.
if let Some(store) = self.store() {
let value = serde_json::Value::String(model.to_string());
if let Err(e) = store.set_setting("default", "selected_model", &value).await {
tracing::warn!("Failed to persist model to DB: {}", e);
}
}
// 2. Update TOML config file if it exists (sync I/O in spawn_blocking).
let model_owned = model.to_string();
if let Err(e) = tokio::task::spawn_blocking(move || {
let toml_path = crate::settings::Settings::default_toml_path();
match crate::settings::Settings::load_toml(&toml_path) {
Ok(Some(mut settings)) => {
settings.selected_model = Some(model_owned);
if let Err(e) = settings.save_toml(&toml_path) {
tracing::warn!("Failed to persist model to config.toml: {}", e);
}
}
Ok(None) => {
// No config file on disk; nothing to update.
}
Err(e) => {
tracing::warn!("Failed to load config.toml for model persistence: {}", e);
}
}
})
.await
{
tracing::warn!("Model TOML persistence task failed: {}", e);
}
}
}
+112 -37
View File
@@ -13,7 +13,6 @@ use crate::agent::context_monitor::{CompactionStrategy, ContextBreakdown};
use crate::agent::session::Thread;
use crate::error::Error;
use crate::llm::{ChatMessage, CompletionRequest, LlmProvider, Reasoning};
use crate::safety::SafetyLayer;
use crate::workspace::Workspace;
/// Result of a compaction operation.
@@ -34,13 +33,12 @@ pub struct CompactionResult {
/// Compacts conversation context to stay within limits.
pub struct ContextCompactor {
llm: Arc<dyn LlmProvider>,
safety: Arc<SafetyLayer>,
}
impl ContextCompactor {
/// Create a new context compactor.
pub fn new(llm: Arc<dyn LlmProvider>, safety: Arc<SafetyLayer>) -> Self {
Self { llm, safety }
pub fn new(llm: Arc<dyn LlmProvider>) -> Self {
Self { llm }
}
/// Compact a thread's context using the given strategy.
@@ -105,27 +103,26 @@ impl ContextCompactor {
// Generate summary
let summary = self.generate_summary(&to_summarize).await?;
// Write to workspace if available
let summary_written = if let Some(ws) = workspace {
// Write to workspace if available.
// If archival fails, preserve turns to avoid context loss.
let (summary_written, turns_removed) = if let Some(ws) = workspace {
match self.write_summary_to_workspace(ws, &summary).await {
Ok(()) => true,
Ok(()) => {
thread.truncate_turns(keep_recent);
(true, turns_to_remove)
}
Err(e) => {
tracing::warn!(
"Compaction summary write failed (turns will still be truncated): {}",
e
);
false
tracing::warn!("Compaction summary write failed (turns preserved): {}", e);
(false, 0)
}
}
} else {
false
thread.truncate_turns(keep_recent);
(false, turns_to_remove)
};
// Truncate thread
thread.truncate_turns(keep_recent);
Ok(CompactionPartial {
turns_removed: turns_to_remove,
turns_removed,
summary_written,
summary: Some(summary),
})
@@ -167,23 +164,20 @@ impl ContextCompactor {
// Format turns for storage
let content = format_turns_for_storage(old_turns);
// Write to workspace
let written = match self.write_context_to_workspace(ws, &content).await {
Ok(()) => true,
// Write to workspace. If archival fails, preserve turns.
let (written, turns_removed) = match self.write_context_to_workspace(ws, &content).await {
Ok(()) => {
thread.truncate_turns(keep_recent);
(true, turns_to_remove)
}
Err(e) => {
tracing::warn!(
"Compaction context write failed (turns will still be truncated): {}",
e
);
false
tracing::warn!("Compaction context write failed (turns preserved): {}", e);
(false, 0)
}
};
// Truncate
thread.truncate_turns(keep_recent);
Ok(CompactionPartial {
turns_removed: turns_to_remove,
turns_removed,
summary_written: written,
summary: None,
})
@@ -233,7 +227,7 @@ Be brief but capture all important details. Use bullet points."#,
.with_max_tokens(1024)
.with_temperature(0.3);
let reasoning = Reasoning::new(self.llm.clone(), self.safety.clone());
let reasoning = Reasoning::new(self.llm.clone());
let (text, _) = reasoning.complete(request).await?;
Ok(text)
}
@@ -346,17 +340,11 @@ mod tests {
// === QA Plan - Compaction strategy tests ===
use crate::agent::context_monitor::CompactionStrategy;
use crate::config::SafetyConfig;
use crate::safety::SafetyLayer;
use crate::testing::StubLlm;
/// Helper: build a `ContextCompactor` with the given `StubLlm`.
fn make_compactor(llm: Arc<StubLlm>) -> ContextCompactor {
let safety = Arc::new(SafetyLayer::new(&SafetyConfig {
max_output_length: 100_000,
injection_check_enabled: false,
}));
ContextCompactor::new(llm, safety)
ContextCompactor::new(llm)
}
/// Helper: build a thread with `n` completed turns.
@@ -370,6 +358,19 @@ mod tests {
thread
}
#[cfg(feature = "libsql")]
async fn make_unmigrated_workspace() -> crate::workspace::Workspace {
use crate::db::Database;
use crate::db::libsql::LibSqlBackend;
// Intentionally skip migrations so workspace append operations fail.
let backend = LibSqlBackend::new_memory()
.await
.expect("should create in-memory libsql backend");
let db: Arc<dyn Database> = Arc::new(backend);
crate::workspace::Workspace::new_with_db("compaction-test", db)
}
// ------------------------------------------------------------------
// 1. compact_truncate keeps last N turns
// ------------------------------------------------------------------
@@ -568,6 +569,43 @@ mod tests {
assert_eq!(llm.calls(), 0);
}
#[cfg(feature = "libsql")]
#[tokio::test]
async fn test_compact_with_summary_preserves_turns_when_workspace_write_fails() {
let llm = Arc::new(StubLlm::new("summary"));
let compactor = make_compactor(llm.clone());
let mut thread = make_thread(8);
let original_inputs: Vec<String> =
thread.turns.iter().map(|t| t.user_input.clone()).collect();
let workspace = make_unmigrated_workspace().await;
let result = compactor
.compact(
&mut thread,
CompactionStrategy::Summarize { keep_recent: 3 },
Some(&workspace),
)
.await
.expect("compact should succeed even when workspace write fails");
// On archival failure, no turns should be removed.
assert_eq!(thread.turns.len(), 8);
assert_eq!(
thread
.turns
.iter()
.map(|t| t.user_input.as_str())
.collect::<Vec<_>>(),
original_inputs
.iter()
.map(|s| s.as_str())
.collect::<Vec<_>>()
);
assert_eq!(result.turns_removed, 0);
assert!(!result.summary_written);
assert_eq!(llm.calls(), 1);
}
// ------------------------------------------------------------------
// 7. compact_to_workspace without workspace falls back to truncation
// ------------------------------------------------------------------
@@ -616,6 +654,43 @@ mod tests {
assert_eq!(result.turns_removed, 0);
}
#[cfg(feature = "libsql")]
#[tokio::test]
async fn test_compact_to_workspace_preserves_turns_when_workspace_write_fails() {
let llm = Arc::new(StubLlm::new("unused"));
let compactor = make_compactor(llm.clone());
let mut thread = make_thread(20);
let original_inputs: Vec<String> =
thread.turns.iter().map(|t| t.user_input.clone()).collect();
let workspace = make_unmigrated_workspace().await;
let result = compactor
.compact(
&mut thread,
CompactionStrategy::MoveToWorkspace,
Some(&workspace),
)
.await
.expect("compact should succeed even when workspace write fails");
// On archival failure, no turns should be removed.
assert_eq!(thread.turns.len(), 20);
assert_eq!(
thread
.turns
.iter()
.map(|t| t.user_input.as_str())
.collect::<Vec<_>>(),
original_inputs
.iter()
.map(|s| s.as_str())
.collect::<Vec<_>>()
);
assert_eq!(result.turns_removed, 0);
assert!(!result.summary_written);
assert_eq!(llm.calls(), 0);
}
// ------------------------------------------------------------------
// 9. format_turns_for_storage includes tool calls
// ------------------------------------------------------------------
+42 -7
View File
@@ -131,10 +131,12 @@ impl CostGuard {
// Check hourly rate
if let Some(limit) = self.config.max_actions_per_hour {
let mut window = self.action_window.lock().await;
let cutoff = Instant::now() - std::time::Duration::from_secs(3600);
// Drain expired entries
while window.front().is_some_and(|t| *t < cutoff) {
window.pop_front();
// checked_sub avoids panic when system uptime < 1 hour (Windows)
if let Some(cutoff) = Instant::now().checked_sub(std::time::Duration::from_secs(3600)) {
// Drain expired entries
while window.front().is_some_and(|t| *t < cutoff) {
window.pop_front();
}
}
let count = window.len() as u64;
if count >= limit {
@@ -260,9 +262,11 @@ impl CostGuard {
/// Number of actions in the current hourly window.
pub async fn actions_this_hour(&self) -> u64 {
let mut window = self.action_window.lock().await;
let cutoff = Instant::now() - std::time::Duration::from_secs(3600);
while window.front().is_some_and(|t| *t < cutoff) {
window.pop_front();
// checked_sub avoids panic when system uptime < 1 hour (Windows)
if let Some(cutoff) = Instant::now().checked_sub(std::time::Duration::from_secs(3600)) {
while window.front().is_some_and(|t| *t < cutoff) {
window.pop_front();
}
}
window.len() as u64
}
@@ -621,4 +625,35 @@ mod tests {
"surcharge should be 100% of input cost for 1h cache writes"
);
}
/// Regression test for #657: Instant::now() - Duration panics on Windows
/// when system uptime is less than the subtracted duration.
#[tokio::test]
async fn test_checked_sub_no_panic_on_fresh_guard() {
// A fresh CostGuard with rate limits should not panic even if
// checked_sub returns None (simulating short uptime).
let guard = CostGuard::new(CostGuardConfig {
max_cost_per_day_cents: None,
max_actions_per_hour: Some(100),
});
// These must not panic regardless of system uptime
assert!(guard.check_allowed().await.is_ok());
assert_eq!(guard.actions_this_hour().await, 0);
// Record some actions and verify again
guard
.record_llm_call("gpt-4o", 10, 10, 0, 0, Decimal::ONE, Decimal::ONE, None)
.await;
assert!(guard.check_allowed().await.is_ok());
assert_eq!(guard.actions_this_hour().await, 1);
}
/// Verify that checked_sub itself behaves as expected for the pattern we use.
#[test]
fn test_instant_checked_sub_returns_none_for_overflow() {
// Duration::MAX will always exceed uptime, so checked_sub must return None
let result = Instant::now().checked_sub(std::time::Duration::MAX);
assert!(result.is_none());
}
}
+221 -31
View File
@@ -50,8 +50,18 @@ impl Agent {
// Load workspace system prompt (identity files: AGENTS.md, SOUL.md, etc.)
// In group chats, MEMORY.md is excluded to prevent leaking personal context.
// Resolve the user's timezone
let user_tz = crate::timezone::resolve_timezone(
message.timezone.as_deref(),
None, // user setting lookup can be added later
&self.config.default_timezone,
);
let system_prompt = if let Some(ws) = self.workspace() {
match ws.system_prompt_for_context(is_group_chat).await {
match ws
.system_prompt_for_context_tz(is_group_chat, user_tz)
.await
{
Ok(prompt) if !prompt.is_empty() => Some(prompt),
Ok(_) => None,
Err(e) => {
@@ -103,7 +113,7 @@ impl Agent {
None
};
let mut reasoning = Reasoning::new(self.llm().clone(), self.safety().clone())
let mut reasoning = Reasoning::new(self.llm().clone())
.with_channel(message.channel.clone())
.with_model_name(self.llm().active_model_name())
.with_group_chat(is_group_chat);
@@ -130,6 +140,7 @@ impl Agent {
let mut job_ctx =
JobContext::with_user(&message.user_id, "chat", "Interactive chat session");
job_ctx.http_interceptor = self.deps.http_interceptor.clone();
job_ctx.user_timezone = user_tz.name().to_string();
// Build system prompts once for this turn. Two variants: with tools
// (normal iterations) and without (force_text final iteration).
@@ -670,8 +681,53 @@ impl Agent {
.into())
});
// Send ToolResult preview
if let Ok(ref output) = tool_result
// Detect image generation sentinel in tool output
// (only from image tools — avoids parsing all tool outputs)
let is_image_sentinel = if let Ok(ref output) = tool_result
&& matches!(tc.name.as_str(), "image_generate" | "image_edit")
{
if let Ok(sentinel) =
serde_json::from_str::<serde_json::Value>(output)
&& sentinel.get("type").and_then(|v| v.as_str())
== Some("image_generated")
{
let data_url = sentinel
.get("data")
.and_then(|v| v.as_str())
.unwrap_or_default()
.to_string();
let path = sentinel
.get("path")
.and_then(|v| v.as_str())
.map(String::from);
// Skip broadcasting if data_url is empty to avoid
// sending a broken ImageGenerated SSE event.
if data_url.is_empty() {
tracing::warn!(
"Image generation sentinel has empty data URL, skipping broadcast"
);
} else {
let _ = self
.channels
.send_status(
&message.channel,
StatusUpdate::ImageGenerated { data_url, path },
&message.metadata,
)
.await;
}
true
} else {
false
}
} else {
false
};
// Send ToolResult preview (skip for image sentinels to avoid
// broadcasting multi-MB base64 data as a preview)
if !is_image_sentinel
&& let Ok(ref output) = tool_result
&& !output.is_empty()
{
let _ = self
@@ -687,23 +743,6 @@ impl Agent {
.await;
}
// Record result in thread
{
let mut sess = session.lock().await;
if let Some(thread) = sess.threads.get_mut(&thread_id)
&& let Some(turn) = thread.last_turn_mut()
{
match &tool_result {
Ok(output) => {
turn.record_tool_result(serde_json::json!(output));
}
Err(e) => {
turn.record_tool_error(e.to_string());
}
}
}
}
// Check for auth awaiting — defer the return
// until all results are recorded.
if deferred_auth.is_none()
@@ -743,6 +782,7 @@ impl Agent {
}
// Sanitize and add tool result to context
let is_tool_error = tool_result.is_err();
let result_content = match tool_result {
Ok(output) => {
let sanitized =
@@ -756,6 +796,23 @@ impl Agent {
Err(e) => format!("Tool '{}' failed: {}", tc.name, e),
};
// Record sanitized result in thread so messages()
// and persist_tool_calls() use cleaned content.
{
let mut sess = session.lock().await;
if let Some(thread) = sess.threads.get_mut(&thread_id)
&& let Some(turn) = thread.last_turn_mut()
{
if is_tool_error {
turn.record_tool_error(result_content.clone());
} else {
turn.record_tool_result(serde_json::json!(
result_content
));
}
}
}
context_messages.push(ChatMessage::tool_result(
&tc.id,
&tc.name,
@@ -785,6 +842,7 @@ impl Agent {
tool_call_id: tc.id.clone(),
context_messages: context_messages.clone(),
deferred_tool_calls: tool_calls[approval_idx + 1..].to_vec(),
user_timezone: Some(user_tz.name().to_string()),
};
return Ok(AgenticLoopResult::NeedApproval { pending });
@@ -1146,6 +1204,8 @@ mod tests {
max_actions_per_hour: None,
max_tool_iterations: 50,
auto_approve_tools: false,
default_timezone: "UTC".to_string(),
max_tokens_per_job: 0,
},
deps,
Arc::new(ChannelManager::new()),
@@ -1204,6 +1264,96 @@ mod tests {
}
}
#[test]
fn test_always_approval_requirement_bypasses_session_auto_approve() {
// Regression test: even if tool is auto-approved in session,
// ApprovalRequirement::Always must still trigger approval.
use crate::tools::ApprovalRequirement;
let mut session = Session::new("user-1");
let tool_name = "tool_remove";
// Manually auto-approve tool_remove in this session
session.auto_approve_tool(tool_name);
assert!(
session.is_tool_auto_approved(tool_name),
"tool should be auto-approved"
);
// However, ApprovalRequirement::Always should always require approval
// This is verified by the dispatcher logic: Always => true (ignores session state)
let always_req = ApprovalRequirement::Always;
let requires_approval = match always_req {
ApprovalRequirement::Never => false,
ApprovalRequirement::UnlessAutoApproved => !session.is_tool_auto_approved(tool_name),
ApprovalRequirement::Always => true,
};
assert!(
requires_approval,
"ApprovalRequirement::Always must require approval even when tool is auto-approved"
);
}
#[test]
fn test_always_approval_requirement_vs_unless_auto_approved() {
// Verify the two requirements behave differently
use crate::tools::ApprovalRequirement;
let mut session = Session::new("user-2");
let tool_name = "http";
// Scenario 1: Tool is auto-approved
session.auto_approve_tool(tool_name);
// UnlessAutoApproved → doesn't require approval if auto-approved
let unless_req = ApprovalRequirement::UnlessAutoApproved;
let unless_needs = match unless_req {
ApprovalRequirement::Never => false,
ApprovalRequirement::UnlessAutoApproved => !session.is_tool_auto_approved(tool_name),
ApprovalRequirement::Always => true,
};
assert!(
!unless_needs,
"UnlessAutoApproved should not need approval when auto-approved"
);
// Always → always requires approval
let always_req = ApprovalRequirement::Always;
let always_needs = match always_req {
ApprovalRequirement::Never => false,
ApprovalRequirement::UnlessAutoApproved => !session.is_tool_auto_approved(tool_name),
ApprovalRequirement::Always => true,
};
assert!(
always_needs,
"Always must always require approval, even when auto-approved"
);
// Scenario 2: Tool is NOT auto-approved
let new_tool = "new_tool";
assert!(!session.is_tool_auto_approved(new_tool));
// UnlessAutoApproved → requires approval
let unless_needs = match unless_req {
ApprovalRequirement::Never => false,
ApprovalRequirement::UnlessAutoApproved => !session.is_tool_auto_approved(new_tool),
ApprovalRequirement::Always => true,
};
assert!(
unless_needs,
"UnlessAutoApproved should need approval when not auto-approved"
);
// Always → always requires approval
let always_needs = match always_req {
ApprovalRequirement::Never => false,
ApprovalRequirement::UnlessAutoApproved => !session.is_tool_auto_approved(new_tool),
ApprovalRequirement::Always => true,
};
assert!(always_needs, "Always must always require approval");
}
#[test]
fn test_pending_approval_serialization_backcompat_without_deferred_calls() {
// PendingApproval from before the deferred_tool_calls field was added
@@ -1248,6 +1398,7 @@ mod tests {
arguments: serde_json::json!({"message": "done"}),
},
],
user_timezone: None,
};
let json = serde_json::to_string(&pending).expect("serialize");
@@ -1595,12 +1746,8 @@ mod tests {
use crate::testing::StubLlm;
let stub = Arc::new(StubLlm::failing_non_transient("ctx-bomb"));
let safety = Arc::new(SafetyLayer::new(&SafetyConfig {
max_output_length: 100_000,
injection_check_enabled: false,
}));
let reasoning = Reasoning::new(stub.clone(), safety);
let reasoning = Reasoning::new(stub.clone());
// Build a fat context with lots of history.
let messages = vec![
@@ -1710,11 +1857,7 @@ mod tests {
use crate::llm::{Reasoning, ReasoningContext, RespondResult, ToolDefinition};
let provider = Arc::new(AlwaysToolCallProvider);
let safety = Arc::new(SafetyLayer::new(&SafetyConfig {
max_output_length: 100_000,
injection_check_enabled: false,
}));
let reasoning = Reasoning::new(provider, safety);
let reasoning = Reasoning::new(provider);
let tool_def = ToolDefinition {
name: "echo".to_string(),
@@ -1900,6 +2043,8 @@ mod tests {
max_actions_per_hour: None,
max_tool_iterations,
auto_approve_tools: true,
default_timezone: "UTC".to_string(),
max_tokens_per_job: 0,
},
deps,
Arc::new(ChannelManager::new()),
@@ -2015,6 +2160,8 @@ mod tests {
max_actions_per_hour: None,
max_tool_iterations: max_iter,
auto_approve_tools: true,
default_timezone: "UTC".to_string(),
max_tokens_per_job: 0,
},
deps,
Arc::new(ChannelManager::new()),
@@ -2116,4 +2263,47 @@ mod tests {
"Error should include the underlying reason, got: {formatted}"
);
}
#[test]
fn test_image_sentinel_empty_data_url_should_be_skipped() {
// Regression: unwrap_or_default() on missing "data" field produces an empty
// string. Broadcasting an empty data_url would send a broken SSE event.
let sentinel = serde_json::json!({
"type": "image_generated",
"path": "/tmp/image.png"
// "data" field is missing
});
let data_url = sentinel
.get("data")
.and_then(|v| v.as_str())
.unwrap_or_default()
.to_string();
assert!(
data_url.is_empty(),
"Missing 'data' field should produce empty string"
);
// The fix: empty data_url means we skip broadcasting
}
#[test]
fn test_image_sentinel_present_data_url_is_valid() {
let sentinel = serde_json::json!({
"type": "image_generated",
"data": "data:image/png;base64,abc123",
"path": "/tmp/image.png"
});
let data_url = sentinel
.get("data")
.and_then(|v| v.as_str())
.unwrap_or_default()
.to_string();
assert!(
!data_url.is_empty(),
"Present 'data' field should produce non-empty string"
);
}
}
+168 -8
View File
@@ -29,8 +29,8 @@ use std::time::Duration;
use tokio::sync::mpsc;
use crate::channels::OutgoingResponse;
use crate::db::Database;
use crate::llm::{ChatMessage, CompletionRequest, LlmProvider, Reasoning};
use crate::safety::SafetyLayer;
use crate::workspace::Workspace;
use crate::workspace::hygiene::HygieneConfig;
@@ -47,6 +47,12 @@ pub struct HeartbeatConfig {
pub notify_user_id: Option<String>,
/// Channel to notify on heartbeat findings.
pub notify_channel: Option<String>,
/// Hour (0-23) when quiet hours start.
pub quiet_hours_start: Option<u32>,
/// Hour (0-23) when quiet hours end.
pub quiet_hours_end: Option<u32>,
/// Timezone for quiet hours evaluation (IANA name).
pub timezone: Option<String>,
}
impl Default for HeartbeatConfig {
@@ -57,6 +63,9 @@ impl Default for HeartbeatConfig {
max_failures: 3,
notify_user_id: None,
notify_channel: None,
quiet_hours_start: None,
quiet_hours_end: None,
timezone: None,
}
}
}
@@ -74,6 +83,26 @@ impl HeartbeatConfig {
self
}
/// Check whether the current time falls within configured quiet hours.
pub fn is_quiet_hours(&self) -> bool {
use chrono::Timelike;
let (Some(start), Some(end)) = (self.quiet_hours_start, self.quiet_hours_end) else {
return false;
};
let tz = self
.timezone
.as_deref()
.and_then(crate::timezone::parse_timezone)
.unwrap_or(chrono_tz::UTC);
let now_hour = crate::timezone::now_in_tz(tz).hour();
if start <= end {
now_hour >= start && now_hour < end
} else {
// Wraps midnight, e.g. 22..06
now_hour >= start || now_hour < end
}
}
/// Set the notification target.
pub fn with_notify(mut self, user_id: impl Into<String>, channel: impl Into<String>) -> Self {
self.notify_user_id = Some(user_id.into());
@@ -101,8 +130,8 @@ pub struct HeartbeatRunner {
hygiene_config: HygieneConfig,
workspace: Arc<Workspace>,
llm: Arc<dyn LlmProvider>,
safety: Arc<SafetyLayer>,
response_tx: Option<mpsc::Sender<OutgoingResponse>>,
store: Option<Arc<dyn Database>>,
consecutive_failures: u32,
}
@@ -113,15 +142,14 @@ impl HeartbeatRunner {
hygiene_config: HygieneConfig,
workspace: Arc<Workspace>,
llm: Arc<dyn LlmProvider>,
safety: Arc<SafetyLayer>,
) -> Self {
Self {
config,
hygiene_config,
workspace,
llm,
safety,
response_tx: None,
store: None,
consecutive_failures: 0,
}
}
@@ -132,6 +160,12 @@ impl HeartbeatRunner {
self
}
/// Set the database store for persistent heartbeat conversations.
pub fn with_store(mut self, store: Arc<dyn Database>) -> Self {
self.store = Some(store);
self
}
/// Run the heartbeat loop.
///
/// This runs forever, checking periodically based on the configured interval.
@@ -153,6 +187,12 @@ impl HeartbeatRunner {
loop {
interval.tick().await;
// Skip during quiet hours
if self.config.is_quiet_hours() {
tracing::debug!("Heartbeat skipped: quiet hours");
continue;
}
// Run memory hygiene in the background so it never delays the
// heartbeat checklist. Failures are logged inside run_if_due.
let hygiene_workspace = Arc::clone(&self.workspace);
@@ -263,7 +303,7 @@ impl HeartbeatRunner {
.with_max_tokens(max_tokens)
.with_temperature(0.3);
let reasoning = Reasoning::new(self.llm.clone(), self.safety.clone());
let reasoning = Reasoning::new(self.llm.clone());
let (content, _usage) = match reasoning.complete(request).await {
Ok(r) => r,
Err(e) => return HeartbeatResult::Failed(format!("LLM call failed: {}", e)),
@@ -292,9 +332,32 @@ impl HeartbeatRunner {
return;
};
let user_id = self.config.notify_user_id.as_deref().unwrap_or("default");
// Persist to heartbeat conversation and get thread_id
let thread_id = if let Some(ref store) = self.store {
match store.get_or_create_heartbeat_conversation(user_id).await {
Ok(conv_id) => {
if let Err(e) = store
.add_conversation_message(conv_id, "assistant", message)
.await
{
tracing::error!("Failed to persist heartbeat message: {}", e);
}
Some(conv_id.to_string())
}
Err(e) => {
tracing::error!("Failed to get heartbeat conversation: {}", e);
None
}
}
} else {
None
};
let response = OutgoingResponse {
content: format!("🔔 *Heartbeat Alert*\n\n{}", message),
thread_id: None,
thread_id,
attachments: Vec::new(),
metadata: serde_json::json!({
"source": "heartbeat",
@@ -354,13 +417,16 @@ pub fn spawn_heartbeat(
hygiene_config: HygieneConfig,
workspace: Arc<Workspace>,
llm: Arc<dyn LlmProvider>,
safety: Arc<SafetyLayer>,
response_tx: Option<mpsc::Sender<OutgoingResponse>>,
store: Option<Arc<dyn Database>>,
) -> tokio::task::JoinHandle<()> {
let mut runner = HeartbeatRunner::new(config, hygiene_config, workspace, llm, safety);
let mut runner = HeartbeatRunner::new(config, hygiene_config, workspace, llm);
if let Some(tx) = response_tx {
runner = runner.with_response_channel(tx);
}
if let Some(s) = store {
runner = runner.with_store(s);
}
tokio::spawn(async move {
runner.run().await;
@@ -495,4 +561,98 @@ mod tests {
let content = "<!-- comment -->\nActual task here";
assert!(!is_effectively_empty(content));
}
// ==================== quiet hours ====================
#[test]
fn test_quiet_hours_inside() {
use chrono::{Timelike, Utc};
let now_utc = Utc::now();
let hour = now_utc.hour();
let start = hour;
let end = (hour + 1) % 24;
let config = HeartbeatConfig {
quiet_hours_start: Some(start),
quiet_hours_end: Some(end),
timezone: Some("UTC".to_string()),
..HeartbeatConfig::default()
};
// Current UTC hour is inside [start, end) by construction
assert!(config.is_quiet_hours());
}
#[test]
fn test_quiet_hours_outside() {
use chrono::{Timelike, Utc};
let now_utc = Utc::now();
let hour = now_utc.hour();
let start = (hour + 1) % 24;
let end = (hour + 2) % 24;
let config = HeartbeatConfig {
quiet_hours_start: Some(start),
quiet_hours_end: Some(end),
timezone: Some("UTC".to_string()),
..HeartbeatConfig::default()
};
// Current UTC hour is outside [start, end) by construction
assert!(!config.is_quiet_hours());
}
#[test]
fn test_quiet_hours_wraparound_excludes_now() {
use chrono::{Timelike, Utc};
let now_utc = Utc::now();
let hour = now_utc.hour();
// Window covers all hours except the current one
let start = (hour + 1) % 24;
let end = hour;
let config = HeartbeatConfig {
quiet_hours_start: Some(start),
quiet_hours_end: Some(end),
timezone: Some("UTC".to_string()),
..HeartbeatConfig::default()
};
assert!(!config.is_quiet_hours());
}
#[test]
fn test_quiet_hours_none_configured() {
let config = HeartbeatConfig::default();
assert!(!config.is_quiet_hours());
}
#[test]
fn test_quiet_hours_same_start_end() {
let config = HeartbeatConfig {
quiet_hours_start: Some(10),
quiet_hours_end: Some(10),
timezone: Some("UTC".to_string()),
..HeartbeatConfig::default()
};
// start == end means zero-width window, should be false
assert!(!config.is_quiet_hours());
}
#[test]
fn test_spawn_heartbeat_accepts_store_param() {
// Regression: spawn_heartbeat must accept an optional Database store
// for persisting heartbeat notifications to a dedicated conversation.
// Compile-time check: the 7th parameter is `Option<Arc<dyn Database>>`.
#[allow(clippy::type_complexity)]
let _fn_ptr: fn(
HeartbeatConfig,
HygieneConfig,
Arc<crate::workspace::Workspace>,
Arc<dyn crate::llm::LlmProvider>,
Option<tokio::sync::mpsc::Sender<crate::channels::OutgoingResponse>>,
Option<Arc<dyn crate::db::Database>>,
) -> tokio::task::JoinHandle<()> = spawn_heartbeat;
let _ = _fn_ptr;
}
}
+87 -9
View File
@@ -57,7 +57,11 @@ pub struct Routine {
#[serde(tag = "type", rename_all = "snake_case")]
pub enum Trigger {
/// Fire on a cron schedule (e.g. "0 9 * * MON-FRI" or "every 2h").
Cron { schedule: String },
Cron {
schedule: String,
#[serde(default)]
timezone: Option<String>,
},
/// Fire when a channel message matches a pattern.
Event {
/// Optional channel filter (e.g. "telegram", "slack").
@@ -99,7 +103,21 @@ impl Trigger {
field: "schedule".into(),
})?
.to_string();
Ok(Trigger::Cron { schedule })
let timezone = config
.get("timezone")
.and_then(|v| v.as_str())
.and_then(|tz| {
if crate::timezone::parse_timezone(tz).is_some() {
Some(tz.to_string())
} else {
tracing::warn!(
"Ignoring invalid timezone '{}' from DB for cron trigger",
tz
);
None
}
});
Ok(Trigger::Cron { schedule, timezone })
}
"event" => {
let pattern = config
@@ -137,7 +155,10 @@ impl Trigger {
/// Serialize trigger-specific config to JSON for DB storage.
pub fn to_config_json(&self) -> serde_json::Value {
match self {
Trigger::Cron { schedule } => serde_json::json!({ "schedule": schedule }),
Trigger::Cron { schedule, timezone } => serde_json::json!({
"schedule": schedule,
"timezone": timezone,
}),
Trigger::Event { channel, pattern } => serde_json::json!({
"pattern": pattern,
"channel": channel,
@@ -415,12 +436,25 @@ pub fn content_hash(content: &str) -> u64 {
}
/// Parse a cron expression and compute the next fire time from now.
pub fn next_cron_fire(schedule: &str) -> Result<Option<DateTime<Utc>>, RoutineError> {
///
/// When `timezone` is provided and valid, the schedule is evaluated in that
/// timezone and the result is converted back to UTC. Otherwise UTC is used.
pub fn next_cron_fire(
schedule: &str,
timezone: Option<&str>,
) -> Result<Option<DateTime<Utc>>, RoutineError> {
let cron_schedule =
cron::Schedule::from_str(schedule).map_err(|e| RoutineError::InvalidCron {
reason: e.to_string(),
})?;
Ok(cron_schedule.upcoming(Utc).next())
if let Some(tz) = timezone.and_then(crate::timezone::parse_timezone) {
Ok(cron_schedule
.upcoming(tz)
.next()
.map(|dt| dt.with_timezone(&Utc)))
} else {
Ok(cron_schedule.upcoming(Utc).next())
}
}
#[cfg(test)]
@@ -433,10 +467,11 @@ mod tests {
fn test_trigger_roundtrip() {
let trigger = Trigger::Cron {
schedule: "0 9 * * MON-FRI".to_string(),
timezone: None,
};
let json = trigger.to_config_json();
let parsed = Trigger::from_db("cron", json).expect("parse cron");
assert!(matches!(parsed, Trigger::Cron { schedule } if schedule == "0 9 * * MON-FRI"));
assert!(matches!(parsed, Trigger::Cron { schedule, .. } if schedule == "0 9 * * MON-FRI"));
}
#[test]
@@ -509,16 +544,58 @@ mod tests {
#[test]
fn test_next_cron_fire_valid() {
// Every minute should always have a next fire
let next = next_cron_fire("* * * * * *").expect("valid cron");
let next = next_cron_fire("* * * * * *", None).expect("valid cron");
assert!(next.is_some());
}
#[test]
fn test_next_cron_fire_invalid() {
let result = next_cron_fire("not a cron");
let result = next_cron_fire("not a cron", None);
assert!(result.is_err());
}
#[test]
fn test_trigger_cron_timezone_roundtrip() {
let trigger = Trigger::Cron {
schedule: "0 9 * * MON-FRI".to_string(),
timezone: Some("America/New_York".to_string()),
};
let json = trigger.to_config_json();
let parsed = Trigger::from_db("cron", json).expect("parse cron");
assert!(matches!(parsed, Trigger::Cron { schedule, timezone }
if schedule == "0 9 * * MON-FRI"
&& timezone.as_deref() == Some("America/New_York")));
}
#[test]
fn test_trigger_cron_no_timezone_backward_compat() {
let json = serde_json::json!({"schedule": "0 9 * * *"});
let parsed = Trigger::from_db("cron", json).expect("parse cron");
assert!(matches!(parsed, Trigger::Cron { timezone, .. } if timezone.is_none()));
}
#[test]
fn test_trigger_cron_invalid_timezone_coerced_to_none() {
let json = serde_json::json!({"schedule": "0 9 * * *", "timezone": "Fake/Zone"});
let parsed = Trigger::from_db("cron", json).expect("parse cron");
assert!(
matches!(parsed, Trigger::Cron { timezone, .. } if timezone.is_none()),
"invalid timezone should be coerced to None"
);
}
#[test]
fn test_next_cron_fire_with_timezone() {
let next_utc = next_cron_fire("0 0 9 * * * *", None)
.expect("valid cron")
.expect("has next");
let next_est = next_cron_fire("0 0 9 * * * *", Some("America/New_York"))
.expect("valid cron")
.expect("has next");
// EST is UTC-5 (or EDT UTC-4), so the UTC result should differ
assert_ne!(next_utc, next_est, "timezone should shift the fire time");
}
#[test]
fn test_guardrails_default() {
let g = RoutineGuardrails::default();
@@ -531,7 +608,8 @@ mod tests {
fn test_trigger_type_tag() {
assert_eq!(
Trigger::Cron {
schedule: String::new()
schedule: String::new(),
timezone: None,
}
.type_tag(),
"cron"
+519 -29
View File
@@ -25,10 +25,14 @@ use crate::agent::routine::{
};
use crate::channels::{IncomingMessage, OutgoingResponse};
use crate::config::RoutineConfig;
use crate::context::JobContext;
use crate::db::Database;
use crate::error::RoutineError;
use crate::llm::{ChatMessage, CompletionRequest, FinishReason, LlmProvider};
use crate::tools::ApprovalContext;
use crate::llm::{
ChatMessage, CompletionRequest, FinishReason, LlmProvider, ToolCall, ToolCompletionRequest,
};
use crate::safety::SafetyLayer;
use crate::tools::{ApprovalContext, ApprovalRequirement, ToolError, ToolRegistry, redact_params};
use crate::workspace::Workspace;
/// The routine execution engine.
@@ -45,9 +49,14 @@ pub struct RoutineEngine {
event_cache: Arc<RwLock<Vec<(Uuid, Routine, Regex)>>>,
/// Scheduler for dispatching jobs (FullJob mode).
scheduler: Option<Arc<Scheduler>>,
/// Tool registry for lightweight routine tool execution.
tools: Arc<ToolRegistry>,
/// Safety layer for tool output sanitization.
safety: Arc<SafetyLayer>,
}
impl RoutineEngine {
#[allow(clippy::too_many_arguments)]
pub fn new(
config: RoutineConfig,
store: Arc<dyn Database>,
@@ -55,6 +64,8 @@ impl RoutineEngine {
workspace: Arc<Workspace>,
notify_tx: mpsc::Sender<OutgoingResponse>,
scheduler: Option<Arc<Scheduler>>,
tools: Arc<ToolRegistry>,
safety: Arc<SafetyLayer>,
) -> Self {
Self {
config,
@@ -65,6 +76,8 @@ impl RoutineEngine {
running_count: Arc::new(AtomicUsize::new(0)),
event_cache: Arc::new(RwLock::new(Vec::new())),
scheduler,
tools,
safety,
}
}
@@ -170,7 +183,7 @@ impl RoutineEngine {
continue;
}
let detail = if let Trigger::Cron { ref schedule } = routine.trigger {
let detail = if let Trigger::Cron { ref schedule, .. } = routine.trigger {
Some(schedule.clone())
} else {
None
@@ -184,7 +197,11 @@ impl RoutineEngine {
///
/// Bypasses cooldown checks (those only apply to cron/event triggers).
/// Still enforces enabled check and concurrent run limit.
pub async fn fire_manual(&self, routine_id: Uuid) -> Result<Uuid, RoutineError> {
pub async fn fire_manual(
&self,
routine_id: Uuid,
user_id: Option<&str>,
) -> Result<Uuid, RoutineError> {
let routine = self
.store
.get_routine(routine_id)
@@ -194,6 +211,13 @@ impl RoutineEngine {
})?
.ok_or(RoutineError::NotFound { id: routine_id })?;
// Enforce ownership when a user_id is provided (gateway calls).
if let Some(uid) = user_id
&& routine.user_id != uid
{
return Err(RoutineError::NotAuthorized { id: routine_id });
}
if !routine.enabled {
return Err(RoutineError::Disabled {
name: routine.name.clone(),
@@ -229,12 +253,15 @@ impl RoutineEngine {
// Execute inline for manual triggers (caller wants to wait)
let engine = EngineContext {
config: self.config.clone(),
store: self.store.clone(),
llm: self.llm.clone(),
workspace: self.workspace.clone(),
notify_tx: self.notify_tx.clone(),
running_count: self.running_count.clone(),
scheduler: self.scheduler.clone(),
tools: self.tools.clone(),
safety: self.safety.clone(),
};
tokio::spawn(async move {
@@ -261,12 +288,15 @@ impl RoutineEngine {
};
let engine = EngineContext {
config: self.config.clone(),
store: self.store.clone(),
llm: self.llm.clone(),
workspace: self.workspace.clone(),
notify_tx: self.notify_tx.clone(),
running_count: self.running_count.clone(),
scheduler: self.scheduler.clone(),
tools: self.tools.clone(),
safety: self.safety.clone(),
};
// Record the run in DB, then spawn execution
@@ -308,12 +338,15 @@ impl RoutineEngine {
/// Shared context passed to the execution function.
struct EngineContext {
config: RoutineConfig,
store: Arc<dyn Database>,
llm: Arc<dyn LlmProvider>,
workspace: Arc<Workspace>,
notify_tx: mpsc::Sender<OutgoingResponse>,
running_count: Arc<AtomicUsize>,
scheduler: Option<Arc<Scheduler>>,
tools: Arc<ToolRegistry>,
safety: Arc<SafetyLayer>,
}
/// Execute a routine run. Handles both lightweight and full_job modes.
@@ -369,8 +402,12 @@ async fn execute_routine(ctx: EngineContext, routine: Routine, run: RoutineRun)
// Update routine runtime state
let now = Utc::now();
let next_fire = if let Trigger::Cron { ref schedule } = routine.trigger {
next_cron_fire(schedule).unwrap_or(None)
let next_fire = if let Trigger::Cron {
ref schedule,
ref timezone,
} = routine.trigger
{
next_cron_fire(schedule, timezone.as_deref()).unwrap_or(None)
} else {
None
};
@@ -396,6 +433,39 @@ async fn execute_routine(ctx: EngineContext, routine: Routine, run: RoutineRun)
tracing::error!(routine = %routine.name, "Failed to update runtime state: {}", e);
}
// Persist routine result to its dedicated conversation thread
let thread_id = match ctx
.store
.get_or_create_routine_conversation(routine.id, &routine.name, &routine.user_id)
.await
{
Ok(conv_id) => {
tracing::debug!(
routine = %routine.name,
routine_id = %routine.id,
conversation_id = %conv_id,
"Resolved routine conversation thread"
);
// Record the run result as a conversation message
let msg = match (&summary, status) {
(Some(s), _) => format!("[{}] {}: {}", run.trigger_type, status, s),
(None, _) => format!("[{}] {}", run.trigger_type, status),
};
if let Err(e) = ctx
.store
.add_conversation_message(conv_id, "assistant", &msg)
.await
{
tracing::error!(routine = %routine.name, "Failed to persist routine message: {}", e);
}
Some(conv_id.to_string())
}
Err(e) => {
tracing::error!(routine = %routine.name, "Failed to get routine conversation: {}", e);
None
}
};
// Send notifications based on config
send_notification(
&ctx.notify_tx,
@@ -403,6 +473,7 @@ async fn execute_routine(ctx: EngineContext, routine: Routine, run: RoutineRun)
&routine.name,
status,
summary.as_deref(),
thread_id.as_deref(),
)
.await;
}
@@ -443,18 +514,13 @@ async fn execute_full_job(
reason: "scheduler not available".to_string(),
})?;
// Set the message tool's default channel/target from the routine's notify config
// so the LLM can send results without triggering cross-channel approval.
// TODO: This mutates shared global state and can race with concurrent jobs.
// Move notify config into JobContext metadata and apply per-job instead.
let mut metadata = serde_json::json!({ "max_iterations": max_iterations });
// Carry the routine's notify config in job metadata so the message tool
// can resolve channel/target per-job without global state mutation.
if let Some(channel) = &routine.notify.channel {
scheduler
.tools()
.set_message_tool_context(Some(channel.clone()), Some(routine.notify.user.clone()))
.await;
metadata["notify_channel"] = serde_json::json!(channel);
}
let metadata = serde_json::json!({ "max_iterations": max_iterations });
metadata["notify_user"] = serde_json::json!(&routine.notify.user);
// Build approval context: UnlessAutoApproved tools are auto-approved for routines;
// Always tools require explicit listing in tool_permissions.
@@ -494,7 +560,10 @@ async fn execute_full_job(
Ok((RunStatus::Ok, Some(summary), None))
}
/// Execute a lightweight routine (single LLM call).
/// Execute a lightweight routine with optional tool support.
///
/// If tools are enabled, this runs a simplified agentic loop (max 3-5 iterations).
/// If tools are disabled, this does a single LLM call (original behavior).
async fn execute_lightweight(
ctx: &EngineContext,
routine: &Routine,
@@ -526,7 +595,7 @@ async fn execute_lightweight(
Err(_) => None,
};
// Build the prompt
// Build the user-facing prompt
let mut full_prompt = String::new();
full_prompt.push_str(prompt);
@@ -554,15 +623,6 @@ async fn execute_lightweight(
}
};
let messages = if system_prompt.is_empty() {
vec![ChatMessage::user(&full_prompt)]
} else {
vec![
ChatMessage::system(&system_prompt),
ChatMessage::user(&full_prompt),
]
};
// Determine max_tokens from model metadata with fallback
let effective_max_tokens = match ctx.llm.model_metadata().await {
Ok(meta) => {
@@ -572,6 +632,45 @@ async fn execute_lightweight(
Err(_) => max_tokens,
};
// If tools are enabled, use the tool execution loop; otherwise, single LLM call
if ctx.config.lightweight_tools_enabled {
execute_lightweight_with_tools(
ctx,
routine,
&system_prompt,
&full_prompt,
effective_max_tokens,
)
.await
} else {
execute_lightweight_no_tools(
ctx,
routine,
&system_prompt,
&full_prompt,
effective_max_tokens,
)
.await
}
}
/// Execute a lightweight routine without tool support (original single-call behavior).
async fn execute_lightweight_no_tools(
ctx: &EngineContext,
_routine: &Routine,
system_prompt: &str,
full_prompt: &str,
effective_max_tokens: u32,
) -> Result<(RunStatus, Option<String>, Option<i32>), RoutineError> {
let messages = if system_prompt.is_empty() {
vec![ChatMessage::user(full_prompt)]
} else {
vec![
ChatMessage::system(system_prompt),
ChatMessage::user(full_prompt),
]
};
let request = CompletionRequest::new(messages)
.with_max_tokens(effective_max_tokens)
.with_temperature(0.3);
@@ -587,7 +686,7 @@ async fn execute_lightweight(
let content = response.content.trim();
let tokens_used = Some((response.input_tokens + response.output_tokens) as i32);
// Empty content guard (same as heartbeat)
// Empty content guard
if content.is_empty() {
return if response.finish_reason == FinishReason::Length {
Err(RoutineError::TruncatedResponse)
@@ -604,6 +703,269 @@ async fn execute_lightweight(
Ok((RunStatus::Attention, Some(content.to_string()), tokens_used))
}
/// Handle a text-only LLM response in lightweight routine execution.
///
/// Checks for the ROUTINE_OK sentinel, validates content, and returns appropriate status.
fn handle_text_response(
content: &str,
finish_reason: FinishReason,
total_input_tokens: u32,
total_output_tokens: u32,
) -> Result<(RunStatus, Option<String>, Option<i32>), RoutineError> {
let content = content.trim();
// Empty content guard
if content.is_empty() {
return if finish_reason == FinishReason::Length {
Err(RoutineError::TruncatedResponse)
} else {
Err(RoutineError::EmptyResponse)
};
}
// Check for the "nothing to do" sentinel
if content == "ROUTINE_OK" || content.contains("ROUTINE_OK") {
let total_tokens = Some((total_input_tokens + total_output_tokens) as i32);
return Ok((RunStatus::Ok, None, total_tokens));
}
let total_tokens = Some((total_input_tokens + total_output_tokens) as i32);
Ok((
RunStatus::Attention,
Some(content.to_string()),
total_tokens,
))
}
/// Execute a lightweight routine with tool execution support (agentic loop).
///
/// This is a simplified version of the full dispatcher loop:
/// - Max 3-5 iterations (configurable)
/// - Sequential tool execution (not parallel)
/// - Auto-approval of non-Always tools
/// - No hooks or approval dialogs
async fn execute_lightweight_with_tools(
ctx: &EngineContext,
routine: &Routine,
system_prompt: &str,
full_prompt: &str,
effective_max_tokens: u32,
) -> Result<(RunStatus, Option<String>, Option<i32>), RoutineError> {
let mut messages = if system_prompt.is_empty() {
vec![ChatMessage::user(full_prompt)]
} else {
vec![
ChatMessage::system(system_prompt),
ChatMessage::user(full_prompt),
]
};
let max_iterations = ctx.config.lightweight_max_iterations.min(5);
let mut iteration = 0;
let mut total_input_tokens = 0;
let mut total_output_tokens = 0;
// Create a minimal job context for tool execution with unique run ID
let run_id = Uuid::new_v4();
let job_ctx = JobContext {
job_id: run_id,
user_id: routine.user_id.clone(),
title: "Lightweight Routine".to_string(),
description: routine.name.clone(),
..Default::default()
};
loop {
iteration += 1;
// Force text-only response at iteration limit
let force_text = iteration >= max_iterations;
if force_text {
// Final iteration: no tools, just get text response
let request = CompletionRequest::new(messages)
.with_max_tokens(effective_max_tokens)
.with_temperature(0.3);
let response =
ctx.llm
.complete(request)
.await
.map_err(|e| RoutineError::LlmFailed {
reason: e.to_string(),
})?;
total_input_tokens += response.input_tokens;
total_output_tokens += response.output_tokens;
return handle_text_response(
&response.content,
response.finish_reason,
total_input_tokens,
total_output_tokens,
);
} else {
// Tool-enabled iteration
let tool_defs = ctx.tools.tool_definitions().await;
let request = ToolCompletionRequest::new(messages.clone(), tool_defs)
.with_max_tokens(effective_max_tokens)
.with_temperature(0.3);
let response = ctx.llm.complete_with_tools(request).await.map_err(|e| {
RoutineError::LlmFailed {
reason: e.to_string(),
}
})?;
total_input_tokens += response.input_tokens;
total_output_tokens += response.output_tokens;
// Check if LLM returned text (no tool calls)
if response.tool_calls.is_empty() {
let content = response.content.unwrap_or_default();
return handle_text_response(
&content,
response.finish_reason,
total_input_tokens,
total_output_tokens,
);
}
// LLM returned tool calls: add assistant message and execute tools
messages.push(ChatMessage::assistant_with_tool_calls(
response.content.clone(),
response.tool_calls.clone(),
));
// Execute tools sequentially
for tc in response.tool_calls {
let result = execute_routine_tool(ctx, &job_ctx, &tc).await;
// Sanitize and wrap result (including errors)
let result_content = match result {
Ok(output) => {
let sanitized = ctx.safety.sanitize_tool_output(&tc.name, &output);
ctx.safety.wrap_for_llm(
&tc.name,
&sanitized.content,
sanitized.was_modified,
)
}
Err(e) => {
let error_msg = format!("Tool '{}' failed: {}", tc.name, e);
let sanitized = ctx.safety.sanitize_tool_output(&tc.name, &error_msg);
ctx.safety.wrap_for_llm(
&tc.name,
&sanitized.content,
sanitized.was_modified,
)
}
};
// Add tool result to context
messages.push(ChatMessage::tool_result(&tc.id, &tc.name, &result_content));
}
// Continue loop to next LLM call
}
}
}
/// Execute a single tool for a lightweight routine.
async fn execute_routine_tool(
ctx: &EngineContext,
job_ctx: &JobContext,
tc: &ToolCall,
) -> Result<String, Box<dyn std::error::Error + Send + Sync>> {
// Check if tool exists
let tool = ctx
.tools
.get(&tc.name)
.await
.ok_or_else(|| format!("Tool '{}' not found", tc.name))?;
// Check approval requirement: only allow Never tools in lightweight routines.
// UnlessAutoApproved and Always tools are blocked to prevent prompt injection attacks.
// Lightweight routines can be triggered by external events and may process untrusted data,
// making them vulnerable to prompt injection that could trick the LLM into calling
// sensitive tools. Blocking these tools entirely is the safest approach.
match tool.requires_approval(&tc.arguments) {
ApprovalRequirement::Never => {}
ApprovalRequirement::UnlessAutoApproved | ApprovalRequirement::Always => {
return Err(format!(
"Tool '{}' requires manual approval and cannot be used in lightweight routines",
tc.name
)
.into());
}
}
// Validate tool parameters
let validation = ctx.safety.validator().validate_tool_params(&tc.arguments);
if !validation.is_valid {
let details = validation
.errors
.iter()
.map(|e| format!("{}: {}", e.field, e.message))
.collect::<Vec<_>>()
.join("; ");
return Err(format!("Invalid tool parameters: {}", details).into());
}
let safe_params = redact_params(&tc.arguments, tool.sensitive_params());
tracing::debug!(
tool = %tc.name,
params = %safe_params,
"Lightweight routine tool call started"
);
// Execute with per-tool timeout
let timeout = tool.execution_timeout();
let start = std::time::Instant::now();
let result = tokio::time::timeout(timeout, async {
tool.execute(tc.arguments.clone(), job_ctx).await
})
.await;
let elapsed = start.elapsed();
match &result {
Ok(Ok(_)) => {
tracing::debug!(
tool = %tc.name,
elapsed_ms = elapsed.as_millis() as u64,
"Lightweight routine tool call succeeded"
);
}
Ok(Err(e)) => {
tracing::debug!(
tool = %tc.name,
elapsed_ms = elapsed.as_millis() as u64,
error = %e,
"Lightweight routine tool call failed"
);
}
Err(_) => {
tracing::debug!(
tool = %tc.name,
elapsed_ms = elapsed.as_millis() as u64,
timeout_secs = timeout.as_secs(),
"Lightweight routine tool call timed out"
);
}
}
let result = result
.map_err(|_| ToolError::Timeout(timeout))
.map_err(|e| Box::new(e) as Box<dyn std::error::Error + Send + Sync>)?
.map_err(|e| Box::new(e) as Box<dyn std::error::Error + Send + Sync>)?;
// Serialize result to JSON string
let result_str =
serde_json::to_string(&result.result).unwrap_or_else(|_| "<serialize error>".to_string());
Ok(result_str)
}
/// Send a notification based on the routine's notify config and run status.
async fn send_notification(
tx: &mpsc::Sender<OutgoingResponse>,
@@ -611,6 +973,7 @@ async fn send_notification(
routine_name: &str,
status: RunStatus,
summary: Option<&str>,
thread_id: Option<&str>,
) {
let should_notify = match status {
RunStatus::Ok => notify.on_success,
@@ -637,7 +1000,7 @@ async fn send_notification(
let response = OutgoingResponse {
content: message,
thread_id: None,
thread_id: thread_id.map(String::from),
attachments: Vec::new(),
metadata: serde_json::json!({
"source": "routine",
@@ -682,6 +1045,7 @@ fn truncate(s: &str, max: usize) -> String {
#[cfg(test)]
mod tests {
use crate::agent::routine::{NotifyConfig, RunStatus};
use crate::config::RoutineConfig;
#[test]
fn test_notification_gating() {
@@ -710,4 +1074,130 @@ mod tests {
let _ = status.to_string();
}
}
#[test]
fn test_routine_config_lightweight_tools_enabled_default() {
let config = RoutineConfig::default();
assert!(
config.lightweight_tools_enabled,
"Tools should be enabled by default"
);
}
#[test]
fn test_routine_config_lightweight_max_iterations_default() {
let config = RoutineConfig::default();
assert_eq!(
config.lightweight_max_iterations, 3,
"Default should be 3 iterations"
);
}
#[test]
fn test_routine_config_can_hold_uncapped_max_iterations() {
// The `RoutineConfig` struct can hold a value greater than the safety cap.
let config = RoutineConfig {
lightweight_max_iterations: 10, // Set a value higher than the cap.
..RoutineConfig::default()
};
// The actual capping to a maximum of 5 is handled at runtime in
// `execute_lightweight_with_tools` and during config resolution from env vars.
assert_eq!(
config.lightweight_max_iterations, 10,
"Config struct should store the provided value"
);
}
#[test]
fn test_sanitize_routine_name_replaces_special_chars() {
let test_cases = vec![
("valid-routine", "valid-routine"),
("routine_with_underscore", "routine_with_underscore"),
("Routine With Spaces", "Routine_With_Spaces"),
("routine/with/slashes", "routine_with_slashes"),
("routine@with#symbols", "routine_with_symbols"),
];
for (input, expected) in test_cases {
let result = super::sanitize_routine_name(input);
assert_eq!(
result, expected,
"sanitize_routine_name({}) should be {}",
input, expected
);
}
}
#[test]
fn test_sanitize_routine_name_preserves_alphanumeric_dash_underscore() {
let names = vec!["routine123", "routine-name", "routine_name", "ROUTINE"];
for name in names {
let result = super::sanitize_routine_name(name);
assert_eq!(result, name, "Should preserve {}", name);
}
}
#[test]
fn test_routine_sentinel_detection_exact_match() {
// The execute_lightweight_no_tools checks: content == "ROUTINE_OK" || content.contains("ROUTINE_OK")
// After trim(), whitespace is removed
let test_cases = vec![
("ROUTINE_OK", true),
(" ROUTINE_OK ", true), // After trim, whitespace is removed so matches
("something ROUTINE_OK something", true),
("ROUTINE_OK is done", true),
("done ROUTINE_OK", true),
("no sentinel here", false),
];
for (content, should_match) in test_cases {
let trimmed = content.trim();
let matches = trimmed == "ROUTINE_OK" || trimmed.contains("ROUTINE_OK");
assert_eq!(
matches, should_match,
"Content '{}' sentinel detection should be {}, got {}",
content, should_match, matches
);
}
}
#[test]
fn test_approval_requirement_pattern_matching() {
// Test the approval requirement logic (Never, UnlessAutoApproved, Always)
use crate::tools::ApprovalRequirement;
let requirements = vec![
(ApprovalRequirement::Never, "auto-approved"),
(ApprovalRequirement::UnlessAutoApproved, "auto-approved"),
(ApprovalRequirement::Always, "blocks"),
];
for (req, expected) in requirements {
let can_auto_approve = matches!(
req,
ApprovalRequirement::Never | ApprovalRequirement::UnlessAutoApproved
);
let label = if can_auto_approve {
"auto-approved"
} else {
"blocks"
};
assert_eq!(label, expected, "Approval pattern should match");
}
}
#[test]
fn test_empty_response_handling() {
// Simulate the empty content guard logic
let empty_content = "";
let finish_reason_length = crate::llm::FinishReason::Length;
let finish_reason_stop = crate::llm::FinishReason::Stop;
assert!(
empty_content.trim().is_empty(),
"Should detect empty content"
);
assert_eq!(finish_reason_length, crate::llm::FinishReason::Length);
assert_eq!(finish_reason_stop, crate::llm::FinishReason::Stop);
}
}
+16
View File
@@ -160,6 +160,13 @@ impl Scheduler {
.create_job_for_user(user_id, title, description)
.await?;
// Apply token budget from config, allowing per-job metadata override.
let max_tokens = metadata
.as_ref()
.and_then(|m| m.get("max_tokens"))
.and_then(|v| v.as_u64())
.unwrap_or(self.config.max_tokens_per_job);
// Apply metadata if provided
if let Some(meta) = metadata {
self.context_manager
@@ -169,6 +176,15 @@ impl Scheduler {
.await?;
}
// Set token budget (separate update to avoid overwriting metadata)
if max_tokens > 0 {
self.context_manager
.update_context(job_id, |ctx| {
ctx.max_tokens = max_tokens;
})
.await?;
}
// Persist to DB before scheduling so the worker's FK references are valid
if let Some(ref store) = self.store {
let ctx = self.context_manager.get_context(job_id).await?;
+324 -11
View File
@@ -16,6 +16,7 @@ use chrono::{DateTime, Utc};
use serde::{Deserialize, Serialize};
use uuid::Uuid;
use crate::channels::web::util::truncate_preview;
use crate::llm::{ChatMessage, ToolCall};
/// A session containing one or more threads.
@@ -164,6 +165,10 @@ pub struct PendingApproval {
/// executed yet when approval was requested.
#[serde(default)]
pub deferred_tool_calls: Vec<ToolCall>,
/// User timezone at the time the approval was requested, so it persists
/// through the approval flow even if the approval message lacks timezone.
#[serde(default)]
pub user_timezone: Option<String>,
}
/// A conversation thread within a session.
@@ -316,7 +321,13 @@ impl Thread {
}
}
/// Get all messages for context building.
/// Get all messages for context building, including tool call history.
///
/// Emits the full LLM-compatible message sequence per turn:
/// `user → [assistant_with_tool_calls → tool_result*] → assistant`
///
/// This ensures the LLM sees prior tool executions and won't re-attempt
/// completed actions in subsequent turns.
pub fn messages(&self) -> Vec<ChatMessage> {
let mut messages = Vec::new();
for turn in &self.turns {
@@ -328,6 +339,42 @@ impl Thread {
turn.image_content_parts.clone(),
));
}
if !turn.tool_calls.is_empty() {
// Build ToolCall objects with synthetic stable IDs
let tool_calls: Vec<ToolCall> = turn
.tool_calls
.iter()
.enumerate()
.map(|(i, tc)| ToolCall {
id: format!("turn{}_{}", turn.turn_number, i),
name: tc.name.clone(),
arguments: tc.parameters.clone(),
})
.collect();
// Assistant message declaring the tool calls (no text content)
messages.push(ChatMessage::assistant_with_tool_calls(None, tool_calls));
// Individual tool result messages, truncated to limit context size.
for (i, tc) in turn.tool_calls.iter().enumerate() {
let call_id = format!("turn{}_{}", turn.turn_number, i);
let content = if let Some(ref err) = tc.error {
// .error already contains the full error text;
// pass through without wrapping to avoid double-prefix.
truncate_preview(err, 1000)
} else if let Some(ref res) = tc.result {
let raw = match res {
serde_json::Value::String(s) => s.clone(),
other => other.to_string(),
};
truncate_preview(&raw, 1000)
} else {
"OK".to_string()
};
messages.push(ChatMessage::tool_result(call_id, &tc.name, content));
}
}
if let Some(ref response) = turn.response {
messages.push(ChatMessage::assistant(response));
}
@@ -349,13 +396,16 @@ impl Thread {
/// Restore thread state from a checkpoint's messages.
///
/// Clears existing turns and rebuilds from message pairs.
/// Messages should alternate: user, assistant, user, assistant...
/// Clears existing turns and rebuilds from the message sequence.
/// Handles the full message pattern including tool messages:
/// `user → [assistant_with_tool_calls → tool_result*] → assistant`
///
/// Also supports the legacy pattern (user/assistant pairs only) for
/// backward compatibility with old checkpoint data.
pub fn restore_from_messages(&mut self, messages: Vec<ChatMessage>) {
self.turns.clear();
self.state = ThreadState::Idle;
// Messages alternate: user, assistant, user, assistant...
let mut iter = messages.into_iter().peekable();
let mut turn_number = 0;
@@ -363,18 +413,58 @@ impl Thread {
if msg.role == crate::llm::Role::User {
let mut turn = Turn::new(turn_number, &msg.content);
// Check if next is assistant response
if let Some(next) = iter.peek()
&& next.role == crate::llm::Role::Assistant
{
// iter.next() is guaranteed Some after a successful peek()
if let Some(response) = iter.next() {
turn.complete(&response.content);
// Consume tool call sequences (assistant_with_tool_calls + tool_results).
// A single turn may contain multiple rounds of tool calls, so we
// track the cumulative base index into turn.tool_calls.
while let Some(next) = iter.peek() {
if next.role == crate::llm::Role::Assistant && next.tool_calls.is_some() {
let call_base_idx = turn.tool_calls.len();
if let Some(assistant_msg) = iter.next()
&& let Some(ref tcs) = assistant_msg.tool_calls
{
for tc in tcs {
turn.record_tool_call(&tc.name, tc.arguments.clone());
}
}
// Consume the corresponding tool_result messages,
// indexing relative to this batch's base offset.
let mut pos = 0;
while let Some(tr) = iter.peek() {
if tr.role != crate::llm::Role::Tool {
break;
}
if let Some(tool_msg) = iter.next() {
let idx = call_base_idx + pos;
if idx < turn.tool_calls.len() {
// Store as result — the error/success distinction
// is for the live turn only; restored context just
// needs the content the LLM originally saw.
turn.tool_calls[idx].result =
Some(serde_json::Value::String(tool_msg.content.clone()));
}
}
pos += 1;
}
} else {
break;
}
}
// Check if next is the final assistant response for this turn
let is_final_assistant = iter.peek().is_some_and(|n| {
n.role == crate::llm::Role::Assistant && n.tool_calls.is_none()
});
if is_final_assistant && let Some(response) = iter.next() {
turn.complete(&response.content);
}
self.turns.push(turn);
turn_number += 1;
} else {
// Skip non-user messages that aren't anchored to a turn
continue;
}
}
@@ -976,6 +1066,7 @@ mod tests {
tool_call_id: "call_123".to_string(),
context_messages: vec![ChatMessage::user("do it")],
deferred_tool_calls: vec![],
user_timezone: None,
};
thread.await_approval(approval);
@@ -1001,6 +1092,7 @@ mod tests {
tool_call_id: "call_456".to_string(),
context_messages: vec![],
deferred_tool_calls: vec![],
user_timezone: None,
};
thread.await_approval(approval);
@@ -1029,4 +1121,225 @@ mod tests {
ThreadState::Processing
);
}
// Regression tests for #568: tool call history must survive hydration.
#[test]
fn test_messages_includes_tool_calls() {
let mut thread = Thread::new(Uuid::new_v4());
thread.start_turn("Search for X");
{
let turn = thread.turns.last_mut().unwrap();
turn.record_tool_call("memory_search", serde_json::json!({"query": "X"}));
turn.record_tool_result(serde_json::json!("Found X in doc.md"));
}
thread.complete_turn("I found X in doc.md.");
let messages = thread.messages();
// user + assistant_with_tool_calls + tool_result + assistant = 4
assert_eq!(messages.len(), 4);
assert_eq!(messages[0].role, crate::llm::Role::User);
assert_eq!(messages[0].content, "Search for X");
assert_eq!(messages[1].role, crate::llm::Role::Assistant);
assert!(messages[1].tool_calls.is_some());
let tcs = messages[1].tool_calls.as_ref().unwrap();
assert_eq!(tcs.len(), 1);
assert_eq!(tcs[0].name, "memory_search");
assert_eq!(messages[2].role, crate::llm::Role::Tool);
assert!(messages[2].content.contains("Found X"));
assert_eq!(messages[3].role, crate::llm::Role::Assistant);
assert_eq!(messages[3].content, "I found X in doc.md.");
}
#[test]
fn test_messages_multiple_tool_calls_per_turn() {
let mut thread = Thread::new(Uuid::new_v4());
thread.start_turn("Do two things");
{
let turn = thread.turns.last_mut().unwrap();
turn.record_tool_call("echo", serde_json::json!({"msg": "a"}));
turn.record_tool_result(serde_json::json!("a"));
turn.record_tool_call("time", serde_json::json!({}));
turn.record_tool_error("timeout");
}
thread.complete_turn("Done.");
let messages = thread.messages();
// user + assistant_with_calls(2) + tool_result + tool_result + assistant = 5
assert_eq!(messages.len(), 5);
let tcs = messages[1].tool_calls.as_ref().unwrap();
assert_eq!(tcs.len(), 2);
// First tool: success
assert_eq!(messages[2].content, "a");
// Second tool: error (passed through directly, no wrapping)
assert!(messages[3].content.contains("timeout"));
}
#[test]
fn test_restore_from_messages_with_tool_calls() {
let mut thread = Thread::new(Uuid::new_v4());
// Build a message sequence with tool calls
let tc = ToolCall {
id: "call_0".to_string(),
name: "search".to_string(),
arguments: serde_json::json!({"q": "test"}),
};
let messages = vec![
ChatMessage::user("Find test"),
ChatMessage::assistant_with_tool_calls(None, vec![tc]),
ChatMessage::tool_result("call_0", "search", "result: found"),
ChatMessage::assistant("Found it."),
];
thread.restore_from_messages(messages);
assert_eq!(thread.turns.len(), 1);
let turn = &thread.turns[0];
assert_eq!(turn.user_input, "Find test");
assert_eq!(turn.tool_calls.len(), 1);
assert_eq!(turn.tool_calls[0].name, "search");
assert_eq!(
turn.tool_calls[0].result,
Some(serde_json::Value::String("result: found".to_string()))
);
assert_eq!(turn.response, Some("Found it.".to_string()));
}
#[test]
fn test_restore_from_messages_with_tool_error() {
let mut thread = Thread::new(Uuid::new_v4());
let tc = ToolCall {
id: "call_0".to_string(),
name: "http".to_string(),
arguments: serde_json::json!({}),
};
let messages = vec![
ChatMessage::user("Fetch URL"),
ChatMessage::assistant_with_tool_calls(None, vec![tc]),
ChatMessage::tool_result("call_0", "http", "Error: timeout"),
ChatMessage::assistant("The request timed out."),
];
thread.restore_from_messages(messages);
// restore_from_messages stores all tool content as result (not error),
// because it can't reliably distinguish errors from results that happen
// to start with "Error: ". The content is preserved for LLM context.
let turn = &thread.turns[0];
assert_eq!(
turn.tool_calls[0].result,
Some(serde_json::Value::String("Error: timeout".to_string()))
);
}
#[test]
fn test_messages_round_trip_with_tools() {
// Build a thread with tool calls, get messages(), restore, get messages() again
// The two message sequences should be equivalent.
let mut thread = Thread::new(Uuid::new_v4());
thread.start_turn("Do search");
{
let turn = thread.turns.last_mut().unwrap();
turn.record_tool_call("search", serde_json::json!({"q": "test"}));
turn.record_tool_result(serde_json::json!("found"));
}
thread.complete_turn("Here are results.");
let messages_original = thread.messages();
// Restore into a new thread
let mut thread2 = Thread::new(Uuid::new_v4());
thread2.restore_from_messages(messages_original.clone());
let messages_restored = thread2.messages();
// Same number of messages
assert_eq!(messages_original.len(), messages_restored.len());
// Same roles
for (orig, rest) in messages_original.iter().zip(messages_restored.iter()) {
assert_eq!(orig.role, rest.role);
}
// Same final response
assert_eq!(
messages_original.last().unwrap().content,
messages_restored.last().unwrap().content
);
}
#[test]
fn test_restore_multi_stage_tool_calls() {
let mut thread = Thread::new(Uuid::new_v4());
let tc1 = ToolCall {
id: "call_a".to_string(),
name: "search".to_string(),
arguments: serde_json::json!({"q": "data"}),
};
let tc2 = ToolCall {
id: "call_b".to_string(),
name: "write".to_string(),
arguments: serde_json::json!({"path": "out.txt"}),
};
let messages = vec![
ChatMessage::user("Find and save"),
ChatMessage::assistant_with_tool_calls(None, vec![tc1]),
ChatMessage::tool_result("call_a", "search", "found data"),
ChatMessage::assistant_with_tool_calls(None, vec![tc2]),
ChatMessage::tool_result("call_b", "write", "written"),
ChatMessage::assistant("Done, saved to out.txt"),
];
thread.restore_from_messages(messages);
assert_eq!(thread.turns.len(), 1);
let turn = &thread.turns[0];
assert_eq!(turn.tool_calls.len(), 2);
assert_eq!(turn.tool_calls[0].name, "search");
assert_eq!(turn.tool_calls[1].name, "write");
assert_eq!(
turn.tool_calls[0].result,
Some(serde_json::Value::String("found data".to_string()))
);
assert_eq!(
turn.tool_calls[1].result,
Some(serde_json::Value::String("written".to_string()))
);
assert_eq!(turn.response, Some("Done, saved to out.txt".to_string()));
}
#[test]
fn test_messages_truncates_large_tool_results() {
let mut thread = Thread::new(Uuid::new_v4());
thread.start_turn("Read big file");
{
let turn = thread.turns.last_mut().unwrap();
turn.record_tool_call("read_file", serde_json::json!({"path": "big.txt"}));
let big_result = "x".repeat(2000);
turn.record_tool_result(serde_json::json!(big_result));
}
thread.complete_turn("Here's the file content.");
let messages = thread.messages();
let tool_result_content = &messages[2].content;
assert!(
tool_result_content.len() <= 1010,
"Tool result should be truncated, got {} chars",
tool_result_content.len()
);
assert!(tool_result_content.ends_with("..."));
}
}
+305 -60
View File
@@ -20,7 +20,7 @@ use crate::channels::web::util::truncate_preview;
use crate::channels::{IncomingMessage, StatusUpdate};
use crate::context::JobContext;
use crate::error::Error;
use crate::llm::ChatMessage;
use crate::llm::{ChatMessage, ToolCall};
use crate::tools::redact_params;
impl Agent {
@@ -66,16 +66,7 @@ impl Agent {
.await
.unwrap_or_default();
msg_count = db_messages.len();
chat_messages = db_messages
.iter()
.filter_map(|m| match m.role.as_str() {
"user" => Some(ChatMessage::user(&m.content)),
"assistant" => Some(ChatMessage::assistant(&m.content)),
// tool_calls rows are UI metadata (tool name + preview),
// not part of the LLM conversation context.
_ => None,
})
.collect();
chat_messages = rebuild_chat_messages_from_db(&db_messages);
} else {
msg_count = 0;
}
@@ -230,7 +221,7 @@ impl Agent {
)
.await;
let compactor = ContextCompactor::new(self.llm().clone(), self.safety().clone());
let compactor = ContextCompactor::new(self.llm().clone());
if let Err(e) = compactor
.compact(thread, strategy, self.workspace().map(|w| w.as_ref()))
.await
@@ -340,10 +331,10 @@ impl Agent {
};
thread.complete_turn(&response);
let tool_calls = thread
let (turn_number, tool_calls) = thread
.turns
.last()
.map(|t| t.tool_calls.clone())
.map(|t| (t.turn_number, t.tool_calls.clone()))
.unwrap_or_default();
let _ = self
.channels
@@ -355,7 +346,7 @@ impl Agent {
.await;
// Persist tool calls then assistant response (user message already persisted at turn start)
self.persist_tool_calls(thread_id, &message.user_id, &tool_calls)
self.persist_tool_calls(thread_id, &message.user_id, turn_number, &tool_calls)
.await;
self.persist_assistant_response(thread_id, &message.user_id, &response)
.await;
@@ -464,6 +455,7 @@ impl Agent {
&self,
thread_id: Uuid,
user_id: &str,
turn_number: usize,
tool_calls: &[crate::agent::session::TurnToolCall],
) {
if tool_calls.is_empty() {
@@ -477,14 +469,24 @@ impl Agent {
let summaries: Vec<serde_json::Value> = tool_calls
.iter()
.map(|tc| {
let mut obj = serde_json::json!({ "name": tc.name });
.enumerate()
.map(|(i, tc)| {
let mut obj = serde_json::json!({
"name": tc.name,
"call_id": format!("turn{}_{}", turn_number, i),
});
if let Some(ref result) = tc.result {
let preview = match result {
serde_json::Value::String(s) => truncate_preview(s, 500),
other => truncate_preview(&other.to_string(), 500),
};
obj["result_preview"] = serde_json::Value::String(preview);
// Store full result (truncated to ~1000 chars) for LLM context rebuild
let full_result = match result {
serde_json::Value::String(s) => truncate_preview(s, 1000),
other => truncate_preview(&other.to_string(), 1000),
};
obj["result"] = serde_json::Value::String(full_result);
}
if let Some(ref error) = tc.error {
obj["error"] = serde_json::Value::String(truncate_preview(error, 200));
@@ -627,7 +629,7 @@ impl Agent {
crate::agent::context_monitor::CompactionStrategy::Summarize { keep_recent: 5 },
);
let compactor = ContextCompactor::new(self.llm().clone(), self.safety().clone());
let compactor = ContextCompactor::new(self.llm().clone());
match compactor
.compact(thread, strategy, self.workspace().map(|w| w.as_ref()))
.await
@@ -746,6 +748,16 @@ impl Agent {
let mut job_ctx =
JobContext::with_user(&message.user_id, "chat", "Interactive chat session");
job_ctx.http_interceptor = self.deps.http_interceptor.clone();
// Prefer a valid timezone from the approval message, fall back to the
// resolved timezone stored when the approval was originally requested.
let tz_candidate = message
.timezone
.as_deref()
.filter(|tz| crate::timezone::parse_timezone(tz).is_some())
.or(pending.user_timezone.as_deref());
if let Some(tz) = tz_candidate {
job_ctx.user_timezone = tz.to_string();
}
let _ = self
.channels
@@ -797,19 +809,33 @@ impl Agent {
let mut context_messages = pending.context_messages;
let deferred_tool_calls = pending.deferred_tool_calls;
// Record result in thread
// Sanitize tool result, then record the cleaned version in the
// thread. Must happen before auth intercept check which may return early.
let is_tool_error = tool_result.is_err();
let result_content = match &tool_result {
Ok(output) => {
let sanitized = self
.safety()
.sanitize_tool_output(&pending.tool_name, output);
self.safety().wrap_for_llm(
&pending.tool_name,
&sanitized.content,
sanitized.was_modified,
)
}
Err(e) => format!("Error: {}", e),
};
// Record sanitized result in thread
{
let mut sess = session.lock().await;
if let Some(thread) = sess.threads.get_mut(&thread_id)
&& let Some(turn) = thread.last_turn_mut()
{
match &tool_result {
Ok(output) => {
turn.record_tool_result(serde_json::json!(output));
}
Err(e) => {
turn.record_tool_error(e.to_string());
}
if is_tool_error {
turn.record_tool_error(result_content.clone());
} else {
turn.record_tool_result(serde_json::json!(result_content));
}
}
}
@@ -831,21 +857,6 @@ impl Agent {
return Ok(SubmissionResult::response(instructions));
}
// Add tool result to context
let result_content = match tool_result {
Ok(output) => {
let sanitized = self
.safety()
.sanitize_tool_output(&pending.tool_name, &output);
self.safety().wrap_for_llm(
&pending.tool_name,
&sanitized.content,
sanitized.was_modified,
)
}
Err(e) => format!("Error: {}", e),
};
context_messages.push(ChatMessage::tool_result(
&pending.tool_call_id,
&pending.tool_name,
@@ -1050,15 +1061,31 @@ impl Agent {
.await;
}
// Record in thread
// Sanitize first, then record the cleaned version in thread.
// Must happen before auth detection which may set deferred_auth.
let is_deferred_error = deferred_result.is_err();
let deferred_content = match &deferred_result {
Ok(output) => {
let sanitized = self.safety().sanitize_tool_output(&tc.name, output);
self.safety().wrap_for_llm(
&tc.name,
&sanitized.content,
sanitized.was_modified,
)
}
Err(e) => format!("Error: {}", e),
};
// Record sanitized result in thread
{
let mut sess = session.lock().await;
if let Some(thread) = sess.threads.get_mut(&thread_id)
&& let Some(turn) = thread.last_turn_mut()
{
match &deferred_result {
Ok(output) => turn.record_tool_result(serde_json::json!(output)),
Err(e) => turn.record_tool_error(e.to_string()),
if is_deferred_error {
turn.record_tool_error(deferred_content.clone());
} else {
turn.record_tool_result(serde_json::json!(deferred_content));
}
}
}
@@ -1080,18 +1107,6 @@ impl Agent {
deferred_auth = Some(instructions);
}
let deferred_content = match deferred_result {
Ok(output) => {
let sanitized = self.safety().sanitize_tool_output(&tc.name, &output);
self.safety().wrap_for_llm(
&tc.name,
&sanitized.content,
sanitized.was_modified,
)
}
Err(e) => format!("Error: {}", e),
};
context_messages.push(ChatMessage::tool_result(&tc.id, &tc.name, deferred_content));
}
@@ -1111,6 +1126,8 @@ impl Agent {
tool_call_id: tc.id.clone(),
context_messages: context_messages.clone(),
deferred_tool_calls: deferred_tool_calls[approval_idx + 1..].to_vec(),
// Carry forward the resolved timezone from the original pending approval
user_timezone: pending.user_timezone.clone(),
};
let request_id = new_pending.request_id;
@@ -1157,13 +1174,13 @@ impl Agent {
match result {
Ok(AgenticLoopResult::Response(response)) => {
thread.complete_turn(&response);
let tool_calls = thread
let (turn_number, tool_calls) = thread
.turns
.last()
.map(|t| t.tool_calls.clone())
.map(|t| (t.turn_number, t.tool_calls.clone()))
.unwrap_or_default();
// User message already persisted at turn start; save tool calls then assistant response
self.persist_tool_calls(thread_id, &message.user_id, &tool_calls)
self.persist_tool_calls(thread_id, &message.user_id, turn_number, &tool_calls)
.await;
self.persist_assistant_response(thread_id, &message.user_id, &response)
.await;
@@ -1478,3 +1495,231 @@ impl Agent {
}
}
}
/// Rebuild full LLM-compatible `ChatMessage` sequence from DB messages.
///
/// Parses `role="tool_calls"` rows to reconstruct `assistant_with_tool_calls`
/// and `tool_result` messages so that the LLM sees the complete tool execution
/// history on thread hydration. Falls back gracefully for legacy rows that
/// lack the enriched fields (`call_id`, `parameters`, `result`).
fn rebuild_chat_messages_from_db(
db_messages: &[crate::history::ConversationMessage],
) -> Vec<ChatMessage> {
let mut result = Vec::new();
for msg in db_messages {
match msg.role.as_str() {
"user" => result.push(ChatMessage::user(&msg.content)),
"assistant" => result.push(ChatMessage::assistant(&msg.content)),
"tool_calls" => {
// Try to parse the enriched JSON and rebuild tool messages.
if let Ok(calls) = serde_json::from_str::<Vec<serde_json::Value>>(&msg.content) {
if calls.is_empty() {
continue;
}
// Check if this is an enriched row (has call_id) or legacy
let has_call_id = calls
.first()
.and_then(|c| c.get("call_id"))
.and_then(|v| v.as_str())
.is_some();
if has_call_id {
// Build assistant_with_tool_calls + tool_result messages
let tool_calls: Vec<ToolCall> = calls
.iter()
.map(|c| ToolCall {
id: c["call_id"].as_str().unwrap_or("call_0").to_string(),
name: c["name"].as_str().unwrap_or("unknown").to_string(),
arguments: c
.get("parameters")
.cloned()
.unwrap_or(serde_json::json!({})),
})
.collect();
// The assistant text for tool_calls is always None here;
// the final assistant response comes as a separate
// "assistant" row after this tool_calls row.
result.push(ChatMessage::assistant_with_tool_calls(None, tool_calls));
// Emit tool_result messages for each call
for c in &calls {
let call_id = c["call_id"].as_str().unwrap_or("call_0").to_string();
let name = c["name"].as_str().unwrap_or("unknown").to_string();
let content = if let Some(err) = c.get("error").and_then(|v| v.as_str())
{
format!("Error: {}", err)
} else if let Some(res) = c.get("result").and_then(|v| v.as_str()) {
res.to_string()
} else if let Some(preview) =
c.get("result_preview").and_then(|v| v.as_str())
{
preview.to_string()
} else {
"OK".to_string()
};
result.push(ChatMessage::tool_result(call_id, name, content));
}
}
// Legacy rows without call_id: skip (will appear as
// simple user/assistant pairs, same as before this fix).
}
}
_ => {} // Skip unknown roles
}
}
result
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_rebuild_chat_messages_user_assistant_only() {
let messages = vec![
make_db_msg("user", "Hello"),
make_db_msg("assistant", "Hi there!"),
];
let result = rebuild_chat_messages_from_db(&messages);
assert_eq!(result.len(), 2);
assert_eq!(result[0].role, crate::llm::Role::User);
assert_eq!(result[1].role, crate::llm::Role::Assistant);
}
#[test]
fn test_rebuild_chat_messages_with_enriched_tool_calls() {
let tool_json = serde_json::json!([
{
"name": "memory_search",
"call_id": "call_0",
"parameters": {"query": "test"},
"result": "Found 3 results",
"result_preview": "Found 3 re..."
},
{
"name": "echo",
"call_id": "call_1",
"parameters": {"message": "hi"},
"error": "timeout"
}
]);
let messages = vec![
make_db_msg("user", "Search for test"),
make_db_msg("tool_calls", &tool_json.to_string()),
make_db_msg("assistant", "I found some results."),
];
let result = rebuild_chat_messages_from_db(&messages);
// user + assistant_with_tool_calls + tool_result*2 + assistant
assert_eq!(result.len(), 5);
// user
assert_eq!(result[0].role, crate::llm::Role::User);
// assistant with tool_calls
assert_eq!(result[1].role, crate::llm::Role::Assistant);
assert!(result[1].tool_calls.is_some());
let tcs = result[1].tool_calls.as_ref().unwrap();
assert_eq!(tcs.len(), 2);
assert_eq!(tcs[0].name, "memory_search");
assert_eq!(tcs[0].id, "call_0");
assert_eq!(tcs[1].name, "echo");
// tool results
assert_eq!(result[2].role, crate::llm::Role::Tool);
assert_eq!(result[2].tool_call_id, Some("call_0".to_string()));
assert!(result[2].content.contains("Found 3 results"));
assert_eq!(result[3].role, crate::llm::Role::Tool);
assert_eq!(result[3].tool_call_id, Some("call_1".to_string()));
assert!(result[3].content.contains("Error: timeout"));
// final assistant
assert_eq!(result[4].role, crate::llm::Role::Assistant);
assert_eq!(result[4].content, "I found some results.");
}
#[test]
fn test_rebuild_chat_messages_legacy_tool_calls_skipped() {
// Legacy format: no call_id field
let tool_json = serde_json::json!([
{"name": "echo", "result_preview": "hello"}
]);
let messages = vec![
make_db_msg("user", "Hi"),
make_db_msg("tool_calls", &tool_json.to_string()),
make_db_msg("assistant", "Done"),
];
let result = rebuild_chat_messages_from_db(&messages);
// Legacy rows are skipped, only user + assistant
assert_eq!(result.len(), 2);
assert_eq!(result[0].role, crate::llm::Role::User);
assert_eq!(result[1].role, crate::llm::Role::Assistant);
}
#[test]
fn test_rebuild_chat_messages_empty() {
let result = rebuild_chat_messages_from_db(&[]);
assert!(result.is_empty());
}
#[test]
fn test_rebuild_chat_messages_malformed_tool_calls_json() {
let messages = vec![
make_db_msg("user", "Hi"),
make_db_msg("tool_calls", "not valid json"),
make_db_msg("assistant", "Done"),
];
let result = rebuild_chat_messages_from_db(&messages);
// Malformed JSON is silently skipped
assert_eq!(result.len(), 2);
}
#[test]
fn test_rebuild_chat_messages_multi_turn_with_tools() {
let tool_json_1 = serde_json::json!([
{"name": "search", "call_id": "call_0", "parameters": {}, "result": "found it"}
]);
let tool_json_2 = serde_json::json!([
{"name": "write", "call_id": "call_0", "parameters": {"path": "a.txt"}, "result": "ok"}
]);
let messages = vec![
make_db_msg("user", "Find X"),
make_db_msg("tool_calls", &tool_json_1.to_string()),
make_db_msg("assistant", "Found X"),
make_db_msg("user", "Write it"),
make_db_msg("tool_calls", &tool_json_2.to_string()),
make_db_msg("assistant", "Written"),
];
let result = rebuild_chat_messages_from_db(&messages);
// Turn 1: user + assistant_with_calls + tool_result + assistant = 4
// Turn 2: user + assistant_with_calls + tool_result + assistant = 4
assert_eq!(result.len(), 8);
// Verify turn boundaries
assert_eq!(result[0].content, "Find X");
assert!(result[1].tool_calls.is_some());
assert_eq!(result[2].role, crate::llm::Role::Tool);
assert_eq!(result[3].content, "Found X");
assert_eq!(result[4].content, "Write it");
assert!(result[5].tool_calls.is_some());
assert_eq!(result[6].role, crate::llm::Role::Tool);
assert_eq!(result[7].content, "Written");
}
fn make_db_msg(role: &str, content: &str) -> crate::history::ConversationMessage {
crate::history::ConversationMessage {
id: uuid::Uuid::new_v4(),
role: role.to_string(),
content: content.to_string(),
created_at: chrono::Utc::now(),
}
}
}
+165 -37
View File
@@ -15,7 +15,8 @@ use crate::db::Database;
use crate::error::Error;
use crate::hooks::HookRegistry;
use crate::llm::{
ActionPlan, ChatMessage, LlmProvider, Reasoning, ReasoningContext, RespondResult, ToolSelection,
ActionPlan, ChatMessage, LlmProvider, Reasoning, ReasoningContext, RespondResult, ToolCall,
ToolSelection,
};
use crate::safety::SafetyLayer;
use crate::tools::rate_limiter::RateLimitResult;
@@ -211,7 +212,7 @@ impl Worker {
let job_ctx = self.context_manager().get_context(self.job_id).await?;
// Create reasoning engine
let reasoning = Reasoning::new(self.llm().clone(), self.safety().clone());
let reasoning = Reasoning::new(self.llm().clone());
// Build initial reasoning context (tool definitions refreshed each iteration in execution_loop)
let mut reason_ctx = ReasoningContext::new().with_job(&job_ctx.description);
@@ -416,7 +417,8 @@ Report when the job is complete or if you encounter issues you cannot resolve."#
iteration += 1;
if iteration > max_iterations {
self.mark_stuck("Maximum iterations exceeded").await?;
self.mark_failed("Maximum iterations exceeded: job hit the iteration cap")
.await?;
return Ok(());
}
@@ -436,7 +438,8 @@ Report when the job is complete or if you encounter issues you cannot resolve."#
"LLM rate limited during tool selection, backing off"
);
if consecutive_rate_limits >= MAX_CONSECUTIVE_RATE_LIMITS {
self.mark_stuck("Persistent rate limiting").await?;
self.mark_failed("Persistent rate limiting: exceeded retry limit")
.await?;
return Ok(());
}
self.log_event(
@@ -466,7 +469,8 @@ Report when the job is complete or if you encounter issues you cannot resolve."#
"LLM rate limited during respond_with_tools, backing off"
);
if consecutive_rate_limits >= MAX_CONSECUTIVE_RATE_LIMITS {
self.mark_stuck("Persistent rate limiting").await?;
self.mark_failed("Persistent rate limiting: exceeded retry limit")
.await?;
return Ok(());
}
self.log_event(
@@ -482,6 +486,20 @@ Report when the job is complete or if you encounter issues you cannot resolve."#
Err(e) => return Err(e.into()),
};
// Track token usage from LLM call against the job budget.
// NOTE: select_tools() also makes LLM calls but doesn't expose
// TokenUsage; only respond_with_tools() usage is tracked here.
let total_tokens = respond_output.usage.total() as u64;
if total_tokens > 0
&& let Err(msg) = self
.context_manager()
.update_context(self.job_id, |ctx| ctx.add_tokens(total_tokens))
.await?
{
self.mark_failed(&msg).await?;
return Ok(());
}
match respond_output.result {
RespondResult::Text(response) => {
// Check for explicit completion phrases. Use word-boundary
@@ -576,37 +594,54 @@ Report when the job is complete or if you encounter issues you cannot resolve."#
}
}
}
} else if selections.len() == 1 {
consecutive_tool_intent_nudges = 0;
// Single tool: execute directly
let selection = &selections[0];
tracing::debug!(
"Job {} selecting tool: {} - {}",
self.job_id,
selection.tool_name,
selection.reasoning
);
let result = self
.execute_tool(&selection.tool_name, &selection.parameters)
.await;
self.process_tool_result(reason_ctx, selection, result)
.await?;
} else {
// Multiple tools: execute in parallel
tracing::debug!(
"Job {} executing {} tools in parallel",
self.job_id,
selections.len()
);
consecutive_tool_intent_nudges = 0;
let results = self.execute_tools_parallel(&selections).await;
// Record the assistant tool_calls message so that tool_result
// messages have a matching parent (prevents orphaned rewrites).
let tool_calls: Vec<ToolCall> = selections
.iter()
.map(|s| ToolCall {
id: s.tool_call_id.clone(),
name: s.tool_name.clone(),
arguments: s.parameters.clone(),
})
.collect();
reason_ctx
.messages
.push(ChatMessage::assistant_with_tool_calls(None, tool_calls));
// Process all results
for (selection, result) in selections.iter().zip(results) {
self.process_tool_result(reason_ctx, selection, result.result)
if selections.len() == 1 {
// Single tool: execute directly
let selection = &selections[0];
tracing::debug!(
"Job {} selecting tool: {} - {}",
self.job_id,
selection.tool_name,
selection.reasoning
);
let result = self
.execute_tool(&selection.tool_name, &selection.parameters)
.await;
self.process_tool_result(reason_ctx, selection, result)
.await?;
} else {
// Multiple tools: execute in parallel
tracing::debug!(
"Job {} executing {} tools in parallel",
self.job_id,
selections.len()
);
let results = self.execute_tools_parallel(&selections).await;
// Process all results
for (selection, result) in selections.iter().zip(results) {
self.process_tool_result(reason_ctx, selection, result.result)
.await?;
}
}
}
@@ -1087,11 +1122,6 @@ Report when the job is complete or if you encounter issues you cannot resolve."#
action.reasoning
);
// Execute the planned tool
let result = self
.execute_tool(&action.tool_name, &action.parameters)
.await;
// Create a synthetic ToolSelection for process_tool_result.
// Plan actions don't originate from an LLM tool_call response so
// there is no real tool_call_id; generate a unique one.
@@ -1103,6 +1133,24 @@ Report when the job is complete or if you encounter issues you cannot resolve."#
tool_call_id: format!("plan_{}_{}", self.job_id, i),
};
// Record the assistant tool_calls message so that the tool_result
// has a matching parent (prevents orphaned rewrites).
reason_ctx
.messages
.push(ChatMessage::assistant_with_tool_calls(
None,
vec![ToolCall {
id: selection.tool_call_id.clone(),
name: selection.tool_name.clone(),
arguments: selection.parameters.clone(),
}],
));
// Execute the planned tool
let result = self
.execute_tool(&action.tool_name, &action.parameters)
.await;
// Process the result
let completed = self
.process_tool_result(reason_ctx, &selection, result)
@@ -1731,4 +1779,84 @@ mod tests {
"Always tool should be allowed with permission"
);
}
#[tokio::test]
async fn test_token_budget_exceeded_fails_job() {
let worker = make_worker(vec![]).await;
// Transition to InProgress (required for mark_failed)
worker
.context_manager()
.update_context(worker.job_id, |ctx| {
ctx.transition_to(JobState::InProgress, None)
})
.await
.unwrap()
.unwrap();
// Set a token budget
worker
.context_manager()
.update_context(worker.job_id, |ctx| {
ctx.max_tokens = 100;
})
.await
.unwrap();
// Simulate adding tokens that exceed the budget
let budget_result = worker
.context_manager()
.update_context(worker.job_id, |ctx| ctx.add_tokens(200))
.await
.unwrap();
assert!(
budget_result.is_err(),
"Should return error when token budget exceeded"
);
// Verify that mark_failed transitions job to Failed
worker
.mark_failed(&budget_result.unwrap_err())
.await
.unwrap();
let ctx = worker
.context_manager()
.get_context(worker.job_id)
.await
.unwrap();
assert_eq!(ctx.state, JobState::Failed);
}
#[tokio::test]
async fn test_iteration_cap_marks_failed_not_stuck() {
let worker = make_worker(vec![]).await;
// Transition to InProgress (required for mark_failed)
worker
.context_manager()
.update_context(worker.job_id, |ctx| {
ctx.transition_to(JobState::InProgress, None)
})
.await
.unwrap()
.unwrap();
// Simulate what the execution loop does when max_iterations is exceeded
worker
.mark_failed("Maximum iterations exceeded: job hit the iteration cap")
.await
.unwrap();
let ctx = worker
.context_manager()
.get_context(worker.job_id)
.await
.unwrap();
assert_eq!(
ctx.state,
JobState::Failed,
"Iteration cap should transition to Failed, not Stuck"
);
}
}
+202 -206
View File
@@ -21,7 +21,7 @@ use crate::secrets::SecretsStore;
use crate::skills::SkillRegistry;
use crate::skills::catalog::SkillCatalog;
use crate::tools::ToolRegistry;
use crate::tools::mcp::McpSessionManager;
use crate::tools::mcp::{McpProcessManager, McpSessionManager};
use crate::tools::wasm::SharedCredentialRegistry;
use crate::tools::wasm::WasmToolRuntime;
use crate::workspace::{EmbeddingProvider, Workspace};
@@ -41,6 +41,7 @@ pub struct AppComponents {
pub workspace: Option<Arc<Workspace>>,
pub extension_manager: Option<Arc<ExtensionManager>>,
pub mcp_session_manager: Arc<McpSessionManager>,
pub mcp_process_manager: Arc<McpProcessManager>,
pub wasm_tool_runtime: Option<Arc<WasmToolRuntime>>,
pub log_broadcaster: Arc<LogBroadcaster>,
pub context_manager: Arc<ContextManager>,
@@ -76,10 +77,7 @@ pub struct AppBuilder {
llm_override: Option<Arc<dyn LlmProvider>>,
// Backend-specific handles needed by secrets store
#[cfg(feature = "postgres")]
pg_pool: Option<deadpool_postgres::Pool>,
#[cfg(feature = "libsql")]
libsql_db: Option<Arc<libsql::Database>>,
handles: Option<crate::db::DatabaseHandles>,
}
impl AppBuilder {
@@ -104,10 +102,7 @@ impl AppBuilder {
db: None,
secrets_store: None,
llm_override: None,
#[cfg(feature = "postgres")]
pg_pool: None,
#[cfg(feature = "libsql")]
libsql_db: None,
handles: None,
}
}
@@ -136,71 +131,10 @@ impl AppBuilder {
return Ok(());
}
let db: Arc<dyn Database> = match self.config.database.backend {
#[cfg(feature = "libsql")]
crate::config::DatabaseBackend::LibSql => {
use crate::db::Database as _;
use crate::db::libsql::LibSqlBackend;
use secrecy::ExposeSecret as _;
let default_path = crate::config::default_libsql_path();
let db_path = self
.config
.database
.libsql_path
.as_deref()
.unwrap_or(&default_path);
let backend = if let Some(ref url) = self.config.database.libsql_url {
let token =
self.config
.database
.libsql_auth_token
.as_ref()
.ok_or_else(|| {
anyhow::anyhow!(
"LIBSQL_AUTH_TOKEN is required when LIBSQL_URL is set"
)
})?;
LibSqlBackend::new_remote_replica(db_path, url, token.expose_secret()).await?
} else {
LibSqlBackend::new_local(db_path).await?
};
backend.run_migrations().await?;
tracing::info!("libSQL database connected and migrations applied");
#[cfg(feature = "libsql")]
{
self.libsql_db = Some(backend.shared_db());
}
Arc::new(backend) as Arc<dyn Database>
}
#[cfg(feature = "postgres")]
_ => {
use crate::db::Database as _;
let pg = crate::db::postgres::PgBackend::new(&self.config.database)
.await
.map_err(|e| anyhow::anyhow!("{}", e))?;
pg.run_migrations()
.await
.map_err(|e| anyhow::anyhow!("{}", e))?;
tracing::info!("PostgreSQL database connected and migrations applied");
#[cfg(feature = "postgres")]
{
self.pg_pool = Some(pg.pool());
}
Arc::new(pg) as Arc<dyn Database>
}
#[cfg(not(feature = "postgres"))]
_ => {
anyhow::bail!(
"No database backend available. Enable 'postgres' or 'libsql' feature."
);
}
};
let (db, handles) = crate::db::connect_with_handles(&self.config.database)
.await
.map_err(|e| anyhow::anyhow!("{}", e))?;
self.handles = Some(handles);
// Post-init: migrate disk config, reload config from DB, attach session, cleanup
if let Err(e) = crate::bootstrap::migrate_disk_to_db(db.as_ref(), "default").await {
@@ -211,7 +145,7 @@ impl AppBuilder {
match Config::from_db_with_toml(db.as_ref(), "default", toml_path).await {
Ok(db_config) => {
self.config = db_config;
tracing::info!("Configuration reloaded from database");
tracing::debug!("Configuration reloaded from database");
}
Err(e) => {
tracing::warn!(
@@ -244,11 +178,28 @@ impl AppBuilder {
let master_key = match self.config.secrets.master_key() {
Some(k) => k,
None => {
// No secrets DB available, but we can still load tokens from
// OS credential stores (e.g., Anthropic OAuth via Claude Code's
// macOS Keychain / Linux ~/.claude/.credentials.json).
crate::config::inject_os_credentials();
// Consume unused handles
#[cfg(feature = "libsql")]
self.handles.take();
// Re-resolve only the LLM config with OS credentials.
let store: Option<&(dyn crate::db::SettingsStore + Sync)> =
self.db.as_ref().map(|db| db.as_ref() as _);
let toml_path = self.toml_path.as_deref();
if let Err(e) = self
.config
.re_resolve_llm(store, "default", toml_path)
.await
{
self.libsql_db.take();
tracing::warn!(
"Failed to re-resolve LLM config after OS credential injection: {e}"
);
}
return Ok(());
}
};
@@ -257,52 +208,31 @@ impl AppBuilder {
Ok(c) => Arc::new(c),
Err(e) => {
tracing::warn!("Failed to initialize secrets crypto: {}", e);
#[cfg(feature = "libsql")]
{
self.libsql_db.take();
}
self.handles.take();
return Ok(());
}
};
let store: Option<Arc<dyn SecretsStore + Send + Sync>> = None;
#[cfg(feature = "libsql")]
let store = store.or_else(|| {
self.libsql_db.take().map(|db| {
Arc::new(crate::secrets::LibSqlSecretsStore::new(
db,
Arc::clone(&crypto),
)) as Arc<dyn SecretsStore + Send + Sync>
})
});
#[cfg(feature = "postgres")]
let store = store.or_else(|| {
self.pg_pool.as_ref().map(|pool| {
Arc::new(crate::secrets::PostgresSecretsStore::new(
pool.clone(),
Arc::clone(&crypto),
)) as Arc<dyn SecretsStore + Send + Sync>
})
});
// Fallback covers the no-database path where `init_database` returned
// early before populating `self.handles`.
let empty_handles = crate::db::DatabaseHandles::default();
let handles = self.handles.as_ref().unwrap_or(&empty_handles);
let store = crate::secrets::create_secrets_store(crypto, handles);
if let Some(ref secrets) = store {
// Inject LLM API keys from encrypted storage
crate::config::inject_llm_keys_from_secrets(secrets.as_ref(), "default").await;
// Re-resolve config with newly available keys
if let Some(ref db) = self.db {
let toml_path = self.toml_path.as_deref();
match Config::from_db_with_toml(db.as_ref(), "default", toml_path).await {
Ok(refreshed) => {
self.config = refreshed;
tracing::debug!("LlmConfig re-resolved after secret injection");
}
Err(e) => {
tracing::warn!("Failed to re-resolve config after secret injection: {}", e);
}
}
// Re-resolve only the LLM config with newly available keys.
let store: Option<&(dyn crate::db::SettingsStore + Sync)> =
self.db.as_ref().map(|db| db.as_ref() as _);
let toml_path = self.toml_path.as_deref();
if let Err(e) = self
.config
.re_resolve_llm(store, "default", toml_path)
.await
{
tracing::warn!("Failed to re-resolve LLM config after secret injection: {e}");
}
}
@@ -315,7 +245,7 @@ impl AppBuilder {
/// Delegates to `build_provider_chain` which applies all decorators
/// (retry, smart routing, failover, circuit breaker, response cache).
#[allow(clippy::type_complexity)]
pub fn init_llm(
pub async fn init_llm(
&self,
) -> Result<
(
@@ -326,7 +256,7 @@ impl AppBuilder {
anyhow::Error,
> {
let (llm, cheap_llm, recording_handle) =
crate::llm::build_provider_chain(&self.config.llm, self.session.clone())?;
crate::llm::build_provider_chain(&self.config.llm, self.session.clone()).await?;
Ok((llm, cheap_llm, recording_handle))
}
@@ -344,7 +274,7 @@ impl AppBuilder {
anyhow::Error,
> {
let safety = Arc::new(SafetyLayer::new(&self.config.safety));
tracing::info!("Safety layer initialized");
tracing::debug!("Safety layer initialized");
// Initialize tool registry with credential injection support
let credential_registry = Arc::new(SharedCredentialRegistry::new());
@@ -381,18 +311,57 @@ impl AppBuilder {
None
};
// Register image/vision tools if we have a workspace and LLM API credentials
if workspace.is_some() {
let (api_base, api_key_opt) = if let Some(ref provider) = self.config.llm.provider {
(
provider.base_url.clone(),
provider.api_key.as_ref().map(|s| {
use secrecy::ExposeSecret;
s.expose_secret().to_string()
}),
)
} else {
(
self.config.llm.nearai.base_url.clone(),
self.config.llm.nearai.api_key.as_ref().map(|s| {
use secrecy::ExposeSecret;
s.expose_secret().to_string()
}),
)
};
if let Some(api_key) = api_key_opt {
// Check for image generation models
let model_name = self
.config
.llm
.provider
.as_ref()
.map(|p| p.model.clone())
.unwrap_or_else(|| self.config.llm.nearai.model.clone());
let models = vec![model_name.clone()];
let gen_model = crate::llm::image_models::suggest_image_model(&models)
.unwrap_or("flux-1.1-pro")
.to_string();
tools.register_image_tools(api_base.clone(), api_key.clone(), gen_model, None);
// Check for vision models
let vision_model = crate::llm::vision_models::suggest_vision_model(&models)
.unwrap_or(&model_name)
.to_string();
tools.register_vision_tools(api_base, api_key, vision_model, None);
}
}
// Register builder tool if enabled
if self.config.builder.enabled
&& (self.config.agent.allow_local_tools || !self.config.sandbox.enabled)
{
tools
.register_builder_tool(
llm.clone(),
safety.clone(),
Some(self.config.builder.to_builder_config()),
)
.register_builder_tool(llm.clone(), Some(self.config.builder.to_builder_config()))
.await;
tracing::info!("Builder mode enabled");
tracing::debug!("Builder mode enabled");
}
Ok((safety, tools, embeddings, workspace))
@@ -406,6 +375,7 @@ impl AppBuilder {
) -> Result<
(
Arc<McpSessionManager>,
Arc<McpProcessManager>,
Option<Arc<WasmToolRuntime>>,
Option<Arc<ExtensionManager>>,
Vec<crate::extensions::RegistryEntry>,
@@ -413,10 +383,11 @@ impl AppBuilder {
),
anyhow::Error,
> {
use crate::tools::mcp::{McpClient, config::load_mcp_servers_from_db, is_authenticated};
use crate::tools::mcp::config::load_mcp_servers_from_db;
use crate::tools::wasm::{WasmToolLoader, load_dev_tools};
let mcp_session_manager = Arc::new(McpSessionManager::new());
let mcp_process_manager = Arc::new(McpProcessManager::new());
// Create WASM tool runtime eagerly so extensions installed after startup
// (e.g. via the web UI) can still be activated. The tools directory is only
@@ -448,7 +419,7 @@ impl AppBuilder {
match loader.load_from_dir(&wasm_config.tools_dir).await {
Ok(results) => {
if !results.loaded.is_empty() {
tracing::info!(
tracing::debug!(
"Loaded {} WASM tools from {}",
results.loaded.len(),
wasm_config.tools_dir.display()
@@ -471,7 +442,7 @@ impl AppBuilder {
Ok(results) => {
dev_loaded_tool_names.extend(results.loaded.iter().cloned());
if !dev_loaded_tool_names.is_empty() {
tracing::info!(
tracing::debug!(
"Loaded {} dev WASM tools from build artifacts",
dev_loaded_tool_names.len()
);
@@ -492,97 +463,107 @@ impl AppBuilder {
let db = self.db.clone();
let tools = Arc::clone(tools);
let mcp_sm = Arc::clone(&mcp_session_manager);
let pm = Arc::clone(&mcp_process_manager);
async move {
if let Some(ref secrets) = secrets_store {
let servers_result = if let Some(ref d) = db {
load_mcp_servers_from_db(d.as_ref(), "default").await
} else {
crate::tools::mcp::config::load_mcp_servers().await
};
match servers_result {
Ok(servers) => {
let enabled: Vec<_> = servers.enabled_servers().cloned().collect();
if !enabled.is_empty() {
tracing::info!(
"Loading {} configured MCP server(s)...",
enabled.len()
);
}
let servers_result = if let Some(ref d) = db {
load_mcp_servers_from_db(d.as_ref(), "default").await
} else {
crate::tools::mcp::config::load_mcp_servers().await
};
match servers_result {
Ok(servers) => {
let enabled: Vec<_> = servers.enabled_servers().cloned().collect();
if !enabled.is_empty() {
tracing::debug!(
"Loading {} configured MCP server(s)...",
enabled.len()
);
}
let mut join_set = tokio::task::JoinSet::new();
for server in enabled {
let mcp_sm = Arc::clone(&mcp_sm);
let secrets = Arc::clone(secrets);
let tools = Arc::clone(&tools);
let mut join_set = tokio::task::JoinSet::new();
for server in enabled {
let mcp_sm = Arc::clone(&mcp_sm);
let secrets = secrets_store.clone();
let tools = Arc::clone(&tools);
let pm = Arc::clone(&pm);
join_set.spawn(async move {
let server_name = server.name.clone();
let has_tokens =
is_authenticated(&server, &secrets, "default").await;
join_set.spawn(async move {
let server_name = server.name.clone();
let client = if has_tokens || server.requires_auth() {
McpClient::new_authenticated(
server, mcp_sm, secrets, "default",
)
} else {
McpClient::new_with_name(&server_name, &server.url)
};
let client = match crate::tools::mcp::create_client_from_config(
server,
&mcp_sm,
&pm,
secrets,
"default",
)
.await
{
Ok(c) => c,
Err(e) => {
tracing::warn!(
"Failed to create MCP client for '{}': {}",
server_name,
e
);
return;
}
};
match client.list_tools().await {
Ok(mcp_tools) => {
let tool_count = mcp_tools.len();
match client.create_tools().await {
Ok(tool_impls) => {
for tool in tool_impls {
tools.register(tool).await;
}
tracing::info!(
"Loaded {} tools from MCP server '{}'",
tool_count,
server_name
);
match client.list_tools().await {
Ok(mcp_tools) => {
let tool_count = mcp_tools.len();
match client.create_tools().await {
Ok(tool_impls) => {
for tool in tool_impls {
tools.register(tool).await;
}
Err(e) => {
tracing::warn!(
"Failed to create tools from MCP server '{}': {}",
server_name,
e
);
}
}
}
Err(e) => {
let err_str = e.to_string();
if err_str.contains("401")
|| err_str.contains("authentication")
{
tracing::warn!(
"MCP server '{}' requires authentication. \
Run: ironclaw mcp auth {}",
server_name,
tracing::debug!(
"Loaded {} tools from MCP server '{}'",
tool_count,
server_name
);
} else {
}
Err(e) => {
tracing::warn!(
"Failed to connect to MCP server '{}': {}",
"Failed to create tools from MCP server '{}': {}",
server_name,
e
);
}
}
}
});
}
while let Some(result) = join_set.join_next().await {
if let Err(e) = result {
tracing::warn!("MCP server loading task panicked: {}", e);
Err(e) => {
let err_str = e.to_string();
if err_str.contains("401")
|| err_str.contains("authentication")
{
tracing::warn!(
"MCP server '{}' requires authentication. \
Run: ironclaw mcp auth {}",
server_name,
server_name
);
} else {
tracing::warn!(
"Failed to connect to MCP server '{}': {}",
server_name,
e
);
}
}
}
});
}
while let Some(result) = join_set.join_next().await {
if let Err(e) = result {
tracing::warn!("MCP server loading task panicked: {}", e);
}
}
Err(e) => {
tracing::debug!("No MCP servers configured ({})", e);
}
}
Err(e) => {
tracing::debug!("No MCP servers configured ({})", e);
}
}
}
@@ -598,7 +579,7 @@ impl AppBuilder {
.iter()
.map(|m| m.to_registry_entry())
.collect();
tracing::info!(
tracing::debug!(
count = entries.len(),
"Loaded registry catalog entries for extension discovery"
);
@@ -627,6 +608,7 @@ impl AppBuilder {
let extension_manager = {
let manager = Arc::new(ExtensionManager::new(
Arc::clone(&mcp_session_manager),
Arc::clone(&mcp_process_manager),
ext_secrets,
Arc::clone(tools),
Some(Arc::clone(hooks)),
@@ -639,7 +621,7 @@ impl AppBuilder {
catalog_entries.clone(),
));
tools.register_extension_tools(Arc::clone(&manager));
tracing::info!("Extension manager initialized with in-chat discovery tools");
tracing::debug!("Extension manager initialized with in-chat discovery tools");
Some(manager)
};
@@ -653,6 +635,7 @@ impl AppBuilder {
Ok((
mcp_session_manager,
mcp_process_manager,
wasm_tool_runtime,
extension_manager,
catalog_entries,
@@ -665,10 +648,21 @@ impl AppBuilder {
self.init_database().await?;
self.init_secrets().await?;
// Post-init validation: if a non-nearai backend was selected but
// credentials were never resolved (deferred resolution found no keys),
// fail early with a clear error instead of a confusing runtime failure.
if self.config.llm.backend != "nearai" && self.config.llm.provider.is_none() {
let backend = &self.config.llm.backend;
anyhow::bail!(
"LLM_BACKEND={backend} is configured but no credentials were found. \
Set the appropriate API key environment variable or run the setup wizard."
);
}
let (llm, cheap_llm, recording_handle) = if let Some(llm) = self.llm_override.take() {
(llm, None, None)
} else {
self.init_llm()?
self.init_llm().await?
};
let (safety, tools, embeddings, workspace) = self.init_tools(&llm).await?;
@@ -677,6 +671,7 @@ impl AppBuilder {
let (
mcp_session_manager,
mcp_process_manager,
wasm_tool_runtime,
extension_manager,
catalog_entries,
@@ -697,7 +692,7 @@ impl AppBuilder {
let import_path = std::path::Path::new(&import_dir);
match ws.import_from_directory(import_path).await {
Ok(count) if count > 0 => {
tracing::info!("Imported {} workspace file(s) from {}", count, import_dir);
tracing::debug!("Imported {} workspace file(s) from {}", count, import_dir);
}
Ok(_) => {}
Err(e) => {
@@ -722,7 +717,7 @@ impl AppBuilder {
tokio::spawn(async move {
match ws_bg.backfill_embeddings().await {
Ok(count) if count > 0 => {
tracing::info!("Backfilled embeddings for {} chunks", count);
tracing::debug!("Backfilled embeddings for {} chunks", count);
}
Ok(_) => {}
Err(e) => {
@@ -739,7 +734,7 @@ impl AppBuilder {
.with_installed_dir(self.config.skills.installed_dir.clone());
let loaded = registry.discover_all().await;
if !loaded.is_empty() {
tracing::info!("Loaded {} skill(s): {}", loaded.len(), loaded.join(", "));
tracing::debug!("Loaded {} skill(s): {}", loaded.len(), loaded.join(", "));
}
let registry = Arc::new(std::sync::RwLock::new(registry));
let catalog = crate::skills::catalog::shared_catalog();
@@ -757,7 +752,7 @@ impl AppBuilder {
},
));
tracing::info!(
tracing::debug!(
"Tool registry initialized with {} total tools",
tools.count()
);
@@ -774,6 +769,7 @@ impl AppBuilder {
workspace,
extension_manager,
mcp_session_manager,
mcp_process_manager,
wasm_tool_runtime,
log_broadcaster: self.log_broadcaster,
context_manager,
+407
View File
@@ -198,6 +198,58 @@ pub fn save_bootstrap_env_to(path: &std::path::Path, vars: &[(&str, &str)]) -> s
Ok(())
}
/// Update or add multiple variables in `~/.ironclaw/.env`, preserving existing content.
///
/// Like `upsert_bootstrap_var` but batched — replaces lines for any key in `vars`
/// and preserves all other existing lines. Use this instead of `save_bootstrap_env`
/// when you want to update specific keys without destroying user-added variables.
pub fn upsert_bootstrap_vars(vars: &[(&str, &str)]) -> std::io::Result<()> {
upsert_bootstrap_vars_to(&ironclaw_env_path(), vars)
}
/// Update or add multiple variables at an arbitrary path (testable variant).
pub fn upsert_bootstrap_vars_to(
path: &std::path::Path,
vars: &[(&str, &str)],
) -> std::io::Result<()> {
if let Some(parent) = path.parent() {
std::fs::create_dir_all(parent)?;
}
let keys_being_written: std::collections::HashSet<&str> =
vars.iter().map(|(k, _)| *k).collect();
let existing = match std::fs::read_to_string(path) {
Ok(contents) => contents,
Err(e) if e.kind() == std::io::ErrorKind::NotFound => String::new(),
Err(e) => return Err(e),
};
let mut result = String::new();
for line in existing.lines() {
// Extract key from lines matching `KEY=...`
let is_overwritten = line
.split_once('=')
.map(|(k, _)| keys_being_written.contains(k.trim()))
.unwrap_or(false);
if !is_overwritten {
result.push_str(line);
result.push('\n');
}
}
// Append all new key=value pairs
for (key, value) in vars {
let escaped = value.replace('\\', "\\\\").replace('"', "\\\"");
result.push_str(&format!("{}=\"{}\"\n", key, escaped));
}
std::fs::write(path, &result)?;
restrict_file_permissions(path)?;
Ok(())
}
/// Update or add a single variable in `~/.ironclaw/.env`, preserving existing content.
///
/// Unlike `save_bootstrap_env` (which overwrites the entire file), this
@@ -414,10 +466,103 @@ pub enum MigrationError {
Io(String),
}
// ── PID Lock ──────────────────────────────────────────────────────────────
/// Path to the PID lock file: `~/.ironclaw/ironclaw.pid`.
pub fn pid_lock_path() -> PathBuf {
ironclaw_base_dir().join("ironclaw.pid")
}
/// A PID-based lock that prevents multiple IronClaw instances from running
/// simultaneously.
///
/// Uses `fs4::try_lock_exclusive()` for atomic locking (no TOCTOU race),
/// then writes the current PID into the locked file for diagnostics.
/// The OS-level lock is held for the lifetime of this struct and
/// automatically released on drop (along with the PID file cleanup).
#[derive(Debug)]
pub struct PidLock {
path: PathBuf,
/// Held open to maintain the OS-level exclusive lock.
_file: std::fs::File,
}
/// Errors from PID lock acquisition.
#[derive(Debug, thiserror::Error)]
pub enum PidLockError {
#[error("Another IronClaw instance is already running (PID {pid})")]
AlreadyRunning { pid: u32 },
#[error("Failed to acquire PID lock: {0}")]
Io(#[from] std::io::Error),
}
impl PidLock {
/// Try to acquire the PID lock.
///
/// Uses an exclusive file lock (`flock`/`LockFileEx`) so that two
/// concurrent processes cannot both acquire the lock — no TOCTOU race.
/// If the lock file exists but the holding process is gone (stale),
/// the lock is reclaimed automatically by the OS.
pub fn acquire() -> Result<Self, PidLockError> {
Self::acquire_at(pid_lock_path())
}
/// Acquire at a specific path (for testing).
fn acquire_at(path: PathBuf) -> Result<Self, PidLockError> {
use fs4::FileExt;
use std::fs::OpenOptions;
use std::io::Write;
// Ensure parent directory exists
if let Some(parent) = path.parent() {
std::fs::create_dir_all(parent)?;
}
// Open (or create) the lock file
let mut file = OpenOptions::new()
.read(true)
.write(true)
.create(true)
.truncate(false)
.open(&path)?;
// Try non-blocking exclusive lock — if another process holds it,
// this fails immediately instead of blocking.
if let Err(e) = file.try_lock_exclusive() {
if e.kind() == std::io::ErrorKind::WouldBlock {
// Lock held by another process — read its PID for the error message
let pid = std::fs::read_to_string(&path)
.ok()
.and_then(|s| s.trim().parse::<u32>().ok())
.unwrap_or(0);
return Err(PidLockError::AlreadyRunning { pid });
}
// Other errors (permissions, unsupported filesystem, etc.)
return Err(PidLockError::Io(e));
}
// We hold the exclusive lock — write our PID
file.set_len(0)?; // truncate
write!(file, "{}", std::process::id())?;
Ok(PidLock { path, _file: file })
}
}
impl Drop for PidLock {
fn drop(&mut self) {
// Remove the PID file; the OS-level lock is released when _file is dropped.
let _ = std::fs::remove_file(&self.path);
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::process::Command;
use std::sync::Mutex;
use std::thread;
use std::time::{Duration, Instant};
use tempfile::tempdir;
static ENV_MUTEX: Mutex<()> = Mutex::new(());
@@ -986,4 +1131,266 @@ INJECTED="pwned"#;
unsafe { std::env::remove_var("IRONCLAW_BASE_DIR") };
}
}
// ── PID Lock tests ───────────────────────────────────────────────
#[test]
fn test_pid_lock_acquire_and_drop() {
let dir = tempdir().unwrap();
let pid_path = dir.path().join("ironclaw.pid");
// Acquire lock
let lock = PidLock::acquire_at(pid_path.clone()).unwrap();
assert!(pid_path.exists());
// PID file should contain our PID
let contents = std::fs::read_to_string(&pid_path).unwrap();
assert_eq!(contents.trim().parse::<u32>().unwrap(), std::process::id());
// Drop should remove the file
drop(lock);
assert!(!pid_path.exists());
}
#[test]
fn test_pid_lock_rejects_second_acquire() {
let dir = tempdir().unwrap();
let pid_path = dir.path().join("ironclaw.pid");
// First lock succeeds
let _lock1 = PidLock::acquire_at(pid_path.clone()).unwrap();
// Second acquire on same file must fail (exclusive flock held)
let result = PidLock::acquire_at(pid_path.clone());
assert!(result.is_err());
match result.unwrap_err() {
PidLockError::AlreadyRunning { pid } => {
assert_eq!(pid, std::process::id());
}
other => panic!("expected AlreadyRunning, got: {}", other),
}
}
#[test]
fn test_pid_lock_reclaims_after_drop() {
let dir = tempdir().unwrap();
let pid_path = dir.path().join("ironclaw.pid");
// Acquire and release
let lock = PidLock::acquire_at(pid_path.clone()).unwrap();
drop(lock);
// Should succeed — OS lock was released on drop
let lock2 = PidLock::acquire_at(pid_path).unwrap();
drop(lock2);
}
#[test]
fn test_pid_lock_reclaims_stale_file_without_flock() {
let dir = tempdir().unwrap();
let pid_path = dir.path().join("ironclaw.pid");
// Write a stale PID file manually (no flock held)
std::fs::write(&pid_path, "4294967294").unwrap();
// Should succeed because no OS lock is held on the file
let lock = PidLock::acquire_at(pid_path.clone()).unwrap();
let contents = std::fs::read_to_string(&pid_path).unwrap();
assert_eq!(contents.trim().parse::<u32>().unwrap(), std::process::id());
drop(lock);
}
#[test]
fn test_pid_lock_handles_corrupt_pid_file() {
let dir = tempdir().unwrap();
let pid_path = dir.path().join("ironclaw.pid");
// Write garbage (no flock held)
std::fs::write(&pid_path, "not-a-number").unwrap();
// Should succeed — no OS lock held, file is reclaimed
let lock = PidLock::acquire_at(pid_path).unwrap();
drop(lock);
}
#[test]
fn test_pid_lock_creates_parent_dirs() {
let dir = tempdir().unwrap();
let pid_path = dir.path().join("nested").join("deep").join("ironclaw.pid");
let lock = PidLock::acquire_at(pid_path.clone()).unwrap();
assert!(pid_path.exists());
drop(lock);
}
#[test]
fn test_pid_lock_child_helper_holds_lock() {
if std::env::var("IRONCLAW_PID_LOCK_CHILD").ok().as_deref() != Some("1") {
return;
}
let pid_path = PathBuf::from(
std::env::var("IRONCLAW_PID_LOCK_PATH").expect("IRONCLAW_PID_LOCK_PATH missing"),
);
let hold_ms = std::env::var("IRONCLAW_PID_LOCK_HOLD_MS")
.ok()
.and_then(|s| s.parse::<u64>().ok())
.unwrap_or(3000);
let _lock = PidLock::acquire_at(pid_path).expect("child failed to acquire pid lock");
thread::sleep(Duration::from_millis(hold_ms));
}
#[test]
fn test_pid_lock_rejects_lock_held_by_other_process() {
let dir = tempdir().unwrap();
let pid_path = dir.path().join("ironclaw.pid");
let current_exe = std::env::current_exe().unwrap();
let mut child = Command::new(current_exe)
.args([
"--exact",
"bootstrap::tests::test_pid_lock_child_helper_holds_lock",
"--nocapture",
"--test-threads=1",
])
.env("IRONCLAW_PID_LOCK_CHILD", "1")
.env("IRONCLAW_PID_LOCK_PATH", pid_path.display().to_string())
.env("IRONCLAW_PID_LOCK_HOLD_MS", "3000")
.spawn()
.unwrap();
let started = Instant::now();
while started.elapsed() < Duration::from_secs(2) {
if pid_path.exists() {
break;
}
if let Some(status) = child.try_wait().unwrap() {
panic!("child exited before acquiring lock: {}", status);
}
thread::sleep(Duration::from_millis(20));
}
assert!(
pid_path.exists(),
"child did not create lock file in time: {}",
pid_path.display()
);
let result = PidLock::acquire_at(pid_path.clone());
match result.unwrap_err() {
PidLockError::AlreadyRunning { .. } => {}
other => panic!("expected AlreadyRunning, got: {}", other),
}
let status = child.wait().unwrap();
assert!(status.success(), "child process failed: {}", status);
// After the child exits, lock should be released and reacquirable.
let lock = PidLock::acquire_at(pid_path).unwrap();
drop(lock);
}
#[test]
fn upsert_bootstrap_vars_preserves_unknown_keys() {
let dir = tempdir().unwrap();
let env_path = dir.path().join(".env");
// Simulate a user-edited .env with custom vars
let initial =
"HTTP_HOST=\"0.0.0.0\"\nDATABASE_BACKEND=\"postgres\"\nCUSTOM_VAR=\"keep_me\"\n";
std::fs::write(&env_path, initial).unwrap();
// Upsert wizard vars — should preserve HTTP_HOST and CUSTOM_VAR
let vars = [("DATABASE_BACKEND", "libsql"), ("LLM_BACKEND", "openai")];
upsert_bootstrap_vars_to(&env_path, &vars).unwrap();
let parsed: Vec<(String, String)> = dotenvy::from_path_iter(&env_path)
.unwrap()
.filter_map(|r| r.ok())
.collect();
assert_eq!(
parsed.len(),
4,
"should have 4 vars (2 preserved + 2 upserted)"
);
// User-added vars must be preserved
assert!(
parsed
.iter()
.any(|(k, v)| k == "HTTP_HOST" && v == "0.0.0.0"),
"HTTP_HOST must be preserved"
);
assert!(
parsed
.iter()
.any(|(k, v)| k == "CUSTOM_VAR" && v == "keep_me"),
"CUSTOM_VAR must be preserved"
);
// Wizard vars must be updated/added
assert!(
parsed
.iter()
.any(|(k, v)| k == "DATABASE_BACKEND" && v == "libsql"),
"DATABASE_BACKEND must be updated to libsql"
);
assert!(
parsed
.iter()
.any(|(k, v)| k == "LLM_BACKEND" && v == "openai"),
"LLM_BACKEND must be added"
);
// Now update LLM_BACKEND and verify HTTP_HOST still preserved
let vars2 = [("LLM_BACKEND", "anthropic")];
upsert_bootstrap_vars_to(&env_path, &vars2).unwrap();
let parsed2: Vec<(String, String)> = dotenvy::from_path_iter(&env_path)
.unwrap()
.filter_map(|r| r.ok())
.collect();
assert_eq!(
parsed2.len(),
4,
"should still have 4 vars after second upsert"
);
assert!(
parsed2
.iter()
.any(|(k, v)| k == "HTTP_HOST" && v == "0.0.0.0"),
"HTTP_HOST must still be preserved after second upsert"
);
assert!(
parsed2
.iter()
.any(|(k, v)| k == "LLM_BACKEND" && v == "anthropic"),
"LLM_BACKEND must be updated to anthropic"
);
}
#[test]
fn upsert_bootstrap_vars_creates_file_if_missing() {
let dir = tempdir().unwrap();
let env_path = dir.path().join("subdir").join(".env");
// File doesn't exist yet
assert!(!env_path.exists());
let vars = [("DATABASE_BACKEND", "libsql")];
upsert_bootstrap_vars_to(&env_path, &vars).unwrap();
assert!(env_path.exists());
let parsed: Vec<(String, String)> = dotenvy::from_path_iter(&env_path)
.unwrap()
.filter_map(|r| r.ok())
.collect();
assert_eq!(parsed.len(), 1);
assert_eq!(
parsed[0],
("DATABASE_BACKEND".to_string(), "libsql".to_string())
);
}
}
+25 -2
View File
@@ -79,6 +79,8 @@ pub struct IncomingMessage {
pub received_at: DateTime<Utc>,
/// Channel-specific metadata.
pub metadata: serde_json::Value,
/// IANA timezone string from the client (e.g. "America/New_York").
pub timezone: Option<String>,
/// File or media attachments on this message.
pub attachments: Vec<IncomingAttachment>,
}
@@ -99,6 +101,7 @@ impl IncomingMessage {
thread_id: None,
received_at: Utc::now(),
metadata: serde_json::Value::Null,
timezone: None,
attachments: Vec::new(),
}
}
@@ -121,6 +124,12 @@ impl IncomingMessage {
self
}
/// Set the client timezone.
pub fn with_timezone(mut self, tz: impl Into<String>) -> Self {
self.timezone = Some(tz.into());
self
}
/// Set attachments.
pub fn with_attachments(mut self, attachments: Vec<IncomingAttachment>) -> Self {
self.attachments = attachments;
@@ -222,6 +231,13 @@ pub enum StatusUpdate {
success: bool,
message: String,
},
/// An image was generated by a tool.
ImageGenerated {
/// Base64 data URL of the generated image.
data_url: String,
/// Optional workspace path where the image was saved.
path: Option<String>,
},
}
impl StatusUpdate {
@@ -331,6 +347,7 @@ pub trait Channel: Send + Sync {
#[cfg(test)]
mod tests {
use super::*;
use crate::testing::credentials::TEST_REDACT_SECRET_123;
/// Stub tool that marks `"value"` as sensitive.
struct SecretTool;
@@ -360,7 +377,7 @@ mod tests {
#[test]
fn tool_completed_redacts_sensitive_params_on_failure() {
let params = serde_json::json!({"name": "api_key", "value": "sk-secret-123"});
let params = serde_json::json!({"name": "api_key", "value": TEST_REDACT_SECRET_123});
let err: Result<String, crate::error::Error> =
Err(crate::error::ToolError::ExecutionFailed {
name: "secret_save".into(),
@@ -395,7 +412,7 @@ mod tests {
param_str
);
assert!(
!param_str.contains("sk-secret-123"),
!param_str.contains(TEST_REDACT_SECRET_123),
"raw secret should not appear: {}",
param_str
);
@@ -454,4 +471,10 @@ mod tests {
panic!("expected ToolCompleted variant");
}
}
#[test]
fn test_incoming_message_with_timezone() {
let msg = IncomingMessage::new("test", "user1", "hello").with_timezone("America/New_York");
assert_eq!(msg.timezone.as_deref(), Some("America/New_York"));
}
}
+126 -6
View File
@@ -17,7 +17,9 @@ use tokio::sync::{RwLock, mpsc, oneshot};
use tokio_stream::wrappers::ReceiverStream;
use uuid::Uuid;
use crate::channels::{Channel, IncomingMessage, MessageStream, OutgoingResponse};
use crate::channels::{
AttachmentKind, Channel, IncomingAttachment, IncomingMessage, MessageStream, OutgoingResponse,
};
use crate::config::HttpConfig;
use crate::error::ChannelError;
@@ -46,8 +48,9 @@ struct RateLimitState {
request_count: u32,
}
/// Maximum JSON body size for webhook requests (64 KB).
const MAX_BODY_BYTES: usize = 64 * 1024;
/// Maximum JSON body size for webhook requests (15 MB, to support base64 image attachments
/// with ~33% overhead from base64 encoding).
const MAX_BODY_BYTES: usize = 15 * 1024 * 1024;
/// Maximum number of pending wait-for-response requests.
const MAX_PENDING_RESPONSES: usize = 100;
@@ -115,8 +118,34 @@ struct WebhookRequest {
/// Whether to wait for a synchronous response.
#[serde(default)]
wait_for_response: bool,
/// Optional file attachments (base64-encoded).
#[serde(default)]
attachments: Vec<AttachmentData>,
}
/// A file attachment in a webhook request.
#[derive(Debug, Deserialize)]
struct AttachmentData {
/// MIME type (e.g. "image/png", "application/pdf").
mime_type: String,
/// Optional filename.
#[serde(default)]
filename: Option<String>,
/// Base64-encoded file data.
#[serde(default)]
data_base64: Option<String>,
/// URL to fetch the file from (not downloaded server-side for SSRF prevention).
#[serde(default)]
url: Option<String>,
}
/// Maximum size per attachment (5 MB decoded).
const MAX_ATTACHMENT_BYTES: usize = 5 * 1024 * 1024;
/// Maximum total attachment size (10 MB decoded).
const MAX_TOTAL_ATTACHMENT_BYTES: usize = 10 * 1024 * 1024;
/// Maximum number of attachments per request.
const MAX_ATTACHMENTS: usize = 5;
#[derive(Debug, Serialize)]
struct WebhookResponse {
/// Message ID assigned to this request.
@@ -211,15 +240,106 @@ async fn webhook_handler(
);
}
let msg = IncomingMessage::new("http", &state.user_id, &req.content).with_metadata(
// Validate and decode attachments
let attachments = if !req.attachments.is_empty() {
if req.attachments.len() > MAX_ATTACHMENTS {
return (
StatusCode::BAD_REQUEST,
Json(WebhookResponse {
message_id: Uuid::nil(),
status: "error".to_string(),
response: Some(format!("Too many attachments (max {})", MAX_ATTACHMENTS)),
}),
);
}
let mut decoded_attachments = Vec::new();
let mut total_bytes: usize = 0;
for att in &req.attachments {
if let Some(ref b64) = att.data_base64 {
use base64::Engine;
let data = match base64::engine::general_purpose::STANDARD.decode(b64) {
Ok(d) => d,
Err(_) => {
return (
StatusCode::BAD_REQUEST,
Json(WebhookResponse {
message_id: Uuid::nil(),
status: "error".to_string(),
response: Some("Invalid base64 in attachment".to_string()),
}),
);
}
};
if data.len() > MAX_ATTACHMENT_BYTES {
return (
StatusCode::PAYLOAD_TOO_LARGE,
Json(WebhookResponse {
message_id: Uuid::nil(),
status: "error".to_string(),
response: Some(format!(
"Attachment too large (max {} bytes)",
MAX_ATTACHMENT_BYTES
)),
}),
);
}
total_bytes += data.len();
if total_bytes > MAX_TOTAL_ATTACHMENT_BYTES {
return (
StatusCode::PAYLOAD_TOO_LARGE,
Json(WebhookResponse {
message_id: Uuid::nil(),
status: "error".to_string(),
response: Some("Total attachment size exceeds limit".to_string()),
}),
);
}
decoded_attachments.push(IncomingAttachment {
id: Uuid::new_v4().to_string(),
kind: AttachmentKind::from_mime_type(&att.mime_type),
mime_type: att.mime_type.clone(),
filename: att.filename.clone(),
size_bytes: Some(data.len() as u64),
source_url: None,
storage_key: None,
extracted_text: None,
data,
duration_secs: None,
});
} else if let Some(ref url) = att.url {
// URL-only attachment: set source_url but don't download (SSRF prevention)
decoded_attachments.push(IncomingAttachment {
id: Uuid::new_v4().to_string(),
kind: AttachmentKind::from_mime_type(&att.mime_type),
mime_type: att.mime_type.clone(),
filename: att.filename.clone(),
size_bytes: None,
source_url: Some(url.clone()),
storage_key: None,
extracted_text: None,
data: Vec::new(),
duration_secs: None,
});
}
}
decoded_attachments
} else {
Vec::new()
};
let mut msg = IncomingMessage::new("http", &state.user_id, &req.content).with_metadata(
serde_json::json!({
"wait_for_response": req.wait_for_response,
}),
);
if !attachments.is_empty() {
msg = msg.with_attachments(attachments);
}
if let Some(thread_id) = &req.thread_id {
let msg = msg.with_thread(thread_id);
return process_message(state, msg, req.wait_for_response).await;
msg = msg.with_thread(thread_id);
}
process_message(state, msg, req.wait_for_response).await
+2 -2
View File
@@ -75,7 +75,7 @@ impl ChannelManager {
break;
}
}
tracing::info!(channel = %name, "Hot-added channel stream ended");
tracing::debug!(channel = %name, "Hot-added channel stream ended");
});
Ok(())
@@ -92,7 +92,7 @@ impl ChannelManager {
for (name, channel) in channels.iter() {
match channel.start().await {
Ok(stream) => {
tracing::info!("Started channel: {}", name);
tracing::debug!("Started channel: {}", name);
streams.push(stream);
}
Err(e) => {
+57 -9
View File
@@ -18,7 +18,7 @@
//! - `Esc` - Interrupt current operation
use std::borrow::Cow;
use std::io::{self, Write};
use std::io::{self, IsTerminal, Write};
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
@@ -297,10 +297,15 @@ impl Channel for ReplChannel {
let esc_interrupt_triggered_for_thread = Arc::new(AtomicBool::new(false));
std::thread::spawn(move || {
let sys_tz = crate::timezone::detect_system_timezone().name().to_string();
// Single message mode: send it and return
if let Some(msg) = single_message {
let incoming = IncomingMessage::new("repl", "default", &msg);
let incoming = IncomingMessage::new("repl", "default", &msg).with_timezone(&sys_tz);
let _ = tx.blocking_send(incoming);
// Ensure the agent exits after handling exactly one turn in -m mode,
// even when other channels (gateway/http) are enabled.
let _ = tx.blocking_send(IncomingMessage::new("repl", "default", "/quit"));
return;
}
@@ -361,7 +366,8 @@ impl Channel for ReplChannel {
"/quit" | "/exit" => {
// Forward shutdown command so the agent loop exits even
// when other channels (e.g. web gateway) are still active.
let msg = IncomingMessage::new("repl", "default", "/quit");
let msg = IncomingMessage::new("repl", "default", "/quit")
.with_timezone(&sys_tz);
let _ = tx.blocking_send(msg);
break;
}
@@ -382,7 +388,8 @@ impl Channel for ReplChannel {
_ => {}
}
let msg = IncomingMessage::new("repl", "default", line);
let msg =
IncomingMessage::new("repl", "default", line).with_timezone(&sys_tz);
if tx.blocking_send(msg).is_err() {
break;
}
@@ -390,21 +397,29 @@ impl Channel for ReplChannel {
Err(ReadlineError::Interrupted) => {
if esc_interrupt_triggered_for_thread.swap(false, Ordering::Relaxed) {
// Esc: interrupt current operation and keep REPL open.
let msg = IncomingMessage::new("repl", "default", "/interrupt");
let msg = IncomingMessage::new("repl", "default", "/interrupt")
.with_timezone(&sys_tz);
if tx.blocking_send(msg).is_err() {
break;
}
} else {
// Ctrl+C (VINTR): request graceful shutdown.
let msg = IncomingMessage::new("repl", "default", "/quit");
let msg = IncomingMessage::new("repl", "default", "/quit")
.with_timezone(&sys_tz);
let _ = tx.blocking_send(msg);
break;
}
}
Err(ReadlineError::Eof) => {
// Ctrl+D: send /quit so the agent loop runs graceful shutdown
let msg = IncomingMessage::new("repl", "default", "/quit");
let _ = tx.blocking_send(msg);
// Ctrl+D in interactive mode: graceful shutdown.
// In daemon mode (stdin = /dev/null, no TTY), EOF arrives
// immediately — just drop the REPL thread silently so other
// channels (gateway, telegram, …) keep running.
if std::io::stdin().is_terminal() {
let msg = IncomingMessage::new("repl", "default", "/quit")
.with_timezone(&sys_tz);
let _ = tx.blocking_send(msg);
}
break;
}
Err(e) => {
@@ -585,6 +600,13 @@ impl Channel for ReplChannel {
eprintln!("\x1b[31m {extension_name}: {message}\x1b[0m");
}
}
StatusUpdate::ImageGenerated { path, .. } => {
if let Some(ref p) = path {
eprintln!("\x1b[36m [image] {p}\x1b[0m");
} else {
eprintln!("\x1b[36m [image generated]\x1b[0m");
}
}
}
Ok(())
}
@@ -614,3 +636,29 @@ impl Channel for ReplChannel {
Ok(())
}
}
#[cfg(test)]
mod tests {
use futures::StreamExt;
use super::*;
#[tokio::test]
async fn single_message_mode_sends_message_then_quit() {
let repl = ReplChannel::with_message("hi".to_string());
let mut stream = repl.start().await.expect("repl start should succeed");
let first = stream.next().await.expect("first message missing");
assert_eq!(first.channel, "repl");
assert_eq!(first.content, "hi");
let second = stream.next().await.expect("quit message missing");
assert_eq!(second.channel, "repl");
assert_eq!(second.content, "/quit");
assert!(
stream.next().await.is_none(),
"stream should end after /quit"
);
}
}
+37 -6
View File
@@ -184,18 +184,32 @@ impl WasmChannelLoader {
/// └── telegram.capabilities.json
/// ```
pub async fn load_from_dir(&self, dir: &Path) -> Result<LoadResults, WasmChannelError> {
if !dir.is_dir() {
return Err(WasmChannelError::Io(std::io::Error::new(
std::io::ErrorKind::NotADirectory,
format!("{} is not a directory", dir.display()),
)));
match fs::metadata(dir).await {
Ok(meta) if meta.is_dir() => {}
Ok(_) => {
return Err(WasmChannelError::Io(std::io::Error::new(
std::io::ErrorKind::NotADirectory,
format!("{} is not a directory", dir.display()),
)));
}
Err(e) if e.kind() == std::io::ErrorKind::NotFound => {
return Ok(LoadResults::default());
}
Err(e) => return Err(WasmChannelError::Io(e)),
}
let mut results = LoadResults::default();
// Collect all .wasm entries first, then load in parallel
let mut channel_entries = Vec::new();
let mut entries = fs::read_dir(dir).await?;
// Handle TOCTOU: if read_dir fails with NotFound, treat as empty
let mut entries = match fs::read_dir(dir).await {
Ok(entries) => entries,
Err(e) if e.kind() == std::io::ErrorKind::NotFound => {
return Ok(LoadResults::default());
}
Err(e) => return Err(WasmChannelError::Io(e)),
};
while let Some(entry) = entries.next_entry().await? {
let path = entry.path();
@@ -486,4 +500,21 @@ mod tests {
let result = loader.load_from_files("", &wasm_path, None).await;
assert!(result.is_err());
}
#[tokio::test]
async fn load_from_dir_returns_empty_when_dir_missing() {
let config = WasmChannelRuntimeConfig::for_testing();
let runtime = Arc::new(WasmChannelRuntime::new(config).unwrap());
let loader = WasmChannelLoader::new(runtime, Arc::new(PairingStore::new()), None);
let dir = TempDir::new().unwrap();
let missing = dir.path().join("nonexistent_channels_dir");
let results = loader.load_from_dir(&missing).await;
// Must succeed with empty results, not error
let results = results.expect("missing dir should return Ok, not Err");
assert!(results.loaded.is_empty());
assert!(results.errors.is_empty());
}
}
+2
View File
@@ -86,6 +86,7 @@ mod loader;
mod router;
mod runtime;
mod schema;
pub mod setup;
pub(crate) mod signature;
#[allow(dead_code)]
pub(crate) mod storage;
@@ -105,4 +106,5 @@ pub use runtime::{PreparedChannelModule, WasmChannelRuntime, WasmChannelRuntimeC
pub use schema::{
ChannelCapabilitiesFile, ChannelConfig, SecretSetupSchema, SetupSchema, WebhookSchema,
};
pub use setup::{WasmChannelSetup, inject_channel_credentials, setup_wasm_channels};
pub use wrapper::{HttpResponse, SharedWasmChannel, WasmChannel};
+324
View File
@@ -0,0 +1,324 @@
//! WASM channel setup and credential injection.
//!
//! Encapsulates the logic for loading WASM channels, registering their
//! webhook routes, and injecting credentials from the secrets store.
use std::collections::HashSet;
use std::sync::Arc;
use crate::channels::wasm::{
LoadedChannel, RegisteredEndpoint, SharedWasmChannel, WasmChannel, WasmChannelLoader,
WasmChannelRouter, WasmChannelRuntime, WasmChannelRuntimeConfig, create_wasm_channel_router,
};
use crate::config::Config;
use crate::db::Database;
use crate::extensions::ExtensionManager;
use crate::pairing::PairingStore;
use crate::secrets::SecretsStore;
/// Result of WASM channel setup.
pub struct WasmChannelSetup {
pub channels: Vec<(String, Box<dyn crate::channels::Channel>)>,
pub channel_names: Vec<String>,
pub webhook_routes: Option<axum::Router>,
/// Runtime objects needed for hot-activation via ExtensionManager.
pub wasm_channel_runtime: Arc<WasmChannelRuntime>,
pub pairing_store: Arc<PairingStore>,
pub wasm_channel_router: Arc<WasmChannelRouter>,
}
/// Load WASM channels and register their webhook routes.
pub async fn setup_wasm_channels(
config: &Config,
secrets_store: &Option<Arc<dyn SecretsStore + Send + Sync>>,
extension_manager: Option<&Arc<ExtensionManager>>,
database: Option<&Arc<dyn Database>>,
) -> Option<WasmChannelSetup> {
let runtime = match WasmChannelRuntime::new(WasmChannelRuntimeConfig::default()) {
Ok(r) => Arc::new(r),
Err(e) => {
tracing::warn!("Failed to initialize WASM channel runtime: {}", e);
return None;
}
};
let pairing_store = Arc::new(PairingStore::new());
let settings_store: Option<Arc<dyn crate::db::SettingsStore>> =
database.map(|db| Arc::clone(db) as Arc<dyn crate::db::SettingsStore>);
let mut loader = WasmChannelLoader::new(
Arc::clone(&runtime),
Arc::clone(&pairing_store),
settings_store,
);
if let Some(secrets) = secrets_store {
loader = loader.with_secrets_store(Arc::clone(secrets));
}
let results = match loader
.load_from_dir(&config.channels.wasm_channels_dir)
.await
{
Ok(r) => r,
Err(e) => {
tracing::warn!("Failed to scan WASM channels directory: {}", e);
return None;
}
};
let wasm_router = Arc::new(WasmChannelRouter::new());
let mut channels: Vec<(String, Box<dyn crate::channels::Channel>)> = Vec::new();
let mut channel_names: Vec<String> = Vec::new();
for loaded in results.loaded {
let (name, channel) = register_channel(loaded, config, secrets_store, &wasm_router).await;
channel_names.push(name.clone());
channels.push((name, channel));
}
for (path, err) in &results.errors {
tracing::warn!("Failed to load WASM channel {}: {}", path.display(), err);
}
// Always create webhook routes (even with no channels loaded) so that
// channels hot-added at runtime can receive webhooks without a restart.
let webhook_routes = {
Some(create_wasm_channel_router(
Arc::clone(&wasm_router),
extension_manager.map(Arc::clone),
))
};
Some(WasmChannelSetup {
channels,
channel_names,
webhook_routes,
wasm_channel_runtime: runtime,
pairing_store,
wasm_channel_router: wasm_router,
})
}
/// Process a single loaded WASM channel: retrieve secrets, inject config,
/// register with the router, and set up signing keys and credentials.
async fn register_channel(
loaded: LoadedChannel,
config: &Config,
secrets_store: &Option<Arc<dyn SecretsStore + Send + Sync>>,
wasm_router: &Arc<WasmChannelRouter>,
) -> (String, Box<dyn crate::channels::Channel>) {
let channel_name = loaded.name().to_string();
tracing::info!("Loaded WASM channel: {}", channel_name);
let secret_name = loaded.webhook_secret_name();
let sig_key_secret_name = loaded.signature_key_secret_name();
let hmac_secret_name = loaded.hmac_secret_name();
let webhook_secret = if let Some(secrets) = secrets_store {
secrets
.get_decrypted("default", &secret_name)
.await
.ok()
.map(|s| s.expose().to_string())
} else {
None
};
let secret_header = loaded.webhook_secret_header().map(|s| s.to_string());
let webhook_path = format!("/webhook/{}", channel_name);
let endpoints = vec![RegisteredEndpoint {
channel_name: channel_name.clone(),
path: webhook_path,
methods: vec!["POST".to_string()],
require_secret: webhook_secret.is_some(),
}];
let channel_arc = Arc::new(loaded.channel);
// Inject runtime config (tunnel URL, webhook secret, owner_id).
{
let mut config_updates = std::collections::HashMap::new();
if let Some(ref tunnel_url) = config.tunnel.public_url {
config_updates.insert(
"tunnel_url".to_string(),
serde_json::Value::String(tunnel_url.clone()),
);
}
if let Some(ref secret) = webhook_secret {
config_updates.insert(
"webhook_secret".to_string(),
serde_json::Value::String(secret.clone()),
);
}
if let Some(&owner_id) = config
.channels
.wasm_channel_owner_ids
.get(channel_name.as_str())
{
config_updates.insert("owner_id".to_string(), serde_json::json!(owner_id));
}
if !config_updates.is_empty() {
channel_arc.update_config(config_updates).await;
tracing::info!(
channel = %channel_name,
has_tunnel = config.tunnel.public_url.is_some(),
has_webhook_secret = webhook_secret.is_some(),
"Injected runtime config into channel"
);
}
}
tracing::info!(
channel = %channel_name,
has_webhook_secret = webhook_secret.is_some(),
secret_header = ?secret_header,
"Registering channel with router"
);
wasm_router
.register(
Arc::clone(&channel_arc),
endpoints,
webhook_secret.clone(),
secret_header,
)
.await;
// Register Ed25519 signature key if declared in capabilities.
if let Some(ref sig_key_name) = sig_key_secret_name
&& let Some(secrets) = secrets_store
&& let Ok(key_secret) = secrets.get_decrypted("default", sig_key_name).await
{
match wasm_router
.register_signature_key(&channel_name, key_secret.expose())
.await
{
Ok(()) => {
tracing::info!(channel = %channel_name, "Registered Ed25519 signature key")
}
Err(e) => {
tracing::error!(channel = %channel_name, error = %e, "Invalid signature key in secrets store")
}
}
}
// Register HMAC signing secret if declared in capabilities.
if let Some(ref hmac_secret_name) = hmac_secret_name
&& let Some(secrets) = secrets_store
&& let Ok(secret) = secrets.get_decrypted("default", hmac_secret_name).await
{
wasm_router
.register_hmac_secret(&channel_name, secret.expose())
.await;
tracing::info!(channel = %channel_name, "Registered HMAC signing secret");
}
// Inject credentials from secrets store / environment.
if let Some(secrets) = secrets_store {
match inject_channel_credentials(&channel_arc, secrets.as_ref(), &channel_name).await {
Ok(count) => {
if count > 0 {
tracing::info!(
channel = %channel_name,
credentials_injected = count,
"Channel credentials injected"
);
}
}
Err(e) => {
tracing::error!(
channel = %channel_name,
error = %e,
"Failed to inject channel credentials"
);
}
}
}
(channel_name, Box::new(SharedWasmChannel::new(channel_arc)))
}
/// Inject credentials for a channel based on naming convention.
///
/// Looks for secrets matching the pattern `{channel_name}_*` and injects them
/// as credential placeholders (e.g., `telegram_bot_token` -> `{TELEGRAM_BOT_TOKEN}`).
///
/// Falls back to environment variables with the uppercase name if not found
/// in the secrets store (e.g., `TELEGRAM_BOT_TOKEN`).
pub async fn inject_channel_credentials(
channel: &Arc<WasmChannel>,
secrets: &dyn SecretsStore,
channel_name: &str,
) -> anyhow::Result<usize> {
let all_secrets = secrets
.list("default")
.await
.map_err(|e| anyhow::anyhow!("Failed to list secrets: {}", e))?;
let prefix = format!("{}_", channel_name);
let mut count = 0;
let mut injected_placeholders = HashSet::new();
for secret_meta in all_secrets {
if !secret_meta.name.starts_with(&prefix) {
continue;
}
let decrypted = match secrets.get_decrypted("default", &secret_meta.name).await {
Ok(d) => d,
Err(e) => {
tracing::warn!(
secret = %secret_meta.name,
error = %e,
"Failed to decrypt secret for channel credential injection"
);
continue;
}
};
let placeholder = secret_meta.name.to_uppercase();
tracing::debug!(
channel = %channel_name,
secret = %secret_meta.name,
placeholder = %placeholder,
"Injecting credential"
);
channel
.set_credential(&placeholder, decrypted.expose().to_string())
.await;
injected_placeholders.insert(placeholder);
count += 1;
}
// Fall back to environment variables for required secrets not found in the store.
// This allows channels to work when configured via env vars (e.g., TELEGRAM_BOT_TOKEN)
// without requiring the setup wizard to have run.
let caps = channel.capabilities();
if let Some(ref http_cap) = caps.tool_capabilities.http {
for cred_mapping in http_cap.credentials.values() {
let placeholder = cred_mapping.secret_name.to_uppercase();
if injected_placeholders.contains(&placeholder) {
continue;
}
if let Ok(env_value) = std::env::var(&placeholder)
&& !env_value.is_empty()
{
tracing::debug!(
channel = %channel_name,
placeholder = %placeholder,
"Injecting credential from environment variable"
);
channel.set_credential(&placeholder, env_value).await;
count += 1;
}
}
}
Ok(count)
}
+16 -5
View File
@@ -2809,6 +2809,14 @@ fn status_to_wit(status: &StatusUpdate, metadata: &serde_json::Value) -> wit_cha
),
metadata_json,
},
StatusUpdate::ImageGenerated { path, .. } => wit_channel::StatusUpdate {
status: wit_channel::StatusType::Status,
message: match path {
Some(p) => format!("[image] {}", p),
None => "[image generated]".to_string(),
},
metadata_json,
},
}
}
@@ -3051,6 +3059,7 @@ mod tests {
};
use crate::channels::wasm::wrapper::{HttpResponse, WasmChannel};
use crate::pairing::PairingStore;
use crate::testing::credentials::TEST_TELEGRAM_BOT_TOKEN;
use crate::tools::wasm::ResourceLimits;
fn create_test_channel() -> WasmChannel {
@@ -4001,7 +4010,7 @@ mod tests {
let mut creds = std::collections::HashMap::new();
creds.insert(
"TELEGRAM_BOT_TOKEN".to_string(),
"8218490433:AAEZeUxwqZ5OO3mOCXv7fKvpdhDgsmBBNis".to_string(),
TEST_TELEGRAM_BOT_TOKEN.to_string(),
);
creds.insert("OTHER_SECRET".to_string(), "s3cret".to_string());
@@ -4014,13 +4023,15 @@ mod tests {
Arc::new(PairingStore::new()),
);
let error = "HTTP request failed: error sending request for url \
(https://api.telegram.org/bot8218490433:AAEZeUxwqZ5OO3mOCXv7fKvpdhDgsmBBNis/getUpdates)";
let error = format!(
"HTTP request failed: error sending request for url \
(https://api.telegram.org/bot{TEST_TELEGRAM_BOT_TOKEN}/getUpdates)"
);
let redacted = store.redact_credentials(error);
let redacted = store.redact_credentials(&error);
assert!(
!redacted.contains("8218490433:AAEZeUxwqZ5OO3mOCXv7fKvpdhDgsmBBNis"),
!redacted.contains(TEST_TELEGRAM_BOT_TOKEN),
"credential value should be redacted"
);
assert!(
+27 -26
View File
@@ -83,14 +83,15 @@ pub async fn auth_middleware(
#[cfg(test)]
mod tests {
use super::*;
use crate::testing::credentials::{TEST_AUTH_SECRET_TOKEN, TEST_BEARER_TOKEN};
#[test]
fn test_auth_state_clone() {
let state = AuthState {
token: "test-token".to_string(),
token: TEST_BEARER_TOKEN.to_string(),
};
let cloned = state.clone();
assert_eq!(cloned.token, "test-token");
assert_eq!(cloned.token, TEST_BEARER_TOKEN);
}
use axum::Router;
@@ -120,10 +121,10 @@ mod tests {
#[tokio::test]
async fn test_valid_bearer_token_passes() {
let app = test_app("secret-token");
let app = test_app(TEST_AUTH_SECRET_TOKEN);
let req = Request::builder()
.uri("/api/chat/events")
.header("Authorization", "Bearer secret-token")
.header("Authorization", format!("Bearer {TEST_AUTH_SECRET_TOKEN}"))
.body(Body::empty())
.unwrap();
let resp = app.oneshot(req).await.unwrap();
@@ -132,7 +133,7 @@ mod tests {
#[tokio::test]
async fn test_invalid_bearer_token_rejected() {
let app = test_app("secret-token");
let app = test_app(TEST_AUTH_SECRET_TOKEN);
let req = Request::builder()
.uri("/api/chat/events")
.header("Authorization", "Bearer wrong-token")
@@ -144,9 +145,9 @@ mod tests {
#[tokio::test]
async fn test_query_token_allowed_for_chat_events() {
let app = test_app("secret-token");
let app = test_app(TEST_AUTH_SECRET_TOKEN);
let req = Request::builder()
.uri("/api/chat/events?token=secret-token")
.uri(format!("/api/chat/events?token={TEST_AUTH_SECRET_TOKEN}"))
.body(Body::empty())
.unwrap();
let resp = app.oneshot(req).await.unwrap();
@@ -155,9 +156,9 @@ mod tests {
#[tokio::test]
async fn test_query_token_allowed_for_logs_events() {
let app = test_app("secret-token");
let app = test_app(TEST_AUTH_SECRET_TOKEN);
let req = Request::builder()
.uri("/api/logs/events?token=secret-token")
.uri(format!("/api/logs/events?token={TEST_AUTH_SECRET_TOKEN}"))
.body(Body::empty())
.unwrap();
let resp = app.oneshot(req).await.unwrap();
@@ -166,9 +167,9 @@ mod tests {
#[tokio::test]
async fn test_query_token_allowed_for_ws_upgrade() {
let app = test_app("secret-token");
let app = test_app(TEST_AUTH_SECRET_TOKEN);
let req = Request::builder()
.uri("/api/chat/ws?token=secret-token")
.uri(format!("/api/chat/ws?token={TEST_AUTH_SECRET_TOKEN}"))
.body(Body::empty())
.unwrap();
let resp = app.oneshot(req).await.unwrap();
@@ -202,9 +203,9 @@ mod tests {
#[tokio::test]
async fn test_query_token_rejected_for_non_sse_get() {
let app = test_app("secret-token");
let app = test_app(TEST_AUTH_SECRET_TOKEN);
let req = Request::builder()
.uri("/api/chat/history?token=secret-token")
.uri(format!("/api/chat/history?token={TEST_AUTH_SECRET_TOKEN}"))
.body(Body::empty())
.unwrap();
let resp = app.oneshot(req).await.unwrap();
@@ -213,10 +214,10 @@ mod tests {
#[tokio::test]
async fn test_query_token_rejected_for_post() {
let app = test_app("secret-token");
let app = test_app(TEST_AUTH_SECRET_TOKEN);
let req = Request::builder()
.method(Method::POST)
.uri("/api/chat/send?token=secret-token")
.uri(format!("/api/chat/send?token={TEST_AUTH_SECRET_TOKEN}"))
.body(Body::empty())
.unwrap();
let resp = app.oneshot(req).await.unwrap();
@@ -225,7 +226,7 @@ mod tests {
#[tokio::test]
async fn test_query_token_invalid_rejected() {
let app = test_app("secret-token");
let app = test_app(TEST_AUTH_SECRET_TOKEN);
let req = Request::builder()
.uri("/api/chat/events?token=wrong-token")
.body(Body::empty())
@@ -236,7 +237,7 @@ mod tests {
#[tokio::test]
async fn test_no_auth_at_all_rejected() {
let app = test_app("secret-token");
let app = test_app(TEST_AUTH_SECRET_TOKEN);
let req = Request::builder()
.uri("/api/chat/events")
.body(Body::empty())
@@ -247,11 +248,11 @@ mod tests {
#[tokio::test]
async fn test_bearer_header_works_for_post() {
let app = test_app("secret-token");
let app = test_app(TEST_AUTH_SECRET_TOKEN);
let req = Request::builder()
.method(Method::POST)
.uri("/api/chat/send")
.header("Authorization", "Bearer secret-token")
.header("Authorization", format!("Bearer {TEST_AUTH_SECRET_TOKEN}"))
.body(Body::empty())
.unwrap();
let resp = app.oneshot(req).await.unwrap();
@@ -260,10 +261,10 @@ mod tests {
#[tokio::test]
async fn test_bearer_prefix_case_insensitive() {
let app = test_app("secret-token");
let app = test_app(TEST_AUTH_SECRET_TOKEN);
let req = Request::builder()
.uri("/api/chat/events")
.header("Authorization", "bearer secret-token")
.header("Authorization", format!("bearer {TEST_AUTH_SECRET_TOKEN}"))
.body(Body::empty())
.unwrap();
let resp = app.oneshot(req).await.unwrap();
@@ -272,10 +273,10 @@ mod tests {
#[tokio::test]
async fn test_bearer_prefix_mixed_case() {
let app = test_app("secret-token");
let app = test_app(TEST_AUTH_SECRET_TOKEN);
let req = Request::builder()
.uri("/api/chat/events")
.header("Authorization", "BEARER secret-token")
.header("Authorization", format!("BEARER {TEST_AUTH_SECRET_TOKEN}"))
.body(Body::empty())
.unwrap();
let resp = app.oneshot(req).await.unwrap();
@@ -284,7 +285,7 @@ mod tests {
#[tokio::test]
async fn test_empty_bearer_token_rejected() {
let app = test_app("secret-token");
let app = test_app(TEST_AUTH_SECRET_TOKEN);
let req = Request::builder()
.uri("/api/chat/events")
.header("Authorization", "Bearer ")
@@ -296,10 +297,10 @@ mod tests {
#[tokio::test]
async fn test_token_with_whitespace_rejected() {
let app = test_app("secret-token");
let app = test_app(TEST_AUTH_SECRET_TOKEN);
let req = Request::builder()
.uri("/api/chat/events")
.header("Authorization", "Bearer secret-token")
.header("Authorization", format!("Bearer {TEST_AUTH_SECRET_TOKEN}"))
.body(Body::empty())
.unwrap();
let resp = app.oneshot(req).await.unwrap();
+38 -33
View File
@@ -426,7 +426,7 @@ pub async fn chat_threads_handler(
.map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, e.to_string()))?;
if let Ok(summaries) = store
.list_conversations_with_preview(&state.user_id, "gateway", 50)
.list_conversations_all_channels(&state.user_id, 50)
.await
{
let mut assistant_thread = None;
@@ -441,6 +441,7 @@ pub async fn chat_threads_handler(
updated_at: s.last_activity.to_rfc3339(),
title: s.title.clone(),
thread_type: s.thread_type.clone(),
channel: Some(s.channel.clone()),
};
if s.id == assistant_id {
@@ -460,6 +461,7 @@ pub async fn chat_threads_handler(
updated_at: chrono::Utc::now().to_rfc3339(),
title: None,
thread_type: Some("assistant".to_string()),
channel: Some("gateway".to_string()),
});
}
@@ -472,9 +474,10 @@ pub async fn chat_threads_handler(
}
// Fallback: in-memory only (no assistant thread without DB)
let threads: Vec<ThreadInfo> = sess
.threads
.values()
let mut sorted_threads: Vec<_> = sess.threads.values().collect();
sorted_threads.sort_by(|a, b| b.updated_at.cmp(&a.updated_at));
let threads: Vec<ThreadInfo> = sorted_threads
.into_iter()
.map(|t| ThreadInfo {
id: t.id,
state: format!("{:?}", t.state),
@@ -483,6 +486,7 @@ pub async fn chat_threads_handler(
updated_at: t.updated_at.to_rfc3339(),
title: None,
thread_type: None,
channel: Some("gateway".to_string()),
})
.collect();
@@ -502,38 +506,39 @@ pub async fn chat_new_thread_handler(
))?;
let session = session_manager.get_or_create_session(&state.user_id).await;
let mut sess = session.lock().await;
let thread = sess.create_thread();
let thread_id = thread.id;
let info = ThreadInfo {
id: thread.id,
state: format!("{:?}", thread.state),
turn_count: thread.turns.len(),
created_at: thread.created_at.to_rfc3339(),
updated_at: thread.updated_at.to_rfc3339(),
title: None,
thread_type: Some("thread".to_string()),
let (thread_id, info) = {
let mut sess = session.lock().await;
let thread = sess.create_thread();
let id = thread.id;
let info = ThreadInfo {
id: thread.id,
state: format!("{:?}", thread.state),
turn_count: thread.turns.len(),
created_at: thread.created_at.to_rfc3339(),
updated_at: thread.updated_at.to_rfc3339(),
title: None,
thread_type: Some("thread".to_string()),
channel: Some("gateway".to_string()),
};
(id, info)
};
// Persist the empty conversation row with thread_type metadata
// Persist the empty conversation row with thread_type metadata synchronously
// so that the subsequent loadThreads() call from the frontend sees it.
if let Some(ref store) = state.store {
let store = Arc::clone(store);
let user_id = state.user_id.clone();
tokio::spawn(async move {
if let Err(e) = store
.ensure_conversation(thread_id, "gateway", &user_id, None)
.await
{
tracing::warn!("Failed to persist new thread: {}", e);
}
let metadata_val = serde_json::json!("thread");
if let Err(e) = store
.update_conversation_metadata_field(thread_id, "thread_type", &metadata_val)
.await
{
tracing::warn!("Failed to set thread_type metadata: {}", e);
}
});
if let Err(e) = store
.ensure_conversation(thread_id, "gateway", &state.user_id, None)
.await
{
tracing::warn!("Failed to persist new thread: {}", e);
}
let metadata_val = serde_json::json!("thread");
if let Err(e) = store
.update_conversation_metadata_field(thread_id, "thread_type", &metadata_val)
.await
{
tracing::warn!("Failed to set thread_type metadata: {}", e);
}
}
Ok(Json(info))
+15 -1
View File
@@ -276,11 +276,25 @@ pub async fn jobs_cancel_handler(
})));
}
// Fall back to agent job cancellation via DB status update.
// Fall back to agent job cancellation: stop the worker via the scheduler
// (which updates the in-memory ContextManager AND aborts the task handle),
// then persist the status to the DB as a fallback.
if let Some(ref store) = state.store
&& let Ok(Some(job)) = store.get_job(job_id).await
{
if job.state.is_active() {
// Try to stop via scheduler (aborts the worker task + updates
// in-memory ContextManager). This is best-effort — the job may
// not be in the scheduler map if it already finished.
if let Some(ref slot) = state.scheduler
&& let Some(ref scheduler) = *slot.read().await
{
let _ = scheduler.stop(job_id).await;
}
// Always persist cancellation to the DB so the state is
// consistent even if the scheduler wasn't available or the
// job wasn't in its in-memory map.
store
.update_job_status(
job_id,
+26 -43
View File
@@ -10,9 +10,9 @@ use axum::{
use serde::Deserialize;
use uuid::Uuid;
use crate::channels::IncomingMessage;
use crate::channels::web::server::GatewayState;
use crate::channels::web::types::*;
use crate::error::RoutineError;
pub async fn routines_list_handler(
State(state): State<Arc<GatewayState>>,
@@ -108,6 +108,7 @@ pub async fn routines_detail_handler(
status: format!("{:?}", run.status),
result_summary: run.result_summary.clone(),
tokens_used: run.tokens_used,
job_id: run.job_id,
})
.collect();
@@ -133,56 +134,27 @@ pub async fn routines_trigger_handler(
State(state): State<Arc<GatewayState>>,
Path(id): Path<String>,
) -> Result<Json<serde_json::Value>, (StatusCode, String)> {
let store = state.store.as_ref().ok_or((
StatusCode::SERVICE_UNAVAILABLE,
"Database not available".to_string(),
))?;
// Clone the Arc out of the lock to avoid holding the RwLock across .await.
let engine = {
let guard = state.routine_engine.read().await;
guard.as_ref().cloned().ok_or((
StatusCode::SERVICE_UNAVAILABLE,
"Routine engine not available".to_string(),
))?
};
let routine_id = Uuid::parse_str(&id)
.map_err(|_| (StatusCode::BAD_REQUEST, "Invalid routine ID".to_string()))?;
let routine = store
.get_routine(routine_id)
let run_id = engine
.fire_manual(routine_id, Some(&state.user_id))
.await
.map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, e.to_string()))?
.ok_or((StatusCode::NOT_FOUND, "Routine not found".to_string()))?;
if routine.user_id != state.user_id {
return Err((StatusCode::FORBIDDEN, "Access denied".to_string()));
}
// Send the routine prompt through the message pipeline as a manual trigger.
let prompt = match &routine.action {
crate::agent::routine::RoutineAction::Lightweight { prompt, .. } => prompt.clone(),
crate::agent::routine::RoutineAction::FullJob {
title, description, ..
} => format!("{}: {}", title, description),
};
let content = format!("[routine:{}] {}", routine.name, prompt);
let thread_id = format!(
"routine-{}-{}",
routine_id,
chrono::Utc::now().timestamp_millis()
);
let msg = IncomingMessage::new("gateway", &state.user_id, content).with_thread(thread_id);
let tx_guard = state.msg_tx.read().await;
let tx = tx_guard.as_ref().ok_or((
StatusCode::SERVICE_UNAVAILABLE,
"Channel not started".to_string(),
))?;
tx.send(msg).await.map_err(|_| {
(
StatusCode::INTERNAL_SERVER_ERROR,
"Channel closed".to_string(),
)
})?;
.map_err(|e| (routine_error_status(&e), e.to_string()))?;
Ok(Json(serde_json::json!({
"status": "triggered",
"routine_id": routine_id,
"run_id": run_id,
})))
}
@@ -281,6 +253,7 @@ pub async fn routines_runs_handler(
status: format!("{:?}", run.status),
result_summary: run.result_summary.clone(),
tokens_used: run.tokens_used,
job_id: run.job_id,
})
.collect();
@@ -293,7 +266,7 @@ pub async fn routines_runs_handler(
/// Convert a Routine to the trimmed RoutineInfo for list display.
fn routine_to_info(r: &crate::agent::routine::Routine) -> RoutineInfo {
let (trigger_type, trigger_summary) = match &r.trigger {
crate::agent::routine::Trigger::Cron { schedule } => {
crate::agent::routine::Trigger::Cron { schedule, .. } => {
("cron".to_string(), format!("cron: {}", schedule))
}
crate::agent::routine::Trigger::Event {
@@ -337,3 +310,13 @@ fn routine_to_info(r: &crate::agent::routine::Routine) -> RoutineInfo {
status: status.to_string(),
}
}
/// Map `RoutineError` variants to appropriate HTTP status codes.
fn routine_error_status(err: &RoutineError) -> StatusCode {
match err {
RoutineError::NotFound { .. } => StatusCode::NOT_FOUND,
RoutineError::NotAuthorized { .. } => StatusCode::FORBIDDEN,
RoutineError::Disabled { .. } | RoutineError::MaxConcurrent { .. } => StatusCode::CONFLICT,
_ => StatusCode::INTERNAL_SERVER_ERROR,
}
}
+26 -2
View File
@@ -99,6 +99,7 @@ impl GatewayChannel {
chat_rate_limiter: server::RateLimiter::new(30, 60),
registry_entries: Vec::new(),
cost_guard: None,
routine_engine: Arc::new(tokio::sync::RwLock::new(None)),
startup_time: std::time::Instant::now(),
});
@@ -134,6 +135,7 @@ impl GatewayChannel {
chat_rate_limiter: server::RateLimiter::new(30, 60),
registry_entries: self.state.registry_entries.clone(),
cost_guard: self.state.cost_guard.clone(),
routine_engine: Arc::clone(&self.state.routine_engine),
startup_time: self.state.startup_time,
};
mutate(&mut new_state);
@@ -281,7 +283,15 @@ impl Channel for GatewayChannel {
msg: &IncomingMessage,
response: OutgoingResponse,
) -> Result<(), ChannelError> {
let thread_id = msg.thread_id.clone().unwrap_or_default();
let thread_id = match &msg.thread_id {
Some(tid) => tid.clone(),
None => {
tracing::warn!(
"Gateway respond with no thread_id — skipping (clients would drop it)"
);
return Ok(());
}
};
self.state.sse.broadcast(SseEvent::Response {
content: response.content,
@@ -376,6 +386,11 @@ impl Channel for GatewayChannel {
success,
message,
},
StatusUpdate::ImageGenerated { data_url, path } => SseEvent::ImageGenerated {
data_url,
path,
thread_id,
},
};
self.state.sse.broadcast(event);
@@ -387,9 +402,18 @@ impl Channel for GatewayChannel {
_user_id: &str,
response: OutgoingResponse,
) -> Result<(), ChannelError> {
let thread_id = match response.thread_id {
Some(tid) => tid,
None => {
tracing::warn!(
"Gateway broadcast with no thread_id — skipping (clients would drop it)"
);
return Ok(());
}
};
self.state.sse.broadcast(SseEvent::Response {
content: response.content,
thread_id: String::new(),
thread_id,
});
Ok(())
}
+151 -75
View File
@@ -57,6 +57,10 @@ pub type PromptQueue = Arc<
>,
>;
/// Slot for the routine engine, filled at runtime after the agent starts.
pub type RoutineEngineSlot =
Arc<tokio::sync::RwLock<Option<Arc<crate::agent::routine_engine::RoutineEngine>>>>;
/// Simple sliding-window rate limiter.
///
/// Tracks the number of requests in the current window. Resets when the window expires.
@@ -165,6 +169,8 @@ pub struct GatewayState {
pub registry_entries: Vec<crate::extensions::RegistryEntry>,
/// Cost guard for token/cost tracking.
pub cost_guard: Option<Arc<crate::agent::cost_guard::CostGuard>>,
/// Routine engine slot for manual routine triggering (filled at runtime).
pub routine_engine: RoutineEngineSlot,
/// Server startup time for uptime calculation.
pub startup_time: std::time::Instant,
}
@@ -345,7 +351,7 @@ pub async fn start_server(
.merge(statics)
.merge(projects)
.merge(protected)
.layer(DefaultBodyLimit::max(1024 * 1024)) // 1 MB max request body
.layer(DefaultBodyLimit::max(10 * 1024 * 1024)) // 10 MB max request body (image uploads)
.layer(cors)
.layer(SetResponseHeaderLayer::if_not_present(
header::X_CONTENT_TYPE_OPTIONS,
@@ -364,7 +370,7 @@ pub async fn start_server(
if let Err(e) = axum::serve(listener, app)
.with_graceful_shutdown(async {
let _ = shutdown_rx.await;
tracing::info!("Web gateway shutting down");
tracing::debug!("Web gateway shutting down");
})
.await
{
@@ -602,8 +608,59 @@ async fn oauth_callback_handler(
// --- Chat handlers ---
/// Convert web gateway `ImageData` to `IncomingAttachment` objects.
pub(crate) fn images_to_attachments(
images: &[ImageData],
) -> Vec<crate::channels::IncomingAttachment> {
use base64::Engine;
images
.iter()
.enumerate()
.filter_map(|(i, img)| {
if !img.media_type.starts_with("image/") {
tracing::warn!(
"Skipping image {i}: invalid media type '{}' (must start with 'image/')",
img.media_type
);
return None;
}
let data = match base64::engine::general_purpose::STANDARD.decode(&img.data) {
Ok(d) => d,
Err(e) => {
tracing::warn!("Skipping image {i}: invalid base64 data: {e}");
return None;
}
};
Some(crate::channels::IncomingAttachment {
id: format!("web-image-{i}"),
kind: crate::channels::AttachmentKind::Image,
mime_type: img.media_type.clone(),
filename: Some(format!("image-{i}.{}", mime_to_ext(&img.media_type))),
size_bytes: Some(data.len() as u64),
source_url: None,
storage_key: None,
extracted_text: None,
data,
duration_secs: None,
})
})
.collect()
}
/// Map MIME type to file extension.
fn mime_to_ext(mime: &str) -> &str {
match mime {
"image/png" => "png",
"image/gif" => "gif",
"image/webp" => "webp",
"image/svg+xml" => "svg",
_ => "jpg",
}
}
async fn chat_send_handler(
State(state): State<Arc<GatewayState>>,
headers: axum::http::HeaderMap,
Json(req): Json<SendMessageRequest>,
) -> Result<(StatusCode, Json<SendMessageResponse>), (StatusCode, String)> {
tracing::debug!(
@@ -620,17 +677,32 @@ async fn chat_send_handler(
}
let mut msg = IncomingMessage::new("gateway", &state.user_id, &req.content);
// Prefer timezone from JSON body, fall back to X-Timezone header
let tz = req
.timezone
.as_deref()
.or_else(|| headers.get("X-Timezone").and_then(|v| v.to_str().ok()));
if let Some(tz) = tz {
msg = msg.with_timezone(tz);
}
if let Some(ref thread_id) = req.thread_id {
msg = msg.with_thread(thread_id);
msg = msg.with_metadata(serde_json::json!({"thread_id": thread_id}));
}
// Convert uploaded images to IncomingAttachments
if !req.images.is_empty() {
let attachments = images_to_attachments(&req.images);
msg = msg.with_attachments(attachments);
}
let msg_id = msg.id;
tracing::debug!(
"[chat_send_handler] Created message id={}, content={:?}",
"[chat_send_handler] Created message id={}, content={:?}, images={}",
msg_id,
req.content
req.content,
req.images.len()
);
let tx_guard = state.msg_tx.read().await;
@@ -1037,7 +1109,7 @@ async fn chat_threads_handler(
.map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, e.to_string()))?;
if let Ok(summaries) = store
.list_conversations_with_preview(&state.user_id, "gateway", 50)
.list_conversations_all_channels(&state.user_id, 50)
.await
{
let mut assistant_thread = None;
@@ -1052,6 +1124,7 @@ async fn chat_threads_handler(
updated_at: s.last_activity.to_rfc3339(),
title: s.title.clone(),
thread_type: s.thread_type.clone(),
channel: Some(s.channel.clone()),
};
if s.id == assistant_id {
@@ -1071,6 +1144,7 @@ async fn chat_threads_handler(
updated_at: chrono::Utc::now().to_rfc3339(),
title: None,
thread_type: Some("assistant".to_string()),
channel: Some("gateway".to_string()),
});
}
@@ -1083,9 +1157,10 @@ async fn chat_threads_handler(
}
// Fallback: in-memory only (no assistant thread without DB)
let threads: Vec<ThreadInfo> = sess
.threads
.values()
let mut sorted_threads: Vec<_> = sess.threads.values().collect();
sorted_threads.sort_by(|a, b| b.updated_at.cmp(&a.updated_at));
let threads: Vec<ThreadInfo> = sorted_threads
.into_iter()
.map(|t| ThreadInfo {
id: t.id,
state: format!("{:?}", t.state),
@@ -1094,6 +1169,7 @@ async fn chat_threads_handler(
updated_at: t.updated_at.to_rfc3339(),
title: None,
thread_type: None,
channel: Some("gateway".to_string()),
})
.collect();
@@ -1113,38 +1189,39 @@ async fn chat_new_thread_handler(
))?;
let session = session_manager.get_or_create_session(&state.user_id).await;
let mut sess = session.lock().await;
let thread = sess.create_thread();
let thread_id = thread.id;
let info = ThreadInfo {
id: thread.id,
state: format!("{:?}", thread.state),
turn_count: thread.turns.len(),
created_at: thread.created_at.to_rfc3339(),
updated_at: thread.updated_at.to_rfc3339(),
title: None,
thread_type: Some("thread".to_string()),
let (thread_id, info) = {
let mut sess = session.lock().await;
let thread = sess.create_thread();
let id = thread.id;
let info = ThreadInfo {
id: thread.id,
state: format!("{:?}", thread.state),
turn_count: thread.turns.len(),
created_at: thread.created_at.to_rfc3339(),
updated_at: thread.updated_at.to_rfc3339(),
title: None,
thread_type: Some("thread".to_string()),
channel: Some("gateway".to_string()),
};
(id, info)
};
// Persist the empty conversation row with thread_type metadata
// Persist the empty conversation row with thread_type metadata synchronously
// so that the subsequent loadThreads() call from the frontend sees it.
if let Some(ref store) = state.store {
let store = Arc::clone(store);
let user_id = state.user_id.clone();
tokio::spawn(async move {
if let Err(e) = store
.ensure_conversation(thread_id, "gateway", &user_id, None)
.await
{
tracing::warn!("Failed to persist new thread: {}", e);
}
let metadata_val = serde_json::json!("thread");
if let Err(e) = store
.update_conversation_metadata_field(thread_id, "thread_type", &metadata_val)
.await
{
tracing::warn!("Failed to set thread_type metadata: {}", e);
}
});
if let Err(e) = store
.ensure_conversation(thread_id, "gateway", &state.user_id, None)
.await
{
tracing::warn!("Failed to persist new thread: {}", e);
}
let metadata_val = serde_json::json!("thread");
if let Err(e) = store
.update_conversation_metadata_field(thread_id, "thread_type", &metadata_val)
.await
{
tracing::warn!("Failed to set thread_type metadata: {}", e);
}
}
Ok(Json(info))
@@ -1940,6 +2017,7 @@ async fn routines_detail_handler(
status: format!("{:?}", run.status),
result_summary: run.result_summary.clone(),
tokens_used: run.tokens_used,
job_id: run.job_id,
})
.collect();
@@ -1965,47 +2043,35 @@ async fn routines_trigger_handler(
State(state): State<Arc<GatewayState>>,
Path(id): Path<String>,
) -> Result<Json<serde_json::Value>, (StatusCode, String)> {
let store = state.store.as_ref().ok_or((
StatusCode::SERVICE_UNAVAILABLE,
"Database not available".to_string(),
))?;
let engine = {
let guard = state.routine_engine.read().await;
guard.as_ref().cloned().ok_or((
StatusCode::SERVICE_UNAVAILABLE,
"Routine engine not available".to_string(),
))?
};
let routine_id = Uuid::parse_str(&id)
.map_err(|_| (StatusCode::BAD_REQUEST, "Invalid routine ID".to_string()))?;
let routine = store
.get_routine(routine_id)
let run_id = engine
.fire_manual(routine_id, Some(&state.user_id))
.await
.map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, e.to_string()))?
.ok_or((StatusCode::NOT_FOUND, "Routine not found".to_string()))?;
// Send the routine prompt through the message pipeline as a manual trigger.
let prompt = match &routine.action {
crate::agent::routine::RoutineAction::Lightweight { prompt, .. } => prompt.clone(),
crate::agent::routine::RoutineAction::FullJob {
title, description, ..
} => format!("{}: {}", title, description),
};
let content = format!("[routine:{}] {}", routine.name, prompt);
let msg = IncomingMessage::new("gateway", &state.user_id, content);
let tx_guard = state.msg_tx.read().await;
let tx = tx_guard.as_ref().ok_or((
StatusCode::SERVICE_UNAVAILABLE,
"Channel not started".to_string(),
))?;
tx.send(msg).await.map_err(|_| {
(
StatusCode::INTERNAL_SERVER_ERROR,
"Channel closed".to_string(),
)
})?;
.map_err(|e| {
let status = match &e {
crate::error::RoutineError::NotFound { .. } => StatusCode::NOT_FOUND,
crate::error::RoutineError::NotAuthorized { .. } => StatusCode::FORBIDDEN,
crate::error::RoutineError::Disabled { .. }
| crate::error::RoutineError::MaxConcurrent { .. } => StatusCode::CONFLICT,
_ => StatusCode::INTERNAL_SERVER_ERROR,
};
(status, e.to_string())
})?;
Ok(Json(serde_json::json!({
"status": "triggered",
"routine_id": routine_id,
"run_id": run_id,
})))
}
@@ -2104,6 +2170,7 @@ async fn routines_runs_handler(
status: format!("{:?}", run.status),
result_summary: run.result_summary.clone(),
tokens_used: run.tokens_used,
job_id: run.job_id,
})
.collect();
@@ -2116,7 +2183,7 @@ async fn routines_runs_handler(
/// Convert a Routine to the trimmed RoutineInfo for list display.
fn routine_to_info(r: &crate::agent::routine::Routine) -> RoutineInfo {
let (trigger_type, trigger_summary) = match &r.trigger {
crate::agent::routine::Trigger::Cron { schedule } => {
crate::agent::routine::Trigger::Cron { schedule, .. } => {
("cron".to_string(), format!("cron: {}", schedule))
}
crate::agent::routine::Trigger::Event {
@@ -2360,6 +2427,7 @@ struct GatewayStatusResponse {
#[cfg(test)]
mod tests {
use super::*;
use crate::testing::credentials::TEST_GATEWAY_CRYPTO_KEY;
#[test]
fn test_build_turns_from_db_messages_complete() {
@@ -2463,6 +2531,7 @@ mod tests {
chat_rate_limiter: RateLimiter::new(30, 60),
registry_entries: vec![],
cost_guard: None,
routine_engine: Arc::new(tokio::sync::RwLock::new(None)),
startup_time: std::time::Instant::now(),
})
}
@@ -2532,7 +2601,7 @@ mod tests {
// Build an ExtensionManager so the handler can look up flows
let secrets = Arc::new(crate::secrets::InMemorySecretsStore::new(Arc::new(
crate::secrets::SecretsCrypto::new(secrecy::SecretString::from(
"test-key-at-least-32-chars-long!!".to_string(),
TEST_GATEWAY_CRYPTO_KEY.to_string(),
))
.expect("crypto"),
)));
@@ -2541,6 +2610,7 @@ mod tests {
let ext_mgr = Arc::new(ExtensionManager::new(
mcp_sm,
Arc::new(crate::tools::mcp::process::McpProcessManager::new()),
secrets,
tool_registry,
None,
@@ -2581,7 +2651,7 @@ mod tests {
let secrets: Arc<dyn crate::secrets::SecretsStore + Send + Sync> =
Arc::new(crate::secrets::InMemorySecretsStore::new(Arc::new(
crate::secrets::SecretsCrypto::new(secrecy::SecretString::from(
"test-key-at-least-32-chars-long!!".to_string(),
TEST_GATEWAY_CRYPTO_KEY.to_string(),
))
.expect("crypto"),
)));
@@ -2590,6 +2660,7 @@ mod tests {
let ext_mgr = Arc::new(ExtensionManager::new(
mcp_sm,
Arc::new(crate::tools::mcp::process::McpProcessManager::new()),
secrets.clone(),
tool_registry,
None,
@@ -2620,7 +2691,9 @@ mod tests {
secrets,
sse_sender: None,
gateway_token: None,
created_at: std::time::Instant::now() - std::time::Duration::from_secs(600),
created_at: std::time::Instant::now()
.checked_sub(std::time::Duration::from_secs(600))
.expect("System uptime is too low to run expired flow test"),
};
ext_mgr
@@ -2684,7 +2757,7 @@ mod tests {
let secrets: Arc<dyn crate::secrets::SecretsStore + Send + Sync> =
Arc::new(crate::secrets::InMemorySecretsStore::new(Arc::new(
crate::secrets::SecretsCrypto::new(secrecy::SecretString::from(
"test-key-at-least-32-chars-long!!".to_string(),
TEST_GATEWAY_CRYPTO_KEY.to_string(),
))
.expect("crypto"),
)));
@@ -2693,6 +2766,7 @@ mod tests {
let ext_mgr = Arc::new(ExtensionManager::new(
mcp_sm,
Arc::new(crate::tools::mcp::process::McpProcessManager::new()),
secrets.clone(),
tool_registry,
None,
@@ -2727,7 +2801,9 @@ mod tests {
sse_sender: None,
gateway_token: None,
// Expired — handler will reject after lookup (no network I/O)
created_at: std::time::Instant::now() - std::time::Duration::from_secs(600),
created_at: std::time::Instant::now()
.checked_sub(std::time::Duration::from_secs(600))
.expect("System uptime is too low to run expired flow test"),
};
ext_mgr
+1
View File
@@ -142,6 +142,7 @@ impl SseManager {
SseEvent::JobStatus { .. } => "job_status",
SseEvent::JobResult { .. } => "job_result",
SseEvent::Heartbeat => "heartbeat",
SseEvent::ImageGenerated { .. } => "image_generated",
SseEvent::ExtensionStatus { .. } => "extension_status",
};
Ok(Event::default().event(event_type).data(data))
+270 -49
View File
@@ -5,6 +5,7 @@ let eventSource = null;
let logEventSource = null;
let currentTab = 'chat';
let currentThreadId = null;
let currentThreadIsReadOnly = false;
let assistantThreadId = null;
let hasMore = false;
let oldestTimestamp = null;
@@ -13,8 +14,11 @@ let sseHasConnectedBefore = false;
let jobEvents = new Map(); // job_id -> Array of events
let jobListRefreshTimer = null;
let pairingPollInterval = null;
let unreadThreads = new Map(); // thread_id -> unread count
let _loadThreadsTimer = null;
const JOB_EVENTS_CAP = 500;
const MEMORY_SEARCH_QUERY_MAX_LENGTH = 100;
let stagedImages = [];
// --- Slash Commands ---
@@ -178,6 +182,7 @@ function confirmRestart() {
body: {
content: '/restart',
thread_id: currentThreadId,
timezone: Intl.DateTimeFormat().resolvedOptions().timeZone,
},
})
.then((response) => {
@@ -220,23 +225,6 @@ function updateRestartButtonVisibility() {
}
}
function startGatewayStatusPolling() {
fetchGatewayStatus();
// Poll every 5 seconds
setInterval(fetchGatewayStatus, 5000);
}
function fetchGatewayStatus() {
apiFetch('/api/gateway/status')
.then((data) => {
restartEnabled = data.restart_enabled || false;
updateRestartButtonVisibility();
})
.catch((err) => {
console.warn('[gateway status] Failed to fetch:', err);
});
}
// --- SSE ---
function connectSSE() {
@@ -273,7 +261,13 @@ function connectSSE() {
eventSource.addEventListener('response', (e) => {
const data = JSON.parse(e.data);
if (!isCurrentThread(data.thread_id)) return;
if (!isCurrentThread(data.thread_id)) {
if (data.thread_id) {
unreadThreads.set(data.thread_id, (unreadThreads.get(data.thread_id) || 0) + 1);
debouncedLoadThreads();
}
return;
}
finalizeActivityGroup();
addMessage('assistant', data.content);
enableChatInput();
@@ -288,7 +282,10 @@ function connectSSE() {
eventSource.addEventListener('thinking', (e) => {
const data = JSON.parse(e.data);
if (!isCurrentThread(data.thread_id)) return;
if (!isCurrentThread(data.thread_id)) {
if (data.thread_id) debouncedLoadThreads();
return;
}
showActivityThinking(data.message);
});
@@ -324,7 +321,10 @@ function connectSSE() {
eventSource.addEventListener('status', (e) => {
const data = JSON.parse(e.data);
if (!isCurrentThread(data.thread_id)) return;
if (!isCurrentThread(data.thread_id)) {
if (data.thread_id) debouncedLoadThreads();
return;
}
// "Done" and "Awaiting approval" are terminal signals from the agent:
// the agentic loop finished, so re-enable input as a safety net in case
// the response SSE event is empty or lost.
@@ -373,6 +373,12 @@ function connectSSE() {
if (currentTab === 'extensions') loadExtensions();
});
eventSource.addEventListener('image_generated', (e) => {
const data = JSON.parse(e.data);
if (!isCurrentThread(data.thread_id)) return;
addGeneratedImage(data.data_url, data.path);
});
eventSource.addEventListener('error', (e) => {
if (e.data) {
const data = JSON.parse(e.data);
@@ -414,9 +420,9 @@ function connectSSE() {
}
// Check if an SSE event belongs to the currently viewed thread.
// Events without a thread_id (legacy) are always shown.
// Events without a thread_id are dropped (prevents notification leaking).
function isCurrentThread(threadId) {
if (!threadId) return true;
if (!threadId) return false;
if (!currentThreadId) return true;
return threadId === currentThreadId;
}
@@ -430,23 +436,135 @@ function sendMessage() {
return;
}
const content = input.value.trim();
if (!content) return;
if (!content && stagedImages.length === 0) return;
addMessage('user', content);
addMessage('user', content || '(images attached)');
input.value = '';
autoResizeTextarea(input);
input.focus();
const body = { content, thread_id: currentThreadId || undefined, timezone: Intl.DateTimeFormat().resolvedOptions().timeZone };
if (stagedImages.length > 0) {
body.images = stagedImages.map(img => ({ media_type: img.media_type, data: img.data }));
stagedImages = [];
renderImagePreviews();
}
apiFetch('/api/chat/send', {
method: 'POST',
body: { content, thread_id: currentThreadId || undefined },
body: body,
}).catch((err) => {
addMessage('system', 'Failed to send: ' + err.message);
});
}
function enableChatInput() {
// no-op: input and send button are always enabled
if (currentThreadIsReadOnly) return;
const input = document.getElementById('chat-input');
const btn = document.getElementById('send-btn');
if (input) {
input.disabled = false;
input.placeholder = 'Message or / for commands...';
}
if (btn) btn.disabled = false;
}
// --- Image Upload ---
function renderImagePreviews() {
const strip = document.getElementById('image-preview-strip');
strip.innerHTML = '';
stagedImages.forEach((img, idx) => {
const container = document.createElement('div');
container.className = 'image-preview-container';
const preview = document.createElement('img');
preview.className = 'image-preview';
preview.src = img.dataUrl;
preview.alt = 'Attached image';
const removeBtn = document.createElement('button');
removeBtn.className = 'image-preview-remove';
removeBtn.textContent = '\u00d7';
removeBtn.addEventListener('click', () => {
stagedImages.splice(idx, 1);
renderImagePreviews();
});
container.appendChild(preview);
container.appendChild(removeBtn);
strip.appendChild(container);
});
}
const MAX_IMAGE_SIZE_BYTES = 5 * 1024 * 1024; // 5 MB per image
const MAX_STAGED_IMAGES = 5;
function handleImageFiles(files) {
Array.from(files).forEach(file => {
if (!file.type.startsWith('image/')) return;
if (file.size > MAX_IMAGE_SIZE_BYTES) {
alert(`Image "${file.name}" exceeds 5 MB limit (${(file.size / 1024 / 1024).toFixed(1)} MB)`);
return;
}
if (stagedImages.length >= MAX_STAGED_IMAGES) {
alert(`Maximum ${MAX_STAGED_IMAGES} images allowed per message`);
return;
}
const reader = new FileReader();
reader.onload = function(e) {
const dataUrl = e.target.result;
const commaIdx = dataUrl.indexOf(',');
const meta = dataUrl.substring(0, commaIdx); // e.g. "data:image/png;base64"
const base64 = dataUrl.substring(commaIdx + 1);
const mediaType = meta.replace('data:', '').replace(';base64', '');
stagedImages.push({ media_type: mediaType, data: base64, dataUrl: dataUrl });
renderImagePreviews();
};
reader.readAsDataURL(file);
});
}
document.getElementById('attach-btn').addEventListener('click', () => {
document.getElementById('image-file-input').click();
});
document.getElementById('image-file-input').addEventListener('change', (e) => {
handleImageFiles(e.target.files);
e.target.value = '';
});
document.getElementById('chat-input').addEventListener('paste', (e) => {
const items = (e.clipboardData || e.originalEvent.clipboardData).items;
for (let i = 0; i < items.length; i++) {
if (items[i].kind === 'file' && items[i].type.startsWith('image/')) {
const file = items[i].getAsFile();
if (file) handleImageFiles([file]);
}
}
});
function addGeneratedImage(dataUrl, path) {
const container = document.getElementById('chat-messages');
const card = document.createElement('div');
card.className = 'generated-image-card';
const img = document.createElement('img');
img.className = 'generated-image';
img.src = dataUrl;
img.alt = 'Generated image';
card.appendChild(img);
if (path) {
const pathLabel = document.createElement('div');
pathLabel.className = 'generated-image-path';
pathLabel.textContent = path;
card.appendChild(pathLabel);
}
container.appendChild(card);
container.scrollTop = container.scrollHeight;
}
// --- Slash Autocomplete ---
@@ -541,6 +659,13 @@ function sendApprovalAction(requestId, action) {
function renderMarkdown(text) {
if (typeof marked !== 'undefined') {
// Escape raw HTML error pages instead of rendering them as markup.
// Only triggers when the text *starts with* a doctype or <html> tag
// (after optional whitespace), so normal messages that mention HTML
// tags in prose or code fences are not affected. See #263.
if (/^\s*<!doctype\s/i.test(text) || /^\s*<html[\s>]/i.test(text)) {
return escapeHtml(text);
}
let html = marked.parse(text);
// Sanitize HTML output to prevent XSS from tool output or LLM responses.
html = sanitizeRenderedHtml(html);
@@ -1134,7 +1259,9 @@ function loadHistory(before) {
// Fresh load: clear and render
container.innerHTML = '';
for (const turn of data.turns) {
addMessage('user', turn.user_input);
if (turn.user_input) {
addMessage('user', turn.user_input);
}
if (turn.tool_calls && turn.tool_calls.length > 0) {
addToolCallsSummary(turn.tool_calls);
}
@@ -1156,8 +1283,10 @@ function loadHistory(before) {
const savedHeight = container.scrollHeight;
const fragment = document.createDocumentFragment();
for (const turn of data.turns) {
const userDiv = createMessageElement('user', turn.user_input);
fragment.appendChild(userDiv);
if (turn.user_input) {
const userDiv = createMessageElement('user', turn.user_input);
fragment.appendChild(userDiv);
}
if (turn.tool_calls && turn.tool_calls.length > 0) {
fragment.appendChild(createToolCallsSummaryElement(turn.tool_calls));
}
@@ -1256,6 +1385,37 @@ function removeScrollSpinner() {
// --- Threads ---
function threadTitle(thread) {
if (thread.title) return thread.title;
const ch = thread.channel || 'gateway';
if (thread.thread_type === 'heartbeat') return 'Heartbeat Alerts';
if (thread.thread_type === 'routine') return 'Routine';
if (ch !== 'gateway') return ch.charAt(0).toUpperCase() + ch.slice(1);
if (thread.turn_count === 0) return 'New chat';
return thread.id.substring(0, 8);
}
function relativeTime(isoStr) {
if (!isoStr) return '';
const diff = Date.now() - new Date(isoStr).getTime();
const mins = Math.floor(diff / 60000);
if (mins < 1) return 'now';
if (mins < 60) return mins + 'm ago';
const hrs = Math.floor(mins / 60);
if (hrs < 24) return hrs + 'h ago';
const days = Math.floor(hrs / 24);
return days + 'd ago';
}
function isReadOnlyChannel(channel) {
return channel && channel !== 'gateway' && channel !== 'routine' && channel !== 'heartbeat';
}
function debouncedLoadThreads() {
if (_loadThreadsTimer) clearTimeout(_loadThreadsTimer);
_loadThreadsTimer = setTimeout(() => { _loadThreadsTimer = null; loadThreads(); }, 500);
}
function loadThreads() {
apiFetch('/api/chat/threads').then((data) => {
// Pinned assistant thread
@@ -1264,9 +1424,13 @@ function loadThreads() {
const el = document.getElementById('assistant-thread');
const isActive = currentThreadId === assistantThreadId;
el.className = 'assistant-item' + (isActive ? ' active' : '');
const labelEl = document.getElementById('assistant-label');
if (labelEl) {
const at = data.assistant_thread;
labelEl.textContent = 'Assistant';
}
const meta = document.getElementById('assistant-meta');
const count = data.assistant_thread.turn_count || 0;
meta.textContent = count > 0 ? count + ' turns' : '';
meta.textContent = relativeTime(data.assistant_thread.updated_at);
}
// Regular threads
@@ -1275,16 +1439,38 @@ function loadThreads() {
const threads = data.threads || [];
for (const thread of threads) {
const item = document.createElement('div');
item.className = 'thread-item' + (thread.id === currentThreadId ? ' active' : '');
const isActive = thread.id === currentThreadId;
item.className = 'thread-item' + (isActive ? ' active' : '');
// Channel badge for non-gateway threads
const ch = thread.channel || 'gateway';
if (ch !== 'gateway') {
const badge = document.createElement('span');
badge.className = 'thread-badge thread-badge-' + ch;
badge.textContent = ch;
item.appendChild(badge);
}
const label = document.createElement('span');
label.className = 'thread-label';
label.textContent = thread.title || thread.id.substring(0, 8);
label.title = thread.title ? thread.title + ' (' + thread.id + ')' : thread.id;
label.textContent = threadTitle(thread);
label.title = (thread.title || '') + ' (' + thread.id + ')';
item.appendChild(label);
const meta = document.createElement('span');
meta.className = 'thread-meta';
meta.textContent = (thread.turn_count || 0) + ' turns';
meta.textContent = relativeTime(thread.updated_at);
item.appendChild(meta);
// Unread dot
const unread = unreadThreads.get(thread.id) || 0;
if (unread > 0 && !isActive) {
const dot = document.createElement('span');
dot.className = 'thread-unread';
dot.textContent = unread > 9 ? '9+' : String(unread);
item.appendChild(dot);
}
item.addEventListener('click', () => switchThread(thread.id));
list.appendChild(item);
}
@@ -1294,17 +1480,36 @@ function loadThreads() {
switchToAssistant();
}
// Enable chat input once a thread is available
// Enable/disable chat input based on channel type
if (currentThreadId) {
enableChatInput();
const currentThread = threads.find(t => t.id === currentThreadId);
const ch = currentThread ? currentThread.channel : 'gateway';
currentThreadIsReadOnly = isReadOnlyChannel(ch);
if (currentThreadIsReadOnly) {
disableChatInputReadOnly();
} else {
enableChatInput();
}
}
}).catch(() => {});
}
function disableChatInputReadOnly() {
const input = document.getElementById('chat-input');
const btn = document.getElementById('send-btn');
if (input) {
input.disabled = true;
input.placeholder = 'Read-only thread (external channel)';
}
if (btn) btn.disabled = true;
}
function switchToAssistant() {
if (!assistantThreadId) return;
finalizeActivityGroup();
currentThreadId = assistantThreadId;
currentThreadIsReadOnly = false;
unreadThreads.delete(assistantThreadId);
hasMore = false;
oldestTimestamp = null;
loadHistory();
@@ -1314,6 +1519,7 @@ function switchToAssistant() {
function switchThread(threadId) {
finalizeActivityGroup();
currentThreadId = threadId;
unreadThreads.delete(threadId);
hasMore = false;
oldestTimestamp = null;
loadHistory();
@@ -1370,7 +1576,7 @@ chatInput.addEventListener('keydown', (e) => {
}
}
if (e.key === 'Enter' && !e.shiftKey) {
if (e.key === 'Enter' && !e.shiftKey && !e.isComposing) {
e.preventDefault();
hideSlashAutocomplete();
sendMessage();
@@ -3305,6 +3511,10 @@ function shortModelName(model) {
function fetchGatewayStatus() {
apiFetch('/api/gateway/status').then(function(data) {
// Update restart button visibility
restartEnabled = data.restart_enabled || false;
updateRestartButtonVisibility();
var popover = document.getElementById('gateway-popover');
var html = '';
@@ -3370,10 +3580,15 @@ let teeReportCache = null;
let teeReportLoading = false;
function teeApiBase() {
var parts = window.location.hostname.split('.');
if (parts.length < 2) return null;
var domain = parts.slice(1).join('.');
return window.location.protocol + '//api.' + domain;
var hostname = window.location.hostname;
// Skip IP addresses (IPv4 and IPv6) and localhost
if (hostname === "localhost" || /^(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)$/.test(hostname) || hostname.indexOf(":") !== -1) {
return null;
}
var parts = hostname.split(".");
if (parts.length < 2) return null;
var domain = parts.slice(1).join(".");
return window.location.protocol + "//api." + domain;
}
function teeInstanceName() {
@@ -3384,13 +3599,19 @@ function checkTeeStatus() {
var base = teeApiBase();
if (!base) return;
var name = teeInstanceName();
fetch(base + '/instances/' + encodeURIComponent(name) + '/attestation').then(function(res) {
if (!res.ok) throw new Error(res.status);
return res.json();
}).then(function(data) {
teeInfo = data;
document.getElementById('tee-shield').style.display = 'flex';
}).catch(function() {});
try {
fetch(base + '/instances/' + encodeURIComponent(name) + '/attestation').then(function(res) {
if (!res.ok) throw new Error(res.status);
return res.json();
}).then(function(data) {
teeInfo = data;
document.getElementById('tee-shield').style.display = 'flex';
}).catch(function(err) {
console.warn('Failed to fetch TEE attestation:', err);
});
} catch (e) {
console.warn("Failed to check TEE status:", e);
}
}
function fetchTeeReport() {
+6 -3
View File
@@ -99,7 +99,7 @@
<span id="sse-status">Connected</span>
<div class="gateway-popover" id="gateway-popover"></div>
</div>
<button class="restart-btn" id="restart-btn" onclick="triggerRestart()" title="Gracefully restart the process">
<button class="restart-btn" id="restart-btn" onclick="triggerRestart()" title="Gracefully restart the process" style="display: none;">
<svg id="restart-icon" width="13" height="13" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">
<path d="M23 4v6h-6"></path>
<path d="M1 20v-6h6"></path>
@@ -113,12 +113,12 @@
<div class="tab-panel active" id="tab-chat">
<div class="thread-sidebar" id="thread-sidebar">
<div class="thread-sidebar-header">
<span>Threads</span>
<button class="thread-new-btn" onclick="createNewThread()" title="New thread (Ctrl/Cmd+N)">+</button>
<div class="spacer"></div>
<button class="thread-toggle-btn" id="thread-toggle-btn" onclick="toggleThreadSidebar()" title="Toggle sidebar">&laquo;</button>
</div>
<div class="assistant-item" id="assistant-thread" onclick="switchToAssistant()">
<span class="assistant-label">Assistant</span>
<span class="assistant-label" id="assistant-label">Assistant</span>
<span class="assistant-meta" id="assistant-meta"></span>
</div>
<div class="threads-section-header">
@@ -130,7 +130,10 @@
<div class="chat-messages" id="chat-messages"></div>
<div id="slash-autocomplete" class="slash-autocomplete" style="display:none"></div>
<div class="chat-input">
<div id="image-preview-strip" class="image-preview-strip"></div>
<textarea id="chat-input" placeholder="Message or / for commands..." rows="1"></textarea>
<input type="file" id="image-file-input" accept="image/*" multiple style="display:none">
<button id="attach-btn" class="attach-btn" title="Attach images" aria-label="Attach images">&#x1F4CE;</button>
<button id="send-btn" onclick="sendMessage()">Send</button>
</div>
</div>
+153 -23
View File
@@ -159,7 +159,7 @@ body {
flex-shrink: 0;
}
.tab-bar button:not(.status-logs-btn) {
.tab-bar button:not(.status-logs-btn):not(.restart-btn) {
padding: 10px 20px;
background: none;
border: none;
@@ -171,11 +171,11 @@ body {
transition: color 0.2s, border-color 0.2s;
}
.tab-bar button:not(.status-logs-btn):hover {
.tab-bar button:not(.status-logs-btn):not(.restart-btn):hover {
color: var(--text);
}
.tab-bar button:not(.status-logs-btn).active {
.tab-bar button:not(.status-logs-btn):not(.restart-btn).active {
color: var(--accent);
border-bottom-color: var(--accent);
}
@@ -260,42 +260,42 @@ body {
}
/* Restart Button */
.restart-btn {
.tab-bar .restart-btn {
display: flex;
align-items: center;
gap: 0.375rem;
margin: 0.375rem;
padding: 0.25rem 0.75rem;
border-radius: 0.5rem;
font-size: 0.8rem;
border: 1px solid;
border-color: #00d894;
border: 1px solid #00d894;
color: #00d894;
background-color: transparent;
cursor: pointer;
transition: color 150ms, background-color 150ms, border-color 150ms;
}
.restart-btn:hover:not(:disabled) {
.tab-bar .restart-btn:hover:not(:disabled) {
background-color: rgba(0, 216, 148, 0.1);
}
.restart-btn:disabled {
.tab-bar .restart-btn:disabled {
border-color: #333;
color: #666;
cursor: not-allowed;
}
.restart-btn:disabled:hover {
.tab-bar .restart-btn:disabled:hover {
background-color: transparent;
}
.restart-btn svg {
.tab-bar .restart-btn svg {
flex-shrink: 0;
width: 13px;
height: 13px;
}
.restart-btn svg.spinning {
.tab-bar .restart-btn svg.spinning {
animation: spin-icon 1s linear infinite;
}
@@ -1272,6 +1272,7 @@ body {
/* Chat input */
.chat-input {
display: flex;
flex-wrap: wrap;
padding: 12px 16px max(12px, env(safe-area-inset-bottom)) 16px;
gap: 8px;
background: var(--bg-secondary);
@@ -3074,7 +3075,7 @@ mark {
}
.thread-sidebar {
width: 200px;
width: 240px;
background: var(--bg-secondary);
border-right: 1px solid var(--border);
display: flex;
@@ -3082,6 +3083,8 @@ mark {
flex-shrink: 0;
transition: width 0.2s ease;
overflow: hidden;
padding: 6px;
gap: 2px;
}
.thread-sidebar.collapsed {
@@ -3099,8 +3102,7 @@ mark {
.thread-sidebar-header {
display: flex;
align-items: center;
padding: 10px 12px;
border-bottom: 1px solid var(--border);
padding: 10px 10px;
font-size: 13px;
font-weight: 600;
gap: 8px;
@@ -3134,21 +3136,22 @@ mark {
display: flex;
align-items: center;
justify-content: space-between;
padding: 10px 12px;
padding: 12px 14px;
cursor: pointer;
font-size: 13px;
font-weight: 600;
color: var(--text);
border-bottom: 1px solid var(--border);
background: var(--bg-secondary);
background: var(--bg-tertiary);
border-radius: var(--radius);
margin-bottom: 2px;
}
.assistant-item:hover {
background: var(--bg-tertiary);
background: rgba(255, 255, 255, 0.06);
}
.assistant-item.active {
background: rgba(52, 211, 153, 0.08);
background: rgba(52, 211, 153, 0.1);
color: var(--accent);
border-left: 2px solid var(--accent);
}
@@ -3166,7 +3169,7 @@ mark {
}
.threads-section-header {
padding: 8px 12px 4px;
padding: 10px 10px 4px;
font-size: 11px;
font-weight: 500;
text-transform: uppercase;
@@ -3196,11 +3199,11 @@ mark {
display: flex;
align-items: center;
justify-content: space-between;
padding: 8px 12px;
padding: 10px 14px;
cursor: pointer;
font-size: 13px;
color: var(--text-secondary);
border-bottom: 1px solid rgba(255, 255, 255, 0.03);
border-radius: var(--radius);
}
.thread-item:hover {
@@ -3222,6 +3225,43 @@ mark {
.thread-meta {
font-size: 11px;
color: var(--text-secondary);
flex-shrink: 0;
}
.thread-badge {
display: inline-block;
font-size: 9px;
font-weight: 600;
text-transform: uppercase;
letter-spacing: 0.5px;
padding: 1px 5px;
border-radius: 3px;
background: rgba(255, 255, 255, 0.08);
color: var(--text-secondary);
margin-right: 6px;
flex-shrink: 0;
}
.thread-badge-routine { background: rgba(52, 211, 153, 0.15); color: var(--accent); }
.thread-badge-heartbeat { background: rgba(245, 166, 35, 0.15); color: var(--warning); }
.thread-badge-telegram { background: rgba(0, 136, 204, 0.15); color: #0088cc; }
.thread-badge-signal { background: rgba(59, 118, 240, 0.15); color: #3b76f0; }
.thread-badge-slack { background: rgba(74, 21, 75, 0.15); color: #e01e5a; }
.thread-unread {
display: inline-flex;
align-items: center;
justify-content: center;
min-width: 16px;
height: 16px;
font-size: 10px;
font-weight: 700;
background: var(--accent);
color: var(--bg);
border-radius: 8px;
padding: 0 4px;
margin-left: auto;
flex-shrink: 0;
}
/* --- Memory editing --- */
@@ -3620,7 +3660,7 @@ mark {
left: 0;
top: 0;
bottom: 0;
width: 200px;
width: 240px;
z-index: 50;
}
@@ -3722,3 +3762,93 @@ mark {
text-overflow: ellipsis;
white-space: nowrap;
}
/* Image Upload */
.attach-btn {
background: none;
border: none;
cursor: pointer;
font-size: 1.2em;
padding: 8px;
align-self: flex-end;
color: var(--text-secondary);
transition: color 0.2s;
min-height: 40px;
display: flex;
align-items: center;
justify-content: center;
}
.attach-btn:hover {
color: var(--text);
}
.image-preview-strip {
display: flex;
flex-direction: row;
gap: 8px;
padding: 4px;
overflow-x: auto;
min-height: 0;
width: 100%;
}
.image-preview-strip:empty {
display: none;
}
.image-preview-container {
position: relative;
display: inline-block;
flex-shrink: 0;
}
.image-preview {
width: 60px;
height: 60px;
border-radius: 6px;
object-fit: cover;
display: block;
}
.image-preview-remove {
position: absolute;
top: -6px;
right: -6px;
width: 18px;
height: 18px;
border-radius: 50%;
background: var(--danger);
color: #fff;
border: none;
font-size: 12px;
line-height: 18px;
text-align: center;
cursor: pointer;
padding: 0;
}
.image-preview-remove:hover {
background: #c33;
}
/* Generated Image */
.generated-image-card {
max-width: 512px;
margin: 8px 0;
border-radius: 8px;
overflow: hidden;
border: 1px solid var(--border);
}
.generated-image {
max-width: 100%;
display: block;
}
.generated-image-path {
font-size: 12px;
color: var(--text-secondary);
padding: 4px 8px;
background: var(--bg-secondary);
}
+1
View File
@@ -84,6 +84,7 @@ impl TestGatewayBuilder {
chat_rate_limiter: RateLimiter::new(30, 60),
registry_entries: Vec::new(),
cost_guard: None,
routine_engine: Arc::new(tokio::sync::RwLock::new(None)),
startup_time: std::time::Instant::now(),
})
}
+73 -2
View File
@@ -5,10 +5,23 @@ use uuid::Uuid;
// --- Chat ---
/// Base64-encoded image data sent from the web frontend.
#[derive(Debug, Clone, Deserialize)]
pub struct ImageData {
/// MIME type (e.g., "image/png", "image/jpeg").
pub media_type: String,
/// Base64-encoded image data (without data: URL prefix).
pub data: String,
}
#[derive(Debug, Deserialize)]
pub struct SendMessageRequest {
pub content: String,
pub thread_id: Option<String>,
pub timezone: Option<String>,
/// Optional images attached to the message.
#[serde(default)]
pub images: Vec<ImageData>,
}
#[derive(Debug, Serialize)]
@@ -28,6 +41,8 @@ pub struct ThreadInfo {
pub title: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub thread_type: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub channel: Option<String>,
}
#[derive(Debug, Serialize)]
@@ -217,6 +232,16 @@ pub enum SseEvent {
session_id: Option<String>,
},
/// An image was generated by a tool.
#[serde(rename = "image_generated")]
ImageGenerated {
data_url: String,
#[serde(skip_serializing_if = "Option::is_none")]
path: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
thread_id: Option<String>,
},
/// Extension activation status change (WASM channels).
#[serde(rename = "extension_status")]
ExtensionStatus {
@@ -611,6 +636,10 @@ pub enum WsClientMessage {
Message {
content: String,
thread_id: Option<String>,
timezone: Option<String>,
/// Optional images attached to the message.
#[serde(default)]
images: Vec<ImageData>,
},
/// Approve or deny a pending tool execution.
#[serde(rename = "approval")]
@@ -677,6 +706,7 @@ impl WsServerMessage {
SseEvent::JobToolResult { .. } => "job_tool_result",
SseEvent::JobStatus { .. } => "job_status",
SseEvent::JobResult { .. } => "job_result",
SseEvent::ImageGenerated { .. } => "image_generated",
SseEvent::ExtensionStatus { .. } => "extension_status",
};
let data = serde_json::to_value(event).unwrap_or(serde_json::Value::Null);
@@ -746,6 +776,7 @@ pub struct RoutineRunInfo {
pub status: String,
pub result_summary: Option<String>,
pub tokens_used: Option<i32>,
pub job_id: Option<Uuid>,
}
// --- Settings ---
@@ -796,7 +827,9 @@ mod tests {
let json = r#"{"type":"message","content":"hello","thread_id":"t1"}"#;
let msg: WsClientMessage = serde_json::from_str(json).unwrap();
match msg {
WsClientMessage::Message { content, thread_id } => {
WsClientMessage::Message {
content, thread_id, ..
} => {
assert_eq!(content, "hello");
assert_eq!(thread_id.as_deref(), Some("t1"));
}
@@ -809,7 +842,9 @@ mod tests {
let json = r#"{"type":"message","content":"hi"}"#;
let msg: WsClientMessage = serde_json::from_str(json).unwrap();
match msg {
WsClientMessage::Message { content, thread_id } => {
WsClientMessage::Message {
content, thread_id, ..
} => {
assert_eq!(content, "hi");
assert!(thread_id.is_none());
}
@@ -1063,4 +1098,40 @@ mod tests {
let req: AuthCancelRequest = serde_json::from_str(json).unwrap();
assert_eq!(req.extension_name, "telegram");
}
// ---- ThreadInfo channel field tests ----
#[test]
fn test_thread_info_channel_serialized() {
let info = ThreadInfo {
id: Uuid::nil(),
state: "Idle".to_string(),
turn_count: 0,
created_at: "2026-01-01T00:00:00Z".to_string(),
updated_at: "2026-01-01T00:00:00Z".to_string(),
title: None,
thread_type: None,
channel: Some("telegram".to_string()),
};
let json = serde_json::to_string(&info).unwrap();
let parsed: serde_json::Value = serde_json::from_str(&json).unwrap();
assert_eq!(parsed["channel"], "telegram");
}
#[test]
fn test_thread_info_channel_omitted_when_none() {
let info = ThreadInfo {
id: Uuid::nil(),
state: "Idle".to_string(),
turn_count: 0,
created_at: "2026-01-01T00:00:00Z".to_string(),
updated_at: "2026-01-01T00:00:00Z".to_string(),
title: None,
thread_type: None,
channel: None,
};
let json = serde_json::to_string(&info).unwrap();
let parsed: serde_json::Value = serde_json::from_str(&json).unwrap();
assert!(parsed.get("channel").is_none());
}
}
+75 -1
View File
@@ -3,6 +3,10 @@
use crate::channels::web::types::{ToolCallInfo, TurnInfo};
/// Truncate a string to at most `max_bytes` bytes at a char boundary, appending "...".
///
/// If the input is wrapped in `<tool_output …>…</tool_output>` and truncation
/// removes the closing tag, the tag is re-appended so downstream XML parsers
/// never see an unclosed element.
pub fn truncate_preview(s: &str, max_bytes: usize) -> String {
if s.len() <= max_bytes {
return s.to_string();
@@ -12,7 +16,14 @@ pub fn truncate_preview(s: &str, max_bytes: usize) -> String {
while end > 0 && !s.is_char_boundary(end) {
end -= 1;
}
format!("{}...", &s[..end])
let mut result = format!("{}...", &s[..end]);
// Re-close <tool_output> if truncation cut through the closing tag.
if s.starts_with("<tool_output") && !result.ends_with("</tool_output>") {
result.push_str("\n</tool_output>");
}
result
}
/// Build TurnInfo pairs from flat DB messages (user/tool_calls/assistant triples).
@@ -83,6 +94,19 @@ pub fn build_turns_from_db_messages(
turns.push(turn);
turn_number += 1;
} else if msg.role == "assistant" {
// Standalone assistant message (e.g. routine output, heartbeat)
// with no preceding user message — render as a turn with empty input.
turns.push(TurnInfo {
turn_number,
user_input: String::new(),
response: Some(msg.content.clone()),
state: "Completed".to_string(),
started_at: msg.created_at.to_rfc3339(),
completed_at: Some(msg.created_at.to_rfc3339()),
tool_calls: Vec::new(),
});
turn_number += 1;
}
}
@@ -149,6 +173,33 @@ mod tests {
assert_eq!(truncate_preview("hello", 0), "...");
}
#[test]
fn test_truncate_preview_closes_tool_output_tag() {
let s = "<tool_output name=\"search\" sanitized=\"true\">\nSome very long content here\n</tool_output>";
// Truncate so it cuts before the closing tag
let result = truncate_preview(s, 60);
assert!(result.ends_with("</tool_output>"));
assert!(result.contains("..."));
}
#[test]
fn test_truncate_preview_no_extra_close_when_intact() {
let s = "<tool_output name=\"echo\" sanitized=\"false\">\nshort\n</tool_output>";
// The string is short enough not to be truncated
let result = truncate_preview(s, 500);
assert_eq!(result, s);
// Should not have a duplicate closing tag
assert_eq!(result.matches("</tool_output>").count(), 1);
}
#[test]
fn test_truncate_preview_non_xml_unaffected() {
let s = "Just a plain long string that gets truncated";
let result = truncate_preview(s, 10);
assert_eq!(result, "Just a pla...");
assert!(!result.contains("</tool_output>"));
}
// ---- build_turns_from_db_messages tests ----
fn make_msg(role: &str, content: &str, offset_ms: i64) -> crate::history::ConversationMessage {
@@ -220,6 +271,29 @@ mod tests {
assert_eq!(turns[0].response.as_deref(), Some("Done"));
}
#[test]
fn test_build_turns_standalone_assistant_messages() {
// Routine conversations only have assistant messages (no user messages).
let messages = vec![
make_msg("assistant", "Routine executed: all checks passed", 0),
make_msg("assistant", "Routine executed: found 2 issues", 5000),
];
let turns = build_turns_from_db_messages(&messages);
assert_eq!(turns.len(), 2);
// Standalone assistant messages should have empty user_input
assert_eq!(turns[0].user_input, "");
assert_eq!(
turns[0].response.as_deref(),
Some("Routine executed: all checks passed")
);
assert_eq!(turns[0].state, "Completed");
assert_eq!(turns[1].user_input, "");
assert_eq!(
turns[1].response.as_deref(),
Some("Routine executed: found 2 issues")
);
}
#[test]
fn test_build_turns_backward_compatible() {
let messages = vec![
+20 -1
View File
@@ -156,12 +156,26 @@ async fn handle_client_message(
direct_tx: &mpsc::Sender<WsServerMessage>,
) {
match msg {
WsClientMessage::Message { content, thread_id } => {
WsClientMessage::Message {
content,
thread_id,
timezone,
images,
} => {
let mut incoming = IncomingMessage::new("gateway", user_id, &content);
if let Some(ref tz) = timezone {
incoming = incoming.with_timezone(tz);
}
if let Some(ref tid) = thread_id {
incoming = incoming.with_thread(tid);
}
// Convert uploaded images to IncomingAttachments
if !images.is_empty() {
let attachments = crate::channels::web::server::images_to_attachments(&images);
incoming = incoming.with_attachments(attachments);
}
let tx_guard = state.msg_tx.read().await;
if let Some(ref tx) = *tx_guard {
if tx.send(incoming).await.is_err() {
@@ -349,6 +363,8 @@ mod tests {
WsClientMessage::Message {
content: "hello agent".to_string(),
thread_id: Some("t1".to_string()),
timezone: None,
images: Vec::new(),
},
&state,
"user1",
@@ -373,6 +389,8 @@ mod tests {
WsClientMessage::Message {
content: "hello".to_string(),
thread_id: None,
timezone: None,
images: Vec::new(),
},
&state,
"user1",
@@ -493,6 +511,7 @@ mod tests {
chat_rate_limiter: crate::channels::web::server::RateLimiter::new(30, 60),
registry_entries: Vec::new(),
cost_guard: None,
routine_engine: Arc::new(tokio::sync::RwLock::new(None)),
startup_time: std::time::Instant::now(),
}
}
+1 -1
View File
@@ -68,7 +68,7 @@ impl WebhookServer {
if let Err(e) = axum::serve(listener, app)
.with_graceful_shutdown(async {
let _ = shutdown_rx.await;
tracing::info!("Webhook server shutting down");
tracing::debug!("Webhook server shutting down");
})
.await
{
+1 -1
View File
@@ -107,7 +107,7 @@ enum CheckResult {
async fn check_nearai_session() -> CheckResult {
// Check if session file exists
let session_path = crate::llm::session::default_session_path();
let session_path = crate::config::llm::default_session_path();
if !session_path.exists() {
// Check for API key mode
if std::env::var("NEARAI_API_KEY").is_ok() {
+243 -131
View File
@@ -2,52 +2,79 @@
//!
//! Commands for adding, removing, authenticating, and testing MCP servers.
use std::collections::HashMap;
use std::io::Write;
use std::sync::Arc;
use clap::Subcommand;
use clap::{Args, Subcommand};
use crate::config::Config;
use crate::db::Database;
#[cfg(feature = "postgres")]
use crate::secrets::PostgresSecretsStore;
use crate::secrets::{SecretsCrypto, SecretsStore};
use crate::secrets::SecretsStore;
use crate::tools::mcp::{
McpClient, McpServerConfig, McpSessionManager, OAuthConfig,
auth::{authorize_mcp_server, is_authenticated},
config::{self, McpServersFile},
config::{self, EffectiveTransport, McpServersFile},
};
/// Arguments for the `mcp add` subcommand.
#[derive(Args, Debug, Clone)]
pub struct McpAddArgs {
/// Server name (e.g., "notion", "github")
pub name: String,
/// Server URL (e.g., "https://mcp.notion.com") -- required for http transport
pub url: Option<String>,
/// Transport type: http (default), stdio, unix
#[arg(long, default_value = "http")]
pub transport: String,
/// Command to run (stdio transport)
#[arg(long)]
pub command: Option<String>,
/// Command arguments (stdio transport, can be repeated)
#[arg(long = "arg", num_args = 1..)]
pub cmd_args: Vec<String>,
/// Environment variables (stdio transport, KEY=VALUE format, can be repeated)
#[arg(long = "env", value_parser = parse_env_var)]
pub env: Vec<(String, String)>,
/// Unix socket path (unix transport)
#[arg(long)]
pub socket: Option<String>,
/// Custom HTTP headers (KEY:VALUE format, can be repeated)
#[arg(long = "header", value_parser = parse_header)]
pub headers: Vec<(String, String)>,
/// OAuth client ID (if authentication is required)
#[arg(long)]
pub client_id: Option<String>,
/// OAuth authorization URL (optional, can be discovered)
#[arg(long)]
pub auth_url: Option<String>,
/// OAuth token URL (optional, can be discovered)
#[arg(long)]
pub token_url: Option<String>,
/// Scopes to request (comma-separated)
#[arg(long)]
pub scopes: Option<String>,
/// Server description
#[arg(long)]
pub description: Option<String>,
}
#[derive(Subcommand, Debug, Clone)]
pub enum McpCommand {
/// Add an MCP server
Add {
/// Server name (e.g., "notion", "github")
name: String,
/// Server URL (e.g., "https://mcp.notion.com")
url: String,
/// OAuth client ID (if authentication is required)
#[arg(long)]
client_id: Option<String>,
/// OAuth authorization URL (optional, can be discovered)
#[arg(long)]
auth_url: Option<String>,
/// OAuth token URL (optional, can be discovered)
#[arg(long)]
token_url: Option<String>,
/// Scopes to request (comma-separated)
#[arg(long)]
scopes: Option<String>,
/// Server description
#[arg(long)]
description: Option<String>,
},
Add(Box<McpAddArgs>),
/// Remove an MCP server
Remove {
@@ -97,29 +124,24 @@ pub enum McpCommand {
},
}
fn parse_header(s: &str) -> Result<(String, String), String> {
let pos = s
.find(':')
.ok_or_else(|| format!("invalid header format '{}', expected KEY:VALUE", s))?;
Ok((s[..pos].trim().to_string(), s[pos + 1..].trim().to_string()))
}
fn parse_env_var(s: &str) -> Result<(String, String), String> {
let pos = s
.find('=')
.ok_or_else(|| format!("invalid env var format '{}', expected KEY=VALUE", s))?;
Ok((s[..pos].to_string(), s[pos + 1..].to_string()))
}
/// Run an MCP command.
pub async fn run_mcp_command(cmd: McpCommand) -> anyhow::Result<()> {
match cmd {
McpCommand::Add {
name,
url,
client_id,
auth_url,
token_url,
scopes,
description,
} => {
add_server(
name,
url,
client_id,
auth_url,
token_url,
scopes,
description,
)
.await
}
McpCommand::Add(args) => add_server(*args).await,
McpCommand::Remove { name } => remove_server(name).await,
McpCommand::List { verbose } => list_servers(verbose).await,
McpCommand::Auth { name, user } => auth_server(name, user).await,
@@ -133,16 +155,58 @@ pub async fn run_mcp_command(cmd: McpCommand) -> anyhow::Result<()> {
}
/// Add a new MCP server.
async fn add_server(
name: String,
url: String,
client_id: Option<String>,
auth_url: Option<String>,
token_url: Option<String>,
scopes: Option<String>,
description: Option<String>,
) -> anyhow::Result<()> {
let mut config = McpServerConfig::new(&name, &url);
async fn add_server(args: McpAddArgs) -> anyhow::Result<()> {
let McpAddArgs {
name,
url,
transport,
command,
cmd_args,
env,
socket,
headers,
client_id,
auth_url,
token_url,
scopes,
description,
} = args;
let transport_lower = transport.to_lowercase();
let mut config = match transport_lower.as_str() {
"stdio" => {
let cmd = command
.clone()
.ok_or_else(|| anyhow::anyhow!("--command is required for stdio transport"))?;
let env_map: HashMap<String, String> = env.into_iter().collect();
McpServerConfig::new_stdio(&name, &cmd, cmd_args.clone(), env_map)
}
"unix" => {
let socket_path = socket
.clone()
.ok_or_else(|| anyhow::anyhow!("--socket is required for unix transport"))?;
McpServerConfig::new_unix(&name, &socket_path)
}
"http" => {
let url_val = url
.as_deref()
.ok_or_else(|| anyhow::anyhow!("URL is required for http transport"))?;
McpServerConfig::new(&name, url_val)
}
other => {
anyhow::bail!(
"Unknown transport type '{}'. Supported: http, stdio, unix",
other
);
}
};
// Apply headers if any
if !headers.is_empty() {
let headers_map: HashMap<String, String> = headers.into_iter().collect();
config = config.with_headers(headers_map);
}
if let Some(desc) = description {
config = config.with_description(desc);
@@ -151,8 +215,12 @@ async fn add_server(
// Track if auth is required
let requires_auth = client_id.is_some();
// Set up OAuth if client_id is provided
// Set up OAuth if client_id is provided (HTTP transport only)
if let Some(client_id) = client_id {
if transport_lower != "http" {
anyhow::bail!("OAuth authentication is only supported with http transport");
}
let mut oauth = OAuthConfig::new(client_id);
if let (Some(auth), Some(token)) = (auth_url, token_url) {
@@ -181,7 +249,24 @@ async fn add_server(
println!();
println!(" ✓ Added MCP server '{}'", name);
println!(" URL: {}", url);
match transport_lower.as_str() {
"stdio" => {
println!(
" Transport: stdio (command: {})",
command.as_deref().unwrap_or("")
);
}
"unix" => {
println!(
" Transport: unix (socket: {})",
socket.as_deref().unwrap_or("")
);
}
_ => {
println!(" URL: {}", url.as_deref().unwrap_or(""));
}
}
if requires_auth {
println!();
@@ -236,9 +321,40 @@ async fn list_servers(verbose: bool) -> anyhow::Result<()> {
""
};
let effective = server.effective_transport();
let transport_label = match &effective {
EffectiveTransport::Http => "http".to_string(),
EffectiveTransport::Stdio { command, .. } => {
format!("stdio ({})", command)
}
EffectiveTransport::Unix { socket_path } => {
format!("unix ({})", socket_path)
}
};
if verbose {
println!(" {} {}{}", status, server.name, auth_status);
println!(" URL: {}", server.url);
println!(" Transport: {}", transport_label);
match &effective {
EffectiveTransport::Http => {
println!(" URL: {}", server.url);
}
EffectiveTransport::Stdio { command, args, env } => {
println!(" Command: {}", command);
if !args.is_empty() {
println!(" Args: {}", args.join(", "));
}
if !env.is_empty() {
// Only print env var names, not values (may contain secrets).
let env_keys: Vec<&str> = env.keys().map(|k| k.as_str()).collect();
println!(" Env: {}", env_keys.join(", "));
}
}
EffectiveTransport::Unix { socket_path } => {
println!(" Socket: {}", socket_path);
}
}
if let Some(ref desc) = server.description {
println!(" Description: {}", desc);
}
@@ -248,11 +364,27 @@ async fn list_servers(verbose: bool) -> anyhow::Result<()> {
println!(" Scopes: {}", oauth.scopes.join(", "));
}
}
if !server.headers.is_empty() {
let header_keys: Vec<&String> = server.headers.keys().collect();
println!(
" Headers: {}",
header_keys
.iter()
.map(|k| k.as_str())
.collect::<Vec<_>>()
.join(", ")
);
}
println!();
} else {
let display = match &effective {
EffectiveTransport::Http => server.url.clone(),
EffectiveTransport::Stdio { command, .. } => command.to_string(),
EffectiveTransport::Unix { socket_path } => socket_path.to_string(),
};
println!(
" {} {} - {}{}",
status, server.name, server.url, auth_status
" {} {} - {} [{}]{}",
status, server.name, display, transport_label, auth_status
);
}
}
@@ -374,7 +506,7 @@ async fn test_server(name: String, user_id: String) -> anyhow::Result<()> {
return Ok(());
} else {
// No OAuth and no tokens - try unauthenticated
McpClient::new_with_name(&server.name, &server.url)
McpClient::new_with_config(server.clone())
};
// Test connection
@@ -496,69 +628,7 @@ async fn save_servers(
/// Initialize and return the secrets store.
async fn get_secrets_store() -> anyhow::Result<Arc<dyn SecretsStore + Send + Sync>> {
let config = Config::from_env().await?;
let master_key = config.secrets.master_key().ok_or_else(|| {
anyhow::anyhow!(
"SECRETS_MASTER_KEY not set. Run 'ironclaw onboard' first or set it in .env"
)
})?;
let crypto = SecretsCrypto::new(master_key.clone())?;
#[cfg(feature = "postgres")]
{
let store = crate::history::Store::new(&config.database).await?;
store.run_migrations().await?;
Ok(Arc::new(PostgresSecretsStore::new(
store.pool(),
Arc::new(crypto),
)))
}
#[cfg(all(feature = "libsql", not(feature = "postgres")))]
{
use crate::db::Database as _;
use crate::db::libsql::LibSqlBackend;
use secrecy::ExposeSecret as _;
let default_path = crate::config::default_libsql_path();
let db_path = config
.database
.libsql_path
.as_deref()
.unwrap_or(&default_path);
let backend = if let Some(ref url) = config.database.libsql_url {
let token = config.database.libsql_auth_token.as_ref().ok_or_else(|| {
anyhow::anyhow!("LIBSQL_AUTH_TOKEN is required when LIBSQL_URL is set")
})?;
LibSqlBackend::new_remote_replica(db_path, url, token.expose_secret())
.await
.map_err(|e| anyhow::anyhow!("{}", e))?
} else {
LibSqlBackend::new_local(db_path)
.await
.map_err(|e| anyhow::anyhow!("{}", e))?
};
backend
.run_migrations()
.await
.map_err(|e| anyhow::anyhow!("{}", e))?;
Ok(Arc::new(crate::secrets::LibSqlSecretsStore::new(
backend.shared_db(),
Arc::new(crypto),
)))
}
#[cfg(not(any(feature = "postgres", feature = "libsql")))]
{
let _ = crypto;
anyhow::bail!(
"No database backend available for secrets. Enable 'postgres' or 'libsql' feature."
);
}
crate::cli::init_secrets_store().await
}
#[cfg(test)]
@@ -579,4 +649,46 @@ mod tests {
TestCli::command().debug_assert();
}
#[test]
fn test_parse_header_valid() {
let result = parse_header("Authorization: Bearer token123").unwrap();
assert_eq!(result.0, "Authorization");
assert_eq!(result.1, "Bearer token123");
}
#[test]
fn test_parse_header_no_spaces() {
let result = parse_header("X-Api-Key:abc123").unwrap();
assert_eq!(result.0, "X-Api-Key");
assert_eq!(result.1, "abc123");
}
#[test]
fn test_parse_header_invalid() {
let result = parse_header("no-colon-here");
assert!(result.is_err());
assert!(result.unwrap_err().contains("invalid header format"));
}
#[test]
fn test_parse_env_var_valid() {
let result = parse_env_var("NODE_ENV=production").unwrap();
assert_eq!(result.0, "NODE_ENV");
assert_eq!(result.1, "production");
}
#[test]
fn test_parse_env_var_with_equals_in_value() {
let result = parse_env_var("KEY=value=with=equals").unwrap();
assert_eq!(result.0, "KEY");
assert_eq!(result.1, "value=with=equals");
}
#[test]
fn test_parse_env_var_invalid() {
let result = parse_env_var("no-equals-here");
assert!(result.is_err());
assert!(result.unwrap_err().contains("invalid env var format"));
}
}
+46 -5
View File
@@ -28,8 +28,6 @@ pub use config::{ConfigCommand, run_config_command};
pub use doctor::run_doctor_command;
pub use mcp::{McpCommand, run_mcp_command};
pub use memory::MemoryCommand;
#[cfg(feature = "postgres")]
pub use memory::run_memory_command;
pub use memory::run_memory_command_with_db;
pub use pairing::{PairingCommand, run_pairing_command, run_pairing_command_with_store};
pub use registry::{RegistryCommand, run_registry_command};
@@ -37,6 +35,8 @@ pub use service::{ServiceCommand, run_service_command};
pub use status::run_status_command;
pub use tool::{ToolCommand, run_tool_command};
use std::sync::Arc;
use clap::{ColorChoice, Parser, Subcommand};
#[derive(Parser, Debug)]
@@ -94,12 +94,16 @@ pub enum Command {
skip_auth: bool,
/// Reconfigure channels only
#[arg(long, conflicts_with = "provider_only")]
#[arg(long, conflicts_with_all = ["provider_only", "quick"])]
channels_only: bool,
/// Reconfigure LLM provider and model only
#[arg(long, conflicts_with = "channels_only")]
#[arg(long, conflicts_with_all = ["channels_only", "quick"])]
provider_only: bool,
/// Quick setup: auto-defaults everything except LLM provider and model
#[arg(long, conflicts_with_all = ["channels_only", "provider_only"])]
quick: bool,
},
/// Manage configuration settings
@@ -132,7 +136,7 @@ pub enum Command {
about = "Manage MCP servers",
long_about = "Add, auth, list, or test MCP servers.\nExample: ironclaw mcp add notion https://mcp.notion.com"
)]
Mcp(McpCommand),
Mcp(Box<McpCommand>),
/// Query and manage workspace memory
#[command(
@@ -225,6 +229,43 @@ impl Cli {
}
}
/// Initialize a secrets store from environment config.
///
/// Shared helper for CLI subcommands (`mcp auth`, `tool auth`, etc.) that need
/// access to encrypted secrets without spinning up the full AppBuilder.
pub async fn init_secrets_store()
-> anyhow::Result<Arc<dyn crate::secrets::SecretsStore + Send + Sync>> {
let config = crate::config::Config::from_env().await?;
let master_key = config.secrets.master_key().ok_or_else(|| {
anyhow::anyhow!(
"SECRETS_MASTER_KEY not set. Run 'ironclaw onboard' first or set it in .env"
)
})?;
let crypto = Arc::new(crate::secrets::SecretsCrypto::new(master_key.clone())?);
Ok(crate::db::create_secrets_store(&config.database, crypto).await?)
}
/// Run the Memory CLI subcommand.
pub async fn run_memory_command(mem_cmd: &MemoryCommand) -> anyhow::Result<()> {
let config = crate::config::Config::from_env()
.await
.map_err(|e| anyhow::anyhow!("{}", e))?;
let session = crate::llm::create_session_manager(config.llm.session.clone()).await;
let embeddings = config
.embeddings
.create_provider(&config.llm.nearai.base_url, session);
let db: Arc<dyn crate::db::Database> = crate::db::connect_from_config(&config.database)
.await
.map_err(|e| anyhow::anyhow!("{}", e))?;
run_memory_command_with_db(mem_cmd.clone(), db, embeddings).await
}
#[cfg(test)]
mod tests {
use super::*;
+6 -342
View File
@@ -24,8 +24,6 @@ use std::time::Duration;
use base64::{Engine, engine::general_purpose::URL_SAFE_NO_PAD};
use rand::RngCore;
use sha2::{Digest, Sha256};
use tokio::io::{AsyncBufReadExt, AsyncWriteExt, BufReader};
use tokio::net::TcpListener;
use tokio::sync::RwLock;
use crate::secrets::{CreateSecretParams, SecretsStore};
@@ -64,259 +62,12 @@ pub fn builtin_credentials(secret_name: &str) -> Option<OAuthCredentials> {
// ── Shared callback server ──────────────────────────────────────────────
/// Fixed port for all OAuth callbacks.
///
/// Every redirect URI registered with providers must use this port:
/// `http://localhost:9876/callback` (or `/auth/callback` for NEAR AI).
pub const OAUTH_CALLBACK_PORT: u16 = 9876;
/// Returns the OAuth callback base URL.
///
/// Checks `IRONCLAW_OAUTH_CALLBACK_URL` env var first (useful for remote/VPS
/// deployments where `127.0.0.1` is unreachable from the user's browser),
/// then falls back to `http://{callback_host()}:{OAUTH_CALLBACK_PORT}`.
pub fn callback_url() -> String {
std::env::var("IRONCLAW_OAUTH_CALLBACK_URL")
.ok()
.filter(|v| !v.is_empty())
.unwrap_or_else(|| format!("http://{}:{}", callback_host(), OAUTH_CALLBACK_PORT))
}
/// Returns the hostname used in OAuth callback URLs.
///
/// Reads `OAUTH_CALLBACK_HOST` from the environment (default: `127.0.0.1`).
///
/// **Remote server usage:** set `OAUTH_CALLBACK_HOST` to the network interface
/// address you want to listen on (e.g. the server's LAN IP or `0.0.0.0`).
/// The callback listener will bind to that specific address instead of the
/// loopback interface, so the OAuth redirect can reach an external browser.
/// Note: this transmits the session token over plain HTTP — prefer SSH port
/// forwarding (`ssh -L 9876:127.0.0.1:9876 user@host`) when possible.
///
/// # Example
///
/// ```bash
/// export OAUTH_CALLBACK_HOST=203.0.113.10
/// ironclaw login
/// # Opens: http://203.0.113.10:9876/auth/callback
/// ```
pub fn callback_host() -> String {
std::env::var("OAUTH_CALLBACK_HOST").unwrap_or_else(|_| "127.0.0.1".to_string())
}
/// Returns `true` if `host` is a loopback address that only accepts local connections.
///
/// Covers `localhost` (case-insensitive), the full `127.0.0.0/8` IPv4 loopback
/// range, and `::1` for IPv6.
pub fn is_loopback_host(host: &str) -> bool {
if host.eq_ignore_ascii_case("localhost") {
return true;
}
host.parse::<std::net::IpAddr>()
.map(|ip| ip.is_loopback())
.unwrap_or(false)
}
/// Error from the OAuth callback listener.
#[derive(Debug, thiserror::Error)]
pub enum OAuthCallbackError {
#[error("Port {0} is in use (another auth flow running?): {1}")]
PortInUse(u16, String),
#[error("Authorization denied by user")]
Denied,
#[error("Timed out waiting for authorization")]
Timeout,
#[error("CSRF state mismatch: expected {expected}, got {actual}")]
StateMismatch { expected: String, actual: String },
#[error("IO error: {0}")]
Io(String),
}
/// Map a `std::io::Error` from a bind attempt to an `OAuthCallbackError`.
fn bind_error(e: std::io::Error) -> OAuthCallbackError {
if e.kind() == std::io::ErrorKind::AddrInUse {
OAuthCallbackError::PortInUse(OAUTH_CALLBACK_PORT, e.to_string())
} else {
OAuthCallbackError::Io(e.to_string())
}
}
/// Bind the OAuth callback listener on the fixed port.
///
/// When `OAUTH_CALLBACK_HOST` is a loopback address (the default `127.0.0.1`),
/// binds to `127.0.0.1` first and falls back to `[::1]` so local-only auth
/// flows remain restricted to the local machine.
///
/// When `OAUTH_CALLBACK_HOST` is set to a remote address, binds to that
/// specific address so only connections directed to it are accepted.
pub async fn bind_callback_listener() -> Result<TcpListener, OAuthCallbackError> {
let host = callback_host();
if is_loopback_host(&host) {
// Local mode: prefer IPv4 loopback, fall back to IPv6.
let ipv4_addr = format!("127.0.0.1:{}", OAUTH_CALLBACK_PORT);
match TcpListener::bind(&ipv4_addr).await {
Ok(listener) => return Ok(listener),
Err(e) if e.kind() == std::io::ErrorKind::AddrInUse => {
return Err(OAuthCallbackError::PortInUse(
OAUTH_CALLBACK_PORT,
e.to_string(),
));
}
Err(_) => {
// IPv4 not available, fall back to IPv6
}
}
TcpListener::bind(format!("[::1]:{}", OAUTH_CALLBACK_PORT))
.await
.map_err(bind_error)
} else {
// Remote mode: bind to the specific configured host address only,
// not 0.0.0.0, to limit exposure to the intended interface.
let addr = format!("{}:{}", host, OAUTH_CALLBACK_PORT);
TcpListener::bind(&addr).await.map_err(bind_error)
}
}
/// Wait for an OAuth callback and extract a query parameter value.
///
/// Listens for a GET request matching `path_prefix` (e.g., "/callback" or "/auth/callback"),
/// extracts the value of `param_name` (e.g., "code" or "token"), and shows a branded
/// landing page using `display_name` (e.g., "Google", "Notion", "NEAR AI").
///
/// When `expected_state` is `Some`, the callback's `state` query parameter is validated
/// against it to prevent CSRF attacks. If the state doesn't match, the callback is
/// rejected with an error page.
///
/// Times out after 5 minutes.
pub async fn wait_for_callback(
listener: TcpListener,
path_prefix: &str,
param_name: &str,
display_name: &str,
expected_state: Option<&str>,
) -> Result<String, OAuthCallbackError> {
let path_prefix = path_prefix.to_string();
let param_name = param_name.to_string();
let display_name = display_name.to_string();
let expected_state = expected_state.map(String::from);
tokio::time::timeout(Duration::from_secs(300), async move {
loop {
let (mut socket, _) = listener
.accept()
.await
.map_err(|e| OAuthCallbackError::Io(e.to_string()))?;
let mut reader = BufReader::new(&mut socket);
let mut request_line = String::new();
reader
.read_line(&mut request_line)
.await
.map_err(|e| OAuthCallbackError::Io(e.to_string()))?;
if let Some(path) = request_line.split_whitespace().nth(1)
&& path.starts_with(&path_prefix)
&& let Some(query) = path.split('?').nth(1)
{
// Check for error first
if query.contains("error=") {
let html = landing_html(&display_name, false);
let response = format!(
"HTTP/1.1 400 Bad Request\r\n\
Content-Type: text/html; charset=utf-8\r\n\
Connection: close\r\n\
\r\n\
{}",
html
);
let _ = socket.write_all(response.as_bytes()).await;
return Err(OAuthCallbackError::Denied);
}
// Parse all query params into a map for validation
let params: HashMap<&str, String> = query
.split('&')
.filter_map(|p| {
let mut parts = p.splitn(2, '=');
let key = parts.next()?;
let val = parts.next().unwrap_or("");
Some((
key,
urlencoding::decode(val)
.unwrap_or_else(|_| val.into())
.into_owned(),
))
})
.collect();
// Validate CSRF state parameter
if let Some(ref expected) = expected_state {
let actual = params.get("state").cloned().unwrap_or_default();
if actual != *expected {
let html = landing_html(&display_name, false);
let response = format!(
"HTTP/1.1 403 Forbidden\r\n\
Content-Type: text/html; charset=utf-8\r\n\
Connection: close\r\n\
\r\n\
{}",
html
);
let _ = socket.write_all(response.as_bytes()).await;
return Err(OAuthCallbackError::StateMismatch {
expected: expected.clone(),
actual,
});
}
}
// Look for the target parameter
if let Some(value) = params.get(param_name.as_str()) {
let html = landing_html(&display_name, true);
let response = format!(
"HTTP/1.1 200 OK\r\n\
Content-Type: text/html; charset=utf-8\r\n\
Connection: close\r\n\
\r\n\
{}",
html
);
let _ = socket.write_all(response.as_bytes()).await;
let _ = socket.shutdown().await;
return Ok(value.clone());
}
}
// Not the callback we're looking for
let response = "HTTP/1.1 404 Not Found\r\nConnection: close\r\n\r\n";
let _ = socket.write_all(response.as_bytes()).await;
}
})
.await
.map_err(|_| OAuthCallbackError::Timeout)?
}
/// Escape a string for safe interpolation into HTML content.
fn html_escape(s: &str) -> String {
let mut out = String::with_capacity(s.len());
for c in s.chars() {
match c {
'&' => out.push_str("&amp;"),
'<' => out.push_str("&lt;"),
'>' => out.push_str("&gt;"),
'"' => out.push_str("&quot;"),
'\'' => out.push_str("&#x27;"),
_ => out.push(c),
}
}
out
}
// Core OAuth callback infrastructure is defined in `crate::llm::oauth_helpers`
// and re-exported here for backward compatibility.
pub use crate::llm::oauth_helpers::{
OAUTH_CALLBACK_PORT, OAuthCallbackError, bind_callback_listener, callback_host, callback_url,
is_loopback_host, landing_html, wait_for_callback,
};
// ── Shared OAuth flow steps ─────────────────────────────────────────
@@ -598,93 +349,6 @@ pub async fn validate_oauth_token(
}
}
// ── Landing pages ───────────────────────────────────────────────────
pub fn landing_html(provider_name: &str, success: bool) -> String {
let safe_name = html_escape(provider_name);
let (icon, heading, subtitle, accent) = if success {
(
r##"<div style="width:64px;height:64px;border-radius:50%;background:#22c55e;display:flex;align-items:center;justify-content:center;margin:0 auto 24px">
<svg width="32" height="32" viewBox="0 0 24 24" fill="none" stroke="#fff" stroke-width="3" stroke-linecap="round" stroke-linejoin="round"><polyline points="20 6 9 17 4 12"/></svg>
</div>"##,
format!("{} Connected", safe_name),
"You can close this window and return to your terminal.",
"#22c55e",
)
} else {
(
r##"<div style="width:64px;height:64px;border-radius:50%;background:#ef4444;display:flex;align-items:center;justify-content:center;margin:0 auto 24px">
<svg width="32" height="32" viewBox="0 0 24 24" fill="none" stroke="#fff" stroke-width="3" stroke-linecap="round" stroke-linejoin="round"><line x1="18" y1="6" x2="6" y2="18"/><line x1="6" y1="6" x2="18" y2="18"/></svg>
</div>"##,
"Authorization Failed".to_string(),
"The request was denied. You can close this window and try again.",
"#ef4444",
)
};
format!(
r#"<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width,initial-scale=1">
<title>IronClaw - {heading}</title>
<style>
* {{ margin:0; padding:0; box-sizing:border-box }}
body {{
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif;
background: #0a0a0a;
color: #e5e5e5;
display: flex;
justify-content: center;
align-items: center;
min-height: 100vh;
}}
.card {{
text-align: center;
padding: 48px 40px;
max-width: 420px;
border: 1px solid #262626;
border-radius: 16px;
background: #141414;
}}
h1 {{
font-size: 22px;
font-weight: 600;
margin-bottom: 8px;
color: #fafafa;
}}
p {{
font-size: 14px;
color: #a3a3a3;
line-height: 1.5;
}}
.accent {{ color: {accent}; }}
.brand {{
margin-top: 32px;
font-size: 12px;
color: #525252;
letter-spacing: 0.5px;
text-transform: uppercase;
}}
</style>
</head>
<body>
<div class="card">
{icon}
<h1>{heading}</h1>
<p>{subtitle}</p>
<div class="brand">IronClaw</div>
</div>
</body>
</html>"#,
heading = heading,
icon = icon,
subtitle = subtitle,
accent = accent,
)
}
// ── Gateway callback support ─────────────────────────────────────────
/// State for an in-progress OAuth flow, keyed by CSRF `state` parameter.
+124 -2
View File
@@ -8,9 +8,35 @@ use std::path::PathBuf;
use crate::bootstrap::ironclaw_base_dir;
use crate::settings::Settings;
/// Load settings from JSON and TOML config files, matching the runtime
/// priority: TOML overlay > settings.json > defaults.
///
/// This mirrors the loading chain in `Config::from_env_with_toml()` but
/// without resolving the full `Config` (which requires async + secrets).
fn load_settings() -> Settings {
load_settings_from(&Settings::default_path(), &Settings::default_toml_path())
}
/// Inner implementation with injectable paths (testable).
fn load_settings_from(json_path: &std::path::Path, toml_path: &std::path::Path) -> Settings {
let mut settings = Settings::load_from(json_path);
match Settings::load_toml(toml_path) {
Ok(Some(toml_settings)) => {
settings.merge_from(&toml_settings);
}
Ok(None) => {} // File not found — fine for default path
Err(e) => {
eprintln!("Warning: failed to parse {}: {}", toml_path.display(), e);
}
}
settings
}
/// Run the status command, printing system health info.
pub async fn run_status_command() -> anyhow::Result<()> {
let settings = Settings::default();
let settings = load_settings();
println!("IronClaw Status");
println!("===============\n");
@@ -57,7 +83,7 @@ pub async fn run_status_command() -> anyhow::Result<()> {
// Session / Auth
print!(" Session: ");
let session_path = crate::llm::session::default_session_path();
let session_path = crate::config::llm::default_session_path();
if session_path.exists() {
println!("found ({})", session_path.display());
} else {
@@ -209,3 +235,99 @@ fn default_tools_dir() -> PathBuf {
fn default_channels_dir() -> PathBuf {
ironclaw_base_dir().join("channels")
}
#[cfg(test)]
mod tests {
use super::load_settings_from;
/// Regression test for #354: load_settings_from must read config.toml.
#[test]
fn reads_toml_heartbeat_enabled() {
let dir = tempfile::tempdir().expect("tempdir");
let json_path = dir.path().join("settings.json");
let toml_path = dir.path().join("config.toml");
// No JSON file — only TOML
std::fs::write(
&toml_path,
"[heartbeat]\nenabled = true\ninterval_secs = 600",
)
.expect("write toml");
let settings = load_settings_from(&json_path, &toml_path);
assert!(settings.heartbeat.enabled);
assert_eq!(settings.heartbeat.interval_secs, 600);
}
/// Without any config files, defaults are returned.
#[test]
fn defaults_without_config_files() {
let dir = tempfile::tempdir().expect("tempdir");
let settings = load_settings_from(
&dir.path().join("nonexistent.json"),
&dir.path().join("nonexistent.toml"),
);
assert!(!settings.heartbeat.enabled);
}
/// settings.json is respected.
#[test]
fn reads_json_heartbeat_enabled() {
let dir = tempfile::tempdir().expect("tempdir");
let json_path = dir.path().join("settings.json");
let toml_path = dir.path().join("nonexistent.toml");
std::fs::write(
&json_path,
r#"{"heartbeat":{"enabled":true,"interval_secs":900}}"#,
)
.expect("write json");
let settings = load_settings_from(&json_path, &toml_path);
assert!(settings.heartbeat.enabled);
assert_eq!(settings.heartbeat.interval_secs, 900);
}
/// TOML overlay wins over JSON settings.
#[test]
fn toml_overlay_wins_over_json() {
let dir = tempfile::tempdir().expect("tempdir");
let json_path = dir.path().join("settings.json");
let toml_path = dir.path().join("config.toml");
std::fs::write(
&json_path,
r#"{"heartbeat":{"enabled":false,"interval_secs":100}}"#,
)
.expect("write json");
std::fs::write(
&toml_path,
"[heartbeat]\nenabled = true\ninterval_secs = 200",
)
.expect("write toml");
let settings = load_settings_from(&json_path, &toml_path);
assert!(settings.heartbeat.enabled);
assert_eq!(settings.heartbeat.interval_secs, 200);
}
/// Invalid TOML is warned but doesn't crash; falls back to JSON/defaults.
#[test]
fn invalid_toml_falls_back_gracefully() {
let dir = tempfile::tempdir().expect("tempdir");
let json_path = dir.path().join("settings.json");
let toml_path = dir.path().join("config.toml");
std::fs::write(
&json_path,
r#"{"heartbeat":{"enabled":true,"interval_secs":500}}"#,
)
.expect("write json");
std::fs::write(&toml_path, "this is not valid toml [[[").expect("write bad toml");
let settings = load_settings_from(&json_path, &toml_path);
// Should fall back to JSON values, not crash
assert!(settings.heartbeat.enabled);
assert_eq!(settings.heartbeat.interval_secs, 500);
}
}
+2 -66
View File
@@ -10,12 +10,7 @@ use clap::Subcommand;
use tokio::fs;
use crate::bootstrap::ironclaw_base_dir;
use crate::config::Config;
#[allow(unused_imports)]
use crate::db::Database;
#[cfg(feature = "postgres")]
use crate::secrets::PostgresSecretsStore;
use crate::secrets::{CreateSecretParams, SecretsCrypto, SecretsStore};
use crate::secrets::{CreateSecretParams, SecretsStore};
use crate::tools::wasm::{CapabilitiesFile, compute_binary_hash};
/// Default tools directory.
@@ -556,66 +551,7 @@ fn validate_tool_name(name: &str) -> anyhow::Result<()> {
/// Initialize the secrets store from environment config.
async fn init_secrets_store() -> anyhow::Result<Arc<dyn SecretsStore + Send + Sync>> {
let config = Config::from_env().await?;
let master_key = config.secrets.master_key().ok_or_else(|| {
anyhow::anyhow!(
"SECRETS_MASTER_KEY not set. Run 'ironclaw onboard' first or set it in .env"
)
})?;
let crypto = SecretsCrypto::new(master_key.clone())?;
let store: Arc<dyn SecretsStore + Send + Sync> = {
#[cfg(feature = "postgres")]
{
let store = crate::history::Store::new(&config.database).await?;
store.run_migrations().await?;
Arc::new(PostgresSecretsStore::new(store.pool(), Arc::new(crypto)))
}
#[cfg(all(feature = "libsql", not(feature = "postgres")))]
{
use crate::db::Database as _;
use crate::db::libsql::LibSqlBackend;
use secrecy::ExposeSecret as _;
let default_path = crate::config::default_libsql_path();
let db_path = config
.database
.libsql_path
.as_deref()
.unwrap_or(&default_path);
let backend = if let Some(ref url) = config.database.libsql_url {
let token = config.database.libsql_auth_token.as_ref().ok_or_else(|| {
anyhow::anyhow!("LIBSQL_AUTH_TOKEN is required when LIBSQL_URL is set")
})?;
LibSqlBackend::new_remote_replica(db_path, url, token.expose_secret())
.await
.map_err(|e| anyhow::anyhow!("{}", e))?
} else {
LibSqlBackend::new_local(db_path)
.await
.map_err(|e| anyhow::anyhow!("{}", e))?
};
backend
.run_migrations()
.await
.map_err(|e| anyhow::anyhow!("{}", e))?;
Arc::new(crate::secrets::LibSqlSecretsStore::new(
backend.shared_db(),
Arc::new(crypto),
))
}
#[cfg(not(any(feature = "postgres", feature = "libsql")))]
{
let _ = crypto;
anyhow::bail!(
"No database backend available for secrets. Enable 'postgres' or 'libsql' feature."
);
}
};
Ok(store)
crate::cli::init_secrets_store().await
}
/// Configure authentication for a tool.
+44
View File
@@ -27,6 +27,10 @@ pub struct AgentConfig {
pub max_tool_iterations: usize,
/// When true, skip tool approval checks entirely. For benchmarks/CI.
pub auto_approve_tools: bool,
/// Default timezone for new sessions (IANA name, e.g. "America/New_York").
pub default_timezone: String,
/// Maximum tokens per job (0 = unlimited).
pub max_tokens_per_job: u64,
}
impl AgentConfig {
@@ -47,6 +51,8 @@ impl AgentConfig {
max_actions_per_hour: None,
max_tool_iterations: 10,
auto_approve_tools: true,
default_timezone: "UTC".to_string(),
max_tokens_per_job: 0,
}
}
@@ -89,6 +95,44 @@ impl AgentConfig {
"AGENT_AUTO_APPROVE_TOOLS",
settings.agent.auto_approve_tools,
)?,
default_timezone: {
let tz: String = parse_optional_env(
"DEFAULT_TIMEZONE",
settings.agent.default_timezone.clone(),
)?;
if crate::timezone::parse_timezone(&tz).is_none() {
return Err(ConfigError::InvalidValue {
key: "DEFAULT_TIMEZONE".into(),
message: format!("invalid IANA timezone: '{tz}'"),
});
}
tz
},
max_tokens_per_job: parse_optional_env(
"AGENT_MAX_TOKENS_PER_JOB",
settings.agent.max_tokens_per_job,
)?,
})
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_default_timezone_rejects_invalid() {
let mut settings = Settings::default();
settings.agent.default_timezone = "Fake/Zone".to_string();
let result = AgentConfig::resolve(&settings);
assert!(result.is_err(), "invalid IANA timezone should be rejected");
}
#[test]
fn test_default_timezone_accepts_valid() {
let settings = Settings::default(); // default is "UTC"
let config = AgentConfig::resolve(&settings).expect("resolve");
assert_eq!(config.default_timezone, "UTC");
}
}
+6 -5
View File
@@ -100,13 +100,13 @@ impl EmbeddingsConfig {
session: Arc<SessionManager>,
) -> Option<Arc<dyn EmbeddingProvider>> {
if !self.enabled {
tracing::info!("Embeddings disabled (set EMBEDDING_ENABLED=true to enable)");
tracing::debug!("Embeddings disabled (set EMBEDDING_ENABLED=true to enable)");
return None;
}
match self.provider.as_str() {
"nearai" => {
tracing::info!(
tracing::debug!(
"Embeddings enabled via NEAR AI (model: {}, dim: {})",
self.model,
self.dimension,
@@ -117,7 +117,7 @@ impl EmbeddingsConfig {
))
}
"ollama" => {
tracing::info!(
tracing::debug!(
"Embeddings enabled via Ollama (model: {}, url: {}, dim: {})",
self.model,
self.ollama_base_url,
@@ -130,7 +130,7 @@ impl EmbeddingsConfig {
}
_ => {
if let Some(api_key) = self.openai_api_key() {
tracing::info!(
tracing::debug!(
"Embeddings enabled via OpenAI (model: {}, dim: {})",
self.model,
self.dimension,
@@ -154,6 +154,7 @@ mod tests {
use super::*;
use crate::config::helpers::ENV_MUTEX;
use crate::settings::{EmbeddingsSettings, Settings};
use crate::testing::credentials::*;
/// Clear all embedding-related env vars.
fn clear_embedding_env() {
@@ -173,7 +174,7 @@ mod tests {
clear_embedding_env();
// SAFETY: Under ENV_MUTEX, no concurrent env access.
unsafe {
std::env::set_var("OPENAI_API_KEY", "sk-test-key-for-issue-129");
std::env::set_var("OPENAI_API_KEY", TEST_OPENAI_API_KEY_ISSUE_129);
}
let settings = Settings {
+102 -1
View File
@@ -1,4 +1,4 @@
use crate::config::helpers::{optional_env, parse_bool_env, parse_optional_env};
use crate::config::helpers::{optional_env, parse_bool_env, parse_option_env, parse_optional_env};
use crate::error::ConfigError;
use crate::settings::Settings;
@@ -13,6 +13,12 @@ pub struct HeartbeatConfig {
pub notify_channel: Option<String>,
/// User ID to notify on heartbeat findings.
pub notify_user: Option<String>,
/// Hour (0-23) when quiet hours start.
pub quiet_hours_start: Option<u32>,
/// Hour (0-23) when quiet hours end.
pub quiet_hours_end: Option<u32>,
/// Timezone for quiet hours evaluation (IANA name).
pub timezone: Option<String>,
}
impl Default for HeartbeatConfig {
@@ -22,6 +28,9 @@ impl Default for HeartbeatConfig {
interval_secs: 1800, // 30 minutes
notify_channel: None,
notify_user: None,
quiet_hours_start: None,
quiet_hours_end: None,
timezone: None,
}
}
}
@@ -38,6 +47,98 @@ impl HeartbeatConfig {
.or_else(|| settings.heartbeat.notify_channel.clone()),
notify_user: optional_env("HEARTBEAT_NOTIFY_USER")?
.or_else(|| settings.heartbeat.notify_user.clone()),
quiet_hours_start: parse_option_env::<u32>("HEARTBEAT_QUIET_START")?
.or(settings.heartbeat.quiet_hours_start)
.map(|h| {
if h > 23 {
return Err(ConfigError::InvalidValue {
key: "HEARTBEAT_QUIET_START".into(),
message: "must be 0-23".into(),
});
}
Ok(h)
})
.transpose()?,
quiet_hours_end: parse_option_env::<u32>("HEARTBEAT_QUIET_END")?
.or(settings.heartbeat.quiet_hours_end)
.map(|h| {
if h > 23 {
return Err(ConfigError::InvalidValue {
key: "HEARTBEAT_QUIET_END".into(),
message: "must be 0-23".into(),
});
}
Ok(h)
})
.transpose()?,
timezone: {
let tz = optional_env("HEARTBEAT_TIMEZONE")?
.or_else(|| settings.heartbeat.timezone.clone());
if let Some(ref tz_str) = tz
&& crate::timezone::parse_timezone(tz_str).is_none()
{
return Err(ConfigError::InvalidValue {
key: "HEARTBEAT_TIMEZONE".into(),
message: format!("invalid IANA timezone: '{tz_str}'"),
});
}
tz
},
})
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_quiet_hours_settings_fallback() {
// When env vars are not set, settings values should be used
let mut settings = Settings::default();
settings.heartbeat.quiet_hours_start = Some(22);
settings.heartbeat.quiet_hours_end = Some(6);
let config = HeartbeatConfig::resolve(&settings).expect("resolve");
assert_eq!(config.quiet_hours_start, Some(22));
assert_eq!(config.quiet_hours_end, Some(6));
}
#[test]
fn test_quiet_hours_rejects_invalid_hour() {
let mut settings = Settings::default();
settings.heartbeat.quiet_hours_start = Some(24);
let result = HeartbeatConfig::resolve(&settings);
assert!(result.is_err());
}
#[test]
fn test_quiet_hours_accepts_boundary_values() {
let mut settings = Settings::default();
settings.heartbeat.quiet_hours_start = Some(0);
settings.heartbeat.quiet_hours_end = Some(23);
let config = HeartbeatConfig::resolve(&settings).expect("resolve");
assert_eq!(config.quiet_hours_start, Some(0));
assert_eq!(config.quiet_hours_end, Some(23));
}
#[test]
fn test_heartbeat_timezone_rejects_invalid() {
let mut settings = Settings::default();
settings.heartbeat.timezone = Some("Fake/Zone".to_string());
let result = HeartbeatConfig::resolve(&settings);
assert!(result.is_err(), "invalid IANA timezone should be rejected");
}
#[test]
fn test_heartbeat_timezone_accepts_valid() {
let mut settings = Settings::default();
settings.heartbeat.timezone = Some("America/New_York".to_string());
let config = HeartbeatConfig::resolve(&settings).expect("resolve");
assert_eq!(config.timezone.as_deref(), Some("America/New_York"));
}
}
+7 -2
View File
@@ -25,8 +25,13 @@ pub(crate) fn optional_env(key: &str) -> Result<Option<String>, ConfigError> {
}
// Fall back to thread-safe overlay (secrets injected from DB)
if let Some(val) = INJECTED_VARS.get().and_then(|map| map.get(key)) {
return Ok(Some(val.clone()));
if let Some(val) = INJECTED_VARS
.lock()
.unwrap_or_else(|p| p.into_inner())
.get(key)
.cloned()
{
return Ok(Some(val));
}
Ok(None)
+228 -129
View File
@@ -5,128 +5,11 @@ use secrecy::SecretString;
use crate::bootstrap::ironclaw_base_dir;
use crate::config::helpers::{optional_env, parse_optional_env};
use crate::error::ConfigError;
use crate::llm::config::*;
use crate::llm::registry::{ProviderProtocol, ProviderRegistry};
use crate::llm::session::SessionConfig;
use crate::settings::Settings;
/// Prompt cache retention policy for Anthropic.
///
/// Controls Anthropic's automatic prompt caching via a top-level
/// `cache_control` field injected through rig-core's `additional_params`.
/// - `None` — caching disabled, no `cache_control` injected.
/// - `Short` — 5-minute TTL (default), `{"type": "ephemeral"}`, 1.25× write surcharge.
/// - `Long` — 1-hour TTL, `{"type": "ephemeral", "ttl": "1h"}`, 2× write surcharge.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
pub enum CacheRetention {
/// No prompt caching.
None,
/// 5-minute TTL (default). Write cost: 1.25× base input.
#[default]
Short,
/// 1-hour TTL. Write cost: 2× base input.
Long,
}
impl std::str::FromStr for CacheRetention {
type Err = String;
fn from_str(s: &str) -> Result<Self, Self::Err> {
match s.to_lowercase().as_str() {
"none" | "off" | "disabled" => Ok(Self::None),
"short" | "5m" | "ephemeral" => Ok(Self::Short),
"long" | "1h" => Ok(Self::Long),
_ => Err(format!(
"invalid cache retention '{}', expected one of: none, short, long",
s
)),
}
}
}
impl std::fmt::Display for CacheRetention {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
Self::None => write!(f, "none"),
Self::Short => write!(f, "short"),
Self::Long => write!(f, "long"),
}
}
}
/// Resolved configuration for a registry-based provider.
///
/// This single struct replaces what used to be five separate config types
/// (`OpenAiDirectConfig`, `AnthropicDirectConfig`, `OllamaConfig`,
/// `OpenAiCompatibleConfig`, `TinfoilConfig`). The `protocol` field
/// determines which rig-core client constructor to use.
#[derive(Debug, Clone)]
pub struct RegistryProviderConfig {
/// Which API protocol to use (determines the rig-core client).
pub protocol: ProviderProtocol,
/// Provider identifier (e.g., "groq", "openai", "tinfoil").
pub provider_id: String,
/// API key (optional for some providers like Ollama).
pub api_key: Option<SecretString>,
/// Base URL for the API endpoint.
pub base_url: String,
/// Model identifier.
pub model: String,
/// Extra HTTP headers injected into every request.
pub extra_headers: Vec<(String, String)>,
}
/// LLM provider configuration.
///
/// NearAI remains the default backend with its own config struct (session auth).
/// All other providers are resolved through the provider registry, producing
/// a generic `RegistryProviderConfig`.
#[derive(Debug, Clone)]
pub struct LlmConfig {
/// Backend identifier (e.g., "nearai", "openai", "groq", "tinfoil").
pub backend: String,
/// Session manager configuration (auth URL, token persistence path).
/// Used by the NearAI provider for OAuth/session-token auth.
pub session: SessionConfig,
/// NEAR AI config (always populated, also used for embeddings).
pub nearai: NearAiConfig,
/// Resolved provider config for registry-based providers.
/// `None` when backend is "nearai".
pub provider: Option<RegistryProviderConfig>,
}
/// NEAR AI configuration.
#[derive(Debug, Clone)]
pub struct NearAiConfig {
/// Model to use (e.g., "claude-3-5-sonnet-20241022", "gpt-4o")
pub model: String,
/// Cheap/fast model for lightweight tasks (heartbeat, routing, evaluation).
pub cheap_model: Option<String>,
/// Base URL for the NEAR AI API.
pub base_url: String,
/// API key for NEAR AI Cloud.
pub api_key: Option<SecretString>,
/// Optional fallback model for failover.
pub fallback_model: Option<String>,
/// Maximum number of retries for transient errors (default: 3).
pub max_retries: u32,
/// Consecutive failures before circuit breaker opens. None = disabled.
pub circuit_breaker_threshold: Option<u32>,
/// Seconds the circuit stays open before probing (default: 30).
pub circuit_breaker_recovery_secs: u64,
/// Enable in-memory response caching. Default: false.
pub response_cache_enabled: bool,
/// TTL in seconds for cached responses (default: 3600).
pub response_cache_ttl_secs: u64,
/// Max cached responses before LRU eviction (default: 1000).
pub response_cache_max_entries: usize,
/// Cooldown duration in seconds for failover (default: 300).
pub failover_cooldown_secs: u64,
/// Consecutive failures before failover cooldown (default: 3).
pub failover_cooldown_threshold: u32,
/// Enable cascade mode for smart routing. Default: true.
pub smart_routing_cascade: bool,
}
impl LlmConfig {
/// Create a test-friendly config without reading env vars.
#[cfg(feature = "libsql")]
@@ -154,6 +37,8 @@ impl LlmConfig {
smart_routing_cascade: false,
},
provider: None,
bedrock: None,
request_timeout_secs: 120,
}
}
@@ -184,8 +69,10 @@ impl LlmConfig {
let backend_lower = backend.to_lowercase();
let is_nearai =
backend_lower == "nearai" || backend_lower == "near_ai" || backend_lower == "near";
let is_bedrock =
backend_lower == "bedrock" || backend_lower == "aws_bedrock" || backend_lower == "aws";
if !is_nearai && registry.find(&backend_lower).is_none() {
if !is_nearai && !is_bedrock && registry.find(&backend_lower).is_none() {
tracing::warn!(
"Unknown LLM backend '{}'. Will attempt as openai_compatible fallback.",
backend
@@ -232,8 +119,8 @@ impl LlmConfig {
smart_routing_cascade: parse_optional_env("SMART_ROUTING_CASCADE", true)?,
};
// Resolve registry provider config (for non-NearAI backends)
let provider = if is_nearai {
// Resolve registry provider config (for non-NearAI, non-Bedrock backends)
let provider = if is_nearai || is_bedrock {
None
} else {
Some(Self::resolve_registry_provider(
@@ -243,9 +130,50 @@ impl LlmConfig {
)?)
};
let bedrock = if is_bedrock {
let explicit_region =
optional_env("BEDROCK_REGION")?.or_else(|| settings.bedrock_region.clone());
if explicit_region.is_none() {
tracing::info!("BEDROCK_REGION not set, defaulting to us-east-1");
}
let region = explicit_region.unwrap_or_else(|| "us-east-1".to_string());
let model = optional_env("BEDROCK_MODEL")?
.or_else(|| settings.selected_model.clone())
.ok_or_else(|| ConfigError::MissingRequired {
key: "BEDROCK_MODEL".to_string(),
hint: "Set BEDROCK_MODEL when LLM_BACKEND=bedrock".to_string(),
})?;
let cross_region = optional_env("BEDROCK_CROSS_REGION")?
.or_else(|| settings.bedrock_cross_region.clone());
if let Some(ref cr) = cross_region
&& !matches!(cr.as_str(), "us" | "eu" | "apac" | "global")
{
return Err(ConfigError::InvalidValue {
key: "BEDROCK_CROSS_REGION".to_string(),
message: format!(
"'{}' is not valid, expected one of: us, eu, apac, global",
cr
),
});
}
let profile = optional_env("AWS_PROFILE")?.or_else(|| settings.bedrock_profile.clone());
Some(BedrockConfig {
region,
model,
cross_region,
profile,
})
} else {
None
};
let request_timeout_secs = parse_optional_env("LLM_REQUEST_TIMEOUT_SECS", 120)?;
Ok(Self {
backend: if is_nearai {
"nearai".to_string()
} else if is_bedrock {
"bedrock".to_string()
} else if let Some(ref p) = provider {
p.provider_id.clone()
} else {
@@ -254,6 +182,8 @@ impl LlmConfig {
session,
nearai,
provider,
bedrock,
request_timeout_secs,
})
}
@@ -366,6 +296,39 @@ impl LlmConfig {
Vec::new()
};
// Resolve OAuth token (Anthropic-specific: `claude login` flow).
// Only check for OAuth token when the provider is actually Anthropic.
let oauth_token = if canonical_id == "anthropic" {
optional_env("ANTHROPIC_OAUTH_TOKEN")?.map(SecretString::from)
} else {
None
};
let api_key = if api_key.is_none() && oauth_token.is_some() {
// OAuth token present but no API key: use a placeholder so the
// config block is populated. The provider factory will route to
// the OAuth provider instead of rig-core's x-api-key client.
Some(SecretString::from(OAUTH_PLACEHOLDER.to_string()))
} else {
api_key
};
// Resolve Anthropic prompt cache retention from env (default: Short).
let cache_retention: CacheRetention = if canonical_id == "anthropic" {
optional_env("ANTHROPIC_CACHE_RETENTION")?
.and_then(|val| match val.parse::<CacheRetention>() {
Ok(r) => Some(r),
Err(e) => {
tracing::warn!(
"Invalid ANTHROPIC_CACHE_RETENTION: {e}; defaulting to short"
);
None
}
})
.unwrap_or_default()
} else {
CacheRetention::default()
};
Ok(RegistryProviderConfig {
protocol,
provider_id: canonical_id.to_string(),
@@ -373,6 +336,8 @@ impl LlmConfig {
base_url,
model,
extra_headers,
oauth_token,
cache_retention,
})
}
}
@@ -411,7 +376,7 @@ fn parse_extra_headers(val: &str) -> Result<Vec<(String, String)>, ConfigError>
}
/// Get the default session file path (~/.ironclaw/session.json).
fn default_session_path() -> PathBuf {
pub fn default_session_path() -> PathBuf {
ironclaw_base_dir().join("session.json")
}
@@ -420,6 +385,7 @@ mod tests {
use super::*;
use crate::config::helpers::ENV_MUTEX;
use crate::settings::Settings;
use crate::testing::credentials::*;
/// Clear all openai-compatible-related env vars.
fn clear_openai_compatible_env() {
@@ -677,14 +643,12 @@ mod tests {
#[test]
fn backend_alias_normalized_to_canonical_id() {
// When the user sets LLM_BACKEND to an alias (e.g., "open_ai"),
// LlmConfig.backend should resolve to the canonical ID ("openai").
let _guard = ENV_MUTEX.lock().expect("env mutex poisoned");
clear_openai_compatible_env();
// SAFETY: Under ENV_MUTEX.
unsafe {
std::env::set_var("LLM_BACKEND", "open_ai");
std::env::set_var("OPENAI_API_KEY", "test-key");
std::env::set_var("OPENAI_API_KEY", TEST_API_KEY);
}
let settings = Settings::default();
@@ -705,8 +669,6 @@ mod tests {
#[test]
fn unknown_backend_falls_back_to_openai_compatible() {
// An unrecognized LLM_BACKEND should fall back to the openai_compatible
// provider definition instead of erroring.
let _guard = ENV_MUTEX.lock().expect("env mutex poisoned");
clear_openai_compatible_env();
// SAFETY: Under ENV_MUTEX.
@@ -717,7 +679,6 @@ mod tests {
let settings = Settings::default();
let cfg = LlmConfig::resolve(&settings).expect("resolve should succeed");
// Falls back to openai_compatible since "some_custom_provider" is unknown
assert_eq!(cfg.backend, "openai_compatible");
let provider = cfg.provider.expect("should have provider config");
assert_eq!(provider.provider_id, "openai_compatible");
@@ -759,7 +720,6 @@ mod tests {
#[test]
fn base_url_resolution_priority() {
// Env var > settings > registry default
let _guard = ENV_MUTEX.lock().expect("env mutex poisoned");
clear_openai_compatible_env();
@@ -800,6 +760,119 @@ mod tests {
}
}
// ── OAuth resolution tests ──────────────────────────────────────
/// Clear all Anthropic-related env vars.
fn clear_anthropic_env() {
// SAFETY: Only called under ENV_MUTEX in tests.
unsafe {
std::env::remove_var("LLM_BACKEND");
std::env::remove_var("ANTHROPIC_API_KEY");
std::env::remove_var("ANTHROPIC_OAUTH_TOKEN");
std::env::remove_var("ANTHROPIC_MODEL");
std::env::remove_var("ANTHROPIC_BASE_URL");
}
}
#[test]
fn anthropic_oauth_token_sets_placeholder_api_key() {
use secrecy::ExposeSecret;
let _guard = ENV_MUTEX.lock().expect("env mutex poisoned");
clear_anthropic_env();
// SAFETY: Under ENV_MUTEX.
unsafe {
std::env::set_var("ANTHROPIC_OAUTH_TOKEN", TEST_ANTHROPIC_OAUTH_TOKEN);
}
let settings = Settings {
llm_backend: Some("anthropic".to_string()),
..Default::default()
};
let cfg = LlmConfig::resolve(&settings).expect("resolve should succeed");
let provider = cfg.provider.expect("provider config should be present");
assert_eq!(
provider
.api_key
.as_ref()
.map(|k| k.expose_secret().to_string()),
Some(OAUTH_PLACEHOLDER.to_string()),
"api_key should be the OAuth placeholder when only OAuth token is set"
);
assert!(
provider.oauth_token.is_some(),
"oauth_token should be populated"
);
assert_eq!(
provider.oauth_token.as_ref().unwrap().expose_secret(),
TEST_ANTHROPIC_OAUTH_TOKEN
);
clear_anthropic_env();
}
#[test]
fn anthropic_api_key_takes_priority_over_oauth() {
use secrecy::ExposeSecret;
let _guard = ENV_MUTEX.lock().expect("env mutex poisoned");
clear_anthropic_env();
// SAFETY: Under ENV_MUTEX.
unsafe {
std::env::set_var("ANTHROPIC_API_KEY", TEST_ANTHROPIC_API_KEY);
std::env::set_var("ANTHROPIC_OAUTH_TOKEN", TEST_ANTHROPIC_OAUTH_TOKEN);
}
let settings = Settings {
llm_backend: Some("anthropic".to_string()),
..Default::default()
};
let cfg = LlmConfig::resolve(&settings).expect("resolve should succeed");
let provider = cfg.provider.expect("provider config should be present");
assert_eq!(
provider
.api_key
.as_ref()
.map(|k| k.expose_secret().to_string()),
Some(TEST_ANTHROPIC_API_KEY.to_string()),
"real API key should take priority over OAuth placeholder"
);
assert!(
provider.oauth_token.is_some(),
"oauth_token should still be populated"
);
clear_anthropic_env();
}
#[test]
fn non_anthropic_provider_has_no_oauth_token() {
let _guard = ENV_MUTEX.lock().expect("env mutex poisoned");
clear_anthropic_env();
// SAFETY: Under ENV_MUTEX.
unsafe {
std::env::set_var("ANTHROPIC_OAUTH_TOKEN", TEST_ANTHROPIC_OAUTH_TOKEN);
}
let settings = Settings {
llm_backend: Some("openai".to_string()),
..Default::default()
};
let cfg = LlmConfig::resolve(&settings).expect("resolve should succeed");
let provider = cfg.provider.expect("provider config should be present");
assert!(
provider.oauth_token.is_none(),
"non-Anthropic providers should not pick up ANTHROPIC_OAUTH_TOKEN"
);
clear_anthropic_env();
}
// ── Cache retention tests ───────────────────────────────────────
#[test]
fn cache_retention_from_str_primary_values() {
assert_eq!(
@@ -881,4 +954,30 @@ mod tests {
assert_eq!(parsed, variant, "round-trip failed for {s}");
}
}
#[test]
fn test_request_timeout_defaults_to_120() {
let _guard = ENV_MUTEX.lock().expect("env mutex poisoned");
// SAFETY: Under ENV_MUTEX.
unsafe {
std::env::remove_var("LLM_REQUEST_TIMEOUT_SECS");
}
let config = LlmConfig::resolve(&Settings::default()).expect("resolve");
assert_eq!(config.request_timeout_secs, 120);
}
#[test]
fn test_request_timeout_configurable() {
let _guard = ENV_MUTEX.lock().expect("env mutex poisoned");
// SAFETY: Under ENV_MUTEX.
unsafe {
std::env::set_var("LLM_REQUEST_TIMEOUT_SECS", "300");
}
let config = LlmConfig::resolve(&Settings::default()).expect("resolve");
assert_eq!(config.request_timeout_secs, 300);
// SAFETY: Cleanup
unsafe {
std::env::remove_var("LLM_REQUEST_TIMEOUT_SECS");
}
}
}
+104 -6
View File
@@ -13,7 +13,7 @@ mod embeddings;
mod heartbeat;
pub(crate) mod helpers;
mod hygiene;
mod llm;
pub(crate) mod llm;
mod routines;
mod safety;
mod sandbox;
@@ -24,7 +24,7 @@ mod tunnel;
mod wasm;
use std::collections::HashMap;
use std::sync::OnceLock;
use std::sync::{LazyLock, Mutex};
use crate::error::ConfigError;
use crate::settings::Settings;
@@ -37,7 +37,7 @@ pub use self::database::{DatabaseBackend, DatabaseConfig, SslMode, default_libsq
pub use self::embeddings::EmbeddingsConfig;
pub use self::heartbeat::HeartbeatConfig;
pub use self::hygiene::HygieneConfig;
pub use self::llm::{CacheRetention, LlmConfig, NearAiConfig, RegistryProviderConfig};
pub use self::llm::default_session_path;
pub use self::routines::RoutineConfig;
pub use self::safety::SafetyConfig;
pub use self::sandbox::{ClaudeCodeConfig, SandboxModeConfig};
@@ -46,6 +46,10 @@ pub use self::skills::SkillsConfig;
pub use self::transcription::TranscriptionConfig;
pub use self::tunnel::TunnelConfig;
pub use self::wasm::WasmConfig;
pub use crate::llm::config::{
BedrockConfig, CacheRetention, LlmConfig, NearAiConfig, OAUTH_PLACEHOLDER,
RegistryProviderConfig,
};
pub use crate::llm::session::SessionConfig;
/// Thread-safe overlay for injected env vars (secrets loaded from DB).
@@ -53,7 +57,12 @@ pub use crate::llm::session::SessionConfig;
/// Used by `inject_llm_keys_from_secrets()` to make API keys available to
/// `optional_env()` without unsafe `set_var` calls. `optional_env()` checks
/// real env vars first, then falls back to this overlay.
static INJECTED_VARS: OnceLock<HashMap<String, String>> = OnceLock::new();
///
/// Uses `Mutex<HashMap>` instead of `OnceLock` so that both
/// `inject_os_credentials()` and `inject_llm_keys_from_secrets()` can merge
/// their data. Whichever runs first initialises the map; the second merges in.
static INJECTED_VARS: LazyLock<Mutex<HashMap<String, String>>> =
LazyLock::new(|| Mutex::new(HashMap::new()));
/// Main configuration for the agent.
#[derive(Debug, Clone)]
@@ -252,6 +261,32 @@ impl Config {
Ok(())
}
/// Re-resolve only the LLM config after credential injection.
///
/// Called by `AppBuilder::init_secrets()` after injecting API keys into
/// the env overlay. Only rebuilds `self.llm` — all other config fields
/// are unaffected, preserving values from the initial config load (or
/// from `Config::for_testing()` in test mode).
pub async fn re_resolve_llm(
&mut self,
store: Option<&(dyn crate::db::SettingsStore + Sync)>,
user_id: &str,
toml_path: Option<&std::path::Path>,
) -> Result<(), ConfigError> {
let settings = if let Some(store) = store {
let mut s = match store.get_all_settings(user_id).await {
Ok(map) => Settings::from_db_map(&map),
Err(_) => Settings::default(),
};
Self::apply_toml_overlay(&mut s, toml_path)?;
s
} else {
Settings::default()
};
self.llm = LlmConfig::resolve(&settings)?;
Ok(())
}
/// Build config from settings (shared by from_env and from_db).
async fn build(settings: &Settings) -> Result<Self, ConfigError> {
Ok(Self {
@@ -285,6 +320,9 @@ impl Config {
/// env-var-first resolution in `LlmConfig::resolve()`. Keys in the overlay
/// are read by `optional_env()` before falling back to `std::env::var()`,
/// so explicit env vars always win.
///
/// Also loads tokens from OS credential stores (macOS Keychain, Linux
/// credentials files) which don't require the secrets DB.
pub async fn inject_llm_keys_from_secrets(
secrets: &dyn crate::secrets::SecretsStore,
user_id: &str,
@@ -292,7 +330,10 @@ pub async fn inject_llm_keys_from_secrets(
// Static mappings for well-known providers.
// The registry's setup hints define secret_name -> env_var mappings,
// so new providers added to providers.json get injection automatically.
let mut mappings: Vec<(&str, &str)> = vec![("llm_nearai_api_key", "NEARAI_API_KEY")];
let mut mappings: Vec<(&str, &str)> = vec![
("llm_nearai_api_key", "NEARAI_API_KEY"),
("llm_anthropic_oauth_token", "ANTHROPIC_OAUTH_TOKEN"),
];
// Dynamically discover secret->env mappings from the provider registry.
// Uses selectable() which deduplicates user overrides correctly.
@@ -331,5 +372,62 @@ pub async fn inject_llm_keys_from_secrets(
}
}
let _ = INJECTED_VARS.set(injected);
inject_os_credential_store_tokens(&mut injected);
merge_injected_vars(injected);
}
/// Load tokens from OS credential stores (no DB required).
///
/// Called unconditionally during startup — even when the encrypted secrets DB
/// is unavailable (no master key, no DB connection). This ensures OAuth tokens
/// from `claude login` (macOS Keychain / Linux credentials.json)
/// are available for config resolution.
pub fn inject_os_credentials() {
let mut injected = HashMap::new();
inject_os_credential_store_tokens(&mut injected);
merge_injected_vars(injected);
}
/// Merge new entries into the global injected-vars overlay.
///
/// New keys are inserted; existing keys are overwritten (later callers win,
/// e.g. fresh OS credential store tokens override stale DB copies).
fn merge_injected_vars(new_entries: HashMap<String, String>) {
if new_entries.is_empty() {
return;
}
match INJECTED_VARS.lock() {
Ok(mut map) => map.extend(new_entries),
Err(poisoned) => poisoned.into_inner().extend(new_entries),
}
}
/// Inject a single key-value pair into the overlay.
///
/// Used by the setup wizard to make credentials available to `optional_env()`
/// without calling `unsafe { std::env::set_var }`.
pub fn inject_single_var(key: &str, value: &str) {
match INJECTED_VARS.lock() {
Ok(mut map) => {
map.insert(key.to_string(), value.to_string());
}
Err(poisoned) => {
poisoned
.into_inner()
.insert(key.to_string(), value.to_string());
}
}
}
/// Shared helper: extract tokens from OS credential stores into the overlay map.
fn inject_os_credential_store_tokens(injected: &mut HashMap<String, String>) {
// Try the OS credential store for a fresh Anthropic OAuth token.
// Tokens from `claude login` expire in 8-12h, so the DB copy may be stale.
// A fresh extraction from macOS Keychain / Linux credentials.json wins
// over the (possibly expired) copy stored in the encrypted secrets DB.
if let Some(fresh) = crate::config::ClaudeCodeConfig::extract_oauth_token() {
injected.insert("ANTHROPIC_OAUTH_TOKEN".to_string(), fresh);
tracing::debug!("Refreshed ANTHROPIC_OAUTH_TOKEN from OS credential store");
}
}

Some files were not shown because too many files have changed in this diff Show More