Compare commits

..
Author SHA1 Message Date
github-actions[bot]GitHubgithub-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>Illia Polosukhin
1f18422b88 chore: release v0.8.0 (#249)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Illia Polosukhin <[email protected]>
2026-02-20 20:45:42 +00:00
448383cfb0 refactor: remove Responses API, consolidate to Chat Completions (#272)
* fix: strip reasoning from LLM responses and persist assistant messages reliably

- Filter out `type: "reasoning"` output items from NEAR AI Responses API
  parsing so chain-of-thought never reaches the UI (nearai.rs)
- Rewrite clean_response with regex-based tag stripping that is
  code-aware (preserves tags inside fenced blocks and inline backticks),
  supports 9+ tag names (think, thought, reasoning, reflection, etc.),
  handles <final> extraction, pipe-delimited tags, and case/whitespace
  tolerance (reasoning.rs)
- Add Reasoning::complete() helper so all non-agentic LLM call sites
  (summarize, suggest, heartbeat, compaction) get automatic response
  cleaning; thread SafetyLayer through to those callers
- Change persist_turn from fire-and-forget tokio::spawn to awaited async
  so both user and assistant messages are written before returning,
  preventing data loss on shutdown/restart
- Pass input_count through seed_response_chain so response chaining
  delta calculation is accurate after thread hydration on restart
- Make NearAiResponse.usage optional and preserve response_id in alt
  response path for chaining continuity
- Persist session token to DB during onboarding wizard so runtime
  loads it without legacy-key fallback; suppress spurious warning on
  fresh installs
- Fix dev tool double-registration when builder already registers them
- Load dotenv/ironclaw env for doctor and status subcommands
- Reduce startup log noise (demote info→debug for skills, remove
  redundant info lines)

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* Nudge to not loop over tools continuesly

* refactor: remove Responses API, consolidate NEAR AI to Chat Completions only

The Responses API provider (nearai.rs, 1278 lines) added significant complexity
(response chaining state machine, delta message calculation, previous_response_id
persistence) for marginal benefit. This consolidates to the Chat Completions API
only, upgrading NearAiChatProvider with dual auth (session token + API key) and
401 retry for session token renewal.

- Delete src/llm/nearai.rs (Responses API provider)
- Upgrade nearai_chat.rs with SessionManager, dual auth, flexible list_models
- Remove response_id from CompletionResponse and ToolCompletionResponse
- Remove seed_response_chain/get_response_chain_id from LlmProvider trait
- Remove response chain persistence from agent (thread_ops, session)
- Remove NearAiApiMode enum and NEARAI_API_MODE config
- Clean up all wrapper providers (retry, circuit_breaker, failover, cache)
- Update documentation (CLAUDE.md, .env.example)

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* feat: runtime log level control via gateway UI and URL parameter

Add server-side log level switching using tracing_subscriber::reload::Layer
so the EnvFilter can be swapped at runtime without restarting. Expose via
GET/PUT /api/logs/level endpoints, a "Server: LEVEL" dropdown in the logs
toolbar, and a ?log_level=debug URL parameter for one-click activation.

Also applies cargo fmt to pre-existing files (llm/, tests/).

Co-Authored-By: Claude Opus 4.6 <[email protected]>

---------

Co-authored-by: Claude Opus 4.6 <[email protected]>
2026-02-20 20:43:32 +00:00
7df356c109 fix: persist WASM channel workspace writes across callbacks (#264)
* fix: persist WASM channel workspace writes across callbacks

WASM channel callbacks (polling, webhooks, on_start) call
workspace_write() to persist state, but the host code never committed
these writes — take_pending_writes() was never called. Additionally,
no WorkspaceReader was injected into channel capabilities, so
workspace_read() always returned None.

This caused Telegram's polling offset to reset to 0 on every tick,
making getUpdates re-deliver already-processed messages and producing
2-4 duplicate LLM responses per user message.

Add ChannelWorkspaceStore (Arc-wrapped HashMap with std::sync::RwLock)
that persists across callback invocations within a channel's lifetime.
Inject it as the WorkspaceReader and commit pending writes after every
callback execution (on_start, on_poll, on_http_request, execute_poll).

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* style: fix formatting

Co-Authored-By: Claude Opus 4.6 <[email protected]>

---------

Co-authored-by: Claude Opus 4.6 <[email protected]>
2026-02-20 15:52:21 +00:00
3829d81269 fix: consolidate per-module ENV_MUTEX into crate-wide test lock (#246)
Each config test module (llm.rs, embeddings.rs) defined its own
ENV_MUTEX, which doesn't prevent cross-module env races since
cargo test runs in parallel. Move to a single shared mutex in
config/helpers.rs so all unsafe set_var/remove_var calls are
serialized crate-wide.

Closes #245

Co-authored-by: Claude Opus 4.6 <[email protected]>
2026-02-20 08:11:10 +00:00
8a4f3b6f88 fix: remove auto-proceed fake user message injection from agent loop (#255)
The agentic loop injected fake user messages ("Please proceed and use
the available tools to complete this task.") when the LLM responded
with text instead of tool calls. This caused hallucinated conversations
during casual chat, 3x wasted LLM calls, and trust issues.

Remove the `resume_after_tool` parameter and `tools_executed` tracking
entirely. Text responses now return immediately, trusting the LLM to
decide when tools are needed (consistent with ZeroClaw and OpenClaw).

Closes #145

Co-authored-by: Claude Opus 4.6 <[email protected]>
2026-02-20 08:09:52 +00:00
140f29decf ci: add automated PR labeling system (#253)
* ci: add automated PR labeling system

Add two independent workflows for PR auto-labeling:
- Scope labels via actions/labeler (path glob matching)
- Size, risk, and contributor tier via custom shell script

Includes idempotent label bootstrap script (create-labels.sh).

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* ci: temporarily use pull_request trigger for testing

Switch to pull_request so workflows run from the PR branch.
Will revert to pull_request_target before merge.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix(ci): use absolute path for search/issues API call

gh api requires a leading slash for REST endpoints.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix(ci): use gh pr list instead of search API for contributor count

The search/issues API returns 404 with the default GITHUB_TOKEN.
gh pr list --state merged works with standard permissions.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* ci: revert to pull_request_target for fork PR support

Restore pull_request_target trigger and base branch checkout
now that testing is complete.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

---------

Co-authored-by: Claude Opus 4.6 <[email protected]>
2026-02-20 12:02:41 +04:00
5725a62c83 fix: onboarding errors reset flow and remote server auth (#185, #186) (#248)
* fix: incremental settings persistence and remote server auth (#185, #186)

Persist settings after each wizard step so failures don't lose prior
progress. Load existing settings on re-run to recover from partial
onboarding. Add manual token paste option for remote/headless servers
where browser OAuth is unreachable, and support IRONCLAW_OAUTH_CALLBACK_URL
for custom callback URLs. Color prompt output (green/red/blue prefixes).

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: replace session token paste with API key entry, address PR review

Replace option 4 in NEAR AI auth menu from session token paste to NEAR
AI Cloud API key entry (cloud.near.ai). Also address all PR review
feedback: restrict .env file permissions to 0o600, mask API key input
with secret_input, fix libsql loaded flag in try_load_existing_settings,
add ENV_MUTEX to oauth_defaults tests, and add NEARAI_API_KEY to secrets
injection.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: deduplicate keys in upsert_bootstrap_var

When the .env file contains duplicate keys (e.g. from manual editing),
only write the replacement once and skip subsequent duplicates.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: NEARAI_SESSION_TOKEN env var takes precedence over file-based tokens

Hosting providers inject session tokens via env var and expect them to
be used directly. Previously the env var was only picked up when no
session file existed and was treated as a legacy migration. Now the env
var always wins, without persisting to disk.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* docs: distinguish NEAR AI Chat and NEAR AI Cloud providers

Split documentation into two clearly named modes:
- NEAR AI Chat: Responses API at private.near.ai, session token auth
- NEAR AI Cloud: Chat Completions API at cloud-api.near.ai, API key auth

Update default base URLs so each mode points to its correct endpoint.
Update .env.example, deploy/env.example, CLAUDE.md, setup spec, and
code comments across config/llm.rs, nearai.rs, nearai_chat.rs, mod.rs.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: wizard recovery ordering — load DB before persist, fresh choices win

Previously, persist_after_step() ran after Step 1 but before
try_load_existing_settings(), bulk-upserting defaults that clobbered
prior settings. Additionally, merge_from gave stale DB values
precedence over fresh Step 1 choices.

Fix: snapshot Step 1 settings, load DB, then re-apply the snapshot.
This ensures prior progress (steps 2-7) is recovered while fresh
Step 1 choices override stale DB values.

Add two tests verifying wizard recovery merge ordering.

Addresses PR review comments from Copilot on wizard.rs:150,
wizard.rs:1607, and wizard.rs:1626.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* style: fix rustfmt formatting in config/llm.rs

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* style: collapse nested if per clippy collapsible_if lint

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: use print_success for API key confirmation, fix menu spacing

- Use print_success() for colored output consistency in api_key_login
- Fix box-drawing alignment: options 1-2 had an extra trailing space

Co-Authored-By: Claude Opus 4.6 <[email protected]>

---------

Co-authored-by: Claude Opus 4.6 <[email protected]>
2026-02-20 08:02:22 +00:00
bfe393eb38 fix: parallelize tool call execution via JoinSet (#219) (#252)
* fix: parallelize tool call execution via JoinSet (#219)

When the LLM returns multiple tool_calls in a single response, they were
executed sequentially. This change makes both the worker and dispatcher
paths concurrent using tokio::task::JoinSet, so N independent tool calls
complete in ~max(latency) instead of sum(latency).

Worker path: migrate execute_tools_parallel from join_all to JoinSet and
route the respond_with_tools branch through the same parallel path.

Dispatcher path: restructure the while-idx loop into three phases —
preflight (sequential approval/hook checks), parallel execution via
JoinSet, and sequential post-flight processing (session recording,
auth detection, sanitization).

Also fixes a pre-existing infinite loop bug where hook rejection used
`continue` inside a `while idx` loop, skipping `idx += 1` and retrying
the same rejected tool forever.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: address PR review — ordered results, deferred auth, dedup standalone fn

- Fix auth early return skipping unrecorded tool results: defer auth
  response until after all results in the batch are recorded in session
  history and context_messages (both dispatcher and thread_ops paths)
- Fix tool results appearing out of order: collect Phase 1 hook
  rejections indexed by original position, merge with Phase 2 execution
  results, and emit all in Phase 3 in original tool_calls order
- Deduplicate execute_chat_tool: Agent method now delegates to the
  standalone function instead of duplicating 90 lines of logic
- Fix benchmark compilation: add missing session_manager arg to Agent::new

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: rustfmt alignment for CI compatibility

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: address second round of PR review comments

- Distinguish JoinError panic vs cancellation in log messages and error
  reasons across all 3 files (dispatcher, thread_ops, worker)
- Simplify deferred_auth from Option<(String, String)> to Option<String>
  since only the instructions string is used
- Add single-tool short-circuit in worker execute_tools_parallel to
  avoid JoinSet overhead for the common single-tool case

Co-Authored-By: Claude Opus 4.6 <[email protected]>

---------

Co-authored-by: Claude Opus 4.6 <[email protected]>
2026-02-20 06:07:30 +00:00
AI-Reviewer-QSandGitHub 9906190de7 fix: prevent pipe deadlock in shell command execution (#140)
Drain stdout and stderr concurrently with child.wait() using tokio::join
to prevent deadlocks when command output exceeds the OS pipe buffer
(64KB on Linux, 16KB on macOS).

Use AsyncReadExt::take() for memory-bounded reads and
tokio::io::copy to sink for draining excess output.

Add regression test that generates 128KB of output to verify the
fix prevents deadlocks.
2026-02-20 03:07:46 +00:00
Illia PolosukhinandClaude Opus 4.6 9349a3baca fix: add missing session_manager arg to Agent::new in benchmark runner
Agent::new gained an 8th parameter (session_manager) but the benchmark
runner was not updated, breaking compilation of the bench crate.

Co-Authored-By: Claude Opus 4.6 <[email protected]>
2026-02-19 18:32:56 -08:00
3f135bdde9 fix: persist turns after approval and add agent-level tests (#250)
* fix: persist turns after approval and add agent-level tests

Port relevant changes from PR #112 that were not carried over to #237:

- Add persist_turn calls in process_approval for the response, error,
  and auth-required paths. Previously, turns completed after tool
  approval were never persisted to DB — if the process crashed after
  approval the entire turn (user message + assistant response) was lost.

- Add agent-level unit tests: StaticLlmProvider mock, make_test_agent
  helper, tests for auto-approval logic, destructive shell command
  detection, and PendingApproval backward-compatible deserialization
  (without deferred_tool_calls field).

- Remove unused _thread_state binding in process_approval.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: address 14 audit findings in src/agent/

Audit of the agent module found 2 High, 7 Medium, 3 Low, and 2 Nit
severity issues. This commit fixes all of them:

High:
- Remove 4 `.expect()` calls in session.rs (entry API, match, direct
  indexing, if-let) to eliminate panic paths in production
- Add typed RoutineError enum replacing Result<_, String> across
  routine.rs, routine_engine.rs, and callers in history/store.rs and
  db/libsql/mod.rs

Medium:
- Sanitize routine names in path construction to prevent directory
  traversal (routine_engine.rs)
- Log warnings for 5 silently-swallowed errors in scheduler.rs,
  compaction.rs, and worker.rs
- Extract shared handle_auth_intercept helper to deduplicate auth
  interception in thread_ops.rs
- Add session count warning threshold in session_manager.rs
- Make FullJob stub degradation visible via warn-level log and
  prepended warning in output

Low:
- Restrict dead code visibility with #[cfg(test)] on 19 unused items
  in submission.rs, task.rs, and undo.rs
- Narrow pub to pub(crate) on self_repair.rs builder methods
- Remove TaskStatus from mod.rs re-exports (test-only type)

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: address PR review comments

- Reorder persist_turn before persist_response_chain so the
  conversation row exists before the metadata UPDATE runs
- Add persist_response_chain call to handle_auth_intercept so
  auth-required paths preserve the response chain
- Harden sanitize_routine_name to use allowlist (alphanumeric,
  dash, underscore) instead of denylist replacements
- Fix stale active_thread ID in get_or_create_thread: fall back
  to create_thread() when the stored ID is missing from the map
- Persist turn on approval rejection so user messages survive
  crashes after a tool is rejected

Co-Authored-By: Claude Opus 4.6 <[email protected]>

---------

Co-authored-by: Claude Opus 4.6 <[email protected]>
2026-02-20 02:28:15 +00:00
97a7637f30 feat: extension registry with metadata catalog and onboarding integration (#238)
* feat: add extension registry with metadata catalog, CLI, and onboarding integration

Adds a central registry that catalogs all 14 available extensions (10 tools,
4 channels) with their capabilities, auth requirements, and artifact references.
The onboarding wizard now shows installable channels from the registry and
offers tool installation as a new Step 7.

- registry/ folder with per-extension JSON manifests and bundle definitions
- src/registry/ module: manifest structs, catalog loader, installer
- `ironclaw registry list|info|install|install-defaults` CLI commands
- Setup wizard enhanced: channels from registry, new extensions step (8 steps)

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix(setup): resolve workspace errors for tool crates and channels-only onboarding

Tool crates in tools-src/ and channels-src/ failed `cargo metadata` during
onboard install because Cargo resolved them as part of the root workspace.
Add `[workspace]` table to each standalone crate and extend the root
`workspace.exclude` list so they build independently.

Channels-only mode (`onboard --channels-only`) failed with "Secrets not
configured" and "No database connection" because it skipped database and
security setup. Add `reconnect_existing_db()` to establish the DB connection
and load saved settings before running channel configuration.

Also improve the tunnel "already configured" display to show full provider
details (domain, mode, command) instead of just the provider name.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix(registry): address PR review feedback on installer and catalog

- Use manifest.name (not crate_name) for installed filenames so
  discovery, auth, and CLI commands all agree on the stem (#1)
- Add AlreadyInstalled error variant instead of misleading
  ExtensionNotFound (#2)
- Add DownloadFailed error variant with URL context instead of
  stuffing URLs into PathBuf (#3)
- Validate HTTP status with error_for_status() before reading
  response bytes in artifact downloads (#4)
- Switch build_wasm_component to tokio::process::Command with
  status() so build output streams to the terminal (#6)
- Find WASM artifact by crate_name specifically instead of picking
  the first .wasm file in the release directory (#7)
- Add is_file() guard in catalog loader to skip directories (#8)
- Detect ambiguous bare-name lookups when both tools/<name> and
  channels/<name> exist, with get_strict() returning an error (#9)
- Fix wizard step_extensions to check tool.name for installed
  detection, consistent with the new naming (#11, #12)
- Fix redundant closures and map_or clippy warnings in changed files

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix(setup): restore DB connection fields after settings reload

reconnect_postgres() and reconnect_libsql() called Settings::from_db_map()
which overwrote database_url / libsql_path / libsql_url set from env vars.
Also use get_strict() in cmd_info to surface ambiguous bare-name errors.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* style: fix clippy collapsible_if and print_literal warnings

Collapse nested if-let chains and inline string literals in format
macros to satisfy CI clippy lint checks (deny warnings).

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix(registry): prefer artifacts for install-defaults and improve dir lookup

- InstallDefaults now defaults to downloading pre-built artifacts
  (matching `registry install` behavior), with --build flag for source builds.
- find_registry_dir() walks up 3 ancestor levels from the exe and adds
  a CARGO_MANIFEST_DIR fallback, matching load_registry_catalog() logic.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

---------

Co-authored-by: Claude Opus 4.6 <[email protected]>
2026-02-20 01:17:44 +00:00
bigguybobbyandGitHub dae26d640e feat(models): add GPT-5.3 Codex, full GPT-5.x family, Claude 4.x series, o4-mini (#197)
Fixes #184 — updates model selection, priority sort, and cost table to
match current OpenAI and Anthropic model catalogs.

OpenAI: GPT-5.3 Codex, GPT-5.2 Codex/Pro, GPT-5.1 Codex/Mini/Max,
GPT-5/Mini/Nano, GPT-4.1/Mini/Nano, o4-mini, o3/Pro
Anthropic: Claude Opus 4.6/4.5/4.1/4.0, Claude Sonnet 4.6/4.5/4.0,
Claude Haiku 4.5, Claude 3.7 Sonnet, Claude 3.5 Haiku

Also resolves stale merge-conflict markers in http.rs and json.rs.
2026-02-20 01:16:13 +00:00
fa64df05ff feat: wire memory hygiene into the heartbeat loop (#195)
* feat: wire memory hygiene into heartbeat loop (#166)

* refactor: address PR review comments for hygiene wiring

* style: fix fmt import ordering and clippy too_many_arguments warning

* fix: update heartbeat integration test to pass HygieneConfig argument

HeartbeatRunner::new() now requires a HygieneConfig as its second
argument after the hygiene wiring refactor. Pass the default config
in the integration test.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

---------

Co-authored-by: Claude Sonnet 4.6 <[email protected]>
Co-authored-by: Illia Polosukhin <[email protected]>
2026-02-20 01:13:25 +00:00
356f56f77c docs: update CLAUDE.md for recently merged features (#183)
* docs: update CLAUDE.md for recently merged features

Document skills system, sandbox network proxy, leak detector,
Tinfoil private inference, setup wizard, and shell env scrubbing
that were merged but not reflected in CLAUDE.md.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* docs: fix SKILL.md format example and scoring description

Align SKILL.md frontmatter example with actual SkillManifest struct:
activation block with patterns/keywords/max_context_tokens, requires
nested under metadata.openclaw. Fix scoring pipeline description to
mention keywords, tags, and regex patterns instead of triggers/intents.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* docs: optimize CLAUDE.md structure and reduce from 959 to 671 lines

- Update llm/ directory tree (4 -> 12 files to match actual codebase)
- Fix "NEAR AI (required)" -> "NEAR AI (when LLM_BACKEND=nearai)"
- Remove 28-item Completed changelog list (no actionable value)
- Deduplicate 3 config blocks with cross-references
- Extract Workspace deep-dive to src/workspace/README.md
- Extract Tool Architecture deep-dive to src/tools/README.md
- Consolidate Code Style and Review Discipline under Key Patterns
- Add workspace and tools to Module Specifications table

Co-Authored-By: Claude Opus 4.6 <[email protected]>

---------

Co-authored-by: Claude Opus 4.6 <[email protected]>
2026-02-20 01:04:39 +00:00
115 changed files with 7592 additions and 2875 deletions
+21 -8
View File
@@ -2,18 +2,25 @@
DATABASE_URL=postgres://localhost/ironclaw
DATABASE_POOL_SIZE=10
# LLM Provider (NEAR AI)
# NEAR AI provides a unified interface to all models with user authentication
# Session token is stored in ~/.ironclaw/session.json and managed automatically.
# On first run, the agent will open a browser for OAuth authentication.
NEARAI_MODEL=claude-3-5-sonnet-20241022
# LLM Provider
# LLM_BACKEND=nearai # default
# Possible values: nearai, ollama, openai_compatible, openai, anthropic, tinfoil
# === NEAR AI (Chat Completions API) ===
# Two auth modes:
# 1. Session token (default): Uses browser OAuth (GitHub/Google) on first run.
# Session token stored in ~/.ironclaw/session.json automatically.
# Base URL defaults to https://private.near.ai
# 2. API key: Set NEARAI_API_KEY to use API key auth from cloud.near.ai.
# Base URL defaults to https://cloud-api.near.ai
NEARAI_MODEL=zai-org/GLM-5-FP8
NEARAI_BASE_URL=https://private.near.ai
NEARAI_AUTH_URL=https://private.near.ai
# NEARAI_SESSION_PATH=~/.ironclaw/session.json # optional, default shown
# NEARAI_SESSION_TOKEN=sess_... # hosting providers: set this
# NEARAI_SESSION_PATH=~/.ironclaw/session.json # optional, default shown
# NEARAI_API_KEY=... # API key from cloud.near.ai
# Local LLM Providers (Ollama, LM Studio, vLLM, LiteLLM)
# LLM_BACKEND=nearai # default
# Possible values: nearai, ollama, openai_compatible, openai, anthropic
# === Ollama ===
# OLLAMA_MODEL=llama3.2
@@ -68,6 +75,12 @@ HEARTBEAT_INTERVAL_SECS=1800
HEARTBEAT_NOTIFY_CHANNEL=cli
HEARTBEAT_NOTIFY_USER=default
# Memory hygiene settings (automatic cleanup of stale workspace documents)
# Runs on each heartbeat tick; identity files (IDENTITY.md, SOUL.md) are never deleted
# MEMORY_HYGIENE_ENABLED=true
# MEMORY_HYGIENE_RETENTION_DAYS=30 # delete daily/ docs older than this many days
# MEMORY_HYGIENE_CADENCE_HOURS=12 # minimum hours between cleanup passes
# Safety settings
SAFETY_MAX_OUTPUT_LENGTH=100000
SAFETY_INJECTION_CHECK_ENABLED=true
+166
View File
@@ -0,0 +1,166 @@
# Scope labels for actions/labeler@v6
# Maps file path globs to scope labels. Multiple labels can apply per PR.
"scope: agent":
- changed-files:
- any-glob-to-any-file:
- src/agent/**
"scope: channel":
- changed-files:
- any-glob-to-any-file:
- src/channels/channel.rs
- src/channels/manager.rs
- src/channels/mod.rs
"scope: channel/cli":
- changed-files:
- any-glob-to-any-file:
- src/channels/cli/**
- src/cli/**
"scope: channel/web":
- changed-files:
- any-glob-to-any-file:
- src/channels/web/**
"scope: channel/wasm":
- changed-files:
- any-glob-to-any-file:
- src/channels/wasm/**
"scope: tool":
- changed-files:
- any-glob-to-any-file:
- src/tools/tool.rs
- src/tools/registry.rs
- src/tools/mod.rs
- src/tools/sandbox.rs
"scope: tool/builtin":
- changed-files:
- any-glob-to-any-file:
- src/tools/builtin/**
"scope: tool/wasm":
- changed-files:
- any-glob-to-any-file:
- src/tools/wasm/**
"scope: tool/mcp":
- changed-files:
- any-glob-to-any-file:
- src/tools/mcp/**
"scope: tool/builder":
- changed-files:
- any-glob-to-any-file:
- src/tools/builder/**
"scope: db":
- changed-files:
- any-glob-to-any-file:
- src/db/mod.rs
"scope: db/postgres":
- changed-files:
- any-glob-to-any-file:
- src/db/postgres.rs
- migrations/**
"scope: db/libsql":
- changed-files:
- any-glob-to-any-file:
- src/db/libsql_backend.rs
- src/db/libsql_migrations.rs
"scope: safety":
- changed-files:
- any-glob-to-any-file:
- src/safety/**
"scope: llm":
- changed-files:
- any-glob-to-any-file:
- src/llm/**
"scope: workspace":
- changed-files:
- any-glob-to-any-file:
- src/workspace/**
"scope: orchestrator":
- changed-files:
- any-glob-to-any-file:
- src/orchestrator/**
"scope: worker":
- changed-files:
- any-glob-to-any-file:
- src/worker/**
"scope: secrets":
- changed-files:
- any-glob-to-any-file:
- src/secrets/**
"scope: config":
- changed-files:
- any-glob-to-any-file:
- src/config.rs
- src/settings.rs
"scope: extensions":
- changed-files:
- any-glob-to-any-file:
- src/extensions/**
"scope: setup":
- changed-files:
- any-glob-to-any-file:
- src/setup/**
"scope: evaluation":
- changed-files:
- any-glob-to-any-file:
- src/evaluation/**
"scope: estimation":
- changed-files:
- any-glob-to-any-file:
- src/estimation/**
"scope: sandbox":
- changed-files:
- any-glob-to-any-file:
- src/sandbox/**
- Dockerfile*
"scope: hooks":
- changed-files:
- any-glob-to-any-file:
- src/hooks/**
"scope: pairing":
- changed-files:
- any-glob-to-any-file:
- src/pairing/**
"scope: ci":
- changed-files:
- any-glob-to-any-file:
- .github/workflows/**
- .github/scripts/**
"scope: docs":
- changed-files:
- any-glob-to-any-file:
- "**/*.md"
- docs/**
- LICENSE*
"scope: dependencies":
- changed-files:
- any-glob-to-any-file:
- Cargo.toml
- Cargo.lock
+71
View File
@@ -0,0 +1,71 @@
#!/usr/bin/env bash
# Idempotent label bootstrap for IronClaw PR automation.
# Uses `gh label create --force` so it can be re-run safely.
#
# Usage: bash .github/scripts/create-labels.sh
# Requires: gh CLI authenticated with repo scope
set -euo pipefail
if ! command -v gh &>/dev/null; then
echo "Error: gh CLI is required. Install from https://cli.github.com" >&2
exit 1
fi
create() {
local name="$1" color="$2" description="$3"
gh label create "$name" --color "$color" --description "$description" --force
}
echo "==> Creating size labels..."
create "size: XS" "F9D0C4" "< 10 changed lines (excluding docs)"
create "size: S" "F5A3A3" "10-49 changed lines"
create "size: M" "E57373" "50-199 changed lines"
create "size: L" "D32F2F" "200-499 changed lines"
create "size: XL" "B71C1C" "500+ changed lines"
echo "==> Creating risk labels..."
create "risk: low" "4CAF50" "Changes to docs, tests, or low-risk modules"
create "risk: medium" "FFC107" "Business logic, config, or moderate-risk modules"
create "risk: high" "F44336" "Safety, secrets, auth, or critical infrastructure"
create "risk: manual" "9E9E9E" "Risk level set manually (sticky, not overwritten)"
echo "==> Creating scope labels..."
create "scope: agent" "006B75" "Agent core (agent loop, router, scheduler)"
create "scope: channel" "00838F" "Channel infrastructure"
create "scope: channel/cli" "00897B" "TUI / CLI channel"
create "scope: channel/web" "00796B" "Web gateway channel"
create "scope: channel/wasm" "00695C" "WASM channel runtime"
create "scope: tool" "1565C0" "Tool infrastructure"
create "scope: tool/builtin" "1976D2" "Built-in tools"
create "scope: tool/wasm" "1E88E5" "WASM tool sandbox"
create "scope: tool/mcp" "2196F3" "MCP client"
create "scope: tool/builder" "42A5F5" "Dynamic tool builder"
create "scope: db" "4A148C" "Database trait / abstraction"
create "scope: db/postgres" "6A1B9A" "PostgreSQL backend"
create "scope: db/libsql" "7B1FA2" "libSQL / Turso backend"
create "scope: safety" "880E4F" "Prompt injection defense"
create "scope: llm" "4527A0" "LLM integration"
create "scope: workspace" "283593" "Persistent memory / workspace"
create "scope: orchestrator" "0D47A1" "Container orchestrator"
create "scope: worker" "01579B" "Container worker"
create "scope: secrets" "BF360C" "Secrets management"
create "scope: config" "E65100" "Configuration"
create "scope: extensions" "33691E" "Extension management"
create "scope: setup" "827717" "Onboarding / setup"
create "scope: evaluation" "558B2F" "Success evaluation"
create "scope: estimation" "9E9D24" "Cost/time estimation"
create "scope: sandbox" "00BFA5" "Docker sandbox"
create "scope: hooks" "6D4C41" "Git/event hooks"
create "scope: pairing" "4E342E" "Pairing mode"
create "scope: ci" "546E7A" "CI/CD workflows"
create "scope: docs" "78909C" "Documentation"
create "scope: dependencies" "90A4AE" "Dependency updates"
echo "==> Creating contributor labels..."
create "contributor: new" "FFF9C4" "First-time contributor"
create "contributor: regular" "FFE082" "2-5 merged PRs"
create "contributor: experienced" "FFB74D" "6-19 merged PRs"
create "contributor: core" "FF8A65" "20+ merged PRs"
echo "Done. All labels created/updated."
+139
View File
@@ -0,0 +1,139 @@
#!/usr/bin/env bash
# Classify a PR by size, risk, and contributor tier.
# Called by the pr-label-classify workflow.
#
# Inputs (env vars):
# PR_NUMBER — pull request number
# REPO — owner/repo (e.g. "user/ironclaw")
#
# Requires: gh CLI, jq
set -euo pipefail
PR_NUMBER="${PR_NUMBER:?PR_NUMBER is required}"
REPO="${REPO:?REPO is required}"
# ─── helpers ────────────────────────────────────────────────────────────────
# Remove all labels in a dimension except the desired one.
# Usage: set_exclusive_label "size" "size: M"
set_exclusive_label() {
local prefix="$1" desired="$2"
# Fetch current labels on the PR
local current
current=$(gh pr view "$PR_NUMBER" --repo "$REPO" --json labels --jq '.labels[].name')
# Remove any existing label with the same prefix
while IFS= read -r label; do
[[ -z "$label" ]] && continue
if [[ "$label" == "${prefix}:"* && "$label" != "$desired" ]]; then
gh pr edit "$PR_NUMBER" --repo "$REPO" --remove-label "$label" 2>/dev/null || true
fi
done <<< "$current"
# Add the desired label
gh pr edit "$PR_NUMBER" --repo "$REPO" --add-label "$desired"
}
# ─── size ───────────────────────────────────────────────────────────────────
classify_size() {
# Sum changed lines across non-doc files
local total
total=$(gh api "repos/${REPO}/pulls/${PR_NUMBER}/files" \
--paginate --jq '
[.[] | select(.filename | test("\\.(md|txt|rst|adoc)$") | not) | .changes]
| add // 0
')
local label
if (( total < 10 )); then label="size: XS"
elif (( total < 50 )); then label="size: S"
elif (( total < 200 )); then label="size: M"
elif (( total < 500 )); then label="size: L"
else label="size: XL"
fi
echo "Size: ${total} changed lines -> ${label}"
set_exclusive_label "size" "$label"
}
# ─── risk ───────────────────────────────────────────────────────────────────
classify_risk() {
# If "risk: manual" is present, skip — it's a sticky override
local current
current=$(gh pr view "$PR_NUMBER" --repo "$REPO" --json labels --jq '.labels[].name')
if echo "$current" | grep -qx "risk: manual"; then
echo "Risk: skipped (manual override)"
return
fi
# Fetch changed file paths
local files
files=$(gh api "repos/${REPO}/pulls/${PR_NUMBER}/files" \
--paginate --jq '.[].filename')
local risk="low"
while IFS= read -r file; do
[[ -z "$file" ]] && continue
case "$file" in
# High risk: safety, secrets, auth, crypto, setup, orchestrator auth
src/safety/*|src/secrets/*|src/llm/session.rs|src/orchestrator/auth.rs|\
src/channels/web/auth.rs|src/setup/*)
risk="high"
break # can't go higher
;;
# Medium risk: agent core, config, database, worker, tools, channels
src/agent/*|src/config.rs|src/settings.rs|src/db/*|src/worker/*|\
src/tools/*|src/channels/*|src/orchestrator/*|src/context/*|\
src/hooks/*|src/sandbox/*|src/extensions/*|Cargo.toml|\
.github/workflows/*)
# Only upgrade, never downgrade
[[ "$risk" != "high" ]] && risk="medium"
;;
# Low risk: docs, tests, estimation, evaluation, history, etc.
*)
;;
esac
done <<< "$files"
echo "Risk: ${risk}"
set_exclusive_label "risk" "risk: ${risk}"
}
# ─── contributor tier ───────────────────────────────────────────────────────
classify_contributor() {
# Get PR author
local author
author=$(gh pr view "$PR_NUMBER" --repo "$REPO" --json author --jq '.author.login')
# Count merged PRs by this author in this repo
local count
count=$(gh pr list --repo "$REPO" --state merged --author "$author" \
--limit 100 --json number --jq 'length')
local label
if (( count == 0 )); then label="contributor: new"
elif (( count < 6 )); then label="contributor: regular"
elif (( count < 20 )); then label="contributor: experienced"
else label="contributor: core"
fi
echo "Contributor: ${author} has ${count} merged PRs -> ${label}"
set_exclusive_label "contributor" "$label"
}
# ─── main ───────────────────────────────────────────────────────────────────
echo "Classifying PR #${PR_NUMBER} in ${REPO}..."
classify_size
classify_risk
classify_contributor
echo "Done."
+26
View File
@@ -0,0 +1,26 @@
name: "PR: Classify (Size, Risk, Contributor)"
on:
pull_request_target:
types: [opened, synchronize, reopened]
permissions:
contents: read
pull-requests: write
issues: read # needed for search/issues API (contributor count)
jobs:
classify:
runs-on: ubuntu-latest
steps:
- name: Checkout base branch
uses: actions/checkout@v4
with:
ref: ${{ github.event.pull_request.base.ref }}
- name: Classify PR
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
PR_NUMBER: ${{ github.event.pull_request.number }}
REPO: ${{ github.repository }}
run: bash .github/scripts/pr-labeler.sh
+18
View File
@@ -0,0 +1,18 @@
name: "PR: Scope Labels"
on:
pull_request_target:
types: [opened, synchronize, reopened]
permissions:
contents: read
pull-requests: write
jobs:
scope:
runs-on: ubuntu-latest
steps:
- uses: actions/labeler@v5
with:
configuration-path: .github/labeler.yml
sync-labels: false # additive only — never remove scope labels
+23
View File
@@ -7,6 +7,29 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
## [0.8.0](https://github.com/nearai/ironclaw/compare/ironclaw-v0.7.0...ironclaw-v0.8.0) - 2026-02-20
### Added
- extension registry with metadata catalog and onboarding integration ([#238](https://github.com/nearai/ironclaw/pull/238))
- *(models)* add GPT-5.3 Codex, full GPT-5.x family, Claude 4.x series, o4-mini ([#197](https://github.com/nearai/ironclaw/pull/197))
- wire memory hygiene into the heartbeat loop ([#195](https://github.com/nearai/ironclaw/pull/195))
### Fixed
- persist WASM channel workspace writes across callbacks ([#264](https://github.com/nearai/ironclaw/pull/264))
- consolidate per-module ENV_MUTEX into crate-wide test lock ([#246](https://github.com/nearai/ironclaw/pull/246))
- remove auto-proceed fake user message injection from agent loop ([#255](https://github.com/nearai/ironclaw/pull/255))
- onboarding errors reset flow and remote server auth (#185, #186) ([#248](https://github.com/nearai/ironclaw/pull/248))
- parallelize tool call execution via JoinSet ([#219](https://github.com/nearai/ironclaw/pull/219)) ([#252](https://github.com/nearai/ironclaw/pull/252))
- prevent pipe deadlock in shell command execution ([#140](https://github.com/nearai/ironclaw/pull/140))
- persist turns after approval and add agent-level tests ([#250](https://github.com/nearai/ironclaw/pull/250))
### Other
- add automated PR labeling system ([#253](https://github.com/nearai/ironclaw/pull/253))
- update CLAUDE.md for recently merged features ([#183](https://github.com/nearai/ironclaw/pull/183))
## [0.7.0](https://github.com/nearai/ironclaw/compare/ironclaw-v0.6.0...ironclaw-v0.7.0) - 2026-02-19
### Added
+206 -330
View File
@@ -13,14 +13,17 @@
### Features
- **Multi-channel input**: TUI (Ratatui), HTTP webhooks, WASM channels (Telegram, Slack), web gateway
- **Parallel job execution** with state machine and self-repair for stuck jobs
- **Sandbox execution**: Docker container isolation with orchestrator/worker pattern
- **Sandbox execution**: Docker container isolation with network proxy and credential injection
- **Claude Code mode**: Delegate jobs to Claude CLI inside containers
- **Skills system**: SKILL.md prompt extensions with trust model, tool attenuation, and ClawHub registry
- **Routines**: Scheduled (cron) and reactive (event, webhook) task execution
- **Web gateway**: Browser UI with SSE/WebSocket real-time streaming
- **Extension management**: Install, auth, activate MCP/WASM extensions
- **Extensible tools**: Built-in tools, WASM sandbox, MCP client, dynamic builder
- **Persistent memory**: Workspace with hybrid search (FTS + vector via RRF)
- **Prompt injection defense**: Sanitizer, validator, policy rules, leak detection
- **Prompt injection defense**: Sanitizer, validator, policy rules, leak detection, shell env scrubbing
- **Multi-provider LLM**: NEAR AI, OpenAI, Anthropic, Ollama, OpenAI-compatible, Tinfoil private inference
- **Setup wizard**: 7-step interactive onboarding for first-run configuration
- **Heartbeat system**: Proactive periodic execution with checklist
## Build & Test
@@ -64,6 +67,7 @@ src/
│ ├── context_monitor.rs # Memory pressure detection
│ ├── undo.rs # Turn-based undo/redo with checkpoints
│ ├── submission.rs # Submission parsing (undo, redo, compact, clear, etc.)
│ ├── dispatcher.rs # Skill-aware job dispatching
│ ├── task.rs # Sub-task execution framework
│ ├── routine.rs # Routine types (Trigger, Action, Guardrails)
│ └── routine_engine.rs # Routine execution (cron ticker, event matcher)
@@ -113,11 +117,18 @@ src/
│ ├── policy.rs # PolicyRule system with severity/actions
│ └── leak_detector.rs # Secret detection (API keys, tokens, etc.)
├── llm/ # LLM integration (NEAR AI only)
├── llm/ # LLM integration (multi-provider)
│ ├── mod.rs # Provider factory, LlmBackend enum
│ ├── provider.rs # LlmProvider trait, message types
│ ├── nearai.rs # NEAR AI chat-api implementation
│ ├── nearai_chat.rs # NEAR AI Chat Completions provider (session token + API key auth)
│ ├── reasoning.rs # Planning, tool selection, evaluation
── session.rs # Session token management with auto-renewal
── session.rs # Session token management with auto-renewal
│ ├── circuit_breaker.rs # Circuit breaker for provider failures
│ ├── retry.rs # Retry with exponential backoff
│ ├── failover.rs # Multi-provider failover chain
│ ├── response_cache.rs # LLM response caching
│ ├── costs.rs # Token cost tracking
│ └── rig_adapter.rs # Rig framework adapter
├── tools/ # Extensible tool system
│ ├── tool.rs # Tool trait, ToolOutput, ToolError
@@ -131,6 +142,7 @@ src/
│ │ ├── job.rs # CreateJob, ListJobs, JobStatus, CancelJob
│ │ ├── routine.rs # routine_create/list/update/delete/history
│ │ ├── extension_tools.rs # Extension install/auth/activate/remove
│ │ ├── skill_tools.rs # skill_list/search/install/remove tools
│ │ └── marketplace.rs, ecommerce.rs, taskrabbit.rs, restaurant.rs (stubs)
│ ├── builder/ # Dynamic tool building
│ │ ├── core.rs # BuildRequirement, SoftwareType, Language
@@ -180,11 +192,38 @@ src/
│ ├── success.rs # SuccessEvaluator trait, RuleBasedEvaluator, LlmEvaluator
│ └── metrics.rs # MetricsCollector, QualityMetrics
├── sandbox/ # Docker execution sandbox
│ ├── mod.rs # Public API, default allowlist
│ ├── config.rs # SandboxConfig, SandboxPolicy enum
│ ├── manager.rs # SandboxManager orchestration
│ ├── container.rs # ContainerRunner, Docker lifecycle
│ ├── error.rs # SandboxError types
│ └── proxy/ # Network proxy for containers
│ ├── mod.rs # NetworkProxyBuilder
│ ├── http.rs # HttpProxy, CredentialResolver trait
│ ├── policy.rs # NetworkPolicyDecider trait
│ └── allowlist.rs # DomainAllowlist validation
├── secrets/ # Secrets management
│ ├── crypto.rs # AES-256-GCM encryption
│ ├── store.rs # Secret storage
│ └── types.rs # Credential types
├── setup/ # Onboarding wizard (spec: src/setup/README.md)
│ ├── mod.rs # Entry point, check_onboard_needed()
│ ├── wizard.rs # 7-step interactive wizard
│ ├── channels.rs # Channel setup helpers
│ └── prompts.rs # Terminal prompts (select, confirm, secret)
├── skills/ # SKILL.md prompt extension system
│ ├── mod.rs # Core types (SkillTrust, LoadedSkill)
│ ├── registry.rs # SkillRegistry: discover, install, remove
│ ├── selector.rs # Deterministic scoring prefilter
│ ├── attenuation.rs # Trust-based tool ceiling
│ ├── gating.rs # Requirement checks (bins, env, config)
│ ├── parser.rs # SKILL.md frontmatter + markdown parser
│ └── catalog.rs # ClawHub registry client
└── history/ # Persistence
├── store.rs # PostgreSQL repositories
└── analytics.rs # Aggregation queries (JobStats, ToolStats)
@@ -214,6 +253,7 @@ When designing new features or systems, always prefer generic/extensible archite
- `LlmProvider` - Add new LLM backends
- `SuccessEvaluator` - Custom evaluation logic
- `EmbeddingProvider` - Add embedding backends (workspace search)
- `NetworkPolicyDecider` - Custom network access policies for sandbox containers
### Tool Implementation
```rust
@@ -252,6 +292,40 @@ Pending -> InProgress -> Completed -> Submitted -> Accepted
\-> Failed
```
### Code Style
- Use `crate::` imports, not `super::`
- No `pub use` re-exports unless exposing to downstream consumers
- Prefer strong types over strings (enums, newtypes)
- Keep functions focused, extract helpers when logic is reused
- Comments for non-obvious logic only
### Review & Fix Discipline
Hard-won lessons from code review -- follow these when fixing bugs or addressing review feedback.
**Fix the pattern, not just the instance:** When a reviewer flags a bug (e.g., TOCTOU race in INSERT + SELECT-back), search the entire codebase for all instances of that same pattern. A fix in `SecretsStore::create()` that doesn't also fix `WasmToolStore::store()` is half a fix.
**Propagate architectural fixes to satellite types:** If a core type changes its concurrency model (e.g., `LibSqlBackend` switches to connection-per-operation), every type that was handed a resource from the old model (e.g., `LibSqlSecretsStore`, `LibSqlWasmToolStore` holding a single `Connection`) must also be updated. Grep for the old type across the codebase.
**Schema translation is more than DDL:** When translating a database schema between backends (PostgreSQL to libSQL, etc.), check for:
- **Indexes** -- diff `CREATE INDEX` statements between the two schemas
- **Seed data** -- check for `INSERT INTO` in migrations (e.g., `leak_detection_patterns`)
- **Semantic differences** -- document where SQL functions behave differently (e.g., `json_patch` vs `jsonb_set`)
**Feature flag testing:** When adding feature-gated code, test compilation with each feature in isolation:
```bash
cargo check # default features
cargo check --no-default-features --features libsql # libsql only
cargo check --all-features # all features
```
Dead code behind the wrong `#[cfg]` gate will only show up when building with a single feature.
**Mechanical verification before committing:** Run these checks on changed files before committing:
- `grep -rnE '\.unwrap\(|\.expect\(' <files>` -- no panics in production
- `grep -rn 'super::' <files>` -- use `crate::` imports
- If you fixed a pattern bug, `grep` for other instances of that pattern across `src/`
## Configuration
Environment variables (see `.env.example`):
@@ -263,10 +337,14 @@ LIBSQL_PATH=~/.ironclaw/ironclaw.db # libSQL local path (default)
# LIBSQL_URL=libsql://xxx.turso.io # Turso cloud (optional)
# LIBSQL_AUTH_TOKEN=xxx # Required with LIBSQL_URL
# NEAR AI (required)
NEARAI_SESSION_TOKEN=sess_...
NEARAI_MODEL=claude-3-5-sonnet-20241022
# NEAR AI (when LLM_BACKEND=nearai, the default)
# Two auth modes: session token (default) or API key
# Session token auth (default): uses browser OAuth on first run
NEARAI_SESSION_TOKEN=sess_... # hosting providers: set this
NEARAI_BASE_URL=https://private.near.ai
# API key auth: set NEARAI_API_KEY, base URL defaults to cloud-api.near.ai
# NEARAI_API_KEY=... # API key from cloud.near.ai
NEARAI_MODEL=claude-3-5-sonnet-20241022
# Agent settings
AGENT_NAME=ironclaw
@@ -297,6 +375,10 @@ SANDBOX_ENABLED=true
SANDBOX_IMAGE=ironclaw-worker:latest
SANDBOX_MEMORY_LIMIT_MB=512
SANDBOX_TIMEOUT_SECS=1800
SANDBOX_CPU_LIMIT=1.0 # CPU cores per container
SANDBOX_NETWORK_PROXY=true # Enable network proxy for containers
SANDBOX_PROXY_PORT=8080 # Proxy listener port
SANDBOX_DEFAULT_POLICY=workspace_write # ReadOnly, WorkspaceWrite, FullAccess
# Claude Code mode (runs inside sandbox containers)
CLAUDE_CODE_ENABLED=false
@@ -308,16 +390,25 @@ CLAUDE_CODE_CONFIG_DIR=/home/worker/.claude
ROUTINES_ENABLED=true
ROUTINES_CRON_INTERVAL=60 # Tick interval in seconds
ROUTINES_MAX_CONCURRENT=3
# Skills system
SKILLS_ENABLED=true
SKILLS_MAX_TOKENS=4000 # Max prompt budget per turn
SKILLS_CATALOG_URL=https://clawhub.dev # ClawHub registry URL
SKILLS_AUTO_DISCOVER=true # Scan skill directories on startup
# Tinfoil private inference
TINFOIL_API_KEY=... # Required when LLM_BACKEND=tinfoil
TINFOIL_MODEL=kimi-k2-5 # Default model
```
### NEAR AI Provider
### LLM Providers
Uses the NEAR AI chat-api (`https://api.near.ai/v1/responses`) which provides:
- Unified access to multiple models (OpenAI, Anthropic, etc.)
- User authentication via session tokens
- Usage tracking and billing through NEAR AI
IronClaw supports multiple LLM backends via the `LLM_BACKEND` env var: `nearai` (default), `openai`, `anthropic`, `ollama`, `openai_compatible`, and `tinfoil`.
Session tokens have the format `sess_xxx` (37 characters). They are authenticated against the NEAR AI auth service.
**NEAR AI** -- Uses the Chat Completions API with dual auth support. Session token auth (default): authenticates with session tokens (`sess_xxx`) obtained via browser OAuth (GitHub/Google), base URL defaults to `https://private.near.ai`. API key auth: set `NEARAI_API_KEY` (from `cloud.near.ai`), base URL defaults to `https://cloud-api.near.ai`. Both modes use the same Chat Completions endpoint. Tool messages are flattened to plain text for compatibility. Set `NEARAI_SESSION_TOKEN` env var for hosting providers that inject tokens via environment.
**Tinfoil** -- Private inference via `https://inference.tinfoil.sh/v1`. Runs models inside hardware-attested TEEs so neither Tinfoil nor the cloud provider can see prompts or responses. Uses the OpenAI-compatible Chat Completions API. Configure with `TINFOIL_API_KEY` and `TINFOIL_MODEL` (default: `kimi-k2-5`).
## Database
@@ -386,22 +477,7 @@ Both backends implement this trait. PostgreSQL delegates to the existing `Store`
- `tool_failures` - Self-repair tracking
- `secrets`, `wasm_tools`, `tool_capabilities` - Extension infrastructure
### Configuration
```bash
# Backend selection (default: postgres)
DATABASE_BACKEND=libsql
# PostgreSQL
DATABASE_URL=postgres://user:pass@localhost/ironclaw
# libSQL (embedded)
LIBSQL_PATH=~/.ironclaw/ironclaw.db # Default path
# libSQL (Turso cloud sync)
LIBSQL_URL=libsql://your-db.turso.io
LIBSQL_AUTH_TOKEN=your-token # Required when LIBSQL_URL is set
```
Database configuration: see Configuration section above.
### Current Limitations (libSQL backend)
@@ -419,6 +495,7 @@ All external tool output passes through `SafetyLayer`:
1. **Sanitizer** - Detects injection patterns, escapes dangerous content
2. **Validator** - Checks length, encoding, forbidden patterns
3. **Policy** - Rules with severity (Critical/High/Medium/Low) and actions (Block/Warn/Review/Sanitize)
4. **Leak Detector** - Scans for 15+ secret patterns (API keys, tokens, private keys, connection strings) at two points: tool output before it reaches the LLM, and LLM responses before they reach the user. Actions per pattern: Block (reject entirely), Redact (mask the secret), or Warn (flag but allow)
Tool outputs are wrapped before reaching LLM:
```xml
@@ -427,6 +504,95 @@ Tool outputs are wrapped before reaching LLM:
</tool_output>
```
### Shell Environment Scrubbing
The shell tool (`src/tools/builtin/shell.rs`) scrubs sensitive environment variables before executing commands, preventing secrets from leaking through `env`, `printenv`, or `$VAR` expansion. The sanitizer (`src/safety/sanitizer.rs`) also detects command injection patterns (chained commands, subshells, path traversal) and blocks or escapes them based on policy rules.
## Skills System
Skills are SKILL.md files that extend the agent's prompt with domain-specific instructions. Each skill is a YAML frontmatter block (metadata, activation criteria, required tools) followed by a markdown body that gets injected into the LLM context when the skill activates.
### Trust Model
| Trust Level | Source | Tool Access |
|-------------|--------|-------------|
| **Trusted** | User-placed in `~/.ironclaw/skills/` or workspace `skills/` | All tools available to the agent |
| **Installed** | Downloaded from ClawHub registry | Read-only tools only (no shell, file write, HTTP) |
### SKILL.md Format
```yaml
---
name: my-skill
version: 0.1.0
description: Does something useful
activation:
patterns:
- "deploy to.*production"
keywords:
- "deployment"
max_context_tokens: 2000
metadata:
openclaw:
requires:
bins: [docker, kubectl]
env: [KUBECONFIG]
---
# Deployment Skill
Instructions for the agent when this skill activates...
```
### Selection Pipeline
1. **Gating** -- Check binary/env/config requirements; skip skills whose prerequisites are missing
2. **Scoring** -- Deterministic scoring against message content using keywords, tags, and regex patterns
3. **Budget** -- Select top-scoring skills that fit within `SKILLS_MAX_TOKENS` prompt budget
4. **Attenuation** -- Apply trust-based tool ceiling; installed skills lose access to dangerous tools
### Skill Tools
Four built-in tools for managing skills at runtime:
- **`skill_list`** -- List all discovered skills with trust level and status
- **`skill_search`** -- Search ClawHub registry for available skills
- **`skill_install`** -- Download and install a skill from ClawHub
- **`skill_remove`** -- Remove an installed skill
### Skill Directories
- `~/.ironclaw/skills/` -- User's global skills (trusted)
- `<workspace>/skills/` -- Per-workspace skills (trusted)
- `~/.ironclaw/installed_skills/` -- Registry-installed skills (installed trust)
Skills configuration: see Configuration section above.
## Docker Sandbox
The `src/sandbox/` module provides Docker-based isolation for job execution with a network proxy that controls outbound access and injects credentials.
### Sandbox Policies
| Policy | Filesystem | Network | Use Case |
|--------|-----------|---------|----------|
| **ReadOnly** | Read-only workspace mount | Allowlisted domains only | Analysis, code review |
| **WorkspaceWrite** | Read-write workspace mount | Allowlisted domains only | Code generation, file edits |
| **FullAccess** | Full filesystem | Unrestricted | Trusted admin tasks |
### Network Proxy
Containers route all HTTP/HTTPS traffic through a host-side proxy (`src/sandbox/proxy/`):
- **Domain allowlist** -- Only allowlisted domains are reachable (default: package registries, docs sites, GitHub, common APIs)
- **Credential injection** -- The `CredentialResolver` trait injects auth headers into proxied requests so secrets never enter the container environment
- **CONNECT tunnel** -- HTTPS traffic uses CONNECT method; the proxy validates the target domain against the allowlist before establishing the tunnel
- **Policy decisions** -- The `NetworkPolicyDecider` trait allows custom logic for allow/deny/inject decisions per request
### Zero-Exposure Credential Model
Secrets (API keys, tokens) are stored encrypted on the host and injected into HTTP requests by the proxy at transit time. Container processes never have access to raw credential values, preventing exfiltration even if container code is compromised.
Sandbox configuration: see Configuration section above.
## Testing
Tests are in `mod tests {}` blocks at the bottom of each file. Run specific module tests:
@@ -451,164 +617,13 @@ Key test patterns:
7. **Webhook trigger endpoint** - Routines webhook trigger not yet exposed in web gateway
8. **Full channel status view** - Gateway status widget exists, but no per-channel connection dashboard
### Completed
## Tool Architecture
-**Workspace integration** - Memory tools registered, workspace passed to Agent and heartbeat
-**WASM sandboxing** - Full implementation in `tools/wasm/` with fuel metering, memory limits, capabilities
-**Dynamic tool building** - `tools/builder/` has LlmSoftwareBuilder with iterative build loop
-**HTTP webhook security** - Secret validation implemented, proper error handling (no panics)
-**Embeddings integration** - OpenAI and NEAR AI providers wired to workspace for semantic search
-**Workspace system prompt** - Identity files (AGENTS.md, SOUL.md, USER.md, IDENTITY.md) injected into LLM context
-**Heartbeat notifications** - Route through channel manager (broadcast API) instead of logging-only
-**Auto-context compaction** - Triggers automatically when context exceeds threshold
-**Embedding backfill** - Runs on startup when embeddings provider is enabled
-**Clippy clean** - All warnings addressed via config struct refactoring
-**Tool approval enforcement** - Tools with `requires_approval()` (shell, http, file write/patch, build_software) now gate execution, track auto-approved tools per session
-**Tool definition refresh** - Tool definitions refreshed each iteration so newly built tools become visible in same session
-**Worker tool call handling** - Uses `respond_with_tools()` to properly execute tool calls when `select_tools()` returns empty
-**Gateway control plane** - Web gateway with 40+ API endpoints, SSE/WebSocket
-**Web Control UI** - Browser-based dashboard with chat, memory, jobs, logs, extensions, routines
-**Slack/Telegram channels** - Implemented as WASM tools
-**Docker sandbox** - Orchestrator/worker containers with per-job auth
-**Claude Code mode** - Delegate jobs to Claude CLI inside containers
-**Routines system** - Cron, event, webhook, and manual triggers with guardrails
-**Extension management** - Install, auth, activate MCP/WASM extensions via CLI and web UI
-**libSQL/Turso backend** - Database trait abstraction (`src/db/`), feature-gated dual backend support (postgres/libsql), embedded SQLite for zero-dependency local mode
**Keep tool-specific logic out of the main agent codebase.** The main agent provides generic infrastructure; tools are self-contained units that declare their requirements through `capabilities.json` files (API endpoints, credentials, rate limits, auth setup). Service-specific auth flows, CLI commands, and configuration do not belong in the main agent.
## Adding a New Tool
Tools can be built as **WASM** (sandboxed, credential-injected, single binary) or **MCP servers** (ecosystem of pre-built servers, any language, but no sandbox). Both are first-class via `ironclaw tool install`. Auth is declared in capabilities files with OAuth and manual token entry support.
### Built-in Tools (Rust)
1. Create `src/tools/builtin/my_tool.rs`
2. Implement the `Tool` trait
3. Add `mod my_tool;` and `pub use` in `src/tools/builtin/mod.rs`
4. Register in `ToolRegistry::register_builtin_tools()` in `registry.rs`
5. Add tests
### WASM Tools (Recommended)
WASM tools are the preferred way to add new capabilities. They run in a sandboxed environment with explicit capabilities.
1. Create a new crate in `tools-src/<name>/`
2. Implement the WIT interface (`wit/tool.wit`)
3. Create `<name>.capabilities.json` declaring required permissions
4. Build with `cargo build --target wasm32-wasip2 --release`
5. Install with `ironclaw tool install path/to/tool.wasm`
See `tools-src/` for examples.
## Tool Architecture Principles
**CRITICAL: Keep tool-specific logic out of the main agent codebase.**
The main agent provides generic infrastructure; tools are self-contained units that declare their requirements through capabilities files.
### What Goes in Tools (capabilities.json)
- API endpoints the tool needs (HTTP allowlist)
- Credentials required (secret names, injection locations)
- Rate limits and timeouts
- Auth setup instructions (see below)
- Workspace paths the tool can read
### What Does NOT Go in Main Agent
- Service-specific auth flows (OAuth for Notion, Slack, etc.)
- Service-specific CLI commands (`auth notion`, `auth slack`)
- Service-specific configuration handling
- Hardcoded API URLs or token formats
### Tool Authentication
Tools declare their auth requirements in `<tool>.capabilities.json` under the `auth` section. Two methods are supported:
#### OAuth (Browser-based login)
For services that support OAuth, users just click through browser login:
```json
{
"auth": {
"secret_name": "notion_api_token",
"display_name": "Notion",
"oauth": {
"authorization_url": "https://api.notion.com/v1/oauth/authorize",
"token_url": "https://api.notion.com/v1/oauth/token",
"client_id_env": "NOTION_OAUTH_CLIENT_ID",
"client_secret_env": "NOTION_OAUTH_CLIENT_SECRET",
"scopes": [],
"use_pkce": false,
"extra_params": { "owner": "user" }
},
"env_var": "NOTION_TOKEN"
}
}
```
To enable OAuth for a tool:
1. Register a public OAuth app with the service (e.g., notion.so/my-integrations)
2. Configure redirect URIs: `http://localhost:9876/callback` through `http://localhost:9886/callback`
3. Set environment variables for client_id and client_secret
#### Manual Token Entry (Fallback)
For services without OAuth or when OAuth isn't configured:
```json
{
"auth": {
"secret_name": "openai_api_key",
"display_name": "OpenAI",
"instructions": "Get your API key from platform.openai.com/api-keys",
"setup_url": "https://platform.openai.com/api-keys",
"token_hint": "Starts with 'sk-'",
"env_var": "OPENAI_API_KEY"
}
}
```
#### Auth Flow Priority
When running `ironclaw tool auth <tool>`:
1. Check `env_var` - if set in environment, use it directly
2. Check `oauth` - if configured, open browser for OAuth flow
3. Fall back to `instructions` + manual token entry
The agent reads auth config from the tool's capabilities file and provides the appropriate flow. No service-specific code in the main agent.
### WASM Tools vs MCP Servers: When to Use Which
Both are first-class in the extension system (`ironclaw tool install` handles both), but they have different strengths.
**WASM Tools (IronClaw native)**
- Sandboxed: fuel metering, memory limits, no access except what's allowlisted
- Credentials injected by host runtime, tool code never sees the actual token
- Output scanned for secret leakage before returning to the LLM
- Auth (OAuth/manual) declared in `capabilities.json`, agent handles the flow
- Single binary, no process management, works offline
- Cost: must build yourself in Rust, no ecosystem, synchronous only
**MCP Servers (Model Context Protocol)**
- Growing ecosystem of pre-built servers (GitHub, Notion, Postgres, etc.)
- Any language (TypeScript/Python most common)
- Can do websockets, streaming, background polling
- Cost: external process with full system access (no sandbox), manages own credentials, IronClaw can't prevent leaks
**Decision guide:**
| Scenario | Use |
|----------|-----|
| Good MCP server already exists | **MCP** |
| Handles sensitive credentials (email send, banking) | **WASM** |
| Quick prototype or one-off integration | **MCP** |
| Core capability you'll maintain long-term | **WASM** |
| Needs background connections (websockets, polling) | **MCP** |
| Multiple tools share one OAuth token (e.g., Google suite) | **WASM** |
The LLM-facing interface is identical for both (tool name, schema, execute), so swapping between them is transparent to the agent.
See `src/tools/README.md` for full tool architecture, adding new tools (built-in Rust and WASM), auth JSON examples, and WASM vs MCP decision guide.
## Adding a New Channel
@@ -645,154 +660,15 @@ for that module's behavior. When modifying code in a module that has a spec:
| Module | Spec File |
|--------|-----------|
| `src/setup/` | `src/setup/README.md` |
## Code Style
- Use `crate::` imports, not `super::`
- No `pub use` re-exports unless exposing to downstream consumers
- Prefer strong types over strings (enums, newtypes)
- Keep functions focused, extract helpers when logic is reused
- Comments for non-obvious logic only
## Review & Fix Discipline
Hard-won lessons from code review -- follow these when fixing bugs or addressing review feedback.
### Fix the pattern, not just the instance
When a reviewer flags a bug (e.g., TOCTOU race in INSERT + SELECT-back), search the entire codebase for all instances of that same pattern. A fix in `SecretsStore::create()` that doesn't also fix `WasmToolStore::store()` is half a fix.
### Propagate architectural fixes to satellite types
If a core type changes its concurrency model (e.g., `LibSqlBackend` switches to connection-per-operation), every type that was handed a resource from the old model (e.g., `LibSqlSecretsStore`, `LibSqlWasmToolStore` holding a single `Connection`) must also be updated. Grep for the old type across the codebase.
### Schema translation is more than DDL
When translating a database schema between backends (PostgreSQL to libSQL, etc.), check for:
- **Indexes** -- diff `CREATE INDEX` statements between the two schemas
- **Seed data** -- check for `INSERT INTO` in migrations (e.g., `leak_detection_patterns`)
- **Semantic differences** -- document where SQL functions behave differently (e.g., `json_patch` vs `jsonb_set`)
### Feature flag testing
When adding feature-gated code, test compilation with each feature in isolation:
```bash
cargo check # default features
cargo check --no-default-features --features libsql # libsql only
cargo check --all-features # all features
```
Dead code behind the wrong `#[cfg]` gate will only show up when building with a single feature.
### Mechanical verification before committing
Run these checks on changed files before committing:
- `grep -rnE '\.unwrap\(|\.expect\(' <files>` -- no panics in production
- `grep -rn 'super::' <files>` -- use `crate::` imports
- If you fixed a pattern bug, `grep` for other instances of that pattern across `src/`
| `src/workspace/` | `src/workspace/README.md` |
| `src/tools/` | `src/tools/README.md` |
## Workspace & Memory System
Inspired by [OpenClaw](https://github.com/openclaw/openclaw), the workspace provides persistent memory for agents with a flexible filesystem-like structure.
OpenClaw-inspired persistent memory with a flexible filesystem-like structure. Principle: "Memory is database, not RAM" -- if you want to remember something, write it explicitly. Uses hybrid search combining FTS (keyword) + vector (semantic) via Reciprocal Rank Fusion.
### Key Principles
Four memory tools for LLM use: `memory_search` (hybrid search -- call before answering questions about prior work), `memory_write`, `memory_read`, `memory_tree`. Identity files (AGENTS.md, SOUL.md, USER.md, IDENTITY.md) are injected into the LLM system prompt.
1. **"Memory is database, not RAM"** - If you want to remember something, write it explicitly
2. **Flexible structure** - Create any directory/file hierarchy you need
3. **Self-documenting** - Use README.md files to describe directory structure
4. **Hybrid search** - Combines FTS (keyword) + vector (semantic) via Reciprocal Rank Fusion
The heartbeat system runs proactive periodic execution (default: 30 minutes), reading `HEARTBEAT.md` and notifying via channel if findings are detected.
### Filesystem Structure
```
workspace/
├── README.md <- Root runbook/index
├── MEMORY.md <- Long-term curated memory
├── HEARTBEAT.md <- Periodic checklist
├── IDENTITY.md <- Agent name, nature, vibe
├── SOUL.md <- Core values
├── AGENTS.md <- Behavior instructions
├── USER.md <- User context
├── context/ <- Identity-related docs
│ ├── vision.md
│ └── priorities.md
├── daily/ <- Daily logs
│ ├── 2024-01-15.md
│ └── 2024-01-16.md
├── projects/ <- Arbitrary structure
│ └── alpha/
│ ├── README.md
│ └── notes.md
└── ...
```
### Using the Workspace
```rust
use crate::workspace::{Workspace, OpenAiEmbeddings, paths};
// Create workspace for a user
let workspace = Workspace::new("user_123", pool)
.with_embeddings(Arc::new(OpenAiEmbeddings::new(api_key)));
// Read/write any path
let doc = workspace.read("projects/alpha/notes.md").await?;
workspace.write("context/priorities.md", "# Priorities\n\n1. Feature X").await?;
workspace.append("daily/2024-01-15.md", "Completed task X").await?;
// Convenience methods for well-known files
workspace.append_memory("User prefers dark mode").await?;
workspace.append_daily_log("Session note").await?;
// List directory contents
let entries = workspace.list("projects/").await?;
// Search (hybrid FTS + vector)
let results = workspace.search("dark mode preference", 5).await?;
// Get system prompt from identity files
let prompt = workspace.system_prompt().await?;
```
### Memory Tools
Four tools for LLM use:
- **`memory_search`** - Hybrid search, MUST be called before answering questions about prior work
- **`memory_write`** - Write to any path (memory, daily_log, or custom paths)
- **`memory_read`** - Read any file by path
- **`memory_tree`** - View workspace structure as a tree (depth parameter, default 1)
### Hybrid Search (RRF)
Combines full-text search and vector similarity using Reciprocal Rank Fusion:
```
score(d) = Σ 1/(k + rank(d)) for each method where d appears
```
Default k=60. Results from both methods are combined, with documents appearing in both getting boosted scores.
**Backend differences:**
- **PostgreSQL:** `ts_rank_cd` for FTS, pgvector cosine distance for vectors, full RRF
- **libSQL:** FTS5 for keyword search only (vector search via `libsql_vector_idx` not yet wired)
### Heartbeat System
Proactive periodic execution (default: 30 minutes):
1. Reads `HEARTBEAT.md` checklist
2. Runs agent turn with checklist prompt
3. If findings, notifies via channel
4. If nothing, agent replies "HEARTBEAT_OK" (no notification)
```rust
use crate::agent::{HeartbeatConfig, spawn_heartbeat};
let config = HeartbeatConfig::default()
.with_interval(Duration::from_secs(60 * 30))
.with_notify("user_123", "telegram");
spawn_heartbeat(config, workspace, llm, response_tx);
```
### Chunking Strategy
Documents are chunked for search indexing:
- Default: 800 words per chunk (roughly 800 tokens for English)
- 15% overlap between chunks for context preservation
- Minimum chunk size: 50 words (tiny trailing chunks merge with previous)
See `src/workspace/README.md` for full API documentation, filesystem structure, hybrid search details, chunking strategy, and heartbeat system.
Generated
+1 -1
View File
@@ -2490,7 +2490,7 @@ dependencies = [
[[package]]
name = "ironclaw"
version = "0.7.0"
version = "0.8.0"
dependencies = [
"aes-gcm",
"aho-corasick",
+11 -1
View File
@@ -1,15 +1,25 @@
[workspace]
members = [".", "benchmarks"]
exclude = [
"channels-src/discord",
"channels-src/telegram",
"channels-src/slack",
"channels-src/whatsapp",
"tools-src/github",
"tools-src/gmail",
"tools-src/google-calendar",
"tools-src/google-docs",
"tools-src/google-drive",
"tools-src/google-sheets",
"tools-src/google-slides",
"tools-src/okta",
"tools-src/slack",
"tools-src/telegram",
]
[package]
name = "ironclaw"
version = "0.7.0"
version = "0.8.0"
edition = "2024"
rust-version = "1.92"
description = "Secure personal AI assistant that protects your data and expands its capabilities on the fly"
-2
View File
@@ -182,7 +182,6 @@ mod tests {
input_tokens: 100,
output_tokens: 50,
finish_reason: FinishReason::Stop,
response_id: None,
})
}
@@ -196,7 +195,6 @@ mod tests {
input_tokens: 200,
output_tokens: 100,
finish_reason: FinishReason::Stop,
response_id: None,
})
}
}
+1 -1
View File
@@ -402,7 +402,7 @@ async fn run_task_isolated(params: TaskRunParams<'_>) -> TaskResult {
let mut channels = ChannelManager::new();
channels.add(Box::new(bench_channel));
let agent = Agent::new(agent_config, deps, channels, None, None, None, None);
let agent = Agent::new(agent_config, deps, channels, None, None, None, None, None);
// Build the full prompt with context
let full_prompt = if let Some(ref ctx) = task.context {
+2
View File
@@ -21,3 +21,5 @@ lto = true
codegen-units = 1
[workspace]
+2
View File
@@ -27,3 +27,5 @@ opt-level = "s"
lto = true
strip = true
codegen-units = 1
[workspace]
+2
View File
@@ -25,3 +25,5 @@ opt-level = "s"
lto = true
strip = true
codegen-units = 1
[workspace]
+2
View File
@@ -16,3 +16,5 @@ serde_json = "1"
opt-level = "s"
lto = true
strip = true
[workspace]
+8 -5
View File
@@ -2,12 +2,15 @@
# Do not use placeholder passwords in production.
DATABASE_URL=postgres://ironclaw:CHANGE_ME@localhost:5432/ironclaw
# NEAR AI
NEARAI_SESSION_TOKEN=CHANGE_ME
# NEAR AI Cloud (API key auth, Chat Completions API)
# Get an API key from https://cloud.near.ai
NEARAI_API_KEY=CHANGE_ME
NEARAI_MODEL=claude-3-5-sonnet-20241022
NEARAI_BASE_URL=https://private.near.ai
NEARAI_AUTH_URL=https://private.near.ai
NEARAI_API_MODE=chat_completions
NEARAI_BASE_URL=https://cloud-api.near.ai
# Or use NEAR AI Chat (session token auth, Responses API):
# NEARAI_SESSION_TOKEN=sess_...
# NEARAI_BASE_URL=https://private.near.ai
# Agent
AGENT_NAME=ironclaw
+42
View File
@@ -0,0 +1,42 @@
{
"bundles": {
"google": {
"display_name": "Google Suite",
"description": "Gmail, Calendar, Drive, Docs, Sheets, Slides",
"extensions": [
"tools/gmail",
"tools/google-calendar",
"tools/google-docs",
"tools/google-drive",
"tools/google-sheets",
"tools/google-slides"
],
"shared_auth": "google_oauth_token"
},
"messaging": {
"display_name": "Messaging Channels",
"description": "Discord, Telegram, Slack, and WhatsApp channels",
"extensions": [
"channels/discord",
"channels/telegram",
"channels/slack",
"channels/whatsapp"
],
"shared_auth": null
},
"default": {
"display_name": "Recommended Set",
"description": "Core tools and channels for a productive setup",
"extensions": [
"tools/github",
"tools/gmail",
"tools/google-calendar",
"tools/google-drive",
"tools/slack",
"channels/telegram",
"channels/slack"
],
"shared_auth": null
}
}
}
+31
View File
@@ -0,0 +1,31 @@
{
"name": "discord",
"display_name": "Discord",
"kind": "channel",
"version": "0.1.0",
"description": "Discord Gateway/Webhook channel for slash commands, buttons, and messages",
"keywords": ["messaging", "chat", "discord", "bot"],
"source": {
"dir": "channels-src/discord",
"capabilities": "discord.capabilities.json",
"crate_name": "discord-channel"
},
"artifacts": {
"wasm32-wasip2": {
"url": null,
"sha256": null
}
},
"auth_summary": {
"method": "manual",
"provider": "Discord",
"secrets": ["discord_bot_token"],
"shared_auth": null,
"setup_url": "https://discord.com/developers/applications"
},
"tags": ["messaging"]
}
+31
View File
@@ -0,0 +1,31 @@
{
"name": "slack",
"display_name": "Slack",
"kind": "channel",
"version": "0.1.0",
"description": "Slack Events API channel for receiving and responding to Slack messages",
"keywords": ["messaging", "chat", "workspace", "slack"],
"source": {
"dir": "channels-src/slack",
"capabilities": "slack.capabilities.json",
"crate_name": "slack-channel"
},
"artifacts": {
"wasm32-wasip2": {
"url": null,
"sha256": null
}
},
"auth_summary": {
"method": "manual",
"provider": "Slack",
"secrets": ["slack_bot_token", "slack_signing_secret"],
"shared_auth": null,
"setup_url": "https://api.slack.com/apps"
},
"tags": ["default", "messaging"]
}
+31
View File
@@ -0,0 +1,31 @@
{
"name": "telegram",
"display_name": "Telegram",
"kind": "channel",
"version": "0.1.0",
"description": "Telegram Bot API channel for receiving and responding to messages",
"keywords": ["messaging", "bot", "chat", "telegram"],
"source": {
"dir": "channels-src/telegram",
"capabilities": "telegram.capabilities.json",
"crate_name": "telegram-channel"
},
"artifacts": {
"wasm32-wasip2": {
"url": null,
"sha256": null
}
},
"auth_summary": {
"method": "manual",
"provider": "Telegram",
"secrets": ["telegram_bot_token"],
"shared_auth": null,
"setup_url": "https://t.me/BotFather"
},
"tags": ["default", "messaging"]
}
+31
View File
@@ -0,0 +1,31 @@
{
"name": "whatsapp",
"display_name": "WhatsApp",
"kind": "channel",
"version": "0.1.0",
"description": "WhatsApp Cloud API channel for receiving and responding to messages",
"keywords": ["messaging", "chat", "whatsapp", "meta"],
"source": {
"dir": "channels-src/whatsapp",
"capabilities": "whatsapp.capabilities.json",
"crate_name": "whatsapp-channel"
},
"artifacts": {
"wasm32-wasip2": {
"url": null,
"sha256": null
}
},
"auth_summary": {
"method": "manual",
"provider": "Meta",
"secrets": ["whatsapp_access_token", "whatsapp_verify_token"],
"shared_auth": null,
"setup_url": "https://developers.facebook.com/apps/"
},
"tags": ["messaging"]
}
+31
View File
@@ -0,0 +1,31 @@
{
"name": "github",
"display_name": "GitHub",
"kind": "tool",
"version": "0.1.0",
"description": "GitHub integration for issues, PRs, repos, and code search",
"keywords": ["git", "code", "issues", "pull-requests", "repositories"],
"source": {
"dir": "tools-src/github",
"capabilities": "github-tool.capabilities.json",
"crate_name": "github-tool"
},
"artifacts": {
"wasm32-wasip2": {
"url": null,
"sha256": null
}
},
"auth_summary": {
"method": "manual",
"provider": "GitHub",
"secrets": ["github_token"],
"shared_auth": null,
"setup_url": "https://github.com/settings/tokens"
},
"tags": ["default", "development"]
}
+31
View File
@@ -0,0 +1,31 @@
{
"name": "gmail",
"display_name": "Gmail",
"kind": "tool",
"version": "0.1.0",
"description": "Read, send, and manage Gmail messages and threads",
"keywords": ["email", "google", "mail", "messaging"],
"source": {
"dir": "tools-src/gmail",
"capabilities": "gmail-tool.capabilities.json",
"crate_name": "gmail-tool"
},
"artifacts": {
"wasm32-wasip2": {
"url": null,
"sha256": null
}
},
"auth_summary": {
"method": "oauth",
"provider": "Google",
"secrets": ["google_oauth_token"],
"shared_auth": "google_oauth_token",
"setup_url": "https://console.cloud.google.com/apis/credentials"
},
"tags": ["default", "google", "messaging"]
}
+31
View File
@@ -0,0 +1,31 @@
{
"name": "google-calendar",
"display_name": "Google Calendar",
"kind": "tool",
"version": "0.1.0",
"description": "Create, read, update, and delete Google Calendar events",
"keywords": ["calendar", "google", "scheduling", "events"],
"source": {
"dir": "tools-src/google-calendar",
"capabilities": "google-calendar-tool.capabilities.json",
"crate_name": "google-calendar-tool"
},
"artifacts": {
"wasm32-wasip2": {
"url": null,
"sha256": null
}
},
"auth_summary": {
"method": "oauth",
"provider": "Google",
"secrets": ["google_oauth_token"],
"shared_auth": "google_oauth_token",
"setup_url": "https://console.cloud.google.com/apis/credentials"
},
"tags": ["default", "google", "productivity"]
}
+31
View File
@@ -0,0 +1,31 @@
{
"name": "google-docs",
"display_name": "Google Docs",
"kind": "tool",
"version": "0.1.0",
"description": "Create and edit Google Docs documents",
"keywords": ["documents", "google", "writing", "docs"],
"source": {
"dir": "tools-src/google-docs",
"capabilities": "google-docs-tool.capabilities.json",
"crate_name": "google-docs-tool"
},
"artifacts": {
"wasm32-wasip2": {
"url": null,
"sha256": null
}
},
"auth_summary": {
"method": "oauth",
"provider": "Google",
"secrets": ["google_oauth_token"],
"shared_auth": "google_oauth_token",
"setup_url": "https://console.cloud.google.com/apis/credentials"
},
"tags": ["google", "productivity"]
}
+31
View File
@@ -0,0 +1,31 @@
{
"name": "google-drive",
"display_name": "Google Drive",
"kind": "tool",
"version": "0.1.0",
"description": "Upload, download, search, and manage Google Drive files and folders",
"keywords": ["storage", "google", "files", "drive"],
"source": {
"dir": "tools-src/google-drive",
"capabilities": "google-drive-tool.capabilities.json",
"crate_name": "google-drive-tool"
},
"artifacts": {
"wasm32-wasip2": {
"url": null,
"sha256": null
}
},
"auth_summary": {
"method": "oauth",
"provider": "Google",
"secrets": ["google_oauth_token"],
"shared_auth": "google_oauth_token",
"setup_url": "https://console.cloud.google.com/apis/credentials"
},
"tags": ["default", "google", "storage"]
}
+31
View File
@@ -0,0 +1,31 @@
{
"name": "google-sheets",
"display_name": "Google Sheets",
"kind": "tool",
"version": "0.1.0",
"description": "Read and write Google Sheets spreadsheet data",
"keywords": ["spreadsheets", "google", "data", "sheets"],
"source": {
"dir": "tools-src/google-sheets",
"capabilities": "google-sheets-tool.capabilities.json",
"crate_name": "google-sheets-tool"
},
"artifacts": {
"wasm32-wasip2": {
"url": null,
"sha256": null
}
},
"auth_summary": {
"method": "oauth",
"provider": "Google",
"secrets": ["google_oauth_token"],
"shared_auth": "google_oauth_token",
"setup_url": "https://console.cloud.google.com/apis/credentials"
},
"tags": ["google", "productivity"]
}
+31
View File
@@ -0,0 +1,31 @@
{
"name": "google-slides",
"display_name": "Google Slides",
"kind": "tool",
"version": "0.1.0",
"description": "Create and edit Google Slides presentations",
"keywords": ["presentations", "google", "slides"],
"source": {
"dir": "tools-src/google-slides",
"capabilities": "google-slides-tool.capabilities.json",
"crate_name": "google-slides-tool"
},
"artifacts": {
"wasm32-wasip2": {
"url": null,
"sha256": null
}
},
"auth_summary": {
"method": "oauth",
"provider": "Google",
"secrets": ["google_oauth_token"],
"shared_auth": "google_oauth_token",
"setup_url": "https://console.cloud.google.com/apis/credentials"
},
"tags": ["google", "productivity"]
}
+31
View File
@@ -0,0 +1,31 @@
{
"name": "okta",
"display_name": "Okta",
"kind": "tool",
"version": "0.1.0",
"description": "Okta SSO for user profile, app catalog, and SSO launch links",
"keywords": ["sso", "identity", "authentication", "okta"],
"source": {
"dir": "tools-src/okta",
"capabilities": "okta-tool.capabilities.json",
"crate_name": "okta-tool"
},
"artifacts": {
"wasm32-wasip2": {
"url": null,
"sha256": null
}
},
"auth_summary": {
"method": "oauth",
"provider": "Okta",
"secrets": ["okta_oauth_token"],
"shared_auth": null,
"setup_url": "https://developer.okta.com/docs/guides/implement-oauth-for-okta/main/"
},
"tags": ["identity"]
}
+31
View File
@@ -0,0 +1,31 @@
{
"name": "slack",
"display_name": "Slack",
"kind": "tool",
"version": "0.1.0",
"description": "Post messages, read channels, and manage conversations via Slack API",
"keywords": ["messaging", "chat", "workspace"],
"source": {
"dir": "tools-src/slack",
"capabilities": "slack-tool.capabilities.json",
"crate_name": "slack-tool"
},
"artifacts": {
"wasm32-wasip2": {
"url": null,
"sha256": null
}
},
"auth_summary": {
"method": "oauth",
"provider": "Slack",
"secrets": ["slack_bot_token"],
"shared_auth": null,
"setup_url": "https://api.slack.com/apps"
},
"tags": ["default", "messaging"]
}
+31
View File
@@ -0,0 +1,31 @@
{
"name": "telegram",
"display_name": "Telegram",
"kind": "tool",
"version": "0.1.0",
"description": "Telegram user-mode integration via MTProto for messages and contacts",
"keywords": ["messaging", "chat", "telegram", "mtproto"],
"source": {
"dir": "tools-src/telegram",
"capabilities": "telegram-tool.capabilities.json",
"crate_name": "telegram-tool"
},
"artifacts": {
"wasm32-wasip2": {
"url": null,
"sha256": null
}
},
"auth_summary": {
"method": "manual",
"provider": "Telegram",
"secrets": ["telegram_api_id", "telegram_api_hash"],
"shared_auth": null,
"setup_url": "https://my.telegram.org/apps"
},
"tags": ["messaging"]
}
+12 -4
View File
@@ -85,6 +85,7 @@ pub struct Agent {
pub(super) session_manager: Arc<SessionManager>,
pub(super) context_monitor: ContextMonitor,
pub(super) heartbeat_config: Option<HeartbeatConfig>,
pub(super) hygiene_config: Option<crate::config::HygieneConfig>,
pub(super) routine_config: Option<RoutineConfig>,
}
@@ -93,11 +94,13 @@ impl Agent {
///
/// Optionally accepts pre-created `ContextManager` and `SessionManager` for sharing
/// with external components (job tools, web gateway). Creates new ones if not provided.
#[allow(clippy::too_many_arguments)]
pub fn new(
config: AgentConfig,
deps: AgentDeps,
channels: ChannelManager,
heartbeat_config: Option<HeartbeatConfig>,
hygiene_config: Option<crate::config::HygieneConfig>,
routine_config: Option<RoutineConfig>,
context_manager: Option<Arc<ContextManager>>,
session_manager: Option<Arc<SessionManager>>,
@@ -127,6 +130,7 @@ impl Agent {
session_manager,
context_monitor: ContextMonitor::new(),
heartbeat_config,
hygiene_config,
routine_config,
}
}
@@ -354,14 +358,18 @@ impl Agent {
}
});
tracing::info!(
"Heartbeat enabled with {}s interval",
hb_config.interval_secs
);
let hygiene = self
.hygiene_config
.as_ref()
.map(|h| h.to_workspace_config())
.unwrap_or_default();
Some(spawn_heartbeat(
config,
hygiene,
workspace.clone(),
self.cheap_llm().clone(),
self.safety().clone(),
Some(notify_tx),
))
} else {
+11 -7
View File
@@ -13,7 +13,7 @@ use crate::agent::submission::SubmissionResult;
use crate::agent::{Agent, MessageIntent};
use crate::channels::{IncomingMessage, StatusUpdate};
use crate::error::Error;
use crate::llm::ChatMessage;
use crate::llm::{ChatMessage, Reasoning};
impl Agent {
/// Handle job-related intents without turn tracking.
@@ -232,8 +232,10 @@ impl Agent {
let runner = crate::agent::HeartbeatRunner::new(
crate::agent::HeartbeatConfig::default(),
crate::workspace::hygiene::HygieneConfig::default(),
workspace.clone(),
self.llm().clone(),
self.safety().clone(),
);
match runner.check_heartbeat().await {
@@ -294,10 +296,11 @@ impl Agent {
.with_max_tokens(512)
.with_temperature(0.3);
match self.llm().complete(request).await {
Ok(response) => Ok(SubmissionResult::response(format!(
let reasoning = Reasoning::new(self.llm().clone(), self.safety().clone());
match reasoning.complete(request).await {
Ok((text, _usage)) => Ok(SubmissionResult::response(format!(
"Thread Summary:\n\n{}",
response.content.trim()
text.trim()
))),
Err(e) => Ok(SubmissionResult::error(format!("Summarize failed: {}", e))),
}
@@ -341,10 +344,11 @@ impl Agent {
.with_max_tokens(512)
.with_temperature(0.5);
match self.llm().complete(request).await {
Ok(response) => Ok(SubmissionResult::response(format!(
let reasoning = Reasoning::new(self.llm().clone(), self.safety().clone());
match reasoning.complete(request).await {
Ok((text, _usage)) => Ok(SubmissionResult::response(format!(
"Suggested Next Steps:\n\n{}",
response.content.trim()
text.trim()
))),
Err(e) => Ok(SubmissionResult::error(format!("Suggest failed: {}", e))),
}
+28 -7
View File
@@ -12,7 +12,8 @@ use chrono::Utc;
use crate::agent::context_monitor::{CompactionStrategy, ContextBreakdown};
use crate::agent::session::Thread;
use crate::error::Error;
use crate::llm::{ChatMessage, CompletionRequest, LlmProvider};
use crate::llm::{ChatMessage, CompletionRequest, LlmProvider, Reasoning};
use crate::safety::SafetyLayer;
use crate::workspace::Workspace;
/// Result of a compaction operation.
@@ -33,12 +34,13 @@ pub struct CompactionResult {
/// Compacts conversation context to stay within limits.
pub struct ContextCompactor {
llm: Arc<dyn LlmProvider>,
safety: Arc<SafetyLayer>,
}
impl ContextCompactor {
/// Create a new context compactor.
pub fn new(llm: Arc<dyn LlmProvider>) -> Self {
Self { llm }
pub fn new(llm: Arc<dyn LlmProvider>, safety: Arc<SafetyLayer>) -> Self {
Self { llm, safety }
}
/// Compact a thread's context using the given strategy.
@@ -105,7 +107,16 @@ impl ContextCompactor {
// Write to workspace if available
let summary_written = if let Some(ws) = workspace {
self.write_summary_to_workspace(ws, &summary).await.is_ok()
match self.write_summary_to_workspace(ws, &summary).await {
Ok(()) => true,
Err(e) => {
tracing::warn!(
"Compaction summary write failed (turns will still be truncated): {}",
e
);
false
}
}
} else {
false
};
@@ -157,7 +168,16 @@ impl ContextCompactor {
let content = format_turns_for_storage(old_turns);
// Write to workspace
let written = self.write_context_to_workspace(ws, &content).await.is_ok();
let written = match self.write_context_to_workspace(ws, &content).await {
Ok(()) => true,
Err(e) => {
tracing::warn!(
"Compaction context write failed (turns will still be truncated): {}",
e
);
false
}
};
// Truncate
thread.truncate_turns(keep_recent);
@@ -213,8 +233,9 @@ Be brief but capture all important details. Use bullet points."#,
.with_max_tokens(1024)
.with_temperature(0.3);
let response = self.llm.complete(request).await?;
Ok(response.content)
let reasoning = Reasoning::new(self.llm.clone(), self.safety.clone());
let (text, _) = reasoning.complete(request).await?;
Ok(text)
}
/// Write a summary to the workspace daily log.
+716 -277
View File
File diff suppressed because it is too large Load Diff
+32 -13
View File
@@ -29,8 +29,10 @@ use std::time::Duration;
use tokio::sync::mpsc;
use crate::channels::OutgoingResponse;
use crate::llm::{ChatMessage, CompletionRequest, FinishReason, LlmProvider};
use crate::llm::{ChatMessage, CompletionRequest, LlmProvider, Reasoning};
use crate::safety::SafetyLayer;
use crate::workspace::Workspace;
use crate::workspace::hygiene::HygieneConfig;
/// Configuration for the heartbeat runner.
#[derive(Debug, Clone)]
@@ -96,8 +98,10 @@ pub enum HeartbeatResult {
/// Heartbeat runner for proactive periodic execution.
pub struct HeartbeatRunner {
config: HeartbeatConfig,
hygiene_config: HygieneConfig,
workspace: Arc<Workspace>,
llm: Arc<dyn LlmProvider>,
safety: Arc<SafetyLayer>,
response_tx: Option<mpsc::Sender<OutgoingResponse>>,
consecutive_failures: u32,
}
@@ -106,13 +110,17 @@ impl HeartbeatRunner {
/// Create a new heartbeat runner.
pub fn new(
config: HeartbeatConfig,
hygiene_config: HygieneConfig,
workspace: Arc<Workspace>,
llm: Arc<dyn LlmProvider>,
safety: Arc<SafetyLayer>,
) -> Self {
Self {
config,
hygiene_config,
workspace,
llm,
safety,
response_tx: None,
consecutive_failures: 0,
}
@@ -145,6 +153,22 @@ impl HeartbeatRunner {
loop {
interval.tick().await;
// Run memory hygiene in the background so it never delays the
// heartbeat checklist. Failures are logged inside run_if_due.
let hygiene_workspace = Arc::clone(&self.workspace);
let hygiene_config = self.hygiene_config.clone();
tokio::spawn(async move {
let report =
crate::workspace::hygiene::run_if_due(&hygiene_workspace, &hygiene_config)
.await;
if report.had_work() {
tracing::info!(
daily_logs_deleted = report.daily_logs_deleted,
"heartbeat: memory hygiene deleted stale documents"
);
}
});
match self.check_heartbeat().await {
HeartbeatResult::Ok => {
tracing::debug!("Heartbeat OK");
@@ -238,25 +262,18 @@ impl HeartbeatRunner {
.with_max_tokens(max_tokens)
.with_temperature(0.3);
let response = match self.llm.complete(request).await {
let reasoning = Reasoning::new(self.llm.clone(), self.safety.clone());
let (content, _usage) = match reasoning.complete(request).await {
Ok(r) => r,
Err(e) => return HeartbeatResult::Failed(format!("LLM call failed: {}", e)),
};
let content = response.content.trim();
let content = content.trim();
// Guard against empty content. Reasoning models (e.g. GLM-4.7) may
// burn all output tokens on chain-of-thought and return content: null.
if content.is_empty() {
return if response.finish_reason == FinishReason::Length {
HeartbeatResult::Failed(
"LLM response was truncated (finish_reason=length) with no content. \
The model may have exhausted its token budget on reasoning."
.to_string(),
)
} else {
HeartbeatResult::Failed("LLM returned empty content.".to_string())
};
return HeartbeatResult::Failed("LLM returned empty content.".to_string());
}
// Check if nothing needs attention
@@ -332,11 +349,13 @@ fn strip_html_comments(content: &str) -> String {
/// Returns a handle that can be used to stop the runner.
pub fn spawn_heartbeat(
config: HeartbeatConfig,
hygiene_config: HygieneConfig,
workspace: Arc<Workspace>,
llm: Arc<dyn LlmProvider>,
safety: Arc<SafetyLayer>,
response_tx: Option<mpsc::Sender<OutgoingResponse>>,
) -> tokio::task::JoinHandle<()> {
let mut runner = HeartbeatRunner::new(config, workspace, llm);
let mut runner = HeartbeatRunner::new(config, hygiene_config, workspace, llm, safety);
if let Some(tx) = response_tx {
runner = runner.with_response_channel(tx);
}
+1 -1
View File
@@ -44,6 +44,6 @@ pub use self_repair::{BrokenTool, RepairResult, RepairTask, SelfRepair, StuckJob
pub use session::{PendingApproval, PendingAuth, Session, Thread, ThreadState, Turn, TurnState};
pub use session_manager::SessionManager;
pub use submission::{Submission, SubmissionParser, SubmissionResult};
pub use task::{Task, TaskContext, TaskHandler, TaskOutput, TaskStatus};
pub use task::{Task, TaskContext, TaskHandler, TaskOutput};
pub use undo::{Checkpoint, UndoManager};
pub use worker::{Worker, WorkerDeps};
+38 -13
View File
@@ -26,6 +26,8 @@ use chrono::{DateTime, Utc};
use serde::{Deserialize, Serialize};
use uuid::Uuid;
use crate::error::RoutineError;
/// A routine is a named, persistent, user-owned task with a trigger and an action.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct Routine {
@@ -86,13 +88,16 @@ impl Trigger {
}
/// Parse a trigger from its DB representation.
pub fn from_db(trigger_type: &str, config: serde_json::Value) -> Result<Self, String> {
pub fn from_db(trigger_type: &str, config: serde_json::Value) -> Result<Self, RoutineError> {
match trigger_type {
"cron" => {
let schedule = config
.get("schedule")
.and_then(|v| v.as_str())
.ok_or("cron trigger missing 'schedule'")?
.ok_or_else(|| RoutineError::MissingField {
context: "cron trigger".into(),
field: "schedule".into(),
})?
.to_string();
Ok(Trigger::Cron { schedule })
}
@@ -100,7 +105,10 @@ impl Trigger {
let pattern = config
.get("pattern")
.and_then(|v| v.as_str())
.ok_or("event trigger missing 'pattern'")?
.ok_or_else(|| RoutineError::MissingField {
context: "event trigger".into(),
field: "pattern".into(),
})?
.to_string();
let channel = config
.get("channel")
@@ -120,7 +128,9 @@ impl Trigger {
Ok(Trigger::Webhook { path, secret })
}
"manual" => Ok(Trigger::Manual),
other => Err(format!("unknown trigger type: {other}")),
other => Err(RoutineError::UnknownTriggerType {
trigger_type: other.to_string(),
}),
}
}
@@ -186,13 +196,16 @@ impl RoutineAction {
}
/// Parse an action from its DB representation.
pub fn from_db(action_type: &str, config: serde_json::Value) -> Result<Self, String> {
pub fn from_db(action_type: &str, config: serde_json::Value) -> Result<Self, RoutineError> {
match action_type {
"lightweight" => {
let prompt = config
.get("prompt")
.and_then(|v| v.as_str())
.ok_or("lightweight action missing 'prompt'")?
.ok_or_else(|| RoutineError::MissingField {
context: "lightweight action".into(),
field: "prompt".into(),
})?
.to_string();
let context_paths = config
.get("context_paths")
@@ -217,12 +230,18 @@ impl RoutineAction {
let title = config
.get("title")
.and_then(|v| v.as_str())
.ok_or("full_job action missing 'title'")?
.ok_or_else(|| RoutineError::MissingField {
context: "full_job action".into(),
field: "title".into(),
})?
.to_string();
let description = config
.get("description")
.and_then(|v| v.as_str())
.ok_or("full_job action missing 'description'")?
.ok_or_else(|| RoutineError::MissingField {
context: "full_job action".into(),
field: "description".into(),
})?
.to_string();
let max_iterations = config
.get("max_iterations")
@@ -235,7 +254,9 @@ impl RoutineAction {
max_iterations,
})
}
other => Err(format!("unknown action type: {other}")),
other => Err(RoutineError::UnknownActionType {
action_type: other.to_string(),
}),
}
}
@@ -334,14 +355,16 @@ impl std::fmt::Display for RunStatus {
}
impl FromStr for RunStatus {
type Err = String;
type Err = RoutineError;
fn from_str(s: &str) -> Result<Self, Self::Err> {
match s {
"running" => Ok(RunStatus::Running),
"ok" => Ok(RunStatus::Ok),
"attention" => Ok(RunStatus::Attention),
"failed" => Ok(RunStatus::Failed),
other => Err(format!("unknown run status: {other}")),
other => Err(RoutineError::UnknownRunStatus {
status: other.to_string(),
}),
}
}
}
@@ -370,9 +393,11 @@ pub fn content_hash(content: &str) -> u64 {
}
/// Parse a cron expression and compute the next fire time from now.
pub fn next_cron_fire(schedule: &str) -> Result<Option<DateTime<Utc>>, String> {
pub fn next_cron_fire(schedule: &str) -> Result<Option<DateTime<Utc>>, RoutineError> {
let cron_schedule =
cron::Schedule::from_str(schedule).map_err(|e| format!("invalid cron: {e}"))?;
cron::Schedule::from_str(schedule).map_err(|e| RoutineError::InvalidCron {
reason: e.to_string(),
})?;
Ok(cron_schedule.upcoming(Utc).next())
}
+58 -25
View File
@@ -25,6 +25,7 @@ use crate::agent::routine::{
use crate::channels::{IncomingMessage, OutgoingResponse};
use crate::config::RoutineConfig;
use crate::db::Database;
use crate::error::RoutineError;
use crate::llm::{ChatMessage, CompletionRequest, FinishReason, LlmProvider};
use crate::workspace::Workspace;
@@ -174,23 +175,26 @@ impl RoutineEngine {
}
/// Fire a routine manually (from tool call or CLI).
pub async fn fire_manual(&self, routine_id: Uuid) -> Result<Uuid, String> {
pub async fn fire_manual(&self, routine_id: Uuid) -> Result<Uuid, RoutineError> {
let routine = self
.store
.get_routine(routine_id)
.await
.map_err(|e| format!("DB error: {e}"))?
.ok_or_else(|| format!("routine {routine_id} not found"))?;
.map_err(|e| RoutineError::Database {
reason: e.to_string(),
})?
.ok_or(RoutineError::NotFound { id: routine_id })?;
if !routine.enabled {
return Err(format!("routine '{}' is disabled", routine.name));
return Err(RoutineError::Disabled {
name: routine.name.clone(),
});
}
if !self.check_concurrent(&routine).await {
return Err(format!(
"routine '{}' already at max concurrent runs",
routine.name
));
return Err(RoutineError::MaxConcurrent {
name: routine.name.clone(),
});
}
let run_id = Uuid::new_v4();
@@ -209,7 +213,9 @@ impl RoutineEngine {
};
if let Err(e) = self.store.create_routine_run(&run).await {
return Err(format!("failed to create run record: {e}"));
return Err(RoutineError::Database {
reason: format!("failed to create run record: {e}"),
});
}
// Execute inline for manual triggers (caller wants to wait)
@@ -313,13 +319,27 @@ async fn execute_routine(ctx: EngineContext, routine: Routine, run: RoutineRun)
max_tokens,
} => execute_lightweight(&ctx, &routine, prompt, context_paths, *max_tokens).await,
RoutineAction::FullJob { description, .. } => {
// Full job mode: for now, execute as lightweight with the description
// as prompt. Full scheduler integration will come as a follow-up.
tracing::info!(
// Full job mode: scheduler integration not yet implemented.
// Execute as lightweight and prepend a warning to the summary.
tracing::warn!(
routine = %routine.name,
"FullJob mode executing as lightweight (scheduler integration pending)"
"FullJob mode not yet implemented; falling back to lightweight execution"
);
execute_lightweight(&ctx, &routine, description, &[], ctx.max_lightweight_tokens).await
match execute_lightweight(&ctx, &routine, description, &[], ctx.max_lightweight_tokens)
.await
{
Ok((status, summary, tokens)) => {
let warning = "[Note: FullJob mode is not yet implemented. This routine ran as \
a single LLM call without tool access. Configure as 'lightweight' \
or wait for full scheduler integration.]";
let summary = match summary {
Some(s) => Some(format!("{warning}\n\n{s}")),
None => Some(warning.to_string()),
};
Ok((status, summary, tokens))
}
Err(e) => Err(e),
}
}
};
@@ -331,7 +351,7 @@ async fn execute_routine(ctx: EngineContext, routine: Routine, run: RoutineRun)
Ok(execution) => execution,
Err(e) => {
tracing::error!(routine = %routine.name, "Execution failed: {}", e);
(RunStatus::Failed, Some(e), None)
(RunStatus::Failed, Some(e.to_string()), None)
}
};
@@ -384,6 +404,20 @@ async fn execute_routine(ctx: EngineContext, routine: Routine, run: RoutineRun)
.await;
}
/// Sanitize a routine name for use in workspace paths.
/// Only keeps alphanumeric, dash, and underscore characters; replaces everything else.
fn sanitize_routine_name(name: &str) -> String {
name.chars()
.map(|c| {
if c.is_ascii_alphanumeric() || c == '-' || c == '_' {
c
} else {
'_'
}
})
.collect()
}
/// Execute a lightweight routine (single LLM call).
async fn execute_lightweight(
ctx: &EngineContext,
@@ -391,7 +425,7 @@ async fn execute_lightweight(
prompt: &str,
context_paths: &[String],
max_tokens: u32,
) -> Result<(RunStatus, Option<String>, Option<i32>), String> {
) -> Result<(RunStatus, Option<String>, Option<i32>), RoutineError> {
// Load context from workspace
let mut context_parts = Vec::new();
for path in context_paths {
@@ -408,8 +442,9 @@ async fn execute_lightweight(
}
}
// Load routine state from workspace
let state_path = format!("routines/{}/state.md", routine.name);
// Load routine state from workspace (name sanitized to prevent path traversal)
let safe_name = sanitize_routine_name(&routine.name);
let state_path = format!("routines/{safe_name}/state.md");
let state_content = match ctx.workspace.read(&state_path).await {
Ok(doc) => Some(doc.content),
Err(_) => None,
@@ -469,7 +504,9 @@ async fn execute_lightweight(
.llm
.complete(request)
.await
.map_err(|e| format!("LLM call failed: {e}"))?;
.map_err(|e| RoutineError::LlmFailed {
reason: e.to_string(),
})?;
let content = response.content.trim();
let tokens_used = Some((response.input_tokens + response.output_tokens) as i32);
@@ -477,13 +514,9 @@ async fn execute_lightweight(
// Empty content guard (same as heartbeat)
if content.is_empty() {
return if response.finish_reason == FinishReason::Length {
Err(
"LLM response truncated (finish_reason=length) with no content. \
Model may have exhausted token budget on reasoning."
.to_string(),
)
Err(RoutineError::TruncatedResponse)
} else {
Err("LLM returned empty content.".to_string())
Err(RoutineError::EmptyResponse)
};
}
+11 -3
View File
@@ -136,7 +136,9 @@ impl Scheduler {
});
// Start the worker
let _ = tx.send(WorkerMessage::Start).await;
if tx.send(WorkerMessage::Start).await.is_err() {
tracing::error!(job_id = %job_id, "Worker died before receiving Start message");
}
// Insert while still holding the write lock
jobs.insert(job_id, ScheduledJob { handle, tx });
@@ -418,10 +420,16 @@ impl Scheduler {
// Update job state
self.context_manager
.update_context(job_id, |ctx| {
let _ = ctx.transition_to(
if let Err(e) = ctx.transition_to(
JobState::Cancelled,
Some("Stopped by scheduler".to_string()),
);
) {
tracing::warn!(
job_id = %job_id,
error = %e,
"Failed to transition job to Cancelled state"
);
}
})
.await?;
+8 -6
View File
@@ -66,12 +66,14 @@ pub trait SelfRepair: Send + Sync {
/// Default self-repair implementation.
pub struct DefaultSelfRepair {
context_manager: Arc<ContextManager>,
#[allow(dead_code)] // Will be used for time-based stuck detection
// TODO: use for time-based stuck detection (currently only max_repair_attempts is checked)
#[allow(dead_code)]
stuck_threshold: Duration,
max_repair_attempts: u32,
store: Option<Arc<dyn Database>>,
builder: Option<Arc<dyn SoftwareBuilder>>,
#[allow(dead_code)] // Will be used for tool hot-reload after repair
// TODO: use for tool hot-reload after repair
#[allow(dead_code)]
tools: Option<Arc<ToolRegistry>>,
}
@@ -93,15 +95,15 @@ impl DefaultSelfRepair {
}
/// Add a Store for tool failure tracking.
#[allow(dead_code)] // Public API for configuring repair with persistence
pub fn with_store(mut self, store: Arc<dyn Database>) -> Self {
#[allow(dead_code)] // TODO: wire up in main.rs when persistence is needed
pub(crate) fn with_store(mut self, store: Arc<dyn Database>) -> Self {
self.store = Some(store);
self
}
/// Add a Builder and ToolRegistry for automatic tool repair.
#[allow(dead_code)] // Public API for enabling automatic tool repair
pub fn with_builder(
#[allow(dead_code)] // TODO: wire up in main.rs when auto-repair is needed
pub(crate) fn with_builder(
mut self,
builder: Arc<dyn SoftwareBuilder>,
tools: Arc<ToolRegistry>,
+19 -16
View File
@@ -70,10 +70,9 @@ impl Session {
pub fn create_thread(&mut self) -> &mut Thread {
let thread = Thread::new(self.id);
let thread_id = thread.id;
self.threads.insert(thread_id, thread);
self.active_thread = Some(thread_id);
self.last_active_at = Utc::now();
self.threads.get_mut(&thread_id).expect("just inserted")
self.threads.entry(thread_id).or_insert(thread)
}
/// Get the active thread.
@@ -88,10 +87,19 @@ impl Session {
/// Get or create the active thread.
pub fn get_or_create_thread(&mut self) -> &mut Thread {
if self.active_thread.is_none() {
self.create_thread();
match self.active_thread {
None => self.create_thread(),
Some(id) => {
if self.threads.contains_key(&id) {
// Safe: contains_key confirmed the entry exists.
self.threads.get_mut(&id).unwrap()
} else {
// Stale active_thread ID: create a new thread, which
// updates self.active_thread to the new thread's ID.
self.create_thread()
}
}
}
self.active_thread_mut().expect("just created")
}
/// Switch to a different thread.
@@ -177,10 +185,6 @@ pub struct Thread {
/// Pending auth token request (thread is in auth mode).
#[serde(default)]
pub pending_auth: Option<PendingAuth>,
/// Last NEAR AI response ID for response chaining. Persisted to DB
/// metadata so we can resume chaining across restarts.
#[serde(default)]
pub last_response_id: Option<String>,
}
impl Thread {
@@ -197,7 +201,6 @@ impl Thread {
metadata: serde_json::Value::Null,
pending_approval: None,
pending_auth: None,
last_response_id: None,
}
}
@@ -214,7 +217,6 @@ impl Thread {
metadata: serde_json::Value::Null,
pending_approval: None,
pending_auth: None,
last_response_id: None,
}
}
@@ -240,7 +242,8 @@ impl Thread {
self.turns.push(turn);
self.state = ThreadState::Processing;
self.updated_at = Utc::now();
self.turns.last_mut().expect("just pushed")
// turn_number was len() before push, so it's a valid index after push
&mut self.turns[turn_number]
}
/// Complete the current turn with a response.
@@ -353,8 +356,10 @@ impl Thread {
if let Some(next) = iter.peek()
&& next.role == crate::llm::Role::Assistant
{
let response = iter.next().expect("peeked");
turn.complete(&response.content);
// iter.next() is guaranteed Some after a successful peek()
if let Some(response) = iter.next() {
turn.complete(&response.content);
}
}
self.turns.push(turn);
@@ -852,7 +857,6 @@ mod tests {
thread.start_turn("hello");
thread.complete_turn("world");
thread.last_response_id = Some("resp_abc123".to_string());
let json = serde_json::to_string(&thread).unwrap();
let restored: Thread = serde_json::from_str(&json).unwrap();
@@ -862,7 +866,6 @@ mod tests {
assert_eq!(restored.turns.len(), 1);
assert_eq!(restored.turns[0].user_input, "hello");
assert_eq!(restored.turns[0].response, Some("world".to_string()));
assert_eq!(restored.last_response_id, Some("resp_abc123".to_string()));
}
#[test]
+11
View File
@@ -13,6 +13,9 @@ use crate::agent::session::Session;
use crate::agent::undo::UndoManager;
use crate::hooks::HookRegistry;
/// Warn when session count exceeds this threshold.
const SESSION_COUNT_WARNING_THRESHOLD: usize = 1000;
/// Key for mapping external thread IDs to internal ones.
#[derive(Clone, Hash, Eq, PartialEq)]
struct ThreadKey {
@@ -68,6 +71,14 @@ impl SessionManager {
let session = Arc::new(Mutex::new(new_session));
sessions.insert(user_id.to_string(), Arc::clone(&session));
if sessions.len() >= SESSION_COUNT_WARNING_THRESHOLD && sessions.len() % 100 == 0 {
tracing::warn!(
"High session count: {} active sessions. \
Pruning runs every 10 minutes; consider reducing session_idle_timeout.",
sessions.len()
);
}
// Fire OnSessionStart hook (fire-and-forget)
if let Some(ref hooks) = self.hooks {
let hooks = hooks.clone();
+8
View File
@@ -234,6 +234,7 @@ impl Submission {
}
/// Create an approval submission.
#[cfg(test)]
pub fn approval(request_id: Uuid, approved: bool) -> Self {
Self::ExecApproval {
request_id,
@@ -243,6 +244,7 @@ impl Submission {
}
/// Create an "always approve" submission.
#[cfg(test)]
pub fn always_approve(request_id: Uuid) -> Self {
Self::ExecApproval {
request_id,
@@ -252,26 +254,31 @@ impl Submission {
}
/// Create an interrupt submission.
#[cfg(test)]
pub fn interrupt() -> Self {
Self::Interrupt
}
/// Create a compact submission.
#[cfg(test)]
pub fn compact() -> Self {
Self::Compact
}
/// Create an undo submission.
#[cfg(test)]
pub fn undo() -> Self {
Self::Undo
}
/// Create a redo submission.
#[cfg(test)]
pub fn redo() -> Self {
Self::Redo
}
/// Check if this submission starts a new turn.
#[cfg(test)]
pub fn starts_turn(&self) -> bool {
matches!(self, Self::UserInput { .. })
}
@@ -340,6 +347,7 @@ impl SubmissionResult {
}
/// Create an OK result.
#[cfg(test)]
pub fn ok() -> Self {
Self::Ok { message: None }
}
+7
View File
@@ -29,6 +29,7 @@ impl TaskOutput {
}
/// Create a text result.
#[cfg(test)]
pub fn text(text: impl Into<String>, duration: Duration) -> Self {
Self {
result: serde_json::Value::String(text.into()),
@@ -37,6 +38,7 @@ impl TaskOutput {
}
/// Create an empty success result.
#[cfg(test)]
pub fn empty(duration: Duration) -> Self {
Self {
result: serde_json::Value::Null,
@@ -130,6 +132,7 @@ impl Task {
}
/// Create a new Job task with a specific ID.
#[cfg(test)]
pub fn job_with_id(id: Uuid, title: impl Into<String>, description: impl Into<String>) -> Self {
Self::Job {
id,
@@ -152,6 +155,7 @@ impl Task {
}
/// Create a new Background task.
#[cfg(test)]
pub fn background(handler: std::sync::Arc<dyn TaskHandler>) -> Self {
Self::Background {
id: Uuid::new_v4(),
@@ -160,6 +164,7 @@ impl Task {
}
/// Create a new Background task with a specific ID.
#[cfg(test)]
pub fn background_with_id(id: Uuid, handler: std::sync::Arc<dyn TaskHandler>) -> Self {
Self::Background { id, handler }
}
@@ -174,6 +179,7 @@ impl Task {
}
/// Get the parent ID for sub-tasks.
#[cfg(test)]
pub fn parent_id(&self) -> Option<Uuid> {
match self {
Self::Job { .. } => None,
@@ -225,6 +231,7 @@ impl fmt::Debug for Task {
}
/// Status of a scheduled task.
#[cfg(test)]
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum TaskStatus {
/// Task is queued waiting for execution.
+316 -209
View File
@@ -6,11 +6,14 @@
use std::sync::Arc;
use tokio::sync::Mutex;
use tokio::task::JoinSet;
use uuid::Uuid;
use crate::agent::Agent;
use crate::agent::compaction::ContextCompactor;
use crate::agent::dispatcher::{AgenticLoopResult, detect_auth_awaiting, parse_auth_result};
use crate::agent::dispatcher::{
AgenticLoopResult, check_auth_required, execute_chat_tool_standalone, parse_auth_result,
};
use crate::agent::session::{PendingApproval, Session, ThreadState};
use crate::agent::submission::SubmissionResult;
use crate::channels::{IncomingMessage, StatusUpdate};
@@ -84,20 +87,6 @@ impl Agent {
thread.restore_from_messages(chat_messages);
}
// Restore response chain from conversation metadata
if let Some(store) = self.store()
&& let Ok(Some(metadata)) = store.get_conversation_metadata(thread_uuid).await
&& let Some(rid) = metadata
.get("last_response_id")
.and_then(|v| v.as_str())
.map(String::from)
{
thread.last_response_id = Some(rid.clone());
self.llm()
.seed_response_chain(&thread_uuid.to_string(), rid);
tracing::debug!("Restored response chain for thread {}", thread_uuid);
}
// Insert into session and register with session manager
{
let mut sess = session.lock().await;
@@ -225,7 +214,7 @@ impl Agent {
)
.await;
let compactor = ContextCompactor::new(self.llm().clone());
let compactor = ContextCompactor::new(self.llm().clone(), self.safety().clone());
if let Err(e) = compactor
.compact(thread, strategy, self.workspace().map(|w| w.as_ref()))
.await
@@ -275,7 +264,7 @@ impl Agent {
// Run the agentic tool execution loop
let result = self
.run_agentic_loop(message, session.clone(), thread_id, turn_messages, false)
.run_agentic_loop(message, session.clone(), thread_id, turn_messages)
.await;
// Re-acquire lock and check if interrupted
@@ -322,7 +311,6 @@ impl Agent {
};
thread.complete_turn(&response);
self.persist_response_chain(thread);
let _ = self
.channels
.send_status(
@@ -332,8 +320,10 @@ impl Agent {
)
.await;
// Fire-and-forget: persist turn to DB
self.persist_turn(thread_id, &message.user_id, content, Some(&response));
// Persist turn to DB before returning so the write
// completes even if the process shuts down right after.
self.persist_turn(thread_id, &message.user_id, content, Some(&response))
.await;
Ok(SubmissionResult::response(response))
}
@@ -363,15 +353,16 @@ impl Agent {
thread.fail_turn(e.to_string());
// Persist the user message even on failure
self.persist_turn(thread_id, &message.user_id, content, None);
self.persist_turn(thread_id, &message.user_id, content, None)
.await;
Ok(SubmissionResult::error(e.to_string()))
}
}
}
/// Fire-and-forget: persist a turn (user message + optional assistant response) to the DB.
pub(super) fn persist_turn(
/// Persist a turn (user message + optional assistant response) to the DB.
pub(super) async fn persist_turn(
&self,
thread_id: Uuid,
user_id: &str,
@@ -383,70 +374,29 @@ impl Agent {
None => return,
};
let user_id = user_id.to_string();
let user_input = user_input.to_string();
let response = response.map(String::from);
if let Err(e) = store
.ensure_conversation(thread_id, "gateway", user_id, None)
.await
{
tracing::warn!("Failed to ensure conversation {}: {}", thread_id, e);
return;
}
tokio::spawn(async move {
if let Err(e) = store
.ensure_conversation(thread_id, "gateway", &user_id, None)
if let Err(e) = store
.add_conversation_message(thread_id, "user", user_input)
.await
{
tracing::warn!("Failed to persist user message: {}", e);
return;
}
if let Some(resp) = response
&& let Err(e) = store
.add_conversation_message(thread_id, "assistant", resp)
.await
{
tracing::warn!("Failed to ensure conversation {}: {}", thread_id, e);
return;
}
if let Err(e) = store
.add_conversation_message(thread_id, "user", &user_input)
.await
{
tracing::warn!("Failed to persist user message: {}", e);
return;
}
if let Some(ref resp) = response
&& let Err(e) = store
.add_conversation_message(thread_id, "assistant", resp)
.await
{
tracing::warn!("Failed to persist assistant message: {}", e);
}
});
}
/// Sync the provider's response chain ID to the thread and DB metadata.
///
/// Call after a successful agentic loop to persist the latest
/// `previous_response_id` so chaining survives restarts.
pub(super) fn persist_response_chain(&self, thread: &mut crate::agent::session::Thread) {
let tid = thread.id.to_string();
let response_id = match self.llm().get_response_chain_id(&tid) {
Some(rid) => rid,
None => return,
};
// Update in-memory thread
thread.last_response_id = Some(response_id.clone());
// Fire-and-forget DB write
let store = match self.store() {
Some(s) => Arc::clone(s),
None => return,
};
let thread_id = thread.id;
tokio::spawn(async move {
let val = serde_json::json!(response_id);
if let Err(e) = store
.update_conversation_metadata_field(thread_id, "last_response_id", &val)
.await
{
tracing::warn!(
"Failed to persist response chain for thread {}: {}",
thread_id,
e
);
}
});
{
tracing::warn!("Failed to persist assistant message: {}", e);
}
}
pub(super) async fn process_undo(
@@ -559,7 +509,7 @@ impl Agent {
crate::agent::context_monitor::CompactionStrategy::Summarize { keep_recent: 5 },
);
let compactor = ContextCompactor::new(self.llm().clone());
let compactor = ContextCompactor::new(self.llm().clone(), self.safety().clone());
match compactor
.compact(thread, strategy, self.workspace().map(|w| w.as_ref()))
.await
@@ -608,8 +558,8 @@ impl Agent {
approved: bool,
always: bool,
) -> Result<SubmissionResult, Error> {
// Get thread state and pending approval
let (_thread_state, pending) = {
// Get pending approval for this thread
let pending = {
let mut sess = session.lock().await;
let thread = sess
.threads
@@ -620,8 +570,7 @@ impl Agent {
return Ok(SubmissionResult::error("No pending approval request."));
}
let pending = thread.take_pending_approval();
(thread.state, pending)
thread.take_pending_approval()
};
let pending = match pending {
@@ -734,29 +683,17 @@ impl Agent {
// If tool_auth returned awaiting_token, enter auth mode and
// return instructions directly (skip agentic loop continuation).
if let Some((ext_name, instructions)) =
detect_auth_awaiting(&pending.tool_name, &tool_result)
check_auth_required(&pending.tool_name, &tool_result)
{
let auth_data = parse_auth_result(&tool_result);
{
let mut sess = session.lock().await;
if let Some(thread) = sess.threads.get_mut(&thread_id) {
thread.enter_auth_mode(ext_name.clone());
thread.complete_turn(&instructions);
}
}
let _ = self
.channels
.send_status(
&message.channel,
StatusUpdate::AuthRequired {
extension_name: ext_name,
instructions: Some(instructions.clone()),
auth_url: auth_data.auth_url,
setup_url: auth_data.setup_url,
},
&message.metadata,
)
.await;
self.handle_auth_intercept(
&session,
thread_id,
message,
&tool_result,
ext_name,
instructions.clone(),
)
.await;
return Ok(SubmissionResult::response(instructions));
}
@@ -798,9 +735,17 @@ impl Agent {
.await;
}
let mut deferred_queue = std::collections::VecDeque::from(deferred_tool_calls);
while let Some(tc) = deferred_queue.pop_front() {
// Re-check approval for each deferred tool call
// === Phase 1: Preflight (sequential) ===
// Walk deferred tools checking approval. Collect runnable
// tools; stop at the first that needs approval.
let mut runnable: Vec<crate::llm::ToolCall> = Vec::new();
let mut approval_needed: Option<(
usize,
crate::llm::ToolCall,
Arc<dyn crate::tools::Tool>,
)> = None;
for (idx, tc) in deferred_tool_calls.iter().enumerate() {
if let Some(tool) = self.tools().get(&tc.name).await
&& tool.requires_approval()
{
@@ -814,73 +759,142 @@ impl Agent {
};
if !is_auto_approved {
let new_pending = PendingApproval {
request_id: Uuid::new_v4(),
tool_name: tc.name.clone(),
parameters: tc.arguments.clone(),
description: tool.description().to_string(),
tool_call_id: tc.id.clone(),
context_messages: context_messages.clone(),
deferred_tool_calls: deferred_queue.iter().cloned().collect(),
};
let request_id = new_pending.request_id;
let tool_name = new_pending.tool_name.clone();
let description = new_pending.description.clone();
let parameters = new_pending.parameters.clone();
{
let mut sess = session.lock().await;
if let Some(thread) = sess.threads.get_mut(&thread_id) {
thread.await_approval(new_pending);
}
}
let _ = self
.channels
.send_status(
&message.channel,
StatusUpdate::Status("Awaiting approval".into()),
&message.metadata,
)
.await;
return Ok(SubmissionResult::NeedApproval {
request_id,
tool_name,
description,
parameters,
});
approval_needed = Some((idx, tc.clone(), tool));
break; // remaining tools stay deferred
}
}
let _ = self
.channels
.send_status(
&message.channel,
StatusUpdate::ToolStarted {
name: tc.name.clone(),
},
&message.metadata,
)
.await;
runnable.push(tc.clone());
}
let deferred_result = self
.execute_chat_tool(&tc.name, &tc.arguments, &job_ctx)
.await;
// === Phase 2: Parallel execution ===
let exec_results: Vec<(crate::llm::ToolCall, Result<String, Error>)> = if runnable.len()
<= 1
{
// Single tool (or none): execute inline
let mut results = Vec::new();
for tc in &runnable {
let _ = self
.channels
.send_status(
&message.channel,
StatusUpdate::ToolStarted {
name: tc.name.clone(),
},
&message.metadata,
)
.await;
let _ = self
.channels
.send_status(
&message.channel,
StatusUpdate::ToolCompleted {
name: tc.name.clone(),
success: deferred_result.is_ok(),
},
&message.metadata,
)
.await;
let result = self
.execute_chat_tool(&tc.name, &tc.arguments, &job_ctx)
.await;
let _ = self
.channels
.send_status(
&message.channel,
StatusUpdate::ToolCompleted {
name: tc.name.clone(),
success: result.is_ok(),
},
&message.metadata,
)
.await;
results.push((tc.clone(), result));
}
results
} else {
// Multiple tools: execute in parallel via JoinSet
let mut join_set = JoinSet::new();
let runnable_count = runnable.len();
for (spawn_idx, tc) in runnable.iter().enumerate() {
let tools = self.tools().clone();
let safety = self.safety().clone();
let channels = self.channels.clone();
let job_ctx = job_ctx.clone();
let tc = tc.clone();
let channel = message.channel.clone();
let metadata = message.metadata.clone();
join_set.spawn(async move {
let _ = channels
.send_status(
&channel,
StatusUpdate::ToolStarted {
name: tc.name.clone(),
},
&metadata,
)
.await;
let result = execute_chat_tool_standalone(
&tools,
&safety,
&tc.name,
&tc.arguments,
&job_ctx,
)
.await;
let _ = channels
.send_status(
&channel,
StatusUpdate::ToolCompleted {
name: tc.name.clone(),
success: result.is_ok(),
},
&metadata,
)
.await;
(spawn_idx, tc, result)
});
}
// Collect and reorder by original index
let mut ordered: Vec<Option<(crate::llm::ToolCall, Result<String, Error>)>> =
(0..runnable_count).map(|_| None).collect();
while let Some(join_result) = join_set.join_next().await {
match join_result {
Ok((idx, tc, result)) => {
ordered[idx] = Some((tc, result));
}
Err(e) => {
if e.is_panic() {
tracing::error!("Deferred tool execution task panicked: {}", e);
} else {
tracing::error!("Deferred tool execution task cancelled: {}", e);
}
}
}
}
// Fill panicked slots with error results
ordered
.into_iter()
.enumerate()
.map(|(i, opt)| {
opt.unwrap_or_else(|| {
let tc = runnable[i].clone();
let err: Error = crate::error::ToolError::ExecutionFailed {
name: tc.name.clone(),
reason: "Task failed during execution".to_string(),
}
.into();
(tc, Err(err))
})
})
.collect()
};
// === Phase 3: Post-flight (sequential, in original order) ===
// Process all results before any conditional return so every
// tool result is recorded in the session audit trail.
let mut deferred_auth: Option<String> = None;
for (tc, deferred_result) in exec_results {
if let Ok(ref output) = deferred_result
&& !output.is_empty()
{
@@ -910,32 +924,21 @@ impl Agent {
}
}
// Auth detection for deferred tools
if let Some((ext_name, instructions)) =
detect_auth_awaiting(&tc.name, &deferred_result)
// Auth detection defer return until all results are recorded
if deferred_auth.is_none()
&& let Some((ext_name, instructions)) =
check_auth_required(&tc.name, &deferred_result)
{
let auth_data = parse_auth_result(&deferred_result);
{
let mut sess = session.lock().await;
if let Some(thread) = sess.threads.get_mut(&thread_id) {
thread.enter_auth_mode(ext_name.clone());
thread.complete_turn(&instructions);
}
}
let _ = self
.channels
.send_status(
&message.channel,
StatusUpdate::AuthRequired {
extension_name: ext_name,
instructions: Some(instructions.clone()),
auth_url: auth_data.auth_url,
setup_url: auth_data.setup_url,
},
&message.metadata,
)
.await;
return Ok(SubmissionResult::response(instructions));
self.handle_auth_intercept(
&session,
thread_id,
message,
&deferred_result,
ext_name,
instructions.clone(),
)
.await;
deferred_auth = Some(instructions);
}
let deferred_content = match deferred_result {
@@ -953,9 +956,55 @@ impl Agent {
context_messages.push(ChatMessage::tool_result(&tc.id, &tc.name, deferred_content));
}
// Return auth response after all results are recorded
if let Some(instructions) = deferred_auth {
return Ok(SubmissionResult::response(instructions));
}
// Handle approval if a tool needed it
if let Some((approval_idx, tc, tool)) = approval_needed {
let new_pending = PendingApproval {
request_id: Uuid::new_v4(),
tool_name: tc.name.clone(),
parameters: tc.arguments.clone(),
description: tool.description().to_string(),
tool_call_id: tc.id.clone(),
context_messages: context_messages.clone(),
deferred_tool_calls: deferred_tool_calls[approval_idx + 1..].to_vec(),
};
let request_id = new_pending.request_id;
let tool_name = new_pending.tool_name.clone();
let description = new_pending.description.clone();
let parameters = new_pending.parameters.clone();
{
let mut sess = session.lock().await;
if let Some(thread) = sess.threads.get_mut(&thread_id) {
thread.await_approval(new_pending);
}
}
let _ = self
.channels
.send_status(
&message.channel,
StatusUpdate::Status("Awaiting approval".into()),
&message.metadata,
)
.await;
return Ok(SubmissionResult::NeedApproval {
request_id,
tool_name,
description,
parameters,
});
}
// Continue the agentic loop (a tool was already executed this turn)
let result = self
.run_agentic_loop(message, session.clone(), thread_id, context_messages, true)
.run_agentic_loop(message, session.clone(), thread_id, context_messages)
.await;
// Handle the result
@@ -967,8 +1016,12 @@ impl Agent {
match result {
Ok(AgenticLoopResult::Response(response)) => {
let user_input = thread.last_turn().map(|t| t.user_input.clone());
thread.complete_turn(&response);
self.persist_response_chain(thread);
if let Some(input) = user_input {
self.persist_turn(thread_id, &message.user_id, &input, Some(&response))
.await;
}
let _ = self
.channels
.send_status(
@@ -1003,16 +1056,32 @@ impl Agent {
})
}
Err(e) => {
let user_input = thread.last_turn().map(|t| t.user_input.clone());
thread.fail_turn(e.to_string());
if let Some(input) = user_input {
self.persist_turn(thread_id, &message.user_id, &input, None)
.await;
}
Ok(SubmissionResult::error(e.to_string()))
}
}
} else {
// Rejected - clear approval and return to idle
// Rejected - complete the turn with a rejection message and persist
let rejection = format!(
"Tool '{}' was rejected. The agent will not execute this tool.\n\n\
You can continue the conversation or try a different approach.",
pending.tool_name
);
{
let mut sess = session.lock().await;
if let Some(thread) = sess.threads.get_mut(&thread_id) {
let user_input = thread.last_turn().map(|t| t.user_input.clone());
thread.clear_pending_approval();
thread.complete_turn(&rejection);
if let Some(input) = user_input {
self.persist_turn(thread_id, &message.user_id, &input, Some(&rejection))
.await;
}
}
}
@@ -1025,14 +1094,52 @@ impl Agent {
)
.await;
Ok(SubmissionResult::response(format!(
"Tool '{}' was rejected. The agent will not execute this tool.\n\n\
You can continue the conversation or try a different approach.",
pending.tool_name
)))
Ok(SubmissionResult::response(rejection))
}
}
/// Handle an auth-required result from a tool execution.
///
/// Enters auth mode on the thread, completes + persists the turn,
/// and sends the AuthRequired status to the channel.
/// Returns the instructions string for the caller to wrap in a response.
async fn handle_auth_intercept(
&self,
session: &Arc<Mutex<Session>>,
thread_id: Uuid,
message: &IncomingMessage,
tool_result: &Result<String, Error>,
ext_name: String,
instructions: String,
) {
let auth_data = parse_auth_result(tool_result);
{
let mut sess = session.lock().await;
if let Some(thread) = sess.threads.get_mut(&thread_id) {
let user_input = thread.last_turn().map(|t| t.user_input.clone());
thread.enter_auth_mode(ext_name.clone());
thread.complete_turn(&instructions);
if let Some(input) = user_input {
self.persist_turn(thread_id, &message.user_id, &input, Some(&instructions))
.await;
}
}
}
let _ = self
.channels
.send_status(
&message.channel,
StatusUpdate::AuthRequired {
extension_name: ext_name,
instructions: Some(instructions.clone()),
auth_url: auth_data.auth_url,
setup_url: auth_data.setup_url,
},
&message.metadata,
)
.await;
}
/// Handle an auth token submitted while the thread is in auth mode.
///
/// The token goes directly to the extension manager's credential store,
+4
View File
@@ -67,6 +67,7 @@ impl UndoManager {
}
/// Create with a custom checkpoint limit.
#[cfg(test)]
pub fn with_max_checkpoints(mut self, max: usize) -> Self {
self.max_checkpoints = max;
self
@@ -126,6 +127,7 @@ impl UndoManager {
}
/// Pop the last checkpoint from the undo stack.
#[cfg(test)]
pub fn pop_undo(&mut self) -> Option<Checkpoint> {
self.undo_stack.pop_back()
}
@@ -178,6 +180,7 @@ impl UndoManager {
}
/// Get a checkpoint by ID.
#[cfg(test)]
pub fn get_checkpoint(&self, id: Uuid) -> Option<&Checkpoint> {
self.undo_stack
.iter()
@@ -186,6 +189,7 @@ impl UndoManager {
}
/// List all available checkpoints (for UI display).
#[cfg(test)]
pub fn list_checkpoints(&self) -> Vec<&Checkpoint> {
self.undo_stack.iter().collect()
}
+329 -46
View File
@@ -3,8 +3,8 @@
use std::sync::Arc;
use std::time::Duration;
use futures::future::join_all;
use tokio::sync::mpsc;
use tokio::task::JoinSet;
use uuid::Uuid;
use crate::agent::scheduler::WorkerMessage;
@@ -292,19 +292,21 @@ Report when the job is complete or if you encounter issues you cannot resolve."#
tool_calls.clone(),
));
for tc in tool_calls {
let result = self.execute_tool(&tc.name, &tc.arguments).await;
// Create synthetic selection for process_tool_result
let selection = ToolSelection {
// Convert ToolCalls to ToolSelections and execute in parallel
let selections: Vec<ToolSelection> = tool_calls
.iter()
.map(|tc| ToolSelection {
tool_name: tc.name.clone(),
parameters: tc.arguments.clone(),
reasoning: String::new(),
alternatives: vec![],
tool_call_id: tc.id.clone(),
};
})
.collect();
self.process_tool_result(reason_ctx, &selection, result)
let results = self.execute_tools_parallel(&selections).await;
for (selection, result) in selections.iter().zip(results) {
self.process_tool_result(reason_ctx, selection, result.result)
.await?;
}
}
@@ -347,24 +349,71 @@ Report when the job is complete or if you encounter issues you cannot resolve."#
}
}
/// Execute multiple tools in parallel.
/// Execute multiple tools in parallel using a JoinSet.
///
/// Each task is tagged with its original index so results are returned
/// in the same order as `selections`, regardless of completion order.
async fn execute_tools_parallel(&self, selections: &[ToolSelection]) -> Vec<ToolExecResult> {
let futures: Vec<_> = selections
.iter()
.map(|selection| {
let tool_name = selection.tool_name.clone();
let params = selection.parameters.clone();
let deps = self.deps.clone();
let job_id = self.job_id;
let count = selections.len();
async move {
let result = Self::execute_tool_inner(&deps, job_id, &tool_name, &params).await;
ToolExecResult { result }
// Short-circuit for single tool: execute directly without JoinSet overhead
if count <= 1 {
let mut results = Vec::with_capacity(count);
for selection in selections {
let result = Self::execute_tool_inner(
&self.deps,
self.job_id,
&selection.tool_name,
&selection.parameters,
)
.await;
results.push(ToolExecResult { result });
}
return results;
}
let mut join_set = JoinSet::new();
for (idx, selection) in selections.iter().enumerate() {
let deps = self.deps.clone();
let job_id = self.job_id;
let tool_name = selection.tool_name.clone();
let params = selection.parameters.clone();
join_set.spawn(async move {
let result = Self::execute_tool_inner(&deps, job_id, &tool_name, &params).await;
(idx, ToolExecResult { result })
});
}
// Collect and reorder by original index
let mut results: Vec<Option<ToolExecResult>> = (0..count).map(|_| None).collect();
while let Some(join_result) = join_set.join_next().await {
match join_result {
Ok((idx, exec_result)) => results[idx] = Some(exec_result),
Err(e) => {
if e.is_panic() {
tracing::error!("Tool execution task panicked: {}", e);
} else {
tracing::error!("Tool execution task cancelled: {}", e);
}
}
})
.collect();
}
}
join_all(futures).await
// Fill any panicked slots with error results
results
.into_iter()
.enumerate()
.map(|(i, opt)| {
opt.unwrap_or_else(|| ToolExecResult {
result: Err(crate::error::ToolError::ExecutionFailed {
name: selections[i].tool_name.clone(),
reason: "Task failed during execution".to_string(),
}
.into()),
})
})
.collect()
}
/// Inner tool execution logic that can be called from both single and parallel paths.
@@ -505,7 +554,8 @@ Report when the job is complete or if you encounter issues you cannot resolve."#
let output_str = serde_json::to_string_pretty(&output.result)
.ok()
.map(|s| deps.safety.sanitize_tool_output(tool_name, &s).content);
deps.context_manager
match deps
.context_manager
.update_memory(job_id, |mem| {
let rec = mem.create_action(tool_name, params.clone()).succeed(
output_str.clone(),
@@ -516,30 +566,52 @@ Report when the job is complete or if you encounter issues you cannot resolve."#
rec
})
.await
.ok()
{
Ok(rec) => Some(rec),
Err(e) => {
tracing::warn!(job_id = %job_id, tool = tool_name, "Failed to record action in memory: {e}");
None
}
}
}
Ok(Err(e)) => {
match deps
.context_manager
.update_memory(job_id, |mem| {
let rec = mem
.create_action(tool_name, params.clone())
.fail(e.to_string(), elapsed);
mem.record_action(rec.clone());
rec
})
.await
{
Ok(rec) => Some(rec),
Err(e) => {
tracing::warn!(job_id = %job_id, tool = tool_name, "Failed to record action in memory: {e}");
None
}
}
}
Err(_) => {
match deps
.context_manager
.update_memory(job_id, |mem| {
let rec = mem
.create_action(tool_name, params.clone())
.fail("Execution timeout", elapsed);
mem.record_action(rec.clone());
rec
})
.await
{
Ok(rec) => Some(rec),
Err(e) => {
tracing::warn!(job_id = %job_id, tool = tool_name, "Failed to record action in memory: {e}");
None
}
}
}
Ok(Err(e)) => deps
.context_manager
.update_memory(job_id, |mem| {
let rec = mem
.create_action(tool_name, params.clone())
.fail(e.to_string(), elapsed);
mem.record_action(rec.clone());
rec
})
.await
.ok(),
Err(_) => deps
.context_manager
.update_memory(job_id, |mem| {
let rec = mem
.create_action(tool_name, params.clone())
.fail("Execution timeout", elapsed);
mem.record_action(rec.clone());
rec
})
.await
.ok(),
};
// Persist action to database (fire-and-forget)
@@ -800,6 +872,102 @@ mod tests {
use crate::llm::ToolSelection;
use crate::util::llm_signals_completion;
use super::*;
use crate::config::SafetyConfig;
use crate::context::JobContext;
use crate::llm::{
CompletionRequest, CompletionResponse, LlmProvider, ToolCompletionRequest,
ToolCompletionResponse,
};
use crate::safety::SafetyLayer;
use crate::tools::{Tool, ToolError, ToolOutput};
/// A test tool that sleeps for a configurable duration before returning.
struct SlowTool {
tool_name: String,
delay: Duration,
}
#[async_trait::async_trait]
impl Tool for SlowTool {
fn name(&self) -> &str {
&self.tool_name
}
fn description(&self) -> &str {
"Test tool with configurable delay"
}
fn parameters_schema(&self) -> serde_json::Value {
serde_json::json!({"type": "object", "properties": {}})
}
async fn execute(
&self,
_params: serde_json::Value,
_ctx: &JobContext,
) -> Result<ToolOutput, ToolError> {
let start = std::time::Instant::now();
tokio::time::sleep(self.delay).await;
Ok(ToolOutput::text(
format!("done_{}", self.tool_name),
start.elapsed(),
))
}
fn requires_sanitization(&self) -> bool {
false
}
}
/// Stub LLM provider (never called in these tests).
struct StubLlm;
#[async_trait::async_trait]
impl LlmProvider for StubLlm {
fn model_name(&self) -> &str {
"stub"
}
fn cost_per_token(&self) -> (rust_decimal::Decimal, rust_decimal::Decimal) {
(rust_decimal::Decimal::ZERO, rust_decimal::Decimal::ZERO)
}
async fn complete(
&self,
_req: CompletionRequest,
) -> Result<CompletionResponse, crate::error::LlmError> {
unimplemented!("stub")
}
async fn complete_with_tools(
&self,
_req: ToolCompletionRequest,
) -> Result<ToolCompletionResponse, crate::error::LlmError> {
unimplemented!("stub")
}
}
/// Build a Worker wired to a ToolRegistry containing the given tools.
async fn make_worker(tools: Vec<Arc<dyn Tool>>) -> Worker {
let registry = ToolRegistry::new();
for t in tools {
registry.register(t).await;
}
let cm = Arc::new(crate::context::ContextManager::new(5));
let job_id = cm.create_job("test", "test job").await.unwrap();
let deps = WorkerDeps {
context_manager: cm,
llm: Arc::new(StubLlm),
safety: Arc::new(SafetyLayer::new(&SafetyConfig {
max_output_length: 100_000,
injection_check_enabled: false,
})),
tools: Arc::new(registry),
store: None,
hooks: Arc::new(crate::hooks::HookRegistry::new()),
timeout: Duration::from_secs(30),
use_planning: false,
};
Worker::new(job_id, deps)
}
#[test]
fn test_tool_selection_preserves_call_id() {
let selection = ToolSelection {
@@ -876,4 +1044,119 @@ mod tests {
"The tool returned: TASK_COMPLETE signal"
));
}
#[tokio::test]
async fn test_parallel_speedup() {
// 3 tools each sleeping 200ms should finish in roughly 200ms (parallel),
// not ~600ms (sequential).
let tools: Vec<Arc<dyn Tool>> = (0..3)
.map(|i| {
Arc::new(SlowTool {
tool_name: format!("slow_{}", i),
delay: Duration::from_millis(200),
}) as Arc<dyn Tool>
})
.collect();
let worker = make_worker(tools).await;
let selections: Vec<ToolSelection> = (0..3)
.map(|i| ToolSelection {
tool_name: format!("slow_{}", i),
parameters: serde_json::json!({}),
reasoning: String::new(),
alternatives: vec![],
tool_call_id: format!("call_{}", i),
})
.collect();
let start = std::time::Instant::now();
let results = worker.execute_tools_parallel(&selections).await;
let elapsed = start.elapsed();
assert_eq!(results.len(), 3);
for r in &results {
assert!(r.result.is_ok(), "Tool should succeed");
}
// Parallel should complete well under the sequential 600ms threshold.
assert!(
elapsed < Duration::from_millis(500),
"Parallel execution took {:?}, expected < 500ms",
elapsed
);
}
#[tokio::test]
async fn test_result_ordering_preserved() {
// Tools with different delays finish in different order.
// Results must be returned in the original request order.
let tools: Vec<Arc<dyn Tool>> = vec![
Arc::new(SlowTool {
tool_name: "tool_a".into(),
delay: Duration::from_millis(300),
}),
Arc::new(SlowTool {
tool_name: "tool_b".into(),
delay: Duration::from_millis(100),
}),
Arc::new(SlowTool {
tool_name: "tool_c".into(),
delay: Duration::from_millis(200),
}),
];
let worker = make_worker(tools).await;
let selections = vec![
ToolSelection {
tool_name: "tool_a".into(),
parameters: serde_json::json!({}),
reasoning: String::new(),
alternatives: vec![],
tool_call_id: "call_a".into(),
},
ToolSelection {
tool_name: "tool_b".into(),
parameters: serde_json::json!({}),
reasoning: String::new(),
alternatives: vec![],
tool_call_id: "call_b".into(),
},
ToolSelection {
tool_name: "tool_c".into(),
parameters: serde_json::json!({}),
reasoning: String::new(),
alternatives: vec![],
tool_call_id: "call_c".into(),
},
];
let results = worker.execute_tools_parallel(&selections).await;
// Results must be in same order as selections, not completion order.
assert!(results[0].result.as_ref().unwrap().contains("done_tool_a"));
assert!(results[1].result.as_ref().unwrap().contains("done_tool_b"));
assert!(results[2].result.as_ref().unwrap().contains("done_tool_c"));
}
#[tokio::test]
async fn test_missing_tool_produces_error_not_panic() {
// If a tool doesn't exist, the result slot should contain an error.
let worker = make_worker(vec![]).await;
let selections = vec![ToolSelection {
tool_name: "nonexistent_tool".into(),
parameters: serde_json::json!({}),
reasoning: String::new(),
alternatives: vec![],
tool_call_id: "call_x".into(),
}];
let results = worker.execute_tools_parallel(&selections).await;
assert_eq!(results.len(), 1);
assert!(
results[0].result.is_err(),
"Missing tool should produce an error, not a panic"
);
}
}
+5 -9
View File
@@ -396,7 +396,6 @@ impl AppBuilder {
let tools = Arc::new(ToolRegistry::new());
tools.register_builtin_tools();
tracing::info!("Registered {} built-in tools", tools.count());
// Create embeddings provider if configured
let embeddings: Option<Arc<dyn EmbeddingProvider>> = if self.config.embeddings.enabled {
@@ -681,12 +680,12 @@ impl AppBuilder {
None
};
// Register dev tools if local tools are enabled
if self.config.agent.allow_local_tools {
// register_builder_tool() already calls register_dev_tools() internally,
// so only register them here when the builder didn't already do it.
let builder_registered_dev_tools = self.config.builder.enabled
&& (self.config.agent.allow_local_tools || !self.config.sandbox.enabled);
if self.config.agent.allow_local_tools && !builder_registered_dev_tools {
tools.register_dev_tools();
tracing::info!(
"Local tools enabled (allow_local_tools=true), dev tools registered directly"
);
}
Ok((mcp_session_manager, wasm_tool_runtime, extension_manager))
@@ -709,9 +708,6 @@ impl AppBuilder {
// Seed workspace and backfill embeddings
if let Some(ref ws) = workspace {
match ws.seed_if_empty().await {
Ok(count) if count > 0 => {
tracing::info!("Workspace seeded with {} core files", count);
}
Ok(_) => {}
Err(e) => {
tracing::warn!("Failed to seed workspace: {}", e);
+61 -1
View File
@@ -103,7 +103,67 @@ pub fn save_bootstrap_env(vars: &[(&str, &str)]) -> std::io::Result<()> {
let escaped = value.replace('\\', "\\\\").replace('"', "\\\"");
content.push_str(&format!("{}=\"{}\"\n", key, escaped));
}
std::fs::write(&path, content)
std::fs::write(&path, &content)?;
restrict_file_permissions(&path)?;
Ok(())
}
/// Update or add a single variable in `~/.ironclaw/.env`, preserving existing content.
///
/// Unlike `save_bootstrap_env` (which overwrites the entire file), this
/// reads the current `.env`, replaces the line for `key` if it exists,
/// or appends it otherwise. Use this when writing a single bootstrap var
/// outside the wizard (which manages the full set via `save_bootstrap_env`).
pub fn upsert_bootstrap_var(key: &str, value: &str) -> std::io::Result<()> {
let path = ironclaw_env_path();
if let Some(parent) = path.parent() {
std::fs::create_dir_all(parent)?;
}
let escaped = value.replace('\\', "\\\\").replace('"', "\\\"");
let new_line = format!("{}=\"{}\"", key, escaped);
let prefix = format!("{}=", key);
let existing = std::fs::read_to_string(&path).unwrap_or_default();
let mut found = false;
let mut result = String::new();
for line in existing.lines() {
if line.starts_with(&prefix) {
if !found {
result.push_str(&new_line);
result.push('\n');
found = true;
}
// Skip duplicate lines for this key
continue;
}
result.push_str(line);
result.push('\n');
}
if !found {
result.push_str(&new_line);
result.push('\n');
}
std::fs::write(&path, result)?;
restrict_file_permissions(&path)?;
Ok(())
}
/// Set restrictive file permissions (0o600) on Unix systems.
///
/// The `.env` file may contain database credentials and API keys,
/// so it should only be readable by the owner.
fn restrict_file_permissions(_path: &std::path::Path) -> std::io::Result<()> {
#[cfg(unix)]
{
use std::os::unix::fs::PermissionsExt;
let perms = std::fs::Permissions::from_mode(0o600);
std::fs::set_permissions(_path, perms)?;
}
Ok(())
}
/// Write `DATABASE_URL` to `~/.ironclaw/.env`.
+97
View File
@@ -300,6 +300,51 @@ impl ChannelHostState {
}
}
/// In-memory workspace store for WASM channels.
///
/// Persists workspace writes across callback invocations within a single
/// channel lifetime. This allows WASM channels to maintain state (e.g.,
/// Telegram polling offsets) between poll ticks without requiring a
/// full database-backed workspace.
///
/// Uses `std::sync::RwLock` (not tokio) because WASM execution runs
/// inside `spawn_blocking`.
pub struct ChannelWorkspaceStore {
data: std::sync::RwLock<std::collections::HashMap<String, String>>,
}
impl ChannelWorkspaceStore {
/// Create a new empty workspace store.
pub fn new() -> Self {
Self {
data: std::sync::RwLock::new(std::collections::HashMap::new()),
}
}
/// Commit pending writes from a callback execution into the store.
pub fn commit_writes(&self, writes: &[PendingWorkspaceWrite]) {
if writes.is_empty() {
return;
}
if let Ok(mut data) = self.data.write() {
for write in writes {
tracing::debug!(
path = %write.path,
content_len = write.content.len(),
"Committing workspace write to channel store"
);
data.insert(write.path.clone(), write.content.clone());
}
}
}
}
impl crate::tools::wasm::WorkspaceReader for ChannelWorkspaceStore {
fn read(&self, path: &str) -> Option<String> {
self.data.read().ok()?.get(path).cloned()
}
}
/// Rate limiter for channel message emission.
///
/// Tracks emission rates across multiple executions.
@@ -497,4 +542,56 @@ mod tests {
assert_eq!(state.channel_name(), "telegram");
}
#[test]
fn test_channel_workspace_store_commit_and_read() {
use crate::channels::wasm::host::{ChannelWorkspaceStore, PendingWorkspaceWrite};
use crate::tools::wasm::WorkspaceReader;
let store = ChannelWorkspaceStore::new();
// Initially empty
assert!(store.read("channels/telegram/offset").is_none());
// Commit some writes
let writes = vec![
PendingWorkspaceWrite {
path: "channels/telegram/offset".to_string(),
content: "103".to_string(),
},
PendingWorkspaceWrite {
path: "channels/telegram/state.json".to_string(),
content: r#"{"ok":true}"#.to_string(),
},
];
store.commit_writes(&writes);
// Should be readable
assert_eq!(
store.read("channels/telegram/offset"),
Some("103".to_string())
);
assert_eq!(
store.read("channels/telegram/state.json"),
Some(r#"{"ok":true}"#.to_string())
);
// Overwrite a value
let writes2 = vec![PendingWorkspaceWrite {
path: "channels/telegram/offset".to_string(),
content: "200".to_string(),
}];
store.commit_writes(&writes2);
assert_eq!(
store.read("channels/telegram/offset"),
Some("200".to_string())
);
// Empty writes are a no-op
store.commit_writes(&[]);
assert_eq!(
store.read("channels/telegram/offset"),
Some("200".to_string())
);
}
}
+70 -10
View File
@@ -42,7 +42,9 @@ use wasmtime_wasi::{ResourceTable, WasiCtx, WasiCtxBuilder, WasiView};
use crate::channels::wasm::capabilities::ChannelCapabilities;
use crate::channels::wasm::error::WasmChannelError;
use crate::channels::wasm::host::{ChannelEmitRateLimiter, ChannelHostState, EmittedMessage};
use crate::channels::wasm::host::{
ChannelEmitRateLimiter, ChannelHostState, ChannelWorkspaceStore, EmittedMessage,
};
use crate::channels::wasm::router::RegisteredEndpoint;
use crate::channels::wasm::runtime::{PreparedChannelModule, WasmChannelRuntime};
use crate::channels::wasm::schema::ChannelConfig;
@@ -547,6 +549,10 @@ pub struct WasmChannel {
/// Pairing store for DM pairing (guest access control).
pairing_store: Arc<PairingStore>,
/// In-memory workspace store persisting writes across callback invocations.
/// Ensures WASM channels can maintain state (e.g., polling offsets) between ticks.
workspace_store: Arc<ChannelWorkspaceStore>,
}
impl WasmChannel {
@@ -577,6 +583,7 @@ impl WasmChannel {
credentials: Arc::new(RwLock::new(HashMap::new())),
typing_task: RwLock::new(None),
pairing_store,
workspace_store: Arc::new(ChannelWorkspaceStore::new()),
}
}
@@ -634,6 +641,26 @@ impl WasmChannel {
self.endpoints.read().await.clone()
}
/// Inject the workspace store as the reader into a capabilities clone.
///
/// Ensures `workspace_read` capability is present with the store as its reader,
/// so WASM callbacks can read previously written workspace state.
fn inject_workspace_reader(
capabilities: &ChannelCapabilities,
store: &Arc<ChannelWorkspaceStore>,
) -> ChannelCapabilities {
let mut caps = capabilities.clone();
let ws_cap = caps
.tool_capabilities
.workspace_read
.get_or_insert_with(|| crate::tools::wasm::WorkspaceCapability {
allowed_prefixes: Vec::new(),
reader: None,
});
ws_cap.reader = Some(Arc::clone(store) as Arc<dyn crate::tools::wasm::WorkspaceReader>);
caps
}
/// Add channel host functions to the linker using generated bindings.
///
/// Uses the wasmtime::component::bindgen! generated `add_to_linker` function
@@ -765,12 +792,13 @@ impl WasmChannel {
let runtime = Arc::clone(&self.runtime);
let prepared = Arc::clone(&self.prepared);
let capabilities = self.capabilities.clone();
let capabilities = Self::inject_workspace_reader(&self.capabilities, &self.workspace_store);
let config_json = self.config_json.read().await.clone();
let timeout = self.runtime.config().callback_timeout;
let channel_name = self.name.clone();
let credentials = self.get_credentials().await;
let pairing_store = self.pairing_store.clone();
let workspace_store = self.workspace_store.clone();
// Execute in blocking task with timeout
let result = tokio::time::timeout(timeout, async move {
@@ -801,8 +829,13 @@ impl WasmChannel {
}
};
let host_state =
let mut host_state =
Self::extract_host_state(&mut store, &prepared.name, &capabilities);
// Commit pending workspace writes to the persistent store
let pending_writes = host_state.take_pending_writes();
workspace_store.commit_writes(&pending_writes);
Ok((config, host_state))
})
.await
@@ -897,10 +930,11 @@ impl WasmChannel {
let runtime = Arc::clone(&self.runtime);
let prepared = Arc::clone(&self.prepared);
let capabilities = self.capabilities.clone();
let capabilities = Self::inject_workspace_reader(&self.capabilities, &self.workspace_store);
let timeout = self.runtime.config().callback_timeout;
let credentials = self.get_credentials().await;
let pairing_store = self.pairing_store.clone();
let workspace_store = self.workspace_store.clone();
// Prepare request data
let method = method.to_string();
@@ -940,8 +974,13 @@ impl WasmChannel {
.map_err(|e| Self::map_wasm_error(e, &prepared.name, prepared.limits.fuel))?;
let response = convert_http_response(wit_response);
let host_state =
let mut host_state =
Self::extract_host_state(&mut store, &prepared.name, &capabilities);
// Commit pending workspace writes to the persistent store
let pending_writes = host_state.take_pending_writes();
workspace_store.commit_writes(&pending_writes);
Ok((response, host_state))
})
.await
@@ -989,11 +1028,12 @@ impl WasmChannel {
let runtime = Arc::clone(&self.runtime);
let prepared = Arc::clone(&self.prepared);
let capabilities = self.capabilities.clone();
let capabilities = Self::inject_workspace_reader(&self.capabilities, &self.workspace_store);
let timeout = self.runtime.config().callback_timeout;
let channel_name = self.name.clone();
let credentials = self.get_credentials().await;
let pairing_store = self.pairing_store.clone();
let workspace_store = self.workspace_store.clone();
// Execute in blocking task with timeout
let result = tokio::time::timeout(timeout, async move {
@@ -1013,8 +1053,13 @@ impl WasmChannel {
.call_on_poll(&mut store)
.map_err(|e| Self::map_wasm_error(e, &prepared.name, prepared.limits.fuel))?;
let host_state =
let mut host_state =
Self::extract_host_state(&mut store, &prepared.name, &capabilities);
// Commit pending workspace writes to the persistent store
let pending_writes = host_state.take_pending_writes();
workspace_store.commit_writes(&pending_writes);
Ok(((), host_state))
})
.await
@@ -1501,6 +1546,7 @@ impl WasmChannel {
let credentials = self.credentials.clone();
let pairing_store = self.pairing_store.clone();
let callback_timeout = self.runtime.config().callback_timeout;
let workspace_store = self.workspace_store.clone();
tokio::spawn(async move {
let mut interval_timer = tokio::time::interval(interval);
@@ -1523,6 +1569,7 @@ impl WasmChannel {
&credentials,
pairing_store.clone(),
callback_timeout,
&workspace_store,
).await;
match result {
@@ -1565,7 +1612,10 @@ impl WasmChannel {
/// Execute a single poll callback with a fresh WASM instance.
///
/// Returns any emitted messages from the callback.
/// Returns any emitted messages from the callback. Pending workspace writes
/// are committed to the shared `ChannelWorkspaceStore` so state persists
/// across poll ticks (e.g., Telegram polling offset).
#[allow(clippy::too_many_arguments)]
async fn execute_poll(
channel_name: &str,
runtime: &Arc<WasmChannelRuntime>,
@@ -1574,6 +1624,7 @@ impl WasmChannel {
credentials: &RwLock<HashMap<String, String>>,
pairing_store: Arc<PairingStore>,
timeout: Duration,
workspace_store: &Arc<ChannelWorkspaceStore>,
) -> Result<Vec<EmittedMessage>, WasmChannelError> {
// Skip if no WASM bytes (testing mode)
if prepared.component_bytes.is_empty() {
@@ -1586,9 +1637,10 @@ impl WasmChannel {
let runtime = Arc::clone(runtime);
let prepared = Arc::clone(prepared);
let capabilities = capabilities.clone();
let capabilities = Self::inject_workspace_reader(capabilities, workspace_store);
let credentials_snapshot = credentials.read().await.clone();
let channel_name_owned = channel_name.to_string();
let workspace_store = Arc::clone(workspace_store);
// Execute in blocking task with timeout
let result = tokio::time::timeout(timeout, async move {
@@ -1608,8 +1660,13 @@ impl WasmChannel {
.call_on_poll(&mut store)
.map_err(|e| Self::map_wasm_error(e, &prepared.name, prepared.limits.fuel))?;
let host_state =
let mut host_state =
Self::extract_host_state(&mut store, &prepared.name, &capabilities);
// Commit pending workspace writes to the persistent store
let pending_writes = host_state.take_pending_writes();
workspace_store.commit_writes(&pending_writes);
Ok(host_state)
})
.await
@@ -2230,6 +2287,8 @@ mod tests {
let credentials = Arc::new(tokio::sync::RwLock::new(std::collections::HashMap::new()));
let timeout = std::time::Duration::from_secs(5);
let workspace_store = Arc::new(crate::channels::wasm::host::ChannelWorkspaceStore::new());
let result = WasmChannel::execute_poll(
"poll-test",
&runtime,
@@ -2238,6 +2297,7 @@ mod tests {
&credentials,
Arc::new(PairingStore::new()),
timeout,
&workspace_store,
)
.await;
+112 -1
View File
@@ -22,7 +22,9 @@ use std::sync::{Arc, Mutex};
use serde::Serialize;
use tokio::sync::broadcast;
use tracing::field::{Field, Visit};
use tracing_subscriber::Layer;
use tracing_subscriber::layer::SubscriberExt;
use tracing_subscriber::util::SubscriberInitExt;
use tracing_subscriber::{EnvFilter, Layer, reload};
use crate::safety::LeakDetector;
@@ -102,6 +104,115 @@ impl Default for LogBroadcaster {
}
}
/// Handle for changing the tracing `EnvFilter` at runtime.
///
/// Wraps a `reload::Handle` so the gateway can switch between log levels
/// (e.g. `ironclaw=debug`) without restarting the process.
pub struct LogLevelHandle {
handle: reload::Handle<EnvFilter, tracing_subscriber::Registry>,
current_level: Mutex<String>,
base_filter: String,
}
impl LogLevelHandle {
pub fn new(
handle: reload::Handle<EnvFilter, tracing_subscriber::Registry>,
initial_level: String,
base_filter: String,
) -> Self {
Self {
handle,
current_level: Mutex::new(initial_level),
base_filter,
}
}
/// Change the `ironclaw=<level>` directive at runtime.
///
/// `level` must be one of: trace, debug, info, warn, error.
pub fn set_level(&self, level: &str) -> Result<(), String> {
const VALID: &[&str] = &["trace", "debug", "info", "warn", "error"];
let level = level.to_lowercase();
if !VALID.contains(&level.as_str()) {
return Err(format!(
"invalid level '{}', must be one of: {}",
level,
VALID.join(", ")
));
}
let filter_str = if self.base_filter.is_empty() {
format!("ironclaw={}", level)
} else {
format!("ironclaw={},{}", level, self.base_filter)
};
let new_filter = EnvFilter::new(&filter_str);
self.handle
.reload(new_filter)
.map_err(|e| format!("failed to reload filter: {}", e))?;
if let Ok(mut current) = self.current_level.lock() {
*current = level;
}
Ok(())
}
/// Returns the current ironclaw log level (e.g. "info", "debug").
pub fn current_level(&self) -> String {
self.current_level
.lock()
.map(|l| l.clone())
.unwrap_or_else(|_| "info".to_string())
}
}
/// Initialise the tracing subscriber with a reloadable `EnvFilter`.
///
/// Returns the `LogLevelHandle` so callers can swap the filter at runtime.
/// The fmt layer and `WebLogLayer` are attached alongside the reloadable filter.
pub fn init_tracing(log_broadcaster: Arc<LogBroadcaster>) -> Arc<LogLevelHandle> {
let raw_filter =
std::env::var("RUST_LOG").unwrap_or_else(|_| "ironclaw=info,tower_http=warn".to_string());
// Split into the ironclaw directive and "everything else" (base_filter).
let mut ironclaw_level = String::from("info");
let mut base_parts: Vec<&str> = Vec::new();
for part in raw_filter.split(',') {
let trimmed = part.trim();
if trimmed.starts_with("ironclaw=") {
if let Some(lvl) = trimmed.strip_prefix("ironclaw=") {
ironclaw_level = lvl.to_string();
}
} else if !trimmed.is_empty() {
base_parts.push(trimmed);
}
}
let base_filter = base_parts.join(",");
let env_filter = EnvFilter::new(&raw_filter);
let (reload_layer, reload_handle) = reload::Layer::new(env_filter);
let handle = Arc::new(LogLevelHandle::new(
reload_handle,
ironclaw_level,
base_filter,
));
tracing_subscriber::registry()
.with(reload_layer)
.with(
tracing_subscriber::fmt::layer()
.with_target(false)
.with_writer(crate::tracing_fmt::TruncatingStderr::default()),
)
.with(WebLogLayer::new(log_broadcaster))
.init();
handle
}
/// Visitor that extracts the `message` field and all extra key-value
/// fields from a tracing event.
///
+9 -1
View File
@@ -41,7 +41,7 @@ use crate::skills::registry::SkillRegistry;
use crate::tools::ToolRegistry;
use crate::workspace::Workspace;
use self::log_layer::LogBroadcaster;
use self::log_layer::{LogBroadcaster, LogLevelHandle};
use self::server::GatewayState;
use self::sse::SseManager;
@@ -76,6 +76,7 @@ impl GatewayChannel {
workspace: None,
session_manager: None,
log_broadcaster: None,
log_level_handle: None,
extension_manager: None,
tool_registry: None,
store: None,
@@ -105,6 +106,7 @@ impl GatewayChannel {
workspace: self.state.workspace.clone(),
session_manager: self.state.session_manager.clone(),
log_broadcaster: self.state.log_broadcaster.clone(),
log_level_handle: self.state.log_level_handle.clone(),
extension_manager: self.state.extension_manager.clone(),
tool_registry: self.state.tool_registry.clone(),
store: self.state.store.clone(),
@@ -140,6 +142,12 @@ impl GatewayChannel {
self
}
/// Inject the log level handle for runtime log level control.
pub fn with_log_level_handle(mut self, h: Arc<LogLevelHandle>) -> Self {
self.rebuild_state(|s| s.log_level_handle = Some(h));
self
}
/// Inject the extension manager for the extensions API.
pub fn with_extension_manager(mut self, em: Arc<ExtensionManager>) -> Self {
self.rebuild_state(|s| s.extension_manager = Some(em));
+39
View File
@@ -122,6 +122,8 @@ pub struct GatewayState {
pub session_manager: Option<Arc<SessionManager>>,
/// Log broadcaster for the logs SSE endpoint.
pub log_broadcaster: Option<Arc<LogBroadcaster>>,
/// Handle for changing the tracing log level at runtime.
pub log_level_handle: Option<Arc<crate::channels::web::log_layer::LogLevelHandle>>,
/// Extension manager for extension management API.
pub extension_manager: Option<Arc<ExtensionManager>>,
/// Tool registry for listing registered tools.
@@ -204,6 +206,11 @@ pub async fn start_server(
.route("/api/jobs/{id}/files/read", get(job_files_read_handler))
// Logs
.route("/api/logs/events", get(logs_events_handler))
.route("/api/logs/level", get(logs_level_get_handler))
.route(
"/api/logs/level",
axum::routing::put(logs_level_set_handler),
)
// Extensions
.route("/api/extensions", get(extensions_list_handler))
.route("/api/extensions/tools", get(extensions_tools_handler))
@@ -1620,6 +1627,38 @@ async fn logs_events_handler(
))
}
async fn logs_level_get_handler(
State(state): State<Arc<GatewayState>>,
) -> Result<Json<serde_json::Value>, (StatusCode, String)> {
let handle = state.log_level_handle.as_ref().ok_or((
StatusCode::SERVICE_UNAVAILABLE,
"Log level control not available".to_string(),
))?;
Ok(Json(serde_json::json!({ "level": handle.current_level() })))
}
async fn logs_level_set_handler(
State(state): State<Arc<GatewayState>>,
Json(body): Json<serde_json::Value>,
) -> Result<Json<serde_json::Value>, (StatusCode, String)> {
let handle = state.log_level_handle.as_ref().ok_or((
StatusCode::SERVICE_UNAVAILABLE,
"Log level control not available".to_string(),
))?;
let level = body
.get("level")
.and_then(|v| v.as_str())
.ok_or((StatusCode::BAD_REQUEST, "missing 'level' field".to_string()))?;
handle
.set_level(level)
.map_err(|e| (StatusCode::BAD_REQUEST, e))?;
tracing::info!("Log level changed to '{}'", handle.current_level());
Ok(Json(serde_json::json!({ "level": handle.current_level() })))
}
// --- Extension handlers ---
async fn extensions_list_handler(
+33 -1
View File
@@ -29,9 +29,11 @@ function authenticate() {
sessionStorage.setItem('ironclaw_token', token);
document.getElementById('auth-screen').style.display = 'none';
document.getElementById('app').style.display = 'flex';
// Strip token from URL so it's not visible in the address bar
// Strip token and log_level from URL so they're not visible in the address bar
const cleaned = new URL(window.location);
const urlLogLevel = cleaned.searchParams.get('log_level');
cleaned.searchParams.delete('token');
cleaned.searchParams.delete('log_level');
window.history.replaceState({}, '', cleaned.pathname + cleaned.search);
connectSSE();
connectLogSSE();
@@ -39,6 +41,12 @@ function authenticate() {
loadThreads();
loadMemoryTree();
loadJobs();
// Apply URL log_level param if present, otherwise just sync the dropdown
if (urlLogLevel) {
setServerLogLevel(urlLogLevel);
} else {
loadServerLogLevel();
}
})
.catch(() => {
sessionStorage.removeItem('ironclaw_token');
@@ -1167,6 +1175,30 @@ function applyLogFilters() {
}
}
// --- Server-side log level control ---
function setServerLogLevel(level) {
apiFetch('/api/logs/level', {
method: 'PUT',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ level: level }),
})
.then(r => r.json())
.then(data => {
document.getElementById('logs-server-level').value = data.level;
})
.catch(err => console.error('Failed to set server log level:', err));
}
function loadServerLogLevel() {
apiFetch('/api/logs/level')
.then(r => r.json())
.then(data => {
document.getElementById('logs-server-level').value = data.level;
})
.catch(() => {}); // ignore if not available
}
// --- Extensions ---
function loadExtensions() {
+6
View File
@@ -127,6 +127,12 @@
<div class="tab-panel" id="tab-logs">
<div class="logs-container">
<div class="logs-toolbar">
<select id="logs-server-level" onchange="setServerLogLevel(this.value)" title="Server-side log level (changes what the server emits)">
<option value="error">Server: ERROR</option>
<option value="warn">Server: WARN</option>
<option value="info" selected>Server: INFO</option>
<option value="debug">Server: DEBUG</option>
</select>
<select id="logs-level-filter">
<option value="all">All Levels</option>
<option value="ERROR">Error</option>
+1
View File
@@ -477,6 +477,7 @@ mod tests {
workspace: None,
session_manager: None,
log_broadcaster: None,
log_level_handle: None,
extension_manager: None,
tool_registry: None,
store: None,
+6
View File
@@ -17,6 +17,7 @@ mod mcp;
pub mod memory;
pub mod oauth_defaults;
mod pairing;
mod registry;
mod service;
pub mod status;
mod tool;
@@ -29,6 +30,7 @@ pub use memory::MemoryCommand;
pub use memory::run_memory_command;
pub use memory::run_memory_command_with_db;
pub use pairing::{PairingCommand, run_pairing_command, run_pairing_command_with_store};
pub use registry::{RegistryCommand, run_registry_command};
pub use service::{ServiceCommand, run_service_command};
pub use status::run_status_command;
pub use tool::{ToolCommand, run_tool_command};
@@ -90,6 +92,10 @@ pub enum Command {
#[command(subcommand)]
Tool(ToolCommand),
/// Browse and install extensions from the registry
#[command(subcommand)]
Registry(RegistryCommand),
/// Manage MCP servers (hosted tool providers)
#[command(subcommand)]
Mcp(McpCommand),
+60 -1
View File
@@ -62,6 +62,18 @@ pub fn builtin_credentials(secret_name: &str) -> Option<OAuthCredentials> {
/// `http://localhost:9876/callback` (or `/auth/callback` for NEAR AI).
pub const OAUTH_CALLBACK_PORT: u16 = 9876;
/// Returns the OAuth callback base URL.
///
/// Checks `IRONCLAW_OAUTH_CALLBACK_URL` env var first (useful for remote/VPS
/// deployments where `127.0.0.1` is unreachable from the user's browser),
/// then falls back to `http://127.0.0.1:{OAUTH_CALLBACK_PORT}`.
pub fn callback_url() -> String {
std::env::var("IRONCLAW_OAUTH_CALLBACK_URL")
.ok()
.filter(|v| !v.is_empty())
.unwrap_or_else(|| format!("http://127.0.0.1:{}", OAUTH_CALLBACK_PORT))
}
/// Error from the OAuth callback listener.
#[derive(Debug, thiserror::Error)]
pub enum OAuthCallbackError {
@@ -297,7 +309,54 @@ pub fn landing_html(provider_name: &str, success: bool) -> String {
#[cfg(test)]
mod tests {
use crate::cli::oauth_defaults::{builtin_credentials, landing_html};
use std::sync::Mutex;
use crate::cli::oauth_defaults::{builtin_credentials, callback_url, landing_html};
/// Serializes env-mutating tests to prevent parallel races.
static ENV_MUTEX: Mutex<()> = Mutex::new(());
#[test]
fn test_callback_url_default() {
let _guard = ENV_MUTEX.lock().expect("env mutex poisoned");
// Clear the env var to test default behavior
let original = std::env::var("IRONCLAW_OAUTH_CALLBACK_URL").ok();
// SAFETY: Under ENV_MUTEX, no concurrent env access.
unsafe {
std::env::remove_var("IRONCLAW_OAUTH_CALLBACK_URL");
}
let url = callback_url();
assert_eq!(url, "http://127.0.0.1:9876");
// Restore
unsafe {
if let Some(val) = original {
std::env::set_var("IRONCLAW_OAUTH_CALLBACK_URL", val);
}
}
}
#[test]
fn test_callback_url_env_override() {
let _guard = ENV_MUTEX.lock().expect("env mutex poisoned");
let original = std::env::var("IRONCLAW_OAUTH_CALLBACK_URL").ok();
// SAFETY: Under ENV_MUTEX, no concurrent env access.
unsafe {
std::env::set_var(
"IRONCLAW_OAUTH_CALLBACK_URL",
"https://myserver.example.com:9876",
);
}
let url = callback_url();
assert_eq!(url, "https://myserver.example.com:9876");
// Restore
unsafe {
if let Some(val) = original {
std::env::set_var("IRONCLAW_OAUTH_CALLBACK_URL", val);
} else {
std::env::remove_var("IRONCLAW_OAUTH_CALLBACK_URL");
}
}
}
#[test]
fn test_unknown_provider_returns_none() {
+339
View File
@@ -0,0 +1,339 @@
//! Registry CLI commands for discovering and installing extensions.
use std::path::PathBuf;
use clap::Subcommand;
use crate::registry::catalog::RegistryCatalog;
use crate::registry::installer::RegistryInstaller;
use crate::registry::manifest::ManifestKind;
#[derive(Subcommand, Debug, Clone)]
pub enum RegistryCommand {
/// List available extensions in the registry
List {
/// Filter by kind: "tool" or "channel"
#[arg(short, long)]
kind: Option<String>,
/// Filter by tag (e.g. "default", "google", "messaging")
#[arg(short, long)]
tag: Option<String>,
/// Show detailed information
#[arg(short, long)]
verbose: bool,
},
/// Show detailed information about an extension or bundle
Info {
/// Extension or bundle name (e.g. "slack", "google", "tools/gmail")
name: String,
},
/// Install an extension or bundle from the registry
Install {
/// Extension or bundle name (e.g. "slack", "google", "default")
name: String,
/// Force overwrite if already installed
#[arg(short, long)]
force: bool,
/// Build from source instead of downloading pre-built artifact
#[arg(long)]
build: bool,
},
/// Install the default bundle of recommended extensions
InstallDefaults {
/// Force overwrite if already installed
#[arg(short, long)]
force: bool,
/// Build from source instead of downloading pre-built artifact
#[arg(long)]
build: bool,
},
}
/// Run a registry command.
pub async fn run_registry_command(cmd: RegistryCommand) -> anyhow::Result<()> {
let registry_dir = find_registry_dir()?;
let catalog = RegistryCatalog::load(&registry_dir)?;
match cmd {
RegistryCommand::List { kind, tag, verbose } => {
cmd_list(&catalog, kind.as_deref(), tag.as_deref(), verbose)
}
RegistryCommand::Info { name } => cmd_info(&catalog, &name),
RegistryCommand::Install { name, force, build } => {
cmd_install(&catalog, &registry_dir, &name, force, build).await
}
RegistryCommand::InstallDefaults { force, build } => {
cmd_install(&catalog, &registry_dir, "default", force, build).await
}
}
}
/// Find the registry directory by looking relative to the current executable or cwd.
fn find_registry_dir() -> anyhow::Result<PathBuf> {
// Try relative to current directory (for dev usage)
let cwd = std::env::current_dir()?;
let candidate = cwd.join("registry");
if candidate.is_dir() {
return Ok(candidate);
}
// Try relative to executable (covers installed binary, target/debug/, target/release/)
if let Ok(exe) = std::env::current_exe()
&& let Some(parent) = exe.parent()
{
// Walk up to 3 levels: exe dir, parent (target/release → target), grandparent (→ repo root)
let mut dir = Some(parent);
for _ in 0..3 {
if let Some(d) = dir {
let candidate = d.join("registry");
if candidate.is_dir() {
return Ok(candidate);
}
dir = d.parent();
}
}
}
// Try CARGO_MANIFEST_DIR (compile-time, works in dev builds)
let manifest_dir = std::path::Path::new(env!("CARGO_MANIFEST_DIR"));
let candidate = manifest_dir.join("registry");
if candidate.is_dir() {
return Ok(candidate);
}
anyhow::bail!(
"Could not find registry/ directory. Run from the ironclaw repo root, \
or ensure registry/ is next to the ironclaw binary."
)
}
fn cmd_list(
catalog: &RegistryCatalog,
kind: Option<&str>,
tag: Option<&str>,
verbose: bool,
) -> anyhow::Result<()> {
let kind_filter = match kind {
Some("tool" | "tools") => Some(ManifestKind::Tool),
Some("channel" | "channels") => Some(ManifestKind::Channel),
Some(other) => anyhow::bail!("Unknown kind '{}'. Use 'tool' or 'channel'.", other),
None => None,
};
let manifests = catalog.list(kind_filter, tag);
if manifests.is_empty() {
println!("No extensions found matching the criteria.");
return Ok(());
}
// Print header
if verbose {
println!(
"{:<20} {:<8} {:<8} {:<10} DESCRIPTION",
"NAME", "KIND", "VERSION", "AUTH"
);
println!("{}", "-".repeat(80));
} else {
println!("{:<20} {:<8} DESCRIPTION", "NAME", "KIND");
println!("{}", "-".repeat(60));
}
for m in &manifests {
if verbose {
let auth = m
.auth_summary
.as_ref()
.and_then(|a| a.method.as_deref())
.unwrap_or("none");
println!(
"{:<20} {:<8} {:<8} {:<10} {}",
m.name, m.kind, m.version, auth, m.description
);
} else {
println!("{:<20} {:<8} {}", m.name, m.kind, m.description);
}
}
println!("\n{} extension(s) found.", manifests.len());
// Show bundles hint
let bundle_names = catalog.bundle_names();
if !bundle_names.is_empty() {
println!("\nBundles available: {}", bundle_names.join(", "));
println!("Use `ironclaw registry info <bundle>` for details.");
}
Ok(())
}
fn cmd_info(catalog: &RegistryCatalog, name: &str) -> anyhow::Result<()> {
// Check if it's a bundle
if let Some(bundle) = catalog.get_bundle(name) {
println!("Bundle: {}", bundle.display_name);
if let Some(desc) = &bundle.description {
println!(" {}", desc);
}
println!("\nExtensions:");
for ext_key in &bundle.extensions {
if let Some(m) = catalog.get(ext_key) {
println!(" {} - {} ({})", ext_key, m.description, m.kind);
} else {
println!(" {} (not found in registry)", ext_key);
}
}
if let Some(shared) = &bundle.shared_auth {
println!("\nShared auth: {}", shared);
}
return Ok(());
}
// Single extension (use get_strict to surface ambiguous bare names)
let manifest = catalog
.get_strict(name)
.map_err(|e| anyhow::anyhow!("{}", e))?;
println!("{} ({})", manifest.display_name, manifest.kind);
println!(" Version: {}", manifest.version);
println!(" {}", manifest.description);
if !manifest.keywords.is_empty() {
println!(" Keywords: {}", manifest.keywords.join(", "));
}
println!("\nSource:");
println!(" Directory: {}", manifest.source.dir);
println!(" Crate: {}", manifest.source.crate_name);
println!(" Capabilities: {}", manifest.source.capabilities);
if let Some(artifact) = manifest.artifacts.get("wasm32-wasip2") {
println!("\nArtifact (wasm32-wasip2):");
match &artifact.url {
Some(url) => println!(" URL: {}", url),
None => println!(" URL: (not yet published)"),
}
match &artifact.sha256 {
Some(sha) => println!(" SHA256: {}", sha),
None => println!(" SHA256: (not yet computed)"),
}
}
if let Some(auth) = &manifest.auth_summary {
println!("\nAuthentication:");
if let Some(method) = &auth.method {
println!(" Method: {}", method);
}
if let Some(provider) = &auth.provider {
println!(" Provider: {}", provider);
}
if !auth.secrets.is_empty() {
println!(" Secrets: {}", auth.secrets.join(", "));
}
if let Some(shared) = &auth.shared_auth {
println!(" Shared with: {}", shared);
}
if let Some(url) = &auth.setup_url {
println!(" Setup: {}", url);
}
}
if !manifest.tags.is_empty() {
println!("\nTags: {}", manifest.tags.join(", "));
}
Ok(())
}
async fn cmd_install(
catalog: &RegistryCatalog,
registry_dir: &std::path::Path,
name: &str,
force: bool,
prefer_build: bool,
) -> anyhow::Result<()> {
// Registry dir parent is the repo root
let repo_root = registry_dir
.parent()
.ok_or_else(|| anyhow::anyhow!("Cannot determine repo root from registry dir"))?;
let installer = RegistryInstaller::with_defaults(repo_root.to_path_buf());
let (manifests, bundle) = catalog.resolve(name)?;
if manifests.is_empty() {
anyhow::bail!("No extensions found for '{}'.", name);
}
if let Some(bundle_def) = bundle {
// Bundle install
println!(
"Installing bundle '{}' ({} extensions)...\n",
bundle_def.display_name,
manifests.len()
);
let (outcomes, hints) = installer
.install_bundle(&manifests, bundle_def, force, prefer_build)
.await;
println!("\n--- Results ---");
for outcome in &outcomes {
let caps_status = if outcome.has_capabilities { "+" } else { "-" };
println!(
" [{}] {} ({}) -> {}",
caps_status,
outcome.name,
outcome.kind,
outcome.wasm_path.display()
);
for w in &outcome.warnings {
println!(" Warning: {}", w);
}
}
if !hints.is_empty() {
println!("\nAuth setup:");
for hint in &hints {
println!("{}", hint);
}
}
println!(
"\nInstalled {}/{} extensions.",
outcomes.len(),
manifests.len()
);
} else {
// Single extension
let manifest = manifests[0];
let outcome = installer.install(manifest, force, prefer_build).await?;
println!("\nInstalled successfully:");
println!(" Name: {}", outcome.name);
println!(" Kind: {}", outcome.kind);
println!(" WASM: {}", outcome.wasm_path.display());
println!(" Capabilities: {}", outcome.has_capabilities);
if let Some(auth) = &manifest.auth_summary
&& auth.method.as_deref() != Some("none")
{
println!(
"\nNext step: authenticate with `ironclaw tool auth {}`",
manifest.name
);
if let Some(url) = &auth.setup_url {
println!(" Setup credentials at: {}", url);
}
}
}
Ok(())
}
+2 -6
View File
@@ -102,16 +102,12 @@ impl EmbeddingsConfig {
#[cfg(test)]
mod tests {
use super::*;
use crate::config::helpers::ENV_MUTEX;
use crate::settings::{EmbeddingsSettings, Settings};
use std::sync::Mutex;
/// Serializes env-mutating tests to prevent parallel races.
static ENV_MUTEX: Mutex<()> = Mutex::new(());
/// Clear all embedding-related env vars.
fn clear_embedding_env() {
// SAFETY: Only called under ENV_MUTEX in tests. No other threads
// observe these vars while the lock is held.
// SAFETY: Only called under ENV_MUTEX in tests.
unsafe {
std::env::remove_var("EMBEDDING_ENABLED");
std::env::remove_var("EMBEDDING_PROVIDER");
+9
View File
@@ -2,6 +2,15 @@ use crate::error::ConfigError;
use super::INJECTED_VARS;
/// Crate-wide mutex for tests that mutate process environment variables.
///
/// The process environment is global state shared across all threads.
/// Per-module mutexes do NOT prevent races between modules running in
/// parallel. Every `unsafe { set_var / remove_var }` call in tests
/// MUST hold this single lock.
#[cfg(test)]
pub(crate) static ENV_MUTEX: std::sync::Mutex<()> = std::sync::Mutex::new(());
pub(crate) fn optional_env(key: &str) -> Result<Option<String>, ConfigError> {
// Check real env vars first (always win over injected secrets)
match std::env::var(key) {
+70
View File
@@ -0,0 +1,70 @@
use crate::config::helpers::optional_env;
use crate::error::ConfigError;
/// Memory hygiene configuration.
///
/// Controls automatic cleanup of stale workspace documents.
/// Maps to `crate::workspace::hygiene::HygieneConfig`.
#[derive(Debug, Clone)]
pub struct HygieneConfig {
/// Whether hygiene is enabled. Env: `MEMORY_HYGIENE_ENABLED` (default: true).
pub enabled: bool,
/// Days before `daily/` documents are deleted. Env: `MEMORY_HYGIENE_RETENTION_DAYS` (default: 30).
pub retention_days: u32,
/// Minimum hours between hygiene passes. Env: `MEMORY_HYGIENE_CADENCE_HOURS` (default: 12).
pub cadence_hours: u32,
}
impl Default for HygieneConfig {
fn default() -> Self {
Self {
enabled: true,
retention_days: 30,
cadence_hours: 12,
}
}
}
impl HygieneConfig {
pub(crate) fn resolve() -> Result<Self, ConfigError> {
Ok(Self {
enabled: optional_env("MEMORY_HYGIENE_ENABLED")?
.map(|s| s.parse())
.transpose()
.map_err(|e| ConfigError::InvalidValue {
key: "MEMORY_HYGIENE_ENABLED".to_string(),
message: format!("must be 'true' or 'false': {e}"),
})?
.unwrap_or(true),
retention_days: optional_env("MEMORY_HYGIENE_RETENTION_DAYS")?
.map(|s| s.parse())
.transpose()
.map_err(|e| ConfigError::InvalidValue {
key: "MEMORY_HYGIENE_RETENTION_DAYS".to_string(),
message: format!("must be a positive integer: {e}"),
})?
.unwrap_or(30),
cadence_hours: optional_env("MEMORY_HYGIENE_CADENCE_HOURS")?
.map(|s| s.parse())
.transpose()
.map_err(|e| ConfigError::InvalidValue {
key: "MEMORY_HYGIENE_CADENCE_HOURS".to_string(),
message: format!("must be a positive integer: {e}"),
})?
.unwrap_or(12),
})
}
/// Convert to the workspace hygiene config, resolving the state directory
/// to the standard `~/.ironclaw` location.
pub fn to_workspace_config(&self) -> crate::workspace::hygiene::HygieneConfig {
crate::workspace::hygiene::HygieneConfig {
enabled: self.enabled,
retention_days: self.retention_days,
cadence_hours: self.cadence_hours,
state_dir: dirs::home_dir()
.unwrap_or_else(|| std::path::PathBuf::from("."))
.join(".ironclaw"),
}
}
}
+12 -50
View File
@@ -121,34 +121,7 @@ pub struct LlmConfig {
pub tinfoil: Option<TinfoilConfig>,
}
/// API mode for NEAR AI.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
pub enum NearAiApiMode {
/// Use the Responses API (chat-api proxy) - session-based auth
#[default]
Responses,
/// Use the Chat Completions API (cloud-api) - API key auth
ChatCompletions,
}
impl std::str::FromStr for NearAiApiMode {
type Err = String;
fn from_str(s: &str) -> Result<Self, Self::Err> {
match s.to_lowercase().as_str() {
"responses" | "response" => Ok(Self::Responses),
"chat_completions" | "chatcompletions" | "chat" | "completions" => {
Ok(Self::ChatCompletions)
}
_ => Err(format!(
"invalid API mode '{}', expected 'responses' or 'chat_completions'",
s
)),
}
}
}
/// NEAR AI chat-api configuration.
/// NEAR AI configuration.
#[derive(Debug, Clone)]
pub struct NearAiConfig {
/// Model to use (e.g., "claude-3-5-sonnet-20241022", "gpt-4o")
@@ -156,15 +129,14 @@ pub struct NearAiConfig {
/// Cheap/fast model for lightweight tasks (heartbeat, routing, evaluation).
/// Falls back to the main model if not set.
pub cheap_model: Option<String>,
/// Base URL for the NEAR AI API (default: https://private.near.ai).
/// Base URL for the NEAR AI API.
/// Default: `https://private.near.ai` (session token) or `https://cloud-api.near.ai` (API key)
pub base_url: String,
/// Base URL for auth/refresh endpoints (default: https://private.near.ai)
pub auth_base_url: String,
/// Path to session file (default: ~/.ironclaw/session.json)
pub session_path: PathBuf,
/// API mode: "responses" (chat-api) or "chat_completions" (cloud-api)
pub api_mode: NearAiApiMode,
/// API key for cloud-api (required for chat_completions mode)
/// API key for NEAR AI Cloud. When set, uses API key auth; otherwise uses session token auth.
pub api_key: Option<SecretString>,
/// Optional fallback model for failover (default: None).
/// When set, a secondary provider is created with this model and wrapped
@@ -224,17 +196,6 @@ impl LlmConfig {
// Resolve NEAR AI config only when backend is NearAi (or when explicitly configured)
let nearai_api_key = optional_env("NEARAI_API_KEY")?.map(SecretString::from);
let api_mode = if let Some(mode_str) = optional_env("NEARAI_API_MODE")? {
mode_str.parse().map_err(|e| ConfigError::InvalidValue {
key: "NEARAI_API_MODE".to_string(),
message: e,
})?
} else if nearai_api_key.is_some() {
NearAiApiMode::ChatCompletions
} else {
NearAiApiMode::Responses
};
let nearai = NearAiConfig {
model: optional_env("NEARAI_MODEL")?
.or_else(|| settings.selected_model.clone())
@@ -243,14 +204,18 @@ impl LlmConfig {
.to_string()
}),
cheap_model: optional_env("NEARAI_CHEAP_MODEL")?,
base_url: optional_env("NEARAI_BASE_URL")?
.unwrap_or_else(|| "https://private.near.ai".to_string()),
base_url: optional_env("NEARAI_BASE_URL")?.unwrap_or_else(|| {
if nearai_api_key.is_some() {
"https://cloud-api.near.ai".to_string()
} else {
"https://private.near.ai".to_string()
}
}),
auth_base_url: optional_env("NEARAI_AUTH_URL")?
.unwrap_or_else(|| "https://private.near.ai".to_string()),
session_path: optional_env("NEARAI_SESSION_PATH")?
.map(PathBuf::from)
.unwrap_or_else(default_session_path),
api_mode,
api_key: nearai_api_key,
fallback_model: optional_env("NEARAI_FALLBACK_MODEL")?,
max_retries: parse_optional_env("NEARAI_MAX_RETRIES", 3)?,
@@ -373,11 +338,8 @@ fn default_session_path() -> PathBuf {
#[cfg(test)]
mod tests {
use super::*;
use crate::config::helpers::ENV_MUTEX;
use crate::settings::Settings;
use std::sync::Mutex;
/// Serializes env-mutating tests to prevent parallel races.
static ENV_MUTEX: Mutex<()> = Mutex::new(());
/// Clear all openai-compatible-related env vars.
fn clear_openai_compatible_env() {
+6 -1
View File
@@ -12,6 +12,7 @@ mod database;
mod embeddings;
mod heartbeat;
pub(crate) mod helpers;
mod hygiene;
mod llm;
mod routines;
mod safety;
@@ -34,8 +35,9 @@ pub use self::channels::{ChannelsConfig, CliConfig, GatewayConfig, HttpConfig};
pub use self::database::{DatabaseBackend, DatabaseConfig, default_libsql_path};
pub use self::embeddings::EmbeddingsConfig;
pub use self::heartbeat::HeartbeatConfig;
pub use self::hygiene::HygieneConfig;
pub use self::llm::{
AnthropicDirectConfig, LlmBackend, LlmConfig, NearAiApiMode, NearAiConfig, OllamaConfig,
AnthropicDirectConfig, LlmBackend, LlmConfig, NearAiConfig, OllamaConfig,
OpenAiCompatibleConfig, OpenAiDirectConfig, TinfoilConfig,
};
pub use self::routines::RoutineConfig;
@@ -67,6 +69,7 @@ pub struct Config {
pub secrets: SecretsConfig,
pub builder: BuilderModeConfig,
pub heartbeat: HeartbeatConfig,
pub hygiene: HygieneConfig,
pub routines: RoutineConfig,
pub sandbox: SandboxModeConfig,
pub claude_code: ClaudeCodeConfig,
@@ -190,6 +193,7 @@ impl Config {
secrets: SecretsConfig::resolve().await?,
builder: BuilderModeConfig::resolve()?,
heartbeat: HeartbeatConfig::resolve(settings)?,
hygiene: HygieneConfig::resolve()?,
routines: RoutineConfig::resolve()?,
sandbox: SandboxModeConfig::resolve()?,
claude_code: ClaudeCodeConfig::resolve()?,
@@ -215,6 +219,7 @@ pub async fn inject_llm_keys_from_secrets(
("llm_openai_api_key", "OPENAI_API_KEY"),
("llm_anthropic_api_key", "ANTHROPIC_API_KEY"),
("llm_compatible_api_key", "LLM_API_KEY"),
("llm_nearai_api_key", "NEARAI_API_KEY"),
];
let mut injected = HashMap::new();
+4 -4
View File
@@ -320,10 +320,10 @@ pub(crate) fn row_to_routine_libsql(row: &libsql::Row) -> Result<Routine, Databa
let max_concurrent = get_i64(row, 10);
let dedup_window_secs: Option<i64> = row.get::<i64>(11).ok();
let trigger =
Trigger::from_db(&trigger_type, trigger_config).map_err(DatabaseError::Serialization)?;
let trigger = Trigger::from_db(&trigger_type, trigger_config)
.map_err(|e| DatabaseError::Serialization(e.to_string()))?;
let action = RoutineAction::from_db(&action_type, action_config)
.map_err(DatabaseError::Serialization)?;
.map_err(|e| DatabaseError::Serialization(e.to_string()))?;
Ok(Routine {
id: get_text(row, 0).parse().unwrap_or_default(),
@@ -359,7 +359,7 @@ pub(crate) fn row_to_routine_run_libsql(row: &libsql::Row) -> Result<RoutineRun,
let status_str = get_text(row, 5);
let status: RunStatus = status_str
.parse()
.map_err(|e: String| DatabaseError::Serialization(e))?;
.map_err(|e: crate::error::RoutineError| DatabaseError::Serialization(e.to_string()))?;
Ok(RoutineRun {
id: get_text(row, 0).parse().unwrap_or_default(),
+43
View File
@@ -48,6 +48,9 @@ pub enum Error {
#[error("Worker error: {0}")]
Worker(#[from] WorkerError),
#[error("Routine error: {0}")]
Routine(#[from] RoutineError),
}
/// Configuration-related errors.
@@ -365,5 +368,45 @@ pub enum WorkerError {
MissingToken,
}
/// Routine-related errors.
#[derive(Debug, thiserror::Error)]
pub enum RoutineError {
#[error("Unknown trigger type: {trigger_type}")]
UnknownTriggerType { trigger_type: String },
#[error("Unknown action type: {action_type}")]
UnknownActionType { action_type: String },
#[error("Missing field in {context}: {field}")]
MissingField { context: String, field: String },
#[error("Invalid cron expression: {reason}")]
InvalidCron { reason: String },
#[error("Unknown run status: {status}")]
UnknownRunStatus { status: String },
#[error("Routine {name} is disabled")]
Disabled { name: String },
#[error("Routine not found: {id}")]
NotFound { id: Uuid },
#[error("Routine {name} at max concurrent runs")]
MaxConcurrent { name: String },
#[error("Database error: {reason}")]
Database { reason: String },
#[error("LLM call failed: {reason}")]
LlmFailed { reason: String },
#[error("LLM returned empty content")]
EmptyResponse,
#[error("LLM response truncated (finish_reason=length) with no content")]
TruncatedResponse,
}
/// Result type alias for the agent.
pub type Result<T> = std::result::Result<T, Error>;
+4 -4
View File
@@ -1179,10 +1179,10 @@ fn row_to_routine(row: &tokio_postgres::Row) -> Result<Routine, DatabaseError> {
let max_concurrent: i32 = row.get("max_concurrent");
let dedup_window_secs: Option<i32> = row.get("dedup_window_secs");
let trigger =
Trigger::from_db(&trigger_type, trigger_config).map_err(DatabaseError::Serialization)?;
let trigger = Trigger::from_db(&trigger_type, trigger_config)
.map_err(|e| DatabaseError::Serialization(e.to_string()))?;
let action = RoutineAction::from_db(&action_type, action_config)
.map_err(DatabaseError::Serialization)?;
.map_err(|e| DatabaseError::Serialization(e.to_string()))?;
Ok(Routine {
id: row.get("id"),
@@ -1219,7 +1219,7 @@ fn row_to_routine_run(row: &tokio_postgres::Row) -> Result<RoutineRun, DatabaseE
let status_str: String = row.get("status");
let status: RunStatus = status_str
.parse()
.map_err(|e: String| DatabaseError::Serialization(e))?;
.map_err(|e: crate::error::RoutineError| DatabaseError::Serialization(e.to_string()))?;
Ok(RoutineRun {
id: row.get("id"),
+1
View File
@@ -57,6 +57,7 @@ pub mod llm;
pub mod observability;
pub mod orchestrator;
pub mod pairing;
pub mod registry;
pub mod safety;
pub mod sandbox;
pub mod secrets;
-8
View File
@@ -296,14 +296,6 @@ impl LlmProvider for CircuitBreakerProvider {
self.inner.set_model(model)
}
fn seed_response_chain(&self, thread_id: &str, response_id: String) {
self.inner.seed_response_chain(thread_id, response_id)
}
fn get_response_chain_id(&self, thread_id: &str) -> Option<String> {
self.inner.get_response_chain_id(thread_id)
}
fn calculate_cost(&self, input_tokens: u32, output_tokens: u32) -> Decimal {
self.inner.calculate_cost(input_tokens, output_tokens)
}
+38 -12
View File
@@ -17,7 +17,17 @@ pub fn model_cost(model_id: &str) -> Option<(Decimal, Decimal)> {
.unwrap_or(model_id);
match id {
// OpenAI models -- prices per token (USD)
// OpenAI — GPT-5.x / Codex
"gpt-5.3-codex" | "gpt-5.3-codex-spark" => Some((dec!(0.000002), dec!(0.000008))),
"gpt-5.2-codex" | "gpt-5.2-pro" | "gpt-5.2" => Some((dec!(0.000002), dec!(0.000008))),
"gpt-5.1-codex" | "gpt-5.1-codex-max" | "gpt-5.1" => Some((dec!(0.000002), dec!(0.000008))),
"gpt-5.1-codex-mini" => Some((dec!(0.0000003), dec!(0.0000012))),
"gpt-5-codex" | "gpt-5-pro" | "gpt-5" => Some((dec!(0.000002), dec!(0.000008))),
"gpt-5-mini" | "gpt-5-nano" => Some((dec!(0.0000003), dec!(0.0000012))),
// OpenAI — GPT-4.x
"gpt-4.1" => Some((dec!(0.000002), dec!(0.000008))),
"gpt-4.1-mini" => Some((dec!(0.0000004), dec!(0.0000016))),
"gpt-4.1-nano" => Some((dec!(0.0000001), dec!(0.0000004))),
"gpt-4o" | "gpt-4o-2024-11-20" | "gpt-4o-2024-08-06" => {
Some((dec!(0.0000025), dec!(0.00001)))
}
@@ -25,20 +35,36 @@ pub fn model_cost(model_id: &str) -> Option<(Decimal, Decimal)> {
"gpt-4-turbo" | "gpt-4-turbo-2024-04-09" => Some((dec!(0.00001), dec!(0.00003))),
"gpt-4" | "gpt-4-0613" => Some((dec!(0.00003), dec!(0.00006))),
"gpt-3.5-turbo" | "gpt-3.5-turbo-0125" => Some((dec!(0.0000005), dec!(0.0000015))),
// OpenAI — reasoning
"o3" => Some((dec!(0.000002), dec!(0.000008))),
"o3-mini" | "o3-mini-2025-01-31" => Some((dec!(0.0000011), dec!(0.0000044))),
"o4-mini" => Some((dec!(0.0000011), dec!(0.0000044))),
"o1" | "o1-2024-12-17" => Some((dec!(0.000015), dec!(0.00006))),
"o1-mini" | "o1-mini-2024-09-12" => Some((dec!(0.000003), dec!(0.000012))),
"o3-mini" | "o3-mini-2025-01-31" => Some((dec!(0.0000011), dec!(0.0000044))),
// Anthropic models
"claude-3-5-sonnet-20241022" | "claude-3-5-sonnet-latest" | "claude-sonnet-4-20250514" => {
Some((dec!(0.000003), dec!(0.000015)))
}
"claude-3-5-haiku-20241022" | "claude-3-5-haiku-latest" => {
Some((dec!(0.0000008), dec!(0.000004)))
}
"claude-3-opus-20240229" | "claude-3-opus-latest" | "claude-opus-4-20250514" => {
Some((dec!(0.000015), dec!(0.000075)))
}
// Anthropic
"claude-opus-4-6"
| "claude-opus-4-5"
| "claude-opus-4-5-20251101"
| "claude-opus-4-1"
| "claude-opus-4-1-20250805"
| "claude-opus-4-0"
| "claude-opus-4-20250514"
| "claude-3-opus-20240229"
| "claude-3-opus-latest" => Some((dec!(0.000015), dec!(0.000075))),
"claude-sonnet-4-6"
| "claude-sonnet-4-5"
| "claude-sonnet-4-5-20250929"
| "claude-sonnet-4-0"
| "claude-sonnet-4-20250514"
| "claude-3-7-sonnet-20250219"
| "claude-3-7-sonnet-latest"
| "claude-3-5-sonnet-20241022"
| "claude-3-5-sonnet-latest" => Some((dec!(0.000003), dec!(0.000015))),
"claude-haiku-4-5"
| "claude-haiku-4-5-20251001"
| "claude-3-5-haiku-20241022"
| "claude-3-5-haiku-latest" => Some((dec!(0.0000008), dec!(0.000004))),
"claude-3-haiku-20240307" => Some((dec!(0.00000025), dec!(0.00000125))),
// Ollama / local models -- free
-13
View File
@@ -359,15 +359,6 @@ impl LlmProvider for FailoverProvider {
.await
}
fn seed_response_chain(&self, thread_id: &str, response_id: String) {
self.providers[self.last_used.load(Ordering::Relaxed)]
.seed_response_chain(thread_id, response_id);
}
fn get_response_chain_id(&self, thread_id: &str) -> Option<String> {
self.providers[self.last_used.load(Ordering::Relaxed)].get_response_chain_id(thread_id)
}
fn calculate_cost(&self, input_tokens: u32, output_tokens: u32) -> Decimal {
self.providers[self.last_used.load(Ordering::Relaxed)]
.calculate_cost(input_tokens, output_tokens)
@@ -413,7 +404,6 @@ mod tests {
input_tokens: 10,
output_tokens: 5,
finish_reason: FinishReason::Stop,
response_id: None,
}))),
tool_complete_result: Mutex::new(Some(Ok(ToolCompletionResponse {
content: Some(content.to_string()),
@@ -421,7 +411,6 @@ mod tests {
input_tokens: 10,
output_tokens: 5,
finish_reason: FinishReason::Stop,
response_id: None,
}))),
}
}
@@ -803,7 +792,6 @@ mod tests {
input_tokens: 10,
output_tokens: 5,
finish_reason: FinishReason::Stop,
response_id: None,
})
}
@@ -829,7 +817,6 @@ mod tests {
input_tokens: 10,
output_tokens: 5,
finish_reason: FinishReason::Stop,
response_id: None,
})
}
+21 -32
View File
@@ -1,7 +1,7 @@
//! LLM integration for the agent.
//!
//! Supports multiple backends:
//! - **NEAR AI** (default): Session-based or API key auth via NEAR AI proxy
//! - **NEAR AI** (default): Session token or API key auth via Chat Completions API
//! - **OpenAI**: Direct API access with your own key
//! - **Anthropic**: Direct API access with your own key
//! - **Ollama**: Local model inference
@@ -10,7 +10,6 @@
pub mod circuit_breaker;
pub mod costs;
pub mod failover;
mod nearai;
mod nearai_chat;
mod provider;
mod reasoning;
@@ -21,8 +20,7 @@ pub mod session;
pub use circuit_breaker::{CircuitBreakerConfig, CircuitBreakerProvider};
pub use failover::{CooldownConfig, FailoverProvider};
pub use nearai::{ModelInfo, NearAiProvider};
pub use nearai_chat::NearAiChatProvider;
pub use nearai_chat::{ModelInfo, NearAiChatProvider};
pub use provider::{
ChatMessage, CompletionRequest, CompletionResponse, FinishReason, LlmProvider, ModelMetadata,
Role, ToolCall, ToolCompletionRequest, ToolCompletionResponse, ToolDefinition, ToolResult,
@@ -41,7 +39,7 @@ use std::sync::Arc;
use rig::client::CompletionClient;
use secrecy::ExposeSecret;
use crate::config::{LlmBackend, LlmConfig, NearAiApiMode, NearAiConfig};
use crate::config::{LlmBackend, LlmConfig, NearAiConfig};
use crate::error::LlmError;
/// Create an LLM provider based on configuration.
@@ -71,22 +69,18 @@ pub fn create_llm_provider_with_config(
config: &NearAiConfig,
session: Arc<SessionManager>,
) -> Result<Arc<dyn LlmProvider>, LlmError> {
match config.api_mode {
NearAiApiMode::Responses => {
tracing::info!(
model = %config.model,
"Using Responses API (chat-api) with session auth"
);
Ok(Arc::new(NearAiProvider::new(config.clone(), session)?))
}
NearAiApiMode::ChatCompletions => {
tracing::info!(
model = %config.model,
"Using Chat Completions API (cloud-api) with API key auth"
);
Ok(Arc::new(NearAiChatProvider::new(config.clone())?))
}
}
let auth_mode = if config.api_key.is_some() {
"API key"
} else {
"session token"
};
tracing::info!(
model = %config.model,
base_url = %config.base_url,
auth = auth_mode,
"Using NEAR AI (Chat Completions API)"
);
Ok(Arc::new(NearAiChatProvider::new(config.clone(), session)?))
}
fn create_openai_provider(config: &LlmConfig) -> Result<Arc<dyn LlmProvider>, LlmError> {
@@ -252,7 +246,7 @@ fn create_openai_compatible_provider(config: &LlmConfig) -> Result<Arc<dyn LlmPr
/// Create a cheap/fast LLM provider for lightweight tasks (heartbeat, routing, evaluation).
///
/// Uses `NEARAI_CHEAP_MODEL` if set, otherwise falls back to the main provider.
/// Currently only supports NEAR AI backends (Responses and ChatCompletions modes).
/// Currently only supports NEAR AI backend.
pub fn create_cheap_llm_provider(
config: &LlmConfig,
session: Arc<SessionManager>,
@@ -273,20 +267,16 @@ pub fn create_cheap_llm_provider(
let mut cheap_config = config.nearai.clone();
cheap_config.model = cheap_model.clone();
tracing::info!("Cheap LLM provider: {}", cheap_model);
match cheap_config.api_mode {
NearAiApiMode::Responses => Ok(Some(Arc::new(NearAiProvider::new(cheap_config, session)?))),
NearAiApiMode::ChatCompletions => {
Ok(Some(Arc::new(NearAiChatProvider::new(cheap_config)?)))
}
}
Ok(Some(Arc::new(NearAiChatProvider::new(
cheap_config,
session,
)?)))
}
#[cfg(test)]
mod tests {
use super::*;
use crate::config::{LlmBackend, NearAiApiMode, NearAiConfig};
use crate::config::{LlmBackend, NearAiConfig};
use std::path::PathBuf;
fn test_nearai_config() -> NearAiConfig {
@@ -296,7 +286,6 @@ mod tests {
base_url: "https://api.near.ai".to_string(),
auth_base_url: "https://private.near.ai".to_string(),
session_path: PathBuf::from("/tmp/test-session.json"),
api_mode: NearAiApiMode::Responses,
api_key: None,
fallback_model: None,
max_retries: 3,
-1203
View File
File diff suppressed because it is too large Load Diff
+200 -61
View File
@@ -1,7 +1,12 @@
//! NEAR AI Chat Completions API provider implementation.
//! NEAR AI provider implementation (Chat Completions API).
//!
//! This provider uses the standard OpenAI-compatible chat completions API
//! with API key authentication (for cloud-api).
//! This provider uses the OpenAI-compatible Chat Completions endpoint with
//! dual auth support:
//! - **API key auth**: When `NEARAI_API_KEY` is set, uses Bearer API key
//! - **Session token auth**: Otherwise, uses `SessionManager` for Bearer session token
//! with automatic renewal on 401 errors
use std::sync::Arc;
use async_trait::async_trait;
use reqwest::Client;
@@ -13,38 +18,51 @@ use serde::{Deserialize, Serialize};
use crate::config::NearAiConfig;
use crate::error::LlmError;
use crate::llm::provider::{
ChatMessage, CompletionRequest, CompletionResponse, FinishReason, LlmProvider, ModelMetadata,
Role, ToolCall, ToolCompletionRequest, ToolCompletionResponse,
ChatMessage, CompletionRequest, CompletionResponse, FinishReason, LlmProvider, Role, ToolCall,
ToolCompletionRequest, ToolCompletionResponse,
};
use crate::llm::session::SessionManager;
/// NEAR AI Chat Completions API provider.
/// Information about an available model from NEAR AI API.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ModelInfo {
/// Model identifier.
#[serde(alias = "id", alias = "model")]
pub name: String,
/// Optional provider name.
#[serde(default)]
pub provider: Option<String>,
}
/// NEAR AI provider (Chat Completions API, dual auth).
pub struct NearAiChatProvider {
client: Client,
config: NearAiConfig,
/// Session manager for session token auth (used when no API key is set).
session: Arc<SessionManager>,
active_model: std::sync::RwLock<String>,
flatten_tool_messages: bool,
}
impl NearAiChatProvider {
/// Create a new NEAR AI chat completions provider with API key auth.
/// Create a new NEAR AI Chat Completions provider.
///
/// Auth mode is determined by `config.api_key`:
/// - If set, uses Bearer API key auth
/// - If not set, uses session token auth via `SessionManager`
///
/// By default this enables tool-message flattening for compatibility with
/// providers that reject `role: "tool"` messages (e.g. NEAR cloud-api).
pub fn new(config: NearAiConfig) -> Result<Self, LlmError> {
Self::new_with_flatten(config, true)
/// providers that reject `role: "tool"` messages.
pub fn new(config: NearAiConfig, session: Arc<SessionManager>) -> Result<Self, LlmError> {
Self::new_with_flatten(config, session, true)
}
/// Create a chat completions provider with configurable tool-message flattening.
pub fn new_with_flatten(
config: NearAiConfig,
session: Arc<SessionManager>,
flatten_tool_messages: bool,
) -> Result<Self, LlmError> {
if config.api_key.is_none() {
return Err(LlmError::AuthFailed {
provider: "nearai_chat".to_string(),
});
}
let client = Client::builder()
.timeout(std::time::Duration::from_secs(120))
.build()
@@ -57,6 +75,7 @@ impl NearAiChatProvider {
Ok(Self {
client,
config,
session,
active_model,
flatten_tool_messages,
})
@@ -73,23 +92,50 @@ impl NearAiChatProvider {
}
}
fn api_key(&self) -> String {
self.config
.api_key
.as_ref()
.map(|k| k.expose_secret().to_string())
.unwrap_or_default()
/// Returns true if using API key auth, false if session token auth.
fn uses_api_key(&self) -> bool {
self.config.api_key.is_some()
}
/// Resolve the Bearer token for the current auth mode.
async fn resolve_bearer_token(&self) -> Result<String, LlmError> {
if let Some(ref api_key) = self.config.api_key {
Ok(api_key.expose_secret().to_string())
} else {
let token = self.session.get_token().await?;
Ok(token.expose_secret().to_string())
}
}
/// Send a single request to the chat completions API.
///
/// Does not retry internally — retries are handled by the external
/// For session token auth, handles 401 by calling `session.handle_auth_failure()`
/// and retrying once.
///
/// Does not retry on other errors — retries are handled by the external
/// `RetryProvider` wrapper in the composition chain.
async fn send_request<T: Serialize, R: for<'de> Deserialize<'de>>(
&self,
body: &T,
) -> Result<R, LlmError> {
match self.send_request_inner(body).await {
Ok(result) => Ok(result),
Err(LlmError::SessionExpired { .. }) if !self.uses_api_key() => {
// Session expired, attempt renewal and retry once
self.session.handle_auth_failure().await?;
self.send_request_inner(body).await
}
Err(e) => Err(e),
}
}
/// Inner request implementation (single attempt).
async fn send_request_inner<T: Serialize, R: for<'de> Deserialize<'de>>(
&self,
body: &T,
) -> Result<R, LlmError> {
let url = self.api_url("chat/completions");
let token = self.resolve_bearer_token().await?;
tracing::debug!("Sending request to NEAR AI Chat: {}", url);
@@ -102,7 +148,7 @@ impl NearAiChatProvider {
let response = self
.client
.post(&url)
.header("Authorization", format!("Bearer {}", self.api_key()))
.header("Authorization", format!("Bearer {}", token))
.header("Content-Type", "application/json")
.json(body)
.send()
@@ -125,6 +171,17 @@ impl NearAiChatProvider {
let status_code = status.as_u16();
if status_code == 401 {
// For session token auth, distinguish session expired from plain auth failure
if !self.uses_api_key() {
let lower = response_text.to_lowercase();
let is_session_expired = lower.contains("session")
&& (lower.contains("expired") || lower.contains("invalid"));
if is_session_expired {
return Err(LlmError::SessionExpired {
provider: "nearai_chat".to_string(),
});
}
}
return Err(LlmError::AuthFailed {
provider: "nearai_chat".to_string(),
});
@@ -153,14 +210,31 @@ impl NearAiChatProvider {
})
}
/// Fetch available models with full metadata from the `/v1/models` endpoint.
async fn fetch_models(&self) -> Result<Vec<ApiModelEntry>, LlmError> {
/// Fetch available models from the NEAR AI API.
///
/// Handles session renewal on 401 (same pattern as `send_request`).
/// Supports multiple response formats: `{models: [...]}`, `{data: [...]}`, and plain array.
pub async fn list_models_full(&self) -> Result<Vec<ModelInfo>, LlmError> {
match self.list_models_inner().await {
Ok(models) => Ok(models),
Err(LlmError::SessionExpired { .. }) if !self.uses_api_key() => {
self.session.handle_auth_failure().await?;
self.list_models_inner().await
}
Err(e) => Err(e),
}
}
async fn list_models_inner(&self) -> Result<Vec<ModelInfo>, LlmError> {
let url = self.api_url("models");
let token = self.resolve_bearer_token().await?;
tracing::debug!("Fetching models from: {}", url);
let response = self
.client
.get(&url)
.header("Authorization", format!("Bearer {}", self.api_key()))
.header("Authorization", format!("Bearer {}", token))
.send()
.await
.map_err(|e| LlmError::RequestFailed {
@@ -175,6 +249,11 @@ impl NearAiChatProvider {
})?;
if !status.is_success() {
if status.as_u16() == 401 && !self.uses_api_key() {
return Err(LlmError::SessionExpired {
provider: "nearai_chat".to_string(),
});
}
let truncated = crate::agent::truncate_for_preview(&response_text, 512);
return Err(LlmError::RequestFailed {
provider: "nearai_chat".to_string(),
@@ -182,29 +261,97 @@ impl NearAiChatProvider {
});
}
// Flexible model entry parsing -- handle various field names
#[derive(Deserialize)]
struct ModelsResponse {
data: Vec<ApiModelEntry>,
struct ModelMetadataInner {
#[serde(default)]
name: Option<String>,
#[serde(default, alias = "modelName", alias = "model_name")]
model_name: Option<String>,
}
let resp: ModelsResponse =
serde_json::from_str(&response_text).map_err(|e| LlmError::InvalidResponse {
provider: "nearai_chat".to_string(),
reason: format!("JSON parse error: {}", e),
})?;
#[derive(Deserialize)]
struct ModelEntry {
#[serde(default)]
name: Option<String>,
#[serde(default)]
id: Option<String>,
#[serde(default)]
model: Option<String>,
#[serde(default, alias = "modelName", alias = "model_name")]
model_name: Option<String>,
#[serde(default, alias = "modelId", alias = "model_id")]
model_id: Option<String>,
#[serde(default)]
metadata: Option<ModelMetadataInner>,
}
Ok(resp.data)
impl ModelEntry {
fn get_name(&self) -> Option<String> {
self.name
.clone()
.or_else(|| self.id.clone())
.or_else(|| self.model.clone())
.or_else(|| self.model_name.clone())
.or_else(|| self.model_id.clone())
.or_else(|| self.metadata.as_ref().and_then(|m| m.name.clone()))
.or_else(|| self.metadata.as_ref().and_then(|m| m.model_name.clone()))
}
}
#[derive(Deserialize)]
struct ModelsResponse {
#[serde(default)]
models: Option<Vec<ModelEntry>>,
#[serde(default)]
data: Option<Vec<ModelEntry>>,
}
// Try {models: [...]} or {data: [...]} format
if let Ok(resp) = serde_json::from_str::<ModelsResponse>(&response_text)
&& let Some(entries) = resp.models.or(resp.data)
{
let models: Vec<ModelInfo> = entries
.into_iter()
.filter_map(|e| {
e.get_name().map(|name| ModelInfo {
name,
provider: None,
})
})
.collect();
if !models.is_empty() {
return Ok(models);
}
}
// Try direct array format
if let Ok(entries) = serde_json::from_str::<Vec<ModelEntry>>(&response_text) {
let models: Vec<ModelInfo> = entries
.into_iter()
.filter_map(|e| {
e.get_name().map(|name| ModelInfo {
name,
provider: None,
})
})
.collect();
if !models.is_empty() {
return Ok(models);
}
}
// Couldn't find model names in response
Err(LlmError::InvalidResponse {
provider: "nearai_chat".to_string(),
reason: format!(
"No model names found in response: {}",
&response_text[..response_text.len().min(300)]
),
})
}
}
/// Model entry as returned by the `/v1/models` API.
#[derive(Debug, Deserialize)]
struct ApiModelEntry {
id: String,
#[serde(default)]
context_length: Option<u32>,
}
#[async_trait]
impl LlmProvider for NearAiChatProvider {
async fn complete(&self, req: CompletionRequest) -> Result<CompletionResponse, LlmError> {
@@ -251,7 +398,6 @@ impl LlmProvider for NearAiChatProvider {
finish_reason,
input_tokens,
output_tokens,
response_id: None,
})
}
@@ -346,7 +492,6 @@ impl LlmProvider for NearAiChatProvider {
finish_reason,
input_tokens,
output_tokens,
response_id: None,
})
}
@@ -360,18 +505,8 @@ impl LlmProvider for NearAiChatProvider {
}
async fn list_models(&self) -> Result<Vec<String>, LlmError> {
let models = self.fetch_models().await?;
Ok(models.into_iter().map(|m| m.id).collect())
}
async fn model_metadata(&self) -> Result<ModelMetadata, LlmError> {
let active = self.active_model_name();
let models = self.fetch_models().await?;
let current = models.iter().find(|m| m.id == active);
Ok(ModelMetadata {
id: active,
context_length: current.and_then(|m| m.context_length),
})
let models = self.list_models_full().await?;
Ok(models.into_iter().map(|m| m.name).collect())
}
fn active_model_name(&self) -> String {
@@ -612,6 +747,7 @@ fn parse_usage(usage: Option<&ChatCompletionUsage>) -> (u32, u32) {
#[cfg(test)]
mod tests {
use super::*;
use crate::llm::session::SessionConfig;
fn test_nearai_config(base_url: &str) -> NearAiConfig {
NearAiConfig {
@@ -619,7 +755,6 @@ mod tests {
base_url: base_url.to_string(),
auth_base_url: "https://private.near.ai".to_string(),
session_path: std::path::PathBuf::from("/tmp/session.json"),
api_mode: crate::config::NearAiApiMode::ChatCompletions,
api_key: Some(secrecy::SecretString::from("test-key".to_string())),
cheap_model: None,
fallback_model: None,
@@ -634,18 +769,22 @@ mod tests {
}
}
fn test_session() -> Arc<SessionManager> {
Arc::new(SessionManager::new(SessionConfig::default()))
}
#[test]
fn test_api_url_with_base_without_v1() {
let mut cfg = test_nearai_config("http://127.0.0.1:8318");
let provider = NearAiChatProvider::new(cfg.clone()).expect("provider");
let provider = NearAiChatProvider::new(cfg.clone(), test_session()).expect("provider");
assert_eq!(
provider.api_url("chat/completions"),
"http://127.0.0.1:8318/v1/chat/completions"
);
cfg.base_url = "http://127.0.0.1:8318/".to_string();
let provider = NearAiChatProvider::new(cfg).expect("provider");
let provider = NearAiChatProvider::new(cfg, test_session()).expect("provider");
assert_eq!(
provider.api_url("/chat/completions"),
"http://127.0.0.1:8318/v1/chat/completions"
@@ -656,7 +795,7 @@ mod tests {
fn test_api_url_with_base_already_v1() {
let cfg = test_nearai_config("http://127.0.0.1:8318/v1");
let provider = NearAiChatProvider::new(cfg).expect("provider");
let provider = NearAiChatProvider::new(cfg, test_session()).expect("provider");
assert_eq!(
provider.api_url("chat/completions"),
"http://127.0.0.1:8318/v1/chat/completions"
-18
View File
@@ -153,8 +153,6 @@ pub struct CompletionResponse {
pub input_tokens: u32,
pub output_tokens: u32,
pub finish_reason: FinishReason,
/// Provider-specific response ID (e.g. for NEAR AI response chaining).
pub response_id: Option<String>,
}
/// Why the completion finished.
@@ -256,8 +254,6 @@ pub struct ToolCompletionResponse {
pub input_tokens: u32,
pub output_tokens: u32,
pub finish_reason: FinishReason,
/// Provider-specific response ID (e.g. for NEAR AI response chaining).
pub response_id: Option<String>,
}
/// Metadata about a model returned by the provider's API.
@@ -327,20 +323,6 @@ pub trait LlmProvider: Send + Sync {
})
}
/// Seed a response chain for a thread (e.g. restoring from DB).
///
/// Providers that support response chaining (e.g. NEAR AI `previous_response_id`)
/// store this so subsequent calls send only delta messages.
fn seed_response_chain(&self, _thread_id: &str, _response_id: String) {}
/// Get the last response chain ID for a thread.
///
/// Returns `None` if the provider doesn't support chaining or has no
/// stored state for this thread.
fn get_response_chain_id(&self, _thread_id: &str) -> Option<String> {
None
}
/// Calculate cost for a completion.
fn calculate_cost(&self, input_tokens: u32, output_tokens: u32) -> Decimal {
let (input_cost, output_cost) = self.cost_per_token();
+687 -159
View File
File diff suppressed because it is too large Load Diff
-8
View File
@@ -228,14 +228,6 @@ impl LlmProvider for CachedProvider {
fn set_model(&self, model: &str) -> Result<(), LlmError> {
self.inner.set_model(model)
}
fn seed_response_chain(&self, thread_id: &str, response_id: String) {
self.inner.seed_response_chain(thread_id, response_id);
}
fn get_response_chain_id(&self, thread_id: &str) -> Option<String> {
self.inner.get_response_chain_id(thread_id)
}
}
#[cfg(test)]
-8
View File
@@ -210,14 +210,6 @@ impl LlmProvider for RetryProvider {
self.inner.set_model(model)
}
fn seed_response_chain(&self, thread_id: &str, response_id: String) {
self.inner.seed_response_chain(thread_id, response_id)
}
fn get_response_chain_id(&self, thread_id: &str) -> Option<String> {
self.inner.get_response_chain_id(thread_id)
}
fn calculate_cost(&self, input_tokens: u32, output_tokens: u32) -> Decimal {
self.inner.calculate_cost(input_tokens, output_tokens)
}
-2
View File
@@ -445,7 +445,6 @@ where
input_tokens: saturate_u32(response.usage.input_tokens),
output_tokens: saturate_u32(response.usage.output_tokens),
finish_reason: finish,
response_id: None,
})
}
@@ -511,7 +510,6 @@ where
input_tokens: saturate_u32(response.usage.input_tokens),
output_tokens: saturate_u32(response.usage.output_tokens),
finish_reason: finish,
response_id: None,
})
}
+139 -52
View File
@@ -217,38 +217,43 @@ impl SessionManager {
self.initiate_login().await
}
/// Start the OAuth login flow.
/// Start the login flow.
///
/// 1. Bind the fixed callback port
/// Shows the auth method menu FIRST (before binding any listener), so
/// that the API-key path can skip network binding entirely. This is
/// important for remote/headless servers where `127.0.0.1` is
/// unreachable from the user's browser.
///
/// For OAuth paths (GitHub, Google):
/// 1. Bind the callback listener
/// 2. Print the auth URL and attempt to open browser
/// 3. Wait for OAuth callback with session token
/// 4. Save and return the token
///
/// For NEAR AI Cloud API key:
/// 1. Prompt user for API key from cloud.near.ai
/// 2. Set NEARAI_API_KEY env var and save to bootstrap .env
/// 3. No session token saved (different auth model)
async fn initiate_login(&self) -> Result<(), LlmError> {
use crate::cli::oauth_defaults::{self, OAUTH_CALLBACK_PORT};
use crate::cli::oauth_defaults;
let listener = oauth_defaults::bind_callback_listener()
.await
.map_err(|e| LlmError::SessionRenewalFailed {
provider: "nearai".to_string(),
reason: e.to_string(),
})?;
let cb_url = oauth_defaults::callback_url();
let callback_url = format!("http://127.0.0.1:{}", OAUTH_CALLBACK_PORT);
// Show auth provider menu
// Show auth provider menu BEFORE binding the listener
println!();
println!("╔════════════════════════════════════════════════════════════════╗");
println!("║ NEAR AI Authentication ║");
println!("╠════════════════════════════════════════════════════════════════╣");
println!("║ Choose an authentication method: ║");
println!("║ ║");
println!("║ [1] GitHub ");
println!("║ [2] Google ");
println!("║ [1] GitHub (requires localhost browser access)");
println!("║ [2] Google (requires localhost browser access)");
println!("║ [3] NEAR Wallet (coming soon) ║");
println!("║ [4] NEAR AI Cloud API key ║");
println!("║ ║");
println!("╚════════════════════════════════════════════════════════════════╝");
println!();
print!("Enter choice [1-3]: ");
print!("Enter choice [1-4]: ");
// Flush stdout to ensure prompt is displayed
use std::io::Write;
@@ -263,23 +268,8 @@ impl SessionManager {
reason: format!("Failed to read input: {}", e),
})?;
let (auth_provider, auth_url) = match choice.trim() {
"1" | "" => {
let url = format!(
"{}/v1/auth/github?frontend_callback={}",
self.config.auth_base_url,
urlencoding::encode(&callback_url)
);
("github", url)
}
"2" => {
let url = format!(
"{}/v1/auth/google?frontend_callback={}",
self.config.auth_base_url,
urlencoding::encode(&callback_url)
);
("google", url)
}
match choice.trim() {
"4" => return self.api_key_login().await,
"3" => {
println!();
println!("NEAR Wallet authentication is not yet implemented.");
@@ -289,12 +279,41 @@ impl SessionManager {
reason: "NEAR Wallet auth not yet implemented".to_string(),
});
}
_ => {
"1" | "" | "2" => {} // handled below after listener bind
other => {
return Err(LlmError::SessionRenewalFailed {
provider: "nearai".to_string(),
reason: format!("Invalid choice: {}", choice.trim()),
reason: format!("Invalid choice: {}", other),
});
}
}
// OAuth paths: bind the callback listener now
let listener = oauth_defaults::bind_callback_listener()
.await
.map_err(|e| LlmError::SessionRenewalFailed {
provider: "nearai".to_string(),
reason: e.to_string(),
})?;
let (auth_provider, auth_url) = match choice.trim() {
"2" => {
let url = format!(
"{}/v1/auth/google?frontend_callback={}",
self.config.auth_base_url,
urlencoding::encode(&cb_url)
);
("google", url)
}
_ => {
// "1" or "" (default)
let url = format!(
"{}/v1/auth/github?frontend_callback={}",
self.config.auth_base_url,
urlencoding::encode(&cb_url)
);
("github", url)
}
};
println!();
@@ -341,6 +360,63 @@ impl SessionManager {
Ok(())
}
/// NEAR AI Cloud API key entry flow.
///
/// Prompts the user to enter a NEAR AI Cloud API key from
/// cloud.near.ai. The key is set as `NEARAI_API_KEY` env var so
/// `LlmConfig::resolve()` auto-selects ChatCompletions mode, and
/// saved to `~/.ironclaw/.env` for persistence across restarts.
/// No session token is saved and no `/v1/users/me` validation is
/// performed (different auth model).
async fn api_key_login(&self) -> Result<(), LlmError> {
println!();
println!("NEAR AI Cloud API key");
println!("─────────────────────");
println!();
println!(" 1. Open https://cloud.near.ai in your browser");
println!(" 2. Sign in and navigate to API Keys");
println!(" 3. Create or copy an existing API key");
println!();
let key_secret =
crate::setup::secret_input("API key").map_err(|e| LlmError::SessionRenewalFailed {
provider: "nearai".to_string(),
reason: format!("Failed to read input: {}", e),
})?;
use secrecy::ExposeSecret;
let key = key_secret.expose_secret().to_string();
if key.is_empty() {
return Err(LlmError::SessionRenewalFailed {
provider: "nearai".to_string(),
reason: "API key cannot be empty".to_string(),
});
}
// Set env var so Config picks it up immediately
// (LlmConfig::resolve() auto-selects ChatCompletions mode when
// NEARAI_API_KEY is present).
//
// SAFETY: called during single-threaded interactive login flow.
#[allow(unused_unsafe)]
unsafe {
std::env::set_var("NEARAI_API_KEY", &key);
}
// Persist to ~/.ironclaw/.env so the key survives restarts
// (bootstrap layer — available before DB is connected).
// Uses upsert to avoid clobbering existing bootstrap vars.
if let Err(e) = crate::bootstrap::upsert_bootstrap_var("NEARAI_API_KEY", &key) {
tracing::warn!("Failed to save API key to bootstrap .env: {}", e);
}
println!();
crate::setup::print_success("NEAR AI Cloud API key saved.");
println!();
Ok(())
}
/// Save session data to disk and (if available) to the database.
async fn save_session(&self, token: &str, auth_provider: Option<&str>) -> Result<(), LlmError> {
let session = SessionData {
@@ -437,20 +513,30 @@ impl SessionManager {
})? {
value
} else {
tracing::warn!(
"nearai.session_token missing; falling back to legacy nearai.session for backwards compatibility"
);
store
// Try the legacy key. Only warn if it actually exists (real
// backwards-compat migration). When neither key is present
// (fresh install), just return the "No session in DB" error.
let legacy = store
.get_setting(&user_id, "nearai.session")
.await
.map_err(|e| LlmError::SessionRenewalFailed {
provider: "nearai".to_string(),
reason: format!("DB query failed: {}", e),
})?
.ok_or(LlmError::SessionRenewalFailed {
provider: "nearai".to_string(),
reason: "No session in DB".to_string(),
})?
})?;
match legacy {
Some(value) => {
tracing::warn!(
"nearai.session_token missing; falling back to legacy nearai.session for backwards compatibility"
);
value
}
None => {
return Err(LlmError::SessionRenewalFailed {
provider: "nearai".to_string(),
reason: "No session in DB".to_string(),
});
}
}
};
let session: SessionData =
@@ -508,20 +594,21 @@ impl SessionManager {
}
}
/// Create a session manager from a config, migrating from env var if present.
/// Create a session manager from a config, loading env var if present.
///
/// When `NEARAI_SESSION_TOKEN` is set, it takes precedence over file-based
/// tokens. This supports hosting providers that inject the token via env var.
pub async fn create_session_manager(config: SessionConfig) -> Arc<SessionManager> {
let manager = SessionManager::new_async(config).await;
// Check for legacy env var and migrate if present and no file token
if !manager.has_token().await
&& let Ok(token) = std::env::var("NEARAI_SESSION_TOKEN")
// NEARAI_SESSION_TOKEN env var always takes precedence over file-based
// tokens. Hosting providers set this env var and expect it to be used
// directly — no file persistence needed.
if let Ok(token) = std::env::var("NEARAI_SESSION_TOKEN")
&& !token.is_empty()
{
tracing::info!("Migrating session token from NEARAI_SESSION_TOKEN env var to file");
manager.set_token(SecretString::from(token.clone())).await;
if let Err(e) = manager.save_session(&token, None).await {
tracing::warn!("Failed to save migrated session: {}", e);
}
tracing::info!("Using session token from NEARAI_SESSION_TOKEN env var");
manager.set_token(SecretString::from(token)).await;
}
Arc::new(manager)
+28 -31
View File
@@ -3,7 +3,7 @@
use std::sync::Arc;
use clap::Parser;
use tracing_subscriber::{EnvFilter, layer::SubscriberExt, util::SubscriberInitExt};
use tracing_subscriber::EnvFilter;
use ironclaw::{
agent::{Agent, AgentDeps, SessionManager},
@@ -14,7 +14,7 @@ use ironclaw::{
RegisteredEndpoint, SharedWasmChannel, WasmChannelLoader, WasmChannelRouter,
WasmChannelRuntime, WasmChannelRuntimeConfig, create_wasm_channel_router,
},
web::log_layer::{LogBroadcaster, WebLogLayer},
web::log_layer::LogBroadcaster,
},
cli::{
Cli, Command, run_mcp_command, run_pairing_command, run_service_command,
@@ -80,6 +80,15 @@ async fn main() -> anyhow::Result<()> {
return ironclaw::cli::run_config_command(config_cmd.clone()).await;
}
Some(Command::Registry(registry_cmd)) => {
tracing_subscriber::fmt()
.with_env_filter(
EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new("warn")),
)
.init();
return ironclaw::cli::run_registry_command(registry_cmd.clone()).await;
}
Some(Command::Mcp(mcp_cmd)) => {
// Simple logging for MCP commands
tracing_subscriber::fmt()
@@ -192,6 +201,9 @@ async fn main() -> anyhow::Result<()> {
)
.init();
let _ = dotenvy::dotenv();
ironclaw::bootstrap::load_ironclaw_env();
return ironclaw::cli::run_doctor_command().await;
}
Some(Command::Status) => {
@@ -201,6 +213,9 @@ async fn main() -> anyhow::Result<()> {
)
.init();
let _ = dotenvy::dotenv();
ironclaw::bootstrap::load_ironclaw_env();
return run_status_command().await;
}
Some(Command::Worker {
@@ -351,23 +366,14 @@ async fn main() -> anyhow::Result<()> {
};
let session = create_session_manager(session_config).await;
// Initialize tracing
let env_filter = EnvFilter::try_from_default_env()
.unwrap_or_else(|_| EnvFilter::new("ironclaw=info,tower_http=warn"));
// Create log broadcaster before tracing init so the WebLogLayer can capture all events.
// This gets wired to the gateway's /api/logs/events SSE endpoint later.
let log_broadcaster = Arc::new(LogBroadcaster::new());
tracing_subscriber::registry()
.with(env_filter)
.with(
tracing_subscriber::fmt::layer()
.with_target(false)
.with_writer(ironclaw::tracing_fmt::TruncatingStderr::default()),
)
.with(WebLogLayer::new(Arc::clone(&log_broadcaster)))
.init();
// Initialize tracing with a reloadable EnvFilter so the gateway can switch
// log levels (e.g. ironclaw=debug) at runtime without restarting.
let log_level_handle =
ironclaw::channels::web::log_layer::init_tracing(Arc::clone(&log_broadcaster));
// Create CLI channel
let repl_channel = if let Some(ref msg) = cli.message {
@@ -721,7 +727,6 @@ async fn main() -> anyhow::Result<()> {
// Initialize tool registry
let tools = Arc::new(ToolRegistry::new());
tools.register_builtin_tools();
tracing::info!("Registered {} built-in tools", tools.count());
// Create embeddings provider if configured
let embeddings: Option<Arc<dyn EmbeddingProvider>> = if config.embeddings.enabled {
@@ -1015,11 +1020,12 @@ async fn main() -> anyhow::Result<()> {
// Set up orchestrator for sandboxed job execution
// When allow_local_tools is false (default), the LLM uses create_job for FS/shell work.
// When allow_local_tools is true, dev tools are also registered directly (current behavior).
if config.agent.allow_local_tools {
// register_builder_tool() already calls register_dev_tools() internally,
// so only register them here when the builder didn't already do it.
let builder_registered_dev_tools =
config.builder.enabled && (config.agent.allow_local_tools || !config.sandbox.enabled);
if config.agent.allow_local_tools && !builder_registered_dev_tools {
tools.register_dev_tools();
tracing::info!(
"Local tools enabled (allow_local_tools=true), dev tools registered directly"
);
}
// Shared state for job events (used by both orchestrator and web gateway)
@@ -1070,7 +1076,6 @@ async fn main() -> anyhow::Result<()> {
}
});
tracing::info!("Orchestrator API started on :50051, sandbox delegation enabled");
if config.claude_code.enabled {
tracing::info!(
"Claude Code sandbox mode available (model: {}, max_turns: {})",
@@ -1324,9 +1329,6 @@ async fn main() -> anyhow::Result<()> {
// Seed workspace with core identity files on first boot
if let Some(ref ws) = workspace {
match ws.seed_if_empty().await {
Ok(count) if count > 0 => {
tracing::info!("Workspace seeded with {} core files", count);
}
Ok(_) => {}
Err(e) => {
tracing::warn!("Failed to seed workspace: {}", e);
@@ -1417,6 +1419,7 @@ async fn main() -> anyhow::Result<()> {
}
gw = gw.with_session_manager(Arc::clone(&session_manager));
gw = gw.with_log_broadcaster(Arc::clone(&log_broadcaster));
gw = gw.with_log_level_handle(Arc::clone(&log_level_handle));
gw = gw.with_tool_registry(Arc::clone(&tools));
if let Some(ref ext_mgr) = extension_manager {
gw = gw.with_extension_manager(Arc::clone(ext_mgr));
@@ -1455,11 +1458,6 @@ async fn main() -> anyhow::Result<()> {
gw.auth_token()
));
tracing::info!(
"Web gateway enabled on {}:{}",
gw_config.host,
gw_config.port
);
tracing::info!("Web UI: http://{}:{}/", gw_config.host, gw_config.port);
channel_names.push("gateway".to_string());
@@ -1496,13 +1494,12 @@ async fn main() -> anyhow::Result<()> {
deps,
channels,
Some(config.heartbeat.clone()),
Some(config.hygiene.clone()),
Some(config.routines.clone()),
Some(context_manager),
Some(session_manager),
);
tracing::info!("Agent initialized, starting main loop...");
// Print boot screen for interactive CLI mode (not single-message mode).
if config.channels.cli.enabled && cli.message.is_none() {
let boot_info = ironclaw::boot_screen::BootInfo {
+580
View File
@@ -0,0 +1,580 @@
//! Registry catalog: loads manifests from disk, provides list/search/resolve operations.
use std::collections::HashMap;
use std::path::{Path, PathBuf};
use crate::registry::manifest::{BundleDefinition, BundlesFile, ExtensionManifest, ManifestKind};
/// Error type for registry operations.
#[derive(Debug, thiserror::Error)]
pub enum RegistryError {
#[error("Registry directory not found: {0}")]
DirectoryNotFound(PathBuf),
#[error("Failed to read manifest {path}: {reason}")]
ManifestRead { path: PathBuf, reason: String },
#[error("Failed to parse manifest {path}: {reason}")]
ManifestParse { path: PathBuf, reason: String },
#[error("Extension not found: {0}")]
ExtensionNotFound(String),
#[error("'{name}' already installed at {path}. Use --force to overwrite.")]
AlreadyInstalled {
name: String,
path: std::path::PathBuf,
},
#[error("Download failed for {url}: {reason}")]
DownloadFailed { url: String, reason: String },
#[error(
"Ambiguous name '{name}': exists as both {kind_a} and {kind_b}. Use '{prefix_a}/{name}' or '{prefix_b}/{name}'."
)]
AmbiguousName {
name: String,
kind_a: &'static str,
prefix_a: &'static str,
kind_b: &'static str,
prefix_b: &'static str,
},
#[error("Bundle not found: {0}")]
BundleNotFound(String),
#[error("Failed to read bundles file: {0}")]
BundlesRead(String),
#[error("IO error: {0}")]
Io(#[from] std::io::Error),
}
/// Central catalog loaded from the `registry/` directory.
#[derive(Debug, Clone)]
pub struct RegistryCatalog {
/// All loaded manifests, keyed by "<kind>/<name>" (e.g. "tools/slack").
manifests: HashMap<String, ExtensionManifest>,
/// Bundle definitions from `_bundles.json`.
bundles: HashMap<String, BundleDefinition>,
/// Root directory of the registry.
root: PathBuf,
}
impl RegistryCatalog {
/// Load the catalog from a registry directory.
///
/// Expects the structure:
/// ```text
/// registry/
/// ├── tools/*.json
/// ├── channels/*.json
/// └── _bundles.json
/// ```
pub fn load(registry_dir: &Path) -> Result<Self, RegistryError> {
if !registry_dir.exists() {
return Err(RegistryError::DirectoryNotFound(registry_dir.to_path_buf()));
}
let mut manifests = HashMap::new();
// Load tools
let tools_dir = registry_dir.join("tools");
if tools_dir.is_dir() {
Self::load_manifests_from_dir(&tools_dir, "tools", &mut manifests)?;
}
// Load channels
let channels_dir = registry_dir.join("channels");
if channels_dir.is_dir() {
Self::load_manifests_from_dir(&channels_dir, "channels", &mut manifests)?;
}
// Load bundles
let bundles_path = registry_dir.join("_bundles.json");
let bundles = if bundles_path.is_file() {
let content = std::fs::read_to_string(&bundles_path).map_err(|e| {
RegistryError::BundlesRead(format!("{}: {}", bundles_path.display(), e))
})?;
let bundles_file: BundlesFile = serde_json::from_str(&content).map_err(|e| {
RegistryError::BundlesRead(format!("{}: {}", bundles_path.display(), e))
})?;
bundles_file.bundles
} else {
HashMap::new()
};
Ok(Self {
manifests,
bundles,
root: registry_dir.to_path_buf(),
})
}
fn load_manifests_from_dir(
dir: &Path,
kind_prefix: &str,
manifests: &mut HashMap<String, ExtensionManifest>,
) -> Result<(), RegistryError> {
let entries = std::fs::read_dir(dir).map_err(|e| RegistryError::ManifestRead {
path: dir.to_path_buf(),
reason: e.to_string(),
})?;
for entry in entries {
let entry = entry.map_err(|e| RegistryError::ManifestRead {
path: dir.to_path_buf(),
reason: e.to_string(),
})?;
let path = entry.path();
if !path.is_file() || path.extension().and_then(|e| e.to_str()) != Some("json") {
continue;
}
let content =
std::fs::read_to_string(&path).map_err(|e| RegistryError::ManifestRead {
path: path.clone(),
reason: e.to_string(),
})?;
let manifest: ExtensionManifest =
serde_json::from_str(&content).map_err(|e| RegistryError::ManifestParse {
path: path.clone(),
reason: e.to_string(),
})?;
let key = format!("{}/{}", kind_prefix, manifest.name);
manifests.insert(key, manifest);
}
Ok(())
}
/// The root directory this catalog was loaded from.
pub fn root(&self) -> &Path {
&self.root
}
/// Get all manifests.
pub fn all(&self) -> Vec<&ExtensionManifest> {
let mut items: Vec<_> = self.manifests.values().collect();
items.sort_by(|a, b| a.name.cmp(&b.name));
items
}
/// List manifests, optionally filtered by kind and/or tag.
pub fn list(&self, kind: Option<ManifestKind>, tag: Option<&str>) -> Vec<&ExtensionManifest> {
let mut results: Vec<_> = self
.manifests
.values()
.filter(|m| kind.is_none_or(|k| m.kind == k))
.filter(|m| tag.is_none_or(|t| m.tags.iter().any(|mt| mt == t)))
.collect();
results.sort_by(|a, b| a.name.cmp(&b.name));
results
}
/// Get a manifest by name. Tries exact key match first ("tools/slack"),
/// then searches by bare name ("slack").
///
/// If a bare name matches both a tool and a channel, returns `None`.
/// Use a qualified key ("tools/slack" or "channels/slack") to disambiguate.
pub fn get(&self, name: &str) -> Option<&ExtensionManifest> {
// Try exact key first
if let Some(m) = self.manifests.get(name) {
return Some(m);
}
// Try with kind prefix, detecting collisions
let tool = self.manifests.get(&format!("tools/{}", name));
let channel = self.manifests.get(&format!("channels/{}", name));
match (tool, channel) {
(Some(_), Some(_)) => None, // ambiguous
(Some(m), None) => Some(m),
(None, Some(m)) => Some(m),
(None, None) => None,
}
}
/// Get a manifest by name, returning a `Result` with an explicit error for
/// ambiguous bare names.
pub fn get_strict(&self, name: &str) -> Result<&ExtensionManifest, RegistryError> {
// Try exact key first
if let Some(m) = self.manifests.get(name) {
return Ok(m);
}
let has_tool = self.manifests.contains_key(&format!("tools/{}", name));
let has_channel = self.manifests.contains_key(&format!("channels/{}", name));
match (has_tool, has_channel) {
(true, true) => Err(RegistryError::AmbiguousName {
name: name.to_string(),
kind_a: "tool",
prefix_a: "tools",
kind_b: "channel",
prefix_b: "channels",
}),
(true, false) => Ok(self.manifests.get(&format!("tools/{}", name)).unwrap()),
(false, true) => Ok(self.manifests.get(&format!("channels/{}", name)).unwrap()),
(false, false) => Err(RegistryError::ExtensionNotFound(name.to_string())),
}
}
/// Get the full key ("tools/slack" or "channels/telegram") for a manifest.
pub fn key_for(&self, name: &str) -> Option<String> {
if self.manifests.contains_key(name) {
return Some(name.to_string());
}
let has_tool = self.manifests.contains_key(&format!("tools/{}", name));
let has_channel = self.manifests.contains_key(&format!("channels/{}", name));
match (has_tool, has_channel) {
(true, true) => None, // ambiguous
(true, false) => Some(format!("tools/{}", name)),
(false, true) => Some(format!("channels/{}", name)),
(false, false) => None,
}
}
/// Search manifests by query string (matches name, display_name, description, keywords).
pub fn search(&self, query: &str) -> Vec<&ExtensionManifest> {
let query_lower = query.to_lowercase();
let tokens: Vec<&str> = query_lower.split_whitespace().collect();
let mut scored: Vec<(&ExtensionManifest, usize)> = self
.manifests
.values()
.filter_map(|m| {
let score = Self::score_manifest(m, &tokens);
if score > 0 { Some((m, score)) } else { None }
})
.collect();
scored.sort_by(|a, b| b.1.cmp(&a.1).then(a.0.name.cmp(&b.0.name)));
scored.into_iter().map(|(m, _)| m).collect()
}
fn score_manifest(manifest: &ExtensionManifest, tokens: &[&str]) -> usize {
let mut score = 0;
let name_lower = manifest.name.to_lowercase();
let display_lower = manifest.display_name.to_lowercase();
let desc_lower = manifest.description.to_lowercase();
for token in tokens {
if name_lower == *token {
score += 10;
} else if name_lower.contains(token) {
score += 5;
}
if display_lower == *token {
score += 8;
} else if display_lower.contains(token) {
score += 4;
}
if desc_lower.contains(token) {
score += 2;
}
for kw in &manifest.keywords {
if kw.to_lowercase() == *token {
score += 6;
} else if kw.to_lowercase().contains(token) {
score += 3;
}
}
for tag in &manifest.tags {
if tag.to_lowercase() == *token {
score += 4;
}
}
}
score
}
/// Get a bundle definition by name.
pub fn get_bundle(&self, name: &str) -> Option<&BundleDefinition> {
self.bundles.get(name)
}
/// List all bundle names.
pub fn bundle_names(&self) -> Vec<&str> {
let mut names: Vec<_> = self.bundles.keys().map(|s| s.as_str()).collect();
names.sort();
names
}
/// Resolve a bundle into its constituent manifests.
/// Returns the manifests and any extension keys that couldn't be found.
pub fn resolve_bundle(
&self,
bundle_name: &str,
) -> Result<(Vec<&ExtensionManifest>, Vec<String>), RegistryError> {
let bundle = self
.bundles
.get(bundle_name)
.ok_or_else(|| RegistryError::BundleNotFound(bundle_name.to_string()))?;
let mut found = Vec::new();
let mut missing = Vec::new();
for ext_key in &bundle.extensions {
if let Some(manifest) = self.manifests.get(ext_key) {
found.push(manifest);
} else {
missing.push(ext_key.clone());
}
}
Ok((found, missing))
}
/// Check if a name refers to a bundle rather than an individual extension.
pub fn is_bundle(&self, name: &str) -> bool {
self.bundles.contains_key(name)
}
/// Resolve a name to either a single manifest or the manifests in a bundle.
/// Returns (manifests, bundle_definition_if_bundle).
pub fn resolve(
&self,
name: &str,
) -> Result<(Vec<&ExtensionManifest>, Option<&BundleDefinition>), RegistryError> {
// Check bundle first
if let Some(bundle) = self.bundles.get(name) {
let (manifests, missing) = self.resolve_bundle(name)?;
if !missing.is_empty() {
tracing::warn!(
"Bundle '{}' references missing extensions: {:?}",
name,
missing
);
}
return Ok((manifests, Some(bundle)));
}
// Single extension (use get_strict to catch ambiguous bare names)
let manifest = self.get_strict(name)?;
Ok((vec![manifest], None))
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::fs;
fn create_test_registry(dir: &Path) {
let tools_dir = dir.join("tools");
let channels_dir = dir.join("channels");
fs::create_dir_all(&tools_dir).unwrap();
fs::create_dir_all(&channels_dir).unwrap();
fs::write(
tools_dir.join("slack.json"),
r#"{
"name": "slack",
"display_name": "Slack",
"kind": "tool",
"version": "0.1.0",
"description": "Post messages via Slack API",
"keywords": ["messaging", "chat"],
"source": {
"dir": "tools-src/slack",
"capabilities": "slack-tool.capabilities.json",
"crate_name": "slack-tool"
},
"auth_summary": {
"method": "oauth",
"provider": "Slack",
"secrets": ["slack_bot_token"]
},
"tags": ["default", "messaging"]
}"#,
)
.unwrap();
fs::write(
tools_dir.join("github.json"),
r#"{
"name": "github",
"display_name": "GitHub",
"kind": "tool",
"version": "0.1.0",
"description": "GitHub integration for issues and PRs",
"keywords": ["code", "git"],
"source": {
"dir": "tools-src/github",
"capabilities": "github-tool.capabilities.json",
"crate_name": "github-tool"
},
"tags": ["default", "development"]
}"#,
)
.unwrap();
fs::write(
channels_dir.join("telegram.json"),
r#"{
"name": "telegram",
"display_name": "Telegram",
"kind": "channel",
"version": "0.1.0",
"description": "Telegram Bot API channel",
"source": {
"dir": "channels-src/telegram",
"capabilities": "telegram.capabilities.json",
"crate_name": "telegram-channel"
},
"tags": ["messaging"]
}"#,
)
.unwrap();
fs::write(
dir.join("_bundles.json"),
r#"{
"bundles": {
"default": {
"display_name": "Recommended",
"extensions": ["tools/slack", "tools/github", "channels/telegram"]
},
"messaging": {
"display_name": "Messaging",
"extensions": ["tools/slack", "channels/telegram"],
"shared_auth": null
}
}
}"#,
)
.unwrap();
}
#[test]
fn test_load_catalog() {
let tmp = tempfile::tempdir().unwrap();
create_test_registry(tmp.path());
let catalog = RegistryCatalog::load(tmp.path()).unwrap();
assert_eq!(catalog.all().len(), 3);
}
#[test]
fn test_list_by_kind() {
let tmp = tempfile::tempdir().unwrap();
create_test_registry(tmp.path());
let catalog = RegistryCatalog::load(tmp.path()).unwrap();
let tools = catalog.list(Some(ManifestKind::Tool), None);
assert_eq!(tools.len(), 2);
let channels = catalog.list(Some(ManifestKind::Channel), None);
assert_eq!(channels.len(), 1);
}
#[test]
fn test_list_by_tag() {
let tmp = tempfile::tempdir().unwrap();
create_test_registry(tmp.path());
let catalog = RegistryCatalog::load(tmp.path()).unwrap();
let defaults = catalog.list(None, Some("default"));
assert_eq!(defaults.len(), 2);
let messaging = catalog.list(None, Some("messaging"));
assert_eq!(messaging.len(), 2); // slack (tool) and telegram (channel) both have "messaging" tag
}
#[test]
fn test_get_by_name() {
let tmp = tempfile::tempdir().unwrap();
create_test_registry(tmp.path());
let catalog = RegistryCatalog::load(tmp.path()).unwrap();
// Full key
assert!(catalog.get("tools/slack").is_some());
// Bare name
assert!(catalog.get("slack").is_some());
assert!(catalog.get("telegram").is_some());
// Missing
assert!(catalog.get("nonexistent").is_none());
}
#[test]
fn test_search() {
let tmp = tempfile::tempdir().unwrap();
create_test_registry(tmp.path());
let catalog = RegistryCatalog::load(tmp.path()).unwrap();
let results = catalog.search("slack");
assert_eq!(results.len(), 1);
assert_eq!(results[0].name, "slack");
let results = catalog.search("messaging");
assert!(!results.is_empty());
let results = catalog.search("nonexistent query");
assert!(results.is_empty());
}
#[test]
fn test_resolve_bundle() {
let tmp = tempfile::tempdir().unwrap();
create_test_registry(tmp.path());
let catalog = RegistryCatalog::load(tmp.path()).unwrap();
let (manifests, missing) = catalog.resolve_bundle("default").unwrap();
assert_eq!(manifests.len(), 3);
assert!(missing.is_empty());
assert!(catalog.resolve_bundle("nonexistent").is_err());
}
#[test]
fn test_resolve_single_or_bundle() {
let tmp = tempfile::tempdir().unwrap();
create_test_registry(tmp.path());
let catalog = RegistryCatalog::load(tmp.path()).unwrap();
// Single extension
let (manifests, bundle) = catalog.resolve("slack").unwrap();
assert_eq!(manifests.len(), 1);
assert!(bundle.is_none());
// Bundle
let (manifests, bundle) = catalog.resolve("default").unwrap();
assert_eq!(manifests.len(), 3);
assert!(bundle.is_some());
}
#[test]
fn test_bundle_names() {
let tmp = tempfile::tempdir().unwrap();
create_test_registry(tmp.path());
let catalog = RegistryCatalog::load(tmp.path()).unwrap();
let names = catalog.bundle_names();
assert_eq!(names, vec!["default", "messaging"]);
}
#[test]
fn test_directory_not_found() {
let result = RegistryCatalog::load(Path::new("/nonexistent/path"));
assert!(result.is_err());
}
}
+415
View File
@@ -0,0 +1,415 @@
//! Install extensions from the registry: build-from-source or download pre-built artifacts.
use std::path::{Path, PathBuf};
use tokio::fs;
use crate::registry::catalog::RegistryError;
use crate::registry::manifest::{BundleDefinition, ExtensionManifest, ManifestKind};
/// Result of installing a single extension from the registry.
#[derive(Debug)]
pub struct InstallOutcome {
/// Extension name.
pub name: String,
/// Whether this is a tool or channel.
pub kind: ManifestKind,
/// Destination path of the installed WASM binary.
pub wasm_path: PathBuf,
/// Whether a capabilities file was also installed.
pub has_capabilities: bool,
/// Any warning messages.
pub warnings: Vec<String>,
}
/// Handles installing extensions from registry manifests.
pub struct RegistryInstaller {
/// Root of the repo (parent of `registry/`), used to resolve `source.dir`.
repo_root: PathBuf,
/// Directory for installed tools (`~/.ironclaw/tools/`).
tools_dir: PathBuf,
/// Directory for installed channels (`~/.ironclaw/channels/`).
channels_dir: PathBuf,
}
impl RegistryInstaller {
pub fn new(repo_root: PathBuf, tools_dir: PathBuf, channels_dir: PathBuf) -> Self {
Self {
repo_root,
tools_dir,
channels_dir,
}
}
/// Default installer using standard paths.
pub fn with_defaults(repo_root: PathBuf) -> Self {
let home = dirs::home_dir().unwrap_or_else(|| PathBuf::from("."));
Self {
repo_root,
tools_dir: home.join(".ironclaw").join("tools"),
channels_dir: home.join(".ironclaw").join("channels"),
}
}
/// Install a single extension by building from source.
pub async fn install_from_source(
&self,
manifest: &ExtensionManifest,
force: bool,
) -> Result<InstallOutcome, RegistryError> {
let source_dir = self.repo_root.join(&manifest.source.dir);
if !source_dir.exists() {
return Err(RegistryError::ManifestRead {
path: source_dir.clone(),
reason: "source directory does not exist".to_string(),
});
}
let target_dir = match manifest.kind {
ManifestKind::Tool => &self.tools_dir,
ManifestKind::Channel => &self.channels_dir,
};
fs::create_dir_all(target_dir)
.await
.map_err(RegistryError::Io)?;
// Use manifest.name for installed filenames so discovery, auth, and
// CLI commands (`ironclaw tool auth <name>`) all agree on the stem.
let target_wasm = target_dir.join(format!("{}.wasm", manifest.name));
// Check if already exists
if target_wasm.exists() && !force {
return Err(RegistryError::AlreadyInstalled {
name: manifest.name.clone(),
path: target_wasm,
});
}
// Build the WASM component
println!(
"Building {} '{}' from {}...",
manifest.kind,
manifest.display_name,
source_dir.display()
);
let crate_name = &manifest.source.crate_name;
let wasm_path = build_wasm_component(&source_dir, crate_name)
.await
.map_err(|e| RegistryError::ManifestRead {
path: source_dir.clone(),
reason: format!("build failed: {}", e),
})?;
// Copy WASM binary
println!(" Installing to {}", target_wasm.display());
fs::copy(&wasm_path, &target_wasm)
.await
.map_err(RegistryError::Io)?;
// Copy capabilities file
let caps_source = source_dir.join(&manifest.source.capabilities);
let target_caps = target_dir.join(format!("{}.capabilities.json", manifest.name));
let has_capabilities = if caps_source.exists() {
fs::copy(&caps_source, &target_caps)
.await
.map_err(RegistryError::Io)?;
true
} else {
false
};
let mut warnings = Vec::new();
if !has_capabilities {
warnings.push(format!(
"No capabilities file found at {}",
caps_source.display()
));
}
Ok(InstallOutcome {
name: manifest.name.clone(),
kind: manifest.kind,
wasm_path: target_wasm,
has_capabilities,
warnings,
})
}
/// Download and install a pre-built artifact.
pub async fn install_from_artifact(
&self,
manifest: &ExtensionManifest,
force: bool,
) -> Result<InstallOutcome, RegistryError> {
let artifact = manifest.artifacts.get("wasm32-wasip2").ok_or_else(|| {
RegistryError::ExtensionNotFound(format!(
"No wasm32-wasip2 artifact for '{}'",
manifest.name
))
})?;
let url = artifact.url.as_ref().ok_or_else(|| {
RegistryError::ExtensionNotFound(format!(
"No artifact URL for '{}'. Use --build to build from source.",
manifest.name
))
})?;
let expected_sha = artifact.sha256.as_ref().ok_or_else(|| {
RegistryError::ExtensionNotFound(format!(
"No SHA256 hash for '{}'. Cannot verify download.",
manifest.name
))
})?;
let target_dir = match manifest.kind {
ManifestKind::Tool => &self.tools_dir,
ManifestKind::Channel => &self.channels_dir,
};
fs::create_dir_all(target_dir)
.await
.map_err(RegistryError::Io)?;
let target_wasm = target_dir.join(format!("{}.wasm", manifest.name));
if target_wasm.exists() && !force {
return Err(RegistryError::AlreadyInstalled {
name: manifest.name.clone(),
path: target_wasm,
});
}
// Download
println!(
"Downloading {} '{}'...",
manifest.kind, manifest.display_name
);
let response = reqwest::get(url)
.await
.map_err(|e| RegistryError::DownloadFailed {
url: url.clone(),
reason: format!("request failed: {}", e),
})?;
let response = response
.error_for_status()
.map_err(|e| RegistryError::DownloadFailed {
url: url.clone(),
reason: e.to_string(),
})?;
let bytes = response
.bytes()
.await
.map_err(|e| RegistryError::DownloadFailed {
url: url.clone(),
reason: format!("failed to read body: {}", e),
})?;
// Verify SHA256
use sha2::{Digest, Sha256};
let mut hasher = Sha256::new();
hasher.update(&bytes);
let actual_sha = format!("{:x}", hasher.finalize());
if actual_sha != *expected_sha {
return Err(RegistryError::DownloadFailed {
url: url.clone(),
reason: format!(
"SHA256 mismatch: expected {}, got {}",
expected_sha, actual_sha
),
});
}
// Write file
fs::write(&target_wasm, &bytes)
.await
.map_err(RegistryError::Io)?;
// Copy capabilities from source dir (still needed even for pre-built artifacts).
// NOTE: This requires the source tree to be present. When pre-built artifact
// distribution is implemented, capabilities should be bundled with the artifact
// or fetched from a separate URL.
let caps_source = self
.repo_root
.join(&manifest.source.dir)
.join(&manifest.source.capabilities);
let target_caps = target_dir.join(format!("{}.capabilities.json", manifest.name));
let has_capabilities = if caps_source.exists() {
fs::copy(&caps_source, &target_caps)
.await
.map_err(RegistryError::Io)?;
true
} else {
false
};
println!(" Installed to {}", target_wasm.display());
Ok(InstallOutcome {
name: manifest.name.clone(),
kind: manifest.kind,
wasm_path: target_wasm,
has_capabilities,
warnings: Vec::new(),
})
}
/// Install a single manifest, choosing build vs download based on artifact availability and flags.
pub async fn install(
&self,
manifest: &ExtensionManifest,
force: bool,
prefer_build: bool,
) -> Result<InstallOutcome, RegistryError> {
let has_artifact = manifest
.artifacts
.get("wasm32-wasip2")
.and_then(|a| a.url.as_ref())
.is_some();
if prefer_build || !has_artifact {
self.install_from_source(manifest, force).await
} else {
self.install_from_artifact(manifest, force).await
}
}
/// Install all extensions in a bundle.
/// Returns the outcomes and any shared auth hints.
pub async fn install_bundle(
&self,
manifests: &[&ExtensionManifest],
bundle: &BundleDefinition,
force: bool,
prefer_build: bool,
) -> (Vec<InstallOutcome>, Vec<String>) {
let mut outcomes = Vec::new();
let mut errors = Vec::new();
for manifest in manifests {
match self.install(manifest, force, prefer_build).await {
Ok(outcome) => outcomes.push(outcome),
Err(e) => errors.push(format!("{}: {}", manifest.name, e)),
}
}
// Collect auth hints
let mut auth_hints = Vec::new();
if let Some(shared) = &bundle.shared_auth {
auth_hints.push(format!(
"Bundle uses shared auth '{}'. Run `ironclaw tool auth <any-member>` to authenticate all members.",
shared
));
}
// Collect unique auth providers that need setup
let mut seen_providers = std::collections::HashSet::new();
for manifest in manifests {
if let Some(auth) = &manifest.auth_summary {
let key = auth
.shared_auth
.as_deref()
.unwrap_or(manifest.name.as_str());
if seen_providers.insert(key.to_string())
&& let Some(url) = &auth.setup_url
{
auth_hints.push(format!(
" {} ({}): {}",
auth.provider.as_deref().unwrap_or(&manifest.name),
auth.method.as_deref().unwrap_or("manual"),
url
));
}
}
}
if !errors.is_empty() {
auth_hints.push(format!(
"\nFailed to install {} extension(s):",
errors.len()
));
for err in errors {
auth_hints.push(format!(" - {}", err));
}
}
(outcomes, auth_hints)
}
}
/// Build a WASM component from a source directory using `cargo component build --release`.
///
/// Uses `tokio::process::Command` with inherited stdio so build progress is visible.
/// Looks for the specific `{crate_name}.wasm` in the release directory rather than
/// picking the first `.wasm` file found.
async fn build_wasm_component(source_dir: &Path, crate_name: &str) -> anyhow::Result<PathBuf> {
use tokio::process::Command;
// Check cargo-component availability
let check = Command::new("cargo")
.args(["component", "--version"])
.stdout(std::process::Stdio::null())
.stderr(std::process::Stdio::null())
.status()
.await;
if check.is_err() || !check.as_ref().map(|s| s.success()).unwrap_or(false) {
anyhow::bail!("cargo-component not found. Install with: cargo install cargo-component");
}
// Use status() with inherited stdio so build output streams to the terminal.
let status = Command::new("cargo")
.current_dir(source_dir)
.args(["component", "build", "--release"])
.status()
.await?;
if !status.success() {
anyhow::bail!("Build failed (exit code: {})", status);
}
// Look for the specific crate's WASM file (Cargo uses underscores in artifact names).
let wasm_filename = format!("{}.wasm", crate_name.replace('-', "_"));
let target_base = source_dir.join("target");
let candidates = [
"wasm32-wasip1",
"wasm32-wasip2",
"wasm32-wasi",
"wasm32-unknown-unknown",
];
for target in &candidates {
let wasm_path = target_base
.join(target)
.join("release")
.join(&wasm_filename);
if wasm_path.exists() {
return Ok(wasm_path);
}
}
anyhow::bail!(
"Could not find {} in {}/target/*/release/",
wasm_filename,
source_dir.display()
)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_installer_creation() {
let installer = RegistryInstaller::new(
PathBuf::from("/repo"),
PathBuf::from("/home/.ironclaw/tools"),
PathBuf::from("/home/.ironclaw/channels"),
);
assert_eq!(installer.repo_root, PathBuf::from("/repo"));
}
}
+271
View File
@@ -0,0 +1,271 @@
//! Serde structs for extension registry manifests.
//!
//! Each manifest describes a single extension (tool or channel) with its source
//! location, build artifacts, authentication requirements, and tags.
use serde::{Deserialize, Serialize};
use crate::extensions::{AuthHint, ExtensionKind, ExtensionSource, RegistryEntry};
/// A single extension manifest loaded from `registry/{tools,channels}/<name>.json`.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ExtensionManifest {
/// Unique identifier (matches crate name stem, e.g. "slack").
pub name: String,
/// Human-readable name (e.g. "Slack").
pub display_name: String,
/// Whether this is a tool or channel.
pub kind: ManifestKind,
/// Semver version from Cargo.toml.
pub version: String,
/// One-line description.
pub description: String,
/// Search keywords beyond the name.
#[serde(default)]
pub keywords: Vec<String>,
/// Source code location and build info.
pub source: SourceSpec,
/// Pre-built binary artifacts keyed by target triple.
#[serde(default)]
pub artifacts: std::collections::HashMap<String, ArtifactSpec>,
/// Summary of authentication requirements.
#[serde(default)]
pub auth_summary: Option<AuthSummary>,
/// Tags for filtering (e.g. "default", "messaging", "google").
#[serde(default)]
pub tags: Vec<String>,
}
/// Extension kind as declared in manifests.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
pub enum ManifestKind {
Tool,
Channel,
}
impl From<ManifestKind> for ExtensionKind {
fn from(kind: ManifestKind) -> Self {
match kind {
ManifestKind::Tool => ExtensionKind::WasmTool,
ManifestKind::Channel => ExtensionKind::WasmChannel,
}
}
}
impl std::fmt::Display for ManifestKind {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
ManifestKind::Tool => write!(f, "tool"),
ManifestKind::Channel => write!(f, "channel"),
}
}
}
/// Source code location for building from source.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct SourceSpec {
/// Path relative to repo root (e.g. "tools-src/slack").
pub dir: String,
/// Capabilities filename relative to source dir.
pub capabilities: String,
/// Rust crate name for `cargo component build`.
pub crate_name: String,
}
/// A pre-built binary artifact.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ArtifactSpec {
/// Download URL (null until release).
pub url: Option<String>,
/// Hex SHA256 of the WASM binary (null until release).
pub sha256: Option<String>,
}
/// Summary of authentication requirements extracted from capabilities.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct AuthSummary {
/// Auth method: "oauth", "manual", or "none".
#[serde(default)]
pub method: Option<String>,
/// Display name for the auth provider (e.g. "Google", "Slack").
#[serde(default)]
pub provider: Option<String>,
/// Secret names required by this extension.
#[serde(default)]
pub secrets: Vec<String>,
/// If this extension shares auth with others (e.g. all Google tools share
/// `google_oauth_token`), this is the shared secret name.
#[serde(default)]
pub shared_auth: Option<String>,
/// URL where users can set up credentials.
#[serde(default)]
pub setup_url: Option<String>,
}
/// Bundle definition grouping related extensions.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct BundleDefinition {
/// Human-readable name.
pub display_name: String,
/// Description of what this bundle contains.
#[serde(default)]
pub description: Option<String>,
/// Extension references as "tools/<name>" or "channels/<name>".
pub extensions: Vec<String>,
/// Shared auth secret across bundle members (if any).
#[serde(default)]
pub shared_auth: Option<String>,
}
/// Top-level structure of `_bundles.json`.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct BundlesFile {
pub bundles: std::collections::HashMap<String, BundleDefinition>,
}
impl ExtensionManifest {
/// Convert this manifest into a [`RegistryEntry`] for use with the in-chat
/// extension discovery system.
pub fn to_registry_entry(&self) -> RegistryEntry {
let source = ExtensionSource::WasmBuildable {
repo_url: self.source.dir.clone(),
build_dir: Some(self.source.dir.clone()),
};
let auth_hint = match self.auth_summary.as_ref().and_then(|a| a.method.as_deref()) {
Some("oauth") => AuthHint::CapabilitiesAuth,
Some("manual") => AuthHint::CapabilitiesAuth,
Some("none") | None => AuthHint::None,
Some(_) => AuthHint::CapabilitiesAuth,
};
RegistryEntry {
name: self.name.clone(),
display_name: self.display_name.clone(),
kind: self.kind.into(),
description: self.description.clone(),
keywords: self.keywords.clone(),
source,
auth_hint,
}
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_parse_tool_manifest() {
let json = r#"{
"name": "slack",
"display_name": "Slack",
"kind": "tool",
"version": "0.1.0",
"description": "Post messages via Slack API",
"keywords": ["messaging"],
"source": {
"dir": "tools-src/slack",
"capabilities": "slack-tool.capabilities.json",
"crate_name": "slack-tool"
},
"artifacts": {
"wasm32-wasip2": { "url": null, "sha256": null }
},
"auth_summary": {
"method": "oauth",
"provider": "Slack",
"secrets": ["slack_bot_token"],
"shared_auth": null,
"setup_url": "https://api.slack.com/apps"
},
"tags": ["default", "messaging"]
}"#;
let manifest: ExtensionManifest = serde_json::from_str(json).expect("parse manifest");
assert_eq!(manifest.name, "slack");
assert_eq!(manifest.kind, ManifestKind::Tool);
assert_eq!(manifest.version, "0.1.0");
assert!(manifest.tags.contains(&"default".to_string()));
let entry = manifest.to_registry_entry();
assert_eq!(entry.kind, ExtensionKind::WasmTool);
}
#[test]
fn test_parse_channel_manifest() {
let json = r#"{
"name": "telegram",
"display_name": "Telegram",
"kind": "channel",
"version": "0.1.0",
"description": "Telegram Bot API channel",
"source": {
"dir": "channels-src/telegram",
"capabilities": "telegram.capabilities.json",
"crate_name": "telegram-channel"
},
"tags": ["messaging"]
}"#;
let manifest: ExtensionManifest = serde_json::from_str(json).expect("parse manifest");
assert_eq!(manifest.kind, ManifestKind::Channel);
assert!(manifest.auth_summary.is_none());
assert!(manifest.artifacts.is_empty());
let entry = manifest.to_registry_entry();
assert_eq!(entry.kind, ExtensionKind::WasmChannel);
}
#[test]
fn test_parse_bundles() {
let json = r#"{
"bundles": {
"google": {
"display_name": "Google Suite",
"description": "All Google tools",
"extensions": ["tools/gmail", "tools/google-calendar"],
"shared_auth": "google_oauth_token"
},
"default": {
"display_name": "Recommended Set",
"extensions": ["tools/github", "tools/slack"]
}
}
}"#;
let bundles: BundlesFile = serde_json::from_str(json).expect("parse bundles");
assert_eq!(bundles.bundles.len(), 2);
assert_eq!(
bundles.bundles["google"].shared_auth.as_deref(),
Some("google_oauth_token")
);
assert!(bundles.bundles["default"].shared_auth.is_none());
}
#[test]
fn test_manifest_kind_display() {
assert_eq!(ManifestKind::Tool.to_string(), "tool");
assert_eq!(ManifestKind::Channel.to_string(), "channel");
}
}
+23
View File
@@ -0,0 +1,23 @@
//! Extension registry: metadata catalog for tools and channels.
//!
//! The registry provides a central index of all available extensions (WASM tools
//! and channels) with their source locations, build artifacts, authentication
//! requirements, and grouping via bundles.
//!
//! ```text
//! registry/
//! ├── tools/ <- One JSON manifest per tool
//! ├── channels/ <- One JSON manifest per channel
//! └── _bundles.json <- Bundle definitions (google, messaging, default)
//! ```
pub mod catalog;
pub mod installer;
pub mod manifest;
pub use catalog::{RegistryCatalog, RegistryError};
pub use installer::RegistryInstaller;
pub use manifest::{
ArtifactSpec, AuthSummary, BundleDefinition, BundlesFile, ExtensionManifest, ManifestKind,
SourceSpec,
};
+103
View File
@@ -1229,4 +1229,107 @@ mod tests {
assert_eq!(s.tunnel.cf_token, Some("cf_tok_xyz".to_string()));
assert!(s.tunnel.ts_funnel);
}
/// Simulates the wizard recovery scenario:
///
/// 1. A prior partial run saved steps 1-4 to the DB
/// 2. User re-runs the wizard, Step 1 sets a new database_url
/// 3. Prior settings are loaded from the DB
/// 4. Step 1's fresh choices must win over stale DB values
///
/// This tests the ordering: load DB → merge_from(step1_overrides).
#[test]
fn wizard_recovery_step1_overrides_stale_db() {
// Simulate prior partial run (steps 1-4 completed):
let prior_run = Settings {
database_backend: Some("postgres".to_string()),
database_url: Some("postgres://old-host/ironclaw".to_string()),
llm_backend: Some("anthropic".to_string()),
selected_model: Some("claude-sonnet-4-5".to_string()),
embeddings: EmbeddingsSettings {
enabled: true,
provider: "openai".to_string(),
..Default::default()
},
..Default::default()
};
// Save to DB and reload (simulates persistence round-trip)
let db_map = prior_run.to_db_map();
let from_db = Settings::from_db_map(&db_map);
// Step 1 of the new wizard run: user enters a NEW database_url
let mut step1_settings = Settings::default();
step1_settings.database_backend = Some("postgres".to_string());
step1_settings.database_url = Some("postgres://new-host/ironclaw".to_string());
// Wizard flow: load DB → merge_from(step1_overrides)
let mut current = step1_settings.clone();
// try_load_existing_settings: merge DB into current
current.merge_from(&from_db);
// Re-apply Step 1 choices on top
current.merge_from(&step1_settings);
// Step 1's fresh database_url wins over stale DB value
assert_eq!(
current.database_url,
Some("postgres://new-host/ironclaw".to_string()),
"Step 1 fresh choice must override stale DB value"
);
// Prior run's steps 2-4 settings are preserved
assert_eq!(
current.llm_backend,
Some("anthropic".to_string()),
"Prior run's LLM backend must be recovered"
);
assert_eq!(
current.selected_model,
Some("claude-sonnet-4-5".to_string()),
"Prior run's model must be recovered"
);
assert!(
current.embeddings.enabled,
"Prior run's embeddings setting must be recovered"
);
}
/// Verifies that persisting defaults doesn't clobber prior settings
/// when the merge ordering is correct.
#[test]
fn wizard_recovery_defaults_dont_clobber_prior() {
// Prior run saved non-default settings
let prior_run = Settings {
llm_backend: Some("openai".to_string()),
selected_model: Some("gpt-4o".to_string()),
heartbeat: HeartbeatSettings {
enabled: true,
interval_secs: 900,
..Default::default()
},
..Default::default()
};
let db_map = prior_run.to_db_map();
let from_db = Settings::from_db_map(&db_map);
// New wizard run: Step 1 only sets DB fields (rest is default)
let step1 = Settings {
database_backend: Some("libsql".to_string()),
..Default::default()
};
// Correct merge ordering
let mut current = step1.clone();
current.merge_from(&from_db);
current.merge_from(&step1);
// Prior settings preserved (Step 1 doesn't touch these)
assert_eq!(current.llm_backend, Some("openai".to_string()));
assert_eq!(current.selected_model, Some("gpt-4o".to_string()));
assert!(current.heartbeat.enabled);
assert_eq!(current.heartbeat.interval_secs, 900);
// Step 1's choice applied
assert_eq!(current.database_backend, Some("libsql".to_string()));
}
}
+128 -24
View File
@@ -50,7 +50,7 @@ The `--no-onboard` CLI flag suppresses auto-detection.
---
## The 7-Step Wizard
## The 8-Step Wizard
### Overview
@@ -61,7 +61,8 @@ Step 3: Inference Provider ← skipped if --skip-auth
Step 4: Model Selection
Step 5: Embeddings
Step 6: Channel Configuration
Step 7: Background Tasks (heartbeat)
Step 7: Extensions (tools)
Step 8: Background Tasks (heartbeat)
save_and_summarize()
```
@@ -166,7 +167,8 @@ env-var mode or skipped secrets.
| Provider | Auth Method | Secret Name | Env Var |
|----------|-------------|-------------|---------|
| NEAR AI | Browser OAuth | (session token) | `NEARAI_SESSION_TOKEN` |
| NEAR AI Chat | Browser OAuth or session token | - | `NEARAI_SESSION_TOKEN` |
| NEAR AI Cloud | API key | `llm_nearai_api_key` | `NEARAI_API_KEY` |
| Anthropic | API key | `anthropic_api_key` | `ANTHROPIC_API_KEY` |
| OpenAI | API key | `openai_api_key` | `OPENAI_API_KEY` |
| Ollama | None | - | - |
@@ -179,8 +181,18 @@ env-var mode or skipped secrets.
4. **Cache key in `self.llm_api_key`** for model fetching in Step 4
**NEAR AI** (`setup_nearai`):
- Calls `session_manager.ensure_authenticated()` which opens browser
- Session token saved to `~/.ironclaw/session.json`
- Calls `session_manager.ensure_authenticated()` which shows the auth menu:
- Options 1-2 (GitHub/Google): browser OAuth → **NEAR AI Chat** mode
(Responses API at `private.near.ai`, session token auth)
- Option 4: NEAR AI Cloud API key → **NEAR AI Cloud** mode
(Chat Completions API at `cloud-api.near.ai`, API key auth)
- **NEAR AI Chat** path: session token saved to `~/.ironclaw/session.json`.
Hosting providers can set `NEARAI_SESSION_TOKEN` env var directly (takes
precedence over file-based tokens).
- **NEAR AI Cloud** path: `NEARAI_API_KEY` saved to `~/.ironclaw/.env`
(bootstrap) and encrypted secrets store (`llm_nearai_api_key`).
`LlmConfig::resolve()` auto-selects `ChatCompletions` mode when the
API key is present.
**`self.llm_api_key` caching:** The wizard caches the API key as
`Option<SecretString>` so that Step 4 (model fetching) and Step 5
@@ -243,13 +255,20 @@ key first, then falls back to the standard env var.
```
6a. Tunnel setup (if webhook channels needed)
6b. Discover WASM channels from ~/.ironclaw/channels/
6c. Multi-select: CLI/TUI, HTTP, discovered channels, bundled channels
6d. Install missing bundled channels (copy WASM binaries)
6e. Initialize SecretsContext (for token storage)
6f. Setup HTTP webhook (if selected)
6g. Setup each WASM channel (secrets, owner binding)
6c. Build channel options: discovered + bundled + registry catalog
6d. Multi-select: CLI/TUI, HTTP, all available channels
6e. Install missing bundled channels (copy WASM binaries)
6f. Install missing registry channels (build from source)
6g. Initialize SecretsContext (for token storage)
6h. Setup HTTP webhook (if selected)
6i. Setup each WASM channel (secrets, owner binding)
```
**Channel sources** (priority order for installation):
1. Already installed in `~/.ironclaw/channels/`
2. Bundled channels (pre-compiled in `channels-src/`)
3. Registry channels (`registry/channels/*.json`, built from source)
**Tunnel setup** (`setup_tunnel`):
- Options: ngrok, Cloudflare Tunnel, localtunnel, custom URL
- Validates HTTPS requirement
@@ -273,7 +292,33 @@ key first, then falls back to the standard env var.
---
### Step 7: Heartbeat
### Step 7: Extensions (Tools)
**Module:** `wizard.rs``step_extensions()`
**Goal:** Install WASM tools from the extension registry.
**Flow:**
1. Load `RegistryCatalog` from `registry/` directory
2. If registry not found, print info and skip
3. List all tool manifests from the catalog
4. Discover already-installed tools in `~/.ironclaw/tools/`
5. Multi-select: show all registry tools with display name, auth method,
and description. Pre-check tools tagged `"default"` and already installed.
6. For each selected tool not yet installed, build from source via
`RegistryInstaller::install_from_source()`
7. Print consolidated auth hints (deduplicated by provider, e.g. one hint
for all Google tools sharing `google_oauth_token`)
**Registry lookup** (`load_registry_catalog`):
Searches for `registry/` directory in order:
1. Current working directory
2. Next to the executable
3. `CARGO_MANIFEST_DIR` (compile-time, dev builds)
---
### Step 8: Heartbeat
**Module:** `wizard.rs``step_heartbeat()`
@@ -338,25 +383,60 @@ heartbeat.enabled = "true"
heartbeat.interval_secs = "300"
```
### Incremental Persistence
Settings are persisted **after every successful step**, not just at the end.
This prevents data loss if a later step fails (e.g., the user enters an
API key in step 3 but step 5 crashes — they won't need to re-enter it).
**`persist_after_step()`** is called after each step in `run()` and:
1. Writes bootstrap vars to `~/.ironclaw/.env` via `write_bootstrap_env()`
2. Writes all current settings to the database via `persist_settings()`
3. Silently ignores errors (e.g., if called before Step 1 establishes a DB)
**`try_load_existing_settings()`** is called after Step 1 establishes a
database connection. It loads any previously saved settings from the
database using `get_all_settings("default")``Settings::from_db_map()`
`merge_from()`. This recovers progress from prior partial wizard runs.
**Ordering after Step 1 is critical:**
```
step_database() → sets DB fields in self.settings
let step1 = self.settings.clone() → snapshot Step 1 choices
try_load_existing_settings() → merge DB values into self.settings
self.settings.merge_from(&step1) → re-apply Step 1 (fresh wins over stale)
persist_after_step() → save merged state
```
This ordering ensures:
- Prior progress (steps 2-7 from a previous partial run) is recovered
- Fresh Step 1 choices override stale DB values (not the reverse)
- The first DB persist doesn't clobber prior settings with defaults
### save_and_summarize()
Final step of the wizard:
```
1. Mark onboard_completed = true
2. Write ALL settings to database (try postgres pool, then libSQL backend)
3. Write bootstrap vars to ~/.ironclaw/.env:
- DATABASE_BACKEND (always)
- DATABASE_URL (if postgres)
- LIBSQL_PATH (if libsql)
- LIBSQL_URL (if turso sync)
- LLM_BACKEND (always, when set)
- LLM_BASE_URL (if openai_compatible)
- OLLAMA_BASE_URL (if ollama)
- ONBOARD_COMPLETED (always, "true")
2. Call persist_settings() for final write (idempotent — ensures
onboard_completed flag is saved)
3. Call write_bootstrap_env() for final .env write (idempotent)
4. Print configuration summary
```
Bootstrap vars written to `~/.ironclaw/.env`:
- `DATABASE_BACKEND` (always)
- `DATABASE_URL` (if postgres)
- `LIBSQL_PATH` (if libsql)
- `LIBSQL_URL` (if turso sync)
- `LLM_BACKEND` (always, when set)
- `LLM_BASE_URL` (if openai_compatible)
- `OLLAMA_BASE_URL` (if ollama)
- `NEARAI_API_KEY` (if API key auth path)
- `ONBOARD_COMPLETED` (always, "true")
**Invariant:** Both Layer 1 and Layer 2 must be written. If the database
write fails, the wizard returns an error and the `.env` file is not written.
@@ -464,9 +544,9 @@ anthropic_api_key → encrypted API key
| `confirm(label, default)` | `[Y/n]` or `[y/N]` prompt |
| `print_header(text)` | Bold section header with underline |
| `print_step(n, total, text)` | `[1/7] Step Name` |
| `print_success(text)` | Green checkmark prefix |
| `print_error(text)` | Red X prefix |
| `print_info(text)` | Blue info prefix |
| `print_success(text)` | Green `✓` prefix (ANSI color), message in default color |
| `print_error(text)` | Red `✗` prefix (ANSI color), message in default color |
| `print_info(text)` | Blue `` prefix (ANSI color), message in default color |
`select_many` uses `crossterm` raw mode for arrow key navigation.
Must properly restore terminal state on all exit paths.
@@ -489,6 +569,30 @@ Must properly restore terminal state on all exit paths.
- May need `gnome-keyring` daemon running
- Collection unlock may prompt for password
### Remote Server Authentication
On remote/VPS servers, the browser-based OAuth flow for NEAR AI may not
work because `http://127.0.0.1:9876` is unreachable from the user's
local browser.
**Solutions:**
1. **NEAR AI Cloud API key (option 4 in auth menu):** Get an API key
from `https://cloud.near.ai` and paste it into the terminal. No
local listener is needed. The key is saved to `~/.ironclaw/.env`
and the encrypted secrets store. Uses the OpenAI-compatible
ChatCompletions API mode.
2. **Custom callback URL:** Set `IRONCLAW_OAUTH_CALLBACK_URL` to a
publicly accessible URL (e.g., via SSH tunnel or reverse proxy) that
forwards to port 9876 on the server:
```bash
export IRONCLAW_OAUTH_CALLBACK_URL=https://myserver.example.com:9876
```
The `callback_url()` function in `oauth_defaults.rs` checks this env var
and falls back to `http://127.0.0.1:{OAUTH_CALLBACK_PORT}`.
### URL Passwords
- `#` is common in URL-encoded passwords (`%23` decoded)
+46 -4
View File
@@ -363,12 +363,54 @@ pub fn setup_tunnel(settings: &Settings) -> Result<TunnelSettings, ChannelSetupE
// Show existing config
let has_existing = settings.tunnel.public_url.is_some() || settings.tunnel.provider.is_some();
if has_existing {
if let Some(ref url) = settings.tunnel.public_url {
print_info(&format!("Existing static tunnel URL: {}", url));
println!();
print_info("Current tunnel configuration:");
let t = &settings.tunnel;
match t.provider.as_deref() {
Some("ngrok") => {
print_info(" Provider: ngrok");
if let Some(ref domain) = t.ngrok_domain {
print_info(&format!(" Domain: {}", domain));
}
if t.ngrok_token.is_some() {
print_info(" Auth: token configured");
}
}
Some("cloudflare") => {
print_info(" Provider: Cloudflare Tunnel");
if t.cf_token.is_some() {
print_info(" Auth: token configured");
}
}
Some("tailscale") => {
let mode = if t.ts_funnel {
"Funnel (public)"
} else {
"Serve (tailnet-only)"
};
print_info(&format!(" Provider: Tailscale {}", mode));
if let Some(ref hostname) = t.ts_hostname {
print_info(&format!(" Hostname: {}", hostname));
}
}
Some("custom") => {
print_info(" Provider: Custom command");
if let Some(ref cmd) = t.custom_command {
print_info(&format!(" Command: {}", cmd));
}
if let Some(ref url) = t.custom_health_url {
print_info(&format!(" Health: {}", url));
}
}
Some(other) => {
print_info(&format!(" Provider: {}", other));
}
None => {}
}
if let Some(ref provider) = settings.tunnel.provider {
print_info(&format!("Existing managed provider: {}", provider));
if let Some(ref url) = t.public_url {
print_info(&format!(" URL: {}", url));
}
println!();
if !confirm("Change tunnel configuration?", false)? {
return Ok(settings.tunnel.clone());
}
+2 -1
View File
@@ -7,7 +7,8 @@
//! 4. Model selection
//! 5. Embeddings
//! 6. Channel configuration (HTTP, Telegram, etc.)
//! 7. Heartbeat (background tasks)
//! 7. Extensions (tool installation from registry)
//! 8. Heartbeat (background tasks)
//!
//! # Example
//!
+29 -6
View File
@@ -293,19 +293,31 @@ pub fn print_step(current: usize, total: usize, name: &str) {
println!();
}
/// Print a success message with checkmark.
/// Print a success message with green checkmark.
pub fn print_success(message: &str) {
println!("{}", message);
let mut stdout = io::stdout();
let _ = execute!(stdout, SetForegroundColor(Color::Green));
print!("");
let _ = execute!(stdout, ResetColor);
println!(" {}", message);
}
/// Print an error message.
/// Print an error message with red X.
pub fn print_error(message: &str) {
eprintln!("{}", message);
let mut stderr = io::stderr();
let _ = execute!(stderr, SetForegroundColor(Color::Red));
eprint!("");
let _ = execute!(stderr, ResetColor);
eprintln!(" {}", message);
}
/// Print an info message.
/// Print an info message with blue info icon.
pub fn print_info(message: &str) {
println!(" {}", message);
let mut stdout = io::stdout();
let _ = execute!(stdout, SetForegroundColor(Color::Blue));
print!("");
let _ = execute!(stdout, ResetColor);
println!(" {}", message);
}
/// Read a simple line of input with a prompt.
@@ -358,4 +370,15 @@ mod tests {
super::print_step(1, 3, "Test Step");
super::print_step(3, 3, "Final Step");
}
#[test]
fn test_print_functions_do_not_panic() {
super::print_success("operation completed");
super::print_error("something went wrong");
super::print_info("here is some information");
// Also test with empty strings
super::print_success("");
super::print_error("");
super::print_info("");
}
}
+744 -124
View File
File diff suppressed because it is too large Load Diff
+1 -1
View File
@@ -197,7 +197,7 @@ impl SkillRegistry {
let source = make_source(path.clone());
match self.load_skill_md(&skill_md, trust, source).await {
Ok((name, skill)) => {
tracing::info!("Loaded skill: {}", name);
tracing::debug!("Loaded skill: {}", name);
results.push((name, skill));
}
Err(e) => {
-2
View File
@@ -168,7 +168,6 @@ impl LlmProvider for StubLlm {
input_tokens: 10,
output_tokens: 5,
finish_reason: FinishReason::Stop,
response_id: None,
})
}
@@ -186,7 +185,6 @@ impl LlmProvider for StubLlm {
input_tokens: 10,
output_tokens: 5,
finish_reason: FinishReason::Stop,
response_id: None,
})
}
}
+136
View File
@@ -0,0 +1,136 @@
# Tool System
## Adding a New Tool
### Built-in Tools (Rust)
1. Create `src/tools/builtin/my_tool.rs`
2. Implement the `Tool` trait
3. Add `mod my_tool;` and `pub use` in `src/tools/builtin/mod.rs`
4. Register in `ToolRegistry::register_builtin_tools()` in `registry.rs`
5. Add tests
### WASM Tools (Recommended)
WASM tools are the preferred way to add new capabilities. They run in a sandboxed environment with explicit capabilities.
1. Create a new crate in `tools-src/<name>/`
2. Implement the WIT interface (`wit/tool.wit`)
3. Create `<name>.capabilities.json` declaring required permissions
4. Build with `cargo build --target wasm32-wasip2 --release`
5. Install with `ironclaw tool install path/to/tool.wasm`
See `tools-src/` for examples.
## Tool Architecture Principles
**CRITICAL: Keep tool-specific logic out of the main agent codebase.**
The main agent provides generic infrastructure; tools are self-contained units that declare their requirements through capabilities files.
### What Goes in Tools (capabilities.json)
- API endpoints the tool needs (HTTP allowlist)
- Credentials required (secret names, injection locations)
- Rate limits and timeouts
- Auth setup instructions (see below)
- Workspace paths the tool can read
### What Does NOT Go in Main Agent
- Service-specific auth flows (OAuth for Notion, Slack, etc.)
- Service-specific CLI commands (`auth notion`, `auth slack`)
- Service-specific configuration handling
- Hardcoded API URLs or token formats
### Tool Authentication
Tools declare their auth requirements in `<tool>.capabilities.json` under the `auth` section. Two methods are supported:
#### OAuth (Browser-based login)
For services that support OAuth, users just click through browser login:
```json
{
"auth": {
"secret_name": "notion_api_token",
"display_name": "Notion",
"oauth": {
"authorization_url": "https://api.notion.com/v1/oauth/authorize",
"token_url": "https://api.notion.com/v1/oauth/token",
"client_id_env": "NOTION_OAUTH_CLIENT_ID",
"client_secret_env": "NOTION_OAUTH_CLIENT_SECRET",
"scopes": [],
"use_pkce": false,
"extra_params": { "owner": "user" }
},
"env_var": "NOTION_TOKEN"
}
}
```
To enable OAuth for a tool:
1. Register a public OAuth app with the service (e.g., notion.so/my-integrations)
2. Configure redirect URIs: `http://localhost:9876/callback` through `http://localhost:9886/callback`
3. Set environment variables for client_id and client_secret
#### Manual Token Entry (Fallback)
For services without OAuth or when OAuth isn't configured:
```json
{
"auth": {
"secret_name": "openai_api_key",
"display_name": "OpenAI",
"instructions": "Get your API key from platform.openai.com/api-keys",
"setup_url": "https://platform.openai.com/api-keys",
"token_hint": "Starts with 'sk-'",
"env_var": "OPENAI_API_KEY"
}
}
```
#### Auth Flow Priority
When running `ironclaw tool auth <tool>`:
1. Check `env_var` - if set in environment, use it directly
2. Check `oauth` - if configured, open browser for OAuth flow
3. Fall back to `instructions` + manual token entry
The agent reads auth config from the tool's capabilities file and provides the appropriate flow. No service-specific code in the main agent.
### WASM Tools vs MCP Servers: When to Use Which
Both are first-class in the extension system (`ironclaw tool install` handles both), but they have different strengths.
**WASM Tools (IronClaw native)**
- Sandboxed: fuel metering, memory limits, no access except what's allowlisted
- Credentials injected by host runtime, tool code never sees the actual token
- Output scanned for secret leakage before returning to the LLM
- Auth (OAuth/manual) declared in `capabilities.json`, agent handles the flow
- Single binary, no process management, works offline
- Cost: must build yourself in Rust, no ecosystem, synchronous only
**MCP Servers (Model Context Protocol)**
- Growing ecosystem of pre-built servers (GitHub, Notion, Postgres, etc.)
- Any language (TypeScript/Python most common)
- Can do websockets, streaming, background polling
- Cost: external process with full system access (no sandbox), manages own credentials, IronClaw can't prevent leaks
**Decision guide:**
| Scenario | Use |
|----------|-----|
| Good MCP server already exists | **MCP** |
| Handles sensitive credentials (email send, banking) | **WASM** |
| Quick prototype or one-off integration | **MCP** |
| Core capability you'll maintain long-term | **WASM** |
| Needs background connections (websockets, polling) | **MCP** |
| Multiple tools share one OAuth token (e.g., Google suite) | **WASM** |
The LLM-facing interface is identical for both (tool name, schema, execute), so swapping between them is transparent to the agent.
+16 -9
View File
@@ -189,8 +189,8 @@ impl Tool for HttpTool {
}
},
"body": {
"type": "string",
"description": "Request body. Use plain text or serialized JSON."
"type": ["object", "array", "string", "number", "boolean", "null"],
"description": "Request body (for POST/PUT/PATCH)"
},
"timeout_secs": {
"type": "integer",
@@ -361,13 +361,6 @@ impl Tool for HttpTool {
mod tests {
use super::*;
#[test]
fn test_http_tool_schema_body_has_type() {
let tool = HttpTool::new();
let schema = tool.parameters_schema();
assert_eq!(schema["properties"]["body"]["type"], "string");
}
#[test]
fn test_http_tool_schema_headers_is_array() {
let tool = HttpTool::new();
@@ -460,4 +453,18 @@ mod tests {
]
);
}
#[test]
fn test_http_tool_schema_body_has_type() {
let schema = HttpTool::new().parameters_schema();
let body = schema
.get("properties")
.and_then(|p| p.get("body"))
.expect("body schema missing");
assert!(
body.get("type").is_some(),
"body schema must include a type for OpenAI-compatible tool validation"
);
}
}
+16 -9
View File
@@ -28,8 +28,8 @@ impl Tool for JsonTool {
"description": "The JSON operation to perform"
},
"data": {
"type": "string",
"description": "JSON input string. For query/stringify/validate, pass serialized JSON."
"type": ["string", "object", "array", "number", "boolean", "null"],
"description": "JSON input data. Pass a string for parse, any type otherwise."
},
"path": {
"type": "string",
@@ -154,13 +154,6 @@ fn query_json(data: &serde_json::Value, path: &str) -> Result<serde_json::Value,
mod tests {
use super::*;
#[test]
fn test_json_tool_schema_data_has_type() {
let tool = JsonTool;
let schema = tool.parameters_schema();
assert_eq!(schema["properties"]["data"]["type"], "string");
}
#[test]
fn test_query_json() {
let data = serde_json::json!({
@@ -197,4 +190,18 @@ mod tests {
let err = parse_json_input(&input).unwrap_err();
assert!(err.to_string().contains("invalid JSON input"));
}
#[test]
fn test_json_tool_schema_data_has_type() {
let schema = JsonTool.parameters_schema();
let data = schema
.get("properties")
.and_then(|p| p.get("data"))
.expect("data schema missing");
assert!(
data.get("type").is_some(),
"data schema must include a type for OpenAI-compatible tool validation"
);
}
}
+58 -16
View File
@@ -507,25 +507,47 @@ impl ShellTool {
.spawn()
.map_err(|e| ToolError::ExecutionFailed(format!("Failed to spawn command: {}", e)))?;
// Wait with timeout
// Drain stdout/stderr concurrently with wait() to prevent deadlocks.
// If we call wait() without draining the pipes and the child's output
// exceeds the OS pipe buffer (64KB Linux, 16KB macOS), the child blocks
// on write and wait() never returns.
let stdout_handle = child.stdout.take();
let stderr_handle = child.stderr.take();
let result = tokio::time::timeout(timeout, async {
let status = child.wait().await?;
let stdout_fut = async {
if let Some(mut out) = stdout_handle {
let mut buf = Vec::new();
(&mut out)
.take(MAX_OUTPUT_SIZE as u64)
.read_to_end(&mut buf)
.await
.ok();
// Drain any remaining output so the child does not block
tokio::io::copy(&mut out, &mut tokio::io::sink()).await.ok();
String::from_utf8_lossy(&buf).to_string()
} else {
String::new()
}
};
// Read stdout
let mut stdout = String::new();
if let Some(mut out) = child.stdout.take() {
let mut buf = vec![0u8; MAX_OUTPUT_SIZE];
let n = out.read(&mut buf).await.unwrap_or(0);
stdout = String::from_utf8_lossy(&buf[..n]).to_string();
}
let stderr_fut = async {
if let Some(mut err) = stderr_handle {
let mut buf = Vec::new();
(&mut err)
.take(MAX_OUTPUT_SIZE as u64)
.read_to_end(&mut buf)
.await
.ok();
tokio::io::copy(&mut err, &mut tokio::io::sink()).await.ok();
String::from_utf8_lossy(&buf).to_string()
} else {
String::new()
}
};
// Read stderr
let mut stderr = String::new();
if let Some(mut err) = child.stderr.take() {
let mut buf = vec![0u8; MAX_OUTPUT_SIZE];
let n = err.read(&mut buf).await.unwrap_or(0);
stderr = String::from_utf8_lossy(&buf[..n]).to_string();
}
let (stdout, stderr, wait_result) = tokio::join!(stdout_fut, stderr_fut, child.wait());
let status = wait_result?;
// Combine output
let output = if stderr.is_empty() {
@@ -1184,6 +1206,26 @@ mod tests {
);
}
#[tokio::test]
async fn test_large_output_command() {
let tool = ShellTool::new().with_timeout(Duration::from_secs(10));
let ctx = JobContext::default();
// Generate output larger than OS pipe buffer (64KB on Linux, 16KB on macOS).
// Without draining pipes before wait(), this would deadlock.
let result = tool
.execute(
serde_json::json!({"command": "python3 -c \"print('A' * 131072)\""}),
&ctx,
)
.await
.unwrap();
let output = result.result.get("output").unwrap().as_str().unwrap();
assert_eq!(output.len(), MAX_OUTPUT_SIZE);
assert_eq!(result.result.get("exit_code").unwrap().as_i64().unwrap(), 0);
}
#[tokio::test]
async fn test_netcat_blocked_at_execution() {
let tool = ShellTool::new();

Some files were not shown because too many files have changed in this diff Show More