mirror of
https://github.com/outbackdingo/optimclaw.git
synced 2026-08-31 00:29:24 +00:00
65062f3cc069ebbd29f6d9be874ae5eff796e43a
17
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Henry ParkandGitHub
|
8a60fa2d37 |
fix: add tool_info schema discovery for WASM tools (#1086)
* fix: add tool_info schema discovery for WASM tools * refactor: simplify WASM schema and hint state * refactor: store tool_info registry reference as Weak |
||
|
|
369741fc60 |
Add generic host-verified /webhook/tools/{tool} ingress (#757)
* Add generic host-verified webhook ingress for tools * Stabilize trace E2E test rig and approval behavior * Fix webhook security issues from review feedback - Reject tools without webhook_capability() (was unauthenticated RCE) - Remove secret-in-query-string fallback (leak via logs/referrers) - Require approval for event_emit tool (escalation via routine triggers) - Simplify header_value() (HeaderMap already case-insensitive) - Redact internal errors from webhook HTTP responses - Remove unused hmac_timestamp_tolerance_secs field - Add regression test for tool without webhook capability [skip-regression-check] Co-Authored-By: Claude Opus 4.6 <[email protected]> * Harden webhook ingress: require auth mechanism, body limit layer, health check - Reject webhook capabilities that declare no auth mechanism (empty WebhookCapability would previously allow unauthenticated access) - Add DefaultBodyLimit layer to reject oversized payloads before buffering - Health check (GET) now verifies tool has webhook_capability(), not just existence - Add regression tests for all three fixes [skip-regression-check] Co-Authored-By: Claude Opus 4.6 <[email protected]> * Fix auto_approve_tools inconsistency between dispatcher and thread_ops dispatcher.rs skips all approval checks (including Always) when auto_approve_tools is true, but thread_ops.rs still required approval for Always tools. This caused deferred tool calls to unexpectedly halt in test rigs and auto-approve configurations. Match dispatcher behavior: short-circuit all approval when auto_approve_tools is enabled. [skip-regression-check] Co-Authored-By: Claude Opus 4.6 <[email protected]> --------- Co-authored-by: Claude Opus 4.6 <[email protected]> |
||
|
|
d144484b06 |
feat: WASM channel attachments with LLM pipeline integration (#596)
* feat: add inbound attachment support to WASM channel system Add attachment record to WIT interface and implement inbound media parsing across all four channel implementations (Telegram, Slack, WhatsApp, Discord). Attachments flow from WASM channels through EmittedMessage to IncomingMessage with validation (size limits, MIME allowlist, count caps) at the host boundary. - Add `attachment` record to `emitted-message` in wit/channel.wit - Add `IncomingAttachment` struct to channel.rs and re-export - Add host-side validation (20MB total, 10 max, MIME allowlist) - Telegram: parse photo, document, audio, video, voice, sticker - Slack: parse file attachments with url_private - WhatsApp: parse image, audio, video, document with captions - Discord: backward-compatible empty attachments - Update FEATURE_PARITY.md section 7 - Add fixture-based tests per channel and host integration tests [skip-regression-check] Co-Authored-By: Claude Opus 4.6 <[email protected]> * feat: integrate outbound attachment support and reconcile WIT types (#409) Reconcile PR #409's outbound attachment work with our inbound attachment support into a unified design: WIT type split: - `inbound-attachment` in channel-host: metadata-only (id, mime_type, filename, size_bytes, source_url, storage_key, extracted_text) - `attachment` in channel: raw bytes (filename, mime_type, data) on agent-response for outbound sending Outbound features (from PR #409): - `on-broadcast` WIT export for proactive messages without prior inbound - Telegram: multipart sendPhoto/sendDocument with auto photo→document fallback for files >10MB - wrapper.rs: `call_on_broadcast`, `read_attachments` from disk, attachment params threaded through `call_on_respond` - HTTP tool: `save_to` param for binary downloads to /tmp/ (50MB limit, path traversal protection, SSRF-safe redirect following) - Message tool: allow /tmp/ paths for attachments alongside base_dir - Credential env var fallback in inject_channel_credentials Channel updates: - All 4 channels implement on_broadcast (Telegram full, others stub) - Telegram: polling_enabled config, adjusted poll timeout - Inbound attachment types renamed to InboundAttachment in all channels Tests: 1965 passing (9 new), 0 clippy warnings [skip-regression-check] Co-Authored-By: Claude Opus 4.6 <[email protected]> * feat: add audio transcription pipeline and extensible WIT attachment design Add host-side transcription middleware (OpenAI Whisper) that detects audio attachments with inline data on incoming messages and transcribes them automatically. Refactor WIT inbound-attachment to use extras-json and a store-attachment-data host function instead of typed fields, so future attachment properties (dimensions, codec, etc.) don't require WIT changes that invalidate all channel plugins. - Add src/transcription/ module: TranscriptionProvider trait, TranscriptionMiddleware, AudioFormat enum, OpenAI Whisper provider - Add src/config/transcription.rs: TRANSCRIPTION_ENABLED/MODEL/BASE_URL - Wire middleware into agent message loop via AgentDeps - WIT: replace data + duration-secs with extras-json + store-attachment-data - Host: parse extras-json for well-known keys, merge stored binary data - Telegram: download voice files via store-attachment-data, add duration to extras-json, add /file/bot to HTTP allowlist, voice-only placeholder - Add reqwest multipart feature for Whisper API uploads - 5 regression tests for transcription middleware Co-Authored-By: Claude Opus 4.6 <[email protected]> * feat: wire attachment processing into LLM pipeline with multimodal image support Attachments on incoming messages are now augmented into user text via XML tags before entering the turn system, and images with data are passed as multimodal content parts (base64 data URIs) to LLM providers. This enables audio transcripts, document text, and image content to reach the LLM without changes to ChatMessage serialization or provider interfaces. - Add src/agent/attachments.rs with augment_with_attachments() and 9 unit tests - Add ContentPart/ImageUrl types to llm::provider with OpenAI-compatible serde - Carry image_content_parts transiently on Turn (skipped in serialization) - Update nearai_chat and rig_adapter to serialize multimodal content - Add 3 e2e tests verifying attachments flow through the full agent loop Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: CI failures — formatting, version bumps, and Telegram voice test - Fix cargo fmt formatting in attachments.rs, nearai_chat.rs, rig_adapter.rs, e2e_attachments.rs - Bump channel registry versions 0.1.0 → 0.2.0 (discord, slack, telegram, whatsapp) to satisfy version-bump CI check - Fix Telegram test_extract_attachments_voice: add missing required `duration` field to voice fixture JSON Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: bump WIT channel version to 0.3.0, fix Telegram voice test, add pre-commit hook - Bump wit/channel.wit package version 0.2.0 → 0.3.0 (interface changed with store-attachment-data) - Update WIT_CHANNEL_VERSION constant and registry wit_version fields to match - Fix Telegram test_extract_attachments_voice: gate voice download behind #[cfg(target_arch = "wasm32")] so host functions aren't called in native tests, update assertions for generated filename and extras_json duration - Add @0.3.0 linker stubs in wit_compat.rs - Add .githooks/pre-commit hook that runs scripts/check-version-bumps.sh when WIT or extension sources are staged - Symlink commit-msg regression hook into .githooks/ [skip-regression-check] Co-Authored-By: Claude Opus 4.6 <[email protected]> * refactor: extract voice download from extract_attachments into handle_message Move download_voice_file + store_attachment_data calls out of extract_attachments into a separate download_and_store_voice function called from handle_message. This keeps extract_attachments as a pure data-mapping function with no host calls, making it fully testable in native unit tests without #[cfg(target_arch)] gates. [skip-regression-check] Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: address PR review comments — security, correctness, and code quality Security fixes: - Add path validation to read_attachments (restrict to /tmp/) preventing arbitrary file reads from compromised tools - Escape XML special characters in attachment filenames, MIME types, and extracted text to prevent prompt injection via tag spoofing - Percent-encode file_id in Telegram getFile URL to prevent query injection - Clone SecretString directly instead of expose_secret().to_string() Correctness fixes: - Fix store_attachment_data overwrite accounting: subtract old entry size before adding new to prevent inflated totals and false rejections - Use max(reported, stored_size) for attachment size accounting to prevent WASM channels from under-reporting size_bytes to bypass limits - Add application/octet-stream to MIME allowlist (channels default unknown types to this) Code quality: - Extract send_response helper in Telegram, deduplicating on_respond and on_broadcast - Rename misleading Discord test to test_parse_slash_command_interaction - Fix .githooks/commit-msg to use relative symlink (portable across machines) [skip-regression-check] Co-Authored-By: Claude Opus 4.6 <[email protected]> * feat: add tool_upgrade command + fix TOCTOU in save_to path validation Add `tool_upgrade` — a new extension management tool that automatically detects and reinstalls WASM extensions with outdated WIT versions. Preserves authentication secrets during upgrade. Supports upgrading a single extension by name or all installed WASM tools/channels at once. Fix TOCTOU in `validate_save_to_path`: validate the path *before* creating parent directories, so traversal paths like `/tmp/../../etc/` cannot cause filesystem mutations outside /tmp before being rejected. [skip-regression-check] Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: unify WIT package version to 0.3.0 across tool.wit and all capabilities tool.wit and channel.wit share the `near:agent` package namespace, so they must declare the same version. Bumps tool.wit from 0.2.0 to 0.3.0 and updates all capabilities files and registry entries to match. Fixes `cargo component build` failure: "package identifier near:[email protected] does not match previous package name of near:[email protected]" [skip-regression-check] Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: move WIT file comments after package declaration WIT treats `//` comments before `package` as doc comments. When both tool.wit and channel.wit had header comments, the parser rejected them as "doc comments on multiple 'package' items". Move comments after the package declaration in both files. Also bumps tool registry versions to 0.2.0 to match the WIT 0.3.0 bump. [skip-regression-check] Co-Authored-By: Claude Opus 4.6 <[email protected]> * feat: display extension versions in gateway Extensions tab Add version field to InstalledExtension and RegistryEntry types, pipe through the web API (ExtensionInfo, RegistryEntryInfo), and render as a badge in the gateway UI for both installed and available extensions. For installed WASM extensions, version is read from the capabilities file with a fallback to the registry entry when the local file has no version (old installations). Bump all extension Cargo.toml and registry JSON versions from 0.1.0 to 0.2.0 to keep them in sync. [skip-regression-check] Co-Authored-By: Claude Opus 4.6 <[email protected]> * feat: add document text extraction middleware for PDF, Office, and text files Extract text from document attachments (PDF, DOCX, PPTX, XLSX, RTF, plain text, code files) so the LLM can reason about uploaded documents. Uses pdf-extract for PDFs, zip+XML parsing for Office XML formats, and UTF-8 decode for text files. Wired into the agent loop after transcription middleware. Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: download document files in Telegram channel for text extraction The DocumentExtractionMiddleware needs file bytes in the attachment `data` field, but only voice files were being downloaded. Document attachments (PDFs, DOCX, etc.) had empty `data` and a source_url with a credential placeholder that only works inside the WASM host's http_request. Add `download_and_store_documents()` that downloads non-voice, non-image, non-audio attachments via the existing two-step getFile→download flow and stores bytes via `store_attachment_data` for host-side extraction. Also rename `download_voice_file` → `download_telegram_file` since it's generic for any file_id. Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: allow Office MIME types and increase file download limit for Telegram Two issues preventing document extraction from Telegram: 1. PPTX/DOCX/XLSX MIME types (application/vnd.*) were dropped by the WASM host attachment allowlist — add application/vnd., application/msword, and application/rtf prefixes. 2. Telegram file downloads over 10 MB failed with "Response body too large" — set max_response_bytes to 20 MB in Telegram capabilities. [skip-regression-check] Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: report document extraction errors back to user instead of silently skipping - Bump max_response_bytes to 50 MB for Telegram file downloads - When document extraction fails (too large, download error, parse error), set extracted_text to a user-friendly error message instead of leaving it None. This ensures the LLM tells the user what went wrong. - On Telegram download failure, set extracted_text with the error so the user sees feedback even when the file never reaches the extraction middleware. Co-Authored-By: Claude Opus 4.6 <[email protected]> * feat: store extracted document text in workspace memory for search/recall After document extraction succeeds, write the extracted text to workspace memory at `documents/{date}/{filename}`. This enables: - Full-text and semantic search over past uploaded documents - Cross-conversation recall ("what did that PDF say?") - Automatic chunking and embedding via the workspace pipeline Documents are stored with metadata header (uploader, channel, date, MIME type). Error messages (extraction failures) are not stored — only successful extractions. Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: CI failures — formatting, unused assignment warning - Run cargo fmt on document_extraction and agent_loop modules - Suppress unused_assignments warning on trace_llm_ref (used only behind #[cfg(feature = "libsql")]) [skip-regression-check] Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: address PR review comments — security, correctness, and code quality Security fixes: - Remove SSRF-prone download() from DocumentExtractionMiddleware (#13) - Sanitize filenames in workspace path to prevent directory traversal (#11) - Pre-check file size before reading in WASM wrapper to prevent OOM (#2) - Percent-encode file_id in Telegram source URLs (#7) Correctness fixes: - Clear image_content_parts on turn end to prevent memory leak (#1) - Find first *successful* transcription instead of first overall (#3) - Enforce data.len() size limit in document extraction (#10) - Use UTF-8 safe truncation with char_indices() (#12) Robustness & code quality: - Add 120s timeout to OpenAI Whisper HTTP client (#5) - Trim trailing slash from Whisper base_url (#6) - Allow ~/.ironclaw/ paths in WASM wrapper (#8) - Return error from on_broadcast in Slack/Discord/WhatsApp (#9) - Fix doc comment in HTTP tool (#4) Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: formatting — cargo fmt Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: address latest PR review — doc comments, error messages, version bumps - Fix DocumentExtractionMiddleware doc comment (no longer downloads from source_url) - Fix error message: "no inline data" instead of "no download URL" - Log error + fallback instead of silent unwrap_or_default on Whisper HTTP client - Bump all capabilities.json versions from 0.1.0 to 0.2.0 to match Cargo.toml Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: remove unsupported profile: minimal from CI workflows [skip-regression-check] dtolnay/rust-toolchain@stable does not accept the 'profile' input (it was a parameter for the deprecated actions-rs/toolchain action). Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: merge with latest main — resolve compilation errors and PR review nits - Add version: None to RegistryEntry/InstalledExtension test constructors - Fix MessageContent type mismatches in nearai_chat tests (String → MessageContent::Text) - Fix .contains() calls on MessageContent — use .as_text().unwrap() - Remove redundant trace_llm_ref = None assignment in test_rig - Check data size before clone in document extraction to avoid unnecessary allocation [skip-regression-check] Co-Authored-By: Claude Opus 4.6 <[email protected]> --------- Co-authored-by: Claude Opus 4.6 <[email protected]> |
||
|
|
13e000dc20 |
fix(wasm): use per-engine cache dirs on Windows to avoid file lock error (#624)
* fix(wasm): use per-engine cache dirs on Windows to avoid file lock error (#448) On Windows, multiple wasmtime Engine instances sharing the default compilation cache directory hit OS error 33 (ERROR_LOCK_VIOLATION) because Windows holds exclusive file locks on memory-mapped cache files. This is especially triggered when the Telegram channel WASM module is loaded at startup and then hot-activated via the Extensions UI. Fix by giving each engine its own cache subdirectory on Windows (~/.cache/ironclaw/wasmtime-tools/ and wasmtime-channels/). On Unix the shared default cache continues to work as before. Also adds Windows CI jobs (cargo check + clippy across all feature flag combinations) to catch Windows-specific issues going forward. Closes #448 Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: silence Windows clippy warnings for platform-gated code Gate PathBuf import behind #[cfg(unix)] in container.rs (only used in Unix socket path), suppress unused_mut on conflicts Vec in channels.rs (mutations are platform-gated), and add cfg gates on keychain constants and hex_to_bytes that are only used on macOS/Linux. Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: escape directory path in TOML cache config to prevent injection Use double-quoted TOML strings with backslash and double-quote escaping for the cache directory path, preventing breakage or injection when paths contain special characters (e.g. single quotes on Unix, backslashes on Windows). Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: resolve cargo fmt formatting errors Fix import ordering in container.rs and line wrapping in runtime.rs to pass the CI formatting check. Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix(ci): restore Path import for all platforms, keep PathBuf unix-only Path is used in non-cfg-gated functions (lines 148, 244) so it must be available on all platforms. Only PathBuf is unix-specific. Co-Authored-By: Claude Opus 4.6 <[email protected]> --------- Co-authored-by: Claude Opus 4.6 <[email protected]> |
||
|
|
04c5c3fe9f |
feat: WASM extension versioning with WIT compat checks (#592)
* feat: add WASM extension versioning with WIT compat checks and CI enforcement Phase 1 — WIT Versioning & Compatibility Checks: - Version WIT packages as `package near:[email protected];` - Add `semver` crate for version parsing and comparison - Add `WIT_TOOL_VERSION` / `WIT_CHANNEL_VERSION` host constants - Add `version` and `wit_version` fields to capabilities schemas - Add `wit_version` column to `wasm_tools` DB table (both backends) - Add load-time `check_wit_version_compat()` with semver rules - Add `IncompatibleWitVersion` error variants for tools and channels - Enhance instantiation errors with WIT version mismatch hints - Update all 14 capabilities JSON and 14 registry JSON files Phase 2 — Upgrade-in-Place & Channel DB Storage: - Change tool store to DELETE-before-INSERT (one version per extension) - Create `wasm_channels` table (PostgreSQL migration + libSQL schema) - Add `WasmChannelStore` trait with PostgreSQL and libSQL backends - Add `extension_info` tool showing version, WIT version, and status - Wire `ExtensionInfoTool` into tool registry (7 extension tools) Phase 3 — CI Version-Bump Enforcement: - Add `scripts/check-version-bumps.sh` checking WIT/tool/channel versions - Add `version-check` CI job (PR-only) to `.github/workflows/test.yml` - Support `[skip-version-check]` label/commit message bypass Includes 7 regression tests for WIT version compatibility checking and 2 integration tests for WIT version annotation verification. [skip-regression-check] Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: address PR review feedback for WASM extension versioning - Wrap PostgreSQL DELETE+INSERT in transactions for both tool and channel store() methods to prevent data loss on partial failure (Gemini, Copilot) - Rename StoredWasmChannelWithBinary.tool → .channel (copy-paste fix) - Remove unused WasmError::IncompatibleWitVersion variant (dead code) - Map channel loader WIT mismatch to IncompatibleWitVersion instead of generic Config error, simplify variant to single String message - Fix extension_info description to match actual returned fields - Add schema test for ExtensionInfoTool matching existing test pattern - Fix CI script to fail fast on git errors instead of silent bypass [skip-regression-check] Co-Authored-By: Claude Opus 4.6 <[email protected]> --------- Co-authored-by: Claude Opus 4.6 <[email protected]> |
||
|
|
dc7d9cce34 |
fix(channels): add host-based credential injection to WASM channel wrapper (#421)
* fix(channels): add host-based credential injection to WASM channel wrapper The channel WASM wrapper was missing the host-based credential injection that the tools wrapper implements. The `credentials` block in channel capabilities files was dead code: Slack's `on_respond` sends requests with no Authorization header, expecting the host to inject the bot token based on `host_patterns`, but the host never did. This caused Slack (and any channel relying on capabilities-declared credentials) to fail all outbound API calls with `not_authed`. Changes: - Add `ResolvedHostCredential` struct mirroring the tools wrapper - Add `host_credentials` field to `ChannelStoreData` - Add `inject_host_credentials()` method on `ChannelStoreData` - Update `redact_credentials()` to also scrub host-injected secret values - Add `secrets_store` field to `WasmChannel` + `with_secrets_store()` builder - Add `resolve_channel_host_credentials()` async helper that decrypts capabilities-declared credentials before each WASM callback - Update `create_store()` and all `call_on_*` / `execute_status` / `execute_poll` call sites to pre-resolve and pass host credentials - Fix leak scan ordering: scan runs on WASM-provided values BEFORE host credential injection, preventing false-positive blocks on injected Bearer tokens (e.g. xoxb- Slack tokens) - Make `credential_injector` module pub(crate) so channels can reuse `inject_credential` and `host_matches_pattern` Fixes #389, root cause of #413 Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * fix(wasm): redact URL-encoded credentials, use url::Url, derive Clone Address review feedback on PR #421: 1. Security: redact_credentials now scrubs URL-encoded forms of secrets in addition to raw values, preventing exfiltration via encoded representations in error strings from reqwest 2. Use url::Url::query_pairs_mut() for query parameter injection instead of manual string manipulation, improving robustness with malformed URLs 3. Derive Clone on ResolvedHostCredential and simplify the per-tick clone in the status repeater loop Co-Authored-By: Claude Opus 4.6 <[email protected]> * style: cargo fmt Co-Authored-By: Claude Opus 4.6 <[email protected]> --------- Co-authored-by: Sprite <[email protected]> Co-authored-by: Claude Sonnet 4.6 <[email protected]> |
||
|
|
ea57447649 |
feat: hot-activate WASM channels, channel-first prompts, unified artifact resolution (#297)
* refactor: unify WASM artifact resolution into registry/artifacts.rs Consolidate duplicated WASM find/build/install logic from 5+ files into a single src/registry/artifacts.rs module. This fixes two bugs: - registry/installer.rs now respects CARGO_TARGET_DIR (was hardcoded) - channels/wasm/bundled.rs now searches all WASM triples (was wasip2 only) Also includes: extension manager hot-activation for WASM channels, extension guidance in LLM prompts, channel manager hot-add support, webhook router channel lookup, and minor cleanups. Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: send approval prompts as messages on WASM channels (Telegram, Slack) WASM channels mapped ApprovalNeeded status to a typing indicator, so users on Telegram never saw tool approval prompts — the agent got stuck in AwaitingApproval and all subsequent messages failed with "Waiting for approval". - Intercept ApprovalNeeded in WasmChannel::handle_status_update and send the prompt as an actual message via call_on_respond, showing tool name, description, parameters, and yes/no/always instructions - Guard against empty LLM responses after clean_response() strips reasoning_content think-tags (defense-in-depth for reasoning models) - Add reasoning_content fallback to NearAiChatProvider::complete() for consistency with complete_with_tools() - Add debug logging when empty responses are suppressed - Improve error logging for channel respond() failures - Register WASM channel webhook routes before credential checks so platforms don't deactivate webhook URLs with 404s Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: address PR #297 review comments - ChannelManager::add: use async write().await instead of try_write() - resolve_target_dir: resolve relative CARGO_TARGET_DIR against crate_dir - install_wasm_files: log warning on capabilities copy failure - refresh_active_channel: load capabilities file for webhook secret name - activate_wasm_channel: validate name against path traversal - Fix cargo fmt formatting in nearai_chat.rs Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: wire up channel runtime for hot-activation and address PR review round 2 - Wire up set_channel_runtime() in main.rs so hot-activation actually works (with_channel_runtime was never called — hot-activation was dead code) - Change ExtensionManager channel runtime fields to RwLock<Option<...>> interior mutability so set_channel_runtime(&self) works after Arc wrapping - Fix artifact tests to use resolve_target_dir() instead of hardcoding "target/" (breaks when CARGO_TARGET_DIR is set) - Fix bundled.rs build hint: cargo component build (not cargo build --target) - Fix wasm_artifact_path doc: binary_name should not include .wasm extension Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: use char-aware truncation to prevent UTF-8 panic in approval prompt &s[..77] panics on multi-byte UTF-8 (CJK, emoji). Use s.chars().take(77) for safe truncation at character boundaries. Co-Authored-By: Claude Opus 4.6 <[email protected]> --------- Co-authored-by: Claude Opus 4.6 <[email protected]> |
||
|
|
2cdd1acb1e |
refactor: consolidate tool approval into single param-aware method (#274)
* refactor: consolidate tool approval into single param-aware method Replace the two confusing approval methods (requires_approval() and requires_approval_for()) with a single requires_approval(&self, params) returning a 3-variant ApprovalRequirement enum (Never, UnlessAutoApproved, Always). This enables param-aware approval decisions: HTTP calls without auth headers now skip approval entirely, while authenticated requests always require it. Shell tool merges its destructive-command detection into the same method. Co-Authored-By: Claude Opus 4.6 <[email protected]> * feat: add credential injection to built-in HTTP tool Wire the WASM credential injection system into the built-in HTTP tool so credentials are auto-injected at the boundary (zero-exposure model). - Add SharedCredentialRegistry: thread-safe, append-only registry of credential mappings populated by WASM tools at registration time - Add credential_detect module with broad auth detection for headers (12 exact + 5 substring matches), header values (7 auth scheme prefixes), and URL query params (17 exact + 5 substring matches) - HttpTool now accepts optional credential registry + secrets store, auto-injects matching credentials in execute(), and uses broader auth detection in requires_approval() - ToolRegistry passes credential registry to HttpTool at startup and populates it when WASM tools register - Remove old hardcoded AUTH_HEADER_NAMES / has_auth_headers in favor of the new params_contain_manual_credentials() Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: address PR #274 review comments (query param injection, lock poisoning, visibility) - Fix injected query params not being sent on outbound HTTP requests by also calling .query() on the RequestBuilder alongside parsed_url mutation - Recover from poisoned RwLock in SharedCredentialRegistry instead of silently ignoring failures, with tracing::warn for visibility - Narrow inject_credential and host_matches_pattern to pub(crate) to avoid committing to them as stable public API Co-Authored-By: Claude Opus 4.6 <[email protected]> --------- Co-authored-by: Claude Opus 4.6 <[email protected]> |
||
|
|
a53b2c10b5 |
fix: Fix wasm tool schemas and runtime (#42)
* feat: Move debug log truncation from agent loop to REPL channel Full tool output now flows through StatusUpdate so the web gateway gets untruncated content. The REPL channel truncates at display time (200 chars for tool results, thinking, and status messages). Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: Flatten WASM tool schemas and fix host HTTP runtime contention LLMs can't reliably follow oneOf + const discriminator patterns in JSON Schema, causing tools like Google Calendar to receive malformed params (e.g., {"operation":"list_events","data":{"calendarId":"primary"}} instead of {"action":"list_events","calendar_id":"primary"}). Replace all 9 WASM tool schemas with flat action enum + top-level properties. The serde #[serde(tag = "action")] deserialization works identically. Also fixes WASM host HTTP requests (channels and tools) stalling during startup by replacing Handle::current().block_on() with a dedicated single-threaded runtime per request, avoiding I/O driver contention. Reduces verbose LLM debug logging (full request/response payloads) and changes tower_http default from debug to warn. Co-Authored-By: Claude Opus 4.6 <[email protected]> * feat: Built-in OAuth credentials and combined Google scopes Add infrastructure for shipping default OAuth credentials with the binary, similar to how gcloud/rclone bake in their client_id. Credentials are set at compile time via IRONCLAW_GOOGLE_CLIENT_ID / IRONCLAW_GOOGLE_CLIENT_SECRET env vars, or can be hardcoded in src/cli/oauth_defaults.rs. The fallback chain is: capabilities file > runtime env var > built-in defaults. Also, when authing any Google tool, scopes from ALL installed Google tools are now combined into a single OAuth request (they all share the same google_oauth_token secret). One login covers Gmail, Calendar, Drive, etc. Co-Authored-By: Claude Opus 4.6 <[email protected]> * feat: Ship default Google OAuth credentials for zero-config auth Google Desktop App credentials are not secret (per Google's own docs). Hardcode them so `ironclaw tool auth <google-tool>` works out of the box without requiring users to register their own OAuth app. Credentials can still be overridden at compile time (IRONCLAW_GOOGLE_CLIENT_ID) or runtime (GOOGLE_OAUTH_CLIENT_ID). Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: Consistent OAuth callback port and polished landing page - Use fixed port 9876 instead of scanning 9876-9886 (one redirect URI to register in provider OAuth apps, deterministic behavior) - Replace broken unicode checkmark with SVG icons (charset was missing, rendered as mojibake) - Dark themed landing page with proper card layout for both success and error states - Add charset=utf-8 to Content-Type headers Co-Authored-By: Claude Opus 4.6 <[email protected]> * refactor: Unify OAuth callback server across all auth flows All three OAuth flows (WASM tool auth, MCP server auth, NEAR AI login) now share the same code from cli::oauth_defaults: - Fixed port 9876 (one redirect URI to register per provider) - Shared landing page HTML (dark card with SVG icons, proper charset) - Parameterized wait_for_callback(listener, path, param, display_name) Removes ~120 lines of duplicated callback/HTML code. Co-Authored-By: Claude Opus 4.6 <[email protected]> * Support for oauth token refresh * refactor: Replace bootstrap.json with ~/.ironclaw/.env for DATABASE_URL Kill the 4-field BootstrapConfig JSON file. Only DATABASE_URL actually needs disk persistence (chicken-and-egg before DB connect). The other three fields are now derived: pool_size defaults to 10 via env var, secrets master key is auto-detected (env then keychain probe), and onboard_completed is inferred from DATABASE_URL presence. The new format is a standard .env file loaded via dotenvy early in main, so DATABASE_URL is available as a regular env var everywhere. Handles three upgrade paths: - Clean start: wizard writes .env, reload after wizard completes - Returning user: .env loaded at startup, business as usual - Legacy upgrade: bootstrap.json auto-migrated to .env on first run Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: Address PR review findings - Fix UTF-8 panic in truncate_for_preview (byte-slice on char boundary) - Cap WASM guest timeout_ms at 5 minutes to prevent resource exhaustion - Fix localhost detection in requires_auth() to avoid substring matches (e.g. "notlocalhost.com" no longer matches) - Fix query param injection to insert before URL fragment - Fix extract_host_from_url for IPv6 bracket notation - Remove misleading schema defaults: Slack limit, Slides insertion_index, Docs index (per-action defaults documented in descriptions instead) Co-Authored-By: Claude Opus 4.6 <[email protected]> * style: Fix cargo fmt formatting Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: IPv6 loopback support for OAuth listener and localhost detection - bind_callback_listener: try [::1] first, fall back to 127.0.0.1, so OAuth redirects work on systems where localhost resolves to ::1 - is_localhost_url: replace manual string parsing with url::Url for correct handling of IPv6 brackets, ports, userinfo, etc. - Add url crate as direct dependency (already a transitive dep) Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: Address PR review feedback on runtime reuse, onboard check, and OAuth binding - Remove session file check from check_onboard_needed(); DATABASE_URL is sufficient - Detect AddrInUse on IPv6 bind and fail immediately instead of falling through to IPv4 - Reuse dedicated tokio runtime across HTTP calls in both tool and channel WASM wrappers Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: HTML-escape provider name in OAuth landing page, simplify Slack limit description - Add html_escape() to prevent XSS in landing_html() where provider_name was interpolated directly into HTML (defense-in-depth, source is trusted but escaping costs nothing) - Remove per-action default numbers from Slack limit field description to avoid confusing LLMs with conflicting defaults Addresses review feedback from zmanian on PR #42. Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: Save all bootstrap fields from wizard, fix config module comment - Wizard now saves secrets_master_key_source and database_pool_size to bootstrap.json (was only saving database_url and onboard_completed, which broke secrets after fresh onboard since SecretsConfig::resolve reads key source from bootstrap) - Update config.rs module doc to reflect bootstrap.json priority chain instead of the removed ~/.ironclaw/.env approach Co-Authored-By: Claude Opus 4.6 <[email protected]> * refactor: Replace BootstrapConfig with .env-based bootstrap DATABASE_URL is the only setting that needs disk persistence before the database is available. Instead of a custom bootstrap.json with 4 fields, use a standard ~/.ironclaw/.env file loaded via dotenvy. - Remove BootstrapConfig struct entirely - Restore ironclaw_env_path(), load_ironclaw_env(), save_database_url() - SecretsConfig::resolve() now auto-detects (env var then keychain probe) instead of reading a saved source from bootstrap.json - DatabaseConfig::resolve() reads DATABASE_URL from env only (dotenvy loads ~/.ironclaw/.env into the environment early in startup) - check_onboard_needed() is now sync (just checks env vars) - Wizard save_and_summarize() works for both postgres and libsql backends - One-time migration from bootstrap.json to .env preserved Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: Ensure load_ironclaw_env() runs in all Config paths, fix .env priority - Config::from_env() and Config::from_db() now call load_ironclaw_env() internally (after dotenvy::dotenv()), so CLI commands like `memory` and `config` correctly load DATABASE_URL from ~/.ironclaw/.env - Fix load order: standard ./.env first (higher priority), then ~/.ironclaw/.env, matching the documented priority chain - Collapse nested if/if-let into let-chains (clippy::collapsible_if) in oauth_defaults.rs, tool.rs, and secrets/store.rs - Fix rename_to_migrated to take &Path instead of &PathBuf Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: Address PR review comments (quoting, SSRF, error mapping) - Quote DATABASE_URL in .env writes so `#` in passwords isn't treated as a dotenv comment (e.g., `DATABASE_URL="postgres://..."`) - Add SSRF defenses to refresh_oauth_token(): require HTTPS, reject private/loopback IPs (with DNS resolution), disable redirects. token_url comes from tool capabilities JSON, so a malicious tool could otherwise exfiltrate refresh tokens. - Fix IPv4 bind error mapping: only map AddrInUse to PortInUse, use generic Io variant for other bind failures Co-Authored-By: Claude Opus 4.6 <[email protected]> --------- Co-authored-by: Claude Opus 4.6 <[email protected]> |
||
|
|
e843c18141 |
feat: add libSQL/Turso embedded database backend (#47)
* feat: add libSQL/Turso database backend with full feature parity Introduce a Database trait abstraction (~60 async methods) enabling compile-time backend selection between PostgreSQL and libSQL/Turso. Convert all modules from concrete Store to Arc<dyn Database>, add LibSqlSecretsStore and LibSqlWasmToolStore implementations, wire libsql stores throughout CLI and main entry points, and make the setup wizard backend-agnostic. Key changes: - src/db/: Database trait, PostgresDatabase adapter, LibSqlBackend with native SQLite-dialect SQL, and idempotent migration system - src/secrets/store.rs: LibSqlSecretsStore (all 8 trait methods) - src/tools/wasm/storage.rs: LibSqlWasmToolStore (all 7 trait methods) - src/main.rs, cli/tool.rs, cli/mcp.rs: backend-conditional wiring - src/setup/channels.rs: SecretsContext uses Arc<dyn SecretsStore> - Feature-gate postgres-only tests and examples Co-Authored-By: Claude Opus 4.6 <[email protected]> * feat: enable onboarding wizard for libSQL builds Refactor the setup wizard to work with both postgres and libsql feature flags. Previously the wizard was gated behind #[cfg(feature = "postgres")] only, so libsql-only builds would print an error on `ironclaw onboard`. - Add libsql fields to Settings (database_backend, libsql_path, libsql_url) - Split wizard database/migration/secrets methods into feature-gated variants - Add step_database_libsql() with local path and Turso remote replica prompts - Update setup/mod.rs and main.rs feature gates to any(postgres, libsql) - Extend check_onboard_needed() to detect libsql database presence Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: address PR review feedback for libSQL backend - P0: Switch libsql_backend to connection-per-operation pattern to fix shared Connection concurrency issue across tokio tasks - P0: Wrap secrets store INSERT+SELECT in transaction to fix TOCTOU race - P0: Document encryption-at-rest limitations and json_patch divergence - P1: Fix get_opt_text removing .filter(|s| !s.is_empty()) that conflated empty strings with NULL - P1: Replace datetime('now') with fmt_ts(&Utc::now()) for consistent RFC 3339 timestamps across all queries - P2: Use explicit _rowid column in FTS5 triggers and joins for stability across VACUUM operations - P2: Add tracing::warn when embedding provided but vector search disabled in hybrid_search - Extract shared connect_from_config() helper to deduplicate DB connection logic across main.rs, cli/config.rs, and cli/mcp.rs Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: add missing JobContext fields and resolve fmt/clippy warnings Add total_tokens_used and max_tokens fields to JobContext in libsql_backend.rs, apply cargo fmt, and fix clippy warnings. Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: review fixes for libSQL backend (shared connections, panics, indexes) - Replace .expect() with proper error propagation in 3 call sites - Share Arc<Database> between backend and stores instead of single Connection - Add connect-per-operation pattern to LibSqlSecretsStore and LibSqlWasmToolStore - Wrap store() INSERT + SELECT-back in a transaction - Add ~22 missing indexes for parity with PostgreSQL schema - Add 18 leak_detection_patterns seed rows matching PostgreSQL V2 migration - Fix super:: import to use crate:: style - Gate mask_password_in_url behind #[cfg(feature = "postgres")] - Rewrite secrets store init with or_else chain for runtime backend selection Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: Resolve clippy lints (collapsible_if, too_many_arguments) Collapse nested if blocks into let_chains to satisfy clippy's collapsible_if lint (CI uses -D warnings). Suppress too_many_arguments on libsql_row_to_tool_at since refactoring the positional index pattern would be a larger change. Co-Authored-By: Claude Opus 4.6 <[email protected]> --------- Co-authored-by: Claude Opus 4.6 <[email protected]> Co-authored-by: Illia Polosukhin <[email protected]> |
||
|
|
ced83d5b4d |
feat: Sandbox jobs (#4)
* Orchestrating jobs and running them in sandboxes * Fix heartbeat: dynamic max_tokens, empty content guard, notification fallback - Query /v1/models API for context_length and set max_tokens to half (floor 4096) instead of hardcoded 1024; reasoning models like GLM-4.7 need much larger budgets - Guard against empty LLM content (reasoning models can burn all tokens on chain-of-thought and return content: null) - Simplify notification routing: try configured channel first, fall back to broadcast_all so heartbeat alerts always reach someone - Add ModelMetadata struct and model_metadata() to LlmProvider trait - Refactor NearAiChatProvider::list_models into shared fetch_models() - Add standalone test_heartbeat example for isolated debugging Co-Authored-By: Claude Opus 4.6 <[email protected]> * Add job detail view with drill-down from jobs list Click a job row to see full details across four sub-tabs: Overview (metadata grid, description, state transitions timeline), Actions (expandable tool call cards with input/output JSON), Thinking (conversation messages styled by role), and Files (embedded workspace tree browser). Co-Authored-By: Claude Opus 4.6 <[email protected]> * Strip model-internal XML tags from LLM responses, fix Telegram parse_mode 400 Some models (GLM-4.7, etc.) emit <tool_call>tool_list</tool_call> in the content field instead of using the OpenAI tool_calls array. This XML leaks through to channels as text, and Telegram's Markdown parser chokes on the underscores, returning 400 "can't parse entities". Two fixes: - Generalize clean_response() to strip <tool_call>, <function_call>, <tool_calls>, and pipe-delimited variants (<|tool_call|>) alongside the existing <thinking> tag stripping - Add Telegram send_message helper with parse_mode fallback: try Markdown first, retry as plain text on "can't parse entities" 400 errors Co-Authored-By: Claude Opus 4.6 <[email protected]> * Add SystemCommand submission type for thread-state-independent commands System commands (/help, /model, /version, /tools, /ping, /debug) now bypass thread-state checks and safety validation via a dedicated Submission::SystemCommand variant. Previously these flowed through process_user_input() which blocked them during Processing/AwaitingApproval /Completed states. - Add /model [name] for runtime model switching with provider validation - Add active_model_name()/set_model() to LlmProvider trait with RwLock hot-swap in both NEAR AI providers - Rewrite /help with aligned columns grouped by category - Expand REPL tab-completion from 10 to 23 slash commands - Remove REPL-local /help interception (now handled by agent) Co-Authored-By: Claude Opus 4.6 <[email protected]> * Add per-tool execution timeouts, auto-create sandbox project dirs, serve built files The sandbox e2e pipeline (agent -> container -> built website -> browsable URL) was broken by three gaps: hardcoded 60s timeouts killed sandbox jobs that need minutes, no auto-created project directory meant container output vanished, and no HTTP route to browse the built files. - Add `execution_timeout()` to the `Tool` trait (default 60s), replace all four hardcoded `Duration::from_secs(60)` call sites (agent_loop, worker, scheduler, worker/runtime) with the per-tool value - Override to 660s in `RunInSandboxTool` (10 min polling + 60s buffer) - Auto-create `~/.ironclaw/projects/{uuid}/` when no `project_dir` is specified, so every sandbox job gets a persistent bind mount - Include `project_dir` and `browse_url` in sandbox tool output JSON - Add `/projects/{id}` and `/projects/{id}/{path}` static file serving routes to the web gateway with path traversal protection and MIME type detection - Add `mime_guess` dependency for content-type detection Co-Authored-By: Claude Opus 4.6 <[email protected]> * Apply cargo fmt to wizard.rs after merge Co-Authored-By: Claude Opus 4.6 <[email protected]> * Persist sandbox jobs in DB, fix web UI, unify job model Sandbox container jobs were invisible to the web UI because they lived only in ContainerJobManager's in-memory HashMap while the API queried ContextManager. This persists them to the agent_jobs table and fixes all six front-end bugs (empty job list, broken back button, empty actions/thinking tabs, wrong files tab, stuck status, no persistence). Key changes: - V4 migration adds project_dir and user_id columns to agent_jobs - Embedded migrations via refinery (no external CLI needed) - SandboxJobRecord CRUD in Store with fire-and-forget DB writes - Unified job_id: sandbox tool generates UUID, passes to ContainerJobManager - Web API queries DB for sandbox jobs, merges with ContextManager direct jobs - New endpoints: restart, project file list/read with path traversal protection - Front-end: rebuild DOM on back navigation, sandbox-aware tabs, job cards in chat stream, source badges, restart button for failed/interrupted jobs - Gateway defaults to enabled, prints Web UI URL on startup - Stale jobs marked "interrupted" on restart for visibility and restartability Co-Authored-By: Claude Opus 4.6 <[email protected]> * Secure in-chat auth: tokens never touch the LLM or chat history Remove the token parameter from tool_auth so the LLM cannot pass raw API keys. Add dedicated REST (POST /api/chat/auth-token) and WebSocket (auth_token) endpoints that route tokens directly to ext_mgr.auth(), completely bypassing the message pipeline, turns, history, and compaction. Web UI shows an auth card (password input + OAuth button) when the agent enters auth mode, submitted via the dedicated endpoint. CLI auth mode interception is unchanged (already secure). New StatusUpdate::AuthRequired/AuthCompleted variants propagate through all channels (SSE, WebSocket, REPL, WASM). Co-Authored-By: Claude Opus 4.6 <[email protected]> * feat: Add Claude Code mode for sandbox jobs Run Claude Code CLI inside Docker containers as an alternative to the standard worker mode. The bridge spawns `claude -p` with stream-json output, posts events to the orchestrator, and supports follow-up prompts via `--resume`. Key additions: - `claude-bridge` CLI subcommand and ClaudeBridgeRuntime - JobMode enum (Worker vs ClaudeCode) with per-mode container config - Orchestrator endpoints for Claude events and prompt polling - SSE event variants for real-time Claude Code streaming to frontend - Claude Code sub-tab in web UI with terminal-style output and input bar - Database migration for job_mode column and claude_code_events table - ClaudeCodeConfig with env var support (CLAUDE_CODE_ENABLED, etc.) - Mode parameter on run_in_sandbox tool schema Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: Skip create_job tool when sandbox is enabled to prevent duplicate jobs When sandbox mode is on, the LLM would call create_job (creating a pending "direct" entry) then run_in_sandbox (creating a second "sandbox" entry), producing two jobs in the list for a single user request. Now register_job_tools() skips create_job when sandbox is enabled since run_in_sandbox already creates tracked jobs. Also improved the run_in_sandbox description to guide the LLM to use it directly and to mention wait=false for async execution. Co-Authored-By: Claude Opus 4.6 <[email protected]> * feat: Web gateway UI quality-of-life improvements Phase 1: Send button disabled state to prevent double-sends, copy button on code blocks, confirm() guards on destructive actions, SSE-driven job list auto-refresh, log filters re-applied on tab switch, jobEvents memory leak fix (cap at 500, cleanup after 60s). Phase 2: Toast notification system replacing chat-based system messages, memory search highlighting with centered snippets, keyboard shortcuts (Ctrl+1-5 tabs, Ctrl+K focus, Ctrl+N new thread, Escape close/blur), activity tab toolbar with event type filter and auto-scroll toggle. Phase 3: Thread sidebar with load/switch/create, thread_id passed with messages, collapsible to hamburger. Memory inline editing with textarea, Save/Cancel, POST to /api/memory/write. Phase 4: Gateway status popover on hover (polls every 30s), extension install form (name/URL/kind), markdown rendering in memory viewer for .md files, mobile responsive layout at 768px breakpoint. Co-Authored-By: Claude Opus 4.6 <[email protected]> * feat: Add routines system, remove non-sandbox job mode from web UI Routines: scheduled & reactive job system with cron and event triggers, lightweight (single LLM call) and full-job execution modes, guardrails (cooldown, max concurrent, dedup), and LLM-facing tools for CRUD. Web UI: remove ContextManager-backed "direct" job mode entirely. Jobs are now exclusively sandbox-backed (DB + container). Simplify job detail response, drop dead types (ActionInfo, MessageInfo, MessageToolCallInfo), fix Browse Files CSS loading (trailing-slash redirect), fix Activity tab event rendering. Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: Re-enable chat input on agent completion, auto-auth on tool_activate, recover tool calls from content XML Three fixes: 1. Chat input stays disabled after agent finishes: the "Done" status SSE event now calls enableChatInput() as a safety net when the response event is empty or lost. Same for auth_completed and cancelAuth(). 2. tool_activate never triggers auth: when activation fails due to missing authentication, it now auto-initiates the auth flow (same pattern as the web API handler). detect_auth_awaiting() also matches tool_activate results now. 3. Models like GLM-4.7 emit tool calls as XML tags in content (<tool_call>tool_list</tool_call>) instead of using the structured tool_calls array. recover_tool_calls_from_content() extracts and validates these before falling back to plain text. Co-Authored-By: Claude Opus 4.6 <[email protected]> * feat: Add routines web UI tab, update docs for sandbox-jobs branch Add full routines management to the web gateway (list, detail, trigger, toggle, delete) with 7 new API endpoints, response types, and frontend (HTML, JS, CSS). Update FEATURE_PARITY.md (~23 rows), CLAUDE.md (new subsystems, config, TODOs), and README.md (architecture diagram, features, components, fix onboard command). Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: Bind Telegram bot to owner account during setup Without owner binding, anyone who discovers the bot can send it messages. The setup wizard now prompts the user to message their bot, captures their Telegram user ID via getUpdates, and persists it as telegram_owner_id in settings. On startup, the owner_id is injected into the WASM channel config so the existing owner restriction logic drops messages from non-owners. Co-Authored-By: Claude Opus 4.6 <[email protected]> * feat: Move settings from disk to PostgreSQL database Settings previously lived in three JSON files on disk (settings.json, mcp-servers.json, session.json). This made them inaccessible from the web UI and caused redundant disk reads (Settings::load() called 8+ times during startup). Now all settings live in a `settings` table (user_id + key -> JSONB) with only 4 bootstrap fields remaining on disk (database_url, pool size, secrets key source, onboard_completed) since they're needed before the DB connection exists. - Add V8 migration for settings table - Add BootstrapConfig (thin disk file) and Settings DB round-trip - Add Store CRUD methods for settings (get/set/delete/list/bulk) - Refactor Config to load from DB (env > DB > default cascade) - Add SessionManager DB persistence for session tokens - Add DB-backed MCP server config load/save functions - Add 6 settings web API endpoints (list/get/set/delete/export/import) - Add one-time disk-to-DB migration on first boot - Make CLI config commands async with DB access (disk fallback) Co-Authored-By: Claude Opus 4.6 <[email protected]> * feat: Seed workspace on boot, fix gateway duplicate logs and URL auto-auth - Add Workspace::seed_if_empty() to create core identity files (README, MEMORY, IDENTITY, SOUL, AGENTS, USER, HEARTBEAT) when missing, called on every boot without overwriting existing user edits - Remove duplicate gateway log lines from web/mod.rs (main.rs has the useful clickable ?token= URL) - Auto-authenticate from ?token= URL parameter in the web UI and strip the token from the address bar after successful auth Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: Harden sandbox security (path traversal + orchestrator auth) Two vulnerabilities fixed: 1. project_dir path traversal: The create_job tool let the LLM specify arbitrary host paths for Docker bind mounts. Removed project_dir from the tool schema entirely, and added canonicalization + prefix validation at both resolve_project_dir() and the job_manager bind mount point. 2. Orchestrator API auth bypass: worker_auth_middleware was defined but never applied. Each handler manually called validate_token(), so any new endpoint that forgot would be publicly accessible. Applied the middleware as route_layer on all /worker/ routes, removed manual auth from all 7 handlers. Bind to 127.0.0.1 on macOS/Windows (Linux keeps 0.0.0.0 since containers reach host via docker bridge, not loopback). Co-Authored-By: Claude Opus 4.6 <[email protected]> * feat: Rework gateway chat with pinned assistant, pagination, and NEAR AI response chaining Implements the 4-phase plan for overhauling the web gateway chat: - Phase 1: Pinned "Assistant" thread at top of sidebar, regular threads below - Phase 2: Cursor-based history pagination with infinite scroll - Phase 3: NEAR AI previous_response_id chaining (delta-only messages), with fallback to full history on chain errors, and DB persistence of chain state across restarts - Phase 4: SSE thread isolation (events filtered by thread_id) Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: Add per-request HTTP timeout to WASM host, redact credentials in errors Three fixes for WASM channel reliability: 1. Per-request timeout: Add optional timeout-ms parameter to http-request in both channel and tool WIT interfaces. Telegram long-poll now specifies 35s (outliving the 30s server-side hold), while regular API calls use the 30s default. Fixes the triple-30s timeout race that caused polling failures. 2. Credential redaction: reqwest::Error includes the full URL (with injected bot tokens) in its Display output. Scrub credential values from error messages before logging or returning to WASM. 3. Webhook route registration: Remove tunnel URL gate so webhook routes are always available when webhook channels exist, not only when TUNNEL_URL is configured. Co-Authored-By: Claude Opus 4.6 <[email protected]> * chore: Fix clippy warnings in WASM tools and channels - slack channel: allow dead_code on signing_secret_name (forward compat field) - gmail tool: use div_ceil() instead of manual (n+2)/3 - google-calendar tool: extract CreateEventParams/UpdateEventParams structs to fix too-many-arguments warnings Co-Authored-By: Claude Opus 4.6 <[email protected]> * Fix approval flow * fix: Rebuild bundled telegram.wasm with updated WIT interface The bundled WASM binary must match the host's WIT definition. Previous binary was compiled against the old 4-arg http-request; this rebuild includes the new timeout-ms parameter. Co-Authored-By: Claude Opus 4.6 <[email protected]> * refactor: Load WASM channels from disk instead of bundling in binary Remove include_bytes! embedding of telegram.wasm. Channels are now loaded from their build output directories (channels-src/<name>/target/) during onboarding, then from ~/.ironclaw/channels/ at runtime. - bundled.rs: locate_channel_artifacts() finds WASM + capabilities from build output; IRONCLAW_CHANNELS_SRC env var overrides the default path - available_channel_names(): only lists channels with build artifacts - bundled_channel_names(): lists all known channels (manifest) - Setup wizard uses available_channel_names() to offer installable channels - Add *.wasm to .gitignore, remove tracked telegram.wasm Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: Persist gateway auth token, fix thread hydration race, polish auth screen Three web gateway UX fixes: 1. Token persistence: Store auth token in sessionStorage so refreshing the page doesn't force re-authentication. Hide the auth screen immediately when a saved token exists to prevent flash. 2. Thread hydration: Remove the !msgs.is_empty() bail-out in maybe_hydrate_thread so that even brand-new (empty) assistant threads get hydrated with their correct DB UUID. Previously resolve_thread would mint a fresh UUID, causing messages to land in the wrong conversation and duplicate threads to appear. 3. Auth screen: Redesign as a centered card with brand, tagline, labeled input, and hint text. Also adds 34 new tests covering session/thread lifecycle, thread resolution isolation (user, channel, external ID), hydration edge cases, serialization round-trips, approval flows, and stale mapping recovery. Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: Use bindgen! for WASM tool wrapper, add dev tool loading Three changes: 1. Rewrite src/tools/wasm/wrapper.rs to use wasmtime::component::bindgen! instead of manual linker.root().func_wrap(). This fixes the "component imports instance 'near:agent/host', but a matching implementation was not found in the linker" error. All 6 host functions (log, now-millis, workspace-read, http-request, secret-exists, tool-invoke) are now properly registered under the near:agent/host namespace. Also adds WASI support, credential injection, and leak detection for HTTP requests made by WASM tools. 2. Add dev tool loading to src/tools/wasm/loader.rs. During startup, the loader now also scans tools-src/*/target/wasm32-wasip2/release/ for build artifacts that are newer than installed copies. This means during development you just rebuild the WASM and restart the host; no manual copy step needed. Set IRONCLAW_TOOLS_SRC to override the source dir. 3. Wire up load_dev_tools() in main.rs alongside the existing load_from_dir() call. Co-Authored-By: Claude Opus 4.6 <[email protected]> * feat: Wire main startup and CLI to use DB-backed settings main.rs now reloads Config from the database after connecting, attaches the store to the session manager for dual-write tokens, and loads MCP servers from DB instead of disk. ExtensionManager and MCP CLI commands use DB when available with disk fallback. Co-Authored-By: Claude Opus 4.6 <[email protected]> --------- Co-authored-by: Claude Opus 4.6 <[email protected]> |
||
|
Illia PolosukhinandClaude Opus 4.5
|
974bc8d407 |
Add hosted MCP server support with OAuth 2.1 and token refresh
Enables connecting to official MCP servers (like Notion) instead of building custom WASM tools. Uses OAuth 2.1 with PKCE and supports Dynamic Client Registration for zero-config authentication. Key features: - OAuth 2.1 flow with PKCE for secure browser-based auth - Dynamic Client Registration (DCR) for servers without pre-configured clients - Automatic token refresh on 401 responses - Session management with Mcp-Session-Id headers - SSE streaming response handling New CLI commands: - `mcp add <name> <url>` - Add an MCP server - `mcp remove <name>` - Remove an MCP server - `mcp list` - List configured servers - `mcp auth <name>` - Authenticate with a server - `mcp test <name>` - Test connection Also removes the Notion WASM tool example since it's superseded by the Notion MCP server which provides 13 official tools. Co-Authored-By: Claude Opus 4.5 <[email protected]> |
||
|
Illia PolosukhinandClaude Opus 4.5
|
598dd43b1c |
Rebrand to IronClaw with security-first mission
Renamed project from "near-agent" to "ironclaw" throughout the codebase. Updated documentation to emphasize the core philosophy: - Your data stays yours (local, encrypted, no telemetry) - Self-expanding capabilities (build tools on the fly) - Defense in depth (WASM sandbox, prompt injection defense) - Always on user's side Key changes: - Package name: near-agent -> ironclaw - Config paths: ~/.near-agent/ -> ~/.ironclaw/ - Database name in docs: near_agent -> ironclaw - CLI binary: near-agent -> ironclaw - Log filters: RUST_LOG=near_agent -> RUST_LOG=ironclaw - All user-facing strings (welcome messages, help text, etc.) Preserved for compatibility: - HKDF salt "near-agent-secrets-v1" (changing would break existing secrets) - WIT interface names (near::agent::*) - NEAR AI provider config (NEARAI_* env vars) Co-Authored-By: Claude Opus 4.5 <[email protected]> |
||
|
Illia Polosukhin
|
74d94d33b0 | Implementing channels to be handled in wasm | ||
|
Illia PolosukhinandClaude Opus 4.5
|
4ae59ef52c |
Fix TuiChannel integration and enable in main.rs
- Store event_tx in Arc<Mutex<>> so respond() can send to TUI - Fix run_event_loop to take owned receiver - Switch main.rs from SimpleCliChannel to TuiChannel - Add proper terminal cleanup on TUI error Co-Authored-By: Claude Opus 4.5 <[email protected]> |
||
|
Illia PolosukhinandClaude Opus 4.5
|
32bfd24154 |
Add WASM sandbox secure API extension
Extends the WASM sandbox with HTTP API capabilities, secrets management, tool aliasing, and leak detection. Key security principle: WASM never sees credentials, injection happens at host boundary. New modules: - secrets: AES-256-GCM encrypted storage with HKDF key derivation - leak_detector: Aho-Corasick + regex pattern matching for secret exfiltration - capabilities: Extended capability system (HTTP, ToolInvoke, Secrets) - allowlist: HTTP endpoint validation with glob patterns - credential_injector: Host-boundary credential injection - rate_limiter: Sliding window per-tool rate limiting - storage: WASM binary storage with BLAKE3 integrity verification Leak detection happens at two points: 1. Before HTTP request (prevents exfiltration via URL/headers/body) 2. After response (prevents exposure in outputs returned to WASM) Co-Authored-By: Claude Opus 4.5 <[email protected]> |
||
|
Illia Polosukhin
|
aea3f47f8b | Implementing WASM runtime |