Files
optimclaw/src/tools
8751a5a9bc feat: add web_fetch built-in tool (#435)
* feat: add web_fetch built-in tool and web-fetch skill

- New web_fetch Rust built-in tool (GET-only, auto-approved, structured
  output: url/title/content/word_count) with HTML to Markdown via Readability
- Full SSRF protection: HTTPS-only, no private IPs, DNS rebinding defence,
  outbound/inbound leak scanning, 5 MB cap, no redirect following
- Rate limited: 30 req/min, 500/hr (same as http tool)
- Protected tool name; registered in register_builtin_tools()
- validate_url made pub(crate) so web_fetch can reuse it from http.rs
- New skills/web-fetch/SKILL.md for agent guidance on web browsing
- Fixes unicode panic in extract_title: use to_ascii_lowercase not
  to_lowercase to preserve byte offsets when indexing original string

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

* chore: remove web-fetch skill (tool description is self-sufficient)

The web_fetch tool's schema description already tells the LLM when and
how to use it. A SKILL.md would only add redundant prompt context.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

* fix: include HTTP status in web_fetch output

The LLM had no way to distinguish a 404 error page from a 200 success.
Including status in the structured output (alongside url/title/content/
word_count) lets the agent report failures correctly and matches the
behaviour of the http tool which always returns status.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

* feat(web_fetch): add Chrome UA and safe redirect following

- Set a Chrome-like User-Agent so sites that block the default reqwest
  string return real content instead of bot-rejection pages.
- Add Accept: text/markdown, text/html header (mirrors OpenClaw).
- Follow up to 3 redirects manually instead of blocking all 3xx.
  Every Location URL is run through validate_url() before the next
  request is sent, so SSRF protection applies to every hop identically
  to how it applies to the original URL.
- Resolve relative Location values against the current URL before
  SSRF-validating them.
- Log each followed hop at DEBUG level.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

* fix(web_fetch): expose final_url after redirect following

When redirects are followed, the original `url` field no longer
reflects where the content actually came from. Add `final_url` so
the LLM can cite the canonical source correctly. Equals `url` when
no redirects occurred.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

* fix(web_fetch): address review comments and fix CI failures

- Store LeakDetector in WebFetchTool struct (init once in new(), not per execute() call)
- Use self.leak_detector for both outbound scan and redirect re-validation
- Simplify HTML/cfg blocks to reduce duplication (gemini-code-assist suggestion)
- Fix pub use ordering in mod.rs (cargo fmt)
- Add web_fetch to core_registration_covers_expected_tools snapshot test

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

---------

Co-authored-by: Claude Sonnet 4.6 <[email protected]>
2026-02-28 19:58:22 -08:00
..

Tool System

Adding a New Tool

Built-in Tools (Rust)

  1. Create src/tools/builtin/my_tool.rs
  2. Implement the Tool trait
  3. Add mod my_tool; and pub use in src/tools/builtin/mod.rs
  4. Register in ToolRegistry::register_builtin_tools() in registry.rs
  5. Add tests

WASM tools are the preferred way to add new capabilities. They run in a sandboxed environment with explicit capabilities.

  1. Create a new crate in tools-src/<name>/
  2. Implement the WIT interface (wit/tool.wit)
  3. Create <name>.capabilities.json declaring required permissions
  4. Build with cargo build --target wasm32-wasip2 --release
  5. Install with ironclaw tool install path/to/tool.wasm

See tools-src/ for examples.

Tool Architecture Principles

CRITICAL: Keep tool-specific logic out of the main agent codebase.

The main agent provides generic infrastructure; tools are self-contained units that declare their requirements through capabilities files.

What Goes in Tools (capabilities.json)

  • API endpoints the tool needs (HTTP allowlist)
  • Credentials required (secret names, injection locations)
  • Rate limits and timeouts
  • Auth setup instructions (see below)
  • Workspace paths the tool can read

What Does NOT Go in Main Agent

  • Service-specific auth flows (OAuth for Notion, Slack, etc.)
  • Service-specific CLI commands (auth notion, auth slack)
  • Service-specific configuration handling
  • Hardcoded API URLs or token formats

Tool Authentication

Tools declare their auth requirements in <tool>.capabilities.json under the auth section. Two methods are supported:

OAuth (Browser-based login)

For services that support OAuth, users just click through browser login:

{
  "auth": {
    "secret_name": "notion_api_token",
    "display_name": "Notion",
    "oauth": {
      "authorization_url": "https://api.notion.com/v1/oauth/authorize",
      "token_url": "https://api.notion.com/v1/oauth/token",
      "client_id_env": "NOTION_OAUTH_CLIENT_ID",
      "client_secret_env": "NOTION_OAUTH_CLIENT_SECRET",
      "scopes": [],
      "use_pkce": false,
      "extra_params": { "owner": "user" }
    },
    "env_var": "NOTION_TOKEN"
  }
}

To enable OAuth for a tool:

  1. Register a public OAuth app with the service (e.g., notion.so/my-integrations)
  2. Configure redirect URIs: http://localhost:9876/callback through http://localhost:9886/callback
  3. Set environment variables for client_id and client_secret

Manual Token Entry (Fallback)

For services without OAuth or when OAuth isn't configured:

{
  "auth": {
    "secret_name": "openai_api_key",
    "display_name": "OpenAI",
    "instructions": "Get your API key from platform.openai.com/api-keys",
    "setup_url": "https://platform.openai.com/api-keys",
    "token_hint": "Starts with 'sk-'",
    "env_var": "OPENAI_API_KEY"
  }
}

Auth Flow Priority

When running ironclaw tool auth <tool>:

  1. Check env_var - if set in environment, use it directly
  2. Check oauth - if configured, open browser for OAuth flow
  3. Fall back to instructions + manual token entry

The agent reads auth config from the tool's capabilities file and provides the appropriate flow. No service-specific code in the main agent.

WASM Tools vs MCP Servers: When to Use Which

Both are first-class in the extension system (ironclaw tool install handles both), but they have different strengths.

WASM Tools (IronClaw native)

  • Sandboxed: fuel metering, memory limits, no access except what's allowlisted
  • Credentials injected by host runtime, tool code never sees the actual token
  • Output scanned for secret leakage before returning to the LLM
  • Auth (OAuth/manual) declared in capabilities.json, agent handles the flow
  • Single binary, no process management, works offline
  • Cost: must build yourself in Rust, no ecosystem, synchronous only

MCP Servers (Model Context Protocol)

  • Growing ecosystem of pre-built servers (GitHub, Notion, Postgres, etc.)
  • Any language (TypeScript/Python most common)
  • Can do websockets, streaming, background polling
  • Cost: external process with full system access (no sandbox), manages own credentials, IronClaw can't prevent leaks

Decision guide:

Scenario Use
Good MCP server already exists MCP
Handles sensitive credentials (email send, banking) WASM
Quick prototype or one-off integration MCP
Core capability you'll maintain long-term WASM
Needs background connections (websockets, polling) MCP
Multiple tools share one OAuth token (e.g., Google suite) WASM

The LLM-facing interface is identical for both (tool name, schema, execute), so swapping between them is transparent to the agent.