diff --git a/docs/development-history.md b/docs/development-history.md index 90c274d8..e6e7678a 100644 --- a/docs/development-history.md +++ b/docs/development-history.md @@ -96,6 +96,30 @@ The pivotal architectural change. Motivated by the question: *"What if we move s **Final state**: 189 tests pass, zero clippy warnings. The Python orchestrator is the execution engine. The Rust layer is the kernel. +## Session 7: Integration Scaling Research (2026-03-26) + +Studied [Pica](https://github.com/withoneai/pica) (formerly IntegrationOS, 200+ third-party API integrations) to understand how to rapidly scale the number of available integrations in IronClaw. + +**Pica's architecture**: Integrations are MongoDB documents, not code. Each platform has a `ConnectionDefinition` (identity + auth schema) and N `ConnectionModelDefinition` records (one per API endpoint: URL, method, auth method, schemas, JS transform functions). A generic executor dispatches requests. OAuth definitions embed JavaScript compute functions executed by a TypeScript service. Adding a new platform = inserting documents, no code changes. + +**Analysis of IronClaw v1 tools**: Audited all 37 built-in tools. Only 3 (image_gen, image_analyze, image_edit) are HTTP API wrappers. The other 34 are local computation, filesystem, orchestration, or system management — none convertible to data-driven definitions. The value isn't converting existing tools; it's enabling hundreds of new integrations. + +**Key finding — deterministic executors don't solve the LLM problem**: Even with a Pica-style executor, each integration action must be registered as a tool in the LLM's context. At 200+ tools: +- ~20,000 tokens always-on cost (tool definitions sent every request) +- LLM tool selection accuracy degrades beyond ~20-30 tools +- The LLM still constructs parameters and can get them wrong +- Deterministic execution only helps *after* the LLM correctly selects the tool and params + +**The realization**: In engine v2, Capabilities already bundle actions + knowledge. For API integrations, a Capability's knowledge text teaches the LLM how to call the platform's API using the generic `http` action. This is superior to dedicated tools because: +- Tool list stays small (just `http` + core actions) — high selection accuracy +- Knowledge loaded on-demand per thread context — zero cost for unused integrations +- ~350 tokens of knowledge covers 4+ API endpoints (the LLM generalizes) +- Adding a new platform = writing markdown knowledge, no Rust code + +**Remaining gap**: OAuth token acquisition requires a dedicated `oauth_init` action (LLM can't do redirect flows). Capability knowledge instructs the LLM to call it before using the API. + +**Decision**: Use Capabilities as knowledge-bearing integration definitions. Write knowledge text for top 20 platforms. Build one `oauth_init` action. Skip the Pica-style deterministic executor — it solves the wrong problem for LLM agents. + ## Architecture Evolution ``` @@ -105,6 +129,8 @@ Session 4: + Self-improvement Mission (fires on issues, fixes prompts) Session 5: + Autoresearch-style goal prompt (concrete, not vague) Session 6: Rust loop → Python orchestrator (self-modifiable) 900 lines Rust → 80 lines Rust bootstrap + 230 lines Python +Session 7: Integration scaling: Capabilities as knowledge → http action + (not Pica-style per-action tools — tool list bloat kills LLM accuracy) ``` ## Key Commits diff --git a/docs/engine-v2-architecture.md b/docs/engine-v2-architecture.md index 01790815..e8c0378f 100644 --- a/docs/engine-v2-architecture.md +++ b/docs/engine-v2-architecture.md @@ -215,6 +215,117 @@ ReadLocal, ReadExternal, WriteLocal, WriteExternal, CredentialedNetwork, Compute, Financial ``` +## Integration Scaling Strategy + +### The Problem: Tool List Bloat + +A naive approach to adding third-party integrations (Slack, GitHub, Stripe, etc.) is to register each API action as a separate tool — `slack_post_message`, `slack_list_channels`, `github_create_issue`, etc. This fails for LLM-based agents: + +- Each tool definition costs ~80-120 tokens in the tool list, sent on **every request** +- 200 actions = ~20,000 tokens always-on context cost +- LLM tool selection accuracy **degrades significantly** beyond ~20-30 tools +- The LLM still has to construct correct parameters — deterministic execution doesn't help if the LLM picks the wrong tool or hallucinates params + +This was confirmed by studying [Pica](https://github.com/withoneai/pica) (formerly IntegrationOS), which supports 200+ platforms via data-driven definitions in MongoDB. Pica's approach works for programmatic API access, but registering all those actions as LLM tools would degrade agent performance. + +### The Solution: Capabilities as Knowledge-Bearing Definitions + +In engine v2, Capabilities replace both Tools and Skills. A Capability bundles **actions** (what it can do) with **knowledge** (how to do it). For API integrations, this means: + +1. The `http` action is always available (one tool in the LLM's action list) +2. Each integration is a Capability with knowledge text that teaches the LLM how to call that platform's API +3. Capabilities are loaded on-demand based on thread context, not registered globally +4. The LLM reads the knowledge, constructs the correct `http` call + +``` +User: "post hello to #general on slack" + ↓ +Capability activation: "slack-api" loaded into thread context + ↓ +LLM reads knowledge: learns endpoints, auth pattern, body format + ↓ +LLM calls `http` action: + POST https://slack.com/api/chat.postMessage + headers: {"Authorization": "Bearer {SLACK_BOT_TOKEN}"} + body: {"channel": "C01234", "text": "hello"} + ↓ +EffectExecutor: policy check → credential injection → SSRF protection → leak detection → response +``` + +### Token Cost Comparison + +| Scenario | Dedicated Tools (200 actions) | Capability + http | +|---|---|---| +| User asks about Slack | ~20,000 (all tools in list) | ~700 (http action + slack knowledge) | +| User asks about nothing | ~20,000 (still there) | ~200 (just http action) | +| Tool selection accuracy | Degrades with count | Always picks `http` — no confusion | +| Adding a new platform | Define N tool schemas + executor | Write knowledge text (markdown) | + +### What a Capability Definition Looks Like + +```yaml +name: slack-api +description: Slack Web API — post messages, manage channels, search, react +knowledge: | + Base: `https://slack.com/api` + Auth header: `Authorization: Bearer {SLACK_BOT_TOKEN}` + All POST bodies are JSON with `Content-Type: application/json`. + + **Post message**: POST `/chat.postMessage` body `{"channel":"","text":""}` + **List channels**: GET `/conversations.list?types=public_channel&limit=100` + **Search**: GET `/search.messages?query=` + **Add reaction**: POST `/reactions.add` body `{"channel":"","timestamp":"","name":""}` + + All responses: `{"ok": true, ...}` or `{"ok": false, "error": ""}`. + Paginate with `cursor` param when `response_metadata.next_cursor` is non-empty. +actions: [http] +effects: [CredentialedNetwork, ReadExternal, WriteExternal] +policies: + requires_secret: SLACK_BOT_TOKEN +``` + +~350 tokens of knowledge covers 4+ actions. The LLM generalizes the pattern to other Slack endpoints from training data. + +### Classification of v1 Built-in Tools + +Studied all 37 v1 built-in tools to determine which fit the knowledge-driven pattern: + +**Can be knowledge-driven (HTTP API wrappers):** +- `image_gen`, `image_analyze`, `image_edit` — pure HTTP calls to external APIs with auth + +**Already a generic action (the execution engine):** +- `http` — the action that knowledge-driven Capabilities delegate to + +**Must remain dedicated actions (complex local logic):** +- `shell` — 4-layer command validation, Docker sandbox, environment scrubbing +- `file` (read/write/list/patch) — local filesystem with path traversal prevention +- `memory_*` — hybrid FTS + vector search, prompt injection detection +- `job_*` — Docker container lifecycle, context isolation +- `routine_*` — database-backed CRON scheduling +- `extension_tools`, `skill_tools` — registry and system management +- `secrets_tools` — encrypted store management +- `json`, `time`, `echo` — pure local computation +- `message`, `restart`, `tool_info` — internal agent control + +**Takeaway**: Only 3 of 37 existing tools are HTTP wrappers. The value is not converting existing tools — it's enabling hundreds of **new** integrations (Slack, GitHub, Jira, Stripe, Salesforce, etc.) without writing Rust. + +### Where Dedicated Actions Still Win + +1. **Autonomous/headless threads** — Missions and background threads with no human oversight benefit from deterministic execution for their 1-2 critical integrations. Register those specific actions via leases. +2. **OAuth token acquisition** — The LLM cannot perform redirect-based OAuth flows. A dedicated `oauth_init` action handles the redirect dance and stores tokens in the secrets system. The Capability knowledge then instructs the LLM to call `oauth_init` before using the API. +3. **High-frequency reliability-critical paths** — If a specific integration is called thousands of times and must never fail, a dedicated action avoids LLM reasoning variance. + +### Comparison with Pica's Approach + +[Pica](https://github.com/withoneai/pica) uses a data-driven model where each API action is a MongoDB document (`ConnectionModelDefinition`) with base URL, path, method, auth method, schemas, and JavaScript transform functions. A generic executor dispatches requests. Key patterns: + +- **Handlebars secret injection** — entire definition rendered as template with user's secrets as context +- **Passthrough + Unified dual mode** — raw HTTP proxy or normalized CRUD via CommonModels +- **JS sandbox transforms** — `fromCommonModel`/`toCommonModel` functions for data mapping +- **`knowledge` field** — free-text documentation per action for AI tool discovery + +Pica's model is optimized for programmatic API access (SDK calls from code). For LLM agents, the Capability-as-knowledge approach is superior because it avoids tool list bloat while leveraging the LLM's ability to construct HTTP calls from documentation. The two approaches share the insight that **integrations should be data, not code**. + ## Key Files | File | Purpose | @@ -248,5 +359,6 @@ cargo test # full suite - **karpathy/autoresearch** — the self-improvement loop as a program.md, fixed-budget evaluation, git as state machine - **Official RLM impl** (alexzhang13/rlm) — 30 max iterations, compaction at 85%, budget inheritance - **fast-rlm** (avbiswas/fast-rlm) — Step 0 orientation, parallel sub-calls, dual model routing +- **Pica/IntegrationOS** (withoneai/pica) — data-driven integration definitions, Handlebars secret injection, knowledge fields for AI tool discovery. Validated the "integrations as data" principle; diverged on execution model (knowledge-driven Capabilities instead of per-action tool registration) See also: `docs/plans/2026-03-20-engine-v2-architecture.md` for the full 8-phase roadmap.