From b621cd503d490e48411f06a067b763e8599cf4fa Mon Sep 17 00:00:00 2001 From: "ilblackdragon@gmail.com" Date: Sun, 22 Mar 2026 14:49:52 -0700 Subject: [PATCH] docs: add crate extraction & cleanup roadmap MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Documents architectural recommendations from the engine v2 design process for future reference: - Root directory consolidation (channels-src + tools-src → extensions/) - Crate extraction tiers: zero-coupling (estimation, observability, tunnel), trivial-coupling (document_extraction, pairing, hooks), medium-coupling (secrets, MCP, db, workspace, llm, skills), heavy-coupling (web gateway, agent, extensions) - src/ module reorganization into logical groups (core, persistence, infra, media, support) - main.rs/app.rs slimming targets (100/500 lines after migration) - WASM module candidates (document_extraction) and non-candidates (REPL, web gateway → separate crates instead) - Priority ordering for extraction work - Tracks completed items (ironclaw_safety, ironclaw_engine, transcription move) Co-Authored-By: Claude Opus 4.6 (1M context) --- ...2026-03-22-crate-extraction-and-cleanup.md | 161 ++++++++++++++++++ 1 file changed, 161 insertions(+) create mode 100644 docs/plans/2026-03-22-crate-extraction-and-cleanup.md diff --git a/docs/plans/2026-03-22-crate-extraction-and-cleanup.md b/docs/plans/2026-03-22-crate-extraction-and-cleanup.md new file mode 100644 index 00000000..83a7bc84 --- /dev/null +++ b/docs/plans/2026-03-22-crate-extraction-and-cleanup.md @@ -0,0 +1,161 @@ +# Crate Extraction & Codebase Cleanup Roadmap + +**Date:** 2026-03-22 +**Status:** Recommendations (some already completed) +**Context:** Architectural analysis of IronClaw's module boundaries, coupling, and organization. These recommendations emerged from the engine v2 design process. + +--- + +## Root-Level Directory Cleanup + +Current root has 30+ items. Proposed consolidation: + +| Current | Proposed | Rationale | +|---------|----------|-----------| +| `channels-src/` + `tools-src/` | `extensions/channels/` + `extensions/tools/` | Unified "extensions" directory for all WASM modules | +| `deploy/` + `docker/` + `scripts/` + `wix/` | `infra/` subdirectories | Build/deploy infrastructure grouped | +| Everything else | Stays | `crates/`, `src/`, `tests/`, `benches/`, `fuzz/`, `migrations/`, `registry/`, `skills/`, `wit/`, `docs/` | + +--- + +## Crate Extraction Tiers + +### Tier 1: Zero coupling — extract immediately + +These modules have no `crate::` imports from the rest of the codebase: + +| Module | Lines | Notes | +|--------|-------|-------| +| `src/estimation/` | ~36 | Pure math (EMA learning). Could be a general-purpose crate | +| `src/observability/` | ~28 | Self-contained Observer trait + impls. Only references itself | +| `src/tunnel/` | ~56 | Clean Tunnel trait, only needs anyhow + tokio | + +### Tier 2: Trivial coupling — one interface to break + +| Module | Lines | Coupling | How to break | +|--------|-------|----------|-------------| +| `src/transcription/` | ~727 | `crate::channels::{AttachmentKind, IncomingMessage}` | **DONE** — moved to `src/llm/transcription/` in staging (PR #1559). Could further extract to `ironclaw_media` crate | +| `src/document_extraction/` | ~798 | `crate::channels::{AttachmentKind, IncomingMessage}` | Extract `AttachmentKind` to shared types | +| `src/pairing/` | ~917 | `crate::bootstrap::ironclaw_base_dir` | Pass base_dir as parameter instead of importing | +| `src/hooks/` | ~84 | Light | Define Hook trait in shared types | + +### Tier 3: Medium coupling — need `ironclaw_types` crate first + +| Module | Lines | Dependencies to untangle | +|--------|-------|--------------------------| +| `src/secrets/` | ~88 | Encryption is self-contained, needs config types | +| `src/tools/mcp/` | ~3K | Generic MCP protocol client. **Highly reusable** outside IronClaw | +| `src/db/` | ~256 | Trait-based (`Database`), needs shared types for schema | +| `src/workspace/` | ~240 | Depends on db + embedding, but has clean `Workspace` trait | +| `src/llm/` | ~888 | Trait-based (`LlmProvider`), depends on config types | +| `src/skills/` | ~120 | Depends on filesystem + trust model | + +### Tier 4: Heavy coupling — longer term + +| Module | Lines | Why it's hard | +|--------|-------|---------------| +| `src/channels/web/` | ~160K | Imports agent, db, extensions, skills, tools, workspace, orchestrator | +| `src/agent/` | ~3K | Core — everything flows through it | +| `src/extensions/` | ~10K | Orchestrates tools + channels + WASM | + +--- + +## src/ Module Reorganization + +Too many top-level concepts. Proposed grouping: + +``` +src/ +├── core/ # The agent brain +│ ├── agent/ # Agent loop, dispatcher, scheduler +│ ├── context/ # Job context isolation +│ └── evaluation/ # Success evaluation +│ +├── channels/ # I/O surface (as-is, well-structured) +│ +├── tools/ # Tool system (as-is) +│ +├── llm/ # LLM providers +│ └── transcription/ # ← DONE (moved from src/transcription/) +│ +├── media/ # Content processing +│ └── document_extraction/ # PDF/DOCX → text +│ +├── persistence/ # Data layer +│ ├── db/ +│ ├── workspace/ +│ ├── history/ +│ └── secrets/ +│ +├── infra/ # Infrastructure +│ ├── config/ +│ ├── bootstrap.rs +│ ├── settings.rs +│ ├── service.rs +│ ├── tunnel/ +│ ├── sandbox/ +│ ├── orchestrator/ +│ └── worker/ +│ +├── extensions/ # Extension system +│ ├── registry/ +│ ├── skills/ +│ ├── hooks/ +│ └── extensions/ # Manager +│ +├── support/ # Small utilities +│ ├── observability/ +│ ├── estimation/ +│ ├── profile.rs +│ ├── timezone.rs +│ └── util.rs +│ +├── bridge/ # ← NEW (engine v2 bridge) +└── cli/ # CLI subcommands +``` + +--- + +## The `main.rs` / `app.rs` Problem + +These files are ~44K and ~37K lines. After engine v2 migration (Phase 7-8): +- `main.rs` should be ~100 lines (parse CLI args, call `app::run()`) +- `app.rs` should be ~500 lines (construct dependencies, wire crates, start event loop) +- All logic lives in crates / modules + +--- + +## WASM Module Candidates + +### Already WASM (channels-src/, tools-src/) +Discord, Slack, Telegram, Feishu, WhatsApp channels + 11 tools. Mature WIT interfaces. + +### Could become WASM tools +| Candidate | Rationale | +|-----------|-----------| +| `document_extraction` | Pure input→output transform. Takes bytes + mime_type, returns text | + +### Cannot become WASM +| Module | Reason | +|--------|--------| +| REPL (`src/channels/repl.rs`) | Needs terminal I/O (rustyline, crossterm). Can become a separate **crate** | +| Web gateway (`src/channels/web/`) | 160K lines, deep coupling. Can become a separate **crate** | + +--- + +## Priority Order + +1. **`ironclaw_types`** — shared traits + types. Keystone for all extractions +2. **Tier 1** (estimation, observability, tunnel) — immediate wins, zero risk +3. **`ironclaw_mcp`** — generic MCP client, independently useful +4. **`ironclaw_llm`** (with transcription) — large module, clean trait boundary +5. **`ironclaw_db`** + **`ironclaw_workspace`** — persistence layer +6. **`ironclaw_gateway`** — extract 160K-line web gateway (biggest compile time win) + +--- + +## Completed + +- [x] `ironclaw_safety` — extracted to `crates/ironclaw_safety/` (already existed) +- [x] `ironclaw_engine` — new crate at `crates/ironclaw_engine/` (engine v2) +- [x] Transcription moved to `src/llm/transcription/` (PR #1559 on staging)