From 97a7637f301b2b62f60c531fdc9c2cf054216390 Mon Sep 17 00:00:00 2001 From: Illia Polosukhin Date: Thu, 19 Feb 2026 17:17:44 -0800 Subject: [PATCH] feat: extension registry with metadata catalog and onboarding integration (#238) * feat: add extension registry with metadata catalog, CLI, and onboarding integration Adds a central registry that catalogs all 14 available extensions (10 tools, 4 channels) with their capabilities, auth requirements, and artifact references. The onboarding wizard now shows installable channels from the registry and offers tool installation as a new Step 7. - registry/ folder with per-extension JSON manifests and bundle definitions - src/registry/ module: manifest structs, catalog loader, installer - `ironclaw registry list|info|install|install-defaults` CLI commands - Setup wizard enhanced: channels from registry, new extensions step (8 steps) Co-Authored-By: Claude Opus 4.6 * fix(setup): resolve workspace errors for tool crates and channels-only onboarding Tool crates in tools-src/ and channels-src/ failed `cargo metadata` during onboard install because Cargo resolved them as part of the root workspace. Add `[workspace]` table to each standalone crate and extend the root `workspace.exclude` list so they build independently. Channels-only mode (`onboard --channels-only`) failed with "Secrets not configured" and "No database connection" because it skipped database and security setup. Add `reconnect_existing_db()` to establish the DB connection and load saved settings before running channel configuration. Also improve the tunnel "already configured" display to show full provider details (domain, mode, command) instead of just the provider name. Co-Authored-By: Claude Opus 4.6 * fix(registry): address PR review feedback on installer and catalog - Use manifest.name (not crate_name) for installed filenames so discovery, auth, and CLI commands all agree on the stem (#1) - Add AlreadyInstalled error variant instead of misleading ExtensionNotFound (#2) - Add DownloadFailed error variant with URL context instead of stuffing URLs into PathBuf (#3) - Validate HTTP status with error_for_status() before reading response bytes in artifact downloads (#4) - Switch build_wasm_component to tokio::process::Command with status() so build output streams to the terminal (#6) - Find WASM artifact by crate_name specifically instead of picking the first .wasm file in the release directory (#7) - Add is_file() guard in catalog loader to skip directories (#8) - Detect ambiguous bare-name lookups when both tools/ and channels/ exist, with get_strict() returning an error (#9) - Fix wizard step_extensions to check tool.name for installed detection, consistent with the new naming (#11, #12) - Fix redundant closures and map_or clippy warnings in changed files Co-Authored-By: Claude Opus 4.6 * fix(setup): restore DB connection fields after settings reload reconnect_postgres() and reconnect_libsql() called Settings::from_db_map() which overwrote database_url / libsql_path / libsql_url set from env vars. Also use get_strict() in cmd_info to surface ambiguous bare-name errors. Co-Authored-By: Claude Opus 4.6 * style: fix clippy collapsible_if and print_literal warnings Collapse nested if-let chains and inline string literals in format macros to satisfy CI clippy lint checks (deny warnings). Co-Authored-By: Claude Opus 4.6 * fix(registry): prefer artifacts for install-defaults and improve dir lookup - InstallDefaults now defaults to downloading pre-built artifacts (matching `registry install` behavior), with --build flag for source builds. - find_registry_dir() walks up 3 ancestor levels from the exe and adds a CARGO_MANIFEST_DIR fallback, matching load_registry_catalog() logic. Co-Authored-By: Claude Opus 4.6 --------- Co-authored-by: Claude Opus 4.6 --- Cargo.toml | 10 + channels-src/discord/Cargo.toml | 2 + channels-src/slack/Cargo.toml | 2 + channels-src/telegram/Cargo.toml | 2 + channels-src/whatsapp/Cargo.toml | 2 + registry/_bundles.json | 42 ++ registry/channels/discord.json | 31 ++ registry/channels/slack.json | 31 ++ registry/channels/telegram.json | 31 ++ registry/channels/whatsapp.json | 31 ++ registry/tools/github.json | 31 ++ registry/tools/gmail.json | 31 ++ registry/tools/google-calendar.json | 31 ++ registry/tools/google-docs.json | 31 ++ registry/tools/google-drive.json | 31 ++ registry/tools/google-sheets.json | 31 ++ registry/tools/google-slides.json | 31 ++ registry/tools/okta.json | 31 ++ registry/tools/slack.json | 31 ++ registry/tools/telegram.json | 31 ++ src/cli/mod.rs | 6 + src/cli/registry.rs | 339 ++++++++++++++++ src/lib.rs | 1 + src/main.rs | 9 + src/registry/catalog.rs | 580 +++++++++++++++++++++++++++ src/registry/installer.rs | 415 +++++++++++++++++++ src/registry/manifest.rs | 271 +++++++++++++ src/registry/mod.rs | 23 ++ src/setup/README.md | 50 ++- src/setup/channels.rs | 50 ++- src/setup/mod.rs | 3 +- src/setup/wizard.rs | 439 +++++++++++++++++++- tools-src/github/Cargo.toml | 2 + tools-src/gmail/Cargo.toml | 2 + tools-src/google-calendar/Cargo.toml | 2 + tools-src/google-docs/Cargo.toml | 2 + tools-src/google-drive/Cargo.toml | 2 + tools-src/google-sheets/Cargo.toml | 2 + tools-src/google-slides/Cargo.toml | 2 + tools-src/okta/Cargo.toml | 2 + tools-src/slack/Cargo.toml | 2 + tools-src/telegram/Cargo.toml | 2 + 42 files changed, 2671 insertions(+), 29 deletions(-) create mode 100644 registry/_bundles.json create mode 100644 registry/channels/discord.json create mode 100644 registry/channels/slack.json create mode 100644 registry/channels/telegram.json create mode 100644 registry/channels/whatsapp.json create mode 100644 registry/tools/github.json create mode 100644 registry/tools/gmail.json create mode 100644 registry/tools/google-calendar.json create mode 100644 registry/tools/google-docs.json create mode 100644 registry/tools/google-drive.json create mode 100644 registry/tools/google-sheets.json create mode 100644 registry/tools/google-slides.json create mode 100644 registry/tools/okta.json create mode 100644 registry/tools/slack.json create mode 100644 registry/tools/telegram.json create mode 100644 src/cli/registry.rs create mode 100644 src/registry/catalog.rs create mode 100644 src/registry/installer.rs create mode 100644 src/registry/manifest.rs create mode 100644 src/registry/mod.rs diff --git a/Cargo.toml b/Cargo.toml index d6301b56..d37fa792 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -1,10 +1,20 @@ [workspace] members = [".", "benchmarks"] exclude = [ + "channels-src/discord", "channels-src/telegram", "channels-src/slack", "channels-src/whatsapp", + "tools-src/github", "tools-src/gmail", + "tools-src/google-calendar", + "tools-src/google-docs", + "tools-src/google-drive", + "tools-src/google-sheets", + "tools-src/google-slides", + "tools-src/okta", + "tools-src/slack", + "tools-src/telegram", ] [package] diff --git a/channels-src/discord/Cargo.toml b/channels-src/discord/Cargo.toml index 6edd6e64..b8a9f196 100644 --- a/channels-src/discord/Cargo.toml +++ b/channels-src/discord/Cargo.toml @@ -21,3 +21,5 @@ lto = true codegen-units = 1 + +[workspace] diff --git a/channels-src/slack/Cargo.toml b/channels-src/slack/Cargo.toml index 18d2fd39..7d77c021 100644 --- a/channels-src/slack/Cargo.toml +++ b/channels-src/slack/Cargo.toml @@ -27,3 +27,5 @@ opt-level = "s" lto = true strip = true codegen-units = 1 + +[workspace] diff --git a/channels-src/telegram/Cargo.toml b/channels-src/telegram/Cargo.toml index 1964e327..06cd9de5 100644 --- a/channels-src/telegram/Cargo.toml +++ b/channels-src/telegram/Cargo.toml @@ -25,3 +25,5 @@ opt-level = "s" lto = true strip = true codegen-units = 1 + +[workspace] diff --git a/channels-src/whatsapp/Cargo.toml b/channels-src/whatsapp/Cargo.toml index 8dd03499..4e334bee 100644 --- a/channels-src/whatsapp/Cargo.toml +++ b/channels-src/whatsapp/Cargo.toml @@ -16,3 +16,5 @@ serde_json = "1" opt-level = "s" lto = true strip = true + +[workspace] diff --git a/registry/_bundles.json b/registry/_bundles.json new file mode 100644 index 00000000..bf332a58 --- /dev/null +++ b/registry/_bundles.json @@ -0,0 +1,42 @@ +{ + "bundles": { + "google": { + "display_name": "Google Suite", + "description": "Gmail, Calendar, Drive, Docs, Sheets, Slides", + "extensions": [ + "tools/gmail", + "tools/google-calendar", + "tools/google-docs", + "tools/google-drive", + "tools/google-sheets", + "tools/google-slides" + ], + "shared_auth": "google_oauth_token" + }, + "messaging": { + "display_name": "Messaging Channels", + "description": "Discord, Telegram, Slack, and WhatsApp channels", + "extensions": [ + "channels/discord", + "channels/telegram", + "channels/slack", + "channels/whatsapp" + ], + "shared_auth": null + }, + "default": { + "display_name": "Recommended Set", + "description": "Core tools and channels for a productive setup", + "extensions": [ + "tools/github", + "tools/gmail", + "tools/google-calendar", + "tools/google-drive", + "tools/slack", + "channels/telegram", + "channels/slack" + ], + "shared_auth": null + } + } +} diff --git a/registry/channels/discord.json b/registry/channels/discord.json new file mode 100644 index 00000000..77aba7dc --- /dev/null +++ b/registry/channels/discord.json @@ -0,0 +1,31 @@ +{ + "name": "discord", + "display_name": "Discord", + "kind": "channel", + "version": "0.1.0", + "description": "Discord Gateway/Webhook channel for slash commands, buttons, and messages", + "keywords": ["messaging", "chat", "discord", "bot"], + + "source": { + "dir": "channels-src/discord", + "capabilities": "discord.capabilities.json", + "crate_name": "discord-channel" + }, + + "artifacts": { + "wasm32-wasip2": { + "url": null, + "sha256": null + } + }, + + "auth_summary": { + "method": "manual", + "provider": "Discord", + "secrets": ["discord_bot_token"], + "shared_auth": null, + "setup_url": "https://discord.com/developers/applications" + }, + + "tags": ["messaging"] +} diff --git a/registry/channels/slack.json b/registry/channels/slack.json new file mode 100644 index 00000000..bd4c85bf --- /dev/null +++ b/registry/channels/slack.json @@ -0,0 +1,31 @@ +{ + "name": "slack", + "display_name": "Slack", + "kind": "channel", + "version": "0.1.0", + "description": "Slack Events API channel for receiving and responding to Slack messages", + "keywords": ["messaging", "chat", "workspace", "slack"], + + "source": { + "dir": "channels-src/slack", + "capabilities": "slack.capabilities.json", + "crate_name": "slack-channel" + }, + + "artifacts": { + "wasm32-wasip2": { + "url": null, + "sha256": null + } + }, + + "auth_summary": { + "method": "manual", + "provider": "Slack", + "secrets": ["slack_bot_token", "slack_signing_secret"], + "shared_auth": null, + "setup_url": "https://api.slack.com/apps" + }, + + "tags": ["default", "messaging"] +} diff --git a/registry/channels/telegram.json b/registry/channels/telegram.json new file mode 100644 index 00000000..65a199d9 --- /dev/null +++ b/registry/channels/telegram.json @@ -0,0 +1,31 @@ +{ + "name": "telegram", + "display_name": "Telegram", + "kind": "channel", + "version": "0.1.0", + "description": "Telegram Bot API channel for receiving and responding to messages", + "keywords": ["messaging", "bot", "chat", "telegram"], + + "source": { + "dir": "channels-src/telegram", + "capabilities": "telegram.capabilities.json", + "crate_name": "telegram-channel" + }, + + "artifacts": { + "wasm32-wasip2": { + "url": null, + "sha256": null + } + }, + + "auth_summary": { + "method": "manual", + "provider": "Telegram", + "secrets": ["telegram_bot_token"], + "shared_auth": null, + "setup_url": "https://t.me/BotFather" + }, + + "tags": ["default", "messaging"] +} diff --git a/registry/channels/whatsapp.json b/registry/channels/whatsapp.json new file mode 100644 index 00000000..a36d1e69 --- /dev/null +++ b/registry/channels/whatsapp.json @@ -0,0 +1,31 @@ +{ + "name": "whatsapp", + "display_name": "WhatsApp", + "kind": "channel", + "version": "0.1.0", + "description": "WhatsApp Cloud API channel for receiving and responding to messages", + "keywords": ["messaging", "chat", "whatsapp", "meta"], + + "source": { + "dir": "channels-src/whatsapp", + "capabilities": "whatsapp.capabilities.json", + "crate_name": "whatsapp-channel" + }, + + "artifacts": { + "wasm32-wasip2": { + "url": null, + "sha256": null + } + }, + + "auth_summary": { + "method": "manual", + "provider": "Meta", + "secrets": ["whatsapp_access_token", "whatsapp_verify_token"], + "shared_auth": null, + "setup_url": "https://developers.facebook.com/apps/" + }, + + "tags": ["messaging"] +} diff --git a/registry/tools/github.json b/registry/tools/github.json new file mode 100644 index 00000000..85ee06f7 --- /dev/null +++ b/registry/tools/github.json @@ -0,0 +1,31 @@ +{ + "name": "github", + "display_name": "GitHub", + "kind": "tool", + "version": "0.1.0", + "description": "GitHub integration for issues, PRs, repos, and code search", + "keywords": ["git", "code", "issues", "pull-requests", "repositories"], + + "source": { + "dir": "tools-src/github", + "capabilities": "github-tool.capabilities.json", + "crate_name": "github-tool" + }, + + "artifacts": { + "wasm32-wasip2": { + "url": null, + "sha256": null + } + }, + + "auth_summary": { + "method": "manual", + "provider": "GitHub", + "secrets": ["github_token"], + "shared_auth": null, + "setup_url": "https://github.com/settings/tokens" + }, + + "tags": ["default", "development"] +} diff --git a/registry/tools/gmail.json b/registry/tools/gmail.json new file mode 100644 index 00000000..04c6fd9e --- /dev/null +++ b/registry/tools/gmail.json @@ -0,0 +1,31 @@ +{ + "name": "gmail", + "display_name": "Gmail", + "kind": "tool", + "version": "0.1.0", + "description": "Read, send, and manage Gmail messages and threads", + "keywords": ["email", "google", "mail", "messaging"], + + "source": { + "dir": "tools-src/gmail", + "capabilities": "gmail-tool.capabilities.json", + "crate_name": "gmail-tool" + }, + + "artifacts": { + "wasm32-wasip2": { + "url": null, + "sha256": null + } + }, + + "auth_summary": { + "method": "oauth", + "provider": "Google", + "secrets": ["google_oauth_token"], + "shared_auth": "google_oauth_token", + "setup_url": "https://console.cloud.google.com/apis/credentials" + }, + + "tags": ["default", "google", "messaging"] +} diff --git a/registry/tools/google-calendar.json b/registry/tools/google-calendar.json new file mode 100644 index 00000000..16d8e89d --- /dev/null +++ b/registry/tools/google-calendar.json @@ -0,0 +1,31 @@ +{ + "name": "google-calendar", + "display_name": "Google Calendar", + "kind": "tool", + "version": "0.1.0", + "description": "Create, read, update, and delete Google Calendar events", + "keywords": ["calendar", "google", "scheduling", "events"], + + "source": { + "dir": "tools-src/google-calendar", + "capabilities": "google-calendar-tool.capabilities.json", + "crate_name": "google-calendar-tool" + }, + + "artifacts": { + "wasm32-wasip2": { + "url": null, + "sha256": null + } + }, + + "auth_summary": { + "method": "oauth", + "provider": "Google", + "secrets": ["google_oauth_token"], + "shared_auth": "google_oauth_token", + "setup_url": "https://console.cloud.google.com/apis/credentials" + }, + + "tags": ["default", "google", "productivity"] +} diff --git a/registry/tools/google-docs.json b/registry/tools/google-docs.json new file mode 100644 index 00000000..90e6859a --- /dev/null +++ b/registry/tools/google-docs.json @@ -0,0 +1,31 @@ +{ + "name": "google-docs", + "display_name": "Google Docs", + "kind": "tool", + "version": "0.1.0", + "description": "Create and edit Google Docs documents", + "keywords": ["documents", "google", "writing", "docs"], + + "source": { + "dir": "tools-src/google-docs", + "capabilities": "google-docs-tool.capabilities.json", + "crate_name": "google-docs-tool" + }, + + "artifacts": { + "wasm32-wasip2": { + "url": null, + "sha256": null + } + }, + + "auth_summary": { + "method": "oauth", + "provider": "Google", + "secrets": ["google_oauth_token"], + "shared_auth": "google_oauth_token", + "setup_url": "https://console.cloud.google.com/apis/credentials" + }, + + "tags": ["google", "productivity"] +} diff --git a/registry/tools/google-drive.json b/registry/tools/google-drive.json new file mode 100644 index 00000000..586c6afd --- /dev/null +++ b/registry/tools/google-drive.json @@ -0,0 +1,31 @@ +{ + "name": "google-drive", + "display_name": "Google Drive", + "kind": "tool", + "version": "0.1.0", + "description": "Upload, download, search, and manage Google Drive files and folders", + "keywords": ["storage", "google", "files", "drive"], + + "source": { + "dir": "tools-src/google-drive", + "capabilities": "google-drive-tool.capabilities.json", + "crate_name": "google-drive-tool" + }, + + "artifacts": { + "wasm32-wasip2": { + "url": null, + "sha256": null + } + }, + + "auth_summary": { + "method": "oauth", + "provider": "Google", + "secrets": ["google_oauth_token"], + "shared_auth": "google_oauth_token", + "setup_url": "https://console.cloud.google.com/apis/credentials" + }, + + "tags": ["default", "google", "storage"] +} diff --git a/registry/tools/google-sheets.json b/registry/tools/google-sheets.json new file mode 100644 index 00000000..f840b6a6 --- /dev/null +++ b/registry/tools/google-sheets.json @@ -0,0 +1,31 @@ +{ + "name": "google-sheets", + "display_name": "Google Sheets", + "kind": "tool", + "version": "0.1.0", + "description": "Read and write Google Sheets spreadsheet data", + "keywords": ["spreadsheets", "google", "data", "sheets"], + + "source": { + "dir": "tools-src/google-sheets", + "capabilities": "google-sheets-tool.capabilities.json", + "crate_name": "google-sheets-tool" + }, + + "artifacts": { + "wasm32-wasip2": { + "url": null, + "sha256": null + } + }, + + "auth_summary": { + "method": "oauth", + "provider": "Google", + "secrets": ["google_oauth_token"], + "shared_auth": "google_oauth_token", + "setup_url": "https://console.cloud.google.com/apis/credentials" + }, + + "tags": ["google", "productivity"] +} diff --git a/registry/tools/google-slides.json b/registry/tools/google-slides.json new file mode 100644 index 00000000..94ed4a4a --- /dev/null +++ b/registry/tools/google-slides.json @@ -0,0 +1,31 @@ +{ + "name": "google-slides", + "display_name": "Google Slides", + "kind": "tool", + "version": "0.1.0", + "description": "Create and edit Google Slides presentations", + "keywords": ["presentations", "google", "slides"], + + "source": { + "dir": "tools-src/google-slides", + "capabilities": "google-slides-tool.capabilities.json", + "crate_name": "google-slides-tool" + }, + + "artifacts": { + "wasm32-wasip2": { + "url": null, + "sha256": null + } + }, + + "auth_summary": { + "method": "oauth", + "provider": "Google", + "secrets": ["google_oauth_token"], + "shared_auth": "google_oauth_token", + "setup_url": "https://console.cloud.google.com/apis/credentials" + }, + + "tags": ["google", "productivity"] +} diff --git a/registry/tools/okta.json b/registry/tools/okta.json new file mode 100644 index 00000000..2b55571a --- /dev/null +++ b/registry/tools/okta.json @@ -0,0 +1,31 @@ +{ + "name": "okta", + "display_name": "Okta", + "kind": "tool", + "version": "0.1.0", + "description": "Okta SSO for user profile, app catalog, and SSO launch links", + "keywords": ["sso", "identity", "authentication", "okta"], + + "source": { + "dir": "tools-src/okta", + "capabilities": "okta-tool.capabilities.json", + "crate_name": "okta-tool" + }, + + "artifacts": { + "wasm32-wasip2": { + "url": null, + "sha256": null + } + }, + + "auth_summary": { + "method": "oauth", + "provider": "Okta", + "secrets": ["okta_oauth_token"], + "shared_auth": null, + "setup_url": "https://developer.okta.com/docs/guides/implement-oauth-for-okta/main/" + }, + + "tags": ["identity"] +} diff --git a/registry/tools/slack.json b/registry/tools/slack.json new file mode 100644 index 00000000..0f876cf3 --- /dev/null +++ b/registry/tools/slack.json @@ -0,0 +1,31 @@ +{ + "name": "slack", + "display_name": "Slack", + "kind": "tool", + "version": "0.1.0", + "description": "Post messages, read channels, and manage conversations via Slack API", + "keywords": ["messaging", "chat", "workspace"], + + "source": { + "dir": "tools-src/slack", + "capabilities": "slack-tool.capabilities.json", + "crate_name": "slack-tool" + }, + + "artifacts": { + "wasm32-wasip2": { + "url": null, + "sha256": null + } + }, + + "auth_summary": { + "method": "oauth", + "provider": "Slack", + "secrets": ["slack_bot_token"], + "shared_auth": null, + "setup_url": "https://api.slack.com/apps" + }, + + "tags": ["default", "messaging"] +} diff --git a/registry/tools/telegram.json b/registry/tools/telegram.json new file mode 100644 index 00000000..cd4835c2 --- /dev/null +++ b/registry/tools/telegram.json @@ -0,0 +1,31 @@ +{ + "name": "telegram", + "display_name": "Telegram", + "kind": "tool", + "version": "0.1.0", + "description": "Telegram user-mode integration via MTProto for messages and contacts", + "keywords": ["messaging", "chat", "telegram", "mtproto"], + + "source": { + "dir": "tools-src/telegram", + "capabilities": "telegram-tool.capabilities.json", + "crate_name": "telegram-tool" + }, + + "artifacts": { + "wasm32-wasip2": { + "url": null, + "sha256": null + } + }, + + "auth_summary": { + "method": "manual", + "provider": "Telegram", + "secrets": ["telegram_api_id", "telegram_api_hash"], + "shared_auth": null, + "setup_url": "https://my.telegram.org/apps" + }, + + "tags": ["messaging"] +} diff --git a/src/cli/mod.rs b/src/cli/mod.rs index ce193013..1ff4d391 100644 --- a/src/cli/mod.rs +++ b/src/cli/mod.rs @@ -17,6 +17,7 @@ mod mcp; pub mod memory; pub mod oauth_defaults; mod pairing; +mod registry; mod service; pub mod status; mod tool; @@ -29,6 +30,7 @@ pub use memory::MemoryCommand; pub use memory::run_memory_command; pub use memory::run_memory_command_with_db; pub use pairing::{PairingCommand, run_pairing_command, run_pairing_command_with_store}; +pub use registry::{RegistryCommand, run_registry_command}; pub use service::{ServiceCommand, run_service_command}; pub use status::run_status_command; pub use tool::{ToolCommand, run_tool_command}; @@ -90,6 +92,10 @@ pub enum Command { #[command(subcommand)] Tool(ToolCommand), + /// Browse and install extensions from the registry + #[command(subcommand)] + Registry(RegistryCommand), + /// Manage MCP servers (hosted tool providers) #[command(subcommand)] Mcp(McpCommand), diff --git a/src/cli/registry.rs b/src/cli/registry.rs new file mode 100644 index 00000000..76dc77a8 --- /dev/null +++ b/src/cli/registry.rs @@ -0,0 +1,339 @@ +//! Registry CLI commands for discovering and installing extensions. + +use std::path::PathBuf; + +use clap::Subcommand; + +use crate::registry::catalog::RegistryCatalog; +use crate::registry::installer::RegistryInstaller; +use crate::registry::manifest::ManifestKind; + +#[derive(Subcommand, Debug, Clone)] +pub enum RegistryCommand { + /// List available extensions in the registry + List { + /// Filter by kind: "tool" or "channel" + #[arg(short, long)] + kind: Option, + + /// Filter by tag (e.g. "default", "google", "messaging") + #[arg(short, long)] + tag: Option, + + /// Show detailed information + #[arg(short, long)] + verbose: bool, + }, + + /// Show detailed information about an extension or bundle + Info { + /// Extension or bundle name (e.g. "slack", "google", "tools/gmail") + name: String, + }, + + /// Install an extension or bundle from the registry + Install { + /// Extension or bundle name (e.g. "slack", "google", "default") + name: String, + + /// Force overwrite if already installed + #[arg(short, long)] + force: bool, + + /// Build from source instead of downloading pre-built artifact + #[arg(long)] + build: bool, + }, + + /// Install the default bundle of recommended extensions + InstallDefaults { + /// Force overwrite if already installed + #[arg(short, long)] + force: bool, + + /// Build from source instead of downloading pre-built artifact + #[arg(long)] + build: bool, + }, +} + +/// Run a registry command. +pub async fn run_registry_command(cmd: RegistryCommand) -> anyhow::Result<()> { + let registry_dir = find_registry_dir()?; + let catalog = RegistryCatalog::load(®istry_dir)?; + + match cmd { + RegistryCommand::List { kind, tag, verbose } => { + cmd_list(&catalog, kind.as_deref(), tag.as_deref(), verbose) + } + RegistryCommand::Info { name } => cmd_info(&catalog, &name), + RegistryCommand::Install { name, force, build } => { + cmd_install(&catalog, ®istry_dir, &name, force, build).await + } + RegistryCommand::InstallDefaults { force, build } => { + cmd_install(&catalog, ®istry_dir, "default", force, build).await + } + } +} + +/// Find the registry directory by looking relative to the current executable or cwd. +fn find_registry_dir() -> anyhow::Result { + // Try relative to current directory (for dev usage) + let cwd = std::env::current_dir()?; + let candidate = cwd.join("registry"); + if candidate.is_dir() { + return Ok(candidate); + } + + // Try relative to executable (covers installed binary, target/debug/, target/release/) + if let Ok(exe) = std::env::current_exe() + && let Some(parent) = exe.parent() + { + // Walk up to 3 levels: exe dir, parent (target/release → target), grandparent (→ repo root) + let mut dir = Some(parent); + for _ in 0..3 { + if let Some(d) = dir { + let candidate = d.join("registry"); + if candidate.is_dir() { + return Ok(candidate); + } + dir = d.parent(); + } + } + } + + // Try CARGO_MANIFEST_DIR (compile-time, works in dev builds) + let manifest_dir = std::path::Path::new(env!("CARGO_MANIFEST_DIR")); + let candidate = manifest_dir.join("registry"); + if candidate.is_dir() { + return Ok(candidate); + } + + anyhow::bail!( + "Could not find registry/ directory. Run from the ironclaw repo root, \ + or ensure registry/ is next to the ironclaw binary." + ) +} + +fn cmd_list( + catalog: &RegistryCatalog, + kind: Option<&str>, + tag: Option<&str>, + verbose: bool, +) -> anyhow::Result<()> { + let kind_filter = match kind { + Some("tool" | "tools") => Some(ManifestKind::Tool), + Some("channel" | "channels") => Some(ManifestKind::Channel), + Some(other) => anyhow::bail!("Unknown kind '{}'. Use 'tool' or 'channel'.", other), + None => None, + }; + + let manifests = catalog.list(kind_filter, tag); + + if manifests.is_empty() { + println!("No extensions found matching the criteria."); + return Ok(()); + } + + // Print header + if verbose { + println!( + "{:<20} {:<8} {:<8} {:<10} DESCRIPTION", + "NAME", "KIND", "VERSION", "AUTH" + ); + println!("{}", "-".repeat(80)); + } else { + println!("{:<20} {:<8} DESCRIPTION", "NAME", "KIND"); + println!("{}", "-".repeat(60)); + } + + for m in &manifests { + if verbose { + let auth = m + .auth_summary + .as_ref() + .and_then(|a| a.method.as_deref()) + .unwrap_or("none"); + println!( + "{:<20} {:<8} {:<8} {:<10} {}", + m.name, m.kind, m.version, auth, m.description + ); + } else { + println!("{:<20} {:<8} {}", m.name, m.kind, m.description); + } + } + + println!("\n{} extension(s) found.", manifests.len()); + + // Show bundles hint + let bundle_names = catalog.bundle_names(); + if !bundle_names.is_empty() { + println!("\nBundles available: {}", bundle_names.join(", ")); + println!("Use `ironclaw registry info ` for details."); + } + + Ok(()) +} + +fn cmd_info(catalog: &RegistryCatalog, name: &str) -> anyhow::Result<()> { + // Check if it's a bundle + if let Some(bundle) = catalog.get_bundle(name) { + println!("Bundle: {}", bundle.display_name); + if let Some(desc) = &bundle.description { + println!(" {}", desc); + } + println!("\nExtensions:"); + for ext_key in &bundle.extensions { + if let Some(m) = catalog.get(ext_key) { + println!(" {} - {} ({})", ext_key, m.description, m.kind); + } else { + println!(" {} (not found in registry)", ext_key); + } + } + if let Some(shared) = &bundle.shared_auth { + println!("\nShared auth: {}", shared); + } + return Ok(()); + } + + // Single extension (use get_strict to surface ambiguous bare names) + let manifest = catalog + .get_strict(name) + .map_err(|e| anyhow::anyhow!("{}", e))?; + + println!("{} ({})", manifest.display_name, manifest.kind); + println!(" Version: {}", manifest.version); + println!(" {}", manifest.description); + + if !manifest.keywords.is_empty() { + println!(" Keywords: {}", manifest.keywords.join(", ")); + } + + println!("\nSource:"); + println!(" Directory: {}", manifest.source.dir); + println!(" Crate: {}", manifest.source.crate_name); + println!(" Capabilities: {}", manifest.source.capabilities); + + if let Some(artifact) = manifest.artifacts.get("wasm32-wasip2") { + println!("\nArtifact (wasm32-wasip2):"); + match &artifact.url { + Some(url) => println!(" URL: {}", url), + None => println!(" URL: (not yet published)"), + } + match &artifact.sha256 { + Some(sha) => println!(" SHA256: {}", sha), + None => println!(" SHA256: (not yet computed)"), + } + } + + if let Some(auth) = &manifest.auth_summary { + println!("\nAuthentication:"); + if let Some(method) = &auth.method { + println!(" Method: {}", method); + } + if let Some(provider) = &auth.provider { + println!(" Provider: {}", provider); + } + if !auth.secrets.is_empty() { + println!(" Secrets: {}", auth.secrets.join(", ")); + } + if let Some(shared) = &auth.shared_auth { + println!(" Shared with: {}", shared); + } + if let Some(url) = &auth.setup_url { + println!(" Setup: {}", url); + } + } + + if !manifest.tags.is_empty() { + println!("\nTags: {}", manifest.tags.join(", ")); + } + + Ok(()) +} + +async fn cmd_install( + catalog: &RegistryCatalog, + registry_dir: &std::path::Path, + name: &str, + force: bool, + prefer_build: bool, +) -> anyhow::Result<()> { + // Registry dir parent is the repo root + let repo_root = registry_dir + .parent() + .ok_or_else(|| anyhow::anyhow!("Cannot determine repo root from registry dir"))?; + + let installer = RegistryInstaller::with_defaults(repo_root.to_path_buf()); + + let (manifests, bundle) = catalog.resolve(name)?; + + if manifests.is_empty() { + anyhow::bail!("No extensions found for '{}'.", name); + } + + if let Some(bundle_def) = bundle { + // Bundle install + println!( + "Installing bundle '{}' ({} extensions)...\n", + bundle_def.display_name, + manifests.len() + ); + + let (outcomes, hints) = installer + .install_bundle(&manifests, bundle_def, force, prefer_build) + .await; + + println!("\n--- Results ---"); + for outcome in &outcomes { + let caps_status = if outcome.has_capabilities { "+" } else { "-" }; + println!( + " [{}] {} ({}) -> {}", + caps_status, + outcome.name, + outcome.kind, + outcome.wasm_path.display() + ); + for w in &outcome.warnings { + println!(" Warning: {}", w); + } + } + + if !hints.is_empty() { + println!("\nAuth setup:"); + for hint in &hints { + println!("{}", hint); + } + } + + println!( + "\nInstalled {}/{} extensions.", + outcomes.len(), + manifests.len() + ); + } else { + // Single extension + let manifest = manifests[0]; + let outcome = installer.install(manifest, force, prefer_build).await?; + + println!("\nInstalled successfully:"); + println!(" Name: {}", outcome.name); + println!(" Kind: {}", outcome.kind); + println!(" WASM: {}", outcome.wasm_path.display()); + println!(" Capabilities: {}", outcome.has_capabilities); + + if let Some(auth) = &manifest.auth_summary + && auth.method.as_deref() != Some("none") + { + println!( + "\nNext step: authenticate with `ironclaw tool auth {}`", + manifest.name + ); + if let Some(url) = &auth.setup_url { + println!(" Setup credentials at: {}", url); + } + } + } + + Ok(()) +} diff --git a/src/lib.rs b/src/lib.rs index 202fcbb0..d14d14d2 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -57,6 +57,7 @@ pub mod llm; pub mod observability; pub mod orchestrator; pub mod pairing; +pub mod registry; pub mod safety; pub mod sandbox; pub mod secrets; diff --git a/src/main.rs b/src/main.rs index e8a3e50c..0189cca1 100644 --- a/src/main.rs +++ b/src/main.rs @@ -80,6 +80,15 @@ async fn main() -> anyhow::Result<()> { return ironclaw::cli::run_config_command(config_cmd.clone()).await; } + Some(Command::Registry(registry_cmd)) => { + tracing_subscriber::fmt() + .with_env_filter( + EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new("warn")), + ) + .init(); + + return ironclaw::cli::run_registry_command(registry_cmd.clone()).await; + } Some(Command::Mcp(mcp_cmd)) => { // Simple logging for MCP commands tracing_subscriber::fmt() diff --git a/src/registry/catalog.rs b/src/registry/catalog.rs new file mode 100644 index 00000000..64e8d5a8 --- /dev/null +++ b/src/registry/catalog.rs @@ -0,0 +1,580 @@ +//! Registry catalog: loads manifests from disk, provides list/search/resolve operations. + +use std::collections::HashMap; +use std::path::{Path, PathBuf}; + +use crate::registry::manifest::{BundleDefinition, BundlesFile, ExtensionManifest, ManifestKind}; + +/// Error type for registry operations. +#[derive(Debug, thiserror::Error)] +pub enum RegistryError { + #[error("Registry directory not found: {0}")] + DirectoryNotFound(PathBuf), + + #[error("Failed to read manifest {path}: {reason}")] + ManifestRead { path: PathBuf, reason: String }, + + #[error("Failed to parse manifest {path}: {reason}")] + ManifestParse { path: PathBuf, reason: String }, + + #[error("Extension not found: {0}")] + ExtensionNotFound(String), + + #[error("'{name}' already installed at {path}. Use --force to overwrite.")] + AlreadyInstalled { + name: String, + path: std::path::PathBuf, + }, + + #[error("Download failed for {url}: {reason}")] + DownloadFailed { url: String, reason: String }, + + #[error( + "Ambiguous name '{name}': exists as both {kind_a} and {kind_b}. Use '{prefix_a}/{name}' or '{prefix_b}/{name}'." + )] + AmbiguousName { + name: String, + kind_a: &'static str, + prefix_a: &'static str, + kind_b: &'static str, + prefix_b: &'static str, + }, + + #[error("Bundle not found: {0}")] + BundleNotFound(String), + + #[error("Failed to read bundles file: {0}")] + BundlesRead(String), + + #[error("IO error: {0}")] + Io(#[from] std::io::Error), +} + +/// Central catalog loaded from the `registry/` directory. +#[derive(Debug, Clone)] +pub struct RegistryCatalog { + /// All loaded manifests, keyed by "/" (e.g. "tools/slack"). + manifests: HashMap, + + /// Bundle definitions from `_bundles.json`. + bundles: HashMap, + + /// Root directory of the registry. + root: PathBuf, +} + +impl RegistryCatalog { + /// Load the catalog from a registry directory. + /// + /// Expects the structure: + /// ```text + /// registry/ + /// ├── tools/*.json + /// ├── channels/*.json + /// └── _bundles.json + /// ``` + pub fn load(registry_dir: &Path) -> Result { + if !registry_dir.exists() { + return Err(RegistryError::DirectoryNotFound(registry_dir.to_path_buf())); + } + + let mut manifests = HashMap::new(); + + // Load tools + let tools_dir = registry_dir.join("tools"); + if tools_dir.is_dir() { + Self::load_manifests_from_dir(&tools_dir, "tools", &mut manifests)?; + } + + // Load channels + let channels_dir = registry_dir.join("channels"); + if channels_dir.is_dir() { + Self::load_manifests_from_dir(&channels_dir, "channels", &mut manifests)?; + } + + // Load bundles + let bundles_path = registry_dir.join("_bundles.json"); + let bundles = if bundles_path.is_file() { + let content = std::fs::read_to_string(&bundles_path).map_err(|e| { + RegistryError::BundlesRead(format!("{}: {}", bundles_path.display(), e)) + })?; + let bundles_file: BundlesFile = serde_json::from_str(&content).map_err(|e| { + RegistryError::BundlesRead(format!("{}: {}", bundles_path.display(), e)) + })?; + bundles_file.bundles + } else { + HashMap::new() + }; + + Ok(Self { + manifests, + bundles, + root: registry_dir.to_path_buf(), + }) + } + + fn load_manifests_from_dir( + dir: &Path, + kind_prefix: &str, + manifests: &mut HashMap, + ) -> Result<(), RegistryError> { + let entries = std::fs::read_dir(dir).map_err(|e| RegistryError::ManifestRead { + path: dir.to_path_buf(), + reason: e.to_string(), + })?; + + for entry in entries { + let entry = entry.map_err(|e| RegistryError::ManifestRead { + path: dir.to_path_buf(), + reason: e.to_string(), + })?; + + let path = entry.path(); + if !path.is_file() || path.extension().and_then(|e| e.to_str()) != Some("json") { + continue; + } + + let content = + std::fs::read_to_string(&path).map_err(|e| RegistryError::ManifestRead { + path: path.clone(), + reason: e.to_string(), + })?; + + let manifest: ExtensionManifest = + serde_json::from_str(&content).map_err(|e| RegistryError::ManifestParse { + path: path.clone(), + reason: e.to_string(), + })?; + + let key = format!("{}/{}", kind_prefix, manifest.name); + manifests.insert(key, manifest); + } + + Ok(()) + } + + /// The root directory this catalog was loaded from. + pub fn root(&self) -> &Path { + &self.root + } + + /// Get all manifests. + pub fn all(&self) -> Vec<&ExtensionManifest> { + let mut items: Vec<_> = self.manifests.values().collect(); + items.sort_by(|a, b| a.name.cmp(&b.name)); + items + } + + /// List manifests, optionally filtered by kind and/or tag. + pub fn list(&self, kind: Option, tag: Option<&str>) -> Vec<&ExtensionManifest> { + let mut results: Vec<_> = self + .manifests + .values() + .filter(|m| kind.is_none_or(|k| m.kind == k)) + .filter(|m| tag.is_none_or(|t| m.tags.iter().any(|mt| mt == t))) + .collect(); + results.sort_by(|a, b| a.name.cmp(&b.name)); + results + } + + /// Get a manifest by name. Tries exact key match first ("tools/slack"), + /// then searches by bare name ("slack"). + /// + /// If a bare name matches both a tool and a channel, returns `None`. + /// Use a qualified key ("tools/slack" or "channels/slack") to disambiguate. + pub fn get(&self, name: &str) -> Option<&ExtensionManifest> { + // Try exact key first + if let Some(m) = self.manifests.get(name) { + return Some(m); + } + + // Try with kind prefix, detecting collisions + let tool = self.manifests.get(&format!("tools/{}", name)); + let channel = self.manifests.get(&format!("channels/{}", name)); + + match (tool, channel) { + (Some(_), Some(_)) => None, // ambiguous + (Some(m), None) => Some(m), + (None, Some(m)) => Some(m), + (None, None) => None, + } + } + + /// Get a manifest by name, returning a `Result` with an explicit error for + /// ambiguous bare names. + pub fn get_strict(&self, name: &str) -> Result<&ExtensionManifest, RegistryError> { + // Try exact key first + if let Some(m) = self.manifests.get(name) { + return Ok(m); + } + + let has_tool = self.manifests.contains_key(&format!("tools/{}", name)); + let has_channel = self.manifests.contains_key(&format!("channels/{}", name)); + + match (has_tool, has_channel) { + (true, true) => Err(RegistryError::AmbiguousName { + name: name.to_string(), + kind_a: "tool", + prefix_a: "tools", + kind_b: "channel", + prefix_b: "channels", + }), + (true, false) => Ok(self.manifests.get(&format!("tools/{}", name)).unwrap()), + (false, true) => Ok(self.manifests.get(&format!("channels/{}", name)).unwrap()), + (false, false) => Err(RegistryError::ExtensionNotFound(name.to_string())), + } + } + + /// Get the full key ("tools/slack" or "channels/telegram") for a manifest. + pub fn key_for(&self, name: &str) -> Option { + if self.manifests.contains_key(name) { + return Some(name.to_string()); + } + + let has_tool = self.manifests.contains_key(&format!("tools/{}", name)); + let has_channel = self.manifests.contains_key(&format!("channels/{}", name)); + + match (has_tool, has_channel) { + (true, true) => None, // ambiguous + (true, false) => Some(format!("tools/{}", name)), + (false, true) => Some(format!("channels/{}", name)), + (false, false) => None, + } + } + + /// Search manifests by query string (matches name, display_name, description, keywords). + pub fn search(&self, query: &str) -> Vec<&ExtensionManifest> { + let query_lower = query.to_lowercase(); + let tokens: Vec<&str> = query_lower.split_whitespace().collect(); + + let mut scored: Vec<(&ExtensionManifest, usize)> = self + .manifests + .values() + .filter_map(|m| { + let score = Self::score_manifest(m, &tokens); + if score > 0 { Some((m, score)) } else { None } + }) + .collect(); + + scored.sort_by(|a, b| b.1.cmp(&a.1).then(a.0.name.cmp(&b.0.name))); + scored.into_iter().map(|(m, _)| m).collect() + } + + fn score_manifest(manifest: &ExtensionManifest, tokens: &[&str]) -> usize { + let mut score = 0; + let name_lower = manifest.name.to_lowercase(); + let display_lower = manifest.display_name.to_lowercase(); + let desc_lower = manifest.description.to_lowercase(); + + for token in tokens { + if name_lower == *token { + score += 10; + } else if name_lower.contains(token) { + score += 5; + } + + if display_lower == *token { + score += 8; + } else if display_lower.contains(token) { + score += 4; + } + + if desc_lower.contains(token) { + score += 2; + } + + for kw in &manifest.keywords { + if kw.to_lowercase() == *token { + score += 6; + } else if kw.to_lowercase().contains(token) { + score += 3; + } + } + + for tag in &manifest.tags { + if tag.to_lowercase() == *token { + score += 4; + } + } + } + + score + } + + /// Get a bundle definition by name. + pub fn get_bundle(&self, name: &str) -> Option<&BundleDefinition> { + self.bundles.get(name) + } + + /// List all bundle names. + pub fn bundle_names(&self) -> Vec<&str> { + let mut names: Vec<_> = self.bundles.keys().map(|s| s.as_str()).collect(); + names.sort(); + names + } + + /// Resolve a bundle into its constituent manifests. + /// Returns the manifests and any extension keys that couldn't be found. + pub fn resolve_bundle( + &self, + bundle_name: &str, + ) -> Result<(Vec<&ExtensionManifest>, Vec), RegistryError> { + let bundle = self + .bundles + .get(bundle_name) + .ok_or_else(|| RegistryError::BundleNotFound(bundle_name.to_string()))?; + + let mut found = Vec::new(); + let mut missing = Vec::new(); + + for ext_key in &bundle.extensions { + if let Some(manifest) = self.manifests.get(ext_key) { + found.push(manifest); + } else { + missing.push(ext_key.clone()); + } + } + + Ok((found, missing)) + } + + /// Check if a name refers to a bundle rather than an individual extension. + pub fn is_bundle(&self, name: &str) -> bool { + self.bundles.contains_key(name) + } + + /// Resolve a name to either a single manifest or the manifests in a bundle. + /// Returns (manifests, bundle_definition_if_bundle). + pub fn resolve( + &self, + name: &str, + ) -> Result<(Vec<&ExtensionManifest>, Option<&BundleDefinition>), RegistryError> { + // Check bundle first + if let Some(bundle) = self.bundles.get(name) { + let (manifests, missing) = self.resolve_bundle(name)?; + if !missing.is_empty() { + tracing::warn!( + "Bundle '{}' references missing extensions: {:?}", + name, + missing + ); + } + return Ok((manifests, Some(bundle))); + } + + // Single extension (use get_strict to catch ambiguous bare names) + let manifest = self.get_strict(name)?; + Ok((vec![manifest], None)) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use std::fs; + + fn create_test_registry(dir: &Path) { + let tools_dir = dir.join("tools"); + let channels_dir = dir.join("channels"); + fs::create_dir_all(&tools_dir).unwrap(); + fs::create_dir_all(&channels_dir).unwrap(); + + fs::write( + tools_dir.join("slack.json"), + r#"{ + "name": "slack", + "display_name": "Slack", + "kind": "tool", + "version": "0.1.0", + "description": "Post messages via Slack API", + "keywords": ["messaging", "chat"], + "source": { + "dir": "tools-src/slack", + "capabilities": "slack-tool.capabilities.json", + "crate_name": "slack-tool" + }, + "auth_summary": { + "method": "oauth", + "provider": "Slack", + "secrets": ["slack_bot_token"] + }, + "tags": ["default", "messaging"] + }"#, + ) + .unwrap(); + + fs::write( + tools_dir.join("github.json"), + r#"{ + "name": "github", + "display_name": "GitHub", + "kind": "tool", + "version": "0.1.0", + "description": "GitHub integration for issues and PRs", + "keywords": ["code", "git"], + "source": { + "dir": "tools-src/github", + "capabilities": "github-tool.capabilities.json", + "crate_name": "github-tool" + }, + "tags": ["default", "development"] + }"#, + ) + .unwrap(); + + fs::write( + channels_dir.join("telegram.json"), + r#"{ + "name": "telegram", + "display_name": "Telegram", + "kind": "channel", + "version": "0.1.0", + "description": "Telegram Bot API channel", + "source": { + "dir": "channels-src/telegram", + "capabilities": "telegram.capabilities.json", + "crate_name": "telegram-channel" + }, + "tags": ["messaging"] + }"#, + ) + .unwrap(); + + fs::write( + dir.join("_bundles.json"), + r#"{ + "bundles": { + "default": { + "display_name": "Recommended", + "extensions": ["tools/slack", "tools/github", "channels/telegram"] + }, + "messaging": { + "display_name": "Messaging", + "extensions": ["tools/slack", "channels/telegram"], + "shared_auth": null + } + } + }"#, + ) + .unwrap(); + } + + #[test] + fn test_load_catalog() { + let tmp = tempfile::tempdir().unwrap(); + create_test_registry(tmp.path()); + + let catalog = RegistryCatalog::load(tmp.path()).unwrap(); + assert_eq!(catalog.all().len(), 3); + } + + #[test] + fn test_list_by_kind() { + let tmp = tempfile::tempdir().unwrap(); + create_test_registry(tmp.path()); + + let catalog = RegistryCatalog::load(tmp.path()).unwrap(); + let tools = catalog.list(Some(ManifestKind::Tool), None); + assert_eq!(tools.len(), 2); + + let channels = catalog.list(Some(ManifestKind::Channel), None); + assert_eq!(channels.len(), 1); + } + + #[test] + fn test_list_by_tag() { + let tmp = tempfile::tempdir().unwrap(); + create_test_registry(tmp.path()); + + let catalog = RegistryCatalog::load(tmp.path()).unwrap(); + let defaults = catalog.list(None, Some("default")); + assert_eq!(defaults.len(), 2); + + let messaging = catalog.list(None, Some("messaging")); + assert_eq!(messaging.len(), 2); // slack (tool) and telegram (channel) both have "messaging" tag + } + + #[test] + fn test_get_by_name() { + let tmp = tempfile::tempdir().unwrap(); + create_test_registry(tmp.path()); + + let catalog = RegistryCatalog::load(tmp.path()).unwrap(); + + // Full key + assert!(catalog.get("tools/slack").is_some()); + + // Bare name + assert!(catalog.get("slack").is_some()); + assert!(catalog.get("telegram").is_some()); + + // Missing + assert!(catalog.get("nonexistent").is_none()); + } + + #[test] + fn test_search() { + let tmp = tempfile::tempdir().unwrap(); + create_test_registry(tmp.path()); + + let catalog = RegistryCatalog::load(tmp.path()).unwrap(); + + let results = catalog.search("slack"); + assert_eq!(results.len(), 1); + assert_eq!(results[0].name, "slack"); + + let results = catalog.search("messaging"); + assert!(!results.is_empty()); + + let results = catalog.search("nonexistent query"); + assert!(results.is_empty()); + } + + #[test] + fn test_resolve_bundle() { + let tmp = tempfile::tempdir().unwrap(); + create_test_registry(tmp.path()); + + let catalog = RegistryCatalog::load(tmp.path()).unwrap(); + + let (manifests, missing) = catalog.resolve_bundle("default").unwrap(); + assert_eq!(manifests.len(), 3); + assert!(missing.is_empty()); + + assert!(catalog.resolve_bundle("nonexistent").is_err()); + } + + #[test] + fn test_resolve_single_or_bundle() { + let tmp = tempfile::tempdir().unwrap(); + create_test_registry(tmp.path()); + + let catalog = RegistryCatalog::load(tmp.path()).unwrap(); + + // Single extension + let (manifests, bundle) = catalog.resolve("slack").unwrap(); + assert_eq!(manifests.len(), 1); + assert!(bundle.is_none()); + + // Bundle + let (manifests, bundle) = catalog.resolve("default").unwrap(); + assert_eq!(manifests.len(), 3); + assert!(bundle.is_some()); + } + + #[test] + fn test_bundle_names() { + let tmp = tempfile::tempdir().unwrap(); + create_test_registry(tmp.path()); + + let catalog = RegistryCatalog::load(tmp.path()).unwrap(); + let names = catalog.bundle_names(); + assert_eq!(names, vec!["default", "messaging"]); + } + + #[test] + fn test_directory_not_found() { + let result = RegistryCatalog::load(Path::new("/nonexistent/path")); + assert!(result.is_err()); + } +} diff --git a/src/registry/installer.rs b/src/registry/installer.rs new file mode 100644 index 00000000..87de6330 --- /dev/null +++ b/src/registry/installer.rs @@ -0,0 +1,415 @@ +//! Install extensions from the registry: build-from-source or download pre-built artifacts. + +use std::path::{Path, PathBuf}; + +use tokio::fs; + +use crate::registry::catalog::RegistryError; +use crate::registry::manifest::{BundleDefinition, ExtensionManifest, ManifestKind}; + +/// Result of installing a single extension from the registry. +#[derive(Debug)] +pub struct InstallOutcome { + /// Extension name. + pub name: String, + /// Whether this is a tool or channel. + pub kind: ManifestKind, + /// Destination path of the installed WASM binary. + pub wasm_path: PathBuf, + /// Whether a capabilities file was also installed. + pub has_capabilities: bool, + /// Any warning messages. + pub warnings: Vec, +} + +/// Handles installing extensions from registry manifests. +pub struct RegistryInstaller { + /// Root of the repo (parent of `registry/`), used to resolve `source.dir`. + repo_root: PathBuf, + /// Directory for installed tools (`~/.ironclaw/tools/`). + tools_dir: PathBuf, + /// Directory for installed channels (`~/.ironclaw/channels/`). + channels_dir: PathBuf, +} + +impl RegistryInstaller { + pub fn new(repo_root: PathBuf, tools_dir: PathBuf, channels_dir: PathBuf) -> Self { + Self { + repo_root, + tools_dir, + channels_dir, + } + } + + /// Default installer using standard paths. + pub fn with_defaults(repo_root: PathBuf) -> Self { + let home = dirs::home_dir().unwrap_or_else(|| PathBuf::from(".")); + Self { + repo_root, + tools_dir: home.join(".ironclaw").join("tools"), + channels_dir: home.join(".ironclaw").join("channels"), + } + } + + /// Install a single extension by building from source. + pub async fn install_from_source( + &self, + manifest: &ExtensionManifest, + force: bool, + ) -> Result { + let source_dir = self.repo_root.join(&manifest.source.dir); + if !source_dir.exists() { + return Err(RegistryError::ManifestRead { + path: source_dir.clone(), + reason: "source directory does not exist".to_string(), + }); + } + + let target_dir = match manifest.kind { + ManifestKind::Tool => &self.tools_dir, + ManifestKind::Channel => &self.channels_dir, + }; + + fs::create_dir_all(target_dir) + .await + .map_err(RegistryError::Io)?; + + // Use manifest.name for installed filenames so discovery, auth, and + // CLI commands (`ironclaw tool auth `) all agree on the stem. + let target_wasm = target_dir.join(format!("{}.wasm", manifest.name)); + + // Check if already exists + if target_wasm.exists() && !force { + return Err(RegistryError::AlreadyInstalled { + name: manifest.name.clone(), + path: target_wasm, + }); + } + + // Build the WASM component + println!( + "Building {} '{}' from {}...", + manifest.kind, + manifest.display_name, + source_dir.display() + ); + let crate_name = &manifest.source.crate_name; + let wasm_path = build_wasm_component(&source_dir, crate_name) + .await + .map_err(|e| RegistryError::ManifestRead { + path: source_dir.clone(), + reason: format!("build failed: {}", e), + })?; + + // Copy WASM binary + println!(" Installing to {}", target_wasm.display()); + fs::copy(&wasm_path, &target_wasm) + .await + .map_err(RegistryError::Io)?; + + // Copy capabilities file + let caps_source = source_dir.join(&manifest.source.capabilities); + let target_caps = target_dir.join(format!("{}.capabilities.json", manifest.name)); + let has_capabilities = if caps_source.exists() { + fs::copy(&caps_source, &target_caps) + .await + .map_err(RegistryError::Io)?; + true + } else { + false + }; + + let mut warnings = Vec::new(); + if !has_capabilities { + warnings.push(format!( + "No capabilities file found at {}", + caps_source.display() + )); + } + + Ok(InstallOutcome { + name: manifest.name.clone(), + kind: manifest.kind, + wasm_path: target_wasm, + has_capabilities, + warnings, + }) + } + + /// Download and install a pre-built artifact. + pub async fn install_from_artifact( + &self, + manifest: &ExtensionManifest, + force: bool, + ) -> Result { + let artifact = manifest.artifacts.get("wasm32-wasip2").ok_or_else(|| { + RegistryError::ExtensionNotFound(format!( + "No wasm32-wasip2 artifact for '{}'", + manifest.name + )) + })?; + + let url = artifact.url.as_ref().ok_or_else(|| { + RegistryError::ExtensionNotFound(format!( + "No artifact URL for '{}'. Use --build to build from source.", + manifest.name + )) + })?; + + let expected_sha = artifact.sha256.as_ref().ok_or_else(|| { + RegistryError::ExtensionNotFound(format!( + "No SHA256 hash for '{}'. Cannot verify download.", + manifest.name + )) + })?; + + let target_dir = match manifest.kind { + ManifestKind::Tool => &self.tools_dir, + ManifestKind::Channel => &self.channels_dir, + }; + + fs::create_dir_all(target_dir) + .await + .map_err(RegistryError::Io)?; + + let target_wasm = target_dir.join(format!("{}.wasm", manifest.name)); + + if target_wasm.exists() && !force { + return Err(RegistryError::AlreadyInstalled { + name: manifest.name.clone(), + path: target_wasm, + }); + } + + // Download + println!( + "Downloading {} '{}'...", + manifest.kind, manifest.display_name + ); + let response = reqwest::get(url) + .await + .map_err(|e| RegistryError::DownloadFailed { + url: url.clone(), + reason: format!("request failed: {}", e), + })?; + + let response = response + .error_for_status() + .map_err(|e| RegistryError::DownloadFailed { + url: url.clone(), + reason: e.to_string(), + })?; + + let bytes = response + .bytes() + .await + .map_err(|e| RegistryError::DownloadFailed { + url: url.clone(), + reason: format!("failed to read body: {}", e), + })?; + + // Verify SHA256 + use sha2::{Digest, Sha256}; + let mut hasher = Sha256::new(); + hasher.update(&bytes); + let actual_sha = format!("{:x}", hasher.finalize()); + + if actual_sha != *expected_sha { + return Err(RegistryError::DownloadFailed { + url: url.clone(), + reason: format!( + "SHA256 mismatch: expected {}, got {}", + expected_sha, actual_sha + ), + }); + } + + // Write file + fs::write(&target_wasm, &bytes) + .await + .map_err(RegistryError::Io)?; + + // Copy capabilities from source dir (still needed even for pre-built artifacts). + // NOTE: This requires the source tree to be present. When pre-built artifact + // distribution is implemented, capabilities should be bundled with the artifact + // or fetched from a separate URL. + let caps_source = self + .repo_root + .join(&manifest.source.dir) + .join(&manifest.source.capabilities); + let target_caps = target_dir.join(format!("{}.capabilities.json", manifest.name)); + let has_capabilities = if caps_source.exists() { + fs::copy(&caps_source, &target_caps) + .await + .map_err(RegistryError::Io)?; + true + } else { + false + }; + + println!(" Installed to {}", target_wasm.display()); + + Ok(InstallOutcome { + name: manifest.name.clone(), + kind: manifest.kind, + wasm_path: target_wasm, + has_capabilities, + warnings: Vec::new(), + }) + } + + /// Install a single manifest, choosing build vs download based on artifact availability and flags. + pub async fn install( + &self, + manifest: &ExtensionManifest, + force: bool, + prefer_build: bool, + ) -> Result { + let has_artifact = manifest + .artifacts + .get("wasm32-wasip2") + .and_then(|a| a.url.as_ref()) + .is_some(); + + if prefer_build || !has_artifact { + self.install_from_source(manifest, force).await + } else { + self.install_from_artifact(manifest, force).await + } + } + + /// Install all extensions in a bundle. + /// Returns the outcomes and any shared auth hints. + pub async fn install_bundle( + &self, + manifests: &[&ExtensionManifest], + bundle: &BundleDefinition, + force: bool, + prefer_build: bool, + ) -> (Vec, Vec) { + let mut outcomes = Vec::new(); + let mut errors = Vec::new(); + + for manifest in manifests { + match self.install(manifest, force, prefer_build).await { + Ok(outcome) => outcomes.push(outcome), + Err(e) => errors.push(format!("{}: {}", manifest.name, e)), + } + } + + // Collect auth hints + let mut auth_hints = Vec::new(); + if let Some(shared) = &bundle.shared_auth { + auth_hints.push(format!( + "Bundle uses shared auth '{}'. Run `ironclaw tool auth ` to authenticate all members.", + shared + )); + } + + // Collect unique auth providers that need setup + let mut seen_providers = std::collections::HashSet::new(); + for manifest in manifests { + if let Some(auth) = &manifest.auth_summary { + let key = auth + .shared_auth + .as_deref() + .unwrap_or(manifest.name.as_str()); + if seen_providers.insert(key.to_string()) + && let Some(url) = &auth.setup_url + { + auth_hints.push(format!( + " {} ({}): {}", + auth.provider.as_deref().unwrap_or(&manifest.name), + auth.method.as_deref().unwrap_or("manual"), + url + )); + } + } + } + + if !errors.is_empty() { + auth_hints.push(format!( + "\nFailed to install {} extension(s):", + errors.len() + )); + for err in errors { + auth_hints.push(format!(" - {}", err)); + } + } + + (outcomes, auth_hints) + } +} + +/// Build a WASM component from a source directory using `cargo component build --release`. +/// +/// Uses `tokio::process::Command` with inherited stdio so build progress is visible. +/// Looks for the specific `{crate_name}.wasm` in the release directory rather than +/// picking the first `.wasm` file found. +async fn build_wasm_component(source_dir: &Path, crate_name: &str) -> anyhow::Result { + use tokio::process::Command; + + // Check cargo-component availability + let check = Command::new("cargo") + .args(["component", "--version"]) + .stdout(std::process::Stdio::null()) + .stderr(std::process::Stdio::null()) + .status() + .await; + + if check.is_err() || !check.as_ref().map(|s| s.success()).unwrap_or(false) { + anyhow::bail!("cargo-component not found. Install with: cargo install cargo-component"); + } + + // Use status() with inherited stdio so build output streams to the terminal. + let status = Command::new("cargo") + .current_dir(source_dir) + .args(["component", "build", "--release"]) + .status() + .await?; + + if !status.success() { + anyhow::bail!("Build failed (exit code: {})", status); + } + + // Look for the specific crate's WASM file (Cargo uses underscores in artifact names). + let wasm_filename = format!("{}.wasm", crate_name.replace('-', "_")); + let target_base = source_dir.join("target"); + let candidates = [ + "wasm32-wasip1", + "wasm32-wasip2", + "wasm32-wasi", + "wasm32-unknown-unknown", + ]; + + for target in &candidates { + let wasm_path = target_base + .join(target) + .join("release") + .join(&wasm_filename); + if wasm_path.exists() { + return Ok(wasm_path); + } + } + + anyhow::bail!( + "Could not find {} in {}/target/*/release/", + wasm_filename, + source_dir.display() + ) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn test_installer_creation() { + let installer = RegistryInstaller::new( + PathBuf::from("/repo"), + PathBuf::from("/home/.ironclaw/tools"), + PathBuf::from("/home/.ironclaw/channels"), + ); + assert_eq!(installer.repo_root, PathBuf::from("/repo")); + } +} diff --git a/src/registry/manifest.rs b/src/registry/manifest.rs new file mode 100644 index 00000000..4a1c5591 --- /dev/null +++ b/src/registry/manifest.rs @@ -0,0 +1,271 @@ +//! Serde structs for extension registry manifests. +//! +//! Each manifest describes a single extension (tool or channel) with its source +//! location, build artifacts, authentication requirements, and tags. + +use serde::{Deserialize, Serialize}; + +use crate::extensions::{AuthHint, ExtensionKind, ExtensionSource, RegistryEntry}; + +/// A single extension manifest loaded from `registry/{tools,channels}/.json`. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct ExtensionManifest { + /// Unique identifier (matches crate name stem, e.g. "slack"). + pub name: String, + + /// Human-readable name (e.g. "Slack"). + pub display_name: String, + + /// Whether this is a tool or channel. + pub kind: ManifestKind, + + /// Semver version from Cargo.toml. + pub version: String, + + /// One-line description. + pub description: String, + + /// Search keywords beyond the name. + #[serde(default)] + pub keywords: Vec, + + /// Source code location and build info. + pub source: SourceSpec, + + /// Pre-built binary artifacts keyed by target triple. + #[serde(default)] + pub artifacts: std::collections::HashMap, + + /// Summary of authentication requirements. + #[serde(default)] + pub auth_summary: Option, + + /// Tags for filtering (e.g. "default", "messaging", "google"). + #[serde(default)] + pub tags: Vec, +} + +/// Extension kind as declared in manifests. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum ManifestKind { + Tool, + Channel, +} + +impl From for ExtensionKind { + fn from(kind: ManifestKind) -> Self { + match kind { + ManifestKind::Tool => ExtensionKind::WasmTool, + ManifestKind::Channel => ExtensionKind::WasmChannel, + } + } +} + +impl std::fmt::Display for ManifestKind { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + ManifestKind::Tool => write!(f, "tool"), + ManifestKind::Channel => write!(f, "channel"), + } + } +} + +/// Source code location for building from source. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct SourceSpec { + /// Path relative to repo root (e.g. "tools-src/slack"). + pub dir: String, + + /// Capabilities filename relative to source dir. + pub capabilities: String, + + /// Rust crate name for `cargo component build`. + pub crate_name: String, +} + +/// A pre-built binary artifact. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct ArtifactSpec { + /// Download URL (null until release). + pub url: Option, + + /// Hex SHA256 of the WASM binary (null until release). + pub sha256: Option, +} + +/// Summary of authentication requirements extracted from capabilities. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct AuthSummary { + /// Auth method: "oauth", "manual", or "none". + #[serde(default)] + pub method: Option, + + /// Display name for the auth provider (e.g. "Google", "Slack"). + #[serde(default)] + pub provider: Option, + + /// Secret names required by this extension. + #[serde(default)] + pub secrets: Vec, + + /// If this extension shares auth with others (e.g. all Google tools share + /// `google_oauth_token`), this is the shared secret name. + #[serde(default)] + pub shared_auth: Option, + + /// URL where users can set up credentials. + #[serde(default)] + pub setup_url: Option, +} + +/// Bundle definition grouping related extensions. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct BundleDefinition { + /// Human-readable name. + pub display_name: String, + + /// Description of what this bundle contains. + #[serde(default)] + pub description: Option, + + /// Extension references as "tools/" or "channels/". + pub extensions: Vec, + + /// Shared auth secret across bundle members (if any). + #[serde(default)] + pub shared_auth: Option, +} + +/// Top-level structure of `_bundles.json`. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct BundlesFile { + pub bundles: std::collections::HashMap, +} + +impl ExtensionManifest { + /// Convert this manifest into a [`RegistryEntry`] for use with the in-chat + /// extension discovery system. + pub fn to_registry_entry(&self) -> RegistryEntry { + let source = ExtensionSource::WasmBuildable { + repo_url: self.source.dir.clone(), + build_dir: Some(self.source.dir.clone()), + }; + + let auth_hint = match self.auth_summary.as_ref().and_then(|a| a.method.as_deref()) { + Some("oauth") => AuthHint::CapabilitiesAuth, + Some("manual") => AuthHint::CapabilitiesAuth, + Some("none") | None => AuthHint::None, + Some(_) => AuthHint::CapabilitiesAuth, + }; + + RegistryEntry { + name: self.name.clone(), + display_name: self.display_name.clone(), + kind: self.kind.into(), + description: self.description.clone(), + keywords: self.keywords.clone(), + source, + auth_hint, + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn test_parse_tool_manifest() { + let json = r#"{ + "name": "slack", + "display_name": "Slack", + "kind": "tool", + "version": "0.1.0", + "description": "Post messages via Slack API", + "keywords": ["messaging"], + "source": { + "dir": "tools-src/slack", + "capabilities": "slack-tool.capabilities.json", + "crate_name": "slack-tool" + }, + "artifacts": { + "wasm32-wasip2": { "url": null, "sha256": null } + }, + "auth_summary": { + "method": "oauth", + "provider": "Slack", + "secrets": ["slack_bot_token"], + "shared_auth": null, + "setup_url": "https://api.slack.com/apps" + }, + "tags": ["default", "messaging"] + }"#; + + let manifest: ExtensionManifest = serde_json::from_str(json).expect("parse manifest"); + assert_eq!(manifest.name, "slack"); + assert_eq!(manifest.kind, ManifestKind::Tool); + assert_eq!(manifest.version, "0.1.0"); + assert!(manifest.tags.contains(&"default".to_string())); + + let entry = manifest.to_registry_entry(); + assert_eq!(entry.kind, ExtensionKind::WasmTool); + } + + #[test] + fn test_parse_channel_manifest() { + let json = r#"{ + "name": "telegram", + "display_name": "Telegram", + "kind": "channel", + "version": "0.1.0", + "description": "Telegram Bot API channel", + "source": { + "dir": "channels-src/telegram", + "capabilities": "telegram.capabilities.json", + "crate_name": "telegram-channel" + }, + "tags": ["messaging"] + }"#; + + let manifest: ExtensionManifest = serde_json::from_str(json).expect("parse manifest"); + assert_eq!(manifest.kind, ManifestKind::Channel); + assert!(manifest.auth_summary.is_none()); + assert!(manifest.artifacts.is_empty()); + + let entry = manifest.to_registry_entry(); + assert_eq!(entry.kind, ExtensionKind::WasmChannel); + } + + #[test] + fn test_parse_bundles() { + let json = r#"{ + "bundles": { + "google": { + "display_name": "Google Suite", + "description": "All Google tools", + "extensions": ["tools/gmail", "tools/google-calendar"], + "shared_auth": "google_oauth_token" + }, + "default": { + "display_name": "Recommended Set", + "extensions": ["tools/github", "tools/slack"] + } + } + }"#; + + let bundles: BundlesFile = serde_json::from_str(json).expect("parse bundles"); + assert_eq!(bundles.bundles.len(), 2); + assert_eq!( + bundles.bundles["google"].shared_auth.as_deref(), + Some("google_oauth_token") + ); + assert!(bundles.bundles["default"].shared_auth.is_none()); + } + + #[test] + fn test_manifest_kind_display() { + assert_eq!(ManifestKind::Tool.to_string(), "tool"); + assert_eq!(ManifestKind::Channel.to_string(), "channel"); + } +} diff --git a/src/registry/mod.rs b/src/registry/mod.rs new file mode 100644 index 00000000..a86fb5fc --- /dev/null +++ b/src/registry/mod.rs @@ -0,0 +1,23 @@ +//! Extension registry: metadata catalog for tools and channels. +//! +//! The registry provides a central index of all available extensions (WASM tools +//! and channels) with their source locations, build artifacts, authentication +//! requirements, and grouping via bundles. +//! +//! ```text +//! registry/ +//! ├── tools/ <- One JSON manifest per tool +//! ├── channels/ <- One JSON manifest per channel +//! └── _bundles.json <- Bundle definitions (google, messaging, default) +//! ``` + +pub mod catalog; +pub mod installer; +pub mod manifest; + +pub use catalog::{RegistryCatalog, RegistryError}; +pub use installer::RegistryInstaller; +pub use manifest::{ + ArtifactSpec, AuthSummary, BundleDefinition, BundlesFile, ExtensionManifest, ManifestKind, + SourceSpec, +}; diff --git a/src/setup/README.md b/src/setup/README.md index 36889d30..9c72d390 100644 --- a/src/setup/README.md +++ b/src/setup/README.md @@ -50,7 +50,7 @@ The `--no-onboard` CLI flag suppresses auto-detection. --- -## The 7-Step Wizard +## The 8-Step Wizard ### Overview @@ -61,7 +61,8 @@ Step 3: Inference Provider ← skipped if --skip-auth Step 4: Model Selection Step 5: Embeddings Step 6: Channel Configuration -Step 7: Background Tasks (heartbeat) +Step 7: Extensions (tools) +Step 8: Background Tasks (heartbeat) ↓ save_and_summarize() ``` @@ -243,13 +244,20 @@ key first, then falls back to the standard env var. ``` 6a. Tunnel setup (if webhook channels needed) 6b. Discover WASM channels from ~/.ironclaw/channels/ -6c. Multi-select: CLI/TUI, HTTP, discovered channels, bundled channels -6d. Install missing bundled channels (copy WASM binaries) -6e. Initialize SecretsContext (for token storage) -6f. Setup HTTP webhook (if selected) -6g. Setup each WASM channel (secrets, owner binding) +6c. Build channel options: discovered + bundled + registry catalog +6d. Multi-select: CLI/TUI, HTTP, all available channels +6e. Install missing bundled channels (copy WASM binaries) +6f. Install missing registry channels (build from source) +6g. Initialize SecretsContext (for token storage) +6h. Setup HTTP webhook (if selected) +6i. Setup each WASM channel (secrets, owner binding) ``` +**Channel sources** (priority order for installation): +1. Already installed in `~/.ironclaw/channels/` +2. Bundled channels (pre-compiled in `channels-src/`) +3. Registry channels (`registry/channels/*.json`, built from source) + **Tunnel setup** (`setup_tunnel`): - Options: ngrok, Cloudflare Tunnel, localtunnel, custom URL - Validates HTTPS requirement @@ -273,7 +281,33 @@ key first, then falls back to the standard env var. --- -### Step 7: Heartbeat +### Step 7: Extensions (Tools) + +**Module:** `wizard.rs` → `step_extensions()` + +**Goal:** Install WASM tools from the extension registry. + +**Flow:** +1. Load `RegistryCatalog` from `registry/` directory +2. If registry not found, print info and skip +3. List all tool manifests from the catalog +4. Discover already-installed tools in `~/.ironclaw/tools/` +5. Multi-select: show all registry tools with display name, auth method, + and description. Pre-check tools tagged `"default"` and already installed. +6. For each selected tool not yet installed, build from source via + `RegistryInstaller::install_from_source()` +7. Print consolidated auth hints (deduplicated by provider, e.g. one hint + for all Google tools sharing `google_oauth_token`) + +**Registry lookup** (`load_registry_catalog`): +Searches for `registry/` directory in order: +1. Current working directory +2. Next to the executable +3. `CARGO_MANIFEST_DIR` (compile-time, dev builds) + +--- + +### Step 8: Heartbeat **Module:** `wizard.rs` → `step_heartbeat()` diff --git a/src/setup/channels.rs b/src/setup/channels.rs index 5b0f66bf..36cc7049 100644 --- a/src/setup/channels.rs +++ b/src/setup/channels.rs @@ -363,12 +363,54 @@ pub fn setup_tunnel(settings: &Settings) -> Result { + print_info(" Provider: ngrok"); + if let Some(ref domain) = t.ngrok_domain { + print_info(&format!(" Domain: {}", domain)); + } + if t.ngrok_token.is_some() { + print_info(" Auth: token configured"); + } + } + Some("cloudflare") => { + print_info(" Provider: Cloudflare Tunnel"); + if t.cf_token.is_some() { + print_info(" Auth: token configured"); + } + } + Some("tailscale") => { + let mode = if t.ts_funnel { + "Funnel (public)" + } else { + "Serve (tailnet-only)" + }; + print_info(&format!(" Provider: Tailscale {}", mode)); + if let Some(ref hostname) = t.ts_hostname { + print_info(&format!(" Hostname: {}", hostname)); + } + } + Some("custom") => { + print_info(" Provider: Custom command"); + if let Some(ref cmd) = t.custom_command { + print_info(&format!(" Command: {}", cmd)); + } + if let Some(ref url) = t.custom_health_url { + print_info(&format!(" Health: {}", url)); + } + } + Some(other) => { + print_info(&format!(" Provider: {}", other)); + } + None => {} } - if let Some(ref provider) = settings.tunnel.provider { - print_info(&format!("Existing managed provider: {}", provider)); + if let Some(ref url) = t.public_url { + print_info(&format!(" URL: {}", url)); } + println!(); if !confirm("Change tunnel configuration?", false)? { return Ok(settings.tunnel.clone()); } diff --git a/src/setup/mod.rs b/src/setup/mod.rs index f2501a57..b556ba92 100644 --- a/src/setup/mod.rs +++ b/src/setup/mod.rs @@ -7,7 +7,8 @@ //! 4. Model selection //! 5. Embeddings //! 6. Channel configuration (HTTP, Telegram, etc.) -//! 7. Heartbeat (background tasks) +//! 7. Extensions (tool installation from registry) +//! 8. Heartbeat (background tasks) //! //! # Example //! diff --git a/src/setup/wizard.rs b/src/setup/wizard.rs index 9dc50fad..7947d511 100644 --- a/src/setup/wizard.rs +++ b/src/setup/wizard.rs @@ -7,7 +7,8 @@ //! 4. Model selection //! 5. Embeddings //! 6. Channel configuration -//! 7. Heartbeat (background tasks) +//! 7. Extensions (tool installation from registry) +//! 8. Heartbeat (background tasks) use std::collections::{HashMap, HashSet}; use std::sync::Arc; @@ -128,11 +129,13 @@ impl SetupWizard { print_header("IronClaw Setup Wizard"); if self.config.channels_only { - // Channels-only mode: just step 6 + // Channels-only mode: reconnect to existing DB and load settings + // before running the channel step, so secrets and save work. + self.reconnect_existing_db().await?; print_step(1, 1, "Channel Configuration"); self.step_channels().await?; } else { - let total_steps = 7; + let total_steps = 8; // Step 1: Database print_step(1, total_steps, "Database Connection"); @@ -162,8 +165,12 @@ impl SetupWizard { print_step(6, total_steps, "Channel Configuration"); self.step_channels().await?; - // Step 7: Heartbeat - print_step(7, total_steps, "Background Tasks"); + // Step 7: Extensions (tools) + print_step(7, total_steps, "Extensions"); + self.step_extensions().await?; + + // Step 8: Heartbeat + print_step(8, total_steps, "Background Tasks"); self.step_heartbeat()?; } @@ -173,6 +180,99 @@ impl SetupWizard { Ok(()) } + /// Reconnect to the existing database and load settings. + /// + /// Used by channels-only mode (and future single-step modes) so that + /// `init_secrets_context()` and `save_and_summarize()` have a live + /// database connection and the wizard's `self.settings` reflects the + /// previously saved configuration. + async fn reconnect_existing_db(&mut self) -> Result<(), SetupError> { + // Determine backend from env (set by bootstrap .env loaded in main). + let backend = std::env::var("DATABASE_BACKEND").unwrap_or_else(|_| "postgres".to_string()); + + // Try libsql first if that's the configured backend. + #[cfg(feature = "libsql")] + if backend == "libsql" || backend == "turso" || backend == "sqlite" { + return self.reconnect_libsql().await; + } + + // Try postgres (either explicitly configured or as default). + #[cfg(feature = "postgres")] + { + let _ = &backend; + return self.reconnect_postgres().await; + } + + #[allow(unreachable_code)] + Err(SetupError::Database( + "No database configured. Run full setup first (ironclaw onboard).".to_string(), + )) + } + + /// Reconnect to an existing PostgreSQL database and load settings. + #[cfg(feature = "postgres")] + async fn reconnect_postgres(&mut self) -> Result<(), SetupError> { + let url = std::env::var("DATABASE_URL").map_err(|_| { + SetupError::Database( + "DATABASE_URL not set. Run full setup first (ironclaw onboard).".to_string(), + ) + })?; + + self.test_database_connection_postgres(&url).await?; + self.settings.database_backend = Some("postgres".to_string()); + self.settings.database_url = Some(url.clone()); + + // Load existing settings from DB, then restore connection fields that + // may not be persisted in the settings map. + if let Some(ref pool) = self.db_pool { + let store = crate::history::Store::from_pool(pool.clone()); + if let Ok(map) = store.get_all_settings("default").await { + self.settings = Settings::from_db_map(&map); + self.settings.database_backend = Some("postgres".to_string()); + self.settings.database_url = Some(url); + } + } + + Ok(()) + } + + /// Reconnect to an existing libSQL database and load settings. + #[cfg(feature = "libsql")] + async fn reconnect_libsql(&mut self) -> Result<(), SetupError> { + let path = std::env::var("LIBSQL_PATH").unwrap_or_else(|_| { + crate::config::default_libsql_path() + .to_string_lossy() + .to_string() + }); + let turso_url = std::env::var("LIBSQL_URL").ok(); + let turso_token = std::env::var("LIBSQL_AUTH_TOKEN").ok(); + + self.test_database_connection_libsql(&path, turso_url.as_deref(), turso_token.as_deref()) + .await?; + + self.settings.database_backend = Some("libsql".to_string()); + self.settings.libsql_path = Some(path.clone()); + if let Some(ref url) = turso_url { + self.settings.libsql_url = Some(url.clone()); + } + + // Load existing settings from DB, then restore connection fields that + // may not be persisted in the settings map. + if let Some(ref db) = self.db_backend { + use crate::db::SettingsStore as _; + if let Ok(map) = db.get_all_settings("default").await { + self.settings = Settings::from_db_map(&map); + self.settings.database_backend = Some("libsql".to_string()); + self.settings.libsql_path = Some(path); + if let Some(url) = turso_url { + self.settings.libsql_url = Some(url); + } + } + } + + Ok(()) + } + /// Step 1: Database connection. async fn step_database(&mut self) -> Result<(), SetupError> { // When both features are compiled, let the user choose. @@ -1284,7 +1384,9 @@ impl SetupWizard { .iter() .map(|(name, _)| name.clone()) .collect(); - let wasm_channel_names = wasm_channel_option_names(&discovered_channels); + + // Build channel list from registry (if available) + bundled + discovered + let wasm_channel_names = build_channel_options(&discovered_channels); // Build options list dynamically let mut options: Vec<(String, bool)> = vec![ @@ -1295,11 +1397,15 @@ impl SetupWizard { ), ]; - // Add available WASM channels (installed + bundled) + // Add available WASM channels (installed + bundled + registry) for name in &wasm_channel_names { let is_enabled = self.settings.channels.wasm_channels.contains(name); - let display_name = format!("{} (WASM)", capitalize_first(name)); - options.push((display_name, is_enabled)); + let label = if installed_names.contains(name) { + format!("{} (installed)", capitalize_first(name)) + } else { + format!("{} (will install)", capitalize_first(name)) + }; + options.push((label, is_enabled)); } let options_refs: Vec<(&str, bool)> = @@ -1320,6 +1426,10 @@ impl SetupWizard { }) .collect(); + // Install selected channels that aren't already on disk + let mut any_installed = false; + + // Try bundled channels first (pre-compiled artifacts from channels-src/) if let Some(installed) = install_selected_bundled_channels( &channels_dir, &selected_wasm_channels, @@ -1328,7 +1438,31 @@ impl SetupWizard { .await? && !installed.is_empty() { - print_success(&format!("Installed channels: {}", installed.join(", "))); + print_success(&format!( + "Installed bundled channels: {}", + installed.join(", ") + )); + any_installed = true; + } + + // Then try registry channels (build from source for any still missing) + let installed_from_registry = install_selected_registry_channels( + &channels_dir, + &selected_wasm_channels, + &installed_names, + ) + .await; + + if !installed_from_registry.is_empty() { + print_success(&format!( + "Built from registry: {}", + installed_from_registry.join(", ") + )); + any_installed = true; + } + + // Re-discover after installs + if any_installed { discovered_channels = discover_wasm_channels(&channels_dir).await; } @@ -1419,7 +1553,134 @@ impl SetupWizard { Ok(()) } - /// Step 7: Heartbeat configuration. + /// Step 7: Extensions (tools) installation from registry. + async fn step_extensions(&mut self) -> Result<(), SetupError> { + let catalog = match load_registry_catalog() { + Some(c) => c, + None => { + print_info("Extension registry not found. Skipping tool installation."); + print_info("Install tools manually with: ironclaw tool install "); + return Ok(()); + } + }; + + let tools: Vec<_> = catalog + .list(Some(crate::registry::manifest::ManifestKind::Tool), None) + .into_iter() + .cloned() + .collect(); + + if tools.is_empty() { + print_info("No tools found in registry."); + return Ok(()); + } + + print_info("Available tools from the extension registry:"); + print_info("Select which tools to install. You can install more later with:"); + print_info(" ironclaw registry install "); + println!(); + + // Check which tools are already installed + let tools_dir = dirs::home_dir() + .ok_or_else(|| SetupError::Config("Could not determine home directory".into()))? + .join(".ironclaw/tools"); + + let installed_tools = discover_installed_tools(&tools_dir).await; + + // Build options: show display_name + description, pre-check "default" tagged + already installed + let mut options: Vec<(String, bool)> = Vec::new(); + for tool in &tools { + let is_installed = installed_tools.contains(&tool.name); + let is_default = tool.tags.contains(&"default".to_string()); + let status = if is_installed { " (installed)" } else { "" }; + let auth_hint = tool + .auth_summary + .as_ref() + .and_then(|a| a.method.as_deref()) + .map(|m| format!(" [{}]", m)) + .unwrap_or_default(); + + let label = format!( + "{}{}{} - {}", + tool.display_name, auth_hint, status, tool.description + ); + options.push((label, is_default || is_installed)); + } + + let options_refs: Vec<(&str, bool)> = + options.iter().map(|(s, b)| (s.as_str(), *b)).collect(); + + let selected = select_many("Which tools do you want to install?", &options_refs) + .map_err(SetupError::Io)?; + + if selected.is_empty() { + print_info("No tools selected."); + return Ok(()); + } + + // Install selected tools that aren't already on disk + let repo_root = catalog.root().parent().unwrap_or(catalog.root()); + let installer = crate::registry::installer::RegistryInstaller::new( + repo_root.to_path_buf(), + tools_dir.clone(), + dirs::home_dir() + .unwrap_or_default() + .join(".ironclaw/channels"), + ); + + let mut installed_count = 0; + let mut auth_needed: Vec = Vec::new(); + + for idx in &selected { + let tool = &tools[*idx]; + if installed_tools.contains(&tool.name) { + continue; // Already installed, skip + } + + match installer.install_from_source(tool, false).await { + Ok(outcome) => { + print_success(&format!("Installed {}", outcome.name)); + installed_count += 1; + + // Track auth needs + if let Some(auth) = &tool.auth_summary + && auth.method.as_deref() != Some("none") + && auth.method.is_some() + { + let provider = auth.provider.as_deref().unwrap_or(&tool.name); + // Only mention unique providers (Google tools share auth) + let hint = format!(" {} - ironclaw tool auth {}", provider, tool.name); + if !auth_needed + .iter() + .any(|h| h.starts_with(&format!(" {} -", provider))) + { + auth_needed.push(hint); + } + } + } + Err(e) => { + print_error(&format!("Failed to install {}: {}", tool.display_name, e)); + } + } + } + + if installed_count > 0 { + println!(); + print_success(&format!("{} tool(s) installed.", installed_count)); + } + + if !auth_needed.is_empty() { + println!(); + print_info("Some tools need authentication. Run after setup:"); + for hint in &auth_needed { + print_info(hint); + } + } + + Ok(()) + } + + /// Step 8: Heartbeat configuration. fn step_heartbeat(&mut self) -> Result<(), SetupError> { print_info("Heartbeat runs periodic background tasks (e.g., checking your calendar,"); print_info("monitoring for notifications, running scheduled workflows)."); @@ -2087,15 +2348,161 @@ async fn install_missing_bundled_channels( Ok(installed) } -fn wasm_channel_option_names(discovered: &[(String, ChannelCapabilitiesFile)]) -> Vec { +/// Build channel options from discovered channels + bundled + registry catalog. +/// +/// Returns a deduplicated, sorted list of channel names available for selection. +fn build_channel_options(discovered: &[(String, ChannelCapabilitiesFile)]) -> Vec { let mut names: Vec = discovered.iter().map(|(name, _)| name.clone()).collect(); + // Add bundled channels for bundled in available_channel_names().iter().copied() { if !names.iter().any(|name| name == bundled) { names.push(bundled.to_string()); } } + // Add registry channels + if let Some(catalog) = load_registry_catalog() { + for manifest in catalog.list(Some(crate::registry::manifest::ManifestKind::Channel), None) { + if !names.iter().any(|n| n == &manifest.name) { + names.push(manifest.name.clone()); + } + } + } + + names.sort(); + names +} + +/// Try to load the registry catalog. Returns None if the registry directory +/// cannot be found (e.g. running from an installed binary without the repo). +fn load_registry_catalog() -> Option { + // Try relative to current directory (dev usage) + let cwd = std::env::current_dir().ok()?; + let candidate = cwd.join("registry"); + if candidate.is_dir() { + return crate::registry::catalog::RegistryCatalog::load(&candidate).ok(); + } + + // Try relative to executable + if let Ok(exe) = std::env::current_exe() + && let Some(parent) = exe.parent() + { + let candidate = parent.join("registry"); + if candidate.is_dir() { + return crate::registry::catalog::RegistryCatalog::load(&candidate).ok(); + } + if let Some(grandparent) = parent.parent() { + let candidate = grandparent.join("registry"); + if candidate.is_dir() { + return crate::registry::catalog::RegistryCatalog::load(&candidate).ok(); + } + } + } + + // Try CARGO_MANIFEST_DIR (compile-time, works in dev builds) + let manifest_dir = std::path::Path::new(env!("CARGO_MANIFEST_DIR")); + let candidate = manifest_dir.join("registry"); + if candidate.is_dir() { + return crate::registry::catalog::RegistryCatalog::load(&candidate).ok(); + } + + None +} + +/// Install selected channels from the registry that aren't already on disk +/// and weren't handled by the bundled installer. +/// +/// This builds channels from source using `cargo component build`. +async fn install_selected_registry_channels( + channels_dir: &std::path::Path, + selected_channels: &[String], + already_installed: &HashSet, +) -> Vec { + let catalog = match load_registry_catalog() { + Some(c) => c, + None => return Vec::new(), + }; + + let repo_root = catalog + .root() + .parent() + .unwrap_or(catalog.root()) + .to_path_buf(); + + let bundled: HashSet<&str> = available_channel_names().iter().copied().collect(); + let mut installed = Vec::new(); + + for name in selected_channels { + // Skip if already installed or handled by bundled installer + if already_installed.contains(name) || bundled.contains(name.as_str()) { + continue; + } + + // Check if already on disk (may have been installed between bundled and here) + let wasm_on_disk = channels_dir.join(format!("{}.wasm", name)).exists() + || channels_dir.join(format!("{}-channel.wasm", name)).exists(); + if wasm_on_disk { + continue; + } + + // Look up in registry + let manifest = match catalog.get(&format!("channels/{}", name)) { + Some(m) => m, + None => continue, + }; + + let installer = crate::registry::installer::RegistryInstaller::new( + repo_root.clone(), + dirs::home_dir().unwrap_or_default().join(".ironclaw/tools"), + channels_dir.to_path_buf(), + ); + + match installer.install_from_source(manifest, false).await { + Ok(_) => { + installed.push(name.clone()); + } + Err(e) => { + tracing::warn!( + channel = %name, + error = %e, + "Failed to install channel from registry" + ); + crate::setup::prompts::print_error(&format!( + "Failed to install channel '{}': {}", + name, e + )); + } + } + } + + installed +} + +/// Discover which tools are already installed in the tools directory. +/// +/// Returns a set of tool names (the stem of .wasm files). +async fn discover_installed_tools(tools_dir: &std::path::Path) -> HashSet { + let mut names = HashSet::new(); + + if !tools_dir.is_dir() { + return names; + } + + let mut entries = match tokio::fs::read_dir(tools_dir).await { + Ok(e) => e, + Err(_) => return names, + }; + + while let Ok(Some(entry)) = entries.next_entry().await { + let path = entry.path(); + if path.extension().and_then(|e| e.to_str()) == Some("wasm") + && let Some(stem) = path.file_stem().and_then(|s| s.to_str()) + { + names.insert(stem.to_string()); + } + } + names } @@ -2209,9 +2616,9 @@ mod tests { } #[test] - fn test_wasm_channel_option_names_includes_available_when_missing() { + fn test_build_channel_options_includes_available_when_missing() { let discovered = Vec::new(); - let options = wasm_channel_option_names(&discovered); + let options = build_channel_options(&discovered); let available = available_channel_names(); // All available (built) channels should appear for name in &available { @@ -2224,9 +2631,9 @@ mod tests { } #[test] - fn test_wasm_channel_option_names_dedupes_available() { + fn test_build_channel_options_dedupes_available() { let discovered = vec![(String::from("telegram"), ChannelCapabilitiesFile::default())]; - let options = wasm_channel_option_names(&discovered); + let options = build_channel_options(&discovered); // telegram should appear exactly once despite being both discovered and available assert_eq!( options.iter().filter(|n| *n == "telegram").count(), diff --git a/tools-src/github/Cargo.toml b/tools-src/github/Cargo.toml index 585e2679..a9cc865d 100644 --- a/tools-src/github/Cargo.toml +++ b/tools-src/github/Cargo.toml @@ -20,3 +20,5 @@ lto = true strip = true codegen-units = 1 + +[workspace] diff --git a/tools-src/gmail/Cargo.toml b/tools-src/gmail/Cargo.toml index 533f2aa4..205292aa 100644 --- a/tools-src/gmail/Cargo.toml +++ b/tools-src/gmail/Cargo.toml @@ -19,3 +19,5 @@ opt-level = "s" lto = true strip = true codegen-units = 1 + +[workspace] diff --git a/tools-src/google-calendar/Cargo.toml b/tools-src/google-calendar/Cargo.toml index a6c9a5a4..0b5ef361 100644 --- a/tools-src/google-calendar/Cargo.toml +++ b/tools-src/google-calendar/Cargo.toml @@ -19,3 +19,5 @@ opt-level = "s" lto = true strip = true codegen-units = 1 + +[workspace] diff --git a/tools-src/google-docs/Cargo.toml b/tools-src/google-docs/Cargo.toml index 7348343d..8590c2be 100644 --- a/tools-src/google-docs/Cargo.toml +++ b/tools-src/google-docs/Cargo.toml @@ -19,3 +19,5 @@ opt-level = "s" lto = true strip = true codegen-units = 1 + +[workspace] diff --git a/tools-src/google-drive/Cargo.toml b/tools-src/google-drive/Cargo.toml index 2b07f666..3385c14a 100644 --- a/tools-src/google-drive/Cargo.toml +++ b/tools-src/google-drive/Cargo.toml @@ -19,3 +19,5 @@ opt-level = "s" lto = true strip = true codegen-units = 1 + +[workspace] diff --git a/tools-src/google-sheets/Cargo.toml b/tools-src/google-sheets/Cargo.toml index 39a52e18..048c44de 100644 --- a/tools-src/google-sheets/Cargo.toml +++ b/tools-src/google-sheets/Cargo.toml @@ -19,3 +19,5 @@ opt-level = "s" lto = true strip = true codegen-units = 1 + +[workspace] diff --git a/tools-src/google-slides/Cargo.toml b/tools-src/google-slides/Cargo.toml index f6a3bfe0..c0e3d42b 100644 --- a/tools-src/google-slides/Cargo.toml +++ b/tools-src/google-slides/Cargo.toml @@ -19,3 +19,5 @@ opt-level = "s" lto = true strip = true codegen-units = 1 + +[workspace] diff --git a/tools-src/okta/Cargo.toml b/tools-src/okta/Cargo.toml index e399d494..5265cf4d 100644 --- a/tools-src/okta/Cargo.toml +++ b/tools-src/okta/Cargo.toml @@ -19,3 +19,5 @@ opt-level = "s" lto = true strip = true codegen-units = 1 + +[workspace] diff --git a/tools-src/slack/Cargo.toml b/tools-src/slack/Cargo.toml index cb3c0ad2..ee22922c 100644 --- a/tools-src/slack/Cargo.toml +++ b/tools-src/slack/Cargo.toml @@ -19,3 +19,5 @@ opt-level = "s" lto = true strip = true codegen-units = 1 + +[workspace] diff --git a/tools-src/telegram/Cargo.toml b/tools-src/telegram/Cargo.toml index ed283acf..9af023c5 100644 --- a/tools-src/telegram/Cargo.toml +++ b/tools-src/telegram/Cargo.toml @@ -24,3 +24,5 @@ opt-level = "s" lto = true strip = true codegen-units = 1 + +[workspace]