feat: extension registry with metadata catalog and onboarding integration (#238)

* feat: add extension registry with metadata catalog, CLI, and onboarding integration

Adds a central registry that catalogs all 14 available extensions (10 tools,
4 channels) with their capabilities, auth requirements, and artifact references.
The onboarding wizard now shows installable channels from the registry and
offers tool installation as a new Step 7.

- registry/ folder with per-extension JSON manifests and bundle definitions
- src/registry/ module: manifest structs, catalog loader, installer
- `ironclaw registry list|info|install|install-defaults` CLI commands
- Setup wizard enhanced: channels from registry, new extensions step (8 steps)

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix(setup): resolve workspace errors for tool crates and channels-only onboarding

Tool crates in tools-src/ and channels-src/ failed `cargo metadata` during
onboard install because Cargo resolved them as part of the root workspace.
Add `[workspace]` table to each standalone crate and extend the root
`workspace.exclude` list so they build independently.

Channels-only mode (`onboard --channels-only`) failed with "Secrets not
configured" and "No database connection" because it skipped database and
security setup. Add `reconnect_existing_db()` to establish the DB connection
and load saved settings before running channel configuration.

Also improve the tunnel "already configured" display to show full provider
details (domain, mode, command) instead of just the provider name.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix(registry): address PR review feedback on installer and catalog

- Use manifest.name (not crate_name) for installed filenames so
  discovery, auth, and CLI commands all agree on the stem (#1)
- Add AlreadyInstalled error variant instead of misleading
  ExtensionNotFound (#2)
- Add DownloadFailed error variant with URL context instead of
  stuffing URLs into PathBuf (#3)
- Validate HTTP status with error_for_status() before reading
  response bytes in artifact downloads (#4)
- Switch build_wasm_component to tokio::process::Command with
  status() so build output streams to the terminal (#6)
- Find WASM artifact by crate_name specifically instead of picking
  the first .wasm file in the release directory (#7)
- Add is_file() guard in catalog loader to skip directories (#8)
- Detect ambiguous bare-name lookups when both tools/<name> and
  channels/<name> exist, with get_strict() returning an error (#9)
- Fix wizard step_extensions to check tool.name for installed
  detection, consistent with the new naming (#11, #12)
- Fix redundant closures and map_or clippy warnings in changed files

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix(setup): restore DB connection fields after settings reload

reconnect_postgres() and reconnect_libsql() called Settings::from_db_map()
which overwrote database_url / libsql_path / libsql_url set from env vars.
Also use get_strict() in cmd_info to surface ambiguous bare-name errors.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* style: fix clippy collapsible_if and print_literal warnings

Collapse nested if-let chains and inline string literals in format
macros to satisfy CI clippy lint checks (deny warnings).

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix(registry): prefer artifacts for install-defaults and improve dir lookup

- InstallDefaults now defaults to downloading pre-built artifacts
  (matching `registry install` behavior), with --build flag for source builds.
- find_registry_dir() walks up 3 ancestor levels from the exe and adds
  a CARGO_MANIFEST_DIR fallback, matching load_registry_catalog() logic.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

---------

Co-authored-by: Claude Opus 4.6 <[email protected]>
This commit is contained in:
Illia Polosukhin
2026-02-20 01:17:44 +00:00
committed by GitHub
co-authored by Claude Opus 4.6
parent dae26d640e
commit 97a7637f30
42 changed files with 2671 additions and 29 deletions
+42 -8
View File
@@ -50,7 +50,7 @@ The `--no-onboard` CLI flag suppresses auto-detection.
---
## The 7-Step Wizard
## The 8-Step Wizard
### Overview
@@ -61,7 +61,8 @@ Step 3: Inference Provider ← skipped if --skip-auth
Step 4: Model Selection
Step 5: Embeddings
Step 6: Channel Configuration
Step 7: Background Tasks (heartbeat)
Step 7: Extensions (tools)
Step 8: Background Tasks (heartbeat)
save_and_summarize()
```
@@ -243,13 +244,20 @@ key first, then falls back to the standard env var.
```
6a. Tunnel setup (if webhook channels needed)
6b. Discover WASM channels from ~/.ironclaw/channels/
6c. Multi-select: CLI/TUI, HTTP, discovered channels, bundled channels
6d. Install missing bundled channels (copy WASM binaries)
6e. Initialize SecretsContext (for token storage)
6f. Setup HTTP webhook (if selected)
6g. Setup each WASM channel (secrets, owner binding)
6c. Build channel options: discovered + bundled + registry catalog
6d. Multi-select: CLI/TUI, HTTP, all available channels
6e. Install missing bundled channels (copy WASM binaries)
6f. Install missing registry channels (build from source)
6g. Initialize SecretsContext (for token storage)
6h. Setup HTTP webhook (if selected)
6i. Setup each WASM channel (secrets, owner binding)
```
**Channel sources** (priority order for installation):
1. Already installed in `~/.ironclaw/channels/`
2. Bundled channels (pre-compiled in `channels-src/`)
3. Registry channels (`registry/channels/*.json`, built from source)
**Tunnel setup** (`setup_tunnel`):
- Options: ngrok, Cloudflare Tunnel, localtunnel, custom URL
- Validates HTTPS requirement
@@ -273,7 +281,33 @@ key first, then falls back to the standard env var.
---
### Step 7: Heartbeat
### Step 7: Extensions (Tools)
**Module:** `wizard.rs``step_extensions()`
**Goal:** Install WASM tools from the extension registry.
**Flow:**
1. Load `RegistryCatalog` from `registry/` directory
2. If registry not found, print info and skip
3. List all tool manifests from the catalog
4. Discover already-installed tools in `~/.ironclaw/tools/`
5. Multi-select: show all registry tools with display name, auth method,
and description. Pre-check tools tagged `"default"` and already installed.
6. For each selected tool not yet installed, build from source via
`RegistryInstaller::install_from_source()`
7. Print consolidated auth hints (deduplicated by provider, e.g. one hint
for all Google tools sharing `google_oauth_token`)
**Registry lookup** (`load_registry_catalog`):
Searches for `registry/` directory in order:
1. Current working directory
2. Next to the executable
3. `CARGO_MANIFEST_DIR` (compile-time, dev builds)
---
### Step 8: Heartbeat
**Module:** `wizard.rs``step_heartbeat()`
+46 -4
View File
@@ -363,12 +363,54 @@ pub fn setup_tunnel(settings: &Settings) -> Result<TunnelSettings, ChannelSetupE
// Show existing config
let has_existing = settings.tunnel.public_url.is_some() || settings.tunnel.provider.is_some();
if has_existing {
if let Some(ref url) = settings.tunnel.public_url {
print_info(&format!("Existing static tunnel URL: {}", url));
println!();
print_info("Current tunnel configuration:");
let t = &settings.tunnel;
match t.provider.as_deref() {
Some("ngrok") => {
print_info(" Provider: ngrok");
if let Some(ref domain) = t.ngrok_domain {
print_info(&format!(" Domain: {}", domain));
}
if t.ngrok_token.is_some() {
print_info(" Auth: token configured");
}
}
Some("cloudflare") => {
print_info(" Provider: Cloudflare Tunnel");
if t.cf_token.is_some() {
print_info(" Auth: token configured");
}
}
Some("tailscale") => {
let mode = if t.ts_funnel {
"Funnel (public)"
} else {
"Serve (tailnet-only)"
};
print_info(&format!(" Provider: Tailscale {}", mode));
if let Some(ref hostname) = t.ts_hostname {
print_info(&format!(" Hostname: {}", hostname));
}
}
Some("custom") => {
print_info(" Provider: Custom command");
if let Some(ref cmd) = t.custom_command {
print_info(&format!(" Command: {}", cmd));
}
if let Some(ref url) = t.custom_health_url {
print_info(&format!(" Health: {}", url));
}
}
Some(other) => {
print_info(&format!(" Provider: {}", other));
}
None => {}
}
if let Some(ref provider) = settings.tunnel.provider {
print_info(&format!("Existing managed provider: {}", provider));
if let Some(ref url) = t.public_url {
print_info(&format!(" URL: {}", url));
}
println!();
if !confirm("Change tunnel configuration?", false)? {
return Ok(settings.tunnel.clone());
}
+2 -1
View File
@@ -7,7 +7,8 @@
//! 4. Model selection
//! 5. Embeddings
//! 6. Channel configuration (HTTP, Telegram, etc.)
//! 7. Heartbeat (background tasks)
//! 7. Extensions (tool installation from registry)
//! 8. Heartbeat (background tasks)
//!
//! # Example
//!
+423 -16
View File
@@ -7,7 +7,8 @@
//! 4. Model selection
//! 5. Embeddings
//! 6. Channel configuration
//! 7. Heartbeat (background tasks)
//! 7. Extensions (tool installation from registry)
//! 8. Heartbeat (background tasks)
use std::collections::{HashMap, HashSet};
use std::sync::Arc;
@@ -128,11 +129,13 @@ impl SetupWizard {
print_header("IronClaw Setup Wizard");
if self.config.channels_only {
// Channels-only mode: just step 6
// Channels-only mode: reconnect to existing DB and load settings
// before running the channel step, so secrets and save work.
self.reconnect_existing_db().await?;
print_step(1, 1, "Channel Configuration");
self.step_channels().await?;
} else {
let total_steps = 7;
let total_steps = 8;
// Step 1: Database
print_step(1, total_steps, "Database Connection");
@@ -162,8 +165,12 @@ impl SetupWizard {
print_step(6, total_steps, "Channel Configuration");
self.step_channels().await?;
// Step 7: Heartbeat
print_step(7, total_steps, "Background Tasks");
// Step 7: Extensions (tools)
print_step(7, total_steps, "Extensions");
self.step_extensions().await?;
// Step 8: Heartbeat
print_step(8, total_steps, "Background Tasks");
self.step_heartbeat()?;
}
@@ -173,6 +180,99 @@ impl SetupWizard {
Ok(())
}
/// Reconnect to the existing database and load settings.
///
/// Used by channels-only mode (and future single-step modes) so that
/// `init_secrets_context()` and `save_and_summarize()` have a live
/// database connection and the wizard's `self.settings` reflects the
/// previously saved configuration.
async fn reconnect_existing_db(&mut self) -> Result<(), SetupError> {
// Determine backend from env (set by bootstrap .env loaded in main).
let backend = std::env::var("DATABASE_BACKEND").unwrap_or_else(|_| "postgres".to_string());
// Try libsql first if that's the configured backend.
#[cfg(feature = "libsql")]
if backend == "libsql" || backend == "turso" || backend == "sqlite" {
return self.reconnect_libsql().await;
}
// Try postgres (either explicitly configured or as default).
#[cfg(feature = "postgres")]
{
let _ = &backend;
return self.reconnect_postgres().await;
}
#[allow(unreachable_code)]
Err(SetupError::Database(
"No database configured. Run full setup first (ironclaw onboard).".to_string(),
))
}
/// Reconnect to an existing PostgreSQL database and load settings.
#[cfg(feature = "postgres")]
async fn reconnect_postgres(&mut self) -> Result<(), SetupError> {
let url = std::env::var("DATABASE_URL").map_err(|_| {
SetupError::Database(
"DATABASE_URL not set. Run full setup first (ironclaw onboard).".to_string(),
)
})?;
self.test_database_connection_postgres(&url).await?;
self.settings.database_backend = Some("postgres".to_string());
self.settings.database_url = Some(url.clone());
// Load existing settings from DB, then restore connection fields that
// may not be persisted in the settings map.
if let Some(ref pool) = self.db_pool {
let store = crate::history::Store::from_pool(pool.clone());
if let Ok(map) = store.get_all_settings("default").await {
self.settings = Settings::from_db_map(&map);
self.settings.database_backend = Some("postgres".to_string());
self.settings.database_url = Some(url);
}
}
Ok(())
}
/// Reconnect to an existing libSQL database and load settings.
#[cfg(feature = "libsql")]
async fn reconnect_libsql(&mut self) -> Result<(), SetupError> {
let path = std::env::var("LIBSQL_PATH").unwrap_or_else(|_| {
crate::config::default_libsql_path()
.to_string_lossy()
.to_string()
});
let turso_url = std::env::var("LIBSQL_URL").ok();
let turso_token = std::env::var("LIBSQL_AUTH_TOKEN").ok();
self.test_database_connection_libsql(&path, turso_url.as_deref(), turso_token.as_deref())
.await?;
self.settings.database_backend = Some("libsql".to_string());
self.settings.libsql_path = Some(path.clone());
if let Some(ref url) = turso_url {
self.settings.libsql_url = Some(url.clone());
}
// Load existing settings from DB, then restore connection fields that
// may not be persisted in the settings map.
if let Some(ref db) = self.db_backend {
use crate::db::SettingsStore as _;
if let Ok(map) = db.get_all_settings("default").await {
self.settings = Settings::from_db_map(&map);
self.settings.database_backend = Some("libsql".to_string());
self.settings.libsql_path = Some(path);
if let Some(url) = turso_url {
self.settings.libsql_url = Some(url);
}
}
}
Ok(())
}
/// Step 1: Database connection.
async fn step_database(&mut self) -> Result<(), SetupError> {
// When both features are compiled, let the user choose.
@@ -1284,7 +1384,9 @@ impl SetupWizard {
.iter()
.map(|(name, _)| name.clone())
.collect();
let wasm_channel_names = wasm_channel_option_names(&discovered_channels);
// Build channel list from registry (if available) + bundled + discovered
let wasm_channel_names = build_channel_options(&discovered_channels);
// Build options list dynamically
let mut options: Vec<(String, bool)> = vec![
@@ -1295,11 +1397,15 @@ impl SetupWizard {
),
];
// Add available WASM channels (installed + bundled)
// Add available WASM channels (installed + bundled + registry)
for name in &wasm_channel_names {
let is_enabled = self.settings.channels.wasm_channels.contains(name);
let display_name = format!("{} (WASM)", capitalize_first(name));
options.push((display_name, is_enabled));
let label = if installed_names.contains(name) {
format!("{} (installed)", capitalize_first(name))
} else {
format!("{} (will install)", capitalize_first(name))
};
options.push((label, is_enabled));
}
let options_refs: Vec<(&str, bool)> =
@@ -1320,6 +1426,10 @@ impl SetupWizard {
})
.collect();
// Install selected channels that aren't already on disk
let mut any_installed = false;
// Try bundled channels first (pre-compiled artifacts from channels-src/)
if let Some(installed) = install_selected_bundled_channels(
&channels_dir,
&selected_wasm_channels,
@@ -1328,7 +1438,31 @@ impl SetupWizard {
.await?
&& !installed.is_empty()
{
print_success(&format!("Installed channels: {}", installed.join(", ")));
print_success(&format!(
"Installed bundled channels: {}",
installed.join(", ")
));
any_installed = true;
}
// Then try registry channels (build from source for any still missing)
let installed_from_registry = install_selected_registry_channels(
&channels_dir,
&selected_wasm_channels,
&installed_names,
)
.await;
if !installed_from_registry.is_empty() {
print_success(&format!(
"Built from registry: {}",
installed_from_registry.join(", ")
));
any_installed = true;
}
// Re-discover after installs
if any_installed {
discovered_channels = discover_wasm_channels(&channels_dir).await;
}
@@ -1419,7 +1553,134 @@ impl SetupWizard {
Ok(())
}
/// Step 7: Heartbeat configuration.
/// Step 7: Extensions (tools) installation from registry.
async fn step_extensions(&mut self) -> Result<(), SetupError> {
let catalog = match load_registry_catalog() {
Some(c) => c,
None => {
print_info("Extension registry not found. Skipping tool installation.");
print_info("Install tools manually with: ironclaw tool install <path>");
return Ok(());
}
};
let tools: Vec<_> = catalog
.list(Some(crate::registry::manifest::ManifestKind::Tool), None)
.into_iter()
.cloned()
.collect();
if tools.is_empty() {
print_info("No tools found in registry.");
return Ok(());
}
print_info("Available tools from the extension registry:");
print_info("Select which tools to install. You can install more later with:");
print_info(" ironclaw registry install <name>");
println!();
// Check which tools are already installed
let tools_dir = dirs::home_dir()
.ok_or_else(|| SetupError::Config("Could not determine home directory".into()))?
.join(".ironclaw/tools");
let installed_tools = discover_installed_tools(&tools_dir).await;
// Build options: show display_name + description, pre-check "default" tagged + already installed
let mut options: Vec<(String, bool)> = Vec::new();
for tool in &tools {
let is_installed = installed_tools.contains(&tool.name);
let is_default = tool.tags.contains(&"default".to_string());
let status = if is_installed { " (installed)" } else { "" };
let auth_hint = tool
.auth_summary
.as_ref()
.and_then(|a| a.method.as_deref())
.map(|m| format!(" [{}]", m))
.unwrap_or_default();
let label = format!(
"{}{}{} - {}",
tool.display_name, auth_hint, status, tool.description
);
options.push((label, is_default || is_installed));
}
let options_refs: Vec<(&str, bool)> =
options.iter().map(|(s, b)| (s.as_str(), *b)).collect();
let selected = select_many("Which tools do you want to install?", &options_refs)
.map_err(SetupError::Io)?;
if selected.is_empty() {
print_info("No tools selected.");
return Ok(());
}
// Install selected tools that aren't already on disk
let repo_root = catalog.root().parent().unwrap_or(catalog.root());
let installer = crate::registry::installer::RegistryInstaller::new(
repo_root.to_path_buf(),
tools_dir.clone(),
dirs::home_dir()
.unwrap_or_default()
.join(".ironclaw/channels"),
);
let mut installed_count = 0;
let mut auth_needed: Vec<String> = Vec::new();
for idx in &selected {
let tool = &tools[*idx];
if installed_tools.contains(&tool.name) {
continue; // Already installed, skip
}
match installer.install_from_source(tool, false).await {
Ok(outcome) => {
print_success(&format!("Installed {}", outcome.name));
installed_count += 1;
// Track auth needs
if let Some(auth) = &tool.auth_summary
&& auth.method.as_deref() != Some("none")
&& auth.method.is_some()
{
let provider = auth.provider.as_deref().unwrap_or(&tool.name);
// Only mention unique providers (Google tools share auth)
let hint = format!(" {} - ironclaw tool auth {}", provider, tool.name);
if !auth_needed
.iter()
.any(|h| h.starts_with(&format!(" {} -", provider)))
{
auth_needed.push(hint);
}
}
}
Err(e) => {
print_error(&format!("Failed to install {}: {}", tool.display_name, e));
}
}
}
if installed_count > 0 {
println!();
print_success(&format!("{} tool(s) installed.", installed_count));
}
if !auth_needed.is_empty() {
println!();
print_info("Some tools need authentication. Run after setup:");
for hint in &auth_needed {
print_info(hint);
}
}
Ok(())
}
/// Step 8: Heartbeat configuration.
fn step_heartbeat(&mut self) -> Result<(), SetupError> {
print_info("Heartbeat runs periodic background tasks (e.g., checking your calendar,");
print_info("monitoring for notifications, running scheduled workflows).");
@@ -2087,15 +2348,161 @@ async fn install_missing_bundled_channels(
Ok(installed)
}
fn wasm_channel_option_names(discovered: &[(String, ChannelCapabilitiesFile)]) -> Vec<String> {
/// Build channel options from discovered channels + bundled + registry catalog.
///
/// Returns a deduplicated, sorted list of channel names available for selection.
fn build_channel_options(discovered: &[(String, ChannelCapabilitiesFile)]) -> Vec<String> {
let mut names: Vec<String> = discovered.iter().map(|(name, _)| name.clone()).collect();
// Add bundled channels
for bundled in available_channel_names().iter().copied() {
if !names.iter().any(|name| name == bundled) {
names.push(bundled.to_string());
}
}
// Add registry channels
if let Some(catalog) = load_registry_catalog() {
for manifest in catalog.list(Some(crate::registry::manifest::ManifestKind::Channel), None) {
if !names.iter().any(|n| n == &manifest.name) {
names.push(manifest.name.clone());
}
}
}
names.sort();
names
}
/// Try to load the registry catalog. Returns None if the registry directory
/// cannot be found (e.g. running from an installed binary without the repo).
fn load_registry_catalog() -> Option<crate::registry::catalog::RegistryCatalog> {
// Try relative to current directory (dev usage)
let cwd = std::env::current_dir().ok()?;
let candidate = cwd.join("registry");
if candidate.is_dir() {
return crate::registry::catalog::RegistryCatalog::load(&candidate).ok();
}
// Try relative to executable
if let Ok(exe) = std::env::current_exe()
&& let Some(parent) = exe.parent()
{
let candidate = parent.join("registry");
if candidate.is_dir() {
return crate::registry::catalog::RegistryCatalog::load(&candidate).ok();
}
if let Some(grandparent) = parent.parent() {
let candidate = grandparent.join("registry");
if candidate.is_dir() {
return crate::registry::catalog::RegistryCatalog::load(&candidate).ok();
}
}
}
// Try CARGO_MANIFEST_DIR (compile-time, works in dev builds)
let manifest_dir = std::path::Path::new(env!("CARGO_MANIFEST_DIR"));
let candidate = manifest_dir.join("registry");
if candidate.is_dir() {
return crate::registry::catalog::RegistryCatalog::load(&candidate).ok();
}
None
}
/// Install selected channels from the registry that aren't already on disk
/// and weren't handled by the bundled installer.
///
/// This builds channels from source using `cargo component build`.
async fn install_selected_registry_channels(
channels_dir: &std::path::Path,
selected_channels: &[String],
already_installed: &HashSet<String>,
) -> Vec<String> {
let catalog = match load_registry_catalog() {
Some(c) => c,
None => return Vec::new(),
};
let repo_root = catalog
.root()
.parent()
.unwrap_or(catalog.root())
.to_path_buf();
let bundled: HashSet<&str> = available_channel_names().iter().copied().collect();
let mut installed = Vec::new();
for name in selected_channels {
// Skip if already installed or handled by bundled installer
if already_installed.contains(name) || bundled.contains(name.as_str()) {
continue;
}
// Check if already on disk (may have been installed between bundled and here)
let wasm_on_disk = channels_dir.join(format!("{}.wasm", name)).exists()
|| channels_dir.join(format!("{}-channel.wasm", name)).exists();
if wasm_on_disk {
continue;
}
// Look up in registry
let manifest = match catalog.get(&format!("channels/{}", name)) {
Some(m) => m,
None => continue,
};
let installer = crate::registry::installer::RegistryInstaller::new(
repo_root.clone(),
dirs::home_dir().unwrap_or_default().join(".ironclaw/tools"),
channels_dir.to_path_buf(),
);
match installer.install_from_source(manifest, false).await {
Ok(_) => {
installed.push(name.clone());
}
Err(e) => {
tracing::warn!(
channel = %name,
error = %e,
"Failed to install channel from registry"
);
crate::setup::prompts::print_error(&format!(
"Failed to install channel '{}': {}",
name, e
));
}
}
}
installed
}
/// Discover which tools are already installed in the tools directory.
///
/// Returns a set of tool names (the stem of .wasm files).
async fn discover_installed_tools(tools_dir: &std::path::Path) -> HashSet<String> {
let mut names = HashSet::new();
if !tools_dir.is_dir() {
return names;
}
let mut entries = match tokio::fs::read_dir(tools_dir).await {
Ok(e) => e,
Err(_) => return names,
};
while let Ok(Some(entry)) = entries.next_entry().await {
let path = entry.path();
if path.extension().and_then(|e| e.to_str()) == Some("wasm")
&& let Some(stem) = path.file_stem().and_then(|s| s.to_str())
{
names.insert(stem.to_string());
}
}
names
}
@@ -2209,9 +2616,9 @@ mod tests {
}
#[test]
fn test_wasm_channel_option_names_includes_available_when_missing() {
fn test_build_channel_options_includes_available_when_missing() {
let discovered = Vec::new();
let options = wasm_channel_option_names(&discovered);
let options = build_channel_options(&discovered);
let available = available_channel_names();
// All available (built) channels should appear
for name in &available {
@@ -2224,9 +2631,9 @@ mod tests {
}
#[test]
fn test_wasm_channel_option_names_dedupes_available() {
fn test_build_channel_options_dedupes_available() {
let discovered = vec![(String::from("telegram"), ChannelCapabilitiesFile::default())];
let options = wasm_channel_option_names(&discovered);
let options = build_channel_options(&discovered);
// telegram should appear exactly once despite being both discovered and available
assert_eq!(
options.iter().filter(|n| *n == "telegram").count(),