Merge remote-tracking branch 'origin/main' into feat/gemini-cli-oauth

This commit is contained in:
Artem
2026-03-17 18:21:02 +03:00
470 changed files with 64449 additions and 10013 deletions
+68 -26
View File
@@ -10,7 +10,7 @@ file first, then adjust the code to match.
## Entry Points
```
ironclaw onboard [--skip-auth] [--channels-only]
ironclaw onboard [--skip-auth] [--channels-only] [--provider-only] [--quick]
```
Explicit invocation. Loads `.env` files, runs the wizard, exits.
@@ -26,6 +26,8 @@ the wizard). Otherwise triggers when no database is configured:
- `LIBSQL_PATH` env var is set
- `~/.ironclaw/ironclaw.db` exists on disk
Auto-triggered onboarding uses **quick mode** by default.
The `--no-onboard` CLI flag suppresses auto-detection.
---
@@ -50,7 +52,41 @@ The `--no-onboard` CLI flag suppresses auto-detection.
---
## The 8-Step Wizard
## Quick Mode
Quick mode (`--quick` flag, or auto-triggered on first run) provides a
near-instant onboarding experience by auto-defaulting everything except
the LLM provider and model selection.
```
auto_setup_database() → libsql at ~/.ironclaw/ironclaw.db (zero prompts)
auto_setup_security() → keychain or env var (zero prompts)
Step 1/2: Inference Provider ← only interactive step
Step 2/2: Model Selection ← only interactive step
save_and_summarize() → includes tip to run `ironclaw onboard`
```
**`auto_setup_database()`:** Uses existing env vars if set (`DATABASE_URL`
for postgres, `LIBSQL_PATH` for libsql) without prompting. Otherwise
defaults to libsql at `~/.ironclaw/ironclaw.db`, creates the database,
and runs migrations silently. Falls back to interactive mode only when
just the postgres feature is compiled and no `DATABASE_URL` is set.
**`auto_setup_security()`:** Checks for existing `SECRETS_MASTER_KEY`
env var or OS keychain key. If neither exists, generates a new key and
stores it in the keychain (macOS) or env var (Linux/other). Zero prompts
except unavoidable macOS keychain dialogs.
**`.env` preservation (fix for #751):** `write_bootstrap_env()` now uses
`upsert_bootstrap_vars()` instead of `save_bootstrap_env()`, preserving
user-added variables like `HTTP_HOST` across re-onboarding.
The full 9-step wizard remains available via `ironclaw onboard`.
---
## The 9-Step Wizard
### Overview
@@ -62,7 +98,8 @@ Step 4: Model Selection
Step 5: Embeddings
Step 6: Channel Configuration
Step 7: Extensions (tools)
Step 8: Background Tasks (heartbeat)
Step 8: Docker Sandbox
Step 9: Background Tasks (heartbeat)
save_and_summarize()
```
@@ -77,6 +114,13 @@ Step 8: Background Tasks (heartbeat)
**Goal:** Select backend, establish connection, run migrations.
**Init delegation:** Backend-specific connection logic lives in `src/db/mod.rs`
(`connect_without_migrations()`), not in the wizard. The wizard calls
`test_database_connection()` which delegates to the db module factory. Feature-flag
branching (`#[cfg(feature = ...)]`) is confined to `src/db/mod.rs`. PostgreSQL
validation (version >= 15, pgvector) is handled by `validate_postgres()` in
`src/db/mod.rs`.
**Decision tree:**
```
@@ -84,26 +128,23 @@ Both features compiled?
├─ Yes → DATABASE_BACKEND env var set?
│ ├─ Yes → use that backend
│ └─ No → interactive selection (PostgreSQL vs libSQL)
├─ Only postgres feature → step_database_postgres()
└─ Only libsql feature → step_database_libsql()
├─ Only postgres feature → prompt for DATABASE_URL, test connection
└─ Only libsql feature → prompt for path, test connection
```
**PostgreSQL path** (`step_database_postgres`):
**PostgreSQL path:**
1. Check `DATABASE_URL` from env or settings
2. Test connection (creates `deadpool_postgres::Pool`)
3. Optionally run refinery migrations
4. Store pool in `self.db_pool`
2. Test connection via `connect_without_migrations()` (validates version, pgvector)
3. Optionally run migrations
**libSQL path** (`step_database_libsql`):
**libSQL path:**
1. Offer local path (default: `~/.ironclaw/ironclaw.db`)
2. Optional Turso cloud sync (URL + auth token)
3. Test connection (creates `LibSqlBackend`)
3. Test connection via `connect_without_migrations()`
4. Always run migrations (idempotent CREATE IF NOT EXISTS)
5. Store backend in `self.db_backend`
**Invariant:** After Step 1, exactly one of `self.db_pool` or
`self.db_backend` is `Some`. This is required for settings persistence
in `save_and_summarize()`.
**Invariant:** After Step 1, `self.db` is `Some(Arc<dyn Database>)`.
This is required for settings persistence in `save_and_summarize()`.
---
@@ -172,25 +213,26 @@ env-var mode or skipped secrets.
| Anthropic | API key | `anthropic_api_key` | `ANTHROPIC_API_KEY` |
| OpenAI | API key | `openai_api_key` | `OPENAI_API_KEY` |
| Ollama | None | - | - |
| OpenRouter¹ | API key | `llm_compatible_api_key` | `LLM_API_KEY` |
| OpenAI-compatible¹ | Optional API key | `llm_compatible_api_key` | `LLM_API_KEY` |
| OpenRouter | API key | `llm_openrouter_api_key` | `OPENROUTER_API_KEY` |
| OpenAI-compatible | Optional API key | `llm_compatible_api_key` | `LLM_API_KEY` |
| AWS Bedrock | AWS credentials (IAM, SSO, instance roles) | - | - |
¹ OpenRouter and OpenAI-compatible share the same secret name and env var because
OpenRouter is stored as `llm_backend = "openai_compatible"` under the hood.
Switching between them overwrites the same credential slot.
**OpenRouter** is a standalone registry provider (`providers.json` id `"openrouter"`)
with its own secret name and env var. It is **not** stored as `openai_compatible`.
**OpenRouter** (`setup_openrouter`):
- Pre-configured OpenAI-compatible preset with base URL `https://openrouter.ai/api/v1`
- Delegates to `setup_api_key_provider()` with a display name override ("OpenRouter")
- Sets `llm_backend = "openai_compatible"` and `openai_compatible_base_url` automatically
- Clears `selected_model` so Step 4 prompts for a model name (manual text input, no API-based model fetching)
**OpenRouter** (`setup.kind = "api_key"` in `providers.json`):
- Standalone provider with base URL `https://openrouter.ai/api/v1`
- Delegates to `setup_api_key_provider()` with display name "OpenRouter"
- API key is required (`api_key_required: true`)
- Default model: `openai/gpt-4o`
**API-key providers** (`setup_api_key_provider`):
1. Check env var → if set, ask to reuse, persist to secrets store
2. Otherwise prompt for key entry via `secret_input()`
3. Store encrypted in secrets via `init_secrets_context()`
4. **Cache key in `self.llm_api_key`** for model fetching in Step 4
5. Preserve `selected_model` on a same-backend re-run; clear it only when
switching to a different backend
**NEAR AI** (`setup_nearai`):
- Calls `session_manager.ensure_authenticated()` which shows the auth menu:
@@ -300,7 +342,7 @@ key first, then falls back to the standard env var.
1. Check `self.secrets_crypto` (set in Step 2) → use if available
2. Else try `SECRETS_MASTER_KEY` env var
3. Else try `get_master_key()` from keychain (only in `channels_only` mode)
4. Create backend-appropriate secrets store (respects selected database backend)
4. Create secrets store using `self.db` (`Arc<dyn Database>`)
---
+379 -7
View File
@@ -804,13 +804,15 @@ pub async fn setup_wasm_channel(
print_success(&format!("{} saved to database", secret_config.name));
}
// TODO: Substitute secrets into the validation URL and make a
// GET request to verify the configured credentials actually work.
if let Some(ref validation_endpoint) = setup.validation_endpoint {
print_info(&format!(
"Validation endpoint configured: {} (validation not yet implemented)",
validation_endpoint
));
print_info("Validating configured credentials...");
match validate_channel_credentials(secrets, validation_endpoint).await {
Ok(()) => print_success("Credentials validated successfully"),
Err(e) => print_warning(&format!(
"Credential validation failed: {}. Setup will continue, but the channel may fail to start until the credentials are fixed.",
e
)),
}
}
print_success(&format!("{} channel configured", channel_name));
@@ -821,6 +823,225 @@ pub async fn setup_wasm_channel(
})
}
async fn validate_channel_credentials(
secrets: &SecretsContext,
validation_endpoint: &str,
) -> Result<(), ChannelSetupError> {
let validation_url = substitute_validation_placeholders(secrets, validation_endpoint).await?;
let (parsed, resolved_addrs) = validate_public_https_url(&validation_url).await?;
let target = validation_target_display(&parsed);
let mut client_builder = reqwest::Client::builder()
.timeout(std::time::Duration::from_secs(5))
.redirect(reqwest::redirect::Policy::none());
if matches!(parsed.host(), Some(url::Host::Domain(_)))
&& let Some(host) = parsed.host_str()
{
client_builder = client_builder.resolve_to_addrs(host, &resolved_addrs);
}
let client = client_builder
.build()
.map_err(|e| ChannelSetupError::Network(format!("Failed to build HTTP client: {}", e)))?;
let response = client.get(parsed.clone()).send().await.map_err(|e| {
ChannelSetupError::Network(format!(
"Validation request to {} failed: {}",
target,
describe_validation_request_error(&e)
))
})?;
if response.status().is_success() {
Ok(())
} else {
Err(ChannelSetupError::Validation(format!(
"Validation endpoint returned HTTP {} from {}",
response.status(),
target
)))
}
}
async fn substitute_validation_placeholders(
secrets: &SecretsContext,
validation_endpoint: &str,
) -> Result<String, ChannelSetupError> {
let mut resolved = validation_endpoint.to_string();
let placeholder_names: std::collections::BTreeSet<String> = validation_placeholder_regex()
.captures_iter(validation_endpoint)
.filter_map(|caps| caps.get(1).map(|m| m.as_str().to_string()))
.collect();
for secret_name in placeholder_names {
let secret_value = secrets.get_secret(&secret_name).await?;
let placeholder = format!("{{{}}}", secret_name);
let encoded_value = urlencoding::encode(secret_value.expose_secret());
resolved = resolved.replace(&placeholder, encoded_value.as_ref());
}
Ok(resolved)
}
async fn validate_public_https_url(
url: &str,
) -> Result<(Url, Vec<std::net::SocketAddr>), ChannelSetupError> {
use std::net::{IpAddr, SocketAddr};
let parsed = Url::parse(url)
.map_err(|e| ChannelSetupError::Validation(format!("Invalid URL: {}", e)))?;
if parsed.scheme() != "https" {
return Err(ChannelSetupError::Validation(
"Validation endpoint must use https".to_string(),
));
}
if !parsed.username().is_empty() || parsed.password().is_some() {
return Err(ChannelSetupError::Validation(
"Validation endpoint cannot contain userinfo".to_string(),
));
}
let host = parsed
.host_str()
.ok_or_else(|| ChannelSetupError::Validation("Validation URL missing host".to_string()))?;
let normalized_host = normalize_validation_domain(host);
let host_lower = normalized_host.to_ascii_lowercase();
if host_lower == "localhost" || host_lower.ends_with(".localhost") {
return Err(ChannelSetupError::Validation(
"Validation endpoint cannot target localhost".to_string(),
));
}
let port = parsed.port_or_known_default().unwrap_or(443);
match parsed
.host()
.ok_or_else(|| ChannelSetupError::Validation("Validation URL missing host".to_string()))?
{
url::Host::Ipv4(v4) => {
let ip = IpAddr::V4(v4);
if is_disallowed_ip(&ip) {
return Err(ChannelSetupError::Validation(format!(
"Validation endpoint cannot target private or local IP {}",
ip
)));
}
Ok((parsed, vec![SocketAddr::new(ip, port)]))
}
url::Host::Ipv6(v6) => {
let ip = normalize_ip(IpAddr::V6(v6));
if is_disallowed_ip(&ip) {
return Err(ChannelSetupError::Validation(format!(
"Validation endpoint cannot target private or local IP {}",
ip
)));
}
Ok((parsed, vec![SocketAddr::new(ip, port)]))
}
url::Host::Domain(domain) => {
let addrs: Vec<SocketAddr> = tokio::net::lookup_host((normalized_host, port))
.await
.map_err(|e| {
ChannelSetupError::Validation(format!(
"DNS resolution failed for {}: {}",
normalized_host, e
))
})?
.map(|addr| SocketAddr::new(normalize_ip(addr.ip()), addr.port()))
.collect();
if addrs.is_empty() {
return Err(ChannelSetupError::Validation(format!(
"Validation hostname '{}' did not resolve to any IP addresses",
domain
)));
}
for addr in &addrs {
if is_disallowed_ip(&addr.ip()) {
return Err(ChannelSetupError::Validation(format!(
"Validation hostname '{}' resolves to disallowed IP {}",
domain,
addr.ip()
)));
}
}
Ok((parsed, addrs))
}
}
}
fn is_disallowed_ip(ip: &std::net::IpAddr) -> bool {
match normalize_ip(*ip) {
std::net::IpAddr::V4(v4) => {
v4.is_private()
|| v4.is_loopback()
|| v4.is_link_local()
|| v4.is_multicast()
|| v4.is_unspecified()
|| v4 == std::net::Ipv4Addr::new(169, 254, 169, 254)
|| (v4.octets()[0] == 100 && (v4.octets()[1] & 0xC0) == 64)
}
std::net::IpAddr::V6(v6) => {
v6.is_loopback()
|| v6.is_unique_local()
|| v6.is_unicast_link_local()
|| v6.is_multicast()
|| v6.is_unspecified()
}
}
}
fn normalize_ip(ip: std::net::IpAddr) -> std::net::IpAddr {
match ip {
std::net::IpAddr::V6(v6) => v6
.to_ipv4_mapped()
.map(std::net::IpAddr::V4)
.unwrap_or(std::net::IpAddr::V6(v6)),
other => other,
}
}
fn normalize_validation_domain(host: &str) -> &str {
host.trim_end_matches('.')
}
fn validation_placeholder_regex() -> &'static regex::Regex {
static PLACEHOLDER_RE: std::sync::OnceLock<regex::Regex> = std::sync::OnceLock::new();
PLACEHOLDER_RE.get_or_init(|| {
regex::Regex::new(r"\{([A-Za-z0-9_]+)\}")
.expect("validation placeholder regex must compile") // safety: hardcoded literal
})
}
fn validation_target_display(parsed: &Url) -> String {
let host = parsed.host_str().unwrap_or("unknown host");
match parsed.port() {
Some(port) => format!("{}:{}", host, port),
None => host.to_string(),
}
}
fn describe_validation_request_error(error: &reqwest::Error) -> &'static str {
if error.is_timeout() {
"request timed out"
} else if error.is_redirect() {
"redirects are not allowed"
} else if error.is_connect() {
"connection failed"
} else if error.is_request() {
"request could not be sent"
} else {
"request failed"
}
}
/// Validate a Cloudflare tunnel token by briefly running `cloudflared`.
///
/// Spawns `cloudflared tunnel run` with a dummy local URL and watches stderr
@@ -911,8 +1132,26 @@ fn generate_secret_with_length(length: usize) -> String {
#[cfg(test)]
mod tests {
use base64::Engine;
use std::sync::Arc;
use crate::setup::channels::{generate_webhook_secret, validate_cloudflare_token_format};
use crate::secrets::{InMemorySecretsStore, SecretsCrypto, SecretsStore};
use crate::setup::channels::{
SecretsContext, generate_webhook_secret, substitute_validation_placeholders,
validate_cloudflare_token_format, validate_public_https_url,
};
fn test_secrets_context() -> SecretsContext {
use secrecy::SecretString;
let crypto = Arc::new(
SecretsCrypto::new(SecretString::from(
"0123456789abcdef0123456789abcdef".to_string(),
))
.unwrap(),
);
let store: Arc<dyn SecretsStore> = Arc::new(InMemorySecretsStore::new(crypto));
SecretsContext::from_store(store, "test-user")
}
#[test]
fn test_generate_webhook_secret() {
@@ -965,4 +1204,137 @@ mod tests {
fn test_validate_cloudflare_token_empty() {
assert!(!validate_cloudflare_token_format(""));
}
#[tokio::test]
async fn test_substitute_validation_placeholders() {
let secrets = test_secrets_context();
secrets
.save_secret(
"telegram_bot_token",
&secrecy::SecretString::from("abc123".to_string()),
)
.await
.unwrap();
secrets
.save_secret(
"workspace_id",
&secrecy::SecretString::from("ws_456".to_string()),
)
.await
.unwrap();
let resolved = substitute_validation_placeholders(
&secrets,
"https://api.example.com/{workspace_id}/verify?token={telegram_bot_token}",
)
.await
.unwrap();
assert_eq!(
resolved,
"https://api.example.com/ws_456/verify?token=abc123"
);
}
#[tokio::test]
async fn test_substitute_validation_placeholders_url_encodes_secrets() {
let secrets = test_secrets_context();
secrets
.save_secret(
"telegram_bot_token",
&secrecy::SecretString::from("abc123?foo=1&bar=#baz/slash".to_string()),
)
.await
.unwrap();
let resolved = substitute_validation_placeholders(
&secrets,
"https://api.example.com/verify?token={telegram_bot_token}",
)
.await
.unwrap();
assert_eq!(
resolved,
"https://api.example.com/verify?token=abc123%3Ffoo%3D1%26bar%3D%23baz%2Fslash"
);
}
#[tokio::test]
async fn test_substitute_validation_placeholders_missing_secret() {
let secrets = test_secrets_context();
let err = substitute_validation_placeholders(
&secrets,
"https://api.example.com/verify?token={missing_secret}",
)
.await
.unwrap_err()
.to_string();
assert!(err.contains("Failed to read secret"));
}
#[tokio::test]
async fn test_validate_public_https_url_rejects_localhost() {
let err = validate_public_https_url("https://localhost/api")
.await
.unwrap_err()
.to_string();
assert!(err.contains("localhost"));
}
#[tokio::test]
async fn test_validate_public_https_url_rejects_localhost_with_trailing_dot() {
let err = validate_public_https_url("https://localhost./api")
.await
.unwrap_err()
.to_string();
assert!(err.contains("localhost"));
}
#[tokio::test]
async fn test_validate_public_https_url_rejects_private_ip() {
let err = validate_public_https_url("https://192.168.1.10/api")
.await
.unwrap_err()
.to_string();
assert!(err.contains("private or local IP"));
}
#[tokio::test]
async fn test_validate_public_https_url_rejects_ipv4_mapped_ipv6() {
let err = validate_public_https_url("https://[::ffff:127.0.0.1]/api")
.await
.unwrap_err()
.to_string();
assert!(err.contains("private or local IP"));
}
#[tokio::test]
async fn test_validate_public_https_url_rejects_http() {
let err = validate_public_https_url("http://example.com/api")
.await
.unwrap_err()
.to_string();
assert!(err.contains("must use https"));
}
#[tokio::test]
async fn test_validate_public_https_url_accepts_public_https_literal_ip() {
let (parsed, addrs) = validate_public_https_url("https://8.8.8.8/api")
.await
.unwrap();
assert_eq!(parsed.as_str(), "https://8.8.8.8/api");
assert_eq!(addrs.len(), 1);
assert_eq!(addrs[0].ip().to_string(), "8.8.8.8");
}
#[tokio::test]
async fn test_validate_public_https_url_fails_closed_on_dns_error() {
let err = validate_public_https_url("https://should-not-resolve.invalid/api")
.await
.unwrap_err()
.to_string();
assert!(err.contains("DNS resolution failed"));
}
}
+32
View File
@@ -31,3 +31,35 @@ pub use prompts::{
};
#[cfg(any(feature = "postgres", feature = "libsql"))]
pub use wizard::{SetupConfig, SetupWizard};
/// Check if onboarding is needed and return the reason.
///
/// Reads environment variables (`DATABASE_URL`, `LIBSQL_PATH`,
/// `ONBOARD_COMPLETED`, `NEARAI_API_KEY`) and checks for the default
/// session file on disk. Not safe to call concurrently with `env::set_var`.
#[cfg(any(feature = "postgres", feature = "libsql"))]
pub fn check_onboard_needed() -> Option<&'static str> {
let has_db = std::env::var("DATABASE_URL").is_ok()
|| std::env::var("LIBSQL_PATH").is_ok()
|| crate::config::default_libsql_path().exists();
if !has_db {
return Some("Database not configured");
}
if std::env::var("ONBOARD_COMPLETED")
.map(|v| v == "true")
.unwrap_or(false)
{
return None;
}
if std::env::var("NEARAI_API_KEY").is_err() {
let session_path = crate::config::default_session_path();
if !session_path.exists() {
return Some("First run");
}
}
None
}
+42 -4
View File
@@ -11,13 +11,25 @@ use std::io::{self, Write};
use crossterm::{
cursor,
event::{self, Event, KeyCode, KeyEvent, KeyModifiers},
event::{self, Event, KeyCode, KeyEvent, KeyEventKind, KeyModifiers},
execute,
style::{Color, Print, ResetColor, SetForegroundColor},
terminal::{self, ClearType},
};
use secrecy::SecretString;
/// Drain any residual key events already queued in the terminal buffer.
///
/// On Windows, transitioning between raw mode and cooked mode (or between
/// successive raw-mode prompts) can leave stale events (e.g. the Release
/// half of an Enter keypress) in the queue. Consuming them with a
/// non-blocking poll prevents the next prompt from mis-firing.
fn drain_pending_events() {
while event::poll(std::time::Duration::ZERO).unwrap_or(false) {
let _ = event::read();
}
}
/// Display a numbered menu and get user selection.
///
/// Returns the index (0-based) of the selected option.
@@ -94,6 +106,7 @@ pub fn select_many(prompt: &str, options: &[(&str, bool)]) -> io::Result<Vec<usi
let mut cursor_pos = 0;
terminal::enable_raw_mode()?;
drain_pending_events();
execute!(stdout, cursor::Hide)?;
let result = (|| {
@@ -124,9 +137,13 @@ pub fn select_many(prompt: &str, options: &[(&str, bool)]) -> io::Result<Vec<usi
stdout.flush()?;
// Read key
// Read key — only act on Press events to avoid double-firing
// from Release/Repeat events on Windows.
if let Event::Key(KeyEvent {
code, modifiers, ..
code,
modifiers,
kind: KeyEventKind::Press,
..
}) = event::read()?
{
match code {
@@ -200,9 +217,16 @@ fn read_secret_line() -> io::Result<SecretString> {
let mut input = String::new();
let mut stdout = io::stdout();
drain_pending_events();
loop {
// Only act on Press events to avoid double-firing from
// Release/Repeat events on Windows.
if let Event::Key(KeyEvent {
code, modifiers, ..
code,
modifiers,
kind: KeyEventKind::Press,
..
}) = event::read()?
{
match code {
@@ -260,6 +284,20 @@ pub fn confirm(prompt: &str, default: bool) -> io::Result<bool> {
})
}
/// Print the IronClaw ASCII art banner in blue.
pub fn print_banner() {
let mut stdout = io::stdout();
let _ = execute!(stdout, SetForegroundColor(Color::Cyan));
println!();
println!(r" ██╗██████╗ ██████╗ ███╗ ██╗ ██████╗██╗ █████╗ ██╗ ██╗");
println!(r" ██║██╔══██╗██╔═══██╗████╗ ██║██╔════╝██║ ██╔══██╗██║ ██║");
println!(r" ██║██████╔╝██║ ██║██╔██╗ ██║██║ ██║ ███████║██║ █╗ ██║");
println!(r" ██║██╔══██╗██║ ██║██║╚██╗██║██║ ██║ ██╔══██║██║███╗██║");
println!(r" ██║██║ ██║╚██████╔╝██║ ╚████║╚██████╗███████╗██║ ██║╚███╔███╔╝");
println!(r" ╚═╝╚═╝ ╚═╝ ╚═════╝ ╚═╝ ╚═══╝ ╚═════╝╚══════╝╚═╝ ╚═╝ ╚══╝╚══╝ ");
let _ = execute!(stdout, ResetColor);
}
/// Print a styled header box.
///
/// # Example
+438 -423
View File
File diff suppressed because it is too large Load Diff