feat: 10 infrastructure improvements from zeroclaw (#126)

* refactor: break up agent_loop.rs into four focused modules

Split the monolithic 2835-line agent_loop.rs into:
- agent_loop.rs (722L): Agent struct, event loop, message dispatch
- dispatcher.rs (635L): Agentic tool loop, tool execution, auth detection
- commands.rs (484L): System commands, job handlers, heartbeat, summarize
- thread_ops.rs (1059L): Thread lifecycle, approval, undo/redo, persistence

Each module gets its own impl Agent block. Agent fields changed to
pub(super) so sibling modules in the agent package can access them.
All 16 existing tests pass in their new locations.

Inspired by ZeroClaw's agent module split (agent.rs, loop_.rs,
dispatcher.rs, prompt.rs, memory_loader.rs).

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* feat: add cost caps and guardrails for autonomous agent spending

Daily budget (MAX_COST_PER_DAY_CENTS) and hourly action rate
(MAX_ACTIONS_PER_HOUR) limits prevent runaway agents from burning
through API credits, especially in daemon/heartbeat modes.

- CostGuard with pre-flight check and post-call recording
- Sliding window for hourly rate, midnight-UTC daily reset
- 80% threshold warning, atomic fast-path for exceeded budget
- Wired into dispatcher loop (check before LLM call, record after)

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* feat: add circuit breaker on LLM providers

Wraps LlmProvider with a Closed/Open/HalfOpen state machine that
trips after consecutive transient failures, preventing request storms
against a degraded backend. Automatically probes for recovery.

- CircuitBreakerProvider implements LlmProvider (drop-in wrapper)
- Transient error classification (server, rate-limit, network, auth infra)
- Client errors (wrong model, context overflow) don't trip the breaker
- Configurable via CIRCUIT_BREAKER_THRESHOLD and CIRCUIT_BREAKER_RECOVERY_SECS
- Composes with existing FailoverProvider (circuit breaker wraps failover)
- 12 tests covering full state machine and error classification

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* feat: add tunnel abstraction for remote access

Trait-based tunnel system with lifecycle management (start/stop/health)
for exposing the agent to the internet through external tunnel binaries.

Five providers:
- Cloudflare Tunnel (cloudflared, Zero Trust token auth)
- Tailscale (serve for tailnet, funnel for public)
- ngrok (with optional custom domain)
- Custom (arbitrary command with {host}/{port} placeholders)
- None (local-only, no external exposure)

Config via TUNNEL_PROVIDER + provider-specific env vars. Extends
existing TunnelConfig with optional managed provider alongside the
static TUNNEL_URL path. Factory, shared process management, and
37 tests covering all providers and edge cases.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* feat: add OS service management (launchd/systemd)

Adds `ironclaw service {install,start,stop,status,uninstall}` for
running the agent as a background daemon. macOS uses launchd plists
under ~/Library/LaunchAgents, Linux uses systemd user units.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* feat: add observability trait system with noop, log, and multi backends

Introduces an Observer trait for recording agent lifecycle events and
metrics, with pluggable backends. The noop backend compiles to zero
overhead, log backend uses tracing, and multi fans out to multiple
observers. Configured via OBSERVABILITY_BACKEND env var.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* feat: add in-memory LLM response cache with TTL and LRU eviction

CachedProvider wraps any LlmProvider and caches complete() responses
keyed by SHA-256(model + messages). Tool-calling requests are never
cached since they trigger side effects. Configurable via
RESPONSE_CACHE_ENABLED, RESPONSE_CACHE_TTL_SECS, and
RESPONSE_CACHE_MAX_ENTRIES env vars.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* feat: add memory hygiene with cadence-gated daily log cleanup

Adds workspace::hygiene module that automatically deletes daily log
documents older than a configurable retention period (default 30 days).
Runs on a 12-hour cadence tracked via a local state file to avoid
redundant passes. Best-effort design: failures are logged, never fatal.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* feat: add doctor diagnostics command for active health probing

Probes external dependencies (Docker, cloudflared, ngrok, tailscale),
validates NEAR AI session, checks database connectivity, and verifies
workspace directory. Complements the passive `status` command.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* feat: add structured TOML config file support

Adds ~/.ironclaw/config.toml as a configuration layer between env vars
and database settings. Priority: env var > TOML file > DB > defaults.

- `ironclaw config init` generates a commented config.toml from current settings
- `ironclaw --config path/to/config.toml` loads a custom config file
- Settings.merge_from() only overlays non-default values from the TOML file
- `ironclaw config path` now shows TOML file status

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: address codex review findings

- apply_toml_overlay now returns Result and errors on explicit missing
  or invalid config paths (was log-only, violating the documented
  contract that explicit paths are fatal)
- custom tunnel url_pattern is now used to filter extracted URLs, not
  just as a gate for scanning stdout
- systemd ExecStart path is now quoted to handle spaces in paths

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: address PR review feedback

- Cache key now includes max_tokens, temperature, and stop_sequences
  so different request parameters produce distinct keys
- to_cents() uses .trunc() + parse::<u64> instead of f64 intermediary,
  avoiding precision loss for large values
- Tailscale public URL no longer includes local port (serve/funnel
  expose on standard HTTPS port 443)

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* feat: wire up tunnel lifecycle and fix audit findings

Connect the tunnel module to the rest of the application so that
setting TUNNEL_PROVIDER actually starts a managed tunnel at boot and
stops it on shutdown. Previously create_tunnel() was never called
outside tests.

Changes:
- Expand TunnelSettings with provider credential fields (settings.rs)
- TunnelConfig::resolve() falls back to DB settings when env vars unset
- Start tunnel at boot, stop on shutdown, show URL in boot screen
- Setup wizard collects provider-specific credentials (ngrok, cloudflare,
  tailscale, custom, static URL)
- Fix public_url() returning None under lock contention (SharedUrl)
- Fix local_host parameter ignored by cloudflare/ngrok/tailscale
- Fix tailscale silent fallback to "localhost" on bad JSON
- Fix ngrok globally mutating config via add-authtoken (use env var)
- Add 10s timeout to tailscale status --json

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: address PR review comments

- Document split_whitespace limitation in CustomTunnel doc comment
- Remove unnecessary quotes from systemd ExecStart directive

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: address PR review feedback (round 3)

- doctor: missing libSQL DB on fresh install is Pass, not Fail
- service: quote ExecStart path for systemd space handling

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: correct cost guard doc comment (LLM calls, not LLM/tool)

Co-Authored-By: Claude Opus 4.6 <[email protected]>

---------

Co-authored-by: Claude Opus 4.6 <[email protected]>

src/settings.rs

@@ -149,11 +149,52 @@ impl Default for EmbeddingsSettings {
 /// Tunnel settings for public webhook endpoints.
 ///
 /// The tunnel URL is shared across all channels that need webhooks.
+/// Two modes:
+/// - **Static URL**: `public_url` set directly (manual tunnel management).
+/// - **Managed provider**: `provider` is set and the agent starts/stops the
+///   tunnel process automatically at boot/shutdown.
 #[derive(Debug, Clone, Serialize, Deserialize, Default)]
 pub struct TunnelSettings {
     /// Public URL from tunnel provider (e.g., "https://abc123.ngrok.io").
+    /// When set without a provider, treated as a static (externally managed) URL.
     #[serde(default)]
     pub public_url: Option<String>,
+
+    /// Managed tunnel provider: "ngrok", "cloudflare", "tailscale", "custom".
+    #[serde(default)]
+    pub provider: Option<String>,
+
+    /// Cloudflare tunnel token.
+    #[serde(default)]
+    pub cf_token: Option<String>,
+
+    /// ngrok auth token.
+    #[serde(default)]
+    pub ngrok_token: Option<String>,
+
+    /// ngrok custom domain (paid plans).
+    #[serde(default)]
+    pub ngrok_domain: Option<String>,
+
+    /// Use Tailscale Funnel (public) instead of Serve (tailnet-only).
+    #[serde(default)]
+    pub ts_funnel: bool,
+
+    /// Tailscale hostname override.
+    #[serde(default)]
+    pub ts_hostname: Option<String>,
+
+    /// Shell command for custom tunnel (with `{port}` / `{host}` placeholders).
+    #[serde(default)]
+    pub custom_command: Option<String>,
+
+    /// Health check URL for custom tunnel.
+    #[serde(default)]
+    pub custom_health_url: Option<String>,
+
+    /// Substring pattern to extract URL from custom tunnel stdout.
+    #[serde(default)]
+    pub custom_url_pattern: Option<String>,
 }
 
 /// Channel-specific settings.
@@ -585,6 +626,83 @@ impl Settings {
         }
     }
 
+    /// Default TOML config file path (~/.ironclaw/config.toml).
+    pub fn default_toml_path() -> PathBuf {
+        dirs::home_dir()
+            .unwrap_or_else(|| PathBuf::from("."))
+            .join(".ironclaw")
+            .join("config.toml")
+    }
+
+    /// Load settings from a TOML file.
+    ///
+    /// Returns `None` if the file doesn't exist. Returns an error only
+    /// if the file exists but can't be parsed.
+    pub fn load_toml(path: &std::path::Path) -> Result<Option<Self>, String> {
+        let data = match std::fs::read_to_string(path) {
+            Ok(d) => d,
+            Err(e) if e.kind() == std::io::ErrorKind::NotFound => return Ok(None),
+            Err(e) => return Err(format!("failed to read {}: {}", path.display(), e)),
+        };
+
+        let settings: Self = toml::from_str(&data)
+            .map_err(|e| format!("invalid TOML in {}: {}", path.display(), e))?;
+        Ok(Some(settings))
+    }
+
+    /// Write a well-commented TOML config file with current settings.
+    pub fn save_toml(&self, path: &std::path::Path) -> Result<(), String> {
+        let raw = toml::to_string_pretty(self)
+            .map_err(|e| format!("failed to serialize settings: {}", e))?;
+
+        let content = format!(
+            "# IronClaw configuration file.\n\
+             #\n\
+             # Priority: env var > this file > database settings > defaults.\n\
+             # Uncomment and edit values to override defaults.\n\
+             # Run `ironclaw config init` to regenerate this file.\n\
+             #\n\
+             # Documentation: https://github.com/nearai/ironclaw\n\
+             \n\
+             {raw}"
+        );
+
+        if let Some(parent) = path.parent() {
+            std::fs::create_dir_all(parent)
+                .map_err(|e| format!("failed to create {}: {}", parent.display(), e))?;
+        }
+
+        std::fs::write(path, content)
+            .map_err(|e| format!("failed to write {}: {}", path.display(), e))
+    }
+
+    /// Merge values from `other` into `self`, preferring `other` for
+    /// fields that differ from the default.
+    ///
+    /// This enables layering: load DB/JSON settings as the base, then
+    /// overlay TOML values on top. Only fields that the TOML file
+    /// explicitly changed (i.e. differ from Default) are applied.
+    pub fn merge_from(&mut self, other: &Self) {
+        let default_json = match serde_json::to_value(Self::default()) {
+            Ok(v) => v,
+            Err(_) => return,
+        };
+        let other_json = match serde_json::to_value(other) {
+            Ok(v) => v,
+            Err(_) => return,
+        };
+        let mut self_json = match serde_json::to_value(&*self) {
+            Ok(v) => v,
+            Err(_) => return,
+        };
+
+        merge_non_default(&mut self_json, &other_json, &default_json);
+
+        if let Ok(merged) = serde_json::from_value(self_json) {
+            *self = merged;
+        }
+    }
+
     /// Get a setting value by dotted path (e.g., "agent.max_parallel_jobs").
     pub fn get(&self, path: &str) -> Option<String> {
         let json = serde_json::to_value(self).ok()?;
@@ -766,9 +884,40 @@ fn collect_settings(
     }
 }
 
+/// Recursively merge `other` into `target`, but only for fields where
+/// `other` differs from `defaults`. This means only explicitly-set values
+/// in the TOML file override the base settings.
+fn merge_non_default(
+    target: &mut serde_json::Value,
+    other: &serde_json::Value,
+    defaults: &serde_json::Value,
+) {
+    match (target, other, defaults) {
+        (
+            serde_json::Value::Object(t),
+            serde_json::Value::Object(o),
+            serde_json::Value::Object(d),
+        ) => {
+            for (key, other_val) in o {
+                let default_val = d.get(key).cloned().unwrap_or(serde_json::Value::Null);
+                if let Some(target_val) = t.get_mut(key) {
+                    merge_non_default(target_val, other_val, &default_val);
+                } else if other_val != &default_val {
+                    t.insert(key.clone(), other_val.clone());
+                }
+            }
+        }
+        (target, other, defaults) => {
+            if other != defaults {
+                *target = other.clone();
+            }
+        }
+    }
+}
+
 #[cfg(test)]
 mod tests {
-    use super::*;
+    use crate::settings::*;
 
     #[test]
     fn test_db_map_round_trip() {
@@ -904,4 +1053,152 @@ mod tests {
             Some("http://my-vllm:8000/v1".to_string())
         );
     }
+
+    #[test]
+    fn toml_round_trip() {
+        let dir = tempfile::tempdir().unwrap();
+        let path = dir.path().join("config.toml");
+
+        let mut settings = Settings::default();
+        settings.agent.name = "toml-bot".to_string();
+        settings.heartbeat.enabled = true;
+        settings.heartbeat.interval_secs = 900;
+
+        settings.save_toml(&path).unwrap();
+        let loaded = Settings::load_toml(&path).unwrap().unwrap();
+
+        assert_eq!(loaded.agent.name, "toml-bot");
+        assert!(loaded.heartbeat.enabled);
+        assert_eq!(loaded.heartbeat.interval_secs, 900);
+    }
+
+    #[test]
+    fn toml_missing_file_returns_none() {
+        let result = Settings::load_toml(std::path::Path::new("/tmp/nonexistent_config.toml"));
+        assert!(result.unwrap().is_none());
+    }
+
+    #[test]
+    fn toml_invalid_content_returns_error() {
+        let dir = tempfile::tempdir().unwrap();
+        let path = dir.path().join("bad.toml");
+        std::fs::write(&path, "this is not valid toml [[[").unwrap();
+
+        let result = Settings::load_toml(&path);
+        assert!(result.is_err());
+    }
+
+    #[test]
+    fn toml_partial_config_uses_defaults() {
+        let dir = tempfile::tempdir().unwrap();
+        let path = dir.path().join("partial.toml");
+
+        // Only set agent name, everything else should be default
+        std::fs::write(&path, "[agent]\nname = \"partial-bot\"\n").unwrap();
+
+        let loaded = Settings::load_toml(&path).unwrap().unwrap();
+        assert_eq!(loaded.agent.name, "partial-bot");
+        // Defaults preserved
+        assert_eq!(loaded.agent.max_parallel_jobs, 5);
+        assert!(!loaded.heartbeat.enabled);
+    }
+
+    #[test]
+    fn toml_header_comment_present() {
+        let dir = tempfile::tempdir().unwrap();
+        let path = dir.path().join("config.toml");
+
+        Settings::default().save_toml(&path).unwrap();
+        let content = std::fs::read_to_string(&path).unwrap();
+
+        assert!(content.starts_with("# IronClaw configuration file."));
+        assert!(content.contains("[agent]"));
+        assert!(content.contains("[heartbeat]"));
+    }
+
+    #[test]
+    fn merge_only_overrides_non_default_values() {
+        let mut base = Settings::default();
+        base.agent.name = "from-db".to_string();
+        base.heartbeat.interval_secs = 600;
+
+        let mut toml_overlay = Settings::default();
+        toml_overlay.agent.name = "from-toml".to_string();
+        // heartbeat.interval_secs stays at default (1800) in the overlay,
+        // so the base value (600) should be preserved.
+
+        base.merge_from(&toml_overlay);
+
+        assert_eq!(base.agent.name, "from-toml");
+        assert_eq!(base.heartbeat.interval_secs, 600);
+    }
+
+    #[test]
+    fn merge_preserves_base_when_overlay_is_default() {
+        let mut base = Settings::default();
+        base.agent.name = "custom-name".to_string();
+        base.heartbeat.enabled = true;
+
+        let overlay = Settings::default();
+        base.merge_from(&overlay);
+
+        // All base values preserved since overlay is entirely default
+        assert_eq!(base.agent.name, "custom-name");
+        assert!(base.heartbeat.enabled);
+    }
+
+    #[test]
+    fn toml_creates_parent_dirs() {
+        let dir = tempfile::tempdir().unwrap();
+        let path = dir.path().join("nested").join("deep").join("config.toml");
+
+        Settings::default().save_toml(&path).unwrap();
+        assert!(path.exists());
+    }
+
+    #[test]
+    fn default_toml_path_under_ironclaw() {
+        let path = Settings::default_toml_path();
+        assert!(path.to_string_lossy().contains(".ironclaw"));
+        assert!(path.to_string_lossy().ends_with("config.toml"));
+    }
+
+    #[test]
+    fn tunnel_settings_round_trip() {
+        let settings = Settings {
+            tunnel: TunnelSettings {
+                provider: Some("ngrok".to_string()),
+                ngrok_token: Some("tok_abc123".to_string()),
+                ngrok_domain: Some("my.ngrok.dev".to_string()),
+                ..Default::default()
+            },
+            ..Default::default()
+        };
+
+        // JSON round-trip
+        let json = serde_json::to_string(&settings).unwrap();
+        let restored: Settings = serde_json::from_str(&json).unwrap();
+        assert_eq!(restored.tunnel.provider, Some("ngrok".to_string()));
+        assert_eq!(restored.tunnel.ngrok_token, Some("tok_abc123".to_string()));
+        assert_eq!(
+            restored.tunnel.ngrok_domain,
+            Some("my.ngrok.dev".to_string())
+        );
+        assert!(restored.tunnel.public_url.is_none());
+
+        // DB map round-trip
+        let map = settings.to_db_map();
+        let from_db = Settings::from_db_map(&map);
+        assert_eq!(from_db.tunnel.provider, Some("ngrok".to_string()));
+        assert_eq!(from_db.tunnel.ngrok_token, Some("tok_abc123".to_string()));
+
+        // get/set round-trip
+        let mut s = Settings::default();
+        s.set("tunnel.provider", "cloudflare").unwrap();
+        s.set("tunnel.cf_token", "cf_tok_xyz").unwrap();
+        s.set("tunnel.ts_funnel", "true").unwrap();
+        assert_eq!(s.tunnel.provider, Some("cloudflare".to_string()));
+        assert_eq!(s.tunnel.cf_token, Some("cf_tok_xyz".to_string()));
+        assert!(s.tunnel.ts_funnel);
+    }
 }