feat: 10 infrastructure improvements from zeroclaw (#126)

* refactor: break up agent_loop.rs into four focused modules Split the monolithic 2835-line agent_loop.rs into: - agent_loop.rs (722L): Agent struct, event loop, message dispatch - dispatcher.rs (635L): Agentic tool loop, tool execution, auth detection - commands.rs (484L): System commands, job handlers, heartbeat, summarize - thread_ops.rs (1059L): Thread lifecycle, approval, undo/redo, persistence Each module gets its own impl Agent block. Agent fields changed to pub(super) so sibling modules in the agent package can access them. All 16 existing tests pass in their new locations. Inspired by ZeroClaw's agent module split (agent.rs, loop_.rs, dispatcher.rs, prompt.rs, memory_loader.rs). Co-Authored-By: Claude Opus 4.6 <[email protected]> * feat: add cost caps and guardrails for autonomous agent spending Daily budget (MAX_COST_PER_DAY_CENTS) and hourly action rate (MAX_ACTIONS_PER_HOUR) limits prevent runaway agents from burning through API credits, especially in daemon/heartbeat modes. - CostGuard with pre-flight check and post-call recording - Sliding window for hourly rate, midnight-UTC daily reset - 80% threshold warning, atomic fast-path for exceeded budget - Wired into dispatcher loop (check before LLM call, record after) Co-Authored-By: Claude Opus 4.6 <[email protected]> * feat: add circuit breaker on LLM providers Wraps LlmProvider with a Closed/Open/HalfOpen state machine that trips after consecutive transient failures, preventing request storms against a degraded backend. Automatically probes for recovery. - CircuitBreakerProvider implements LlmProvider (drop-in wrapper) - Transient error classification (server, rate-limit, network, auth infra) - Client errors (wrong model, context overflow) don't trip the breaker - Configurable via CIRCUIT_BREAKER_THRESHOLD and CIRCUIT_BREAKER_RECOVERY_SECS - Composes with existing FailoverProvider (circuit breaker wraps failover) - 12 tests covering full state machine and error classification Co-Authored-By: Claude Opus 4.6 <[email protected]> * feat: add tunnel abstraction for remote access Trait-based tunnel system with lifecycle management (start/stop/health) for exposing the agent to the internet through external tunnel binaries. Five providers: - Cloudflare Tunnel (cloudflared, Zero Trust token auth) - Tailscale (serve for tailnet, funnel for public) - ngrok (with optional custom domain) - Custom (arbitrary command with {host}/{port} placeholders) - None (local-only, no external exposure) Config via TUNNEL_PROVIDER + provider-specific env vars. Extends existing TunnelConfig with optional managed provider alongside the static TUNNEL_URL path. Factory, shared process management, and 37 tests covering all providers and edge cases. Co-Authored-By: Claude Opus 4.6 <[email protected]> * feat: add OS service management (launchd/systemd) Adds `ironclaw service {install,start,stop,status,uninstall}` for running the agent as a background daemon. macOS uses launchd plists under ~/Library/LaunchAgents, Linux uses systemd user units. Co-Authored-By: Claude Opus 4.6 <[email protected]> * feat: add observability trait system with noop, log, and multi backends Introduces an Observer trait for recording agent lifecycle events and metrics, with pluggable backends. The noop backend compiles to zero overhead, log backend uses tracing, and multi fans out to multiple observers. Configured via OBSERVABILITY_BACKEND env var. Co-Authored-By: Claude Opus 4.6 <[email protected]> * feat: add in-memory LLM response cache with TTL and LRU eviction CachedProvider wraps any LlmProvider and caches complete() responses keyed by SHA-256(model + messages). Tool-calling requests are never cached since they trigger side effects. Configurable via RESPONSE_CACHE_ENABLED, RESPONSE_CACHE_TTL_SECS, and RESPONSE_CACHE_MAX_ENTRIES env vars. Co-Authored-By: Claude Opus 4.6 <[email protected]> * feat: add memory hygiene with cadence-gated daily log cleanup Adds workspace::hygiene module that automatically deletes daily log documents older than a configurable retention period (default 30 days). Runs on a 12-hour cadence tracked via a local state file to avoid redundant passes. Best-effort design: failures are logged, never fatal. Co-Authored-By: Claude Opus 4.6 <[email protected]> * feat: add doctor diagnostics command for active health probing Probes external dependencies (Docker, cloudflared, ngrok, tailscale), validates NEAR AI session, checks database connectivity, and verifies workspace directory. Complements the passive `status` command. Co-Authored-By: Claude Opus 4.6 <[email protected]> * feat: add structured TOML config file support Adds ~/.ironclaw/config.toml as a configuration layer between env vars and database settings. Priority: env var > TOML file > DB > defaults. - `ironclaw config init` generates a commented config.toml from current settings - `ironclaw --config path/to/config.toml` loads a custom config file - Settings.merge_from() only overlays non-default values from the TOML file - `ironclaw config path` now shows TOML file status Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: address codex review findings - apply_toml_overlay now returns Result and errors on explicit missing or invalid config paths (was log-only, violating the documented contract that explicit paths are fatal) - custom tunnel url_pattern is now used to filter extracted URLs, not just as a gate for scanning stdout - systemd ExecStart path is now quoted to handle spaces in paths Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: address PR review feedback - Cache key now includes max_tokens, temperature, and stop_sequences so different request parameters produce distinct keys - to_cents() uses .trunc() + parse::<u64> instead of f64 intermediary, avoiding precision loss for large values - Tailscale public URL no longer includes local port (serve/funnel expose on standard HTTPS port 443) Co-Authored-By: Claude Opus 4.6 <[email protected]> * feat: wire up tunnel lifecycle and fix audit findings Connect the tunnel module to the rest of the application so that setting TUNNEL_PROVIDER actually starts a managed tunnel at boot and stops it on shutdown. Previously create_tunnel() was never called outside tests. Changes: - Expand TunnelSettings with provider credential fields (settings.rs) - TunnelConfig::resolve() falls back to DB settings when env vars unset - Start tunnel at boot, stop on shutdown, show URL in boot screen - Setup wizard collects provider-specific credentials (ngrok, cloudflare, tailscale, custom, static URL) - Fix public_url() returning None under lock contention (SharedUrl) - Fix local_host parameter ignored by cloudflare/ngrok/tailscale - Fix tailscale silent fallback to "localhost" on bad JSON - Fix ngrok globally mutating config via add-authtoken (use env var) - Add 10s timeout to tailscale status --json Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: address PR review comments - Document split_whitespace limitation in CustomTunnel doc comment - Remove unnecessary quotes from systemd ExecStart directive Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: address PR review feedback (round 3) - doctor: missing libSQL DB on fresh install is Pass, not Fail - service: quote ExecStart path for systemd space handling Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: correct cost guard doc comment (LLM calls, not LLM/tool) Co-Authored-By: Claude Opus 4.6 <[email protected]> --------- Co-authored-by: Claude Opus 4.6 <[email protected]>
2026-08-26 15:40:18 +00:00 · 2026-02-17 19:35:56 +00:00
parent 436dda0f2f
commit a158eee1b0
37 changed files with 7272 additions and 2177 deletions
@@ -0,0 +1,181 @@
+//! Tracing-based observer that emits structured log events.
+//!
+//! Uses the existing `tracing` infrastructure so events appear alongside
+//! normal application logs, with no extra dependencies. Good for local
+//! development and debugging.
+
+use crate::observability::traits::{Observer, ObserverEvent, ObserverMetric};
+
+/// Observer that logs events and metrics via `tracing`.
+pub struct LogObserver;
+
+impl Observer for LogObserver {
+    fn record_event(&self, event: &ObserverEvent) {
+        match event {
+            ObserverEvent::AgentStart { provider, model } => {
+                tracing::info!(provider, model, "observer: agent.start");
+            }
+            ObserverEvent::LlmRequest {
+                provider,
+                model,
+                message_count,
+            } => {
+                tracing::info!(provider, model, message_count, "observer: llm.request");
+            }
+            ObserverEvent::LlmResponse {
+                provider,
+                model,
+                duration,
+                success,
+                error_message,
+            } => {
+                tracing::info!(
+                    provider,
+                    model,
+                    duration_ms = duration.as_millis() as u64,
+                    success,
+                    error = error_message.as_deref().unwrap_or(""),
+                    "observer: llm.response"
+                );
+            }
+            ObserverEvent::ToolCallStart { tool } => {
+                tracing::info!(tool, "observer: tool.start");
+            }
+            ObserverEvent::ToolCallEnd {
+                tool,
+                duration,
+                success,
+            } => {
+                tracing::info!(
+                    tool,
+                    duration_ms = duration.as_millis() as u64,
+                    success,
+                    "observer: tool.end"
+                );
+            }
+            ObserverEvent::TurnComplete => {
+                tracing::info!("observer: turn.complete");
+            }
+            ObserverEvent::ChannelMessage { channel, direction } => {
+                tracing::info!(channel, direction, "observer: channel.message");
+            }
+            ObserverEvent::HeartbeatTick => {
+                tracing::debug!("observer: heartbeat.tick");
+            }
+            ObserverEvent::AgentEnd {
+                duration,
+                tokens_used,
+            } => {
+                tracing::info!(
+                    duration_secs = duration.as_secs_f64(),
+                    tokens_used = tokens_used.unwrap_or(0),
+                    "observer: agent.end"
+                );
+            }
+            ObserverEvent::Error { component, message } => {
+                tracing::warn!(component, error = message.as_str(), "observer: error");
+            }
+        }
+    }
+
+    fn record_metric(&self, metric: &ObserverMetric) {
+        match metric {
+            ObserverMetric::RequestLatency(d) => {
+                tracing::debug!(
+                    latency_ms = d.as_millis() as u64,
+                    "observer: metric.request_latency"
+                );
+            }
+            ObserverMetric::TokensUsed(n) => {
+                tracing::debug!(tokens = n, "observer: metric.tokens_used");
+            }
+            ObserverMetric::ActiveJobs(n) => {
+                tracing::debug!(active_jobs = n, "observer: metric.active_jobs");
+            }
+            ObserverMetric::QueueDepth(n) => {
+                tracing::debug!(queue_depth = n, "observer: metric.queue_depth");
+            }
+        }
+    }
+
+    fn name(&self) -> &str {
+        "log"
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use std::time::Duration;
+
+    use crate::observability::log::LogObserver;
+    use crate::observability::traits::*;
+
+    #[test]
+    fn name_is_log() {
+        assert_eq!(LogObserver.name(), "log");
+    }
+
+    #[test]
+    fn record_event_does_not_panic() {
+        let obs = LogObserver;
+        obs.record_event(&ObserverEvent::AgentStart {
+            provider: "nearai".into(),
+            model: "test".into(),
+        });
+        obs.record_event(&ObserverEvent::LlmRequest {
+            provider: "nearai".into(),
+            model: "test".into(),
+            message_count: 5,
+        });
+        obs.record_event(&ObserverEvent::LlmResponse {
+            provider: "nearai".into(),
+            model: "test".into(),
+            duration: Duration::from_millis(150),
+            success: true,
+            error_message: None,
+        });
+        obs.record_event(&ObserverEvent::LlmResponse {
+            provider: "nearai".into(),
+            model: "test".into(),
+            duration: Duration::from_millis(1500),
+            success: false,
+            error_message: Some("timeout".into()),
+        });
+        obs.record_event(&ObserverEvent::ToolCallStart {
+            tool: "shell".into(),
+        });
+        obs.record_event(&ObserverEvent::ToolCallEnd {
+            tool: "shell".into(),
+            duration: Duration::from_millis(20),
+            success: true,
+        });
+        obs.record_event(&ObserverEvent::TurnComplete);
+        obs.record_event(&ObserverEvent::ChannelMessage {
+            channel: "tui".into(),
+            direction: "inbound".into(),
+        });
+        obs.record_event(&ObserverEvent::HeartbeatTick);
+        obs.record_event(&ObserverEvent::AgentEnd {
+            duration: Duration::from_secs(30),
+            tokens_used: Some(2500),
+        });
+        obs.record_event(&ObserverEvent::Error {
+            component: "llm".into(),
+            message: "connection refused".into(),
+        });
+    }
+
+    #[test]
+    fn record_metric_does_not_panic() {
+        let obs = LogObserver;
+        obs.record_metric(&ObserverMetric::RequestLatency(Duration::from_millis(200)));
+        obs.record_metric(&ObserverMetric::TokensUsed(1000));
+        obs.record_metric(&ObserverMetric::ActiveJobs(5));
+        obs.record_metric(&ObserverMetric::QueueDepth(12));
+    }
+
+    #[test]
+    fn flush_does_not_panic() {
+        LogObserver.flush();
+    }
+}
@@ -0,0 +1,105 @@
+//! Observability subsystem: trait-based event and metric recording.
+//!
+//! Provides a pluggable [`Observer`] trait with multiple backends:
+//!
+//! | Backend | Description |
+//! |---------|-------------|
+//! | `noop`  | Zero overhead, discards everything (default) |
+//! | `log`   | Emits structured events via `tracing` |
+//! | `multi` | Fan-out to multiple backends simultaneously |
+//!
+//! The [`create_observer`] factory builds the right backend from
+//! [`ObservabilityConfig`]. Future backends (OpenTelemetry, Prometheus)
+//! can be added by implementing [`Observer`].
+
+mod log;
+mod multi;
+mod noop;
+pub mod traits;
+
+pub use self::log::LogObserver;
+pub use self::multi::MultiObserver;
+pub use self::noop::NoopObserver;
+pub use self::traits::{Observer, ObserverEvent, ObserverMetric};
+
+/// Configuration for the observability backend.
+#[derive(Debug, Clone)]
+pub struct ObservabilityConfig {
+    /// Backend name: "none", "noop", "log".
+    pub backend: String,
+}
+
+impl Default for ObservabilityConfig {
+    fn default() -> Self {
+        Self {
+            backend: "none".into(),
+        }
+    }
+}
+
+/// Create an observer from configuration.
+///
+/// Returns a [`NoopObserver`] for "none"/"noop" (or unknown values),
+/// and a [`LogObserver`] for "log".
+pub fn create_observer(config: &ObservabilityConfig) -> Box<dyn Observer> {
+    match config.backend.as_str() {
+        "log" => Box::new(LogObserver),
+        _ => Box::new(NoopObserver),
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use crate::observability::*;
+
+    #[test]
+    fn default_config_is_none() {
+        let cfg = ObservabilityConfig::default();
+        assert_eq!(cfg.backend, "none");
+    }
+
+    #[test]
+    fn factory_returns_noop_for_none() {
+        let cfg = ObservabilityConfig {
+            backend: "none".into(),
+        };
+        let obs = create_observer(&cfg);
+        assert_eq!(obs.name(), "noop");
+    }
+
+    #[test]
+    fn factory_returns_noop_for_empty() {
+        let cfg = ObservabilityConfig {
+            backend: String::new(),
+        };
+        let obs = create_observer(&cfg);
+        assert_eq!(obs.name(), "noop");
+    }
+
+    #[test]
+    fn factory_returns_noop_for_unknown() {
+        let cfg = ObservabilityConfig {
+            backend: "prometheus".into(),
+        };
+        let obs = create_observer(&cfg);
+        assert_eq!(obs.name(), "noop");
+    }
+
+    #[test]
+    fn factory_returns_log_for_log() {
+        let cfg = ObservabilityConfig {
+            backend: "log".into(),
+        };
+        let obs = create_observer(&cfg);
+        assert_eq!(obs.name(), "log");
+    }
+
+    #[test]
+    fn factory_returns_noop_for_noop() {
+        let cfg = ObservabilityConfig {
+            backend: "noop".into(),
+        };
+        let obs = create_observer(&cfg);
+        assert_eq!(obs.name(), "noop");
+    }
+}
@@ -0,0 +1,139 @@
+//! Fan-out observer that dispatches to multiple backends.
+//!
+//! Useful for combining backends, e.g. log + OpenTelemetry simultaneously.
+
+use crate::observability::traits::{Observer, ObserverEvent, ObserverMetric};
+
+/// Dispatches events and metrics to all inner observers.
+pub struct MultiObserver {
+    observers: Vec<Box<dyn Observer>>,
+}
+
+impl MultiObserver {
+    /// Create from a list of observers. If the list is empty the result
+    /// behaves like a noop.
+    pub fn new(observers: Vec<Box<dyn Observer>>) -> Self {
+        Self { observers }
+    }
+}
+
+impl Observer for MultiObserver {
+    fn record_event(&self, event: &ObserverEvent) {
+        for obs in &self.observers {
+            obs.record_event(event);
+        }
+    }
+
+    fn record_metric(&self, metric: &ObserverMetric) {
+        for obs in &self.observers {
+            obs.record_metric(metric);
+        }
+    }
+
+    fn flush(&self) {
+        for obs in &self.observers {
+            obs.flush();
+        }
+    }
+
+    fn name(&self) -> &str {
+        "multi"
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use std::sync::Arc;
+    use std::sync::atomic::{AtomicUsize, Ordering};
+    use std::time::Duration;
+
+    use crate::observability::multi::MultiObserver;
+    use crate::observability::traits::*;
+
+    /// Test observer that counts calls via shared atomic counters.
+    struct CountingObserver {
+        events: Arc<AtomicUsize>,
+        metrics: Arc<AtomicUsize>,
+        flushes: Arc<AtomicUsize>,
+    }
+
+    impl CountingObserver {
+        fn new() -> (Self, Arc<AtomicUsize>, Arc<AtomicUsize>, Arc<AtomicUsize>) {
+            let events = Arc::new(AtomicUsize::new(0));
+            let metrics = Arc::new(AtomicUsize::new(0));
+            let flushes = Arc::new(AtomicUsize::new(0));
+            (
+                Self {
+                    events: Arc::clone(&events),
+                    metrics: Arc::clone(&metrics),
+                    flushes: Arc::clone(&flushes),
+                },
+                events,
+                metrics,
+                flushes,
+            )
+        }
+    }
+
+    impl Observer for CountingObserver {
+        fn record_event(&self, _event: &ObserverEvent) {
+            self.events.fetch_add(1, Ordering::Relaxed);
+        }
+        fn record_metric(&self, _metric: &ObserverMetric) {
+            self.metrics.fetch_add(1, Ordering::Relaxed);
+        }
+        fn flush(&self) {
+            self.flushes.fetch_add(1, Ordering::Relaxed);
+        }
+        fn name(&self) -> &str {
+            "counting"
+        }
+    }
+
+    #[test]
+    fn name_is_multi() {
+        let multi = MultiObserver::new(vec![]);
+        assert_eq!(multi.name(), "multi");
+    }
+
+    #[test]
+    fn empty_multi_does_not_panic() {
+        let multi = MultiObserver::new(vec![]);
+        multi.record_event(&ObserverEvent::TurnComplete);
+        multi.record_metric(&ObserverMetric::TokensUsed(100));
+        multi.flush();
+    }
+
+    #[test]
+    fn dispatches_to_all_observers() {
+        let (a, a_events, a_metrics, a_flushes) = CountingObserver::new();
+        let (b, b_events, b_metrics, b_flushes) = CountingObserver::new();
+
+        let multi = MultiObserver::new(vec![Box::new(a), Box::new(b)]);
+
+        multi.record_event(&ObserverEvent::TurnComplete);
+        multi.record_event(&ObserverEvent::HeartbeatTick);
+        multi.record_metric(&ObserverMetric::TokensUsed(50));
+        multi.flush();
+
+        assert_eq!(a_events.load(Ordering::Relaxed), 2);
+        assert_eq!(a_metrics.load(Ordering::Relaxed), 1);
+        assert_eq!(a_flushes.load(Ordering::Relaxed), 1);
+        assert_eq!(b_events.load(Ordering::Relaxed), 2);
+        assert_eq!(b_metrics.load(Ordering::Relaxed), 1);
+        assert_eq!(b_flushes.load(Ordering::Relaxed), 1);
+    }
+
+    #[test]
+    fn single_observer_works() {
+        let (obs, events, _, _) = CountingObserver::new();
+
+        let multi = MultiObserver::new(vec![Box::new(obs)]);
+        multi.record_event(&ObserverEvent::AgentEnd {
+            duration: Duration::from_secs(1),
+            tokens_used: None,
+        });
+
+        assert_eq!(events.load(Ordering::Relaxed), 1);
+    }
+}
@@ -0,0 +1,60 @@
+//! Zero-overhead no-op observer.
+//!
+//! Default backend when observability is disabled. All methods compile to
+//! nothing, so there is zero runtime cost.
+
+use crate::observability::traits::{Observer, ObserverEvent, ObserverMetric};
+
+/// Observer that discards all events and metrics.
+pub struct NoopObserver;
+
+impl Observer for NoopObserver {
+    #[inline(always)]
+    fn record_event(&self, _event: &ObserverEvent) {}
+
+    #[inline(always)]
+    fn record_metric(&self, _metric: &ObserverMetric) {}
+
+    fn name(&self) -> &str {
+        "noop"
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use std::time::Duration;
+
+    use crate::observability::traits::*;
+
+    use crate::observability::noop::NoopObserver;
+
+    #[test]
+    fn name_is_noop() {
+        assert_eq!(NoopObserver.name(), "noop");
+    }
+
+    #[test]
+    fn record_event_does_not_panic() {
+        let obs = NoopObserver;
+        obs.record_event(&ObserverEvent::TurnComplete);
+        obs.record_event(&ObserverEvent::HeartbeatTick);
+        obs.record_event(&ObserverEvent::AgentStart {
+            provider: "x".into(),
+            model: "y".into(),
+        });
+    }
+
+    #[test]
+    fn record_metric_does_not_panic() {
+        let obs = NoopObserver;
+        obs.record_metric(&ObserverMetric::TokensUsed(100));
+        obs.record_metric(&ObserverMetric::RequestLatency(Duration::from_millis(50)));
+        obs.record_metric(&ObserverMetric::ActiveJobs(2));
+        obs.record_metric(&ObserverMetric::QueueDepth(0));
+    }
+
+    #[test]
+    fn flush_does_not_panic() {
+        NoopObserver.flush();
+    }
+}
@@ -0,0 +1,143 @@
+//! Core observer trait and event/metric types.
+
+use std::time::Duration;
+
+/// Provider-agnostic observer for agent lifecycle events and metrics.
+///
+/// Implementations can log to tracing, export to OpenTelemetry, write to
+/// Prometheus, or do nothing at all. The agent records events at key
+/// lifecycle points and the observer decides what to do with them.
+///
+/// Thread-safe and cheaply cloneable behind `Arc<dyn Observer>`.
+pub trait Observer: Send + Sync {
+    /// Record a discrete lifecycle event.
+    fn record_event(&self, event: &ObserverEvent);
+
+    /// Record a numeric metric sample.
+    fn record_metric(&self, metric: &ObserverMetric);
+
+    /// Flush any buffered data (e.g. OTLP batch exporter). No-op by default.
+    fn flush(&self) {}
+
+    /// Human-readable backend name (e.g. "noop", "log", "otel").
+    fn name(&self) -> &str;
+}
+
+/// Discrete lifecycle events the agent can emit.
+#[derive(Debug, Clone)]
+pub enum ObserverEvent {
+    /// Agent started processing.
+    AgentStart { provider: String, model: String },
+
+    /// An LLM request was sent.
+    LlmRequest {
+        provider: String,
+        model: String,
+        message_count: usize,
+    },
+
+    /// An LLM response was received.
+    LlmResponse {
+        provider: String,
+        model: String,
+        duration: Duration,
+        success: bool,
+        error_message: Option<String>,
+    },
+
+    /// A tool call is about to start.
+    ToolCallStart { tool: String },
+
+    /// A tool call finished.
+    ToolCallEnd {
+        tool: String,
+        duration: Duration,
+        success: bool,
+    },
+
+    /// One reasoning turn completed.
+    TurnComplete,
+
+    /// A message was sent or received on a channel.
+    ChannelMessage { channel: String, direction: String },
+
+    /// The heartbeat system ran a tick.
+    HeartbeatTick,
+
+    /// Agent finished processing.
+    AgentEnd {
+        duration: Duration,
+        tokens_used: Option<u64>,
+    },
+
+    /// An error occurred in a component.
+    Error { component: String, message: String },
+}
+
+/// Numeric metric samples.
+#[derive(Debug, Clone)]
+pub enum ObserverMetric {
+    /// Latency of a single request (histogram-style).
+    RequestLatency(Duration),
+    /// Cumulative tokens consumed.
+    TokensUsed(u64),
+    /// Current number of active jobs (gauge).
+    ActiveJobs(u64),
+    /// Current message queue depth (gauge).
+    QueueDepth(u64),
+}
+
+#[cfg(test)]
+mod tests {
+    use crate::observability::traits::*;
+
+    #[test]
+    fn event_variants_are_constructible() {
+        let _ = ObserverEvent::AgentStart {
+            provider: "nearai".into(),
+            model: "test".into(),
+        };
+        let _ = ObserverEvent::LlmRequest {
+            provider: "nearai".into(),
+            model: "test".into(),
+            message_count: 3,
+        };
+        let _ = ObserverEvent::LlmResponse {
+            provider: "nearai".into(),
+            model: "test".into(),
+            duration: Duration::from_millis(100),
+            success: true,
+            error_message: None,
+        };
+        let _ = ObserverEvent::ToolCallStart {
+            tool: "echo".into(),
+        };
+        let _ = ObserverEvent::ToolCallEnd {
+            tool: "echo".into(),
+            duration: Duration::from_millis(5),
+            success: true,
+        };
+        let _ = ObserverEvent::TurnComplete;
+        let _ = ObserverEvent::ChannelMessage {
+            channel: "tui".into(),
+            direction: "inbound".into(),
+        };
+        let _ = ObserverEvent::HeartbeatTick;
+        let _ = ObserverEvent::AgentEnd {
+            duration: Duration::from_secs(10),
+            tokens_used: Some(1500),
+        };
+        let _ = ObserverEvent::Error {
+            component: "llm".into(),
+            message: "timeout".into(),
+        };
+    }
+
+    #[test]
+    fn metric_variants_are_constructible() {
+        let _ = ObserverMetric::RequestLatency(Duration::from_millis(200));
+        let _ = ObserverMetric::TokensUsed(500);
+        let _ = ObserverMetric::ActiveJobs(3);
+        let _ = ObserverMetric::QueueDepth(10);
+    }
+}