mirror of
https://github.com/outbackdingo/optimclaw.git
synced 2026-08-25 14:53:34 +00:00
* feat: complete multi-tenant isolation — per-user budgets, model selection, heartbeat cycling Finishes the remaining isolation work from phases 2–4 of #59: Phase 2 (DB scoping): Fix /status and /list commands to use _for_user DB variants instead of global queries that leaked cross-user job data. Phase 3 (Runtime isolation): Per-user workspace in routine engine's spawn_fire so lightweight routines run in the correct user context. Per-user daily cost tracking in CostGuard with configurable budget via MAX_COST_PER_USER_PER_DAY_CENTS. Multi-user heartbeat that cycles through all users with routines, auto-detected from GATEWAY_USER_TOKENS. Phase 4 (Provider/tools): Per-user model selection via preferred_model setting — looked up from SettingsStore on first iteration, threaded through ReasoningContext.model_override to CompletionRequest. Works with providers that support per-request model overrides (NearAI). Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> * fix: use selected_model setting key to match /model command persistence The dispatcher was reading "preferred_model" but the /model command (merged from staging) persists to "selected_model". Since set_setting is already per-user scoped, using the same key makes /model work as the per-user model override in multi-tenant mode. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> * fix: heartbeat hygiene, /model multi-tenant guard, RigAdapter model override Three follow-up fixes for multi-tenant isolation: 1. Multi-user heartbeat now runs memory hygiene per user before each heartbeat check, matching single-user heartbeat behavior. 2. /model command in multi-tenant mode only persists to per-user settings (selected_model) without calling set_model() on the shared LlmProvider. The per-request model_override in the dispatcher reads from the same setting. Added multi_tenant flag to AgentConfig (auto-detected from GATEWAY_USER_TOKENS). 3. RigAdapter now supports per-request model overrides by injecting the model name into rig-core's additional_params. OpenAI/Anthropic/Ollama API servers use last-key-wins for duplicate JSON keys, so the override takes effect via serde's flatten serialization order. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> * fix: address PR review — cost model attribution, heartbeat concurrency, pruning Fixes from review comments on #1614: - Cost tracking now uses the override model name (not active_model_name) when a per-user model override is active, for accurate attribution. - Multi-user heartbeat runs per-user checks concurrently via JoinSet instead of sequentially, preventing one slow user from blocking others. - Per-user failure counts tracked independently; users exceeding max_failures are skipped (matching single-user semantics). - per_user_daily_cost HashMap pruned on day rollover to prevent unbounded growth in long-lived deployments. - Doc comment fixed: says "routines" not "active routines". Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> * fix: /status ownership, model persistence scoping, heartbeat robustness Addresses second round of PR review on #1614: - /status <job_id> DB path now validates job.user_id == requesting user before returning data (was missing ownership check, security fix). - persist_selected_model takes user_id param instead of owner_id, and skips .env/TOML writes in multi-tenant mode (these are shared global files). handle_system_command now receives user_id from caller. - JoinSet collection handles Err(JoinError) explicitly instead of silently dropping panicked tasks. - Notification forwarder extracts owner_id from response metadata in multi-tenant mode for per-user routing instead of broadcasting to the agent owner. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> * fix: cost pricing, fire_manual workspace, heartbeat concurrency cap Round 3 review fixes: - Cost tracking passes None for cost_per_token when model override is active, letting CostGuard look up pricing by model name instead of using the default provider's rates (serrrfirat). - fire_manual() now uses per-user workspace, matching spawn_fire() pattern (serrrfirat). - Removed MULTI_TENANT env var — multi-tenant mode is auto-detected solely from GATEWAY_USER_TOKENS presence (serrrfirat + Copilot). - Multi-user heartbeat capped at 8 concurrent tasks to avoid flooding the LLM provider (serrrfirat + Copilot). - Fixed inject_model_override doc comment accuracy (Copilot). - Added comment explaining multi-tenant notification routing priority (Copilot). Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> * feat: user-scoped webhook endpoint for multi-tenant isolation Adds POST /api/webhooks/u/{user_id}/{path} — a user-scoped webhook endpoint that filters the routine lookup by user_id, preventing cross-user webhook triggering when paths collide. The existing /api/webhooks/{path} endpoint remains unchanged for backward compatibility in single-user deployments. Changes: - get_webhook_routine_by_path gains user_id: Option<&str> param - Both postgres and libsql implementations add AND user_id = ? filter when user_id is provided - New webhook_trigger_user_scoped_handler extracts (user_id, path) from URL and passes to shared fire_webhook_inner logic - Route registered on public router (webhooks are called by external services that can't send bearer tokens) Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> * feat: add TenantCtx for compile-time tenant isolation Implements zmanian's architectural proposal from #1614 review: two-tier scoped database access (TenantScope/AdminScope) so handler code cannot accidentally bypass tenant scoping. TenantScope (default): wraps user_id + Arc<dyn Database>, auto-binds user_id on every operation. ID-based lookups return None for cross- tenant resources. No escape hatch — forgetting to scope is a compile error. AdminScope (explicit opt-in): cross-tenant access for system-level components (heartbeat, routine engine, self-repair, scheduler, worker). TenantCtx bundles TenantScope + workspace + cost guard + per-user rate limiting. Constructed once per request in handle_message, threaded through all command handlers and ChatDelegate. Key changes: - New src/tenant.rs (~920 lines): TenantScope, AdminScope, TenantCtx, TenantRateState, TenantRateRegistry - All command handlers: user_id: &str → ctx: &TenantCtx - ChatDelegate: cost check/record/settings via self.tenant - System components: store field changed to AdminScope - Config: TENANT_MAX_LLM_CONCURRENT, TENANT_MAX_JOBS_CONCURRENT env vars - Fixes bug: /status <job_id> cross-tenant leak (now auto-filtered) Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> --------- Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>
172 lines
6.4 KiB
Rust
172 lines
6.4 KiB
Rust
use crate::config::helpers::{optional_env, parse_bool_env, parse_option_env, parse_optional_env};
|
|
use crate::error::ConfigError;
|
|
use crate::settings::Settings;
|
|
|
|
/// Heartbeat configuration.
|
|
#[derive(Debug, Clone)]
|
|
pub struct HeartbeatConfig {
|
|
/// Whether heartbeat is enabled.
|
|
pub enabled: bool,
|
|
/// Interval between heartbeat checks in seconds (used when fire_at is not set).
|
|
pub interval_secs: u64,
|
|
/// Channel to notify on heartbeat findings.
|
|
pub notify_channel: Option<String>,
|
|
/// User ID to notify on heartbeat findings.
|
|
pub notify_user: Option<String>,
|
|
/// Fixed time-of-day to fire (HH:MM, 24h). When set, interval_secs is ignored.
|
|
pub fire_at: Option<chrono::NaiveTime>,
|
|
/// Hour (0-23) when quiet hours start.
|
|
pub quiet_hours_start: Option<u32>,
|
|
/// Hour (0-23) when quiet hours end.
|
|
pub quiet_hours_end: Option<u32>,
|
|
/// Timezone for fire_at and quiet hours evaluation (IANA name).
|
|
pub timezone: Option<String>,
|
|
/// When true, cycle through all users with routines. Auto-detected from
|
|
/// GATEWAY_USER_TOKENS or set explicitly via HEARTBEAT_MULTI_TENANT.
|
|
pub multi_tenant: bool,
|
|
}
|
|
|
|
impl Default for HeartbeatConfig {
|
|
fn default() -> Self {
|
|
Self {
|
|
enabled: false,
|
|
interval_secs: 1800, // 30 minutes
|
|
notify_channel: None,
|
|
notify_user: None,
|
|
fire_at: None,
|
|
quiet_hours_start: None,
|
|
quiet_hours_end: None,
|
|
timezone: None,
|
|
multi_tenant: false,
|
|
}
|
|
}
|
|
}
|
|
|
|
impl HeartbeatConfig {
|
|
pub(crate) fn resolve(settings: &Settings) -> Result<Self, ConfigError> {
|
|
let fire_at_str =
|
|
optional_env("HEARTBEAT_FIRE_AT")?.or_else(|| settings.heartbeat.fire_at.clone());
|
|
let fire_at = fire_at_str
|
|
.map(|s| {
|
|
chrono::NaiveTime::parse_from_str(&s, "%H:%M").map_err(|e| {
|
|
ConfigError::InvalidValue {
|
|
key: "HEARTBEAT_FIRE_AT".to_string(),
|
|
message: format!("must be HH:MM (24h), e.g. '14:00': {e}"),
|
|
}
|
|
})
|
|
})
|
|
.transpose()?;
|
|
|
|
Ok(Self {
|
|
enabled: parse_bool_env("HEARTBEAT_ENABLED", settings.heartbeat.enabled)?,
|
|
interval_secs: parse_optional_env(
|
|
"HEARTBEAT_INTERVAL_SECS",
|
|
settings.heartbeat.interval_secs,
|
|
)?,
|
|
notify_channel: optional_env("HEARTBEAT_NOTIFY_CHANNEL")?
|
|
.or_else(|| settings.heartbeat.notify_channel.clone()),
|
|
notify_user: optional_env("HEARTBEAT_NOTIFY_USER")?
|
|
.or_else(|| settings.heartbeat.notify_user.clone()),
|
|
fire_at,
|
|
quiet_hours_start: parse_option_env::<u32>("HEARTBEAT_QUIET_START")?
|
|
.or(settings.heartbeat.quiet_hours_start)
|
|
.map(|h| {
|
|
if h > 23 {
|
|
return Err(ConfigError::InvalidValue {
|
|
key: "HEARTBEAT_QUIET_START".into(),
|
|
message: "must be 0-23".into(),
|
|
});
|
|
}
|
|
Ok(h)
|
|
})
|
|
.transpose()?,
|
|
quiet_hours_end: parse_option_env::<u32>("HEARTBEAT_QUIET_END")?
|
|
.or(settings.heartbeat.quiet_hours_end)
|
|
.map(|h| {
|
|
if h > 23 {
|
|
return Err(ConfigError::InvalidValue {
|
|
key: "HEARTBEAT_QUIET_END".into(),
|
|
message: "must be 0-23".into(),
|
|
});
|
|
}
|
|
Ok(h)
|
|
})
|
|
.transpose()?,
|
|
timezone: {
|
|
let tz = optional_env("HEARTBEAT_TIMEZONE")?
|
|
.or_else(|| settings.heartbeat.timezone.clone());
|
|
if let Some(ref tz_str) = tz
|
|
&& crate::timezone::parse_timezone(tz_str).is_none()
|
|
{
|
|
return Err(ConfigError::InvalidValue {
|
|
key: "HEARTBEAT_TIMEZONE".into(),
|
|
message: format!("invalid IANA timezone: '{tz_str}'"),
|
|
});
|
|
}
|
|
tz
|
|
},
|
|
// Auto-detect multi-tenant mode from GATEWAY_USER_TOKENS presence,
|
|
// or allow explicit override via HEARTBEAT_MULTI_TENANT.
|
|
multi_tenant: parse_bool_env(
|
|
"HEARTBEAT_MULTI_TENANT",
|
|
optional_env("GATEWAY_USER_TOKENS")?.is_some(),
|
|
)?,
|
|
})
|
|
}
|
|
}
|
|
|
|
#[cfg(test)]
|
|
mod tests {
|
|
use super::*;
|
|
|
|
#[test]
|
|
fn test_quiet_hours_settings_fallback() {
|
|
// When env vars are not set, settings values should be used
|
|
let mut settings = Settings::default();
|
|
settings.heartbeat.quiet_hours_start = Some(22);
|
|
settings.heartbeat.quiet_hours_end = Some(6);
|
|
|
|
let config = HeartbeatConfig::resolve(&settings).expect("resolve");
|
|
assert_eq!(config.quiet_hours_start, Some(22));
|
|
assert_eq!(config.quiet_hours_end, Some(6));
|
|
}
|
|
|
|
#[test]
|
|
fn test_quiet_hours_rejects_invalid_hour() {
|
|
let mut settings = Settings::default();
|
|
settings.heartbeat.quiet_hours_start = Some(24);
|
|
|
|
let result = HeartbeatConfig::resolve(&settings);
|
|
assert!(result.is_err());
|
|
}
|
|
|
|
#[test]
|
|
fn test_quiet_hours_accepts_boundary_values() {
|
|
let mut settings = Settings::default();
|
|
settings.heartbeat.quiet_hours_start = Some(0);
|
|
settings.heartbeat.quiet_hours_end = Some(23);
|
|
|
|
let config = HeartbeatConfig::resolve(&settings).expect("resolve");
|
|
assert_eq!(config.quiet_hours_start, Some(0));
|
|
assert_eq!(config.quiet_hours_end, Some(23));
|
|
}
|
|
|
|
#[test]
|
|
fn test_heartbeat_timezone_rejects_invalid() {
|
|
let mut settings = Settings::default();
|
|
settings.heartbeat.timezone = Some("Fake/Zone".to_string());
|
|
|
|
let result = HeartbeatConfig::resolve(&settings);
|
|
assert!(result.is_err(), "invalid IANA timezone should be rejected");
|
|
}
|
|
|
|
#[test]
|
|
fn test_heartbeat_timezone_accepts_valid() {
|
|
let mut settings = Settings::default();
|
|
settings.heartbeat.timezone = Some("America/New_York".to_string());
|
|
|
|
let config = HeartbeatConfig::resolve(&settings).expect("resolve");
|
|
assert_eq!(config.timezone.as_deref(), Some("America/New_York"));
|
|
}
|
|
}
|