Files
optimclaw/src/db/libsql/jobs.rs
T
b441ebec02 feat: multi-tenant auth with per-user workspace isolation (#1118)
* feat: multi-tenant auth with per-user scoping

Multi-user authentication and authorization for IronClaw gateway:
- Token-based auth mapping tokens to user IDs via GATEWAY_USER_TOKENS
- Per-user SSE broadcast scoping
- Per-user rate limiting with poisoned lock recovery
- Handler auth and ownership checks for jobs, settings, routines
- Extension secrets scoped per-user
- Chat handlers use authenticated identity
- Reverse proxy deployment documentation
- Comprehensive integration tests for auth, SSE, rate limiting, and job isolation

* fix: scope memory tools per-user in multi-tenant mode

Memory tools (search, write, read, tree) held a single workspace
created at startup with GATEWAY_USER_ID. In multi-tenant mode, all
users' tool calls searched the default user's scope.

Add WorkspaceResolver trait that resolves workspaces per-request using
JobContext.user_id. In single-user mode, returns the startup workspace.
In multi-tenant mode (GATEWAY_USER_TOKENS configured), creates and
caches per-user workspaces on demand.

Includes regression tests for workspace resolution and user isolation.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* fix: comprehensive multi-tenant isolation audit

Address all review findings from @serrrfirat plus 7 additional gaps
found via full security audit:

Reviewer findings (5):
- WorkspacePool now applies search config, memory layers, embedding
  cache, identity read scopes, and global config scopes (was bare)
- jobs_summary_handler uses per-user queries instead of global counters
- jobs_prompt_handler restructured to not 404 agent jobs + ownership check
- jobs_restart_handler agent branch now verifies user ownership
- agent_job_summary_for_user added to Database trait + both backends

Audit findings (7):
- Delete dead handlers/memory.rs (stale copies with no auth)
- Add AuthenticatedUser to logs_events, logs_level_get, logs_level_set
- Add AuthenticatedUser to extensions_tools_handler, gateway_status_handler
- Add auth + ownership checks to all 6 routines handlers
- Add auth to all 4 skills handlers with audit logging on mutations
- Scope extension setup SSE broadcast to user (broadcast_for_user)
- Fix pre-existing test compilation errors in extensions/manager.rs

17 new multi-tenant isolation tests covering:
- WorkspacePool config propagation and scope merging
- Jobs handler per-user isolation (summary, restart, prompt, cancel)
- Routines handler auth enforcement and cross-user rejection
- Auth middleware enforcement on logs, skills, status endpoints

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* fix: second-pass multi-tenant audit — scope SSE broadcasts, DB queries, dead handlers

Second audit pass applying learned patterns across the codebase:

- OAuth callback SSE broadcasts now use broadcast_for_user (lines 773, 912)
- jobs_list_handler uses list_agent_jobs_for_user instead of fetching
  all users' jobs and filtering in Rust
- list_agent_jobs_for_user added to Database trait + postgres + libsql
- Dead handler files (extensions.rs, static_files.rs) hardened with
  AuthenticatedUser to prevent auth regression if migrated

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* fix: address review findings — token hashing, broadcast scoping, error handling

Security fixes:
- Hash tokens with SHA-256 at construction time so authentication
  compares fixed-size 32-byte digests, eliminating length-oracle
  timing leaks
- Scope auth SSE broadcasts per-user in chat_auth_token_handler —
  AuthRequired/AuthCompleted events were leaking across tenants
- Propagate DB errors in restart handlers instead of silently
  swallowing via `if let Ok(Some(...))` pattern

Code quality:
- Log SSE serialization failures instead of silently producing empty
  strings via unwrap_or_default()
- Remove dead `pub type AuthState = MultiAuthState` alias
- Replace `.unwrap()` with `Arc::clone(db)` in app.rs multi-tenant
  workspace setup (db is guaranteed Some in context, but unwrap
  violates project convention)
- Fix telegram setup test to inject UserIdentity into request
  extensions (handler now requires AuthenticatedUser)
- Add safety comments on test-only expect/unwrap calls for CI
- Apply cargo fmt to fix pre-existing formatting

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* fix: address review findings — unify workspace pool, fix SSE regression, cache job owners

- Unify WorkspacePool and PerUserWorkspaceResolver: WorkspacePool now
  implements WorkspaceResolver, eliminating duplicate per-user workspace
  construction logic. app.rs uses WorkspacePool directly.

- Fix sse_tx: None scheduler regression: change scheduler/worker SSE
  broadcasting from broadcast::Sender<SseEvent> to Arc<SseManager>,
  restoring SSE event delivery for scheduled agent jobs.

- Cache job owner in orchestrator: add job_owner_cache to
  OrchestratorState so job_event_handler avoids a DB round-trip on
  every event after the first per job.

- Deduplicate ext_user_id computation in main.rs.

- Remove unused _gateway_state variable.

- Fix pre-existing test: first_token() returns None in multi-user mode
  by design; align test assertion.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* style: fix formatting in app.rs

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* refactor: extract memory handlers back into handlers/memory.rs

Move memory API handlers out of server.rs into their own module,
consistent with how jobs, routines, and skills handlers are organized.
The resolve_workspace() helper moves with them since it is only used
by memory handlers.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>
Co-authored-by: [email protected] <[email protected]>
2026-03-23 20:50:05 -07:00

504 lines
19 KiB
Rust

//! Job-related JobStore implementation for LibSqlBackend.
use async_trait::async_trait;
use libsql::params;
use rust_decimal::Decimal;
use uuid::Uuid;
use super::{
LibSqlBackend, fmt_opt_ts, fmt_ts, get_decimal, get_i64, get_json, get_opt_decimal,
get_opt_text, get_opt_ts, get_text, get_ts, opt_text, opt_text_owned, parse_job_state,
};
use crate::context::{ActionRecord, JobContext, JobState};
use crate::db::JobStore;
use crate::error::DatabaseError;
use crate::history::{AgentJobRecord, AgentJobSummary, LlmCallRecord};
use chrono::Utc;
#[async_trait]
impl JobStore for LibSqlBackend {
async fn save_job(&self, ctx: &JobContext) -> Result<(), DatabaseError> {
let conn = self.connect().await?;
let status = ctx.state.to_string();
let estimated_time_secs = ctx.estimated_duration.map(|d| d.as_secs() as i64);
conn
.execute(
r#"
INSERT INTO agent_jobs (
id, conversation_id, title, description, category, status, source,
user_id,
budget_amount, budget_token, bid_amount, estimated_cost, estimated_time_secs,
actual_cost, repair_attempts, max_tokens, total_tokens_used,
created_at, started_at, completed_at
) VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10, ?11, ?12, ?13, ?14, ?15, ?16, ?17, ?18, ?19, ?20)
ON CONFLICT (id) DO UPDATE SET
title = excluded.title,
description = excluded.description,
category = excluded.category,
status = excluded.status,
user_id = excluded.user_id,
estimated_cost = excluded.estimated_cost,
estimated_time_secs = excluded.estimated_time_secs,
actual_cost = excluded.actual_cost,
repair_attempts = excluded.repair_attempts,
max_tokens = excluded.max_tokens,
total_tokens_used = excluded.total_tokens_used,
started_at = excluded.started_at,
completed_at = excluded.completed_at
"#,
params![
ctx.job_id.to_string(),
opt_text_owned(ctx.conversation_id.map(|id| id.to_string())),
ctx.title.as_str(),
ctx.description.as_str(),
opt_text(ctx.category.as_deref()),
status,
"direct",
ctx.user_id.as_str(),
opt_text_owned(ctx.budget.map(|d| d.to_string())),
opt_text(ctx.budget_token.as_deref()),
opt_text_owned(ctx.bid_amount.map(|d| d.to_string())),
opt_text_owned(ctx.estimated_cost.map(|d| d.to_string())),
estimated_time_secs,
ctx.actual_cost.to_string(),
ctx.repair_attempts as i64,
ctx.max_tokens as i64,
ctx.total_tokens_used as i64,
fmt_ts(&ctx.created_at),
fmt_opt_ts(&ctx.started_at),
fmt_opt_ts(&ctx.completed_at),
],
)
.await
.map_err(|e| DatabaseError::Query(e.to_string()))?;
Ok(())
}
async fn get_job(&self, id: Uuid) -> Result<Option<JobContext>, DatabaseError> {
let conn = self.connect().await?;
let mut rows = conn
.query(
r#"
SELECT id, conversation_id, title, description, category, status, user_id,
budget_amount, budget_token, bid_amount, estimated_cost, estimated_time_secs,
actual_cost, repair_attempts, max_tokens, total_tokens_used,
created_at, started_at, completed_at
FROM agent_jobs WHERE id = ?1
"#,
params![id.to_string()],
)
.await
.map_err(|e| DatabaseError::Query(e.to_string()))?;
match rows
.next()
.await
.map_err(|e| DatabaseError::Query(e.to_string()))?
{
Some(row) => {
let status_str = get_text(&row, 5);
let state = parse_job_state(&status_str);
let estimated_time_secs: Option<i64> = row.get::<i64>(11).ok();
Ok(Some(JobContext {
job_id: get_text(&row, 0).parse().unwrap_or_default(),
state,
user_id: get_text(&row, 6),
requester_id: None,
conversation_id: get_opt_text(&row, 1).and_then(|s| s.parse().ok()),
title: get_text(&row, 2),
description: get_text(&row, 3),
category: get_opt_text(&row, 4),
budget: get_opt_decimal(&row, 7),
budget_token: get_opt_text(&row, 8),
bid_amount: get_opt_decimal(&row, 9),
estimated_cost: get_opt_decimal(&row, 10),
estimated_duration: estimated_time_secs
.map(|s| std::time::Duration::from_secs(s as u64)),
actual_cost: get_decimal(&row, 12),
max_tokens: get_i64(&row, 14) as u64,
total_tokens_used: get_i64(&row, 15) as u64,
repair_attempts: get_i64(&row, 13) as u32,
created_at: get_ts(&row, 16),
started_at: get_opt_ts(&row, 17),
completed_at: get_opt_ts(&row, 18),
transitions: Vec::new(),
metadata: serde_json::Value::Null,
extra_env: std::sync::Arc::new(std::collections::HashMap::new()),
http_interceptor: None,
tool_output_stash: std::sync::Arc::new(tokio::sync::RwLock::new(
std::collections::HashMap::new(),
)),
// TODO(#661): persist user_timezone in agent_jobs table so
// background/routine jobs retain the session's timezone context.
user_timezone: "UTC".to_string(),
}))
}
None => Ok(None),
}
}
async fn update_job_status(
&self,
id: Uuid,
status: JobState,
failure_reason: Option<&str>,
) -> Result<(), DatabaseError> {
let conn = self.connect().await?;
conn.execute(
"UPDATE agent_jobs SET status = ?2, failure_reason = ?3 WHERE id = ?1",
params![id.to_string(), status.to_string(), opt_text(failure_reason)],
)
.await
.map_err(|e| DatabaseError::Query(e.to_string()))?;
Ok(())
}
async fn mark_job_stuck(&self, id: Uuid) -> Result<(), DatabaseError> {
let conn = self.connect().await?;
let now = fmt_ts(&Utc::now());
conn.execute(
"UPDATE agent_jobs SET status = 'stuck', stuck_since = ?2 WHERE id = ?1",
params![id.to_string(), now],
)
.await
.map_err(|e| DatabaseError::Query(e.to_string()))?;
Ok(())
}
async fn get_stuck_jobs(&self) -> Result<Vec<Uuid>, DatabaseError> {
let conn = self.connect().await?;
let mut rows = conn
.query("SELECT id FROM agent_jobs WHERE status = 'stuck'", ())
.await
.map_err(|e| DatabaseError::Query(e.to_string()))?;
let mut ids = Vec::new();
while let Some(row) = rows
.next()
.await
.map_err(|e| DatabaseError::Query(e.to_string()))?
{
if let Ok(id_str) = row.get::<String>(0)
&& let Ok(id) = id_str.parse()
{
ids.push(id);
}
}
Ok(ids)
}
async fn list_agent_jobs(&self) -> Result<Vec<AgentJobRecord>, DatabaseError> {
let conn = self.connect().await?;
let mut rows = conn
.query(
r#"
SELECT id, title, status, user_id, failure_reason,
created_at, started_at, completed_at
FROM agent_jobs WHERE source = 'direct'
ORDER BY created_at DESC
"#,
(),
)
.await
.map_err(|e| DatabaseError::Query(e.to_string()))?;
let mut jobs = Vec::new();
while let Some(row) = rows
.next()
.await
.map_err(|e| DatabaseError::Query(e.to_string()))?
{
let id_str = get_text(&row, 0);
let Ok(id) = id_str.parse() else {
tracing::warn!("Skipping agent job with invalid UUID: {}", id_str);
continue;
};
jobs.push(AgentJobRecord {
id,
title: get_text(&row, 1),
status: get_text(&row, 2),
user_id: get_text(&row, 3),
failure_reason: get_opt_text(&row, 4),
created_at: get_ts(&row, 5),
started_at: get_opt_ts(&row, 6),
completed_at: get_opt_ts(&row, 7),
});
}
Ok(jobs)
}
async fn list_agent_jobs_for_user(
&self,
user_id: &str,
) -> Result<Vec<AgentJobRecord>, DatabaseError> {
let conn = self.connect().await?;
let mut rows = conn
.query(
r#"
SELECT id, title, status, user_id, failure_reason,
created_at, started_at, completed_at
FROM agent_jobs WHERE source = 'direct' AND user_id = ?1
ORDER BY created_at DESC
"#,
params![user_id],
)
.await
.map_err(|e| DatabaseError::Query(e.to_string()))?;
let mut jobs = Vec::new();
while let Some(row) = rows
.next()
.await
.map_err(|e| DatabaseError::Query(e.to_string()))?
{
let id_str = get_text(&row, 0);
let Ok(id) = id_str.parse() else {
tracing::warn!("Skipping agent job with invalid UUID: {}", id_str);
continue;
};
jobs.push(AgentJobRecord {
id,
title: get_text(&row, 1),
status: get_text(&row, 2),
user_id: get_text(&row, 3),
failure_reason: get_opt_text(&row, 4),
created_at: get_ts(&row, 5),
started_at: get_opt_ts(&row, 6),
completed_at: get_opt_ts(&row, 7),
});
}
Ok(jobs)
}
async fn get_agent_job_failure_reason(
&self,
id: Uuid,
) -> Result<Option<String>, DatabaseError> {
let conn = self.connect().await?;
let mut rows = conn
.query(
"SELECT failure_reason FROM agent_jobs WHERE id = ?1",
[id.to_string()],
)
.await
.map_err(|e| DatabaseError::Query(e.to_string()))?;
if let Some(row) = rows
.next()
.await
.map_err(|e| DatabaseError::Query(e.to_string()))?
{
Ok(get_opt_text(&row, 0))
} else {
Ok(None)
}
}
async fn agent_job_summary(&self) -> Result<AgentJobSummary, DatabaseError> {
let conn = self.connect().await?;
let mut rows = conn
.query(
"SELECT status, COUNT(*) as cnt FROM agent_jobs WHERE source = 'direct' GROUP BY status",
(),
)
.await
.map_err(|e| DatabaseError::Query(e.to_string()))?;
let mut summary = AgentJobSummary::default();
while let Some(row) = rows
.next()
.await
.map_err(|e| DatabaseError::Query(e.to_string()))?
{
let status = get_text(&row, 0);
let count = get_i64(&row, 1) as usize;
summary.add_count(&status, count);
}
Ok(summary)
}
async fn agent_job_summary_for_user(
&self,
user_id: &str,
) -> Result<AgentJobSummary, DatabaseError> {
let conn = self.connect().await?;
let mut rows = conn
.query(
"SELECT status, COUNT(*) as cnt FROM agent_jobs WHERE source = 'direct' AND user_id = ?1 GROUP BY status",
params![user_id],
)
.await
.map_err(|e| DatabaseError::Query(e.to_string()))?;
let mut summary = AgentJobSummary::default();
while let Some(row) = rows
.next()
.await
.map_err(|e| DatabaseError::Query(e.to_string()))?
{
let status = get_text(&row, 0);
let count = get_i64(&row, 1) as usize;
summary.add_count(&status, count);
}
Ok(summary)
}
async fn save_action(&self, job_id: Uuid, action: &ActionRecord) -> Result<(), DatabaseError> {
let conn = self.connect().await?;
let duration_ms = action.duration.as_millis() as i64;
let warnings_json = serde_json::to_string(&action.sanitization_warnings)
.map_err(|e| DatabaseError::Serialization(e.to_string()))?;
conn.execute(
r#"
INSERT INTO job_actions (
id, job_id, sequence_num, tool_name, input, output_raw, output_sanitized,
sanitization_warnings, cost, duration_ms, success, error_message, created_at
) VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10, ?11, ?12, ?13)
"#,
params![
action.id.to_string(),
job_id.to_string(),
action.sequence as i64,
action.tool_name.as_str(),
action.input.to_string(),
opt_text(action.output_raw.as_deref()),
opt_text_owned(action.output_sanitized.as_ref().map(|v| v.to_string())),
warnings_json,
opt_text_owned(action.cost.map(|d| d.to_string())),
duration_ms,
action.success as i64,
opt_text(action.error.as_deref()),
fmt_ts(&action.executed_at),
],
)
.await
.map_err(|e| DatabaseError::Query(e.to_string()))?;
Ok(())
}
async fn get_job_actions(&self, job_id: Uuid) -> Result<Vec<ActionRecord>, DatabaseError> {
let conn = self.connect().await?;
let mut rows = conn
.query(
r#"
SELECT id, sequence_num, tool_name, input, output_raw, output_sanitized,
sanitization_warnings, cost, duration_ms, success, error_message, created_at
FROM job_actions WHERE job_id = ?1 ORDER BY sequence_num
"#,
params![job_id.to_string()],
)
.await
.map_err(|e| DatabaseError::Query(e.to_string()))?;
let mut actions = Vec::new();
while let Some(row) = rows
.next()
.await
.map_err(|e| DatabaseError::Query(e.to_string()))?
{
let warnings: Vec<String> =
serde_json::from_str(&get_text(&row, 6)).unwrap_or_default();
actions.push(ActionRecord {
id: get_text(&row, 0).parse().unwrap_or_default(),
sequence: get_i64(&row, 1) as u32,
tool_name: get_text(&row, 2),
input: get_json(&row, 3),
output_raw: get_opt_text(&row, 4),
output_sanitized: get_opt_text(&row, 5).and_then(|s| serde_json::from_str(&s).ok()),
sanitization_warnings: warnings,
cost: get_opt_decimal(&row, 7),
duration: std::time::Duration::from_millis(get_i64(&row, 8) as u64),
success: get_i64(&row, 9) != 0,
error: get_opt_text(&row, 10),
executed_at: get_ts(&row, 11),
});
}
Ok(actions)
}
async fn record_llm_call(&self, record: &LlmCallRecord<'_>) -> Result<Uuid, DatabaseError> {
let conn = self.connect().await?;
let id = Uuid::new_v4();
conn.execute(
r#"
INSERT INTO llm_calls (id, job_id, conversation_id, provider, model, input_tokens, output_tokens, cost, purpose)
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9)
"#,
params![
id.to_string(),
opt_text_owned(record.job_id.map(|id| id.to_string())),
opt_text_owned(record.conversation_id.map(|id| id.to_string())),
record.provider,
record.model,
record.input_tokens as i64,
record.output_tokens as i64,
record.cost.to_string(),
opt_text(record.purpose),
],
)
.await
.map_err(|e| DatabaseError::Query(e.to_string()))?;
Ok(id)
}
async fn save_estimation_snapshot(
&self,
job_id: Uuid,
category: &str,
tool_names: &[String],
estimated_cost: Decimal,
estimated_time_secs: i32,
estimated_value: Decimal,
) -> Result<Uuid, DatabaseError> {
let conn = self.connect().await?;
let id = Uuid::new_v4();
let tools_json = serde_json::to_string(tool_names)
.map_err(|e| DatabaseError::Serialization(e.to_string()))?;
conn.execute(
r#"
INSERT INTO estimation_snapshots (id, job_id, category, tool_names, estimated_cost, estimated_time_secs, estimated_value)
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7)
"#,
params![
id.to_string(),
job_id.to_string(),
category,
tools_json,
estimated_cost.to_string(),
estimated_time_secs as i64,
estimated_value.to_string(),
],
)
.await
.map_err(|e| DatabaseError::Query(e.to_string()))?;
Ok(id)
}
async fn update_estimation_actuals(
&self,
id: Uuid,
actual_cost: Decimal,
actual_time_secs: i32,
actual_value: Option<Decimal>,
) -> Result<(), DatabaseError> {
let conn = self.connect().await?;
conn.execute(
"UPDATE estimation_snapshots SET actual_cost = ?2, actual_time_secs = ?3, actual_value = ?4 WHERE id = ?1",
params![
id.to_string(),
actual_cost.to_string(),
actual_time_secs as i64,
actual_value.map(|d| d.to_string()).unwrap_or_default(),
],
)
.await
.map_err(|e| DatabaseError::Query(e.to_string()))?;
Ok(())
}
}