feat: multi-tenant auth with per-user workspace isolation (#1118)

* feat: multi-tenant auth with per-user scoping

Multi-user authentication and authorization for IronClaw gateway:
- Token-based auth mapping tokens to user IDs via GATEWAY_USER_TOKENS
- Per-user SSE broadcast scoping
- Per-user rate limiting with poisoned lock recovery
- Handler auth and ownership checks for jobs, settings, routines
- Extension secrets scoped per-user
- Chat handlers use authenticated identity
- Reverse proxy deployment documentation
- Comprehensive integration tests for auth, SSE, rate limiting, and job isolation

* fix: scope memory tools per-user in multi-tenant mode

Memory tools (search, write, read, tree) held a single workspace
created at startup with GATEWAY_USER_ID. In multi-tenant mode, all
users' tool calls searched the default user's scope.

Add WorkspaceResolver trait that resolves workspaces per-request using
JobContext.user_id. In single-user mode, returns the startup workspace.
In multi-tenant mode (GATEWAY_USER_TOKENS configured), creates and
caches per-user workspaces on demand.

Includes regression tests for workspace resolution and user isolation.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* fix: comprehensive multi-tenant isolation audit

Address all review findings from @serrrfirat plus 7 additional gaps
found via full security audit:

Reviewer findings (5):
- WorkspacePool now applies search config, memory layers, embedding
  cache, identity read scopes, and global config scopes (was bare)
- jobs_summary_handler uses per-user queries instead of global counters
- jobs_prompt_handler restructured to not 404 agent jobs + ownership check
- jobs_restart_handler agent branch now verifies user ownership
- agent_job_summary_for_user added to Database trait + both backends

Audit findings (7):
- Delete dead handlers/memory.rs (stale copies with no auth)
- Add AuthenticatedUser to logs_events, logs_level_get, logs_level_set
- Add AuthenticatedUser to extensions_tools_handler, gateway_status_handler
- Add auth + ownership checks to all 6 routines handlers
- Add auth to all 4 skills handlers with audit logging on mutations
- Scope extension setup SSE broadcast to user (broadcast_for_user)
- Fix pre-existing test compilation errors in extensions/manager.rs

17 new multi-tenant isolation tests covering:
- WorkspacePool config propagation and scope merging
- Jobs handler per-user isolation (summary, restart, prompt, cancel)
- Routines handler auth enforcement and cross-user rejection
- Auth middleware enforcement on logs, skills, status endpoints

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* fix: second-pass multi-tenant audit — scope SSE broadcasts, DB queries, dead handlers

Second audit pass applying learned patterns across the codebase:

- OAuth callback SSE broadcasts now use broadcast_for_user (lines 773, 912)
- jobs_list_handler uses list_agent_jobs_for_user instead of fetching
  all users' jobs and filtering in Rust
- list_agent_jobs_for_user added to Database trait + postgres + libsql
- Dead handler files (extensions.rs, static_files.rs) hardened with
  AuthenticatedUser to prevent auth regression if migrated

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* fix: address review findings — token hashing, broadcast scoping, error handling

Security fixes:
- Hash tokens with SHA-256 at construction time so authentication
  compares fixed-size 32-byte digests, eliminating length-oracle
  timing leaks
- Scope auth SSE broadcasts per-user in chat_auth_token_handler —
  AuthRequired/AuthCompleted events were leaking across tenants
- Propagate DB errors in restart handlers instead of silently
  swallowing via `if let Ok(Some(...))` pattern

Code quality:
- Log SSE serialization failures instead of silently producing empty
  strings via unwrap_or_default()
- Remove dead `pub type AuthState = MultiAuthState` alias
- Replace `.unwrap()` with `Arc::clone(db)` in app.rs multi-tenant
  workspace setup (db is guaranteed Some in context, but unwrap
  violates project convention)
- Fix telegram setup test to inject UserIdentity into request
  extensions (handler now requires AuthenticatedUser)
- Add safety comments on test-only expect/unwrap calls for CI
- Apply cargo fmt to fix pre-existing formatting

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* fix: address review findings — unify workspace pool, fix SSE regression, cache job owners

- Unify WorkspacePool and PerUserWorkspaceResolver: WorkspacePool now
  implements WorkspaceResolver, eliminating duplicate per-user workspace
  construction logic. app.rs uses WorkspacePool directly.

- Fix sse_tx: None scheduler regression: change scheduler/worker SSE
  broadcasting from broadcast::Sender<SseEvent> to Arc<SseManager>,
  restoring SSE event delivery for scheduled agent jobs.

- Cache job owner in orchestrator: add job_owner_cache to
  OrchestratorState so job_event_handler avoids a DB round-trip on
  every event after the first per job.

- Deduplicate ext_user_id computation in main.rs.

- Remove unused _gateway_state variable.

- Fix pre-existing test: first_token() returns None in multi-user mode
  by design; align test assertion.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* style: fix formatting in app.rs

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* refactor: extract memory handlers back into handlers/memory.rs

Move memory API handlers out of server.rs into their own module,
consistent with how jobs, routines, and skills handlers are organized.
The resolve_workspace() helper moves with them since it is only used
by memory handlers.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>
Co-authored-by: [email protected] <[email protected]>
This commit is contained in:
standardtoaster
2026-03-23 20:50:05 -07:00
committed by GitHub
co-authored by Claude Opus 4.6 [email protected] <[email protected]>
parent fa51b9f52d
commit b441ebec02
46 changed files with 5074 additions and 1204 deletions
+53 -6
View File
@@ -40,7 +40,8 @@ pub struct OrchestratorState {
pub job_manager: Arc<ContainerJobManager>,
pub token_store: TokenStore,
/// Broadcast channel for job events (consumed by the web gateway SSE).
pub job_event_tx: Option<broadcast::Sender<(Uuid, SseEvent)>>,
/// Tuple: (job_id, user_id, event).
pub job_event_tx: Option<broadcast::Sender<(Uuid, String, SseEvent)>>,
/// Buffered follow-up prompts for sandbox jobs, keyed by job_id.
pub prompt_queue: Arc<Mutex<HashMap<Uuid, VecDeque<PendingPrompt>>>>,
/// Database handle for persisting job events.
@@ -49,6 +50,9 @@ pub struct OrchestratorState {
pub secrets_store: Option<Arc<dyn SecretsStore + Send + Sync>>,
/// User ID for secret lookups (single-tenant, typically "default").
pub user_id: String,
/// In-memory cache of job_id → user_id for SSE scoping. Populated when
/// sandbox jobs are created, avoiding a DB round-trip on every job event.
pub job_owner_cache: Arc<std::sync::RwLock<HashMap<Uuid, String>>>,
}
/// The orchestrator's internal API server.
@@ -351,9 +355,45 @@ async fn job_event_handler(
},
};
// Broadcast via the channel (if configured)
// Broadcast via the channel (if configured).
// Look up the job owner from the in-memory cache (populated at job creation).
if let Some(ref tx) = state.job_event_tx {
let _ = tx.send((job_id, sse_event));
let cached_uid = state
.job_owner_cache
.read()
.unwrap_or_else(|e| e.into_inner())
.get(&job_id)
.cloned();
let user_id = match cached_uid {
Some(uid) => uid,
None => {
// Cache miss: fall back to DB lookup and populate cache.
let uid = match state.store.as_ref() {
Some(store) => store
.get_sandbox_job(job_id)
.await
.ok()
.flatten()
.map(|j| j.user_id),
None => None,
};
if let Some(ref uid) = uid {
state
.job_owner_cache
.write()
.unwrap_or_else(|e| e.into_inner())
.insert(job_id, uid.clone());
}
uid.unwrap_or_default()
}
};
if user_id.is_empty() {
let _ = tx.send((job_id, String::new(), sse_event));
} else {
let _ = tx.send((job_id, user_id, sse_event));
}
}
Ok(StatusCode::OK)
@@ -480,6 +520,7 @@ mod tests {
store: None,
secrets_store: None,
user_id: "default".to_string(),
job_owner_cache: Arc::new(std::sync::RwLock::new(HashMap::new())),
}
}
@@ -709,6 +750,7 @@ mod tests {
store: None,
secrets_store: Some(secrets_store),
user_id: "default".to_string(),
job_owner_cache: Arc::new(std::sync::RwLock::new(HashMap::new())),
};
let router = OrchestratorApi::router(state);
@@ -744,6 +786,7 @@ mod tests {
store: None,
secrets_store: None,
user_id: "default".to_string(),
job_owner_cache: Arc::new(std::sync::RwLock::new(HashMap::new())),
};
let job_id = Uuid::new_v4();
@@ -769,8 +812,10 @@ mod tests {
let resp = router.oneshot(req).await.unwrap();
assert_eq!(resp.status(), StatusCode::OK);
let (recv_id, event) = rx.recv().await.unwrap();
let (recv_id, recv_uid, event) = rx.recv().await.unwrap();
assert_eq!(recv_id, job_id);
// No store configured, so user_id falls back to empty string.
assert_eq!(recv_uid, "");
match event {
SseEvent::JobMessage {
job_id: jid,
@@ -799,6 +844,7 @@ mod tests {
store: None,
secrets_store: None,
user_id: "default".to_string(),
job_owner_cache: Arc::new(std::sync::RwLock::new(HashMap::new())),
};
let job_id = Uuid::new_v4();
@@ -824,7 +870,7 @@ mod tests {
let resp = router.oneshot(req).await.unwrap();
assert_eq!(resp.status(), StatusCode::OK);
let (_recv_id, event) = rx.recv().await.unwrap();
let (_recv_id, _recv_uid, event) = rx.recv().await.unwrap();
match event {
SseEvent::JobToolUse { tool_name, .. } => {
assert_eq!(tool_name, "shell");
@@ -847,6 +893,7 @@ mod tests {
store: None,
secrets_store: None,
user_id: "default".to_string(),
job_owner_cache: Arc::new(std::sync::RwLock::new(HashMap::new())),
};
let job_id = Uuid::new_v4();
@@ -869,7 +916,7 @@ mod tests {
let resp = router.oneshot(req).await.unwrap();
assert_eq!(resp.status(), StatusCode::OK);
let (_recv_id, event) = rx.recv().await.unwrap();
let (_recv_id, _recv_uid, event) = rx.recv().await.unwrap();
// Unknown event types fall through to JobStatus
assert!(matches!(event, SseEvent::JobStatus { .. }));
}
+2 -1
View File
@@ -63,7 +63,7 @@ fn resolve_orchestrator_port() -> u16 {
/// Result of orchestrator setup, containing all handles needed by the agent.
pub struct OrchestratorSetup {
pub container_job_manager: Option<Arc<ContainerJobManager>>,
pub job_event_tx: Option<broadcast::Sender<(Uuid, SseEvent)>>,
pub job_event_tx: Option<broadcast::Sender<(Uuid, String, SseEvent)>>,
pub prompt_queue: Arc<Mutex<HashMap<Uuid, VecDeque<api::PendingPrompt>>>>,
pub docker_status: crate::sandbox::DockerStatus,
}
@@ -134,6 +134,7 @@ pub async fn setup_orchestrator(
store: db.cloned(),
secrets_store: secrets_store.cloned(),
user_id: "default".to_string(),
job_owner_cache: Arc::new(std::sync::RwLock::new(std::collections::HashMap::new())),
};
tokio::spawn(async move {