Files
optimclaw/src/channels/web/sse.rs
T
b441ebec02 feat: multi-tenant auth with per-user workspace isolation (#1118)
* feat: multi-tenant auth with per-user scoping

Multi-user authentication and authorization for IronClaw gateway:
- Token-based auth mapping tokens to user IDs via GATEWAY_USER_TOKENS
- Per-user SSE broadcast scoping
- Per-user rate limiting with poisoned lock recovery
- Handler auth and ownership checks for jobs, settings, routines
- Extension secrets scoped per-user
- Chat handlers use authenticated identity
- Reverse proxy deployment documentation
- Comprehensive integration tests for auth, SSE, rate limiting, and job isolation

* fix: scope memory tools per-user in multi-tenant mode

Memory tools (search, write, read, tree) held a single workspace
created at startup with GATEWAY_USER_ID. In multi-tenant mode, all
users' tool calls searched the default user's scope.

Add WorkspaceResolver trait that resolves workspaces per-request using
JobContext.user_id. In single-user mode, returns the startup workspace.
In multi-tenant mode (GATEWAY_USER_TOKENS configured), creates and
caches per-user workspaces on demand.

Includes regression tests for workspace resolution and user isolation.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* fix: comprehensive multi-tenant isolation audit

Address all review findings from @serrrfirat plus 7 additional gaps
found via full security audit:

Reviewer findings (5):
- WorkspacePool now applies search config, memory layers, embedding
  cache, identity read scopes, and global config scopes (was bare)
- jobs_summary_handler uses per-user queries instead of global counters
- jobs_prompt_handler restructured to not 404 agent jobs + ownership check
- jobs_restart_handler agent branch now verifies user ownership
- agent_job_summary_for_user added to Database trait + both backends

Audit findings (7):
- Delete dead handlers/memory.rs (stale copies with no auth)
- Add AuthenticatedUser to logs_events, logs_level_get, logs_level_set
- Add AuthenticatedUser to extensions_tools_handler, gateway_status_handler
- Add auth + ownership checks to all 6 routines handlers
- Add auth to all 4 skills handlers with audit logging on mutations
- Scope extension setup SSE broadcast to user (broadcast_for_user)
- Fix pre-existing test compilation errors in extensions/manager.rs

17 new multi-tenant isolation tests covering:
- WorkspacePool config propagation and scope merging
- Jobs handler per-user isolation (summary, restart, prompt, cancel)
- Routines handler auth enforcement and cross-user rejection
- Auth middleware enforcement on logs, skills, status endpoints

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* fix: second-pass multi-tenant audit — scope SSE broadcasts, DB queries, dead handlers

Second audit pass applying learned patterns across the codebase:

- OAuth callback SSE broadcasts now use broadcast_for_user (lines 773, 912)
- jobs_list_handler uses list_agent_jobs_for_user instead of fetching
  all users' jobs and filtering in Rust
- list_agent_jobs_for_user added to Database trait + postgres + libsql
- Dead handler files (extensions.rs, static_files.rs) hardened with
  AuthenticatedUser to prevent auth regression if migrated

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* fix: address review findings — token hashing, broadcast scoping, error handling

Security fixes:
- Hash tokens with SHA-256 at construction time so authentication
  compares fixed-size 32-byte digests, eliminating length-oracle
  timing leaks
- Scope auth SSE broadcasts per-user in chat_auth_token_handler —
  AuthRequired/AuthCompleted events were leaking across tenants
- Propagate DB errors in restart handlers instead of silently
  swallowing via `if let Ok(Some(...))` pattern

Code quality:
- Log SSE serialization failures instead of silently producing empty
  strings via unwrap_or_default()
- Remove dead `pub type AuthState = MultiAuthState` alias
- Replace `.unwrap()` with `Arc::clone(db)` in app.rs multi-tenant
  workspace setup (db is guaranteed Some in context, but unwrap
  violates project convention)
- Fix telegram setup test to inject UserIdentity into request
  extensions (handler now requires AuthenticatedUser)
- Add safety comments on test-only expect/unwrap calls for CI
- Apply cargo fmt to fix pre-existing formatting

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* fix: address review findings — unify workspace pool, fix SSE regression, cache job owners

- Unify WorkspacePool and PerUserWorkspaceResolver: WorkspacePool now
  implements WorkspaceResolver, eliminating duplicate per-user workspace
  construction logic. app.rs uses WorkspacePool directly.

- Fix sse_tx: None scheduler regression: change scheduler/worker SSE
  broadcasting from broadcast::Sender<SseEvent> to Arc<SseManager>,
  restoring SSE event delivery for scheduled agent jobs.

- Cache job owner in orchestrator: add job_owner_cache to
  OrchestratorState so job_event_handler avoids a DB round-trip on
  every event after the first per job.

- Deduplicate ext_user_id computation in main.rs.

- Remove unused _gateway_state variable.

- Fix pre-existing test: first_token() returns None in multi-user mode
  by design; align test assertion.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* style: fix formatting in app.rs

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* refactor: extract memory handlers back into handlers/memory.rs

Move memory API handlers out of server.rs into their own module,
consistent with how jobs, routines, and skills handlers are organized.
The resolve_workspace() helper moves with them since it is only used
by memory handlers.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>
Co-authored-by: [email protected] <[email protected]>
2026-03-23 20:50:05 -07:00

398 lines
14 KiB
Rust

//! SSE connection manager for broadcasting events to browser tabs.
use std::convert::Infallible;
use std::sync::Arc;
use std::sync::atomic::{AtomicU64, Ordering};
use std::time::Duration;
use axum::response::sse::{Event, KeepAlive, Sse};
use futures::Stream;
use tokio::sync::broadcast;
use tokio_stream::StreamExt;
use tokio_stream::wrappers::BroadcastStream;
use crate::channels::web::types::SseEvent;
/// Maximum number of concurrent SSE/WebSocket connections.
/// Prevents resource exhaustion from connection flooding.
const MAX_CONNECTIONS: u64 = 100;
/// Envelope for broadcast events: carries an optional user scope.
///
/// `user_id = None` means the event is global (e.g. Heartbeat) and delivered
/// to all subscribers. `user_id = Some(id)` means the event is only delivered
/// to subscribers that match that user_id.
#[derive(Debug, Clone)]
pub(crate) struct ScopedEvent {
pub(crate) user_id: Option<String>,
pub(crate) event: SseEvent,
}
/// Manages SSE broadcast to all connected browser tabs.
///
/// In multi-user mode, events are scoped by user_id so that each subscriber
/// only receives events intended for their user (plus global events like
/// Heartbeat). In single-user mode, all events are delivered to all subscribers
/// (backwards compatible).
pub struct SseManager {
tx: broadcast::Sender<ScopedEvent>,
connection_count: Arc<AtomicU64>,
max_connections: u64,
}
impl SseManager {
/// Create a new SSE manager.
pub fn new() -> Self {
// Buffer 256 events; slow clients will miss events (acceptable for SSE with reconnect)
let (tx, _) = broadcast::channel(256);
Self {
tx,
connection_count: Arc::new(AtomicU64::new(0)),
max_connections: MAX_CONNECTIONS,
}
}
/// Create an SSE manager that reuses an existing broadcast sender.
///
/// This preserves the broadcast channel across `rebuild_state` calls so
/// that sender handles captured by other components remain valid.
///
/// **Important:** The connection counter is reset to zero. This method must
/// only be called before the server starts accepting connections (i.e.,
/// during startup wiring). Calling it after connections are established
/// will break connection tracking and allow exceeding `MAX_CONNECTIONS`.
pub(crate) fn from_sender(tx: broadcast::Sender<ScopedEvent>) -> Self {
Self {
tx,
connection_count: Arc::new(AtomicU64::new(0)),
max_connections: MAX_CONNECTIONS,
}
}
/// Get a clone of the broadcast sender for use by other components.
pub(crate) fn sender(&self) -> broadcast::Sender<ScopedEvent> {
self.tx.clone()
}
/// Broadcast an event to all connected clients (global/unscoped).
pub fn broadcast(&self, event: SseEvent) {
let _ = self.tx.send(ScopedEvent {
user_id: None,
event,
});
}
/// Broadcast an event scoped to a specific user.
///
/// Only subscribers for this user_id (or unscoped subscribers) will
/// receive the event.
pub fn broadcast_for_user(&self, user_id: &str, event: SseEvent) {
let _ = self.tx.send(ScopedEvent {
user_id: Some(user_id.to_string()),
event,
});
}
/// Get current number of active connections.
pub fn connection_count(&self) -> u64 {
self.connection_count.load(Ordering::Relaxed)
}
/// Create a raw broadcast subscription for non-SSE consumers (e.g. WebSocket).
///
/// When `user_id` is `Some`, only events scoped to that user (or global
/// events) are delivered. When `None`, all events are delivered (single-user
/// backwards compatibility).
///
/// Returns `None` if the maximum connection limit has been reached.
pub fn subscribe_raw(
&self,
user_id: Option<String>,
) -> Option<impl Stream<Item = SseEvent> + Send + 'static + use<>> {
// Atomically increment only if below the limit. This prevents
// concurrent callers from overshooting max_connections.
let counter = Arc::clone(&self.connection_count);
let max = self.max_connections;
counter
.fetch_update(Ordering::Relaxed, Ordering::Relaxed, |current| {
if current < max {
Some(current + 1)
} else {
None
}
})
.ok()?;
let rx = self.tx.subscribe();
let stream = BroadcastStream::new(rx).filter_map(move |result| match result {
Ok(scoped) => {
// Global events (user_id=None) always pass through.
// Scoped events only pass if the subscriber matches (or subscriber is unscoped).
match (&user_id, &scoped.user_id) {
(_, None) => Some(scoped.event), // global -> all
(None, _) => Some(scoped.event), // unscoped subscriber -> all
(Some(sub), Some(ev)) if sub == ev => Some(scoped.event), // match
_ => None, // different user -> skip
}
}
Err(_) => None,
});
Some(CountedStream {
inner: stream,
counter,
})
}
/// Create a new SSE stream for a client connection.
///
/// When `user_id` is `Some`, only events for that user (or global events)
/// are delivered. When `None`, all events are delivered.
///
/// Returns `None` if the maximum connection limit has been reached.
pub fn subscribe(
&self,
user_id: Option<String>,
) -> Option<Sse<impl Stream<Item = Result<Event, Infallible>> + Send + 'static + use<>>> {
// Atomically increment only if below the limit.
let counter = Arc::clone(&self.connection_count);
let max = self.max_connections;
counter
.fetch_update(Ordering::Relaxed, Ordering::Relaxed, |current| {
if current < max {
Some(current + 1)
} else {
None
}
})
.ok()?;
let rx = self.tx.subscribe();
let stream = BroadcastStream::new(rx)
.filter_map(move |result| match result {
Ok(scoped) => match (&user_id, &scoped.user_id) {
(_, None) => Some(scoped.event),
(None, _) => Some(scoped.event),
(Some(sub), Some(ev)) if sub == ev => Some(scoped.event),
_ => None,
},
Err(_) => None,
})
.filter_map(|event| {
let data = match serde_json::to_string(&event) {
Ok(s) => s,
Err(e) => {
tracing::warn!("Failed to serialize SSE event: {}", e);
return None;
}
};
let event_type = match &event {
SseEvent::Response { .. } => "response",
SseEvent::Thinking { .. } => "thinking",
SseEvent::ToolStarted { .. } => "tool_started",
SseEvent::ToolCompleted { .. } => "tool_completed",
SseEvent::ToolResult { .. } => "tool_result",
SseEvent::StreamChunk { .. } => "stream_chunk",
SseEvent::Status { .. } => "status",
SseEvent::ApprovalNeeded { .. } => "approval_needed",
SseEvent::AuthRequired { .. } => "auth_required",
SseEvent::AuthCompleted { .. } => "auth_completed",
SseEvent::Error { .. } => "error",
SseEvent::JobStarted { .. } => "job_started",
SseEvent::JobMessage { .. } => "job_message",
SseEvent::JobToolUse { .. } => "job_tool_use",
SseEvent::JobToolResult { .. } => "job_tool_result",
SseEvent::JobStatus { .. } => "job_status",
SseEvent::JobResult { .. } => "job_result",
SseEvent::Heartbeat => "heartbeat",
SseEvent::ImageGenerated { .. } => "image_generated",
SseEvent::Suggestions { .. } => "suggestions",
SseEvent::TurnCost { .. } => "turn_cost",
SseEvent::ExtensionStatus { .. } => "extension_status",
};
Some(Ok(Event::default().event(event_type).data(data)))
});
// Wrap in a stream that decrements on drop
let counted_stream = CountedStream {
inner: stream,
counter,
};
Some(
Sse::new(counted_stream)
.keep_alive(KeepAlive::new().interval(Duration::from_secs(30)).text("")),
)
}
}
impl Default for SseManager {
fn default() -> Self {
Self::new()
}
}
/// Stream wrapper that decrements connection count on drop.
///
/// When the SSE client disconnects, this stream is dropped
/// and the counter is decremented.
struct CountedStream<S> {
inner: S,
counter: Arc<AtomicU64>,
}
impl<S: Stream + Unpin> Stream for CountedStream<S> {
type Item = S::Item;
fn poll_next(
mut self: std::pin::Pin<&mut Self>,
cx: &mut std::task::Context<'_>,
) -> std::task::Poll<Option<Self::Item>> {
std::pin::Pin::new(&mut self.inner).poll_next(cx)
}
}
impl<S> Drop for CountedStream<S> {
fn drop(&mut self) {
self.counter.fetch_sub(1, Ordering::Relaxed);
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_sse_manager_creation() {
let manager = SseManager::new();
assert_eq!(manager.connection_count(), 0);
}
#[test]
fn test_broadcast_without_receivers() {
let manager = SseManager::new();
// Should not panic even with no receivers
manager.broadcast(SseEvent::Heartbeat);
}
#[tokio::test]
async fn test_broadcast_to_receiver() {
let manager = SseManager::new();
let mut stream = Box::pin(manager.subscribe_raw(None).expect("should subscribe"));
manager.broadcast(SseEvent::Status {
message: "test".to_string(),
thread_id: None,
});
let event = stream.next().await.unwrap();
match event {
SseEvent::Status { message, .. } => assert_eq!(message, "test"),
_ => panic!("unexpected event type"),
}
}
#[tokio::test]
async fn test_subscribe_raw_receives_events() {
let manager = SseManager::new();
let mut stream = Box::pin(manager.subscribe_raw(None).expect("should subscribe"));
assert_eq!(manager.connection_count(), 1);
manager.broadcast(SseEvent::Thinking {
message: "working".to_string(),
thread_id: None,
});
let event = stream.next().await.unwrap();
match event {
SseEvent::Thinking { message, .. } => assert_eq!(message, "working"),
_ => panic!("Expected Thinking event"),
}
}
#[tokio::test]
async fn test_subscribe_raw_decrements_on_drop() {
let manager = SseManager::new();
{
let _stream = Box::pin(manager.subscribe_raw(None).expect("should subscribe"));
assert_eq!(manager.connection_count(), 1);
}
// Stream dropped, counter should decrement
assert_eq!(manager.connection_count(), 0);
}
#[tokio::test]
async fn test_subscribe_raw_multiple_subscribers() {
let manager = SseManager::new();
let mut s1 = Box::pin(manager.subscribe_raw(None).expect("should subscribe"));
let mut s2 = Box::pin(manager.subscribe_raw(None).expect("should subscribe"));
assert_eq!(manager.connection_count(), 2);
manager.broadcast(SseEvent::Heartbeat);
let e1 = s1.next().await.unwrap();
let e2 = s2.next().await.unwrap();
assert!(matches!(e1, SseEvent::Heartbeat));
assert!(matches!(e2, SseEvent::Heartbeat));
drop(s1);
assert_eq!(manager.connection_count(), 1);
drop(s2);
assert_eq!(manager.connection_count(), 0);
}
#[tokio::test]
async fn test_subscribe_raw_rejects_over_limit() {
let mut manager = SseManager::new();
manager.max_connections = 2; // Low limit for testing
let _s1 = Box::pin(manager.subscribe_raw(None).expect("first should succeed"));
let _s2 = Box::pin(manager.subscribe_raw(None).expect("second should succeed"));
assert_eq!(manager.connection_count(), 2);
// Third should be rejected
assert!(manager.subscribe_raw(None).is_none());
assert!(manager.subscribe(None).is_none());
}
#[tokio::test]
async fn test_scoped_events_filtered_by_user() {
let manager = SseManager::new();
let mut alice = Box::pin(
manager
.subscribe_raw(Some("alice".to_string()))
.expect("subscribe"),
);
let mut bob = Box::pin(
manager
.subscribe_raw(Some("bob".to_string()))
.expect("subscribe"),
);
// Send event scoped to alice
manager.broadcast_for_user(
"alice",
SseEvent::Status {
message: "alice only".to_string(),
thread_id: None,
},
);
// Send global event
manager.broadcast(SseEvent::Heartbeat);
// Alice gets her scoped event
let e = alice.next().await.unwrap();
assert!(matches!(e, SseEvent::Status { .. }));
// Alice also gets the global heartbeat
let e = alice.next().await.unwrap();
assert!(matches!(e, SseEvent::Heartbeat));
// Bob only gets the global heartbeat (alice's event was filtered)
let e = bob.next().await.unwrap(); // safety: test-only
assert!(matches!(e, SseEvent::Heartbeat)); // safety: test assertion
}
}