mirror of
https://github.com/outbackdingo/optimclaw.git
synced 2026-08-31 16:49:34 +00:00
* feat: multi-tenant auth with per-user scoping Multi-user authentication and authorization for IronClaw gateway: - Token-based auth mapping tokens to user IDs via GATEWAY_USER_TOKENS - Per-user SSE broadcast scoping - Per-user rate limiting with poisoned lock recovery - Handler auth and ownership checks for jobs, settings, routines - Extension secrets scoped per-user - Chat handlers use authenticated identity - Reverse proxy deployment documentation - Comprehensive integration tests for auth, SSE, rate limiting, and job isolation * fix: scope memory tools per-user in multi-tenant mode Memory tools (search, write, read, tree) held a single workspace created at startup with GATEWAY_USER_ID. In multi-tenant mode, all users' tool calls searched the default user's scope. Add WorkspaceResolver trait that resolves workspaces per-request using JobContext.user_id. In single-user mode, returns the startup workspace. In multi-tenant mode (GATEWAY_USER_TOKENS configured), creates and caches per-user workspaces on demand. Includes regression tests for workspace resolution and user isolation. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> * fix: comprehensive multi-tenant isolation audit Address all review findings from @serrrfirat plus 7 additional gaps found via full security audit: Reviewer findings (5): - WorkspacePool now applies search config, memory layers, embedding cache, identity read scopes, and global config scopes (was bare) - jobs_summary_handler uses per-user queries instead of global counters - jobs_prompt_handler restructured to not 404 agent jobs + ownership check - jobs_restart_handler agent branch now verifies user ownership - agent_job_summary_for_user added to Database trait + both backends Audit findings (7): - Delete dead handlers/memory.rs (stale copies with no auth) - Add AuthenticatedUser to logs_events, logs_level_get, logs_level_set - Add AuthenticatedUser to extensions_tools_handler, gateway_status_handler - Add auth + ownership checks to all 6 routines handlers - Add auth to all 4 skills handlers with audit logging on mutations - Scope extension setup SSE broadcast to user (broadcast_for_user) - Fix pre-existing test compilation errors in extensions/manager.rs 17 new multi-tenant isolation tests covering: - WorkspacePool config propagation and scope merging - Jobs handler per-user isolation (summary, restart, prompt, cancel) - Routines handler auth enforcement and cross-user rejection - Auth middleware enforcement on logs, skills, status endpoints Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> * fix: second-pass multi-tenant audit — scope SSE broadcasts, DB queries, dead handlers Second audit pass applying learned patterns across the codebase: - OAuth callback SSE broadcasts now use broadcast_for_user (lines 773, 912) - jobs_list_handler uses list_agent_jobs_for_user instead of fetching all users' jobs and filtering in Rust - list_agent_jobs_for_user added to Database trait + postgres + libsql - Dead handler files (extensions.rs, static_files.rs) hardened with AuthenticatedUser to prevent auth regression if migrated Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> * fix: address review findings — token hashing, broadcast scoping, error handling Security fixes: - Hash tokens with SHA-256 at construction time so authentication compares fixed-size 32-byte digests, eliminating length-oracle timing leaks - Scope auth SSE broadcasts per-user in chat_auth_token_handler — AuthRequired/AuthCompleted events were leaking across tenants - Propagate DB errors in restart handlers instead of silently swallowing via `if let Ok(Some(...))` pattern Code quality: - Log SSE serialization failures instead of silently producing empty strings via unwrap_or_default() - Remove dead `pub type AuthState = MultiAuthState` alias - Replace `.unwrap()` with `Arc::clone(db)` in app.rs multi-tenant workspace setup (db is guaranteed Some in context, but unwrap violates project convention) - Fix telegram setup test to inject UserIdentity into request extensions (handler now requires AuthenticatedUser) - Add safety comments on test-only expect/unwrap calls for CI - Apply cargo fmt to fix pre-existing formatting Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> * fix: address review findings — unify workspace pool, fix SSE regression, cache job owners - Unify WorkspacePool and PerUserWorkspaceResolver: WorkspacePool now implements WorkspaceResolver, eliminating duplicate per-user workspace construction logic. app.rs uses WorkspacePool directly. - Fix sse_tx: None scheduler regression: change scheduler/worker SSE broadcasting from broadcast::Sender<SseEvent> to Arc<SseManager>, restoring SSE event delivery for scheduled agent jobs. - Cache job owner in orchestrator: add job_owner_cache to OrchestratorState so job_event_handler avoids a DB round-trip on every event after the first per job. - Deduplicate ext_user_id computation in main.rs. - Remove unused _gateway_state variable. - Fix pre-existing test: first_token() returns None in multi-user mode by design; align test assertion. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> * style: fix formatting in app.rs Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> * refactor: extract memory handlers back into handlers/memory.rs Move memory API handlers out of server.rs into their own module, consistent with how jobs, routines, and skills handlers are organized. The resolve_workspace() helper moves with them since it is only used by memory handlers. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> --------- Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]> Co-authored-by: [email protected] <[email protected]>
398 lines
14 KiB
Rust
398 lines
14 KiB
Rust
//! SSE connection manager for broadcasting events to browser tabs.
|
|
|
|
use std::convert::Infallible;
|
|
use std::sync::Arc;
|
|
use std::sync::atomic::{AtomicU64, Ordering};
|
|
use std::time::Duration;
|
|
|
|
use axum::response::sse::{Event, KeepAlive, Sse};
|
|
use futures::Stream;
|
|
use tokio::sync::broadcast;
|
|
use tokio_stream::StreamExt;
|
|
use tokio_stream::wrappers::BroadcastStream;
|
|
|
|
use crate::channels::web::types::SseEvent;
|
|
|
|
/// Maximum number of concurrent SSE/WebSocket connections.
|
|
/// Prevents resource exhaustion from connection flooding.
|
|
const MAX_CONNECTIONS: u64 = 100;
|
|
|
|
/// Envelope for broadcast events: carries an optional user scope.
|
|
///
|
|
/// `user_id = None` means the event is global (e.g. Heartbeat) and delivered
|
|
/// to all subscribers. `user_id = Some(id)` means the event is only delivered
|
|
/// to subscribers that match that user_id.
|
|
#[derive(Debug, Clone)]
|
|
pub(crate) struct ScopedEvent {
|
|
pub(crate) user_id: Option<String>,
|
|
pub(crate) event: SseEvent,
|
|
}
|
|
|
|
/// Manages SSE broadcast to all connected browser tabs.
|
|
///
|
|
/// In multi-user mode, events are scoped by user_id so that each subscriber
|
|
/// only receives events intended for their user (plus global events like
|
|
/// Heartbeat). In single-user mode, all events are delivered to all subscribers
|
|
/// (backwards compatible).
|
|
pub struct SseManager {
|
|
tx: broadcast::Sender<ScopedEvent>,
|
|
connection_count: Arc<AtomicU64>,
|
|
max_connections: u64,
|
|
}
|
|
|
|
impl SseManager {
|
|
/// Create a new SSE manager.
|
|
pub fn new() -> Self {
|
|
// Buffer 256 events; slow clients will miss events (acceptable for SSE with reconnect)
|
|
let (tx, _) = broadcast::channel(256);
|
|
Self {
|
|
tx,
|
|
connection_count: Arc::new(AtomicU64::new(0)),
|
|
max_connections: MAX_CONNECTIONS,
|
|
}
|
|
}
|
|
|
|
/// Create an SSE manager that reuses an existing broadcast sender.
|
|
///
|
|
/// This preserves the broadcast channel across `rebuild_state` calls so
|
|
/// that sender handles captured by other components remain valid.
|
|
///
|
|
/// **Important:** The connection counter is reset to zero. This method must
|
|
/// only be called before the server starts accepting connections (i.e.,
|
|
/// during startup wiring). Calling it after connections are established
|
|
/// will break connection tracking and allow exceeding `MAX_CONNECTIONS`.
|
|
pub(crate) fn from_sender(tx: broadcast::Sender<ScopedEvent>) -> Self {
|
|
Self {
|
|
tx,
|
|
connection_count: Arc::new(AtomicU64::new(0)),
|
|
max_connections: MAX_CONNECTIONS,
|
|
}
|
|
}
|
|
|
|
/// Get a clone of the broadcast sender for use by other components.
|
|
pub(crate) fn sender(&self) -> broadcast::Sender<ScopedEvent> {
|
|
self.tx.clone()
|
|
}
|
|
|
|
/// Broadcast an event to all connected clients (global/unscoped).
|
|
pub fn broadcast(&self, event: SseEvent) {
|
|
let _ = self.tx.send(ScopedEvent {
|
|
user_id: None,
|
|
event,
|
|
});
|
|
}
|
|
|
|
/// Broadcast an event scoped to a specific user.
|
|
///
|
|
/// Only subscribers for this user_id (or unscoped subscribers) will
|
|
/// receive the event.
|
|
pub fn broadcast_for_user(&self, user_id: &str, event: SseEvent) {
|
|
let _ = self.tx.send(ScopedEvent {
|
|
user_id: Some(user_id.to_string()),
|
|
event,
|
|
});
|
|
}
|
|
|
|
/// Get current number of active connections.
|
|
pub fn connection_count(&self) -> u64 {
|
|
self.connection_count.load(Ordering::Relaxed)
|
|
}
|
|
|
|
/// Create a raw broadcast subscription for non-SSE consumers (e.g. WebSocket).
|
|
///
|
|
/// When `user_id` is `Some`, only events scoped to that user (or global
|
|
/// events) are delivered. When `None`, all events are delivered (single-user
|
|
/// backwards compatibility).
|
|
///
|
|
/// Returns `None` if the maximum connection limit has been reached.
|
|
pub fn subscribe_raw(
|
|
&self,
|
|
user_id: Option<String>,
|
|
) -> Option<impl Stream<Item = SseEvent> + Send + 'static + use<>> {
|
|
// Atomically increment only if below the limit. This prevents
|
|
// concurrent callers from overshooting max_connections.
|
|
let counter = Arc::clone(&self.connection_count);
|
|
let max = self.max_connections;
|
|
counter
|
|
.fetch_update(Ordering::Relaxed, Ordering::Relaxed, |current| {
|
|
if current < max {
|
|
Some(current + 1)
|
|
} else {
|
|
None
|
|
}
|
|
})
|
|
.ok()?;
|
|
let rx = self.tx.subscribe();
|
|
|
|
let stream = BroadcastStream::new(rx).filter_map(move |result| match result {
|
|
Ok(scoped) => {
|
|
// Global events (user_id=None) always pass through.
|
|
// Scoped events only pass if the subscriber matches (or subscriber is unscoped).
|
|
match (&user_id, &scoped.user_id) {
|
|
(_, None) => Some(scoped.event), // global -> all
|
|
(None, _) => Some(scoped.event), // unscoped subscriber -> all
|
|
(Some(sub), Some(ev)) if sub == ev => Some(scoped.event), // match
|
|
_ => None, // different user -> skip
|
|
}
|
|
}
|
|
Err(_) => None,
|
|
});
|
|
|
|
Some(CountedStream {
|
|
inner: stream,
|
|
counter,
|
|
})
|
|
}
|
|
|
|
/// Create a new SSE stream for a client connection.
|
|
///
|
|
/// When `user_id` is `Some`, only events for that user (or global events)
|
|
/// are delivered. When `None`, all events are delivered.
|
|
///
|
|
/// Returns `None` if the maximum connection limit has been reached.
|
|
pub fn subscribe(
|
|
&self,
|
|
user_id: Option<String>,
|
|
) -> Option<Sse<impl Stream<Item = Result<Event, Infallible>> + Send + 'static + use<>>> {
|
|
// Atomically increment only if below the limit.
|
|
let counter = Arc::clone(&self.connection_count);
|
|
let max = self.max_connections;
|
|
counter
|
|
.fetch_update(Ordering::Relaxed, Ordering::Relaxed, |current| {
|
|
if current < max {
|
|
Some(current + 1)
|
|
} else {
|
|
None
|
|
}
|
|
})
|
|
.ok()?;
|
|
let rx = self.tx.subscribe();
|
|
|
|
let stream = BroadcastStream::new(rx)
|
|
.filter_map(move |result| match result {
|
|
Ok(scoped) => match (&user_id, &scoped.user_id) {
|
|
(_, None) => Some(scoped.event),
|
|
(None, _) => Some(scoped.event),
|
|
(Some(sub), Some(ev)) if sub == ev => Some(scoped.event),
|
|
_ => None,
|
|
},
|
|
Err(_) => None,
|
|
})
|
|
.filter_map(|event| {
|
|
let data = match serde_json::to_string(&event) {
|
|
Ok(s) => s,
|
|
Err(e) => {
|
|
tracing::warn!("Failed to serialize SSE event: {}", e);
|
|
return None;
|
|
}
|
|
};
|
|
let event_type = match &event {
|
|
SseEvent::Response { .. } => "response",
|
|
SseEvent::Thinking { .. } => "thinking",
|
|
SseEvent::ToolStarted { .. } => "tool_started",
|
|
SseEvent::ToolCompleted { .. } => "tool_completed",
|
|
SseEvent::ToolResult { .. } => "tool_result",
|
|
SseEvent::StreamChunk { .. } => "stream_chunk",
|
|
SseEvent::Status { .. } => "status",
|
|
SseEvent::ApprovalNeeded { .. } => "approval_needed",
|
|
SseEvent::AuthRequired { .. } => "auth_required",
|
|
SseEvent::AuthCompleted { .. } => "auth_completed",
|
|
SseEvent::Error { .. } => "error",
|
|
SseEvent::JobStarted { .. } => "job_started",
|
|
SseEvent::JobMessage { .. } => "job_message",
|
|
SseEvent::JobToolUse { .. } => "job_tool_use",
|
|
SseEvent::JobToolResult { .. } => "job_tool_result",
|
|
SseEvent::JobStatus { .. } => "job_status",
|
|
SseEvent::JobResult { .. } => "job_result",
|
|
SseEvent::Heartbeat => "heartbeat",
|
|
SseEvent::ImageGenerated { .. } => "image_generated",
|
|
SseEvent::Suggestions { .. } => "suggestions",
|
|
SseEvent::TurnCost { .. } => "turn_cost",
|
|
SseEvent::ExtensionStatus { .. } => "extension_status",
|
|
};
|
|
Some(Ok(Event::default().event(event_type).data(data)))
|
|
});
|
|
|
|
// Wrap in a stream that decrements on drop
|
|
let counted_stream = CountedStream {
|
|
inner: stream,
|
|
counter,
|
|
};
|
|
|
|
Some(
|
|
Sse::new(counted_stream)
|
|
.keep_alive(KeepAlive::new().interval(Duration::from_secs(30)).text("")),
|
|
)
|
|
}
|
|
}
|
|
|
|
impl Default for SseManager {
|
|
fn default() -> Self {
|
|
Self::new()
|
|
}
|
|
}
|
|
|
|
/// Stream wrapper that decrements connection count on drop.
|
|
///
|
|
/// When the SSE client disconnects, this stream is dropped
|
|
/// and the counter is decremented.
|
|
struct CountedStream<S> {
|
|
inner: S,
|
|
counter: Arc<AtomicU64>,
|
|
}
|
|
|
|
impl<S: Stream + Unpin> Stream for CountedStream<S> {
|
|
type Item = S::Item;
|
|
|
|
fn poll_next(
|
|
mut self: std::pin::Pin<&mut Self>,
|
|
cx: &mut std::task::Context<'_>,
|
|
) -> std::task::Poll<Option<Self::Item>> {
|
|
std::pin::Pin::new(&mut self.inner).poll_next(cx)
|
|
}
|
|
}
|
|
|
|
impl<S> Drop for CountedStream<S> {
|
|
fn drop(&mut self) {
|
|
self.counter.fetch_sub(1, Ordering::Relaxed);
|
|
}
|
|
}
|
|
|
|
#[cfg(test)]
|
|
mod tests {
|
|
use super::*;
|
|
|
|
#[test]
|
|
fn test_sse_manager_creation() {
|
|
let manager = SseManager::new();
|
|
assert_eq!(manager.connection_count(), 0);
|
|
}
|
|
|
|
#[test]
|
|
fn test_broadcast_without_receivers() {
|
|
let manager = SseManager::new();
|
|
// Should not panic even with no receivers
|
|
manager.broadcast(SseEvent::Heartbeat);
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn test_broadcast_to_receiver() {
|
|
let manager = SseManager::new();
|
|
let mut stream = Box::pin(manager.subscribe_raw(None).expect("should subscribe"));
|
|
|
|
manager.broadcast(SseEvent::Status {
|
|
message: "test".to_string(),
|
|
thread_id: None,
|
|
});
|
|
|
|
let event = stream.next().await.unwrap();
|
|
match event {
|
|
SseEvent::Status { message, .. } => assert_eq!(message, "test"),
|
|
_ => panic!("unexpected event type"),
|
|
}
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn test_subscribe_raw_receives_events() {
|
|
let manager = SseManager::new();
|
|
let mut stream = Box::pin(manager.subscribe_raw(None).expect("should subscribe"));
|
|
|
|
assert_eq!(manager.connection_count(), 1);
|
|
|
|
manager.broadcast(SseEvent::Thinking {
|
|
message: "working".to_string(),
|
|
thread_id: None,
|
|
});
|
|
|
|
let event = stream.next().await.unwrap();
|
|
match event {
|
|
SseEvent::Thinking { message, .. } => assert_eq!(message, "working"),
|
|
_ => panic!("Expected Thinking event"),
|
|
}
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn test_subscribe_raw_decrements_on_drop() {
|
|
let manager = SseManager::new();
|
|
{
|
|
let _stream = Box::pin(manager.subscribe_raw(None).expect("should subscribe"));
|
|
assert_eq!(manager.connection_count(), 1);
|
|
}
|
|
// Stream dropped, counter should decrement
|
|
assert_eq!(manager.connection_count(), 0);
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn test_subscribe_raw_multiple_subscribers() {
|
|
let manager = SseManager::new();
|
|
let mut s1 = Box::pin(manager.subscribe_raw(None).expect("should subscribe"));
|
|
let mut s2 = Box::pin(manager.subscribe_raw(None).expect("should subscribe"));
|
|
assert_eq!(manager.connection_count(), 2);
|
|
|
|
manager.broadcast(SseEvent::Heartbeat);
|
|
|
|
let e1 = s1.next().await.unwrap();
|
|
let e2 = s2.next().await.unwrap();
|
|
assert!(matches!(e1, SseEvent::Heartbeat));
|
|
assert!(matches!(e2, SseEvent::Heartbeat));
|
|
|
|
drop(s1);
|
|
assert_eq!(manager.connection_count(), 1);
|
|
drop(s2);
|
|
assert_eq!(manager.connection_count(), 0);
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn test_subscribe_raw_rejects_over_limit() {
|
|
let mut manager = SseManager::new();
|
|
manager.max_connections = 2; // Low limit for testing
|
|
|
|
let _s1 = Box::pin(manager.subscribe_raw(None).expect("first should succeed"));
|
|
let _s2 = Box::pin(manager.subscribe_raw(None).expect("second should succeed"));
|
|
assert_eq!(manager.connection_count(), 2);
|
|
|
|
// Third should be rejected
|
|
assert!(manager.subscribe_raw(None).is_none());
|
|
assert!(manager.subscribe(None).is_none());
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn test_scoped_events_filtered_by_user() {
|
|
let manager = SseManager::new();
|
|
let mut alice = Box::pin(
|
|
manager
|
|
.subscribe_raw(Some("alice".to_string()))
|
|
.expect("subscribe"),
|
|
);
|
|
let mut bob = Box::pin(
|
|
manager
|
|
.subscribe_raw(Some("bob".to_string()))
|
|
.expect("subscribe"),
|
|
);
|
|
|
|
// Send event scoped to alice
|
|
manager.broadcast_for_user(
|
|
"alice",
|
|
SseEvent::Status {
|
|
message: "alice only".to_string(),
|
|
thread_id: None,
|
|
},
|
|
);
|
|
|
|
// Send global event
|
|
manager.broadcast(SseEvent::Heartbeat);
|
|
|
|
// Alice gets her scoped event
|
|
let e = alice.next().await.unwrap();
|
|
assert!(matches!(e, SseEvent::Status { .. }));
|
|
|
|
// Alice also gets the global heartbeat
|
|
let e = alice.next().await.unwrap();
|
|
assert!(matches!(e, SseEvent::Heartbeat));
|
|
|
|
// Bob only gets the global heartbeat (alice's event was filtered)
|
|
let e = bob.next().await.unwrap(); // safety: test-only
|
|
assert!(matches!(e, SseEvent::Heartbeat)); // safety: test assertion
|
|
}
|
|
}
|