Files
optimclaw/src/channels/web/ws.rs
T
553c306c52 feat: full image support across all channels (#725)
* feat: full image support across all channels

End-to-end image handling: upload, generation, analysis, editing, and
rendering across web gateway, HTTP webhook, WASM (Telegram/Slack), and
REPL channels. Builds on the attachment infrastructure from #596 and
draws inspiration from PR #641's image pipeline approach — credit to
that PR's author for the sentinel JSON pattern and base64-in-JSON
upload design.

Key changes:
- Image upload in web UI (file picker, paste, preview strip)
- Image generation tool (FLUX/DALL-E via /v1/images/generations)
- Image edit tool (multipart /v1/images/edits with fallback)
- Image analysis tool (vision model for workspace images)
- Model detection utilities (image_models.rs, vision_models.rs)
- Sentinel JSON detection in dispatcher for generated image rendering
- StatusUpdate::ImageGenerated → SSE/WS/REPL/WASM broadcast
- HTTP webhook attachment support (base64, 5MB/file, 10MB total)
- WASM channel image download (Telegram via file API, Slack via host HTTP)
- Tool registration wiring in app.rs

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: address PR #725 review comments (16 issues)

- SecretString for API keys in all image tools (image_gen, image_edit, image_analyze)
- Binary image read via tokio::fs::read instead of DB-backed workspace.read()
- Replace Arc<Workspace> with Option<PathBuf> base_dir (workspace has no filesystem API)
- ApprovalRequirement::UnlessAutoApproved for cost-sensitive image tools
- Scope sentinel detection to image_generate/image_edit tool names only
- Skip ToolResult preview broadcast for image sentinels (avoids multi-MB base64 in SSE)
- Extract shared media_type_from_path() to builtin/mod.rs
- Rename fallback_chat_edit → fallback_generate with tracing::warn
- Increase gateway body limit from 1MB to 10MB for image uploads
- Increase webhook body limit to 15MB (base64 overhead)
- Log warning on invalid base64 in images_to_attachments
- Client-side image size limits (5MB/file, 5 images max) in app.js
- aria-label on attach button for accessibility
- Update body_too_large test for new 10MB limit

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: add Slack file size check before download (PR review item #15)

Skip downloading files larger than 20 MB in the Slack WASM channel to
avoid excessive memory use and slow downloads in the WASM runtime.
Logs a warning when a file is skipped. Also bumps channel versions
for Slack and Telegram (prior branch changes).

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* style: cargo fmt

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix(security): add path validation and approval requirement to image tools

Add sandbox path validation via validate_path() to both ImageAnalyzeTool
and ImageEditTool to prevent path traversal attacks that could exfiltrate
arbitrary files through external vision/edit APIs. Also fix
ImageAnalyzeTool::requires_approval to return UnlessAutoApproved,
consistent with ImageEditTool and ImageGenerateTool.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: post-download size guards and empty data_url sentinel check

- Slack: add post-download size check on actual bytes when metadata
  size_bytes is absent, preventing bypass of the 20MB limit
- Telegram: add 20MB download size limit (matching Slack) enforced
  in download_telegram_file() after receiving response bytes
- Dispatcher: skip broadcasting ImageGenerated SSE event when
  data_url is empty from unwrap_or_default(), log warning instead

Closes correctness issues #3, #4, #5 from PR #725 review.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: use mime_guess for media type detection, add alt attrs and media_type validation

- Replace hardcoded media type mapping with mime_guess crate (already in deps)
- Add alt attributes to img elements in web UI for accessibility
- Validate media_type starts with "image/" in images_to_attachments()
- Update bmp test assertion to match mime_guess behavior

Co-Authored-By: Claude Opus 4.6 <[email protected]>

---------

Co-authored-by: Claude Opus 4.6 <[email protected]>
Co-authored-by: Zaki <[email protected]>
2026-03-09 03:41:27 +00:00

519 lines
18 KiB
Rust

//! WebSocket handler for bidirectional client communication.
//!
//! Provides the same event stream as SSE but also accepts incoming messages
//! (chat, approvals) over a single persistent connection.
//!
//! ```text
//! Client ──── WS frame: {"type":"message","content":"hello"} ──► Agent Loop
//! ◄─── WS frame: {"type":"event","event_type":"response","data":{...}} ── Broadcast
//! ──── WS frame: {"type":"ping"} ──────────────────────────────────────►
//! ◄─── WS frame: {"type":"pong"} ──────────────────────────────────────
//! ```
use std::sync::Arc;
use std::sync::atomic::{AtomicU64, Ordering};
use axum::extract::ws::{Message, WebSocket};
use futures::{SinkExt, StreamExt};
use tokio::sync::mpsc;
use uuid::Uuid;
use crate::agent::submission::Submission;
use crate::channels::IncomingMessage;
use crate::channels::web::server::GatewayState;
use crate::channels::web::types::{WsClientMessage, WsServerMessage};
/// Tracks active WebSocket connections.
pub struct WsConnectionTracker {
count: AtomicU64,
}
impl WsConnectionTracker {
pub fn new() -> Self {
Self {
count: AtomicU64::new(0),
}
}
pub fn connection_count(&self) -> u64 {
self.count.load(Ordering::Relaxed)
}
fn increment(&self) {
self.count.fetch_add(1, Ordering::Relaxed);
}
fn decrement(&self) {
self.count.fetch_sub(1, Ordering::Relaxed);
}
}
impl Default for WsConnectionTracker {
fn default() -> Self {
Self::new()
}
}
/// Handle an upgraded WebSocket connection.
///
/// Spawns two tasks:
/// - **sender**: forwards broadcast events to the WebSocket client
/// - **receiver**: reads client frames and routes them to the agent
///
/// When either task ends (client disconnect or broadcast closed), both are
/// cleaned up.
pub async fn handle_ws_connection(socket: WebSocket, state: Arc<GatewayState>) {
let (mut ws_sink, mut ws_stream) = socket.split();
// Track connection
if let Some(ref tracker) = state.ws_tracker {
tracker.increment();
}
let tracker_for_drop = state.ws_tracker.clone();
// Subscribe to broadcast events (same source as SSE).
// Reject if we've hit the connection limit.
let Some(raw_stream) = state.sse.subscribe_raw() else {
tracing::warn!("WebSocket rejected: too many connections");
// Decrement the WS tracker we already incremented above.
if let Some(ref tracker) = tracker_for_drop {
tracker.decrement();
}
return;
};
let mut event_stream = Box::pin(raw_stream);
// Channel for the sender task to receive messages from both
// the broadcast stream and any direct sends (like Pong)
let (direct_tx, mut direct_rx) = mpsc::channel::<WsServerMessage>(64);
// Sender task: forward broadcast events + direct messages to WS client
let sender_handle = tokio::spawn(async move {
loop {
let msg = tokio::select! {
event = event_stream.next() => {
match event {
Some(sse_event) => WsServerMessage::from_sse_event(&sse_event),
None => break, // Broadcast channel closed
}
}
direct = direct_rx.recv() => {
match direct {
Some(msg) => msg,
None => break, // Direct channel closed
}
}
};
let json = match serde_json::to_string(&msg) {
Ok(j) => j,
Err(_) => continue,
};
if ws_sink.send(Message::Text(json.into())).await.is_err() {
break; // Client disconnected
}
}
});
// Receiver task: read client frames and route to agent
let user_id = state.user_id.clone();
while let Some(Ok(frame)) = ws_stream.next().await {
match frame {
Message::Text(text) => {
let parsed: Result<WsClientMessage, _> = serde_json::from_str(&text);
match parsed {
Ok(client_msg) => {
handle_client_message(client_msg, &state, &user_id, &direct_tx).await;
}
Err(e) => {
let _ = direct_tx
.send(WsServerMessage::Error {
message: format!("Invalid message: {}", e),
})
.await;
}
}
}
Message::Close(_) => break,
// Ignore binary, ping/pong (axum handles protocol-level pings)
_ => {}
}
}
// Clean up: abort sender, decrement counter
sender_handle.abort();
if let Some(ref tracker) = tracker_for_drop {
tracker.decrement();
}
}
/// Route a parsed client message to the appropriate handler.
async fn handle_client_message(
msg: WsClientMessage,
state: &GatewayState,
user_id: &str,
direct_tx: &mpsc::Sender<WsServerMessage>,
) {
match msg {
WsClientMessage::Message {
content,
thread_id,
timezone,
images,
} => {
let mut incoming = IncomingMessage::new("gateway", user_id, &content);
if let Some(ref tz) = timezone {
incoming = incoming.with_timezone(tz);
}
if let Some(ref tid) = thread_id {
incoming = incoming.with_thread(tid);
}
// Convert uploaded images to IncomingAttachments
if !images.is_empty() {
let attachments = crate::channels::web::server::images_to_attachments(&images);
incoming = incoming.with_attachments(attachments);
}
let tx_guard = state.msg_tx.read().await;
if let Some(ref tx) = *tx_guard {
if tx.send(incoming).await.is_err() {
let _ = direct_tx
.send(WsServerMessage::Error {
message: "Channel closed".to_string(),
})
.await;
}
} else {
let _ = direct_tx
.send(WsServerMessage::Error {
message: "Channel not started".to_string(),
})
.await;
}
}
WsClientMessage::Approval {
request_id,
action,
thread_id,
} => {
let (approved, always) = match action.as_str() {
"approve" => (true, false),
"always" => (true, true),
"deny" => (false, false),
other => {
let _ = direct_tx
.send(WsServerMessage::Error {
message: format!("Unknown approval action: {}", other),
})
.await;
return;
}
};
let request_uuid = match Uuid::parse_str(&request_id) {
Ok(id) => id,
Err(_) => {
let _ = direct_tx
.send(WsServerMessage::Error {
message: "Invalid request_id (expected UUID)".to_string(),
})
.await;
return;
}
};
let approval = Submission::ExecApproval {
request_id: request_uuid,
approved,
always,
};
let content = match serde_json::to_string(&approval) {
Ok(c) => c,
Err(e) => {
let _ = direct_tx
.send(WsServerMessage::Error {
message: format!("Failed to serialize approval: {}", e),
})
.await;
return;
}
};
let mut msg = IncomingMessage::new("gateway", user_id, content);
if let Some(ref tid) = thread_id {
msg = msg.with_thread(tid);
}
let tx_guard = state.msg_tx.read().await;
if let Some(ref tx) = *tx_guard {
let _ = tx.send(msg).await;
}
}
WsClientMessage::AuthToken {
extension_name,
token,
} => {
if let Some(ref ext_mgr) = state.extension_manager {
match ext_mgr.auth(&extension_name, Some(&token)).await {
Ok(result) if result.is_authenticated() => {
let msg = match ext_mgr.activate(&extension_name).await {
Ok(r) => format!(
"{} authenticated ({} tools loaded)",
extension_name,
r.tools_loaded.len()
),
Err(e) => format!(
"{} authenticated but activation failed: {}",
extension_name, e
),
};
crate::channels::web::server::clear_auth_mode(state).await;
state
.sse
.broadcast(crate::channels::web::types::SseEvent::AuthCompleted {
extension_name,
success: true,
message: msg,
});
}
Ok(result) => {
state
.sse
.broadcast(crate::channels::web::types::SseEvent::AuthRequired {
extension_name,
instructions: result.instructions().map(String::from),
auth_url: result.auth_url().map(String::from),
setup_url: result.setup_url().map(String::from),
});
}
Err(e) => {
let _ = direct_tx
.send(WsServerMessage::Error {
message: format!("Auth failed: {}", e),
})
.await;
}
}
} else {
let _ = direct_tx
.send(WsServerMessage::Error {
message: "Extension manager not available".to_string(),
})
.await;
}
}
WsClientMessage::AuthCancel { .. } => {
crate::channels::web::server::clear_auth_mode(state).await;
}
WsClientMessage::Ping => {
let _ = direct_tx.send(WsServerMessage::Pong).await;
}
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_ws_connection_tracker() {
let tracker = WsConnectionTracker::new();
assert_eq!(tracker.connection_count(), 0);
tracker.increment();
assert_eq!(tracker.connection_count(), 1);
tracker.increment();
assert_eq!(tracker.connection_count(), 2);
tracker.decrement();
assert_eq!(tracker.connection_count(), 1);
tracker.decrement();
assert_eq!(tracker.connection_count(), 0);
}
#[test]
fn test_ws_connection_tracker_default() {
let tracker = WsConnectionTracker::default();
assert_eq!(tracker.connection_count(), 0);
}
#[tokio::test]
async fn test_handle_client_message_ping() {
// Ping should produce a Pong on the direct channel
let (direct_tx, mut direct_rx) = mpsc::channel(16);
let state = make_test_state(None).await;
handle_client_message(WsClientMessage::Ping, &state, "user1", &direct_tx).await;
let response = direct_rx.recv().await.unwrap();
assert!(matches!(response, WsServerMessage::Pong));
}
#[tokio::test]
async fn test_handle_client_message_sends_to_agent() {
// A Message should be forwarded to the agent's msg_tx
let (agent_tx, mut agent_rx) = mpsc::channel(16);
let state = make_test_state(Some(agent_tx)).await;
let (direct_tx, _direct_rx) = mpsc::channel(16);
handle_client_message(
WsClientMessage::Message {
content: "hello agent".to_string(),
thread_id: Some("t1".to_string()),
timezone: None,
images: Vec::new(),
},
&state,
"user1",
&direct_tx,
)
.await;
let incoming = agent_rx.recv().await.unwrap();
assert_eq!(incoming.content, "hello agent");
assert_eq!(incoming.thread_id.as_deref(), Some("t1"));
assert_eq!(incoming.channel, "gateway");
assert_eq!(incoming.user_id, "user1");
}
#[tokio::test]
async fn test_handle_client_message_no_channel() {
// When msg_tx is None, should send an error back
let state = make_test_state(None).await;
let (direct_tx, mut direct_rx) = mpsc::channel(16);
handle_client_message(
WsClientMessage::Message {
content: "hello".to_string(),
thread_id: None,
timezone: None,
images: Vec::new(),
},
&state,
"user1",
&direct_tx,
)
.await;
let response = direct_rx.recv().await.unwrap();
match response {
WsServerMessage::Error { message } => {
assert!(message.contains("not started"));
}
_ => panic!("Expected Error variant"),
}
}
#[tokio::test]
async fn test_handle_client_approval_approve() {
let (agent_tx, mut agent_rx) = mpsc::channel(16);
let state = make_test_state(Some(agent_tx)).await;
let (direct_tx, _direct_rx) = mpsc::channel(16);
let request_id = Uuid::new_v4();
handle_client_message(
WsClientMessage::Approval {
request_id: request_id.to_string(),
action: "approve".to_string(),
thread_id: Some("thread-42".to_string()),
},
&state,
"user1",
&direct_tx,
)
.await;
let incoming = agent_rx.recv().await.unwrap();
// The content should be a serialized ExecApproval
assert!(incoming.content.contains("ExecApproval"));
// Thread should be forwarded onto the IncomingMessage.
assert_eq!(incoming.thread_id.as_deref(), Some("thread-42"));
}
#[tokio::test]
async fn test_handle_client_approval_invalid_action() {
let state = make_test_state(None).await;
let (direct_tx, mut direct_rx) = mpsc::channel(16);
handle_client_message(
WsClientMessage::Approval {
request_id: Uuid::new_v4().to_string(),
action: "maybe".to_string(),
thread_id: None,
},
&state,
"user1",
&direct_tx,
)
.await;
let response = direct_rx.recv().await.unwrap();
match response {
WsServerMessage::Error { message } => {
assert!(message.contains("Unknown approval action"));
}
_ => panic!("Expected Error variant"),
}
}
#[tokio::test]
async fn test_handle_client_approval_invalid_uuid() {
let state = make_test_state(None).await;
let (direct_tx, mut direct_rx) = mpsc::channel(16);
handle_client_message(
WsClientMessage::Approval {
request_id: "not-a-uuid".to_string(),
action: "approve".to_string(),
thread_id: None,
},
&state,
"user1",
&direct_tx,
)
.await;
let response = direct_rx.recv().await.unwrap();
match response {
WsServerMessage::Error { message } => {
assert!(message.contains("Invalid request_id"));
}
_ => panic!("Expected Error variant"),
}
}
/// Helper to create a GatewayState for testing.
async fn make_test_state(msg_tx: Option<mpsc::Sender<IncomingMessage>>) -> GatewayState {
use crate::channels::web::sse::SseManager;
GatewayState {
msg_tx: tokio::sync::RwLock::new(msg_tx),
sse: SseManager::new(),
workspace: None,
session_manager: None,
log_broadcaster: None,
log_level_handle: None,
extension_manager: None,
tool_registry: None,
store: None,
job_manager: None,
prompt_queue: None,
scheduler: None,
user_id: "test".to_string(),
shutdown_tx: tokio::sync::RwLock::new(None),
ws_tracker: Some(Arc::new(WsConnectionTracker::new())),
llm_provider: None,
skill_registry: None,
skill_catalog: None,
chat_rate_limiter: crate::channels::web::server::RateLimiter::new(30, 60),
registry_entries: Vec::new(),
cost_guard: None,
routine_engine: Arc::new(tokio::sync::RwLock::new(None)),
startup_time: std::time::Instant::now(),
}
}
}