mirror of
https://github.com/outbackdingo/optimclaw.git
synced 2026-09-01 00:59:33 +00:00
* feat: full image support across all channels End-to-end image handling: upload, generation, analysis, editing, and rendering across web gateway, HTTP webhook, WASM (Telegram/Slack), and REPL channels. Builds on the attachment infrastructure from #596 and draws inspiration from PR #641's image pipeline approach — credit to that PR's author for the sentinel JSON pattern and base64-in-JSON upload design. Key changes: - Image upload in web UI (file picker, paste, preview strip) - Image generation tool (FLUX/DALL-E via /v1/images/generations) - Image edit tool (multipart /v1/images/edits with fallback) - Image analysis tool (vision model for workspace images) - Model detection utilities (image_models.rs, vision_models.rs) - Sentinel JSON detection in dispatcher for generated image rendering - StatusUpdate::ImageGenerated → SSE/WS/REPL/WASM broadcast - HTTP webhook attachment support (base64, 5MB/file, 10MB total) - WASM channel image download (Telegram via file API, Slack via host HTTP) - Tool registration wiring in app.rs [skip-regression-check] Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: address PR #725 review comments (16 issues) - SecretString for API keys in all image tools (image_gen, image_edit, image_analyze) - Binary image read via tokio::fs::read instead of DB-backed workspace.read() - Replace Arc<Workspace> with Option<PathBuf> base_dir (workspace has no filesystem API) - ApprovalRequirement::UnlessAutoApproved for cost-sensitive image tools - Scope sentinel detection to image_generate/image_edit tool names only - Skip ToolResult preview broadcast for image sentinels (avoids multi-MB base64 in SSE) - Extract shared media_type_from_path() to builtin/mod.rs - Rename fallback_chat_edit → fallback_generate with tracing::warn - Increase gateway body limit from 1MB to 10MB for image uploads - Increase webhook body limit to 15MB (base64 overhead) - Log warning on invalid base64 in images_to_attachments - Client-side image size limits (5MB/file, 5 images max) in app.js - aria-label on attach button for accessibility - Update body_too_large test for new 10MB limit [skip-regression-check] Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: add Slack file size check before download (PR review item #15) Skip downloading files larger than 20 MB in the Slack WASM channel to avoid excessive memory use and slow downloads in the WASM runtime. Logs a warning when a file is skipped. Also bumps channel versions for Slack and Telegram (prior branch changes). [skip-regression-check] Co-Authored-By: Claude Opus 4.6 <[email protected]> * style: cargo fmt Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix(security): add path validation and approval requirement to image tools Add sandbox path validation via validate_path() to both ImageAnalyzeTool and ImageEditTool to prevent path traversal attacks that could exfiltrate arbitrary files through external vision/edit APIs. Also fix ImageAnalyzeTool::requires_approval to return UnlessAutoApproved, consistent with ImageEditTool and ImageGenerateTool. Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: post-download size guards and empty data_url sentinel check - Slack: add post-download size check on actual bytes when metadata size_bytes is absent, preventing bypass of the 20MB limit - Telegram: add 20MB download size limit (matching Slack) enforced in download_telegram_file() after receiving response bytes - Dispatcher: skip broadcasting ImageGenerated SSE event when data_url is empty from unwrap_or_default(), log warning instead Closes correctness issues #3, #4, #5 from PR #725 review. Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: use mime_guess for media type detection, add alt attrs and media_type validation - Replace hardcoded media type mapping with mime_guess crate (already in deps) - Add alt attributes to img elements in web UI for accessibility - Validate media_type starts with "image/" in images_to_attachments() - Update bmp test assertion to match mime_guess behavior Co-Authored-By: Claude Opus 4.6 <[email protected]> --------- Co-authored-by: Claude Opus 4.6 <[email protected]> Co-authored-by: Zaki <[email protected]>
61 lines
1.8 KiB
Rust
61 lines
1.8 KiB
Rust
//! Built-in tools that come with the agent.
|
|
|
|
mod echo;
|
|
pub mod extension_tools;
|
|
mod file;
|
|
mod http;
|
|
mod job;
|
|
mod json;
|
|
mod memory;
|
|
mod message;
|
|
pub mod path_utils;
|
|
mod restart;
|
|
pub mod routine;
|
|
pub mod secrets_tools;
|
|
pub(crate) mod shell;
|
|
pub mod skill_tools;
|
|
mod time;
|
|
|
|
pub use echo::EchoTool;
|
|
pub use extension_tools::{
|
|
ExtensionInfoTool, ToolActivateTool, ToolAuthTool, ToolInstallTool, ToolListTool,
|
|
ToolRemoveTool, ToolSearchTool, ToolUpgradeTool,
|
|
};
|
|
pub use file::{ApplyPatchTool, ListDirTool, ReadFileTool, WriteFileTool};
|
|
pub use http::HttpTool;
|
|
pub use job::{
|
|
CancelJobTool, CreateJobTool, JobEventsTool, JobPromptTool, JobStatusTool, ListJobsTool,
|
|
PromptQueue, SchedulerSlot,
|
|
};
|
|
pub use json::JsonTool;
|
|
pub use memory::{MemoryReadTool, MemorySearchTool, MemoryTreeTool, MemoryWriteTool};
|
|
pub use message::MessageTool;
|
|
pub use restart::RestartTool;
|
|
pub use routine::{
|
|
RoutineCreateTool, RoutineDeleteTool, RoutineFireTool, RoutineHistoryTool, RoutineListTool,
|
|
RoutineUpdateTool,
|
|
};
|
|
pub use secrets_tools::{SecretDeleteTool, SecretListTool};
|
|
pub use shell::ShellTool;
|
|
pub use skill_tools::{SkillInstallTool, SkillListTool, SkillRemoveTool, SkillSearchTool};
|
|
pub use time::TimeTool;
|
|
mod html_converter;
|
|
pub mod image_analyze;
|
|
pub mod image_edit;
|
|
pub mod image_gen;
|
|
|
|
pub use html_converter::convert_html_to_markdown;
|
|
pub use image_analyze::ImageAnalyzeTool;
|
|
pub use image_edit::ImageEditTool;
|
|
pub use image_gen::ImageGenerateTool;
|
|
|
|
/// Detect image media type from file extension via `mime_guess`.
|
|
/// Falls back to `image/jpeg` for unrecognized or non-image extensions.
|
|
pub(crate) fn media_type_from_path(path: &str) -> String {
|
|
mime_guess::from_path(path)
|
|
.first_raw()
|
|
.filter(|m| m.starts_with("image/"))
|
|
.unwrap_or("image/jpeg")
|
|
.to_string()
|
|
}
|