mirror of
https://github.com/outbackdingo/optimclaw.git
synced 2026-08-27 08:00:17 +00:00
* feat: full image support across all channels End-to-end image handling: upload, generation, analysis, editing, and rendering across web gateway, HTTP webhook, WASM (Telegram/Slack), and REPL channels. Builds on the attachment infrastructure from #596 and draws inspiration from PR #641's image pipeline approach — credit to that PR's author for the sentinel JSON pattern and base64-in-JSON upload design. Key changes: - Image upload in web UI (file picker, paste, preview strip) - Image generation tool (FLUX/DALL-E via /v1/images/generations) - Image edit tool (multipart /v1/images/edits with fallback) - Image analysis tool (vision model for workspace images) - Model detection utilities (image_models.rs, vision_models.rs) - Sentinel JSON detection in dispatcher for generated image rendering - StatusUpdate::ImageGenerated → SSE/WS/REPL/WASM broadcast - HTTP webhook attachment support (base64, 5MB/file, 10MB total) - WASM channel image download (Telegram via file API, Slack via host HTTP) - Tool registration wiring in app.rs [skip-regression-check] Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: address PR #725 review comments (16 issues) - SecretString for API keys in all image tools (image_gen, image_edit, image_analyze) - Binary image read via tokio::fs::read instead of DB-backed workspace.read() - Replace Arc<Workspace> with Option<PathBuf> base_dir (workspace has no filesystem API) - ApprovalRequirement::UnlessAutoApproved for cost-sensitive image tools - Scope sentinel detection to image_generate/image_edit tool names only - Skip ToolResult preview broadcast for image sentinels (avoids multi-MB base64 in SSE) - Extract shared media_type_from_path() to builtin/mod.rs - Rename fallback_chat_edit → fallback_generate with tracing::warn - Increase gateway body limit from 1MB to 10MB for image uploads - Increase webhook body limit to 15MB (base64 overhead) - Log warning on invalid base64 in images_to_attachments - Client-side image size limits (5MB/file, 5 images max) in app.js - aria-label on attach button for accessibility - Update body_too_large test for new 10MB limit [skip-regression-check] Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: add Slack file size check before download (PR review item #15) Skip downloading files larger than 20 MB in the Slack WASM channel to avoid excessive memory use and slow downloads in the WASM runtime. Logs a warning when a file is skipped. Also bumps channel versions for Slack and Telegram (prior branch changes). [skip-regression-check] Co-Authored-By: Claude Opus 4.6 <[email protected]> * style: cargo fmt Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix(security): add path validation and approval requirement to image tools Add sandbox path validation via validate_path() to both ImageAnalyzeTool and ImageEditTool to prevent path traversal attacks that could exfiltrate arbitrary files through external vision/edit APIs. Also fix ImageAnalyzeTool::requires_approval to return UnlessAutoApproved, consistent with ImageEditTool and ImageGenerateTool. Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: post-download size guards and empty data_url sentinel check - Slack: add post-download size check on actual bytes when metadata size_bytes is absent, preventing bypass of the 20MB limit - Telegram: add 20MB download size limit (matching Slack) enforced in download_telegram_file() after receiving response bytes - Dispatcher: skip broadcasting ImageGenerated SSE event when data_url is empty from unwrap_or_default(), log warning instead Closes correctness issues #3, #4, #5 from PR #725 review. Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: use mime_guess for media type detection, add alt attrs and media_type validation - Replace hardcoded media type mapping with mime_guess crate (already in deps) - Add alt attributes to img elements in web UI for accessibility - Validate media_type starts with "image/" in images_to_attachments() - Update bmp test assertion to match mime_guess behavior Co-Authored-By: Claude Opus 4.6 <[email protected]> --------- Co-authored-by: Claude Opus 4.6 <[email protected]> Co-authored-by: Zaki <[email protected]>
96 lines
2.5 KiB
Rust
96 lines
2.5 KiB
Rust
//! Image generation model detection utilities.
|
|
|
|
/// Known image generation model families.
|
|
const IMAGE_GEN_PATTERNS: &[&str] = &[
|
|
"flux",
|
|
"dall-e",
|
|
"dalle",
|
|
"stable-diffusion",
|
|
"sdxl",
|
|
"imagen",
|
|
"midjourney",
|
|
"ideogram",
|
|
"playground",
|
|
];
|
|
|
|
/// Check if a model name indicates an image generation model.
|
|
pub fn is_image_generation_model(model: &str) -> bool {
|
|
let lower = model.to_lowercase();
|
|
IMAGE_GEN_PATTERNS.iter().any(|p| lower.contains(p))
|
|
}
|
|
|
|
/// Suggest the best image generation model from a list of available models.
|
|
///
|
|
/// Priority: FLUX > DALL-E > Stable Diffusion > others.
|
|
pub fn suggest_image_model(models: &[String]) -> Option<&str> {
|
|
let priorities: &[&str] = &[
|
|
"flux",
|
|
"dall-e",
|
|
"dalle",
|
|
"stable-diffusion",
|
|
"sdxl",
|
|
"imagen",
|
|
];
|
|
for priority in priorities {
|
|
if let Some(model) = models.iter().find(|m| m.to_lowercase().contains(priority)) {
|
|
return Some(model);
|
|
}
|
|
}
|
|
// Fall back to any image gen model
|
|
models.iter().find_map(|m| {
|
|
if is_image_generation_model(m) {
|
|
Some(m.as_str())
|
|
} else {
|
|
None
|
|
}
|
|
})
|
|
}
|
|
|
|
#[cfg(test)]
|
|
mod tests {
|
|
use super::*;
|
|
|
|
#[test]
|
|
fn detects_flux_models() {
|
|
assert!(is_image_generation_model(
|
|
"black-forest-labs/FLUX.1-schnell"
|
|
));
|
|
assert!(is_image_generation_model("flux-pro"));
|
|
}
|
|
|
|
#[test]
|
|
fn detects_dalle_models() {
|
|
assert!(is_image_generation_model("dall-e-3"));
|
|
assert!(is_image_generation_model("dalle-3"));
|
|
}
|
|
|
|
#[test]
|
|
fn rejects_non_image_models() {
|
|
assert!(!is_image_generation_model("gpt-4o"));
|
|
assert!(!is_image_generation_model("claude-3-sonnet"));
|
|
assert!(!is_image_generation_model("llama-3.1-70b"));
|
|
}
|
|
|
|
#[test]
|
|
fn suggests_flux_first() {
|
|
let models = vec![
|
|
"gpt-4o".to_string(),
|
|
"dall-e-3".to_string(),
|
|
"flux-pro".to_string(),
|
|
];
|
|
assert_eq!(suggest_image_model(&models), Some("flux-pro"));
|
|
}
|
|
|
|
#[test]
|
|
fn suggests_dalle_without_flux() {
|
|
let models = vec!["gpt-4o".to_string(), "dall-e-3".to_string()];
|
|
assert_eq!(suggest_image_model(&models), Some("dall-e-3"));
|
|
}
|
|
|
|
#[test]
|
|
fn returns_none_when_no_image_models() {
|
|
let models = vec!["gpt-4o".to_string(), "claude-3-sonnet".to_string()];
|
|
assert_eq!(suggest_image_model(&models), None);
|
|
}
|
|
}
|