Files
optimclaw/src/llm/image_models.rs
T
553c306c52 feat: full image support across all channels (#725)
* feat: full image support across all channels

End-to-end image handling: upload, generation, analysis, editing, and
rendering across web gateway, HTTP webhook, WASM (Telegram/Slack), and
REPL channels. Builds on the attachment infrastructure from #596 and
draws inspiration from PR #641's image pipeline approach — credit to
that PR's author for the sentinel JSON pattern and base64-in-JSON
upload design.

Key changes:
- Image upload in web UI (file picker, paste, preview strip)
- Image generation tool (FLUX/DALL-E via /v1/images/generations)
- Image edit tool (multipart /v1/images/edits with fallback)
- Image analysis tool (vision model for workspace images)
- Model detection utilities (image_models.rs, vision_models.rs)
- Sentinel JSON detection in dispatcher for generated image rendering
- StatusUpdate::ImageGenerated → SSE/WS/REPL/WASM broadcast
- HTTP webhook attachment support (base64, 5MB/file, 10MB total)
- WASM channel image download (Telegram via file API, Slack via host HTTP)
- Tool registration wiring in app.rs

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: address PR #725 review comments (16 issues)

- SecretString for API keys in all image tools (image_gen, image_edit, image_analyze)
- Binary image read via tokio::fs::read instead of DB-backed workspace.read()
- Replace Arc<Workspace> with Option<PathBuf> base_dir (workspace has no filesystem API)
- ApprovalRequirement::UnlessAutoApproved for cost-sensitive image tools
- Scope sentinel detection to image_generate/image_edit tool names only
- Skip ToolResult preview broadcast for image sentinels (avoids multi-MB base64 in SSE)
- Extract shared media_type_from_path() to builtin/mod.rs
- Rename fallback_chat_edit → fallback_generate with tracing::warn
- Increase gateway body limit from 1MB to 10MB for image uploads
- Increase webhook body limit to 15MB (base64 overhead)
- Log warning on invalid base64 in images_to_attachments
- Client-side image size limits (5MB/file, 5 images max) in app.js
- aria-label on attach button for accessibility
- Update body_too_large test for new 10MB limit

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: add Slack file size check before download (PR review item #15)

Skip downloading files larger than 20 MB in the Slack WASM channel to
avoid excessive memory use and slow downloads in the WASM runtime.
Logs a warning when a file is skipped. Also bumps channel versions
for Slack and Telegram (prior branch changes).

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* style: cargo fmt

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix(security): add path validation and approval requirement to image tools

Add sandbox path validation via validate_path() to both ImageAnalyzeTool
and ImageEditTool to prevent path traversal attacks that could exfiltrate
arbitrary files through external vision/edit APIs. Also fix
ImageAnalyzeTool::requires_approval to return UnlessAutoApproved,
consistent with ImageEditTool and ImageGenerateTool.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: post-download size guards and empty data_url sentinel check

- Slack: add post-download size check on actual bytes when metadata
  size_bytes is absent, preventing bypass of the 20MB limit
- Telegram: add 20MB download size limit (matching Slack) enforced
  in download_telegram_file() after receiving response bytes
- Dispatcher: skip broadcasting ImageGenerated SSE event when
  data_url is empty from unwrap_or_default(), log warning instead

Closes correctness issues #3, #4, #5 from PR #725 review.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: use mime_guess for media type detection, add alt attrs and media_type validation

- Replace hardcoded media type mapping with mime_guess crate (already in deps)
- Add alt attributes to img elements in web UI for accessibility
- Validate media_type starts with "image/" in images_to_attachments()
- Update bmp test assertion to match mime_guess behavior

Co-Authored-By: Claude Opus 4.6 <[email protected]>

---------

Co-authored-by: Claude Opus 4.6 <[email protected]>
Co-authored-by: Zaki <[email protected]>
2026-03-09 03:41:27 +00:00

96 lines
2.5 KiB
Rust

//! Image generation model detection utilities.
/// Known image generation model families.
const IMAGE_GEN_PATTERNS: &[&str] = &[
"flux",
"dall-e",
"dalle",
"stable-diffusion",
"sdxl",
"imagen",
"midjourney",
"ideogram",
"playground",
];
/// Check if a model name indicates an image generation model.
pub fn is_image_generation_model(model: &str) -> bool {
let lower = model.to_lowercase();
IMAGE_GEN_PATTERNS.iter().any(|p| lower.contains(p))
}
/// Suggest the best image generation model from a list of available models.
///
/// Priority: FLUX > DALL-E > Stable Diffusion > others.
pub fn suggest_image_model(models: &[String]) -> Option<&str> {
let priorities: &[&str] = &[
"flux",
"dall-e",
"dalle",
"stable-diffusion",
"sdxl",
"imagen",
];
for priority in priorities {
if let Some(model) = models.iter().find(|m| m.to_lowercase().contains(priority)) {
return Some(model);
}
}
// Fall back to any image gen model
models.iter().find_map(|m| {
if is_image_generation_model(m) {
Some(m.as_str())
} else {
None
}
})
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn detects_flux_models() {
assert!(is_image_generation_model(
"black-forest-labs/FLUX.1-schnell"
));
assert!(is_image_generation_model("flux-pro"));
}
#[test]
fn detects_dalle_models() {
assert!(is_image_generation_model("dall-e-3"));
assert!(is_image_generation_model("dalle-3"));
}
#[test]
fn rejects_non_image_models() {
assert!(!is_image_generation_model("gpt-4o"));
assert!(!is_image_generation_model("claude-3-sonnet"));
assert!(!is_image_generation_model("llama-3.1-70b"));
}
#[test]
fn suggests_flux_first() {
let models = vec![
"gpt-4o".to_string(),
"dall-e-3".to_string(),
"flux-pro".to_string(),
];
assert_eq!(suggest_image_model(&models), Some("flux-pro"));
}
#[test]
fn suggests_dalle_without_flux() {
let models = vec!["gpt-4o".to_string(), "dall-e-3".to_string()];
assert_eq!(suggest_image_model(&models), Some("dall-e-3"));
}
#[test]
fn returns_none_when_no_image_models() {
let models = vec!["gpt-4o".to_string(), "claude-3-sonnet".to_string()];
assert_eq!(suggest_image_model(&models), None);
}
}