mirror of
https://github.com/outbackdingo/optimclaw.git
synced 2026-08-25 14:53:34 +00:00
8e6e84a08d
* feat: Add benchmarking harness for agent evaluation Introduces ironclaw-bench, a Rust-native benchmarking crate that drives the real agent loop headlessly. Supports standard benchmarks (GAIA, Tau-bench, SWE-bench Pro) and custom JSONL task sets with parallel execution, resume support, and incremental JSONL output. Key components: - BenchChannel: headless Channel impl with auto-approval and response capture - InstrumentedLlm: LlmProvider wrapper recording per-call token/cost metrics - BenchRunner: task orchestration with parallel execution and JSONL resume - Scoring utilities: exact match, contains, regex (all with normalization) - CLI: run, results, compare, list subcommands via clap - Four suite adapters: custom, gaia, tau_bench, swe_bench Also fixes a pre-existing missing SseEvent::ToolResult match arm in the web gateway and adds FinishReason to the LLM module's public re-exports. Co-Authored-By: Claude Opus 4.6 <[email protected]> * feat: Add spot benchmark suite for end-to-end agent verification Adds a "spot" suite with 13 scenarios across 4 categories (smoke, tool use, multi-tool chaining, robustness) using multi-criterion assertions instead of simple text matching. Also adds an `error` field to TaskSubmission so suites can hard-fail on agent errors. Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: address audit findings in benchmarks crate - Fix O(n²) scoring loop by indexing tasks in a HashMap (was re-parsing JSONL per result) - Add UTF-8-safe truncation to prevent panic on multi-byte chars in channel capture - Wire setup_task/teardown_task into both sequential and parallel runner paths - Convert BenchRunner.suite from Box to Arc for parallel task setup/teardown - Add tracing::warn for placeholder scores in custom, swe_bench, tau_bench adapters - Add spot suite to CLI help text - Add doc comment clarifying tools_used HashSet behavior in SpotAssertions - Reorder match arms in create_suite to match KNOWN_SUITES alphabetical order Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: rewrite tasks.jsonl with scored results after scoring The JSONL file was only written during execution (pre-scoring), so the `results` command showed "pending" scores even after scoring completed. Now the runner rewrites the JSONL with final scored results, keeping task-level and aggregate data consistent. Co-Authored-By: Claude Opus 4.6 <[email protected]> * feat: prefix benchmark runs with model name and commit hash Run logs and results table now show the base model and short git commit hash, making it easy to correlate results with code versions. The commit hash is also persisted in run.json for historical tracking. Co-Authored-By: Claude Opus 4.6 <[email protected]> * feat: add 8 memory benchmark scenarios to spot suite Tests save-and-recall workflows using file tools: - daily tasks, reminders, meeting notes, append logs - detail extraction, todo priorities, multi-file ops - context updates (write-read-rewrite-verify) Total spot scenarios: 13 -> 21 Co-Authored-By: Claude Opus 4.6 <[email protected]> * chore: fmt channel.rs and gitignore bench-results Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: address critical and high findings from PR review - Fix race condition: parallel mode now writes JSONL after all tasks complete instead of concurrent unsynchronized appends - Fix UTF-8 panic: use .chars().take(25) instead of byte slicing on task_id which could panic on multi-byte characters - Remove dead code: max_iterations (parsed but never used), tool_whitelist() (declared but never called), MatrixEntry.tools (declared but never applied) - Eliminate double load_tasks(): cache task list on first load and reuse the index for scoring instead of re-reading from disk Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: relax smoke-greeting assertion to not demand parrot greeting The LLM often introduces itself without echoing "hello" back. Use a regex that accepts any reasonable self-introduction (hello, hi, hey, assistant, agent, help) instead of demanding a specific word. Co-Authored-By: Claude Opus 4.6 <[email protected]> * feat: 100% spot baseline (GPT-5.2 @ 2c43b83, 21/21 pass) Relax two brittle assertions: - smoke-greeting: use regex for any reasonable self-intro instead of demanding the model parrot "hello" - memory-update-context: drop response_not_contains PST since the model correctly says "not PST" which triggers the literal check - memory-multifile: lower min_tool_calls from 4 to 3, the model can batch two writes in one LLM turn Baseline results committed to benchmarks/baselines/ for regression tracking. Local runs stay in bench-results/ (gitignored). Results: 100.0% pass, 1.000 avg, $0.31 cost, 111s wall time Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: address remaining PR review comments - Replace .expect("semaphore closed") with proper error handling - Derive PartialEq on BenchScore for cleaner test assertions - Use ToPrimitive::to_f64() instead of string roundtrip in estimated_cost() - Validate SWE-bench inputs: task_id (path traversal), repo (owner/repo format), base_commit (valid git ref) with 5 new tests - Skip "pending" (unscored) entries during resume so they get re-executed - Use run.json mtime for find_latest_run (falls back to tasks.jsonl, then dir) - Move additional_tools() outside parallel loop to share Arc<[Tool]> across tasks - Add doc comments documenting known limitations (single-turn, resources, conversation) Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: reject absolute paths in SWE-bench and validate matrix config - is_safe_path_component now rejects paths starting with '/' - BenchConfig::from_file validates matrix is non-empty - Added tests for both validations Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: fail tasks on setup_task error and compute git hash once - setup_task failure now records an error TaskResult instead of continuing to run the task (both sequential and parallel paths) - git_short_hash() computed once per run instead of twice Co-Authored-By: Claude Opus 4.6 <[email protected]> --------- Co-authored-by: Claude Opus 4.6 <[email protected]>
22 lines
6.3 KiB
JSON
22 lines
6.3 KiB
JSON
{"id": "smoke-greeting", "prompt": "Hello! Introduce yourself briefly.", "tags": ["smoke"], "assertions": {"response_matches": "(?i)(hello|hi|hey|assistant|agent|help)", "no_error": true, "max_tool_calls": 0}}
|
|
{"id": "smoke-math", "prompt": "What is 47 * 23? Reply with just the number.", "tags": ["smoke"], "assertions": {"response_contains": ["1081"], "no_error": true, "max_tool_calls": 0}}
|
|
{"id": "tool-echo", "prompt": "Use the echo tool to repeat the message: 'Spot check passed'", "tags": ["tool"], "assertions": {"tools_used": ["echo"], "response_contains": ["Spot check passed"], "no_error": true}}
|
|
{"id": "tool-time", "prompt": "What is the current date and time? Use the time tool.", "tags": ["tool"], "assertions": {"tools_used": ["time"], "response_matches": "20\\d{2}", "no_error": true}}
|
|
{"id": "tool-json-query", "prompt": "Given this JSON: {\"users\": [{\"name\": \"Alice\"}, {\"name\": \"Bob\"}]}, use the json tool to extract the second user's name.", "tags": ["tool"], "assertions": {"tools_used": ["json"], "response_contains": ["Bob"], "no_error": true}}
|
|
{"id": "tool-shell-echo", "prompt": "Use the shell tool to run: echo 'benchmark test'", "tags": ["tool"], "assertions": {"tools_used": ["shell"], "response_contains": ["benchmark test"], "no_error": true}}
|
|
{"id": "tool-list-dir", "prompt": "Use the list_dir tool to list the contents of the /tmp directory.", "tags": ["tool"], "assertions": {"tools_used": ["list_dir"], "no_error": true}}
|
|
{"id": "chain-write-read", "prompt": "Write the text 'ironclaw spot check' to /tmp/ironclaw_spot_test.txt using the write_file tool, then read it back using the read_file tool and tell me what it says.", "tags": ["chain"], "assertions": {"tools_used": ["write_file", "read_file"], "response_contains": ["ironclaw spot check"], "no_error": true}}
|
|
{"id": "chain-shell-json", "prompt": "Run a shell command to output the JSON string '{\"status\": \"ok\", \"code\": 200}', then use the json tool to extract the status field.", "tags": ["chain"], "assertions": {"response_contains": ["ok"], "min_tool_calls": 1, "no_error": true}}
|
|
{"id": "chain-time-echo", "prompt": "First get the current time using the time tool, then use the echo tool to repeat it back.", "tags": ["chain"], "assertions": {"tools_used": ["time", "echo"], "no_error": true}}
|
|
{"id": "robust-no-tool", "prompt": "What is the capital of France? Answer directly without using any tools.", "tags": ["robust"], "assertions": {"response_contains": ["Paris"], "max_tool_calls": 0, "no_error": true}}
|
|
{"id": "robust-correct-tool", "prompt": "What time is it right now?", "tags": ["robust"], "assertions": {"tools_used": ["time"], "tools_not_used": ["shell", "echo"], "no_error": true}}
|
|
{"id": "robust-json-validate", "prompt": "Use the json tool to validate whether this is valid JSON: {\"key\": \"value\", \"num\": 42}", "tags": ["robust"], "assertions": {"tools_used": ["json"], "tools_not_used": ["shell"], "no_error": true}}
|
|
{"id": "memory-save-daily", "prompt": "Save these daily tasks to /tmp/bench-daily.md:\n1. Review PR #42\n2. Update API docs\n3. Deploy to staging\nThen read the file back and confirm what was saved.", "tags": ["memory"], "assertions": {"tools_used": ["write_file", "read_file"], "response_contains": ["PR #42", "docs", "staging"], "no_error": true}}
|
|
{"id": "memory-save-reminder", "prompt": "Write a reminder to /tmp/bench-reminder.md: Dentist appointment on March 5th at 2pm with Dr. Smith. Then read the file back and tell me when the appointment is and with whom.", "tags": ["memory"], "assertions": {"tools_used": ["write_file", "read_file"], "response_contains": ["March 5", "Smith"], "response_matches": "2(:00)?\\s*[Pp][Mm]", "no_error": true}}
|
|
{"id": "memory-save-meeting", "prompt": "Save these meeting notes to /tmp/bench-meeting.md:\nMeeting: Project Phoenix sync\nAttendees: Alice, Bob, Carol\nDecisions:\n- Launch date: April 15th\n- Budget: $50k approved\n- Bob owns frontend, Carol owns backend\nThen read the file back and tell me who owns the frontend and what the launch date is.", "tags": ["memory"], "assertions": {"tools_used": ["write_file", "read_file"], "response_contains": ["Bob", "frontend", "April 15"], "no_error": true}}
|
|
{"id": "memory-append-log", "prompt": "Write 'Morning standup: discussed blockers on auth service' to /tmp/bench-log.md. Then append a new line 'Afternoon: resolved API timeout issue in payment gateway' to the same file. Finally read the full file and summarize what happened.", "tags": ["memory"], "assertions": {"tools_used": ["write_file", "read_file"], "response_contains": ["auth", "timeout"], "min_tool_calls": 3, "no_error": true}}
|
|
{"id": "memory-recall-detail", "prompt": "Save the following project context to /tmp/bench-project.md:\nProject Ironclad uses Rust for the backend, React for the frontend, and PostgreSQL for the database. The API is deployed on AWS ECS. The lead developer is Sarah and the PM is James. The sprint ends on March 20th.\nThen read it back and answer: What database does the project use, and who is the PM?", "tags": ["memory"], "assertions": {"tools_used": ["write_file", "read_file"], "response_contains": ["PostgreSQL", "James"], "no_error": true}}
|
|
{"id": "memory-todo-priority", "prompt": "Write the following to /tmp/bench-todo.md:\n- [ ] Fix login bug (priority: HIGH)\n- [ ] Write unit tests (priority: medium)\n- [ ] Update README (priority: low)\n- [ ] Security audit (priority: HIGH)\nThen read it back and tell me which tasks are high priority.", "tags": ["memory"], "assertions": {"tools_used": ["write_file", "read_file"], "response_contains": ["login bug", "security audit"], "no_error": true}}
|
|
{"id": "memory-multifile", "prompt": "Save 'Team standup at 9am, then client demo at 2pm' to /tmp/bench-monday.md and 'Sprint retro at 11am, team lunch at noon' to /tmp/bench-tuesday.md. Then read both files and tell me what's happening on Tuesday.", "tags": ["memory"], "assertions": {"tools_used": ["write_file", "read_file"], "response_contains": ["retro", "lunch"], "min_tool_calls": 3, "no_error": true}}
|
|
{"id": "memory-update-context", "prompt": "Write 'User preference: dark mode, timezone: PST, language: English' to /tmp/bench-prefs.md. Then read it back, and rewrite the file changing the timezone to EST. Finally read it one more time and confirm the timezone is now EST.", "tags": ["memory"], "assertions": {"tools_used": ["write_file", "read_file"], "response_contains": ["EST"], "min_tool_calls": 4, "no_error": true}}
|