mirror of
https://github.com/outbackdingo/optimclaw.git
synced 2026-08-25 14:53:34 +00:00
37bba72397
* test: add 29 E2E trace tests for worker, threading, tools, workspace, and routines (#571-575) Add comprehensive E2E test coverage across five test files: - e2e_worker_coverage (7 tests): parallel tool calls, error feedback, unknown tools, invalid params, rate limiting, iteration limits, planning mode - e2e_thread_scheduling (3 tests + 2 deferred): multi-turn state, undo/redo, concurrent dispatch - e2e_builtin_tool_coverage (8 tests): time parse/diff/invalid, routine CRUD/history, job create/status/list/cancel, HTTP replay - e2e_workspace_coverage (6 tests): chunked search, multi-doc search, hybrid search, directory tree, document lifecycle, identity in system prompt - e2e_routine_heartbeat (5 tests): cron triggers, event matching, cooldown enforcement, heartbeat findings, empty checklist skip Infrastructure: extend TestRig with database/workspace/trace_llm accessors, register job and routine tools by default, add with_extra_tools() for custom stub tools. Includes 24 JSON trace fixtures across worker/, threading/, tools/, and workspace/. Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: use 6-field cron format in routine_create_list fixture The cron 0.13 crate accepts both 6 and 7 fields, but the routine_create tool documents 6-field format. Align the fixture to match. [skip-regression-check] Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: eliminate vacuous passes and silently-skipped assertions in E2E tests - job_create_status: replace job_status (needs dynamic UUID) with list_jobs, assert both succeed via completed() not just started() - job_list_cancel: keep cancel_job but explicitly assert it fails with invalid canned job_id "latest", verify create_job + list_jobs succeed - unknown_tool_name: add !is_empty() guard before .all() to prevent vacuous pass on empty iterator - workspace tests: change `if let Some(ws)` to `.expect()` so assertions are never silently skipped when workspace/trace_llm is available [skip-regression-check] Co-Authored-By: Claude Opus 4.6 <[email protected]> * feat: add template substitution to TraceLlm for dynamic tool result forwarding Add {{call_id.json_path}} template syntax to trace fixtures, enabling tool results from one step to flow into subsequent steps' arguments. TraceLlm extracts variables from Role::Tool messages (stripping the safety layer's <tool_output> XML wrapper and unescaping entities) and substitutes them in canned tool_call arguments before returning. This fixes job_create_status and job_list_cancel tests to properly test job_status and cancel_job with real dynamic UUIDs from create_job, instead of using invalid canned IDs that silently failed. Also adds tool result content assertions to job_create_status to verify the actual tool output contains expected data (job_id, title). [skip-regression-check] Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: address PR review feedback on E2E tests - undo_redo_cycle: assert exactly 3 turns instead of >= 2 - tool_error_feedback: use tempfile::tempdir() instead of hardcoded /tmp path, patch fixture path at runtime for CI portability - worker_timeout → iteration_limit: rename to accurately describe what's tested - post_plan_work_remaining → simple_echo_flow: rename, test doesn't exercise planning - identity_in_system_prompt: seed IDENTITY.md before test, assert system prompt contains the seeded content instead of just checking Role::System exists [skip-regression-check] Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: strengthen workspace E2E test assertions per PR review - write_chunk_search: assert memory_search was called and returned payment/architecture-related results - multi_document_search: assert memory_search was called for cross-document search - hybrid_search_with_embeddings: assert both memory_write and memory_search were called to confirm write-then-search pipeline - directory_tree: assert tree output contains expected alpha/beta project paths [skip-regression-check] Co-Authored-By: Claude Opus 4.6 <[email protected]> --------- Co-authored-by: Claude Opus 4.6 <[email protected]>
71 lines
1.7 KiB
JSON
71 lines
1.7 KiB
JSON
{
|
|
"model_name": "test-directory-tree",
|
|
"expects": {
|
|
"tools_used": [
|
|
"memory_write",
|
|
"memory_tree"
|
|
],
|
|
"all_tools_succeeded": true,
|
|
"min_responses": 1
|
|
},
|
|
"steps": [
|
|
{
|
|
"response": {
|
|
"type": "tool_calls",
|
|
"tool_calls": [
|
|
{
|
|
"id": "call_mw_t1",
|
|
"name": "memory_write",
|
|
"arguments": {
|
|
"content": "# Alpha Project\n\nMain readme for the Alpha project.",
|
|
"target": "projects/alpha/readme.md"
|
|
}
|
|
},
|
|
{
|
|
"id": "call_mw_t2",
|
|
"name": "memory_write",
|
|
"arguments": {
|
|
"content": "# Alpha Config\n\nConfiguration details for Alpha.",
|
|
"target": "projects/alpha/config.md"
|
|
}
|
|
},
|
|
{
|
|
"id": "call_mw_t3",
|
|
"name": "memory_write",
|
|
"arguments": {
|
|
"content": "# Beta Project\n\nMain readme for the Beta project.",
|
|
"target": "projects/beta/readme.md"
|
|
}
|
|
}
|
|
],
|
|
"input_tokens": 100,
|
|
"output_tokens": 50
|
|
}
|
|
},
|
|
{
|
|
"response": {
|
|
"type": "tool_calls",
|
|
"tool_calls": [
|
|
{
|
|
"id": "call_mt_1",
|
|
"name": "memory_tree",
|
|
"arguments": {
|
|
"path": "projects"
|
|
}
|
|
}
|
|
],
|
|
"input_tokens": 200,
|
|
"output_tokens": 20
|
|
}
|
|
},
|
|
{
|
|
"response": {
|
|
"type": "text",
|
|
"content": "The workspace tree under 'projects/' shows two subdirectories: alpha (with readme.md and config.md) and beta (with readme.md).",
|
|
"input_tokens": 300,
|
|
"output_tokens": 25
|
|
}
|
|
}
|
|
]
|
|
}
|