mirror of
https://github.com/outbackdingo/optimclaw.git
synced 2026-08-25 14:53:34 +00:00
37bba72397
* test: add 29 E2E trace tests for worker, threading, tools, workspace, and routines (#571-575) Add comprehensive E2E test coverage across five test files: - e2e_worker_coverage (7 tests): parallel tool calls, error feedback, unknown tools, invalid params, rate limiting, iteration limits, planning mode - e2e_thread_scheduling (3 tests + 2 deferred): multi-turn state, undo/redo, concurrent dispatch - e2e_builtin_tool_coverage (8 tests): time parse/diff/invalid, routine CRUD/history, job create/status/list/cancel, HTTP replay - e2e_workspace_coverage (6 tests): chunked search, multi-doc search, hybrid search, directory tree, document lifecycle, identity in system prompt - e2e_routine_heartbeat (5 tests): cron triggers, event matching, cooldown enforcement, heartbeat findings, empty checklist skip Infrastructure: extend TestRig with database/workspace/trace_llm accessors, register job and routine tools by default, add with_extra_tools() for custom stub tools. Includes 24 JSON trace fixtures across worker/, threading/, tools/, and workspace/. Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: use 6-field cron format in routine_create_list fixture The cron 0.13 crate accepts both 6 and 7 fields, but the routine_create tool documents 6-field format. Align the fixture to match. [skip-regression-check] Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: eliminate vacuous passes and silently-skipped assertions in E2E tests - job_create_status: replace job_status (needs dynamic UUID) with list_jobs, assert both succeed via completed() not just started() - job_list_cancel: keep cancel_job but explicitly assert it fails with invalid canned job_id "latest", verify create_job + list_jobs succeed - unknown_tool_name: add !is_empty() guard before .all() to prevent vacuous pass on empty iterator - workspace tests: change `if let Some(ws)` to `.expect()` so assertions are never silently skipped when workspace/trace_llm is available [skip-regression-check] Co-Authored-By: Claude Opus 4.6 <[email protected]> * feat: add template substitution to TraceLlm for dynamic tool result forwarding Add {{call_id.json_path}} template syntax to trace fixtures, enabling tool results from one step to flow into subsequent steps' arguments. TraceLlm extracts variables from Role::Tool messages (stripping the safety layer's <tool_output> XML wrapper and unescaping entities) and substitutes them in canned tool_call arguments before returning. This fixes job_create_status and job_list_cancel tests to properly test job_status and cancel_job with real dynamic UUIDs from create_job, instead of using invalid canned IDs that silently failed. Also adds tool result content assertions to job_create_status to verify the actual tool output contains expected data (job_id, title). [skip-regression-check] Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: address PR review feedback on E2E tests - undo_redo_cycle: assert exactly 3 turns instead of >= 2 - tool_error_feedback: use tempfile::tempdir() instead of hardcoded /tmp path, patch fixture path at runtime for CI portability - worker_timeout → iteration_limit: rename to accurately describe what's tested - post_plan_work_remaining → simple_echo_flow: rename, test doesn't exercise planning - identity_in_system_prompt: seed IDENTITY.md before test, assert system prompt contains the seeded content instead of just checking Role::System exists [skip-regression-check] Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: strengthen workspace E2E test assertions per PR review - write_chunk_search: assert memory_search was called and returned payment/architecture-related results - multi_document_search: assert memory_search was called for cross-document search - hybrid_search_with_embeddings: assert both memory_write and memory_search were called to confirm write-then-search pipeline - directory_tree: assert tree output contains expected alpha/beta project paths [skip-regression-check] Co-Authored-By: Claude Opus 4.6 <[email protected]> --------- Co-authored-by: Claude Opus 4.6 <[email protected]>
55 lines
1.6 KiB
JSON
55 lines
1.6 KiB
JSON
{
|
|
"model_name": "test-hybrid-search",
|
|
"expects": {
|
|
"tools_used": [
|
|
"memory_write",
|
|
"memory_search"
|
|
],
|
|
"all_tools_succeeded": true,
|
|
"min_responses": 1
|
|
},
|
|
"steps": [
|
|
{
|
|
"response": {
|
|
"type": "tool_calls",
|
|
"tool_calls": [
|
|
{
|
|
"id": "call_mw_hybrid",
|
|
"name": "memory_write",
|
|
"arguments": {
|
|
"content": "# Machine Learning Pipeline\n\nOur ML pipeline uses PyTorch for model training and ONNX for inference. Feature engineering is done with Pandas and the feature store uses Feast. Model versioning is handled by MLflow with experiment tracking. The training infrastructure runs on GPU-enabled Kubernetes pods.",
|
|
"target": "context/ml-pipeline.md"
|
|
}
|
|
}
|
|
],
|
|
"input_tokens": 100,
|
|
"output_tokens": 35
|
|
}
|
|
},
|
|
{
|
|
"response": {
|
|
"type": "tool_calls",
|
|
"tool_calls": [
|
|
{
|
|
"id": "call_ms_hybrid",
|
|
"name": "memory_search",
|
|
"arguments": {
|
|
"query": "deep learning model training infrastructure"
|
|
}
|
|
}
|
|
],
|
|
"input_tokens": 200,
|
|
"output_tokens": 20
|
|
}
|
|
},
|
|
{
|
|
"response": {
|
|
"type": "text",
|
|
"content": "The hybrid search found the ML pipeline document. Even though the exact phrase 'deep learning' isn't in the document, the semantic similarity between 'deep learning model training' and 'PyTorch model training' helped surface the relevant content.",
|
|
"input_tokens": 300,
|
|
"output_tokens": 35
|
|
}
|
|
}
|
|
]
|
|
}
|