Files
optimclaw/tests/e2e_routine_heartbeat.rs
T
a868b14221 Fix/lightweight action tool (#785)
* feat: add tool execution support to lightweight routines

Lightweight routines now execute tools instead of outputting raw tool-call XML.

**Problem:** Lightweight routines had no tool execution loop, causing the LLM to generate
tool-call XML as text output (visible to users as garbage on Telegram). All 4 scheduled
routines were disabled and Emil saw the same issue in health-ping routine.

**Solution:** Implement a simplified agentic loop for lightweight routines that:
- Supports up to 3-5 tool iterations (configurable, capped at 5)
- Executes tools sequentially (not parallel, keeps overhead low)
- Auto-approves non-Always tools (lightweight routines are autonomous)
- Sanitizes and wraps tool outputs via SafetyLayer (same as dispatcher)
- Forces text-only response at iteration limit (guarantees termination)
- Maintains backward compatibility (disabled by default, toggled by config)

**Changes:**
1. **src/config/routines.rs:**
   - Added lightweight_tools_enabled (default: true)
   - Added lightweight_max_iterations (default: 3, capped at 5)
   - Added env var support: ROUTINES_LIGHTWEIGHT_TOOLS, ROUTINES_LIGHTWEIGHT_MAX_ITERATIONS

2. **src/agent/routine_engine.rs:**
   - Extended EngineContext with tools and safety fields
   - Split execute_lightweight into three functions:
     - execute_lightweight: router that dispatches to tool or no-tool version
     - execute_lightweight_no_tools: original single-call behavior
     - execute_lightweight_with_tools: new agentic loop with tool support
   - Added execute_routine_tool: isolated tool execution with validation and timeout
   - Uses ToolCompletionRequest/ToolCompletionResponse for tool-aware LLM calls
   - Integrates SafetyLayer for tool output sanitization

3. **src/agent/agent_loop.rs:**
   - Updated RoutineEngine::new call to pass tools and safety

**Tool Execution Loop:**
1. Build initial messages (system + user prompt)
2. Get tool definitions (empty at iteration limit)
3. Call LLM with ToolCompletionRequest
4. If text response: check for ROUTINE_OK sentinel, return result
5. If tool calls: execute sequentially, sanitize, wrap, add to context, loop
6. Safety ceiling at 5 iterations prevents runaway execution

**Approval Handling:** Auto-approves UnlessAutoApproved and Never tools;
blocks Always tools with error message (routines are autonomous by design).

**Testing:** All 2756 tests pass. Zero clippy warnings.

Co-Authored-By: Claude Haiku 4.5 <[email protected]>

* test: add comprehensive unit tests for lightweight routine tool execution

Added 9 new unit tests covering:
- Configuration defaults (lightweight_tools_enabled, lightweight_max_iterations)
- Max iterations capped at 5 (safety ceiling)
- Routine name sanitization (special chars, alphanumeric preservation)
- Sentinel detection for ROUTINE_OK (exact match, contains, whitespace handling)
- Iteration limit safety ceiling enforcement
- Approval requirement pattern matching (Never, UnlessAutoApproved, Always)
- Empty response handling (finish_reason detection)

All 2765 tests pass (11 routine_engine tests, +9 new).

The tests cover the core logic paths of:
- Configuration validation
- Response parsing and sentinel detection
- Name sanitization for workspace paths
- Approval requirement logic
- Iteration limits and safety ceilings

Note: These are unit tests for core logic. Full integration tests with mock LLM
and tool registry would require more complex test infrastructure and are a future enhancement.

Co-Authored-By: Claude Haiku 4.5 <[email protected]>

* style: format routine_engine.rs per cargo fmt

Apply consistent formatting to match Rust style guidelines:
- Break long import lines
- Reformat method chains for readability
- Format multi-line return tuples

No functional changes.

Co-Authored-By: Claude Haiku 4.5 <[email protected]>

* fix: address security and code quality issues in lightweight routine tool execution

**Security Fixes:**

1. Sanitize tool error messages (medium severity)
   - Tool error messages were sent directly to LLM without sanitization
   - Now wrapped through SafetyLayer like successful outputs
   - Prevents leakage of API keys, internal paths, or PII from errors

2. Use unique job_id for each routine run (medium severity)
   - Previously reused routine.id across all executions
   - Caused state collisions and race conditions
   - Now generates unique run_id (Uuid::new_v4()) for each execution
   - Matches behavior of full_job routines

**Code Quality Fixes:**

3. Remove unreachable code
   - Deleted dead if iteration > 5 check
   - max_iterations is capped at 5 via .min(5), so check was impossible
   - Improves code clarity

4. Extract duplicated response handling logic
   - Created handle_text_response() helper function
   - Eliminated 20+ lines of duplicated ROUTINE_OK sentinel detection
   - Reduces maintenance burden and risk of inconsistencies

5. Fix test duplication
   - Tests now call actual super::sanitize_routine_name()
   - Removes duplicate implementation in tests
   - Ensures tests detect changes to original function

**Testing:**
- All 2765 tests pass (no regressions)
- Zero clippy warnings
- Test coverage maintained

Co-Authored-By: Claude Haiku 4.5 <[email protected]>

* fix: address security issue and improve code quality in lightweight routine tool execution

**SECURITY FIX (High Severity):**

1. Block UnlessAutoApproved tools in lightweight routines
   - Previously auto-approved UnlessAutoApproved tools, creating prompt injection vulnerability
   - Lightweight routines can be triggered by external events (channel messages, webhooks)
   - If susceptible to prompt injection, attacker could trick LLM into calling sensitive tools
   - Now blocks both UnlessAutoApproved and Always tools (only Never tools allowed)
   - Only safe approach without requiring tool_permissions allowlist in routine data model
   - Prevents unauthorized file access, network requests, and other sensitive operations

**Code Quality Improvements:**

2. Use ToolError::Timeout for consistent error handling (medium)
   - Changed from std::io::Error to proper ToolError::Timeout variant
   - More idiomatic and consistent with tool execution error handling
   - Makes errors easier to debug and handle uniformly

3. Fix misleading test names and remove tautological tests (medium)
   - Renamed test_routine_config_lightweight_max_iterations_capped_at_five to
     test_routine_config_can_hold_uncapped_max_iterations
   - Clarified comments to explain where capping actually occurs
   - Removed test_iteration_limit_safety_ceiling (tautological: asserts x.min(5) <= 5)
   - Improves test clarity and prevents false sense of coverage

**Testing:**
- 2764 tests passing (1 test removed, no regressions)
- Zero clippy warnings
- Security vulnerability eliminated

Co-Authored-By: Claude Haiku 4.5 <[email protected]>

* style: format routine_engine.rs per cargo fmt

Apply consistent formatting:
- Fix method chain indentation for LLM completion calls
- Reformat error handling closures for readability
- Break long method calls (wrap_for_llm) across multiple lines

No functional changes.

Co-Authored-By: Claude Haiku 4.5 <[email protected]>

* style: apply cargo fmt formatting fixes to routine_engine.rs

Align formatting with project standards:
- Break long method chains across multiple lines for readability
- Reformat error return statements for consistency
- Split long assert/assert_eq statements across multiple lines

No logic changes; purely cosmetic formatting.

Co-Authored-By: Claude Haiku 4.5 <[email protected]>

* test: update routine engine tests for tool/safety layer parameters

Update test code to pass newly required ToolRegistry and SafetyLayer
parameters to RoutineEngine::new(). Also add missing lightweight_tools_enabled
and lightweight_max_iterations fields to RoutineConfig initializers in tests.

Tests affected:
- tests/support/test_rig.rs: Added tools and safety layer to RoutineEngine::new()
- tests/e2e_routine_heartbeat.rs: Added three instances of tools and safety layer construction

All tests pass (2764 tests).

Co-Authored-By: Claude Haiku 4.5 <[email protected]>

---------

Co-authored-by: Claude Haiku 4.5 <[email protected]>
Co-authored-by: Henry Park <[email protected]>
2026-03-09 20:22:10 -07:00

447 lines
15 KiB
Rust

//! E2E tests: routine engine and heartbeat (#575).
//!
//! These tests construct RoutineEngine and HeartbeatRunner directly
//! with a TraceLlm and libSQL database, bypassing the full TestRig.
#[cfg(feature = "libsql")]
mod support;
#[cfg(feature = "libsql")]
mod tests {
use std::sync::Arc;
use std::time::Duration;
use chrono::Utc;
use uuid::Uuid;
use ironclaw::agent::routine::{
NotifyConfig, Routine, RoutineAction, RoutineGuardrails, Trigger,
};
use ironclaw::agent::routine_engine::RoutineEngine;
use ironclaw::agent::{HeartbeatConfig, HeartbeatRunner};
use ironclaw::channels::IncomingMessage;
use ironclaw::config::{RoutineConfig, SafetyConfig};
use ironclaw::db::Database;
use ironclaw::safety::SafetyLayer;
use ironclaw::tools::ToolRegistry;
use ironclaw::workspace::Workspace;
use ironclaw::workspace::hygiene::HygieneConfig;
use crate::support::trace_llm::{LlmTrace, TraceLlm, TraceResponse, TraceStep};
/// Create a temp libSQL database with migrations applied.
async fn create_test_db() -> (Arc<dyn Database>, tempfile::TempDir) {
use ironclaw::db::libsql::LibSqlBackend;
let temp_dir = tempfile::tempdir().expect("tempdir");
let db_path = temp_dir.path().join("test.db");
let backend = LibSqlBackend::new_local(&db_path)
.await
.expect("LibSqlBackend");
backend.run_migrations().await.expect("migrations");
let db: Arc<dyn Database> = Arc::new(backend);
(db, temp_dir)
}
/// Create a workspace backed by the test database.
fn create_workspace(db: &Arc<dyn Database>) -> Arc<Workspace> {
Arc::new(Workspace::new_with_db("default", db.clone()))
}
/// Helper to insert a routine directly into the database.
fn make_routine(name: &str, trigger: Trigger, prompt: &str) -> Routine {
Routine {
id: Uuid::new_v4(),
name: name.to_string(),
description: format!("Test routine: {name}"),
user_id: "default".to_string(),
enabled: true,
trigger,
action: RoutineAction::Lightweight {
prompt: prompt.to_string(),
context_paths: vec![],
max_tokens: 1000,
},
guardrails: RoutineGuardrails {
cooldown: Duration::from_secs(0),
max_concurrent: 5,
dedup_window: None,
},
notify: NotifyConfig::default(),
last_run_at: None,
next_fire_at: None,
run_count: 0,
consecutive_failures: 0,
state: serde_json::json!({}),
created_at: Utc::now(),
updated_at: Utc::now(),
}
}
// -----------------------------------------------------------------------
// Test 1: cron_routine_fires
// -----------------------------------------------------------------------
#[tokio::test]
async fn cron_routine_fires() {
let (db, _tmp) = create_test_db().await;
let ws = create_workspace(&db);
// Create a TraceLlm that responds with ROUTINE_OK.
let trace = LlmTrace::single_turn(
"test-cron-fire",
"check",
vec![TraceStep {
request_hint: None,
response: TraceResponse::Text {
content: "ROUTINE_OK".to_string(),
input_tokens: 50,
output_tokens: 5,
},
expected_tool_results: vec![],
}],
);
let llm = Arc::new(TraceLlm::from_trace(trace));
let (notify_tx, mut notify_rx) = tokio::sync::mpsc::channel(16);
// Create minimal ToolRegistry and SafetyLayer for test.
let tools = Arc::new(ToolRegistry::new());
let safety_config = SafetyConfig {
max_output_length: 100_000,
injection_check_enabled: true,
};
let safety = Arc::new(SafetyLayer::new(&safety_config));
let engine = Arc::new(RoutineEngine::new(
RoutineConfig::default(),
db.clone(),
llm,
ws,
notify_tx,
None,
tools,
safety,
));
// Insert a cron routine with next_fire_at in the past.
let mut routine = make_routine(
"cron-test",
Trigger::Cron {
schedule: "* * * * *".to_string(),
timezone: None,
},
"Check system status.",
);
routine.next_fire_at = Some(Utc::now() - chrono::Duration::minutes(5));
db.create_routine(&routine).await.expect("create_routine");
// Fire cron triggers.
engine.check_cron_triggers().await;
// Give the spawned task time to execute.
tokio::time::sleep(Duration::from_millis(500)).await;
// Verify a run was recorded.
let runs = db
.list_routine_runs(routine.id, 10)
.await
.expect("list_routine_runs");
assert!(
!runs.is_empty(),
"Expected at least one routine run after cron trigger"
);
// Notification may or may not be sent depending on config;
// just verify no panic occurred. Drain the channel.
let _ = notify_rx.try_recv();
}
// -----------------------------------------------------------------------
// Test 2: event_trigger_matches
// -----------------------------------------------------------------------
#[tokio::test]
async fn event_trigger_matches() {
let (db, _tmp) = create_test_db().await;
let ws = create_workspace(&db);
let trace = LlmTrace::single_turn(
"test-event-match",
"deploy",
vec![TraceStep {
request_hint: None,
response: TraceResponse::Text {
content: "Deployment detected".to_string(),
input_tokens: 50,
output_tokens: 10,
},
expected_tool_results: vec![],
}],
);
let llm = Arc::new(TraceLlm::from_trace(trace));
let (notify_tx, _notify_rx) = tokio::sync::mpsc::channel(16);
// Create minimal ToolRegistry and SafetyLayer for test.
let tools = Arc::new(ToolRegistry::new());
let safety_config = SafetyConfig {
max_output_length: 100_000,
injection_check_enabled: true,
};
let safety = Arc::new(SafetyLayer::new(&safety_config));
let engine = Arc::new(RoutineEngine::new(
RoutineConfig::default(),
db.clone(),
llm,
ws,
notify_tx,
None,
tools,
safety,
));
// Insert an event routine matching "deploy.*production".
let routine = make_routine(
"deploy-watcher",
Trigger::Event {
channel: None,
pattern: "deploy.*production".to_string(),
},
"Report on deployment.",
);
db.create_routine(&routine).await.expect("create_routine");
// Refresh the event cache so the engine knows about the routine.
engine.refresh_event_cache().await;
// Positive match: message containing "deploy to production".
let matching_msg = IncomingMessage {
id: Uuid::new_v4(),
channel: "test".to_string(),
user_id: "default".to_string(),
user_name: None,
content: "deploy to production now".to_string(),
thread_id: None,
received_at: Utc::now(),
metadata: serde_json::json!({}),
timezone: None,
attachments: Vec::new(),
};
let fired = engine.check_event_triggers(&matching_msg).await;
assert!(
fired >= 1,
"Expected >= 1 routine fired on match, got {fired}"
);
// Give spawn time.
tokio::time::sleep(Duration::from_millis(500)).await;
// Negative match: message that doesn't match.
let non_matching_msg = IncomingMessage {
id: Uuid::new_v4(),
channel: "test".to_string(),
user_id: "default".to_string(),
user_name: None,
content: "check the staging environment".to_string(),
thread_id: None,
received_at: Utc::now(),
metadata: serde_json::json!({}),
timezone: None,
attachments: Vec::new(),
};
let fired_neg = engine.check_event_triggers(&non_matching_msg).await;
assert_eq!(fired_neg, 0, "Expected 0 routines fired on non-match");
}
// -----------------------------------------------------------------------
// Test 3: routine_cooldown
// -----------------------------------------------------------------------
#[tokio::test]
async fn routine_cooldown() {
let (db, _tmp) = create_test_db().await;
let ws = create_workspace(&db);
// Need two LLM responses (one for the first fire).
let trace = LlmTrace::single_turn(
"test-cooldown",
"check",
vec![TraceStep {
request_hint: None,
response: TraceResponse::Text {
content: "ROUTINE_OK".to_string(),
input_tokens: 50,
output_tokens: 5,
},
expected_tool_results: vec![],
}],
);
let llm = Arc::new(TraceLlm::from_trace(trace));
let (notify_tx, _notify_rx) = tokio::sync::mpsc::channel(16);
// Create minimal ToolRegistry and SafetyLayer for test.
let tools = Arc::new(ToolRegistry::new());
let safety_config = SafetyConfig {
max_output_length: 100_000,
injection_check_enabled: true,
};
let safety = Arc::new(SafetyLayer::new(&safety_config));
let engine = Arc::new(RoutineEngine::new(
RoutineConfig::default(),
db.clone(),
llm,
ws,
notify_tx,
None,
tools,
safety,
));
// Insert an event routine with 1-hour cooldown.
let mut routine = make_routine(
"cooldown-test",
Trigger::Event {
channel: None,
pattern: "test-cooldown".to_string(),
},
"Check status.",
);
routine.guardrails.cooldown = Duration::from_secs(3600);
db.create_routine(&routine).await.expect("create_routine");
engine.refresh_event_cache().await;
// First fire should work.
let msg = IncomingMessage {
id: Uuid::new_v4(),
channel: "test".to_string(),
user_id: "default".to_string(),
user_name: None,
content: "test-cooldown trigger".to_string(),
thread_id: None,
received_at: Utc::now(),
metadata: serde_json::json!({}),
timezone: None,
attachments: Vec::new(),
};
let fired1 = engine.check_event_triggers(&msg).await;
assert!(fired1 >= 1, "First fire should work");
// Give spawn time, then update last_run_at to simulate recent execution.
tokio::time::sleep(Duration::from_millis(300)).await;
// Update the routine's last_run_at to now (simulating it just ran).
db.update_routine_runtime(routine.id, Utc::now(), None, 1, 0, &serde_json::json!({}))
.await
.expect("update_routine_runtime");
// Refresh cache to pick up updated last_run_at.
engine.refresh_event_cache().await;
// Second fire should be blocked by cooldown.
let fired2 = engine.check_event_triggers(&msg).await;
assert_eq!(fired2, 0, "Second fire should be blocked by cooldown");
}
// -----------------------------------------------------------------------
// Test 4: heartbeat_findings
// -----------------------------------------------------------------------
#[tokio::test]
async fn heartbeat_findings() {
let (db, _tmp) = create_test_db().await;
let ws = create_workspace(&db);
// Write a real heartbeat checklist.
ws.write(
"HEARTBEAT.md",
"# Heartbeat Checklist\n\n- [ ] Check if the server is running\n- [ ] Review error logs",
)
.await
.expect("write heartbeat");
// LLM responds with findings (not HEARTBEAT_OK).
let trace = LlmTrace::single_turn(
"test-heartbeat-findings",
"heartbeat",
vec![TraceStep {
request_hint: None,
response: TraceResponse::Text {
content: "The server has elevated error rates. Review the logs immediately."
.to_string(),
input_tokens: 100,
output_tokens: 20,
},
expected_tool_results: vec![],
}],
);
let llm = Arc::new(TraceLlm::from_trace(trace));
let (tx, mut rx) = tokio::sync::mpsc::channel(16);
let hygiene_config = HygieneConfig {
enabled: false,
daily_retention_days: 30,
conversation_retention_days: 7,
cadence_hours: 24,
state_dir: _tmp.path().to_path_buf(),
};
let runner = HeartbeatRunner::new(HeartbeatConfig::default(), hygiene_config, ws, llm)
.with_response_channel(tx);
let result = runner.check_heartbeat().await;
match result {
ironclaw::agent::HeartbeatResult::NeedsAttention(msg) => {
assert!(
msg.contains("error"),
"Expected 'error' in attention message: {msg}"
);
}
other => panic!("Expected NeedsAttention, got: {other:?}"),
}
// No notification since we called check_heartbeat directly (not run).
let _ = rx.try_recv();
}
// -----------------------------------------------------------------------
// Test 5: heartbeat_empty_skip
// -----------------------------------------------------------------------
#[tokio::test]
async fn heartbeat_empty_skip() {
let (db, _tmp) = create_test_db().await;
let ws = create_workspace(&db);
// Write an effectively empty heartbeat (just headers and comments).
ws.write(
"HEARTBEAT.md",
"# Heartbeat Checklist\n\n<!-- No tasks yet -->\n",
)
.await
.expect("write heartbeat");
// LLM should NOT be called, so provide a trace that would panic if called.
let trace = LlmTrace::single_turn("test-heartbeat-skip", "skip", vec![]);
let llm = Arc::new(TraceLlm::from_trace(trace));
let hygiene_config = HygieneConfig {
enabled: false,
daily_retention_days: 30,
conversation_retention_days: 7,
cadence_hours: 24,
state_dir: _tmp.path().to_path_buf(),
};
let runner = HeartbeatRunner::new(HeartbeatConfig::default(), hygiene_config, ws, llm);
let result = runner.check_heartbeat().await;
assert!(
matches!(result, ironclaw::agent::HeartbeatResult::Skipped),
"Expected Skipped for empty checklist, got: {result:?}"
);
}
}