mirror of
https://github.com/outbackdingo/optimclaw.git
synced 2026-09-01 17:19:24 +00:00
fix: job token budget, iteration cap → Failed, web cancel stops worker (#788)
* feat: persist user_id in save_job and expose job_id on routine runs (#709) * feat: persist worker events to DB and fix activity tab rendering In-process Worker (used by Scheduler::dispatch_job) now persists events via save_job_event at key execution points: plan creation, LLM responses, tool_use, tool_result, and job completion/failure/stuck. Event data shapes match the container worker format so the gateway activity tab renders them correctly. Frontend: tool_result errors now show a red X icon with danger styling instead of a silent empty output. The result event falls back to the error field when message is absent. Co-Authored-By: Claude Opus 4.6 <[email protected]> * feat: wire RoutineEngine into gateway for direct manual trigger firing Replace the message-channel hack in routines_trigger_handler with a direct call to RoutineEngine::fire_manual(), ensuring FullJob routines dispatch correctly when triggered from the web UI. Inject the engine into GatewayState from Agent::run after construction. Also persists user_id in save_job for both PG and libSQL backends, removes the source='sandbox' filter so all jobs are visible, and exposes job_id on RoutineRunInfo for the frontend job link. Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: remove stale gateway_state argument from Agent::new test call sites The gateway_state parameter was removed from Agent::new during rebase (replaced by post-construction set_routine_engine_slot), but three test call sites still passed the extra None argument. Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: address PR review — restore sandbox source filter, remove blank lines - Revert removal of `source = 'sandbox'` filter in all SandboxStore queries (8 sites across PG and libSQL). Sandbox-specific APIs should stay scoped to sandbox jobs; unified job listing for the Jobs tab should use a separate query path. - Remove extra blank lines in agent_loop.rs and worker.rs that caused formatting CI failure. [skip-regression-check] Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: address review — regenerate Cargo.lock, add user_id regression test - Regenerate Cargo.lock from main's lockfile to eliminate dependency version downgrades (anyhow, syn, etc.) that were churn from rebase. - Add regression test verifying user_id round-trips through save_job and get_job in the libSQL backend. Co-Authored-By: Claude Opus 4.6 <[email protected]> * style: remove trailing blank line in libsql jobs.rs [skip-regression-check] Co-Authored-By: Claude Opus 4.6 <[email protected]> * test: add Postgres-side regression test for user_id persistence in save_job Mirrors the existing libSQL test (test_save_job_persists_user_id) for the Postgres backend. Gated behind #[cfg(feature = "postgres")] + #[ignore] since it requires a running PostgreSQL instance (integration tier). Co-Authored-By: Claude Opus 4.6 <[email protected]> --------- Co-authored-by: Claude Opus 4.6 <[email protected]> * fix: add job token budget, change iteration cap to Failed, fix web cancel (#698) Jobs could enter infinite retry loops because: (1) no token budget was enforced, (2) iteration cap marked jobs as Stuck (allowing self-repair to restart them), and (3) the web UI cancel button only updated the DB without stopping the running worker. - Add `max_tokens_per_job` config (settings.json + AGENT_MAX_TOKENS_PER_JOB env var, default 0 = unlimited) with per-job metadata override - Track token usage after respond_with_tools() and fail the job on budget exceeded - Change iteration cap and persistent rate limiting from mark_stuck to mark_failed, preventing self-repair restart loops - Fix web cancel handler to call scheduler.stop() which updates in-memory state AND aborts the worker task, falling back to DB-only update Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: address PR review — always persist cancel to DB, simplify token check - Cancel handler now always persists Cancelled to DB regardless of whether scheduler.stop() ran, fixing the edge case where stop() returns Ok(()) for jobs not in the scheduler map - Collapse nested ifs per clippy (let-chains) - Add NOTE comment about select_tools() not exposing TokenUsage [skip-regression-check] Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: rustfmt formatting in wizard.rs (pre-existing) [skip-regression-check] Co-Authored-By: Claude Opus 4.6 <[email protected]> --------- Co-authored-by: Claude Opus 4.6 <[email protected]>
This commit is contained in:
Illia Polosukhin
co-authored by
Claude Opus 4.6
Claude Opus 4.6
parent
764be8547f
commit
83950d11a4
@@ -1205,6 +1205,7 @@ mod tests {
|
||||
max_tool_iterations: 50,
|
||||
auto_approve_tools: false,
|
||||
default_timezone: "UTC".to_string(),
|
||||
max_tokens_per_job: 0,
|
||||
},
|
||||
deps,
|
||||
Arc::new(ChannelManager::new()),
|
||||
@@ -2043,6 +2044,7 @@ mod tests {
|
||||
max_tool_iterations,
|
||||
auto_approve_tools: true,
|
||||
default_timezone: "UTC".to_string(),
|
||||
max_tokens_per_job: 0,
|
||||
},
|
||||
deps,
|
||||
Arc::new(ChannelManager::new()),
|
||||
@@ -2159,6 +2161,7 @@ mod tests {
|
||||
max_tool_iterations: max_iter,
|
||||
auto_approve_tools: true,
|
||||
default_timezone: "UTC".to_string(),
|
||||
max_tokens_per_job: 0,
|
||||
},
|
||||
deps,
|
||||
Arc::new(ChannelManager::new()),
|
||||
|
||||
@@ -160,6 +160,13 @@ impl Scheduler {
|
||||
.create_job_for_user(user_id, title, description)
|
||||
.await?;
|
||||
|
||||
// Apply token budget from config, allowing per-job metadata override.
|
||||
let max_tokens = metadata
|
||||
.as_ref()
|
||||
.and_then(|m| m.get("max_tokens"))
|
||||
.and_then(|v| v.as_u64())
|
||||
.unwrap_or(self.config.max_tokens_per_job);
|
||||
|
||||
// Apply metadata if provided
|
||||
if let Some(meta) = metadata {
|
||||
self.context_manager
|
||||
@@ -169,6 +176,15 @@ impl Scheduler {
|
||||
.await?;
|
||||
}
|
||||
|
||||
// Set token budget (separate update to avoid overwriting metadata)
|
||||
if max_tokens > 0 {
|
||||
self.context_manager
|
||||
.update_context(job_id, |ctx| {
|
||||
ctx.max_tokens = max_tokens;
|
||||
})
|
||||
.await?;
|
||||
}
|
||||
|
||||
// Persist to DB before scheduling so the worker's FK references are valid
|
||||
if let Some(ref store) = self.store {
|
||||
let ctx = self.context_manager.get_context(job_id).await?;
|
||||
|
||||
+100
-3
@@ -417,7 +417,8 @@ Report when the job is complete or if you encounter issues you cannot resolve."#
|
||||
|
||||
iteration += 1;
|
||||
if iteration > max_iterations {
|
||||
self.mark_stuck("Maximum iterations exceeded").await?;
|
||||
self.mark_failed("Maximum iterations exceeded: job hit the iteration cap")
|
||||
.await?;
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
@@ -437,7 +438,8 @@ Report when the job is complete or if you encounter issues you cannot resolve."#
|
||||
"LLM rate limited during tool selection, backing off"
|
||||
);
|
||||
if consecutive_rate_limits >= MAX_CONSECUTIVE_RATE_LIMITS {
|
||||
self.mark_stuck("Persistent rate limiting").await?;
|
||||
self.mark_failed("Persistent rate limiting: exceeded retry limit")
|
||||
.await?;
|
||||
return Ok(());
|
||||
}
|
||||
self.log_event(
|
||||
@@ -467,7 +469,8 @@ Report when the job is complete or if you encounter issues you cannot resolve."#
|
||||
"LLM rate limited during respond_with_tools, backing off"
|
||||
);
|
||||
if consecutive_rate_limits >= MAX_CONSECUTIVE_RATE_LIMITS {
|
||||
self.mark_stuck("Persistent rate limiting").await?;
|
||||
self.mark_failed("Persistent rate limiting: exceeded retry limit")
|
||||
.await?;
|
||||
return Ok(());
|
||||
}
|
||||
self.log_event(
|
||||
@@ -483,6 +486,20 @@ Report when the job is complete or if you encounter issues you cannot resolve."#
|
||||
Err(e) => return Err(e.into()),
|
||||
};
|
||||
|
||||
// Track token usage from LLM call against the job budget.
|
||||
// NOTE: select_tools() also makes LLM calls but doesn't expose
|
||||
// TokenUsage; only respond_with_tools() usage is tracked here.
|
||||
let total_tokens = respond_output.usage.total() as u64;
|
||||
if total_tokens > 0
|
||||
&& let Err(msg) = self
|
||||
.context_manager()
|
||||
.update_context(self.job_id, |ctx| ctx.add_tokens(total_tokens))
|
||||
.await?
|
||||
{
|
||||
self.mark_failed(&msg).await?;
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
match respond_output.result {
|
||||
RespondResult::Text(response) => {
|
||||
// Check for explicit completion phrases. Use word-boundary
|
||||
@@ -1762,4 +1779,84 @@ mod tests {
|
||||
"Always tool should be allowed with permission"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_token_budget_exceeded_fails_job() {
|
||||
let worker = make_worker(vec![]).await;
|
||||
|
||||
// Transition to InProgress (required for mark_failed)
|
||||
worker
|
||||
.context_manager()
|
||||
.update_context(worker.job_id, |ctx| {
|
||||
ctx.transition_to(JobState::InProgress, None)
|
||||
})
|
||||
.await
|
||||
.unwrap()
|
||||
.unwrap();
|
||||
|
||||
// Set a token budget
|
||||
worker
|
||||
.context_manager()
|
||||
.update_context(worker.job_id, |ctx| {
|
||||
ctx.max_tokens = 100;
|
||||
})
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
// Simulate adding tokens that exceed the budget
|
||||
let budget_result = worker
|
||||
.context_manager()
|
||||
.update_context(worker.job_id, |ctx| ctx.add_tokens(200))
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
assert!(
|
||||
budget_result.is_err(),
|
||||
"Should return error when token budget exceeded"
|
||||
);
|
||||
|
||||
// Verify that mark_failed transitions job to Failed
|
||||
worker
|
||||
.mark_failed(&budget_result.unwrap_err())
|
||||
.await
|
||||
.unwrap();
|
||||
let ctx = worker
|
||||
.context_manager()
|
||||
.get_context(worker.job_id)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(ctx.state, JobState::Failed);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_iteration_cap_marks_failed_not_stuck() {
|
||||
let worker = make_worker(vec![]).await;
|
||||
|
||||
// Transition to InProgress (required for mark_failed)
|
||||
worker
|
||||
.context_manager()
|
||||
.update_context(worker.job_id, |ctx| {
|
||||
ctx.transition_to(JobState::InProgress, None)
|
||||
})
|
||||
.await
|
||||
.unwrap()
|
||||
.unwrap();
|
||||
|
||||
// Simulate what the execution loop does when max_iterations is exceeded
|
||||
worker
|
||||
.mark_failed("Maximum iterations exceeded: job hit the iteration cap")
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
let ctx = worker
|
||||
.context_manager()
|
||||
.get_context(worker.job_id)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(
|
||||
ctx.state,
|
||||
JobState::Failed,
|
||||
"Iteration cap should transition to Failed, not Stuck"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user