fix: full_job routine runs stay running until linked job completion (#1374)

* fix: full_job routine runs stay running until linked job completion (#1317)

Previously, execute_full_job() returned RunStatus::Ok immediately after
dispatching the job, causing routine runs to be marked as completed before
the linked worker job had actually finished. This meant failure notifications
were never sent and max_concurrent guardrails stopped applying once the run
was prematurely finalized.

Changes:
- execute_full_job() now returns RunStatus::Running instead of Ok
- execute_routine() skips finalization for Running status (leaves run open)
- New sync_dispatched_runs() polls on each cron tick, checks linked job
  state, and finalizes runs when jobs reach terminal states
- New list_dispatched_routine_runs() DB method on both backends
- Deferred notifications are sent when the run is actually finalized
- consecutive_failures is preserved (not reset) while outcome is unknown

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* fix: address PR review feedback (watcher predicate, running_count safety)

- FullJobWatcher: use is_parallel_blocking() instead of is_active() so
  the watcher exits when a job reaches Completed (not terminal but
  finished executing). Fixes infinite-poll for routine jobs.
- Remove running_count decrement from sync_dispatched_runs() — in normal
  flow execute_routine() handles it; sync only runs for crash recovery
  where the counter is already 0.
- Update PR description to match actual FullJobWatcher behavior.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* fix: sync only at startup to prevent double-completion race

- Move sync_dispatched_runs() out of cron loop into startup-only path.
  During normal operation FullJobWatcher handles finalization inline;
  running sync on every tick would race with the watcher.
- Update complete_dispatched_run() to properly advance runtime fields
  (last_run_at, next_fire_at, run_count) for crash recovery — in that
  scenario execute_routine() never reached its runtime update.
- Fix stale doc comment on complete_dispatched_run().

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* fix: use boot_time filter for safe periodic sync of orphaned runs

- Add boot_time field to RoutineEngine, set to Utc::now() at creation.
- sync_dispatched_runs() now filters runs by started_at < boot_time,
  so it only processes orphans from a previous process — never races
  with FullJobWatcher instances from the current process.
- Move sync back into the cron loop (safe with boot_time filter) and
  run it BEFORE check_cron_triggers to avoid picking up freshly
  dispatched runs.
- Fix doc comments to match actual behavior.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>

src/db/libsql/routines.rs

@@ -476,4 +476,28 @@ impl RoutineStore for LibSqlBackend {
         .map_err(|e| DatabaseError::Query(e.to_string()))?;
         Ok(())
     }
+
+    async fn list_dispatched_routine_runs(&self) -> Result<Vec<RoutineRun>, DatabaseError> {
+        let conn = self.connect().await?;
+        let mut rows = conn
+            .query(
+                &format!(
+                    "SELECT {} FROM routine_runs WHERE status = 'running' AND job_id IS NOT NULL",
+                    ROUTINE_RUN_COLUMNS
+                ),
+                params![],
+            )
+            .await
+            .map_err(|e| DatabaseError::Query(e.to_string()))?;
+
+        let mut runs = Vec::new();
+        while let Some(row) = rows
+            .next()
+            .await
+            .map_err(|e| DatabaseError::Query(e.to_string()))?
+        {
+            runs.push(row_to_routine_run_libsql(&row)?);
+        }
+        Ok(runs)
+    }
 }

src/db/mod.rs

@@ -525,6 +525,9 @@ pub trait RoutineStore: Send + Sync {
         run_id: Uuid,
         job_id: Uuid,
     ) -> Result<(), DatabaseError>;
+    /// List routine runs that were dispatched as full_job but have not yet
+    /// been finalized (status='running' with a linked job_id).
+    async fn list_dispatched_routine_runs(&self) -> Result<Vec<RoutineRun>, DatabaseError>;
 }
 
 #[async_trait]

src/db/postgres.rs

@@ -503,6 +503,10 @@ impl RoutineStore for PgBackend {
     ) -> Result<(), DatabaseError> {
         self.store.link_routine_run_to_job(run_id, job_id).await
     }
+
+    async fn list_dispatched_routine_runs(&self) -> Result<Vec<RoutineRun>, DatabaseError> {
+        self.store.list_dispatched_routine_runs().await
+    }
 }
 
 // ==================== ToolFailureStore ====================