feat(agent): queue and merge messages during active turns (#1412)

* feat(agent): queue and merge messages during active turns Replace the hard rejection ("Turn in progress") when messages arrive during an active turn with a bounded queue (max 10) that auto-drains after the turn completes. Queued messages are merged with newlines into a single turn so the LLM receives full context from rapid consecutive inputs instead of producing fragmented responses from partial context. Key changes: - Thread.pending_messages (VecDeque) with queue_message/drain_pending_messages - Drain loop in agent_loop.rs merges all queued messages per iteration - interrupt() and /clear both clear the pending queue - MAX_PENDING_MESSAGES constant with cap enforced inside queue_message() - Drain loop continues on soft errors, stops on NeedApproval/Interrupted - Drain loop logs respond() failures instead of silently swallowing them Fixes #259 — debounces rapid inbound messages during processing Fixes #826 — drain loop is bounded by MAX_PENDING_MESSAGES cap Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> * fix: address PR review — drain loop busy-loop guard and stale state re-check - Add Ok(SubmissionResult::Ok) to drain loop break conditions to prevent a tight busy-loop if process_user_input returns a queued-ack (e.g. from a corrupted/hydrated session stuck in Processing state) - Re-check thread.state under the mutable lock in the Processing arm to guard against the turn completing between the snapshot read and the queue operation Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> * fix: clear attachments on drain-loop queued message processing Queued messages are text-only (queued as strings during Processing state). The drain loop was reusing the original IncomingMessage reference which carried the first message's attachments, causing augment_with_attachments to incorrectly re-apply them to unrelated queued text. Clone the message with cleared attachments for drain-loop turns. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> * fix: address PR review round 2 — stale state fallthrough and thread-not-found guard - Processing arm: when re-checked state is no longer Processing, fall through to normal processing instead of dropping user input - Processing arm: return error when thread not found instead of false "queued" ack - Document intermediate drain-loop responses as best-effort for one-shot channels (HttpChannel) - Add regression tests for both edge cases Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> * fix: address PR review feedback for message queue drain loop [skip-regression-check] — test modifications present but hook has SIGPIPE/pipefail false negative when awk exits early on match - Replace wildcard match in drain loop with explicit `while let Ok(Response)` guard — stops on Error variant too, preventing confusing interleaved output after soft errors (review issue #1) - Reject queueing messages with attachments during Processing state instead of silently dropping them (review issue #2) - Document response routing limitation: all drain-loop responses route via original message identity (review issue #3) - Document why SubmissionResult::Ok is correct for queued ack and how it interacts with drain loop break condition (review issue #4) - Rewrite two dead regression tests to assert actual behavior: thread-gone returns error, state-changed does not queue (review #5) - Document MAX_PENDING_MESSAGES=10 as acceptable for personal assistant use case (review issue #6) - Fix misleading one-shot channel comment — HttpChannel consumes sender on first call, subsequent calls are dropped (review issue #8) - Simplify drain loop intermediate response since while-let guard guarantees Response variant Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> * fix: add missing extension_manager field in webhook EngineContext The fire_webhook method's EngineContext initializer was missing the extension_manager field added in staging, causing CI compilation failure. [skip-regression-check] Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> * fix: gate TestRig::session_manager() behind libsql feature flag The field is #[cfg(feature = "libsql")] so the accessor must match. All callers are already inside #[cfg(feature = "libsql")] blocks. [skip-regression-check] Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> * fix: re-queue drained messages on drain loop failure If process_user_input fails after drain_pending_messages() removed all queued content, that user input was permanently lost. Now the merged content is re-queued at the front of pending_messages on any non-Response result so it will be processed on the next successful turn. Adds Thread::requeue_drained() helper and unit test. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> * fix: remove unreachable!() from drain loop, add lock-drop comments - Extract content binding in `while let` pattern instead of using a separate match with unreachable!() — satisfies the no-panic-in- production convention (zmanian review item #1) - Add comment clarifying session lock is dropped at Processing arm boundary before fall-through (zmanian review item #5) - Document bounded cap overshoot on requeue_drained (review item #2) [skip-regression-check] Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> * fix(security): validate queued messages and touch updated_at on queue ops - Run safety validation, policy checks, and secret scanning on messages before queueing during Processing state. Previously, content with leaked secrets could be stored in pending_messages and serialized without hitting the inbound scanner. - Touch updated_at in queue_message(), drain_pending_messages(), and requeue_drained() so thread timestamps reflect queue activity. [skip-regression-check] — safety validation requires full Agent; updated_at is a data-level fix on existing tested methods Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> --------- Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>
2026-08-25 14:53:34 +00:00 · 2026-03-21 21:53:14 -07:00
parent 89394ebd29
commit ccdea40e9d
6 changed files with 702 additions and 16 deletions
@@ -707,7 +707,115 @@ mod advanced {
    }

    // -----------------------------------------------------------------------
-    // 9. Bootstrap greeting fires on fresh workspace
+    // 9. Message queue during tool execution
+    //
+    // Verifies that messages queued on a thread's pending_messages are
+    // auto-processed by the drain loop after the current turn completes.
+    // -----------------------------------------------------------------------
+
+    #[tokio::test]
+    async fn message_queue_drains_after_tool_turn() {
+        let trace =
+            LlmTrace::from_file(format!("{FIXTURES}/message_queue_during_tools.json")).unwrap();
+        let rig = TestRigBuilder::new()
+            .with_trace(trace.clone())
+            .build()
+            .await;
+
+        // Turn 1: Send initial message to establish the session and thread.
+        rig.send_message("Echo hello for me").await;
+        let r1 = rig.wait_for_responses(1, TIMEOUT).await;
+        assert!(!r1.is_empty(), "Turn 1: no response");
+        assert!(
+            r1[0].content.to_lowercase().contains("hello"),
+            "Turn 1: missing 'hello' in: {}",
+            r1[0].content,
+        );
+
+        // Verify the echo tool was used in turn 1.
+        let started = rig.tool_calls_started();
+        assert!(
+            started.iter().any(|s| s == "echo"),
+            "Turn 1: echo tool not called: {started:?}",
+        );
+
+        // Pre-populate the thread's pending_messages queue.
+        // This simulates what happens when a concurrent request (e.g. gateway
+        // POST) arrives while the thread is in Processing state.
+        {
+            let session = rig
+                .session_manager()
+                .get_or_create_session("test-user")
+                .await;
+            let mut sess = session.lock().await;
+            // Find the active thread and queue a message.
+            let thread = sess
+                .active_thread
+                .and_then(|tid| sess.threads.get_mut(&tid))
+                .expect("active thread should exist after turn 1");
+            thread.queue_message("What is 2+2?".to_string());
+            assert_eq!(thread.pending_messages.len(), 1);
+        }
+
+        // Turn 2: Send a message that triggers tool calls.
+        // After this turn completes, the drain loop should find "What is 2+2?"
+        // in pending_messages and process it automatically.
+        rig.send_message("Now echo world and check the time").await;
+
+        // Wait for 3 total responses:
+        //   r1 = turn 1 response ("hello")
+        //   r2 = turn 2 response ("echo world + time") — sent inline by drain loop
+        //   r3 = queued message response ("2+2 = 4") — processed by drain loop
+        let all = rig.wait_for_responses(3, TIMEOUT).await;
+        assert!(
+            all.len() >= 3,
+            "Expected 3 responses (turn1 + turn2 + queued), got {}:\n{:?}",
+            all.len(),
+            all.iter().map(|r| &r.content).collect::<Vec<_>>(),
+        );
+
+        // The third response should be from the queued message ("What is 2+2?")
+        let queued_response = &all[2].content;
+        assert!(
+            queued_response.contains("4"),
+            "Queued message response should contain '4', got: {queued_response}",
+        );
+
+        // Verify the pending queue was fully drained.
+        {
+            let session = rig
+                .session_manager()
+                .get_or_create_session("test-user")
+                .await;
+            let sess = session.lock().await;
+            let thread = sess
+                .active_thread
+                .and_then(|tid| sess.threads.get(&tid))
+                .expect("active thread should still exist");
+            assert!(
+                thread.pending_messages.is_empty(),
+                "Pending queue should be empty after drain, got: {:?}",
+                thread.pending_messages,
+            );
+        }
+
+        // Verify tool usage across all turns.
+        let all_started = rig.tool_calls_started();
+        let echo_count = all_started.iter().filter(|s| *s == "echo").count();
+        assert_eq!(
+            echo_count, 2,
+            "Expected 2 echo calls (turn 1 + turn 2), got {echo_count}",
+        );
+        assert!(
+            all_started.iter().any(|s| s == "time"),
+            "time tool should have been called in turn 2: {all_started:?}",
+        );
+
+        rig.shutdown();
+    }
+
+    // -----------------------------------------------------------------------
+    // 10. Bootstrap greeting fires on fresh workspace
    // -----------------------------------------------------------------------

    /// Verifies that a fresh workspace triggers a static bootstrap greeting
@@ -740,7 +848,7 @@ mod advanced {
    }

    // -----------------------------------------------------------------------
-    // 10. Bootstrap onboarding completes and clears BOOTSTRAP.md
+    // 11. Bootstrap onboarding completes and clears BOOTSTRAP.md
    // -----------------------------------------------------------------------

    /// Exercises the full onboarding flow: bootstrap greeting fires, user
@@ -0,0 +1,104 @@
+{
+  "model_name": "advanced-message-queue-during-tools",
+  "turns": [
+    {
+      "user_input": "Echo hello for me",
+      "steps": [
+        {
+          "request_hint": { "last_user_message_contains": "Echo hello" },
+          "response": {
+            "type": "tool_calls",
+            "tool_calls": [
+              {
+                "id": "call_echo_setup",
+                "name": "echo",
+                "arguments": { "message": "hello" }
+              }
+            ],
+            "input_tokens": 80,
+            "output_tokens": 20
+          }
+        },
+        {
+          "response": {
+            "type": "text",
+            "content": "I echoed hello for you. The tool returned: hello",
+            "input_tokens": 120,
+            "output_tokens": 25
+          }
+        }
+      ],
+      "expects": {
+        "tools_used": ["echo"],
+        "all_tools_succeeded": true,
+        "response_contains": ["hello"]
+      }
+    },
+    {
+      "user_input": "Now echo world and check the time",
+      "steps": [
+        {
+          "request_hint": { "last_user_message_contains": "echo world" },
+          "response": {
+            "type": "tool_calls",
+            "tool_calls": [
+              {
+                "id": "call_echo_main",
+                "name": "echo",
+                "arguments": { "message": "world" }
+              }
+            ],
+            "input_tokens": 160,
+            "output_tokens": 20
+          }
+        },
+        {
+          "response": {
+            "type": "tool_calls",
+            "tool_calls": [
+              {
+                "id": "call_time_main",
+                "name": "time",
+                "arguments": {}
+              }
+            ],
+            "input_tokens": 200,
+            "output_tokens": 15
+          }
+        },
+        {
+          "response": {
+            "type": "text",
+            "content": "Done! I echoed world and checked the time for you.",
+            "input_tokens": 250,
+            "output_tokens": 20
+          }
+        }
+      ],
+      "expects": {
+        "tools_used": ["echo", "time"],
+        "all_tools_succeeded": true
+      }
+    },
+    {
+      "user_input": "What is 2+2?",
+      "steps": [
+        {
+          "response": {
+            "type": "text",
+            "content": "2+2 equals 4.",
+            "input_tokens": 80,
+            "output_tokens": 10
+          }
+        }
+      ],
+      "expects": {
+        "response_contains": ["4"]
+      }
+    }
+  ],
+  "expects": {
+    "tools_used": ["echo", "time"],
+    "min_responses": 3
+  }
+}
@@ -53,6 +53,9 @@ pub struct TestRig {
    /// Extension manager for direct extension operations in tests.
    #[cfg(feature = "libsql")]
    extension_manager: Option<Arc<ironclaw::extensions::ExtensionManager>>,
+    /// Session manager for direct session/thread access in tests.
+    #[cfg(feature = "libsql")]
+    session_manager: Arc<ironclaw::agent::SessionManager>,
    /// Temp directory guard -- keeps the libSQL database file alive.
    #[cfg(feature = "libsql")]
    _temp_dir: tempfile::TempDir,
@@ -84,6 +87,12 @@ impl TestRig {
        self.extension_manager.as_ref()
    }

+    /// Return the session manager for direct session/thread access in tests.
+    #[cfg(feature = "libsql")]
+    pub fn session_manager(&self) -> &Arc<ironclaw::agent::SessionManager> {
+        &self.session_manager
+    }
+
    /// Wait until at least `n` responses have been captured, or `timeout` elapses.
    pub async fn wait_for_responses(&self, n: usize, timeout: Duration) -> Vec<OutgoingResponse> {
        self.channel.wait_for_responses(n, timeout).await
@@ -736,6 +745,7 @@ impl TestRigBuilder {
        let db_ref = components.db.clone().expect("test rig requires a database");
        let workspace_ref = components.workspace.clone();
        let ext_mgr_ref = components.extension_manager.clone();
+        let session_manager_ref = Arc::new(ironclaw::agent::SessionManager::new());

        // 7. Construct AgentDeps from AppComponents (mirrors main.rs).
        let deps = AgentDeps {
@@ -800,7 +810,7 @@ impl TestRigBuilder {
            None, // hygiene_config
            routine_config,
            Some(Arc::clone(&components.context_manager)),
-            None, // session_manager
+            Some(Arc::clone(&session_manager_ref)),
        );

        // Match main.rs: fill the scheduler slot once Agent::new has created it.
@@ -828,6 +838,7 @@ impl TestRigBuilder {
            workspace: workspace_ref,
            trace_llm: trace_llm_ref,
            extension_manager: ext_mgr_ref,
+            session_manager: session_manager_ref,
            _temp_dir: temp_dir,
        }
    }