fix: parallelize tool call execution via JoinSet (#219) (#252)

* fix: parallelize tool call execution via JoinSet (#219)

When the LLM returns multiple tool_calls in a single response, they were
executed sequentially. This change makes both the worker and dispatcher
paths concurrent using tokio::task::JoinSet, so N independent tool calls
complete in ~max(latency) instead of sum(latency).

Worker path: migrate execute_tools_parallel from join_all to JoinSet and
route the respond_with_tools branch through the same parallel path.

Dispatcher path: restructure the while-idx loop into three phases —
preflight (sequential approval/hook checks), parallel execution via
JoinSet, and sequential post-flight processing (session recording,
auth detection, sanitization).

Also fixes a pre-existing infinite loop bug where hook rejection used
`continue` inside a `while idx` loop, skipping `idx += 1` and retrying
the same rejected tool forever.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: address PR review — ordered results, deferred auth, dedup standalone fn

- Fix auth early return skipping unrecorded tool results: defer auth
  response until after all results in the batch are recorded in session
  history and context_messages (both dispatcher and thread_ops paths)
- Fix tool results appearing out of order: collect Phase 1 hook
  rejections indexed by original position, merge with Phase 2 execution
  results, and emit all in Phase 3 in original tool_calls order
- Deduplicate execute_chat_tool: Agent method now delegates to the
  standalone function instead of duplicating 90 lines of logic
- Fix benchmark compilation: add missing session_manager arg to Agent::new

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: rustfmt alignment for CI compatibility

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: address second round of PR review comments

- Distinguish JoinError panic vs cancellation in log messages and error
  reasons across all 3 files (dispatcher, thread_ops, worker)
- Simplify deferred_auth from Option<(String, String)> to Option<String>
  since only the instructions string is used
- Add single-tool short-circuit in worker execute_tools_parallel to
  avoid JoinSet overhead for the common single-tool case

Co-Authored-By: Claude Opus 4.6 <[email protected]>

---------

Co-authored-by: Claude Opus 4.6 <[email protected]>
This commit is contained in:
Illia Polosukhin
2026-02-20 06:07:30 +00:00
committed by GitHub
co-authored by Claude Opus 4.6
parent 9906190de7
commit bfe393eb38
4 changed files with 951 additions and 331 deletions
+1
View File
@@ -91,6 +91,7 @@ impl Session {
None => self.create_thread(),
Some(id) => {
if self.threads.contains_key(&id) {
// Safe: contains_key confirmed the entry exists.
self.threads.get_mut(&id).unwrap()
} else {
// Stale active_thread ID: create a new thread, which