perf: build system prompt once per turn, skip tools on force-text (#583)

* perf: build system prompt once per turn, skip tools on force-text, fix nudge role (#565)

Three fixes to agentic loop prompt handling:

1. Build system prompt once per turn instead of every tool iteration.
   `build_system_prompt_with_tools` is now pub; callers pass the result
   via `ReasoningContext::system_prompt` to avoid rebuilding ~1,500 tokens
   per iteration.

2. Skip `## Available Tools` section when `force_text = true`. The
   dispatcher passes a no-tools prompt variant on the final iteration,
   saving ~460 tokens and removing misleading instructions.

3. Change nudge message from `Role::System` to `Role::User`. A second
   system message mid-conversation is unsupported by most providers.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* fix: revert nudge role change to keep ChatMessage::system

Copilot review correctly identified that using Role::User for the nudge
breaks compact_messages_for_retry, which uses rposition for Role::User
to find the last real user message. Role::Assistant would cause
back-to-back assistant messages. Since no production issues were reported
with the original system role, revert to ChatMessage::system.

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* fix: address PR review — omit tool guidance when tools empty, rename shadowed var

- Conditionalize "Call tools…" guidelines and "## Tool Call Style" section
  in the system prompt so they are only included when tools are non-empty.
  Previously the force-text (no-tools) prompt still contained misleading
  tool-calling instructions. (Copilot review comment)

- Rename `system_prompt` → `cached_prompt` in dispatcher to avoid shadowing
  the earlier workspace identity `system_prompt` variable. (Copilot review)

- Add regression tests: `test_system_prompt_with_tools_contains_tool_guidance`
  and extended assertions in `test_system_prompt_without_tools_omits_tools_section`.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>
Co-authored-by: [email protected] <[email protected]>
This commit is contained in:
Henry Park
2026-03-07 09:15:00 +00:00
committed by GitHub
co-authored by Claude Opus 4.6 [email protected] <[email protected]>
parent 424a0366a9
commit 30790439ee
14 changed files with 400 additions and 321 deletions
+20 -2
View File
@@ -131,6 +131,17 @@ impl Agent {
JobContext::with_user(&message.user_id, "chat", "Interactive chat session");
job_ctx.http_interceptor = self.deps.http_interceptor.clone();
// Build system prompts once for this turn. Two variants: with tools
// (normal iterations) and without (force_text final iteration).
let initial_tool_defs = self.tools().tool_definitions().await;
let initial_tool_defs = if !active_skills.is_empty() {
crate::skills::attenuate_tools(&initial_tool_defs, &active_skills).tools
} else {
initial_tool_defs
};
let cached_prompt = reasoning.build_system_prompt_with_tools(&initial_tool_defs);
let cached_prompt_no_tools = reasoning.build_system_prompt_with_tools(&[]);
let max_tool_iterations = self.config.max_tool_iterations;
// Force a text-only response on the last iteration to guarantee termination
// instead of hard-erroring. The penultimate iteration also gets a nudge
@@ -208,10 +219,16 @@ impl Agent {
};
// Call LLM with current context; force_text drops tools to guarantee a
// text response on the final iteration.
// text response on the final iteration. The pre-built system prompt
// avoids rebuilding the same ~1,500-token string each iteration.
let mut context = ReasoningContext::new()
.with_messages(context_messages.clone())
.with_tools(tool_defs)
.with_system_prompt(if force_text {
cached_prompt_no_tools.clone()
} else {
cached_prompt.clone()
})
.with_metadata({
let mut m = std::collections::HashMap::new();
m.insert("thread_id".to_string(), thread_id.to_string());
@@ -248,7 +265,7 @@ impl Agent {
// Compact: keep system messages + last user message + current turn
context_messages = compact_messages_for_retry(&context_messages);
// Rebuild context with compacted messages
// Rebuild context with compacted messages, reusing cached prompt
let mut retry_context = ReasoningContext::new()
.with_messages(context_messages.clone())
.with_tools(if force_text {
@@ -258,6 +275,7 @@ impl Agent {
})
.with_metadata(context.metadata.clone());
retry_context.force_text = force_text;
retry_context.system_prompt = context.system_prompt.clone();
reasoning
.respond_with_tools(&retry_context)