feat: configurable tool iterations, auto-approve, and policy fix (#251)

* feat: direct agentic loop for SWE-bench benchmarks

Replace the full Agent-based runner with a purpose-built agentic loop
that directly calls the LLM with tools. The old path routed through
SafetyLayer (which blocked SWE-bench prompts), dispatcher (capped at
10 iterations), approval flow (wasted iterations), and 20+ irrelevant
builtin tools (diluted the model's focus).

New architecture:
- AgenticLoop: LLM call -> tool execution -> repeat (up to 30 iters)
- Per-task tool scoping via BenchSuite::task_tools() with working dirs
- Suite-provided system prompts via BenchSuite::system_prompt()
- No safety layer, no approval flow, no sessions/threads overhead
- Configurable max_iterations in BenchConfig and TOML

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: apply --model CLI override to LLM provider

The --model flag was updating matrix entry labels but not the actual
LLM provider, so requests were still sent using the model from .env.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* feat: configurable tool iterations and auto-approve for benchmarks

Add max_tool_iterations and auto_approve_tools settings to AgentConfig,
replacing the hardcoded MAX_TOOL_ITERATIONS constant. Fix shell_injection
policy rule to not block markdown backtick code snippets.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: address benchmarks crate audit findings

High:
- Fix truncate_output UTF-8 panic on multi-byte char boundaries
- Fix parallel results durability (write JSONL per-task, not after all)

Medium:
- Fix --sample to use random shuffle instead of first-N
- Delegate all LlmProvider methods in InstrumentedLlm
- Fix LLM-as-judge to return fail instead of misleading 0.5
- Remove unnecessary shallow clone (always gets unshallowed)
- Replace .unwrap() with .expect() in LazyLock regex init

Low:
- Remove dead code: unused error variants, trait methods, struct fields
- Remove BenchSuite::name() (redundant with id())
- Remove TaskSubmission::conversation, ConversationTurn, TurnRole
- Remove unused methods from BenchChannel, results, config
- Clean up ChannelCapture conversation tracking

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* feat: add SWE-bench dataset and Docker scoring infrastructure

Add the SWE-bench Lite dataset (300 tasks) and Docker files for
isolated test execution and scoring of SWE-bench patches.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* chore: remove benchmarks (extracted to separate repo)

Benchmarks crate has been extracted to its own repository.
Remove the workspace member and all benchmarks/ files.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: add missing AgentConfig fields in test initializer

Co-Authored-By: Claude Opus 4.6 <[email protected]>

---------

Co-authored-by: Claude Opus 4.6 <[email protected]>

src/settings.rs

@@ -302,6 +302,14 @@ pub struct AgentSettings {
     /// longer than this are pruned from memory.
     #[serde(default = "default_session_idle_timeout")]
     pub session_idle_timeout_secs: u64,
+
+    /// Maximum tool-call iterations per agentic loop invocation (default: 50).
+    #[serde(default = "default_max_tool_iterations")]
+    pub max_tool_iterations: usize,
+
+    /// When true, skip tool approval checks entirely. For benchmarks/CI.
+    #[serde(default)]
+    pub auto_approve_tools: bool,
 }
 
 fn default_agent_name() -> String {
@@ -332,6 +340,10 @@ fn default_max_repair_attempts() -> u32 {
     3
 }
 
+fn default_max_tool_iterations() -> usize {
+    50
+}
+
 fn default_true() -> bool {
     true
 }
@@ -347,6 +359,8 @@ impl Default for AgentSettings {
             repair_check_interval_secs: default_repair_interval(),
             max_repair_attempts: default_max_repair_attempts(),
             session_idle_timeout_secs: default_session_idle_timeout(),
+            max_tool_iterations: default_max_tool_iterations(),
+            auto_approve_tools: false,
         }
     }
 }