feat: configurable tool iterations, auto-approve, and policy fix (#251)

* feat: direct agentic loop for SWE-bench benchmarks

Replace the full Agent-based runner with a purpose-built agentic loop
that directly calls the LLM with tools. The old path routed through
SafetyLayer (which blocked SWE-bench prompts), dispatcher (capped at
10 iterations), approval flow (wasted iterations), and 20+ irrelevant
builtin tools (diluted the model's focus).

New architecture:
- AgenticLoop: LLM call -> tool execution -> repeat (up to 30 iters)
- Per-task tool scoping via BenchSuite::task_tools() with working dirs
- Suite-provided system prompts via BenchSuite::system_prompt()
- No safety layer, no approval flow, no sessions/threads overhead
- Configurable max_iterations in BenchConfig and TOML

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: apply --model CLI override to LLM provider

The --model flag was updating matrix entry labels but not the actual
LLM provider, so requests were still sent using the model from .env.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* feat: configurable tool iterations and auto-approve for benchmarks

Add max_tool_iterations and auto_approve_tools settings to AgentConfig,
replacing the hardcoded MAX_TOOL_ITERATIONS constant. Fix shell_injection
policy rule to not block markdown backtick code snippets.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: address benchmarks crate audit findings

High:
- Fix truncate_output UTF-8 panic on multi-byte char boundaries
- Fix parallel results durability (write JSONL per-task, not after all)

Medium:
- Fix --sample to use random shuffle instead of first-N
- Delegate all LlmProvider methods in InstrumentedLlm
- Fix LLM-as-judge to return fail instead of misleading 0.5
- Remove unnecessary shallow clone (always gets unshallowed)
- Replace .unwrap() with .expect() in LazyLock regex init

Low:
- Remove dead code: unused error variants, trait methods, struct fields
- Remove BenchSuite::name() (redundant with id())
- Remove TaskSubmission::conversation, ConversationTurn, TurnRole
- Remove unused methods from BenchChannel, results, config
- Clean up ChannelCapture conversation tracking

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* feat: add SWE-bench dataset and Docker scoring infrastructure

Add the SWE-bench Lite dataset (300 tasks) and Docker files for
isolated test execution and scoring of SWE-bench patches.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* chore: remove benchmarks (extracted to separate repo)

Benchmarks crate has been extracted to its own repository.
Remove the workspace member and all benchmarks/ files.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: add missing AgentConfig fields in test initializer

Co-Authored-By: Claude Opus 4.6 <[email protected]>

---------

Co-authored-by: Claude Opus 4.6 <[email protected]>

src/config/agent.rs

@@ -23,6 +23,10 @@ pub struct AgentConfig {
     pub max_cost_per_day_cents: Option<u64>,
     /// Maximum LLM/tool actions per hour. None = unlimited.
     pub max_actions_per_hour: Option<u64>,
+    /// Maximum tool-call iterations per agentic loop invocation. Default 50.
+    pub max_tool_iterations: usize,
+    /// When true, skip tool approval checks entirely. For benchmarks/CI.
+    pub auto_approve_tools: bool,
 }
 
 impl AgentConfig {
@@ -115,6 +119,22 @@ impl AgentConfig {
                     key: "MAX_ACTIONS_PER_HOUR".to_string(),
                     message: format!("must be a positive integer: {e}"),
                 })?,
+            max_tool_iterations: optional_env("AGENT_MAX_TOOL_ITERATIONS")?
+                .map(|s| s.parse())
+                .transpose()
+                .map_err(|e| ConfigError::InvalidValue {
+                    key: "AGENT_MAX_TOOL_ITERATIONS".to_string(),
+                    message: format!("must be a positive integer: {e}"),
+                })?
+                .unwrap_or(settings.agent.max_tool_iterations),
+            auto_approve_tools: optional_env("AGENT_AUTO_APPROVE_TOOLS")?
+                .map(|s| s.parse())
+                .transpose()
+                .map_err(|e| ConfigError::InvalidValue {
+                    key: "AGENT_AUTO_APPROVE_TOOLS".to_string(),
+                    message: format!("must be 'true' or 'false': {e}"),
+                })?
+                .unwrap_or(settings.agent.auto_approve_tools),
         })
     }
 }