feat: configurable tool iterations, auto-approve, and policy fix (#251)

* feat: direct agentic loop for SWE-bench benchmarks

Replace the full Agent-based runner with a purpose-built agentic loop
that directly calls the LLM with tools. The old path routed through
SafetyLayer (which blocked SWE-bench prompts), dispatcher (capped at
10 iterations), approval flow (wasted iterations), and 20+ irrelevant
builtin tools (diluted the model's focus).

New architecture:
- AgenticLoop: LLM call -> tool execution -> repeat (up to 30 iters)
- Per-task tool scoping via BenchSuite::task_tools() with working dirs
- Suite-provided system prompts via BenchSuite::system_prompt()
- No safety layer, no approval flow, no sessions/threads overhead
- Configurable max_iterations in BenchConfig and TOML

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: apply --model CLI override to LLM provider

The --model flag was updating matrix entry labels but not the actual
LLM provider, so requests were still sent using the model from .env.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* feat: configurable tool iterations and auto-approve for benchmarks

Add max_tool_iterations and auto_approve_tools settings to AgentConfig,
replacing the hardcoded MAX_TOOL_ITERATIONS constant. Fix shell_injection
policy rule to not block markdown backtick code snippets.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: address benchmarks crate audit findings

High:
- Fix truncate_output UTF-8 panic on multi-byte char boundaries
- Fix parallel results durability (write JSONL per-task, not after all)

Medium:
- Fix --sample to use random shuffle instead of first-N
- Delegate all LlmProvider methods in InstrumentedLlm
- Fix LLM-as-judge to return fail instead of misleading 0.5
- Remove unnecessary shallow clone (always gets unshallowed)
- Replace .unwrap() with .expect() in LazyLock regex init

Low:
- Remove dead code: unused error variants, trait methods, struct fields
- Remove BenchSuite::name() (redundant with id())
- Remove TaskSubmission::conversation, ConversationTurn, TurnRole
- Remove unused methods from BenchChannel, results, config
- Clean up ChannelCapture conversation tracking

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* feat: add SWE-bench dataset and Docker scoring infrastructure

Add the SWE-bench Lite dataset (300 tasks) and Docker files for
isolated test execution and scoring of SWE-bench patches.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* chore: remove benchmarks (extracted to separate repo)

Benchmarks crate has been extracted to its own repository.
Remove the workspace member and all benchmarks/ files.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: add missing AgentConfig fields in test initializer

Co-Authored-By: Claude Opus 4.6 <[email protected]>

---------

Co-authored-by: Claude Opus 4.6 <[email protected]>

src/safety/policy.rs

@@ -159,11 +159,13 @@ impl Default for Policy {
             PolicyAction::Warn,
         ));
 
-        // Block shell command injection patterns
+        // Block shell command injection patterns.
+        // Only match actual dangerous command sequences, NOT backticked content
+        // (backticks are standard markdown code formatting, not shell injection).
         policy.add_rule(PolicyRule::new(
             "shell_injection",
             "Potential shell command injection",
-            r"(?i)(;\s*rm\s+-rf|;\s*curl\s+.*\|\s*sh|`.*`)",
+            r"(?i)(;\s*rm\s+-rf|;\s*curl\s+.*\|\s*sh)",
             Severity::Critical,
             PolicyAction::Block,
         ));
@@ -233,6 +235,17 @@ mod tests {
         assert!(violations.iter().any(|r| r.action == PolicyAction::Warn));
     }
 
+    #[test]
+    fn test_backticked_code_is_not_blocked() {
+        let policy = Policy::default();
+        // Markdown code snippets should never be blocked
+        assert!(!policy.is_blocked("Use `print('hello')` to debug"));
+        assert!(!policy.is_blocked("Run `pytest tests/` to check"));
+        assert!(!policy.is_blocked("The error is in `foo.bar.baz`"));
+        // Multi-backtick code fences should also pass
+        assert!(!policy.is_blocked("```python\ndef foo():\n    pass\n```"));
+    }
+
     #[test]
     fn test_severity_ordering() {
         assert!(Severity::Critical > Severity::High);