From d313f44a1977a52af023abfdfc52a378fed2c8f0 Mon Sep 17 00:00:00 2001 From: Henry Park Date: Wed, 11 Mar 2026 14:05:33 -0700 Subject: [PATCH] fix(ci): improve Claude Code review reliability (#955) The Claude review step was failing ~40% of the time because: - --allowedTools didn't include Read, Glob, Grep, Agent, causing 8-9 permission denials per run and preventing Claude from reading files or spawning the subagents the prompt required - Step 4 spawned N additional scoring agents per issue found, exhausting the 50-turn budget before the PR comment could be posted - Subagents could independently post PR comments, causing fragmented output Fix: add missing tools to --allowedTools, merge per-issue scoring into the review agents themselves, and add guardrails ensuring exactly one consolidated comment is always posted. Co-authored-by: Claude Opus 4.6 --- .github/workflows/claude-review.yml | 55 +++++++++++++++++------------ 1 file changed, 32 insertions(+), 23 deletions(-) diff --git a/.github/workflows/claude-review.yml b/.github/workflows/claude-review.yml index 3836a5f9..26c15d89 100644 --- a/.github/workflows/claude-review.yml +++ b/.github/workflows/claude-review.yml @@ -29,18 +29,36 @@ jobs: with: anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }} allowed_bots: "ironclaw-ci[bot]" - claude_args: "--max-turns 50 --model claude-haiku-4-5-20251001 --allowedTools 'Bash(gh pr comment:*),Bash(gh pr diff:*),Bash(gh pr view:*),Bash(gh pr list:*),Bash(gh issue view:*),Bash(gh issue list:*),Bash(gh search:*),Bash(git blame:*),Bash(git log:*),Bash(git diff:*)'" + claude_args: "--max-turns 50 --model claude-haiku-4-5-20251001 --allowedTools 'Read,Glob,Grep,Agent,Bash(gh pr comment:*),Bash(gh pr diff:*),Bash(gh pr view:*),Bash(gh pr list:*),Bash(gh issue view:*),Bash(gh issue list:*),Bash(gh search:*),Bash(git blame:*),Bash(git log:*),Bash(git diff:*)'" prompt: | Code review this pull request. Follow these steps precisely: - 1. Use a Haiku agent to find relevant CLAUDE.md files: the root CLAUDE.md - and any CLAUDE.md files in directories whose files this PR modifies. + 1. Find relevant CLAUDE.md files: the root CLAUDE.md and any CLAUDE.md files + in directories whose files this PR modifies. Use Glob to find them, then Read + to load their contents. - 2. Use a Haiku agent to summarize the PR change (use `gh pr diff`). + 2. Get the PR diff with `gh pr diff` and summarize the change. 3. Launch 4 parallel agents to review the change independently. Each agent should read the PR diff with `gh pr diff` and the full source files for changed - code, then return a list of issues found: + code (using Read), then return a list of issues. Each agent MUST score its + own findings inline using the severity and confidence rubric below. + + Severity levels: + - CRITICAL: security vulns, panics in prod (.unwrap/.expect), data exfiltration, race conditions + - HIGH: logic bugs, missing error handling, breaking API/schema changes + - MEDIUM: missing tests, unnecessary complexity, performance issues + - LOW: documentation gaps, naming suggestions + + Confidence scoring (0-100): + 0: False positive, doesn't stand up to scrutiny, or pre-existing issue. + 25: Might be real, but may be false positive. Stylistic issues not in CLAUDE.md. + 50: Real issue but nitpick or rare in practice. Not very important. + 75: Verified real issue, will be hit in practice. Directly impacts functionality + or explicitly mentioned in CLAUDE.md. + 100: Certain, confirmed, will happen frequently. Evidence directly confirms. + + Each agent returns findings as: [SEVERITY:CONFIDENCE] Agent 1 — Security & Safety Check for: command injection, path traversal, SSRF, XSS, auth bypass, @@ -63,22 +81,9 @@ jobs: timeouts, resource leaks (file handles, connections), large allocations in hot paths. - 4. For each issue found, launch a parallel Haiku agent to: - a. Assign a severity: - - CRITICAL: security vulns, panics in prod (.unwrap/.expect), data exfiltration, race conditions - - HIGH: logic bugs, missing error handling, breaking API/schema changes - - MEDIUM: missing tests, unnecessary complexity, performance issues - - LOW: documentation gaps, naming suggestions - b. Score confidence 0-100 (give this rubric verbatim): - 0: False positive, doesn't stand up to scrutiny, or pre-existing issue. - 25: Might be real, but may be false positive. Stylistic issues not in CLAUDE.md. - 50: Real issue but nitpick or rare in practice. Not very important. - 75: Verified real issue, will be hit in practice. Directly impacts functionality - or explicitly mentioned in CLAUDE.md. - 100: Certain, confirmed, will happen frequently. Evidence directly confirms. - - 5. Post a single comment on the PR using `gh pr comment` with this format. - If no issues were found, post "No issues found." instead: + 4. Consolidate all agent findings and post exactly one comment on the PR + using `gh pr comment` with this format. If no issues were found, + post "No issues found." instead: ### Code review @@ -93,8 +98,12 @@ jobs: You MUST use the full git SHA in links (not HEAD or branch name). Provide 1 line of context before and after each linked range. - Notes: - - Use `gh` for all GitHub interactions, not web fetch + IMPORTANT rules: + - Only YOU (the main process) may call `gh pr comment`. Agents must return + their findings to you — they must NOT post comments themselves. + - You MUST post exactly one `gh pr comment` before finishing, even if agents + fail or return empty results. If review is incomplete, post "No issues found." + - Use Read/Glob for file access, `gh` for GitHub interactions, not web fetch - Do NOT check build signal or attempt to build/test the code - Ignore pre-existing issues not introduced by this PR - Ignore issues a linter/compiler would catch (formatting, imports, types)