mirror of
https://github.com/outbackdingo/optimclaw.git
synced 2026-08-25 14:53:34 +00:00
Compare commits
51
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
f714b0a3dd | ||
|
|
83950d11a4 | ||
|
|
764be8547f | ||
|
|
7de639e782 | ||
|
|
a5f88b32fd | ||
|
|
7d8576a464 | ||
|
|
f4b7309523 | ||
|
|
577e26eff4 | ||
|
|
bcbdc273a5 | ||
|
|
c541220ea4 | ||
|
|
14aadd3063 | ||
|
|
45923ef360 | ||
|
|
fcb152e408 | ||
|
|
e86b372fa6 | ||
|
|
63f140d391 | ||
|
|
ab0a2e05de | ||
|
|
290d925c7f | ||
|
|
d73e35cfb0 | ||
|
|
30d81fcdee | ||
|
|
d8dcc34319 | ||
|
|
652f30a826 | ||
|
|
98e9a40762 | ||
|
|
553c306c52 | ||
|
|
7fb2f47999 | ||
|
|
02f85a8ad5 | ||
|
|
9401ab0d58 | ||
|
|
7d1461fc74 | ||
|
|
605a4ba46e | ||
|
|
fe91ba2ab4 | ||
|
|
da2569bb77 | ||
|
|
732b3ecfeb | ||
|
|
461d7712e8 | ||
|
|
1c5117eded | ||
|
|
33b02eabb7 | ||
|
|
068ad2d4b7 | ||
|
|
56b7218897 | ||
|
|
200aed16cd | ||
|
|
4c0275bcdc | ||
|
|
272d31797e | ||
|
|
edff54b0b1 | ||
|
|
4d61d3eedf | ||
|
|
df3635d6be | ||
|
|
a20e19ab16 | ||
|
|
3b57d5bec9 | ||
|
|
11c5e25422 | ||
|
|
12ba79ffc3 | ||
|
|
d3cf637d4a | ||
|
|
b6cf2a6b73 | ||
|
|
9851f2a6ae | ||
|
|
8dc4ca5a98 | ||
|
|
9f71bd0d44 |
@@ -0,0 +1,303 @@
|
||||
---
|
||||
description: Full PR lifecycle — review, fix findings, address comments, quality gate, push, CI fix loop, merge
|
||||
disable-model-invocation: true
|
||||
allowed-tools: Bash(gh pr view:*), Bash(gh pr diff:*), Bash(gh pr comment:*), Bash(gh pr merge:*), Bash(gh pr checks:*), Bash(gh pr edit:*), Bash(gh pr list:*), Bash(gh pr checkout:*), Bash(gh api:*), Bash(gh repo view:*), Bash(gh run view:*), Bash(gh run watch:*), Bash(git diff:*), Bash(git log:*), Bash(git fetch:*), Bash(git checkout:*), Bash(git status:*), Bash(git branch:*), Bash(git add:*), Bash(git commit:*), Bash(git push:*), Bash(git merge:*), Bash(git rebase:*), Bash(cargo fmt:*), Bash(cargo clippy:*), Bash(cargo test:*), Bash(cargo check:*), Read, Edit, Write, Grep, Glob, Agent
|
||||
argument-hint: "<pr-number or url> [--fix] [--merge] [--review-only]"
|
||||
---
|
||||
|
||||
# PR Shepherd
|
||||
|
||||
Full PR lifecycle: review → fix → quality gate → push → CI → merge.
|
||||
|
||||
Parse `$ARGUMENTS`:
|
||||
- Extract PR number from bare number or `https://github.com/owner/repo/pull/123` URL.
|
||||
- Flags: `--fix` (auto-fix without asking), `--merge` (merge when CI green), `--review-only` (stop after review, don't fix).
|
||||
- If no PR number, detect from current branch: `gh pr list --head $(git branch --show-current) --json number --jq '.[0].number'`
|
||||
- If still nothing, stop and ask the user.
|
||||
|
||||
---
|
||||
|
||||
## Phase 1: Situational Awareness
|
||||
|
||||
Gather everything in parallel:
|
||||
|
||||
**PR metadata:**
|
||||
```
|
||||
gh pr view {number} --json number,title,body,author,baseRefName,headRefName,headRefOid,state,isDraft,mergeable,mergeStateStatus,files,additions,deletions,labels,reviewRequests
|
||||
```
|
||||
|
||||
**Diff:**
|
||||
```
|
||||
gh pr diff {number}
|
||||
gh pr diff {number} --name-only
|
||||
```
|
||||
|
||||
**CI status:**
|
||||
```
|
||||
gh pr checks {number} --json name,status,conclusion,detailsUrl
|
||||
```
|
||||
|
||||
**Review comments (human + bot):**
|
||||
```
|
||||
gh api --paginate repos/{owner}/{repo}/pulls/{number}/comments
|
||||
gh api --paginate repos/{owner}/{repo}/pulls/{number}/reviews
|
||||
```
|
||||
|
||||
Resolve `{owner}/{repo}`:
|
||||
```
|
||||
gh repo view --json owner,name --jq '"\(.owner.login)/\(.name)"'
|
||||
```
|
||||
|
||||
Save `headRefOid` — needed for posting line comments later.
|
||||
|
||||
**Assess the situation and print a status card:**
|
||||
|
||||
```
|
||||
PR #{number}: {title}
|
||||
Author: {author} Base: {base} ← {head}
|
||||
Size: +{additions} -{deletions} across {file_count} files
|
||||
CI: {PASS|FAIL|PENDING|NONE} Mergeable: {yes|no|conflict}
|
||||
Reviews: {N approved, N changes_requested, N comments-only, N bot-only}
|
||||
Unresolved comments: {N}
|
||||
Draft: {yes|no}
|
||||
```
|
||||
|
||||
**Decide the mode** based on situation:
|
||||
- **Has unresolved review comments** → Phase 2a (address comments first, then review remaining)
|
||||
- **No reviews yet / bot-only reviews** → Phase 2b (full deep review)
|
||||
- **CI failing, no review issues** → Phase 4 (jump to CI fix)
|
||||
- **Everything green + approved** → Phase 6 (ready to merge)
|
||||
|
||||
---
|
||||
|
||||
## Phase 2a: Address Existing Review Comments
|
||||
|
||||
For each unresolved review comment or review with CHANGES_REQUESTED:
|
||||
|
||||
1. **Read the referenced code** at the file and line mentioned. Never assess without reading.
|
||||
2. **Classify each comment:**
|
||||
- ✅ **Valid & unresolved** — needs a code fix
|
||||
- ✅ **Already fixed** — a later commit addressed it
|
||||
- ❌ **False positive** — explain why the code is correct
|
||||
- 🔧 **Nit** — optional improvement, not blocking
|
||||
|
||||
3. **Deduplicate** — bots (Copilot, Gemini) often post the same finding. Group by actual issue.
|
||||
|
||||
Present a table:
|
||||
|
||||
| # | Source | File:Line | Issue | Status | Planned Fix |
|
||||
|---|--------|-----------|-------|--------|-------------|
|
||||
|
||||
Wait for user confirmation (unless `--fix` flag set), then proceed to Phase 3.
|
||||
|
||||
---
|
||||
|
||||
## Phase 2b: Deep Review (6 Lenses)
|
||||
|
||||
Read EVERY changed file in full (not just diff hunks). For PRs touching >20 files, prioritize: service logic > handlers > types > tests > docs. Batch reads in parallel via Agent tool.
|
||||
|
||||
### IronClaw-specific checks (always)
|
||||
- No `.unwrap()` or `.expect()` in production code
|
||||
- Prefer `crate::` for cross-module imports (`super::` OK in tests/intra-module)
|
||||
- Error types use `thiserror`
|
||||
- If persistence touched, both backends updated (postgres.rs AND libsql/)
|
||||
- New tools implement `Tool` trait correctly and registered
|
||||
- External tool output passes through safety layer
|
||||
- Tool parameters redacted before logging/SSE
|
||||
- No byte-index slicing on external strings
|
||||
- Case-insensitive comparisons where needed
|
||||
|
||||
### Correctness
|
||||
Off-by-one, wrong operators, inverted conditions, unreachable code, type confusion, error propagation, broken invariants, TOCTOU races.
|
||||
|
||||
### Edge cases & failure handling
|
||||
Empty/None/zero-length input, external service failures, integer boundaries, malformed/adversarial input, partial failure handling.
|
||||
|
||||
### Security (assume adversarial actors)
|
||||
Auth/authz bypass, IDOR, injection (SQL/command/log/header), data leakage in logs/errors/API responses, resource exhaustion, replay/race conditions.
|
||||
|
||||
### Test coverage
|
||||
New public functions tested? Error paths tested? Edge cases covered? Existing tests still valid?
|
||||
|
||||
### Architecture
|
||||
Follows existing patterns? Unnecessary abstractions? Duplicated logic? Clean module dependencies?
|
||||
|
||||
**Present findings as a table:**
|
||||
|
||||
| # | Severity | Category | File:Line | Finding | Suggested Fix |
|
||||
|---|----------|----------|-----------|---------|---------------|
|
||||
|
||||
Severity: Critical > High > Medium > Low > Nit
|
||||
|
||||
If `--review-only` flag is set, post findings as GitHub comments (see Phase 2c) and STOP.
|
||||
|
||||
Otherwise, ask which findings to fix (default: all Critical + High + Medium). Then proceed to Phase 3.
|
||||
|
||||
---
|
||||
|
||||
## Phase 2c: Post Review Comments on GitHub
|
||||
|
||||
For each finding the user approved (or all Critical/High/Medium if `--fix`):
|
||||
|
||||
**Line-specific findings** — post as PR review comments:
|
||||
```
|
||||
gh api repos/{owner}/{repo}/pulls/{number}/comments \
|
||||
-f body="**{Severity}**: {finding}\n\n{explanation}\n\n**Suggested fix:** {suggestion}" \
|
||||
-f path="{file}" \
|
||||
-f commit_id="{headRefOid}" \
|
||||
-F line={line} \
|
||||
-f side="RIGHT"
|
||||
```
|
||||
|
||||
**Cross-cutting/architectural findings** — post as regular PR comment:
|
||||
```
|
||||
gh pr comment {number} --body "..."
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Phase 3: Fix
|
||||
|
||||
Checkout the PR branch if not already on it (handles fork PRs automatically):
|
||||
```
|
||||
gh pr checkout {number}
|
||||
```
|
||||
|
||||
**Implement fixes** for:
|
||||
1. All approved review comment fixes (from Phase 2a)
|
||||
2. All approved review findings (from Phase 2b)
|
||||
|
||||
Follow IronClaw conventions:
|
||||
- `thiserror` for errors
|
||||
- `crate::` imports
|
||||
- No `.unwrap()` in production
|
||||
- Both DB backends if persistence touched
|
||||
- Regression test for every bug fix (enforced by commit-msg hook; bypass only with `[skip-regression-check]` if genuinely not feasible)
|
||||
|
||||
After all fixes implemented, proceed to Phase 4.
|
||||
|
||||
---
|
||||
|
||||
## Phase 4: Quality Gate
|
||||
|
||||
Run the full IronClaw shipping checklist:
|
||||
|
||||
```bash
|
||||
cargo fmt
|
||||
```
|
||||
|
||||
```bash
|
||||
cargo clippy --all --benches --tests --examples --all-features
|
||||
```
|
||||
|
||||
```bash
|
||||
cargo test --lib
|
||||
```
|
||||
|
||||
If persistence changes are present, also verify feature isolation:
|
||||
```bash
|
||||
cargo check --no-default-features --features libsql
|
||||
cargo check --all-features
|
||||
```
|
||||
|
||||
**If any step fails:** fix the issue and re-run. Do NOT proceed past a failing step. Loop up to 3 times per step. If still failing after 3 attempts, report the failure and stop.
|
||||
|
||||
---
|
||||
|
||||
## Phase 5: Commit & Push
|
||||
|
||||
Stage changed files by name (never `git add -A` — it can include unintended files):
|
||||
```bash
|
||||
git add path/to/changed/file1 path/to/changed/file2
|
||||
git commit -m "{message}"
|
||||
```
|
||||
|
||||
Commit message format:
|
||||
- For review fixes: `fix: address review findings on PR #{number}`
|
||||
- For comment responses: `fix: address review comments on PR #{number}`
|
||||
- For CI fixes: `fix: resolve CI failures on PR #{number}`
|
||||
- Include specifics in the body (which findings/comments were addressed)
|
||||
|
||||
Push:
|
||||
```bash
|
||||
git push origin {headRefName}
|
||||
```
|
||||
|
||||
**Reply to addressed review comments on GitHub.** For each comment that was fixed, reply with the commit SHA and a brief description of what was done. For false positives, reply explaining why no change was needed.
|
||||
|
||||
---
|
||||
|
||||
## Phase 6: CI Monitor & Fix Loop
|
||||
|
||||
Wait briefly for CI to start, then poll (do NOT use `--watch` as it can hang indefinitely):
|
||||
```
|
||||
gh pr checks {number} --json name,status,conclusion
|
||||
```
|
||||
|
||||
Re-check every 30 seconds, up to 10 minutes. If still pending after 10 minutes, report status and ask the user whether to keep waiting.
|
||||
|
||||
**If CI passes** → proceed to Phase 7.
|
||||
|
||||
**If CI fails** (up to 3 fix attempts):
|
||||
|
||||
1. Identify the failing check:
|
||||
```
|
||||
gh run view {run_id} --log-failed
|
||||
```
|
||||
If `--log-failed` shows nothing useful:
|
||||
```
|
||||
gh run view {run_id} --log | tail -100
|
||||
```
|
||||
|
||||
2. Diagnose and fix the failure.
|
||||
3. Re-run Phase 4 (quality gate).
|
||||
4. Commit and push (Phase 5).
|
||||
5. Go back to top of Phase 6.
|
||||
|
||||
**After 3 failed CI fix attempts:** Report what's failing and why, then stop. Don't keep looping.
|
||||
|
||||
---
|
||||
|
||||
## Phase 7: Merge Decision
|
||||
|
||||
Print final status:
|
||||
```
|
||||
PR #{number}: {title}
|
||||
CI: ✅ PASS
|
||||
Reviews: {summary}
|
||||
Findings fixed: {N}
|
||||
Comments addressed: {N}
|
||||
Commits added: {N}
|
||||
```
|
||||
|
||||
**Auto-merge conditions** (if `--merge` flag or user confirms):
|
||||
- CI is passing
|
||||
- No unresolved CHANGES_REQUESTED reviews
|
||||
- PR is not draft
|
||||
- PR is mergeable (no conflicts)
|
||||
|
||||
If all conditions met, ask the user for merge strategy:
|
||||
|
||||
"CI is green. Merge this PR? [squash/rebase/merge/no]"
|
||||
|
||||
Then execute:
|
||||
```
|
||||
gh pr merge {number} --{strategy} --delete-branch
|
||||
```
|
||||
|
||||
If any condition NOT met, report what's blocking and let the user decide.
|
||||
|
||||
---
|
||||
|
||||
## Rules
|
||||
|
||||
- **Read before judging.** Never comment on code you haven't read in full. Verify line numbers.
|
||||
- **Be specific.** "Line 42 returns 404 but should return 400 because X" not "this might have issues."
|
||||
- **Fix the pattern, not just the instance.** When fixing a bug, grep for the same pattern across `src/`.
|
||||
- **Respect the commit-msg hook.** Bug fixes need regression tests. Use `[skip-regression-check]` only if genuinely not feasible.
|
||||
- **Don't over-fix.** Only change what was flagged. Don't refactor surrounding code or add improvements beyond the review scope.
|
||||
- **Credit original authors.** If taking over someone else's PR, credit them in commits and comments.
|
||||
- **No secrets in comments.** Never include customer data, credentials, or PII in GitHub comments.
|
||||
- **Distinguish certainty.** "This IS a bug" vs "This COULD be a bug if X." Be honest.
|
||||
- **Round up severity when uncertain.** Cheaper to dismiss a false alarm than miss a real bug.
|
||||
- **Parallel where possible.** Use Agent tool for parallel file reads on large PRs. Batch `gh api` calls.
|
||||
@@ -0,0 +1,63 @@
|
||||
---
|
||||
paths:
|
||||
- "src/db/**"
|
||||
- "src/history/**"
|
||||
- "migrations/**"
|
||||
---
|
||||
# Database Rules
|
||||
|
||||
Dual-backend persistence: PostgreSQL + libSQL/Turso. **All new persistence features must support both backends.**
|
||||
|
||||
See `src/db/CLAUDE.md` for full schema, dialect differences, and libSQL limitations.
|
||||
|
||||
## Adding a New Operation
|
||||
|
||||
1. Decide which sub-trait it belongs to (`ConversationStore`, `JobStore`, `SandboxStore`, `RoutineStore`, `ToolFailureStore`, `SettingsStore`, `WorkspaceStore`) or create a new one
|
||||
2. Add the async method signature to that sub-trait in `src/db/mod.rs`
|
||||
3. Implement in `src/db/postgres.rs` (delegate to `Store`/`Repository`)
|
||||
4. Implement in `src/db/libsql/<module>.rs` (use `self.connect().await?` per operation)
|
||||
5. Add migration if needed:
|
||||
- PostgreSQL: new `migrations/VN__description.sql`
|
||||
- libSQL: add `CREATE TABLE IF NOT EXISTS` to `libsql_migrations.rs`
|
||||
6. Test feature isolation:
|
||||
```bash
|
||||
cargo check # postgres (default)
|
||||
cargo check --no-default-features --features libsql # libsql only
|
||||
cargo check --all-features # both
|
||||
```
|
||||
|
||||
## SQL Dialect Translation Checklist
|
||||
|
||||
When writing SQL for both backends, translate these types:
|
||||
|
||||
| PostgreSQL | libSQL |
|
||||
|-----------|--------|
|
||||
| `UUID` | `TEXT` |
|
||||
| `TIMESTAMPTZ` | `TEXT` (ISO-8601, write with `fmt_ts()`, read with `get_ts()`) |
|
||||
| `JSONB` | `TEXT` (JSON string) |
|
||||
| `BOOLEAN` | `INTEGER` (0/1 -- use `get_i64(row, idx) != 0` to read) |
|
||||
| `NUMERIC` | `TEXT` (preserves `rust_decimal` precision) |
|
||||
| `TEXT[]` | `TEXT` (JSON-encoded array) |
|
||||
| `VECTOR` | `BLOB` (flexible dimensions; vector index dropped, brute-force search fallback) |
|
||||
| `jsonb_set(col, '{key}', val)` | `json_patch(col, '{"key": val}')` -- replaces top-level keys entirely, cannot do partial nested updates |
|
||||
| `DEFAULT NOW()` | `DEFAULT (datetime('now'))` |
|
||||
| `tsvector` + `ts_rank_cd` | FTS5 virtual table + sync triggers |
|
||||
|
||||
## Schema Translation Beyond DDL
|
||||
|
||||
Don't just translate `CREATE TABLE`. Also check:
|
||||
- **Indexes** -- diff `CREATE INDEX` statements between backends
|
||||
- **Seed data** -- check for `INSERT INTO` in migrations (e.g., `leak_detection_patterns`)
|
||||
- **Triggers** -- PostgreSQL functions vs SQLite triggers (no stored procs in SQLite)
|
||||
|
||||
## Transaction Safety
|
||||
|
||||
Multi-step operations (INSERT+INSERT, UPDATE+DELETE, read-modify-write) MUST be wrapped in a transaction. Ask: "If this crashes between step N and N+1, is the database consistent?" If not, wrap in a transaction. Applies to both backends.
|
||||
|
||||
## libSQL Connection Model
|
||||
|
||||
`LibSqlBackend::connect()` creates a fresh connection per operation with `PRAGMA busy_timeout = 5000`. This is intentional -- no pool exists. Never hold connections open across `await` points. Satellite stores (`LibSqlSecretsStore`, `LibSqlWasmToolStore`) receive `Arc<LibSqlDatabase>` via `shared_db()` and call `.connect()` themselves -- never pass a live `Connection`.
|
||||
|
||||
## Fix the Pattern, Not the Instance
|
||||
|
||||
When fixing a bug in one backend's SQL, always grep for the same pattern in the other. A fix to `postgres.rs` that doesn't also fix `libsql/jobs.rs` is half a fix. Same applies to satellite stores.
|
||||
@@ -0,0 +1,48 @@
|
||||
---
|
||||
paths:
|
||||
- "src/**/*.rs"
|
||||
---
|
||||
# Review & Fix Discipline
|
||||
|
||||
Hard-won lessons from code review -- follow these when fixing bugs or addressing review feedback.
|
||||
|
||||
**Fix the pattern, not just the instance:** When a reviewer flags a bug (e.g., TOCTOU race in INSERT + SELECT-back), search the entire codebase for all instances of that same pattern. A fix in `SecretsStore::create()` that doesn't also fix `WasmToolStore::store()` is half a fix.
|
||||
|
||||
**Propagate architectural fixes to satellite types:** If a core type changes its concurrency model (e.g., `LibSqlBackend` switches to connection-per-operation), every type that was handed a resource from the old model must also be updated. Grep for the old type across the codebase.
|
||||
|
||||
**Schema translation is more than DDL:** When translating a database schema between backends (PostgreSQL to libSQL, etc.), check for:
|
||||
- **Indexes** -- diff `CREATE INDEX` statements between the two schemas
|
||||
- **Seed data** -- check for `INSERT INTO` in migrations (e.g., `leak_detection_patterns`)
|
||||
- **Semantic differences** -- document where SQL functions behave differently (e.g., `json_patch` vs `jsonb_set`)
|
||||
|
||||
**Feature flag testing:** When adding feature-gated code, test compilation with each feature in isolation:
|
||||
```bash
|
||||
cargo check # default features
|
||||
cargo check --no-default-features --features libsql # libsql only
|
||||
cargo check --all-features # all features
|
||||
```
|
||||
|
||||
**Regression test with every fix:** Every bug fix must include a test that would have caught the bug. Add a `#[test]` or `#[tokio::test]` that reproduces the original failure. Exempt: changes limited to `src/channels/web/static/` or `.md` files. Use `[skip-regression-check]` in commit message or PR label if genuinely not feasible. The `commit-msg` hook and CI workflow enforce this automatically.
|
||||
|
||||
**Zero clippy warnings policy:** Fix ALL clippy warnings before committing, including pre-existing ones in files you didn't change. Never leave warnings behind.
|
||||
|
||||
**Transaction safety:** Multi-step database operations (INSERT+INSERT, UPDATE+DELETE, read-then-write) MUST be wrapped in a transaction. Never assume sequential calls are atomic. This applies to both postgres and libsql backends.
|
||||
|
||||
**UTF-8 string safety:** Never use byte-index slicing (`&s[..n]`) on user-supplied or external strings -- it panics on multi-byte characters. Use `is_char_boundary()` or `char_indices()`. Grep for `[..` in changed files.
|
||||
|
||||
**Case-insensitive comparisons:** When comparing user-supplied strings (file paths, media types, extension names), normalize to lowercase with `.to_ascii_lowercase()`. Path comparisons must be case-insensitive on macOS/Windows.
|
||||
|
||||
**Decorator/wrapper trait delegation:** When adding a new method to `LlmProvider` (or any trait with decorator wrappers), update ALL wrapper types to delegate. Grep for `impl LlmProvider for` to find all implementations. Test through the full provider chain.
|
||||
|
||||
**Sensitive data in logs & events:** Tool parameters and outputs MUST be redacted before logging or broadcasting via SSE/WebSocket. Use `redact_params()` before any `tracing::info!`, `JobEvent`, or SSE emission that includes tool call data.
|
||||
|
||||
**Test temporary files:** Use the `tempfile` crate. Never hardcode `/tmp/...` paths.
|
||||
|
||||
**Trust boundaries in multi-process architecture:** Data from worker containers is untrusted. The orchestrator MUST validate: tool domain, nesting depth (server-side tracking), and parameter sensitivity.
|
||||
|
||||
**Mechanical verification before committing:**
|
||||
- `cargo clippy --all --benches --tests --examples --all-features` -- zero warnings
|
||||
- `grep -rnE '\.unwrap\(|\.expect\(' <files>` -- no panics in production
|
||||
- `grep -rn 'super::' <files>` -- prefer `crate::` for cross-module imports (`super::` OK in tests/intra-module)
|
||||
- If you fixed a pattern bug, `grep` for other instances across `src/`
|
||||
- Run `scripts/pre-commit-safety.sh` to catch UTF-8, case-sensitivity, hardcoded /tmp, and logging issues
|
||||
@@ -0,0 +1,34 @@
|
||||
---
|
||||
paths:
|
||||
- "src/safety/**"
|
||||
- "src/sandbox/**"
|
||||
- "src/secrets/**"
|
||||
- "src/tools/wasm/**"
|
||||
---
|
||||
# Safety Layer & Sandbox Rules
|
||||
|
||||
## Safety Layer
|
||||
|
||||
All external tool output passes through `SafetyLayer`:
|
||||
1. **Sanitizer** - Detects injection patterns, escapes dangerous content
|
||||
2. **Validator** - Checks length, encoding, forbidden patterns
|
||||
3. **Policy** - Rules with severity (Critical/High/Medium/Low) and actions (Block/Warn/Review/Sanitize)
|
||||
4. **Leak Detector** - Scans for 15+ secret patterns at two points: tool output before LLM, and LLM responses before user
|
||||
|
||||
Tool outputs are wrapped in `<tool_output>` XML before reaching the LLM.
|
||||
|
||||
## Shell Environment Scrubbing
|
||||
|
||||
The shell tool scrubs sensitive env vars before executing commands. The sanitizer detects command injection patterns (chained commands, subshells, path traversal).
|
||||
|
||||
## Sandbox Policies
|
||||
|
||||
| Policy | Filesystem | Network |
|
||||
|--------|-----------|---------|
|
||||
| ReadOnly | Read-only workspace | Allowlisted domains |
|
||||
| WorkspaceWrite | Read-write workspace | Allowlisted domains |
|
||||
| FullAccess | Full filesystem | Unrestricted |
|
||||
|
||||
## Zero-Exposure Credential Model
|
||||
|
||||
Secrets are stored encrypted on the host and injected into HTTP requests by the proxy at transit time. Container processes never see raw credential values.
|
||||
@@ -0,0 +1,56 @@
|
||||
---
|
||||
paths:
|
||||
- "src/skills/**"
|
||||
- "skills/**"
|
||||
---
|
||||
# Skills System
|
||||
|
||||
SKILL.md files extend the agent's prompt with domain-specific instructions. Each skill is a YAML frontmatter block (metadata, activation criteria, required tools) followed by a markdown body injected into the LLM context.
|
||||
|
||||
## Trust Model
|
||||
|
||||
| Trust Level | Source | Tool Access |
|
||||
|-------------|--------|-------------|
|
||||
| **Trusted** | User-placed in `~/.ironclaw/skills/` or workspace `skills/` | All tools available to the agent |
|
||||
| **Installed** | Downloaded from ClawHub registry (`~/.ironclaw/installed_skills/`) | Read-only tools only (no shell, file write, HTTP) |
|
||||
|
||||
## SKILL.md Format
|
||||
|
||||
```yaml
|
||||
---
|
||||
name: my-skill
|
||||
version: 0.1.0
|
||||
description: Does something useful
|
||||
activation:
|
||||
patterns:
|
||||
- "deploy to.*production"
|
||||
keywords:
|
||||
- "deployment"
|
||||
exclude_keywords:
|
||||
- "rollback"
|
||||
tags:
|
||||
- "devops"
|
||||
max_context_tokens: 2000
|
||||
metadata:
|
||||
openclaw:
|
||||
requires:
|
||||
bins: [docker, kubectl]
|
||||
env: [KUBECONFIG]
|
||||
---
|
||||
|
||||
# Skill instructions here...
|
||||
```
|
||||
|
||||
## Selection Pipeline
|
||||
|
||||
1. **Gating** -- Check binary/env/config requirements; skip skills whose prerequisites are missing
|
||||
2. **Scoring** -- Deterministic scoring: keywords (10/5 pts, cap 30) + patterns (20 pts, cap 40) + tags (3 pts, cap 15). `exclude_keywords` veto (score = 0 if any present)
|
||||
3. **Budget** -- Select top-scoring skills within `SKILLS_MAX_TOKENS` prompt budget
|
||||
4. **Attenuation** -- Minimum trust across active skills determines tool ceiling; installed skills lose dangerous tools
|
||||
|
||||
## Skill Tools
|
||||
|
||||
- `skill_list` -- List all discovered skills with trust level and status
|
||||
- `skill_search` -- Search ClawHub registry for available skills
|
||||
- `skill_install` -- Download and install a skill from ClawHub
|
||||
- `skill_remove` -- Remove an installed skill
|
||||
@@ -0,0 +1,25 @@
|
||||
---
|
||||
paths:
|
||||
- "src/**/*.rs"
|
||||
- "tests/**"
|
||||
---
|
||||
# Testing Rules
|
||||
|
||||
## Test Tiers
|
||||
|
||||
| Tier | Command | External deps |
|
||||
|------|---------|---------------|
|
||||
| Unit | `cargo test` | None |
|
||||
| Integration | `cargo test --features integration` | Running PostgreSQL |
|
||||
| Live | `cargo test --features integration -- --ignored` | PostgreSQL + LLM API keys |
|
||||
|
||||
Run `bash scripts/check-boundaries.sh` to verify test tier gating.
|
||||
|
||||
## Key Patterns
|
||||
|
||||
- Unit tests in `mod tests {}` at the bottom of each file
|
||||
- Async tests with `#[tokio::test]`
|
||||
- No mocks, prefer real implementations or stubs
|
||||
- Use `tempfile` crate for test directories, never hardcode `/tmp/`
|
||||
- Regression test with every bug fix (enforced by commit-msg hook)
|
||||
- Integration tests (`--test workspace_integration`) require PostgreSQL; skipped if DB is unreachable
|
||||
@@ -0,0 +1,39 @@
|
||||
---
|
||||
paths:
|
||||
- "src/tools/**"
|
||||
- "tools-src/**"
|
||||
---
|
||||
# Tool Architecture
|
||||
|
||||
**Keep tool-specific logic out of the main agent codebase.** The main agent provides generic infrastructure; tools are self-contained units that declare requirements through `<name>.capabilities.json` sidecar files (in dev mode: `tools-src/<name>/<name>-tool.capabilities.json`).
|
||||
|
||||
Tools can be WASM (sandboxed, credential-injected, single binary) or MCP servers (ecosystem, any language, no sandbox). Both are first-class via `ironclaw tool install`.
|
||||
|
||||
See `src/tools/README.md` for full architecture, adding new tools, auth JSON examples, and WASM vs MCP decision guide.
|
||||
|
||||
## Tool Implementation Pattern
|
||||
|
||||
```rust
|
||||
#[async_trait]
|
||||
impl Tool for MyTool {
|
||||
fn name(&self) -> &str { "my_tool" }
|
||||
fn description(&self) -> &str { "Does something useful" }
|
||||
fn parameters_schema(&self) -> serde_json::Value {
|
||||
serde_json::json!({
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"param": { "type": "string", "description": "A parameter" }
|
||||
},
|
||||
"required": ["param"]
|
||||
})
|
||||
}
|
||||
async fn execute(&self, params: serde_json::Value, ctx: &JobContext)
|
||||
-> Result<ToolOutput, ToolError>
|
||||
{
|
||||
let start = std::time::Instant::now();
|
||||
// ... do work ...
|
||||
Ok(ToolOutput::text("result", start.elapsed()))
|
||||
}
|
||||
fn requires_sanitization(&self) -> bool { true } // External data
|
||||
}
|
||||
```
|
||||
@@ -5,6 +5,19 @@ DATABASE_POOL_SIZE=10
|
||||
# LLM Provider
|
||||
# LLM_BACKEND=nearai # default
|
||||
# Possible values: nearai, ollama, openai_compatible, openai, anthropic, tinfoil
|
||||
# LLM_REQUEST_TIMEOUT_SECS=120 # Increase for local LLMs (Ollama, vLLM, LM Studio)
|
||||
|
||||
# === Anthropic Direct ===
|
||||
# Two auth modes:
|
||||
# 1. API key: Set ANTHROPIC_API_KEY (from console.anthropic.com/settings/keys)
|
||||
# 2. OAuth token: Set ANTHROPIC_OAUTH_TOKEN (from `claude login`)
|
||||
# OAuth tokens use Authorization: Bearer instead of x-api-key header.
|
||||
# ANTHROPIC_API_KEY=sk-ant-...
|
||||
# ANTHROPIC_OAUTH_TOKEN=sk-ant-oat01-... # from `claude login` credentials
|
||||
# ANTHROPIC_MODEL=claude-sonnet-4-20250514
|
||||
|
||||
# === OpenAI Direct ===
|
||||
# OPENAI_API_KEY=sk-...
|
||||
|
||||
# === NEAR AI (Chat Completions API) ===
|
||||
# Two auth modes:
|
||||
@@ -102,6 +115,8 @@ AGENT_NAME=ironclaw
|
||||
AGENT_MAX_PARALLEL_JOBS=5
|
||||
AGENT_JOB_TIMEOUT_SECS=3600
|
||||
AGENT_STUCK_THRESHOLD_SECS=300
|
||||
# Maximum tokens per job (0 = unlimited, also settable via settings.json agent.max_tokens_per_job)
|
||||
# AGENT_MAX_TOKENS_PER_JOB=0
|
||||
# Enable planning phase before tool execution (default: true)
|
||||
AGENT_USE_PLANNING=true
|
||||
|
||||
|
||||
@@ -0,0 +1,100 @@
|
||||
name: Claude Code Review
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
types: [labeled]
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
pull-requests: write
|
||||
issues: write
|
||||
id-token: write
|
||||
|
||||
concurrency:
|
||||
group: claude-review-${{ github.event.pull_request.number || github.run_id }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
review:
|
||||
name: Claude Code Review
|
||||
if: contains(github.event.pull_request.labels.*.name, 'staging-promotion')
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
with:
|
||||
fetch-depth: 0
|
||||
|
||||
- name: Run Claude Code review
|
||||
uses: anthropics/claude-code-action@v1
|
||||
with:
|
||||
anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
|
||||
allowed_bots: "ironclaw-ci[bot]"
|
||||
claude_args: "--max-turns 50 --model claude-haiku-4-5-20251001 --allowedTools 'Bash(gh pr comment:*),Bash(gh pr diff:*),Bash(gh pr view:*),Bash(gh pr list:*),Bash(gh issue view:*),Bash(gh issue list:*),Bash(gh search:*),Bash(git blame:*),Bash(git log:*),Bash(git diff:*)'"
|
||||
prompt: |
|
||||
Code review this pull request. Follow these steps precisely:
|
||||
|
||||
1. Use a Haiku agent to find relevant CLAUDE.md files: the root CLAUDE.md
|
||||
and any CLAUDE.md files in directories whose files this PR modifies.
|
||||
|
||||
2. Use a Haiku agent to summarize the PR change (use `gh pr diff`).
|
||||
|
||||
3. Launch 4 parallel agents to review the change independently. Each agent should
|
||||
read the PR diff with `gh pr diff` and the full source files for changed
|
||||
code, then return a list of issues found:
|
||||
|
||||
Agent 1 — Security & Safety
|
||||
Check for: command injection, path traversal, SSRF, XSS, auth bypass,
|
||||
secrets in logs, .unwrap()/.expect() in production code (not tests),
|
||||
race conditions, TOCTOU, unsafe blocks, panics in async, unbounded allocations.
|
||||
|
||||
Agent 2 — Architecture & Patterns
|
||||
Check for: extensible design (traits/enums over nested conditionals),
|
||||
clean abstractions, proper error types (thiserror), CLAUDE.md compliance,
|
||||
type-driven design over stringly-typed code, DRY violations.
|
||||
|
||||
Agent 3 — Bug Scan
|
||||
Shallow diff-only scan for obvious bugs: logic errors, off-by-one,
|
||||
missing error handling, division by zero, incorrect return values.
|
||||
Ignore nitpicks and likely false positives. Do NOT read extra context
|
||||
beyond the diff — focus only on the changes.
|
||||
|
||||
Agent 4 — Performance & Production
|
||||
Check for: blocking in async, N+1 queries, unbounded loops, missing
|
||||
timeouts, resource leaks (file handles, connections), large allocations
|
||||
in hot paths.
|
||||
|
||||
4. For each issue found, launch a parallel Haiku agent to:
|
||||
a. Assign a severity:
|
||||
- CRITICAL: security vulns, panics in prod (.unwrap/.expect), data exfiltration, race conditions
|
||||
- HIGH: logic bugs, missing error handling, breaking API/schema changes
|
||||
- MEDIUM: missing tests, unnecessary complexity, performance issues
|
||||
- LOW: documentation gaps, naming suggestions
|
||||
b. Score confidence 0-100 (give this rubric verbatim):
|
||||
0: False positive, doesn't stand up to scrutiny, or pre-existing issue.
|
||||
25: Might be real, but may be false positive. Stylistic issues not in CLAUDE.md.
|
||||
50: Real issue but nitpick or rare in practice. Not very important.
|
||||
75: Verified real issue, will be hit in practice. Directly impacts functionality
|
||||
or explicitly mentioned in CLAUDE.md.
|
||||
100: Certain, confirmed, will happen frequently. Evidence directly confirms.
|
||||
|
||||
5. Post a single comment on the PR using `gh pr comment` with this format.
|
||||
If no issues were found, post "No issues found." instead:
|
||||
|
||||
### Code review
|
||||
|
||||
Found N issues:
|
||||
|
||||
1. [SEVERITY:CONFIDENCE] <brief description>
|
||||
|
||||
<permalink to file:line using full SHA, eg https://github.com/owner/repo/blob/abc123def/src/file.rs#L10-L15>
|
||||
|
||||
Example: [CRITICAL:92] `.unwrap()` can panic in production when config is missing
|
||||
|
||||
You MUST use the full git SHA in links (not HEAD or branch name).
|
||||
Provide 1 line of context before and after each linked range.
|
||||
|
||||
Notes:
|
||||
- Use `gh` for all GitHub interactions, not web fetch
|
||||
- Do NOT check build signal or attempt to build/test the code
|
||||
- Ignore pre-existing issues not introduced by this PR
|
||||
- Ignore issues a linter/compiler would catch (formatting, imports, types)
|
||||
@@ -44,6 +44,7 @@ jobs:
|
||||
|
||||
clippy-windows:
|
||||
name: Clippy Windows (${{ matrix.name }})
|
||||
if: github.base_ref == 'main'
|
||||
runs-on: windows-latest
|
||||
strategy:
|
||||
fail-fast: false
|
||||
@@ -76,7 +77,12 @@ jobs:
|
||||
needs: [format, clippy, clippy-windows]
|
||||
steps:
|
||||
- run: |
|
||||
if [[ "${{ needs.format.result }}" != "success" || "${{ needs.clippy.result }}" != "success" || "${{ needs.clippy-windows.result }}" != "success" ]]; then
|
||||
if [[ "${{ needs.format.result }}" != "success" || "${{ needs.clippy.result }}" != "success" ]]; then
|
||||
echo "One or more jobs failed"
|
||||
exit 1
|
||||
fi
|
||||
# clippy-windows only runs on main PRs, so skip/success are both acceptable
|
||||
if [[ "${{ needs.clippy-windows.result }}" == "failure" ]]; then
|
||||
echo "Windows clippy failed"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
@@ -1,3 +1,31 @@
|
||||
# Code Coverage Workflow
|
||||
#
|
||||
# This workflow runs test coverage analysis and uploads reports to Codecov.
|
||||
# Coverage reports help identify untested code paths and maintain code quality.
|
||||
#
|
||||
# What it does:
|
||||
# - Runs unit and integration tests with coverage instrumentation
|
||||
# - Runs E2E tests with coverage instrumentation
|
||||
# - Uploads coverage reports to Codecov (https://codecov.io/gh/nearai/ironclaw)
|
||||
#
|
||||
# Viewing coverage reports:
|
||||
# - PRs automatically get coverage comments showing changes in coverage
|
||||
# - Visit https://codecov.io/gh/nearai/ironclaw for detailed coverage reports
|
||||
# - Coverage reports are generated for three configurations:
|
||||
# 1. all-features: Full feature set
|
||||
# 2. default: Default features
|
||||
# 3. libsql-only: Minimal libSQL-only configuration
|
||||
# - E2E coverage tracks end-to-end test coverage separately
|
||||
#
|
||||
# Coverage files:
|
||||
# - Unit/integration: lcov.info (uploaded to Codecov with "unit" flag)
|
||||
# - E2E: e2e-coverage.info (uploaded to Codecov with "e2e" flag)
|
||||
#
|
||||
# Requirements:
|
||||
# - Uses cargo-llvm-cov for coverage instrumentation
|
||||
# - Requires PostgreSQL for integration tests (pgvector/pgvector:pg16)
|
||||
# - E2E tests require Python 3.12 and Playwright
|
||||
|
||||
name: Code Coverage
|
||||
on:
|
||||
push:
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
name: E2E Tests
|
||||
on:
|
||||
workflow_call:
|
||||
schedule:
|
||||
- cron: "0 6 * * 1" # Weekly Monday 6 AM UTC
|
||||
workflow_dispatch:
|
||||
|
||||
@@ -0,0 +1,471 @@
|
||||
name: Staging CI (Batched)
|
||||
|
||||
on:
|
||||
schedule:
|
||||
- cron: "0 * * * *" # Every 60 minutes
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
force:
|
||||
description: "Force run even if no new commits"
|
||||
type: boolean
|
||||
default: false
|
||||
skip_claude_gate:
|
||||
description: "Skip Claude review gate (bypass blocking findings)"
|
||||
type: boolean
|
||||
default: false
|
||||
|
||||
permissions:
|
||||
contents: write
|
||||
issues: write
|
||||
pull-requests: write
|
||||
checks: read
|
||||
|
||||
concurrency:
|
||||
group: staging-ci
|
||||
cancel-in-progress: false # Let running suites finish
|
||||
|
||||
jobs:
|
||||
# ── Check for new commits ──────────────────────────────────────
|
||||
check-changes:
|
||||
name: Check for new commits
|
||||
runs-on: ubuntu-latest
|
||||
outputs:
|
||||
has_changes: ${{ steps.check.outputs.has_changes }}
|
||||
current_head: ${{ steps.check.outputs.current_head }}
|
||||
diff_range: ${{ steps.check.outputs.diff_range }}
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
with:
|
||||
ref: staging
|
||||
fetch-depth: 0
|
||||
fetch-tags: true
|
||||
|
||||
- name: Check for changes since last tested
|
||||
id: check
|
||||
env:
|
||||
FORCE_RUN: ${{ inputs.force }}
|
||||
run: |
|
||||
CURRENT_HEAD=$(git rev-parse HEAD)
|
||||
echo "current_head=${CURRENT_HEAD}" >> "$GITHUB_OUTPUT"
|
||||
|
||||
if git rev-parse staging-tested >/dev/null 2>&1; then
|
||||
LAST_TESTED=$(git rev-parse staging-tested)
|
||||
else
|
||||
LAST_TESTED=""
|
||||
fi
|
||||
|
||||
DIFF_RANGE=""
|
||||
if [ -n "$LAST_TESTED" ] && [ "$LAST_TESTED" = "$CURRENT_HEAD" ]; then
|
||||
echo "No new commits since last tested (${CURRENT_HEAD})"
|
||||
HAS_CHANGES=false
|
||||
else
|
||||
HAS_CHANGES=true
|
||||
if [ -n "$LAST_TESTED" ]; then
|
||||
COMMIT_COUNT=$(git rev-list --count "${LAST_TESTED}..HEAD")
|
||||
echo "Found ${COMMIT_COUNT} new commit(s) since last tested"
|
||||
DIFF_RANGE="${LAST_TESTED}..${CURRENT_HEAD}"
|
||||
else
|
||||
git fetch origin main
|
||||
MERGE_BASE=$(git merge-base origin/main HEAD)
|
||||
echo "First run -- reviewing from merge-base ${MERGE_BASE}"
|
||||
DIFF_RANGE="${MERGE_BASE}..${CURRENT_HEAD}"
|
||||
fi
|
||||
fi
|
||||
|
||||
# Force override from workflow_dispatch
|
||||
if [ "$FORCE_RUN" = "true" ]; then
|
||||
echo "Force run requested"
|
||||
HAS_CHANGES=true
|
||||
if [ -z "$DIFF_RANGE" ]; then
|
||||
DIFF_RANGE="${CURRENT_HEAD}..${CURRENT_HEAD}"
|
||||
fi
|
||||
fi
|
||||
|
||||
echo "has_changes=${HAS_CHANGES}" >> "$GITHUB_OUTPUT"
|
||||
echo "diff_range=${DIFF_RANGE}" >> "$GITHUB_OUTPUT"
|
||||
|
||||
# ── Run full test suite ──────────────────────────────────────────
|
||||
tests:
|
||||
name: Test Suite
|
||||
needs: check-changes
|
||||
if: needs.check-changes.outputs.has_changes == 'true'
|
||||
uses: ./.github/workflows/test.yml
|
||||
|
||||
# ── Run E2E browser tests ────────────────────────────────────────
|
||||
e2e:
|
||||
name: E2E Browser Tests
|
||||
needs: check-changes
|
||||
if: needs.check-changes.outputs.has_changes == 'true'
|
||||
uses: ./.github/workflows/e2e.yml
|
||||
|
||||
# ── Create promotion PR (triggers claude-review.yml on the PR) ──
|
||||
create-promotion-pr:
|
||||
name: Create Promotion PR
|
||||
needs: check-changes
|
||||
if: needs.check-changes.outputs.has_changes == 'true'
|
||||
runs-on: ubuntu-latest
|
||||
outputs:
|
||||
pr_number: ${{ steps.create-pr.outputs.pr_number }}
|
||||
promotion_branch: ${{ steps.branch.outputs.branch }}
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
with:
|
||||
ref: staging
|
||||
fetch-depth: 0
|
||||
|
||||
- name: Generate GitHub App token
|
||||
id: app-token
|
||||
uses: actions/create-github-app-token@v2
|
||||
with:
|
||||
app-id: ${{ secrets.GH_RELEASES_MANAGER_APP_ID }}
|
||||
private-key: ${{ secrets.GH_RELEASES_MANAGER_APP_PRIVATE_KEY }}
|
||||
|
||||
- name: Set token
|
||||
id: token
|
||||
run: |
|
||||
if [ -n "${{ steps.app-token.outputs.token }}" ]; then
|
||||
echo "token=${{ steps.app-token.outputs.token }}" >> "$GITHUB_OUTPUT"
|
||||
else
|
||||
echo "token=${{ github.token }}" >> "$GITHUB_OUTPUT"
|
||||
fi
|
||||
|
||||
- name: Check if staging is ahead of main
|
||||
id: ahead-check
|
||||
env:
|
||||
GH_TOKEN: ${{ steps.token.outputs.token }}
|
||||
run: |
|
||||
git fetch origin main
|
||||
AHEAD=$(git rev-list --count origin/main..origin/staging)
|
||||
echo "commits_ahead=${AHEAD}" >> "$GITHUB_OUTPUT"
|
||||
if [ "$AHEAD" -eq 0 ]; then
|
||||
echo "Staging is not ahead of main. Nothing to promote."
|
||||
else
|
||||
echo "Staging is ${AHEAD} commits ahead of main."
|
||||
fi
|
||||
|
||||
- name: Create promotion branch
|
||||
id: branch
|
||||
if: steps.ahead-check.outputs.commits_ahead != '0'
|
||||
run: |
|
||||
SHORT_SHA=$(echo "${{ needs.check-changes.outputs.current_head }}" | cut -c1-8)
|
||||
BRANCH="staging-promote/${SHORT_SHA}-${{ github.run_id }}"
|
||||
git checkout -b "$BRANCH"
|
||||
git push origin "$BRANCH"
|
||||
echo "branch=${BRANCH}" >> "$GITHUB_OUTPUT"
|
||||
echo "Created promotion branch: ${BRANCH}"
|
||||
|
||||
- name: Find base branch
|
||||
id: find-base
|
||||
if: steps.ahead-check.outputs.commits_ahead != '0'
|
||||
env:
|
||||
GH_TOKEN: ${{ steps.token.outputs.token }}
|
||||
run: |
|
||||
# Find the newest open promotion PR with a staging-promote/* head branch
|
||||
LATEST=$(gh pr list --label staging-promotion --state open \
|
||||
--json headRefName,createdAt \
|
||||
--jq '[.[] | select(.headRefName | startswith("staging-promote/"))] | sort_by(.createdAt) | last | .headRefName // empty')
|
||||
if [ -n "$LATEST" ]; then
|
||||
echo "base=${LATEST}" >> "$GITHUB_OUTPUT"
|
||||
echo "Chaining onto existing promotion branch: ${LATEST}"
|
||||
else
|
||||
echo "base=main" >> "$GITHUB_OUTPUT"
|
||||
echo "No existing promotion PR — targeting main"
|
||||
fi
|
||||
|
||||
- name: Create promotion PR
|
||||
id: create-pr
|
||||
if: steps.ahead-check.outputs.commits_ahead != '0'
|
||||
env:
|
||||
GH_TOKEN: ${{ steps.token.outputs.token }}
|
||||
run: |
|
||||
RANGE="${{ needs.check-changes.outputs.diff_range }}"
|
||||
TIMESTAMP=$(date -u +"%Y-%m-%d %H:%M UTC")
|
||||
BRANCH="${{ steps.branch.outputs.branch }}"
|
||||
BASE="${{ steps.find-base.outputs.base }}"
|
||||
|
||||
PR_URL=$(gh pr create \
|
||||
--base "$BASE" \
|
||||
--head "$BRANCH" \
|
||||
--title "chore: promote staging to main (${TIMESTAMP})" \
|
||||
--body "## Auto-promotion from staging CI
|
||||
|
||||
**Batch range:** \`${RANGE}\`
|
||||
**Promotion branch:** \`${BRANCH}\`
|
||||
**Base:** \`${BASE}\`
|
||||
**Triggered by:** Staging CI batch at ${TIMESTAMP}
|
||||
|
||||
Waiting for gates:
|
||||
- Tests: pending
|
||||
- E2E: pending
|
||||
- Claude Code review: pending (will post comments on this PR)
|
||||
|
||||
---
|
||||
*Auto-created by staging-ci workflow*" \
|
||||
--label "staging-promotion")
|
||||
|
||||
PR_NUM=$(echo "$PR_URL" | grep -oE '[0-9]+$')
|
||||
echo "pr_number=${PR_NUM}" >> "$GITHUB_OUTPUT"
|
||||
echo "Created promotion PR #${PR_NUM}"
|
||||
|
||||
# ── Gate: wait for review, process findings, merge or block ─────
|
||||
gate:
|
||||
name: Staging Gate
|
||||
needs: [check-changes, tests, e2e, create-promotion-pr]
|
||||
if: >
|
||||
always() &&
|
||||
needs.check-changes.outputs.has_changes == 'true' &&
|
||||
needs.tests.result == 'success' &&
|
||||
needs.e2e.result == 'success' &&
|
||||
needs.create-promotion-pr.result == 'success'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 25
|
||||
outputs:
|
||||
gate_passed: ${{ steps.evaluate.outputs.passed }}
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
with:
|
||||
ref: staging
|
||||
fetch-depth: 1
|
||||
|
||||
- name: Generate GitHub App token
|
||||
id: app-token
|
||||
uses: actions/create-github-app-token@v2
|
||||
with:
|
||||
app-id: ${{ secrets.GH_RELEASES_MANAGER_APP_ID }}
|
||||
private-key: ${{ secrets.GH_RELEASES_MANAGER_APP_PRIVATE_KEY }}
|
||||
|
||||
- name: Set token
|
||||
id: token
|
||||
run: |
|
||||
if [ -n "${{ steps.app-token.outputs.token }}" ]; then
|
||||
echo "token=${{ steps.app-token.outputs.token }}" >> "$GITHUB_OUTPUT"
|
||||
else
|
||||
echo "token=${{ github.token }}" >> "$GITHUB_OUTPUT"
|
||||
fi
|
||||
|
||||
- name: Wait for Claude review job
|
||||
env:
|
||||
GH_TOKEN: ${{ steps.token.outputs.token }}
|
||||
PR_NUMBER: ${{ needs.create-promotion-pr.outputs.pr_number }}
|
||||
REPO: ${{ github.repository }}
|
||||
run: |
|
||||
if [ -z "$PR_NUMBER" ]; then
|
||||
echo "No PR number — skipping wait"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
PR_SHA=$(gh pr view "$PR_NUMBER" --json headRefOid --jq '.headRefOid' || echo "")
|
||||
if [ -z "$PR_SHA" ]; then
|
||||
echo "::warning::Could not get PR head SHA"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo "Polling for Claude Code Review job on PR #${PR_NUMBER} (SHA: ${PR_SHA})..."
|
||||
TIMEOUT=1200 # 20 minutes
|
||||
ELAPSED=0
|
||||
INTERVAL=30
|
||||
|
||||
while [ "$ELAPSED" -lt "$TIMEOUT" ]; do
|
||||
STATUS=$(gh api "repos/${REPO}/commits/${PR_SHA}/check-runs" \
|
||||
--jq '[.check_runs[] | select(.name == "Claude Code Review") | .conclusion // .status] | first // "pending"' 2>/dev/null || echo "pending")
|
||||
|
||||
if [ "$STATUS" = "success" ] || [ "$STATUS" = "failure" ] || [ "$STATUS" = "cancelled" ]; then
|
||||
echo "Claude review job completed with status: ${STATUS} (${ELAPSED}s)"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo "Claude review status: ${STATUS} (${ELAPSED}s elapsed)"
|
||||
sleep "$INTERVAL"
|
||||
ELAPSED=$((ELAPSED + INTERVAL))
|
||||
done
|
||||
|
||||
echo "::warning::Claude review job not completed after ${TIMEOUT}s"
|
||||
|
||||
- name: Process Claude review comments and create issues
|
||||
id: process-findings
|
||||
env:
|
||||
GH_TOKEN: ${{ steps.token.outputs.token }}
|
||||
PR_NUMBER: ${{ needs.create-promotion-pr.outputs.pr_number }}
|
||||
REPO: ${{ github.repository }}
|
||||
run: |
|
||||
HAS_BLOCKING=false
|
||||
ISSUES_CREATED=0
|
||||
|
||||
if [ -z "$PR_NUMBER" ]; then
|
||||
echo "No PR — skipping finding processing"
|
||||
echo "has_blocking=false" >> "$GITHUB_OUTPUT"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Check for "No issues found" first (clean pass)
|
||||
NO_ISSUES=$(gh api "repos/${REPO}/issues/${PR_NUMBER}/comments" \
|
||||
--jq '[.[] | select(.user.login == "claude[bot]") | select(.body | test("No issues found"))] | length' 2>/dev/null || echo "0")
|
||||
if [ "$NO_ISSUES" -gt 0 ]; then
|
||||
echo "Claude review found no issues — gate passes"
|
||||
echo "has_blocking=false" >> "$GITHUB_OUTPUT"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Get the last Claude comment that contains findings
|
||||
JQ_FILTER='[.[] | select(.user.login == "claude[bot]") | select(.body | test("Found [0-9]+ issue"))] | last'
|
||||
BODY=$(gh api "repos/${REPO}/issues/${PR_NUMBER}/comments" \
|
||||
--jq "${JQ_FILTER} | .body // empty" 2>/dev/null || echo "")
|
||||
COMMENT_URL=$(gh api "repos/${REPO}/issues/${PR_NUMBER}/comments" \
|
||||
--jq "${JQ_FILTER} | .html_url // empty" 2>/dev/null || echo "")
|
||||
|
||||
if [ -z "$BODY" ]; then
|
||||
echo "::warning::No Claude review comment found for PR #${PR_NUMBER} — treating as blocking"
|
||||
echo "has_blocking=true" >> "$GITHUB_OUTPUT"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Parse [SEVERITY:CONFIDENCE] tags from each numbered finding
|
||||
# Matrix: CRITICAL always→issue, ≥80→block. HIGH ≥50→issue. MEDIUM ≥80→issue. LOW ≥80→issue.
|
||||
# Use process substitution so variables propagate to parent shell
|
||||
while read -r line; do
|
||||
TAG=$(echo "$line" | grep -oE '^\[(CRITICAL|HIGH|MEDIUM|LOW):[0-9]+\]')
|
||||
SEVERITY=$(echo "$TAG" | sed 's/\[\(.*\):\(.*\)\]/\1/')
|
||||
CONFIDENCE=$(echo "$TAG" | sed 's/\[\(.*\):\(.*\)\]/\2/')
|
||||
DESC=$(echo "$line" | sed "s/\[${SEVERITY}:${CONFIDENCE}\] *//" | head -1)
|
||||
|
||||
echo "Found: [${SEVERITY}:${CONFIDENCE}] ${DESC}"
|
||||
|
||||
# Check if blocking (CRITICAL ≥80)
|
||||
if [ "$SEVERITY" = "CRITICAL" ] && [ "$CONFIDENCE" -ge 80 ]; then
|
||||
HAS_BLOCKING=true
|
||||
fi
|
||||
|
||||
# Determine if this should create an issue
|
||||
CREATE_ISSUE=false
|
||||
case "$SEVERITY" in
|
||||
CRITICAL) CREATE_ISSUE=true ;;
|
||||
HIGH) [ "$CONFIDENCE" -ge 50 ] && CREATE_ISSUE=true ;;
|
||||
MEDIUM) [ "$CONFIDENCE" -ge 80 ] && CREATE_ISSUE=true ;;
|
||||
LOW) [ "$CONFIDENCE" -ge 80 ] && CREATE_ISSUE=true ;;
|
||||
esac
|
||||
|
||||
if [ "$CREATE_ISSUE" = "true" ]; then
|
||||
case "$SEVERITY" in
|
||||
CRITICAL) LABELS="bug,risk: high,staging-ci-review" ;;
|
||||
HIGH) LABELS="bug,risk: medium,staging-ci-review" ;;
|
||||
MEDIUM) LABELS="risk: medium,staging-ci-review" ;;
|
||||
LOW) LABELS="risk: low,staging-ci-review" ;;
|
||||
esac
|
||||
|
||||
TITLE=$(echo "$DESC" | cut -c1-80)
|
||||
{
|
||||
echo "## [${SEVERITY}:${CONFIDENCE}] Issue Found by Staging CI Review"
|
||||
echo ""
|
||||
echo "**Severity:** ${SEVERITY}"
|
||||
echo "**Confidence:** ${CONFIDENCE}/100"
|
||||
echo "**PR comment:** ${COMMENT_URL}"
|
||||
echo ""
|
||||
echo "### Description"
|
||||
echo "$DESC"
|
||||
echo ""
|
||||
echo "---"
|
||||
echo "*Auto-created by staging-ci Claude Code review*"
|
||||
} > /tmp/issue-body.md
|
||||
|
||||
if gh issue create \
|
||||
--title "[${SEVERITY}] ${TITLE}" \
|
||||
--body-file /tmp/issue-body.md \
|
||||
--label "${LABELS}"; then
|
||||
ISSUES_CREATED=$((ISSUES_CREATED + 1))
|
||||
else
|
||||
echo "::warning::Failed to create issue for ${SEVERITY} finding"
|
||||
fi
|
||||
fi
|
||||
done < <(echo "$BODY" | grep -oE '\[(CRITICAL|HIGH|MEDIUM|LOW):[0-9]+\].*')
|
||||
|
||||
echo "Created ${ISSUES_CREATED} issues"
|
||||
echo "has_blocking=${HAS_BLOCKING}" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Evaluate gate
|
||||
id: evaluate
|
||||
env:
|
||||
PR_NUMBER: ${{ needs.create-promotion-pr.outputs.pr_number }}
|
||||
SKIP_GATE: ${{ inputs.skip_claude_gate }}
|
||||
HAS_BLOCKING: ${{ steps.process-findings.outputs.has_blocking }}
|
||||
run: |
|
||||
SKIP_INPUT="$SKIP_GATE"
|
||||
|
||||
if [ "$HAS_BLOCKING" = "true" ]; then
|
||||
echo "::warning::Claude review found blocking issues (CRITICAL ≥80 confidence)"
|
||||
if [ "$SKIP_INPUT" = "true" ]; then
|
||||
echo "::warning::Gate overridden by skip_claude_gate workflow input"
|
||||
echo "passed=true" >> "$GITHUB_OUTPUT"
|
||||
else
|
||||
echo "::error::Blocking promotion due to CRITICAL findings (≥80 confidence)"
|
||||
echo "::error::PR #${PR_NUMBER} left open with review comments"
|
||||
echo "passed=false" >> "$GITHUB_OUTPUT"
|
||||
exit 1
|
||||
fi
|
||||
else
|
||||
echo "No blocking findings. Gate passed."
|
||||
echo "passed=true" >> "$GITHUB_OUTPUT"
|
||||
fi
|
||||
|
||||
- name: Merge promotion PR
|
||||
id: merge
|
||||
if: steps.evaluate.outputs.passed == 'true'
|
||||
env:
|
||||
GH_TOKEN: ${{ steps.token.outputs.token }}
|
||||
PR_NUMBER: ${{ needs.create-promotion-pr.outputs.pr_number }}
|
||||
run: |
|
||||
if [ -n "$PR_NUMBER" ]; then
|
||||
echo "Merging promotion PR #${PR_NUMBER}"
|
||||
# Do NOT use --delete-branch: deleting a promotion branch closes
|
||||
# any chained PRs that use it as their base (verified in ironclaw-ci-test).
|
||||
# Stale promotion branches are cleaned up separately.
|
||||
gh pr merge "$PR_NUMBER" --merge
|
||||
echo "merged=true" >> "$GITHUB_OUTPUT"
|
||||
fi
|
||||
|
||||
# ── Update tested tag (always, so next batch covers only new commits) ──
|
||||
update-tag:
|
||||
name: Update staging-tested tag
|
||||
needs: [check-changes, tests, e2e, create-promotion-pr, gate]
|
||||
if: >
|
||||
always() &&
|
||||
needs.check-changes.outputs.has_changes == 'true' &&
|
||||
needs.tests.result == 'success' &&
|
||||
needs.e2e.result == 'success' &&
|
||||
needs.create-promotion-pr.result == 'success'
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
with:
|
||||
ref: staging
|
||||
fetch-depth: 1
|
||||
|
||||
- name: Update staging-tested tag
|
||||
run: |
|
||||
git tag -f staging-tested "${{ needs.check-changes.outputs.current_head }}"
|
||||
git push origin staging-tested --force
|
||||
echo "Updated staging-tested tag to ${{ needs.check-changes.outputs.current_head }}"
|
||||
|
||||
# ── Report ───────────────────────────────────────────────────────
|
||||
report:
|
||||
name: Staging CI Summary
|
||||
needs: [check-changes, tests, e2e, create-promotion-pr, gate, update-tag]
|
||||
if: always() && needs.check-changes.outputs.has_changes == 'true'
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Summary
|
||||
run: |
|
||||
echo "## Staging CI Batch Results" >> "$GITHUB_STEP_SUMMARY"
|
||||
echo "" >> "$GITHUB_STEP_SUMMARY"
|
||||
echo "| Check | Result |" >> "$GITHUB_STEP_SUMMARY"
|
||||
echo "|-------|--------|" >> "$GITHUB_STEP_SUMMARY"
|
||||
echo "| Tests | ${{ needs.tests.result }} |" >> "$GITHUB_STEP_SUMMARY"
|
||||
echo "| E2E | ${{ needs.e2e.result }} |" >> "$GITHUB_STEP_SUMMARY"
|
||||
echo "| Promotion PR | ${{ needs.create-promotion-pr.result }} |" >> "$GITHUB_STEP_SUMMARY"
|
||||
echo "| Gate | ${{ needs.gate.result }} |" >> "$GITHUB_STEP_SUMMARY"
|
||||
echo "| Tag Updated | ${{ needs.update-tag.result }} |" >> "$GITHUB_STEP_SUMMARY"
|
||||
echo "" >> "$GITHUB_STEP_SUMMARY"
|
||||
echo "Range: ${{ needs.check-changes.outputs.diff_range }}" >> "$GITHUB_STEP_SUMMARY"
|
||||
PR_NUM="${{ needs.create-promotion-pr.outputs.pr_number }}"
|
||||
if [ -n "$PR_NUM" ]; then
|
||||
echo "Promotion PR: #${PR_NUM}" >> "$GITHUB_STEP_SUMMARY"
|
||||
fi
|
||||
@@ -1,6 +1,9 @@
|
||||
name: Run Tests
|
||||
on:
|
||||
workflow_call:
|
||||
pull_request:
|
||||
branches:
|
||||
- main
|
||||
push:
|
||||
branches:
|
||||
- main
|
||||
@@ -38,6 +41,10 @@ jobs:
|
||||
|
||||
telegram-tests:
|
||||
name: Telegram Channel Tests
|
||||
if: >
|
||||
github.event_name == 'push' ||
|
||||
github.event_name == 'workflow_call' ||
|
||||
(github.event_name == 'pull_request' && github.base_ref != 'staging')
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
@@ -50,6 +57,10 @@ jobs:
|
||||
|
||||
windows-build:
|
||||
name: Windows Build (${{ matrix.name }})
|
||||
if: >
|
||||
github.event_name == 'push' ||
|
||||
github.event_name == 'workflow_call' ||
|
||||
(github.event_name == 'pull_request' && github.base_ref != 'staging')
|
||||
runs-on: windows-latest
|
||||
strategy:
|
||||
fail-fast: false
|
||||
@@ -74,6 +85,10 @@ jobs:
|
||||
|
||||
wasm-wit-compat:
|
||||
name: WASM WIT Compatibility
|
||||
if: >
|
||||
github.event_name == 'push' ||
|
||||
github.event_name == 'workflow_call' ||
|
||||
(github.event_name == 'pull_request' && github.base_ref != 'staging')
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
@@ -94,6 +109,10 @@ jobs:
|
||||
|
||||
docker-build:
|
||||
name: Docker Build
|
||||
if: >
|
||||
github.event_name == 'push' ||
|
||||
github.event_name == 'workflow_call' ||
|
||||
(github.event_name == 'pull_request' && github.base_ref != 'staging')
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
@@ -123,12 +142,22 @@ jobs:
|
||||
needs: [tests, telegram-tests, wasm-wit-compat, docker-build, windows-build, version-check]
|
||||
steps:
|
||||
- run: |
|
||||
if [[ "${{ needs.tests.result }}" != "success" || "${{ needs.telegram-tests.result }}" != "success" || "${{ needs.wasm-wit-compat.result }}" != "success" || "${{ needs.docker-build.result }}" != "success" || "${{ needs.windows-build.result }}" != "success" ]]; then
|
||||
echo "One or more jobs failed"
|
||||
exit 1
|
||||
fi
|
||||
# version-check only runs on PRs, so skip/success are both acceptable
|
||||
if [[ "${{ needs.version-check.result }}" == "failure" ]]; then
|
||||
echo "Version bump check failed"
|
||||
# Unit tests must always pass
|
||||
if [[ "${{ needs.tests.result }}" != "success" ]]; then
|
||||
echo "Unit tests failed"
|
||||
exit 1
|
||||
fi
|
||||
# Gated jobs: must pass on promotion PRs / push, skipped on developer PRs
|
||||
for job in telegram-tests wasm-wit-compat docker-build windows-build version-check; do
|
||||
case "$job" in
|
||||
telegram-tests) result="${{ needs.telegram-tests.result }}" ;;
|
||||
wasm-wit-compat) result="${{ needs.wasm-wit-compat.result }}" ;;
|
||||
docker-build) result="${{ needs.docker-build.result }}" ;;
|
||||
windows-build) result="${{ needs.windows-build.result }}" ;;
|
||||
version-check) result="${{ needs.version-check.result }}" ;;
|
||||
esac
|
||||
if [[ "$result" == "failure" || "$result" == "cancelled" ]]; then
|
||||
echo "$job failed"
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
|
||||
+7
-1
@@ -4,8 +4,9 @@
|
||||
.env.*
|
||||
!.env.example
|
||||
|
||||
# Claude Code worktrees
|
||||
# Claude Code worktrees and lock files
|
||||
.claude/worktrees/
|
||||
.claude/scheduled_tasks.lock
|
||||
|
||||
# Sidecar tool data
|
||||
.sidecar/
|
||||
@@ -22,3 +23,8 @@ bench-results/
|
||||
# WASM build artifacts (loaded from disk, not bundled)
|
||||
*.wasm
|
||||
|
||||
# Traces
|
||||
trace_*.json
|
||||
|
||||
# Local Claude Code settings (machine-specific, should not be committed)
|
||||
.claude/settings.local.json
|
||||
|
||||
@@ -7,6 +7,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
### Added
|
||||
|
||||
- AWS Bedrock LLM provider via native Converse API with IAM and SSO auth support (feature-gated: `--features bedrock`)
|
||||
|
||||
## [0.16.1](https://github.com/nearai/ironclaw/compare/v0.16.0...v0.16.1) - 2026-03-06
|
||||
|
||||
### Fixed
|
||||
|
||||
@@ -1,73 +1,37 @@
|
||||
# IronClaw Development Guide
|
||||
|
||||
## Project Overview
|
||||
|
||||
**IronClaw** is a secure personal AI assistant that protects your data and expands its capabilities on the fly.
|
||||
|
||||
### Core Philosophy
|
||||
- **User-first security** - Your data stays yours, encrypted and local
|
||||
- **Self-expanding** - Build new tools dynamically without vendor dependency
|
||||
- **Defense in depth** - Multiple security layers against prompt injection and data exfiltration
|
||||
- **Always available** - Multi-channel access with proactive background execution
|
||||
|
||||
### Features
|
||||
- **Multi-channel input**: TUI (Ratatui), HTTP webhooks, WASM channels (Telegram, Slack), web gateway
|
||||
- **Parallel job execution** with state machine and self-repair for stuck jobs
|
||||
- **Sandbox execution**: Docker container isolation with network proxy and credential injection
|
||||
- **Claude Code mode**: Delegate jobs to Claude CLI inside containers
|
||||
- **Skills system**: SKILL.md prompt extensions with trust model, tool attenuation, and ClawHub registry
|
||||
- **Routines**: Scheduled (cron) and reactive (event, webhook) task execution
|
||||
- **Web gateway**: Browser UI with SSE/WebSocket real-time streaming
|
||||
- **Extension management**: Install, auth, activate MCP/WASM extensions
|
||||
- **Extensible tools**: Built-in tools, WASM sandbox, MCP client, dynamic builder
|
||||
- **Persistent memory**: Workspace with hybrid search (FTS + vector via RRF)
|
||||
- **Prompt injection defense**: Sanitizer, validator, policy rules, leak detection, shell env scrubbing
|
||||
- **Multi-provider LLM**: NEAR AI, OpenAI, Anthropic, Ollama, OpenAI-compatible, Tinfoil private inference
|
||||
- **Setup wizard**: 7-step interactive onboarding for first-run configuration
|
||||
- **Heartbeat system**: Proactive periodic execution with checklist
|
||||
**IronClaw** is a secure personal AI assistant — user-first security, self-expanding tools, defense in depth, multi-channel access with proactive background execution.
|
||||
|
||||
## Build & Test
|
||||
|
||||
```bash
|
||||
# Format code
|
||||
cargo fmt
|
||||
|
||||
# Lint (fix ALL warnings before committing, including pre-existing ones)
|
||||
cargo clippy --all --benches --tests --examples --all-features
|
||||
|
||||
# Run all tests
|
||||
cargo test
|
||||
|
||||
# Run specific test
|
||||
cargo test test_name
|
||||
|
||||
# Run with logging
|
||||
RUST_LOG=ironclaw=debug cargo run
|
||||
|
||||
# Run integration tests (may require running services/DB)
|
||||
cargo test --test workspace_integration
|
||||
cargo test --test ws_gateway_integration
|
||||
cargo test --test heartbeat_integration
|
||||
|
||||
# Run E2E tests (Python/Playwright — requires a running ironclaw instance)
|
||||
# See tests/e2e/CLAUDE.md for full setup instructions
|
||||
cd tests/e2e
|
||||
python -m venv .venv && source .venv/bin/activate # On Windows: .venv\Scripts\activate
|
||||
pip install -e .
|
||||
playwright install chromium
|
||||
pytest scenarios/ # all scenarios
|
||||
pytest scenarios/test_chat.py # specific scenario
|
||||
cargo fmt # format
|
||||
cargo clippy --all --benches --tests --examples --all-features # lint (zero warnings)
|
||||
cargo test # unit tests
|
||||
cargo test --features integration # + PostgreSQL tests
|
||||
RUST_LOG=ironclaw=debug cargo run # run with logging
|
||||
```
|
||||
|
||||
### Test Tiers
|
||||
E2E tests: see `tests/e2e/CLAUDE.md`.
|
||||
|
||||
| Tier | Command | What runs | External deps |
|
||||
|------|---------|-----------|---------------|
|
||||
| Unit | `cargo test` | All `mod tests` + self-contained integration tests | None |
|
||||
| Integration | `cargo test --features integration` | + PostgreSQL-dependent tests | Running PostgreSQL |
|
||||
| Live | `cargo test --features integration -- --ignored` | + LLM-dependent tests | PostgreSQL + LLM API keys |
|
||||
## Code Style
|
||||
|
||||
Run `bash scripts/check-boundaries.sh` to verify test tier gating and other architecture rules.
|
||||
- Prefer `crate::` for cross-module imports; `super::` is fine in tests and intra-module refs
|
||||
- No `pub use` re-exports unless exposing to downstream consumers
|
||||
- No `.unwrap()` or `.expect()` in production code (tests are fine)
|
||||
- Use `thiserror` for error types in `error.rs`
|
||||
- Map errors with context: `.map_err(|e| SomeError::Variant { reason: e.to_string() })?`
|
||||
- Prefer strong types over strings (enums, newtypes)
|
||||
- Keep functions focused, extract helpers when logic is reused
|
||||
- Comments for non-obvious logic only
|
||||
|
||||
## Architecture
|
||||
|
||||
Prefer generic/extensible architectures over hardcoding specific integrations. Ask clarifying questions about the desired abstraction level before implementing.
|
||||
|
||||
Key traits for extensibility: `Database`, `Channel`, `Tool`, `LlmProvider`, `SuccessEvaluator`, `EmbeddingProvider`, `NetworkPolicyDecider`, `Hook`, `Observer`, `Tunnel`.
|
||||
|
||||
All I/O is async with tokio. Use `Arc<T>` for shared state, `RwLock` for concurrent access.
|
||||
|
||||
## Project Structure
|
||||
|
||||
@@ -95,78 +59,35 @@ src/
|
||||
│ ├── channel.rs # Channel trait, IncomingMessage, OutgoingResponse
|
||||
│ ├── manager.rs # ChannelManager merges streams
|
||||
│ ├── cli/ # Full TUI with Ratatui
|
||||
│ │ ├── mod.rs # TuiChannel implementation
|
||||
│ │ ├── app.rs # Application state
|
||||
│ │ ├── render.rs # UI rendering
|
||||
│ │ ├── events.rs # Input handling
|
||||
│ │ ├── overlay.rs # Approval overlays
|
||||
│ │ └── composer.rs # Message composition
|
||||
│ ├── http.rs # HTTP webhook (axum) with secret validation
|
||||
│ ├── webhook_server.rs # Unified HTTP server composing all webhook routes
|
||||
│ ├── repl.rs # Simple REPL (for testing)
|
||||
│ ├── web/ # Web gateway (browser UI) — see src/channels/web/CLAUDE.md
|
||||
│ └── wasm/ # WASM channel runtime
|
||||
│ ├── mod.rs
|
||||
│ ├── bundled.rs # Bundled channel discovery
|
||||
│ ├── capabilities.rs # Channel-specific capabilities (HTTP endpoint, emit rate)
|
||||
│ ├── error.rs # WASM channel error types
|
||||
│ ├── runtime.rs # WASM channel execution runtime
|
||||
│ └── wrapper.rs # Channel trait wrapper for WASM modules
|
||||
│
|
||||
├── cli/ # CLI subcommands (clap)
|
||||
│ ├── mod.rs # Cli struct, Command enum (run/onboard/config/tool/registry/mcp/memory/pairing/service/doctor/status/completion)
|
||||
│ ├── config.rs # config list/get/set subcommands
|
||||
│ ├── tool.rs # tool install/list/remove subcommands
|
||||
│ ├── registry.rs # registry list/install subcommands
|
||||
│ ├── mcp.rs # mcp add/auth/list/test subcommands
|
||||
│ ├── memory.rs # memory search/read/write subcommands
|
||||
│ ├── pairing.rs # pairing list/approve subcommands
|
||||
│ ├── service.rs # service install/start/stop subcommands
|
||||
│ ├── doctor.rs # Active health diagnostics
|
||||
│ ├── status.rs # System health/status display
|
||||
│ ├── completion.rs # Shell completion script generation
|
||||
│ └── oauth_defaults.rs # Default OAuth redirect URIs
|
||||
│ └── config.rs, tool.rs, registry.rs, mcp.rs, memory.rs, pairing.rs, service.rs, doctor.rs, status.rs, completion.rs
|
||||
│
|
||||
├── registry/ # Extension registry catalog
|
||||
│ ├── mod.rs # Public API; re-exports RegistryCatalog, RegistryInstaller, manifest types
|
||||
│ ├── manifest.rs # ExtensionManifest, ArtifactSpec, BundleDefinition types
|
||||
│ ├── catalog.rs # RegistryCatalog: load from filesystem and embedded JSON
|
||||
│ ├── installer.rs # RegistryInstaller: download, verify, install WASM artifacts
|
||||
│ ├── artifacts.rs # Artifact download and caching
|
||||
│ └── embedded.rs # Catalog compiled into binary at build time (via build.rs)
|
||||
│ └── installer.rs # RegistryInstaller: download, verify, install WASM artifacts
|
||||
│
|
||||
├── hooks/ # Lifecycle hooks for intercepting agent operations
|
||||
│ ├── mod.rs # 6 HookPoints: BeforeInbound, BeforeToolCall, BeforeOutbound, OnSessionStart, OnSessionEnd, TransformResponse
|
||||
│ ├── hook.rs # Hook trait, HookContext, HookEvent, HookOutcome, HookFailureMode
|
||||
│ ├── registry.rs # HookRegistry: register, prioritize, execute hooks
|
||||
│ └── bundled.rs # Built-in hooks: rule-based filters, webhook forwarders, HookBundleConfig
|
||||
├── hooks/ # Lifecycle hooks (6 points: BeforeInbound, BeforeToolCall, BeforeOutbound, OnSessionStart, OnSessionEnd, TransformResponse)
|
||||
│
|
||||
├── tunnel/ # Tunnel abstraction for public internet exposure
|
||||
│ ├── mod.rs # Tunnel trait, TunnelProviderConfig, create_tunnel() factory
|
||||
│ ├── cloudflare.rs # CloudflareTunnel (cloudflared binary)
|
||||
│ ├── ngrok.rs # NgrokTunnel
|
||||
│ ├── tailscale.rs # TailscaleTunnel (serve/funnel modes)
|
||||
│ ├── custom.rs # CustomTunnel (arbitrary command with {host}/{port})
|
||||
│ └── none.rs # NoneTunnel (local-only, no exposure)
|
||||
├── tunnel/ # Tunnel abstraction (cloudflare, ngrok, tailscale, custom, none)
|
||||
│
|
||||
├── observability/ # Pluggable event/metric recording
|
||||
│ ├── mod.rs # create_observer() factory, ObservabilityConfig
|
||||
│ ├── traits.rs # Observer trait, ObserverEvent, ObserverMetric
|
||||
│ ├── noop.rs # NoopObserver (zero overhead, default)
|
||||
│ ├── log.rs # LogObserver (tracing-based)
|
||||
│ └── multi.rs # MultiObserver (fan-out to multiple backends)
|
||||
├── observability/ # Pluggable event/metric recording (noop, log, multi)
|
||||
│
|
||||
├── orchestrator/ # Internal HTTP API for sandbox containers
|
||||
│ ├── mod.rs
|
||||
│ ├── api.rs # Axum endpoints (LLM proxy, events, prompts)
|
||||
│ ├── auth.rs # Per-job bearer token store
|
||||
│ └── job_manager.rs # Container lifecycle (create, stop, cleanup)
|
||||
│
|
||||
├── worker/ # Runs inside Docker containers
|
||||
│ ├── mod.rs
|
||||
│ ├── runtime.rs # Worker execution loop (tool calls, LLM)
|
||||
│ ├── claude_bridge.rs # Claude Code bridge (spawns claude CLI)
|
||||
│ ├── api.rs # HTTP client to orchestrator
|
||||
│ └── proxy_llm.rs # LlmProvider that proxies through orchestrator
|
||||
│
|
||||
├── safety/ # Prompt injection defense
|
||||
@@ -174,179 +95,68 @@ src/
|
||||
│ ├── validator.rs # Input validation (length, encoding, patterns)
|
||||
│ ├── policy.rs # PolicyRule system with severity/actions
|
||||
│ ├── leak_detector.rs # Secret detection (API keys, tokens, etc.)
|
||||
│ └── credential_detect.rs # HTTP request credential detection (headers, URL params)
|
||||
│ └── credential_detect.rs # HTTP request credential detection
|
||||
│
|
||||
├── llm/ # Multi-provider LLM integration — see src/llm/CLAUDE.md
|
||||
│
|
||||
├── tools/ # Extensible tool system
|
||||
│ ├── tool.rs # Tool trait, ToolOutput, ToolError
|
||||
│ ├── registry.rs # ToolRegistry for discovery
|
||||
│ ├── sandbox.rs # Process-based sandbox (stub, superseded by wasm/)
|
||||
│ ├── rate_limiter.rs # Shared sliding-window rate limiter for built-in and WASM tools
|
||||
│ ├── builtin/ # Built-in tools
|
||||
│ │ ├── echo.rs, time.rs, json.rs, http.rs
|
||||
│ │ ├── web_fetch.rs # GET URL → clean Markdown (readability + html-to-md conversion)
|
||||
│ │ ├── file.rs # ReadFile, WriteFile, ListDir, ApplyPatch
|
||||
│ │ ├── shell.rs # Shell command execution
|
||||
│ │ ├── memory.rs # Memory tools (search, write, read, tree)
|
||||
│ │ ├── message.rs # MessageTool: agent proactively messages users on any channel
|
||||
│ │ ├── job.rs # CreateJob, ListJobs, JobStatus, CancelJob
|
||||
│ │ ├── routine.rs # routine_create/list/update/delete/history
|
||||
│ │ ├── extension_tools.rs # Extension install/auth/activate/remove
|
||||
│ │ ├── skill_tools.rs # skill_list/search/install/remove tools
|
||||
│ │ ├── secrets_tools.rs # secret_list/secret_delete (zero-exposure: no values exposed)
|
||||
│ │ ├── html_converter.rs # HTML→Markdown via readability + html-to-markdown-rs
|
||||
│ │ ├── path_utils.rs # Shared path validation/canonicalization helpers
|
||||
│ │ └── marketplace.rs, ecommerce.rs, taskrabbit.rs, restaurant.rs (stubs)
|
||||
│ ├── rate_limiter.rs # Shared sliding-window rate limiter
|
||||
│ ├── builtin/ # Built-in tools (echo, time, json, http, web_fetch, file, shell, memory, message, job, routine, extension_tools, skill_tools, secrets_tools)
|
||||
│ ├── builder/ # Dynamic tool building
|
||||
│ │ ├── core.rs # BuildRequirement, SoftwareType, Language
|
||||
│ │ ├── templates.rs # Project scaffolding
|
||||
│ │ ├── testing.rs # Test harness integration
|
||||
│ │ └── validation.rs # WASM validation
|
||||
│ ├── mcp/ # Model Context Protocol
|
||||
│ │ ├── client.rs # MCP client over HTTP
|
||||
│ │ ├── protocol.rs # JSON-RPC types
|
||||
│ │ └── session.rs # MCP session management (Mcp-Session-Id header, per-server state)
|
||||
│ └── wasm/ # Full WASM sandbox (wasmtime)
|
||||
│ ├── runtime.rs # Module compilation and caching
|
||||
│ ├── wrapper.rs # Tool trait wrapper for WASM modules
|
||||
│ ├── host.rs # Host functions (logging, time, workspace)
|
||||
│ ├── limits.rs # Fuel metering and memory limiting
|
||||
│ ├── allowlist.rs # Network endpoint allowlisting
|
||||
│ ├── credential_injector.rs # Safe credential injection
|
||||
│ ├── loader.rs # WASM tool discovery from filesystem
|
||||
│ ├── rate_limiter.rs # Per-tool rate limiting
|
||||
│ ├── error.rs # WASM-specific error types
|
||||
│ └── storage.rs # Linear memory persistence
|
||||
│ ├── mcp/ # Model Context Protocol client
|
||||
│ └── wasm/ # Full WASM sandbox (wasmtime) — runtime, host functions, fuel metering, allowlist, credential injection
|
||||
│
|
||||
├── db/ # Dual-backend persistence (PostgreSQL + libSQL) — see src/db/CLAUDE.md
|
||||
│
|
||||
├── workspace/ # Persistent memory system (OpenClaw-inspired)
|
||||
│ ├── mod.rs # Workspace struct, memory operations
|
||||
│ ├── document.rs # MemoryDocument, MemoryChunk, WorkspaceEntry
|
||||
│ ├── chunker.rs # Document chunking (800 tokens, 15% overlap)
|
||||
│ ├── embeddings.rs # EmbeddingProvider trait, OpenAI implementation
|
||||
│ ├── search.rs # Hybrid search with RRF algorithm
|
||||
│ └── repository.rs # PostgreSQL CRUD and search operations
|
||||
├── workspace/ # Persistent memory system — see src/workspace/README.md
|
||||
│
|
||||
├── context/ # Job context isolation
|
||||
│ ├── state.rs # JobState enum, JobContext, state machine
|
||||
│ ├── memory.rs # ActionRecord, ConversationMemory
|
||||
│ └── manager.rs # ContextManager for concurrent jobs
|
||||
│
|
||||
├── estimation/ # Cost/time/value estimation
|
||||
│ ├── cost.rs # CostEstimator
|
||||
│ ├── time.rs # TimeEstimator
|
||||
│ ├── value.rs # ValueEstimator (profit margins)
|
||||
│ └── learner.rs # Exponential moving average learning
|
||||
│
|
||||
├── evaluation/ # Success evaluation
|
||||
│ ├── success.rs # SuccessEvaluator trait, RuleBasedEvaluator, LlmEvaluator
|
||||
│ └── metrics.rs # MetricsCollector, QualityMetrics
|
||||
├── context/ # Job context isolation (JobState, JobContext, ContextManager)
|
||||
├── estimation/ # Cost/time/value estimation with EMA learning
|
||||
├── evaluation/ # Success evaluation (rule-based, LLM-based)
|
||||
│
|
||||
├── sandbox/ # Docker execution sandbox
|
||||
│ ├── mod.rs # Public API, default allowlist
|
||||
│ ├── config.rs # SandboxConfig, SandboxPolicy enum
|
||||
│ ├── config.rs # SandboxConfig, SandboxPolicy enum (ReadOnly/WorkspaceWrite/FullAccess)
|
||||
│ ├── manager.rs # SandboxManager orchestration
|
||||
│ ├── container.rs # ContainerRunner, Docker lifecycle
|
||||
│ ├── error.rs # SandboxError types
|
||||
│ └── proxy/ # Network proxy for containers
|
||||
│ ├── mod.rs # NetworkProxyBuilder
|
||||
│ ├── http.rs # HttpProxy, CredentialResolver trait
|
||||
│ ├── policy.rs # NetworkPolicyDecider trait
|
||||
│ └── allowlist.rs # DomainAllowlist validation
|
||||
│ └── proxy/ # Network proxy: domain allowlist, credential injection, CONNECT tunnel
|
||||
│
|
||||
├── secrets/ # Secrets management
|
||||
│ ├── mod.rs # SecretsStore trait, public API
|
||||
│ ├── types.rs # Core types (Secret, SecretRef, SecretMetadata)
|
||||
│ ├── crypto.rs # AES-256-GCM encryption
|
||||
│ ├── keychain.rs # OS keychain integration (macOS Keychain, GNOME Keyring) for master key
|
||||
│ └── store.rs # Encrypted secret storage
|
||||
├── secrets/ # Secrets management (AES-256-GCM, OS keychain for master key)
|
||||
│
|
||||
├── setup/ # Onboarding wizard (spec: src/setup/README.md)
|
||||
│ ├── mod.rs # Entry point, check_onboard_needed()
|
||||
│ ├── wizard.rs # 7-step interactive wizard
|
||||
│ ├── channels.rs # Channel setup helpers
|
||||
│ └── prompts.rs # Terminal prompts (select, confirm, secret)
|
||||
├── setup/ # 7-step onboarding wizard — see src/setup/README.md
|
||||
│
|
||||
├── skills/ # SKILL.md prompt extension system
|
||||
│ ├── mod.rs # Core types (SkillTrust, LoadedSkill)
|
||||
│ ├── registry.rs # SkillRegistry: discover, install, remove
|
||||
│ ├── selector.rs # Deterministic scoring prefilter
|
||||
│ ├── attenuation.rs # Trust-based tool ceiling
|
||||
│ ├── gating.rs # Requirement checks (bins, env, config)
|
||||
│ ├── parser.rs # SKILL.md frontmatter + markdown parser
|
||||
│ └── catalog.rs # ClawHub registry client
|
||||
├── skills/ # SKILL.md prompt extension system — see .claude/rules/skills.md
|
||||
│
|
||||
└── history/ # Persistence
|
||||
├── store.rs # PostgreSQL repositories
|
||||
└── analytics.rs # Aggregation queries (JobStats, ToolStats)
|
||||
└── history/ # Persistence (PostgreSQL repositories, analytics)
|
||||
|
||||
tests/
|
||||
├── *.rs # Integration tests (workspace, heartbeat, WS gateway, pairing, etc.)
|
||||
├── test-pages/ # HTML→Markdown conversion fixtures (CNN, Medium, Yahoo)
|
||||
├── test-pages/ # HTML→Markdown conversion fixtures
|
||||
└── e2e/ # Python/Playwright E2E scenarios (see tests/e2e/CLAUDE.md)
|
||||
```
|
||||
|
||||
## Key Patterns
|
||||
## Database
|
||||
|
||||
### Architecture
|
||||
Dual-backend: PostgreSQL + libSQL/Turso. **All new persistence features must support both backends.** See `src/db/CLAUDE.md` and `.claude/rules/database.md`.
|
||||
|
||||
When designing new features or systems, always prefer generic/extensible architectures over hardcoding specific integrations. Ask clarifying questions about the desired abstraction level before implementing.
|
||||
## Module Specs
|
||||
|
||||
### Error Handling
|
||||
- Use `thiserror` for error types in `error.rs`
|
||||
- Never use `.unwrap()` or `.expect()` in production code (tests are fine)
|
||||
- Map errors with context: `.map_err(|e| SomeError::Variant { reason: e.to_string() })?`
|
||||
- Before committing, grep for `.unwrap()` and `.expect(` in changed files to catch violations mechanically
|
||||
When modifying a module with a spec, read the spec first. Code follows spec; spec is the tiebreaker.
|
||||
|
||||
### Async
|
||||
- All I/O is async with tokio
|
||||
- Use `Arc<T>` for shared state across tasks
|
||||
- Use `RwLock` for concurrent read/write access
|
||||
| Module | Spec |
|
||||
|--------|------|
|
||||
| `src/agent/` | `src/agent/CLAUDE.md` |
|
||||
| `src/channels/web/` | `src/channels/web/CLAUDE.md` |
|
||||
| `src/db/` | `src/db/CLAUDE.md` |
|
||||
| `src/llm/` | `src/llm/CLAUDE.md` |
|
||||
| `src/setup/` | `src/setup/README.md` |
|
||||
| `src/tools/` | `src/tools/README.md` |
|
||||
| `src/workspace/` | `src/workspace/README.md` |
|
||||
| `tests/e2e/` | `tests/e2e/CLAUDE.md` |
|
||||
|
||||
### Traits for Extensibility
|
||||
- `Database` - Add new database backends (must implement all ~78 methods)
|
||||
- `Channel` - Add new input sources
|
||||
- `Tool` - Add new capabilities
|
||||
- `LlmProvider` - Add new LLM backends
|
||||
- `SuccessEvaluator` - Custom evaluation logic
|
||||
- `EmbeddingProvider` - Add embedding backends (workspace search)
|
||||
- `NetworkPolicyDecider` - Custom network access policies for sandbox containers
|
||||
- `Hook` - Lifecycle hook at 6 interception points (BeforeInbound, BeforeToolCall, BeforeOutbound, OnSessionStart, OnSessionEnd, TransformResponse)
|
||||
- `Observer` - Observability backend (noop/log/multi; future: OpenTelemetry, Prometheus)
|
||||
- `Tunnel` - Tunnel provider for public internet exposure
|
||||
## Job State Machine
|
||||
|
||||
### Tool Implementation
|
||||
```rust
|
||||
#[async_trait]
|
||||
impl Tool for MyTool {
|
||||
fn name(&self) -> &str { "my_tool" }
|
||||
fn description(&self) -> &str { "Does something useful" }
|
||||
fn parameters_schema(&self) -> serde_json::Value {
|
||||
serde_json::json!({
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"param": { "type": "string", "description": "A parameter" }
|
||||
},
|
||||
"required": ["param"]
|
||||
})
|
||||
}
|
||||
|
||||
async fn execute(&self, params: serde_json::Value, ctx: &JobContext)
|
||||
-> Result<ToolOutput, ToolError>
|
||||
{
|
||||
let start = std::time::Instant::now();
|
||||
// ... do work ...
|
||||
Ok(ToolOutput::text("result", start.elapsed()))
|
||||
}
|
||||
|
||||
fn requires_sanitization(&self) -> bool { true } // External data
|
||||
}
|
||||
```
|
||||
|
||||
### State Transitions
|
||||
Job states follow a defined state machine in `context/state.rs`:
|
||||
```
|
||||
Pending -> InProgress -> Completed -> Submitted -> Accepted
|
||||
\-> Failed
|
||||
@@ -354,291 +164,17 @@ Pending -> InProgress -> Completed -> Submitted -> Accepted
|
||||
\-> Failed
|
||||
```
|
||||
|
||||
### Code Style
|
||||
## Skills System
|
||||
|
||||
- Use `crate::` imports, not `super::`
|
||||
- No `pub use` re-exports unless exposing to downstream consumers
|
||||
- Prefer strong types over strings (enums, newtypes)
|
||||
- Keep functions focused, extract helpers when logic is reused
|
||||
- Comments for non-obvious logic only
|
||||
SKILL.md files extend the agent's prompt with domain-specific instructions. See `.claude/rules/skills.md` for full details.
|
||||
|
||||
### Review & Fix Discipline
|
||||
|
||||
Hard-won lessons from code review -- follow these when fixing bugs or addressing review feedback.
|
||||
|
||||
**Fix the pattern, not just the instance:** When a reviewer flags a bug (e.g., TOCTOU race in INSERT + SELECT-back), search the entire codebase for all instances of that same pattern. A fix in `SecretsStore::create()` that doesn't also fix `WasmToolStore::store()` is half a fix.
|
||||
|
||||
**Propagate architectural fixes to satellite types:** If a core type changes its concurrency model (e.g., `LibSqlBackend` switches to connection-per-operation), every type that was handed a resource from the old model (e.g., `LibSqlSecretsStore`, `LibSqlWasmToolStore` holding a single `Connection`) must also be updated. Grep for the old type across the codebase.
|
||||
|
||||
**Schema translation is more than DDL:** When translating a database schema between backends (PostgreSQL to libSQL, etc.), check for:
|
||||
- **Indexes** -- diff `CREATE INDEX` statements between the two schemas
|
||||
- **Seed data** -- check for `INSERT INTO` in migrations (e.g., `leak_detection_patterns`)
|
||||
- **Semantic differences** -- document where SQL functions behave differently (e.g., `json_patch` vs `jsonb_set`)
|
||||
|
||||
**Feature flag testing:** When adding feature-gated code, test compilation with each feature in isolation:
|
||||
```bash
|
||||
cargo check # default features
|
||||
cargo check --no-default-features --features libsql # libsql only
|
||||
cargo check --all-features # all features
|
||||
```
|
||||
Dead code behind the wrong `#[cfg]` gate will only show up when building with a single feature.
|
||||
|
||||
**Regression test with every fix:** Every bug fix must include a test that would have caught the bug. Add a `#[test]` or `#[tokio::test]` that reproduces the original failure. Exempt: changes limited to `src/channels/web/static/` or `.md` files. Use `[skip-regression-check]` in commit message or PR label if genuinely not feasible. The `commit-msg` hook and CI workflow enforce this automatically.
|
||||
|
||||
**Zero clippy warnings policy:** Fix ALL clippy warnings before committing, including pre-existing ones in files you didn't change. Never leave warnings behind — treat `cargo clippy` output as a zero-tolerance gate.
|
||||
|
||||
**Mechanical verification before committing:** Run these checks on changed files before committing:
|
||||
- `cargo clippy --all --benches --tests --examples --all-features` -- zero warnings
|
||||
- `grep -rnE '\.unwrap\(|\.expect\(' <files>` -- no panics in production
|
||||
- `grep -rn 'super::' <files>` -- use `crate::` imports
|
||||
- If you fixed a pattern bug, `grep` for other instances of that pattern across `src/`
|
||||
- Fix commits must include regression tests (enforced by `commit-msg` hook; bypass with `[skip-regression-check]`)
|
||||
- **Trust model**: Trusted (user-placed in `~/.ironclaw/skills/` or workspace `skills/`, full tool access) vs Installed (registry, read-only tools)
|
||||
- **Selection pipeline**: gating (check bin/env/config requirements) -> scoring (keywords/patterns/tags) -> budget (fit within `SKILLS_MAX_TOKENS`) -> attenuation (trust-based tool ceiling)
|
||||
- **Skill tools**: `skill_list`, `skill_search`, `skill_install`, `skill_remove`
|
||||
|
||||
## Configuration
|
||||
|
||||
Environment variables (see `.env.example`):
|
||||
```bash
|
||||
# Database backend (default: postgres)
|
||||
DATABASE_BACKEND=postgres # or "libsql" / "turso"
|
||||
DATABASE_URL=postgres://user:pass@localhost/ironclaw
|
||||
LIBSQL_PATH=~/.ironclaw/ironclaw.db # libSQL local path (default)
|
||||
# LIBSQL_URL=libsql://xxx.turso.io # Turso cloud (optional)
|
||||
# LIBSQL_AUTH_TOKEN=xxx # Required with LIBSQL_URL
|
||||
|
||||
# NEAR AI (when LLM_BACKEND=nearai, the default)
|
||||
# Two auth modes: session token (default) or API key
|
||||
# Session token auth (default): uses browser OAuth on first run
|
||||
NEARAI_SESSION_TOKEN=sess_... # hosting providers: set this
|
||||
NEARAI_BASE_URL=https://private.near.ai
|
||||
# API key auth: set NEARAI_API_KEY, base URL defaults to cloud-api.near.ai
|
||||
# NEARAI_API_KEY=... # API key from cloud.near.ai
|
||||
NEARAI_MODEL=claude-3-5-sonnet-20241022
|
||||
|
||||
# Agent settings
|
||||
AGENT_NAME=ironclaw
|
||||
MAX_PARALLEL_JOBS=5
|
||||
|
||||
# Embeddings (for semantic memory search)
|
||||
OPENAI_API_KEY=sk-... # For OpenAI embeddings
|
||||
# Or use NEAR AI embeddings:
|
||||
# EMBEDDING_PROVIDER=nearai
|
||||
# EMBEDDING_ENABLED=true
|
||||
EMBEDDING_MODEL=text-embedding-3-small # or text-embedding-3-large
|
||||
|
||||
# Heartbeat (proactive periodic execution)
|
||||
HEARTBEAT_ENABLED=true
|
||||
HEARTBEAT_INTERVAL_SECS=1800 # 30 minutes
|
||||
HEARTBEAT_NOTIFY_CHANNEL=tui
|
||||
HEARTBEAT_NOTIFY_USER=default
|
||||
|
||||
# Web gateway
|
||||
GATEWAY_ENABLED=true
|
||||
GATEWAY_HOST=127.0.0.1
|
||||
GATEWAY_PORT=3001
|
||||
GATEWAY_AUTH_TOKEN=changeme # Required for API access
|
||||
GATEWAY_USER_ID=default
|
||||
|
||||
# Docker sandbox
|
||||
SANDBOX_ENABLED=true
|
||||
SANDBOX_IMAGE=ironclaw-worker:latest
|
||||
SANDBOX_MEMORY_LIMIT_MB=512
|
||||
SANDBOX_TIMEOUT_SECS=1800
|
||||
SANDBOX_CPU_LIMIT=1.0 # CPU cores per container
|
||||
SANDBOX_NETWORK_PROXY=true # Enable network proxy for containers
|
||||
SANDBOX_PROXY_PORT=8080 # Proxy listener port
|
||||
SANDBOX_DEFAULT_POLICY=workspace_write # ReadOnly, WorkspaceWrite, FullAccess
|
||||
|
||||
# Claude Code mode (runs inside sandbox containers)
|
||||
CLAUDE_CODE_ENABLED=false
|
||||
CLAUDE_CODE_MODEL=claude-sonnet-4-20250514
|
||||
CLAUDE_CODE_MAX_TURNS=50
|
||||
CLAUDE_CODE_CONFIG_DIR=/home/worker/.claude
|
||||
|
||||
# Routines (scheduled/reactive execution)
|
||||
ROUTINES_ENABLED=true
|
||||
ROUTINES_CRON_INTERVAL=60 # Tick interval in seconds
|
||||
ROUTINES_MAX_CONCURRENT=3
|
||||
|
||||
# Skills system
|
||||
SKILLS_ENABLED=true
|
||||
SKILLS_MAX_TOKENS=4000 # Max prompt budget per turn
|
||||
SKILLS_CATALOG_URL=https://clawhub.dev # ClawHub registry URL
|
||||
SKILLS_AUTO_DISCOVER=true # Scan skill directories on startup
|
||||
|
||||
# Tinfoil private inference
|
||||
TINFOIL_API_KEY=... # Required when LLM_BACKEND=tinfoil
|
||||
TINFOIL_MODEL=kimi-k2-5 # Default model
|
||||
|
||||
# Tunnel (public internet exposure for webhooks)
|
||||
TUNNEL_URL=https://abc123.ngrok.io # Static public URL (manual tunnel)
|
||||
# Or use a managed tunnel provider:
|
||||
TUNNEL_PROVIDER=none # none (default), cloudflare, tailscale, ngrok, custom
|
||||
TUNNEL_CF_TOKEN=... # Required for TUNNEL_PROVIDER=cloudflare
|
||||
TUNNEL_NGROK_TOKEN=... # Required for TUNNEL_PROVIDER=ngrok
|
||||
# TUNNEL_NGROK_DOMAIN=... # Custom domain (paid ngrok plan)
|
||||
# TUNNEL_TS_FUNNEL=true # Use tailscale funnel (public) vs serve (tailnet)
|
||||
TUNNEL_CUSTOM_COMMAND=... # Command with {host}/{port} for custom providers
|
||||
|
||||
# Observability backend
|
||||
OBSERVABILITY_BACKEND=none # none/noop (default) or log
|
||||
```
|
||||
|
||||
### LLM Providers
|
||||
|
||||
Backends: `nearai` (default), `openai`, `anthropic`, `ollama`, `openai_compatible`, `tinfoil` — set via `LLM_BACKEND`. See [src/llm/CLAUDE.md](src/llm/CLAUDE.md) for per-provider auth and configuration details.
|
||||
|
||||
## Database
|
||||
|
||||
Dual-backend persistence (PostgreSQL + libSQL/Turso). **All new persistence features must support both backends** — see [src/db/CLAUDE.md](src/db/CLAUDE.md) for schema, SQL dialect differences, adding operations, and libSQL limitations.
|
||||
|
||||
Implement every new operation in both `src/db/postgres.rs` and `src/db/libsql/mod.rs`. Test in isolation:
|
||||
```bash
|
||||
cargo check # postgres (default)
|
||||
cargo check --no-default-features --features libsql # libsql only
|
||||
cargo check --all-features # both
|
||||
```
|
||||
|
||||
Database configuration: see Configuration section above.
|
||||
|
||||
## Safety Layer
|
||||
|
||||
All external tool output passes through `SafetyLayer`:
|
||||
1. **Sanitizer** - Detects injection patterns, escapes dangerous content
|
||||
2. **Validator** - Checks length, encoding, forbidden patterns
|
||||
3. **Policy** - Rules with severity (Critical/High/Medium/Low) and actions (Block/Warn/Review/Sanitize)
|
||||
4. **Leak Detector** - Scans for 15+ secret patterns (API keys, tokens, private keys, connection strings) at two points: tool output before it reaches the LLM, and LLM responses before they reach the user. Actions per pattern: Block (reject entirely), Redact (mask the secret), or Warn (flag but allow)
|
||||
|
||||
Tool outputs are wrapped before reaching LLM:
|
||||
```xml
|
||||
<tool_output name="search" sanitized="true">
|
||||
[escaped content]
|
||||
</tool_output>
|
||||
```
|
||||
|
||||
### Shell Environment Scrubbing
|
||||
|
||||
The shell tool (`src/tools/builtin/shell.rs`) scrubs sensitive environment variables before executing commands, preventing secrets from leaking through `env`, `printenv`, or `$VAR` expansion. The sanitizer (`src/safety/sanitizer.rs`) also detects command injection patterns (chained commands, subshells, path traversal) and blocks or escapes them based on policy rules.
|
||||
|
||||
## Skills System
|
||||
|
||||
Skills are SKILL.md files that extend the agent's prompt with domain-specific instructions. Each skill is a YAML frontmatter block (metadata, activation criteria, required tools) followed by a markdown body that gets injected into the LLM context when the skill activates.
|
||||
|
||||
### Trust Model
|
||||
|
||||
| Trust Level | Source | Tool Access |
|
||||
|-------------|--------|-------------|
|
||||
| **Trusted** | User-placed in `~/.ironclaw/skills/` or workspace `skills/` | All tools available to the agent |
|
||||
| **Installed** | Downloaded from ClawHub registry | Read-only tools only (no shell, file write, HTTP) |
|
||||
|
||||
### SKILL.md Format
|
||||
|
||||
```yaml
|
||||
---
|
||||
name: my-skill
|
||||
version: 0.1.0
|
||||
description: Does something useful
|
||||
activation:
|
||||
patterns:
|
||||
- "deploy to.*production"
|
||||
keywords:
|
||||
- "deployment"
|
||||
max_context_tokens: 2000
|
||||
metadata:
|
||||
openclaw:
|
||||
requires:
|
||||
bins: [docker, kubectl]
|
||||
env: [KUBECONFIG]
|
||||
---
|
||||
|
||||
# Deployment Skill
|
||||
|
||||
Instructions for the agent when this skill activates...
|
||||
```
|
||||
|
||||
### Selection Pipeline
|
||||
|
||||
1. **Gating** -- Check binary/env/config requirements; skip skills whose prerequisites are missing
|
||||
2. **Scoring** -- Deterministic scoring against message content using keywords, tags, and regex patterns
|
||||
3. **Budget** -- Select top-scoring skills that fit within `SKILLS_MAX_TOKENS` prompt budget
|
||||
4. **Attenuation** -- Apply trust-based tool ceiling; installed skills lose access to dangerous tools
|
||||
|
||||
### Skill Tools
|
||||
|
||||
Four built-in tools for managing skills at runtime:
|
||||
- **`skill_list`** -- List all discovered skills with trust level and status
|
||||
- **`skill_search`** -- Search ClawHub registry for available skills
|
||||
- **`skill_install`** -- Download and install a skill from ClawHub
|
||||
- **`skill_remove`** -- Remove an installed skill
|
||||
|
||||
### Skill Directories
|
||||
|
||||
- `~/.ironclaw/skills/` -- User's global skills (trusted)
|
||||
- `<workspace>/skills/` -- Per-workspace skills (trusted)
|
||||
- `~/.ironclaw/installed_skills/` -- Registry-installed skills (installed trust)
|
||||
|
||||
### Testing Skills
|
||||
|
||||
- `skills/web-ui-test/` -- Manual test checklist for the web gateway UI via Claude for Chrome extension. Covers connection, chat, skills search/install/remove, and other tabs.
|
||||
|
||||
Skills configuration: see Configuration section above.
|
||||
|
||||
## Docker Sandbox
|
||||
|
||||
The `src/sandbox/` module provides Docker-based isolation for job execution with a network proxy that controls outbound access and injects credentials.
|
||||
|
||||
### Sandbox Policies
|
||||
|
||||
| Policy | Filesystem | Network | Use Case |
|
||||
|--------|-----------|---------|----------|
|
||||
| **ReadOnly** | Read-only workspace mount | Allowlisted domains only | Analysis, code review |
|
||||
| **WorkspaceWrite** | Read-write workspace mount | Allowlisted domains only | Code generation, file edits |
|
||||
| **FullAccess** | Full filesystem | Unrestricted | Trusted admin tasks |
|
||||
|
||||
### Network Proxy
|
||||
|
||||
Containers route all HTTP/HTTPS traffic through a host-side proxy (`src/sandbox/proxy/`):
|
||||
- **Domain allowlist** -- Only allowlisted domains are reachable (default: package registries, docs sites, GitHub, common APIs)
|
||||
- **Credential injection** -- The `CredentialResolver` trait injects auth headers into proxied requests so secrets never enter the container environment
|
||||
- **CONNECT tunnel** -- HTTPS traffic uses CONNECT method; the proxy validates the target domain against the allowlist before establishing the tunnel
|
||||
- **Policy decisions** -- The `NetworkPolicyDecider` trait allows custom logic for allow/deny/inject decisions per request
|
||||
|
||||
### Zero-Exposure Credential Model
|
||||
|
||||
Secrets (API keys, tokens) are stored encrypted on the host and injected into HTTP requests by the proxy at transit time. Container processes never have access to raw credential values, preventing exfiltration even if container code is compromised.
|
||||
|
||||
Sandbox configuration: see Configuration section above.
|
||||
|
||||
## Testing
|
||||
|
||||
Tests are in `mod tests {}` blocks at the bottom of each file. Run specific module tests:
|
||||
```bash
|
||||
cargo test safety::sanitizer::tests
|
||||
cargo test tools::registry::tests
|
||||
```
|
||||
|
||||
Key test patterns:
|
||||
- Unit tests for pure functions
|
||||
- Async tests with `#[tokio::test]`
|
||||
- No mocks, prefer real implementations or stubs
|
||||
|
||||
## Current Limitations / TODOs
|
||||
|
||||
1. **Domain-specific tools** - `marketplace.rs`, `restaurant.rs`, `taskrabbit.rs`, `ecommerce.rs` return placeholder responses; need real API integrations
|
||||
2. **Integration tests** - Need testcontainers setup for PostgreSQL
|
||||
3. **MCP stdio transport** - Only HTTP transport implemented
|
||||
4. **WIT bindgen integration** - Auto-extract tool description/schema from WASM modules (stubbed)
|
||||
5. **Capability granting after tool build** - Built tools get empty capabilities; need UX for granting HTTP/secrets access
|
||||
6. **Tool versioning workflow** - No version tracking or rollback for dynamically built tools
|
||||
7. **Full channel status view** - Gateway status widget exists, but no per-channel connection dashboard
|
||||
8. **Observability backends** - Only `log` and `noop` implemented; OpenTelemetry/Prometheus not yet supported
|
||||
|
||||
## Tool Architecture
|
||||
|
||||
**Keep tool-specific logic out of the main agent codebase.** The main agent provides generic infrastructure; tools are self-contained units that declare their requirements through `capabilities.json` files (API endpoints, credentials, rate limits, auth setup). Service-specific auth flows, CLI commands, and configuration do not belong in the main agent.
|
||||
|
||||
Tools can be built as **WASM** (sandboxed, credential-injected, single binary) or **MCP servers** (ecosystem of pre-built servers, any language, but no sandbox). Both are first-class via `ironclaw tool install`. Auth is declared in capabilities files with OAuth and manual token entry support.
|
||||
|
||||
See `src/tools/README.md` for full tool architecture, adding new tools (built-in Rust and WASM), auth JSON examples, and WASM vs MCP decision guide.
|
||||
See `.env.example` for all environment variables. LLM backends (`nearai`, `openai`, `anthropic`, `ollama`, `openai_compatible`, `tinfoil`, `bedrock`) documented in `src/llm/CLAUDE.md`.
|
||||
|
||||
## Adding a New Channel
|
||||
|
||||
@@ -647,48 +183,24 @@ See `src/tools/README.md` for full tool architecture, adding new tools (built-in
|
||||
3. Add config in `src/config/channels.rs`
|
||||
4. Wire up in `src/app.rs` channel setup section
|
||||
|
||||
## Workspace & Memory
|
||||
|
||||
Persistent memory with hybrid search (FTS + vector via RRF). Four tools: `memory_search`, `memory_write`, `memory_read`, `memory_tree`. Identity files (AGENTS.md, SOUL.md, USER.md, IDENTITY.md) injected into system prompt. Heartbeat system runs proactive periodic execution (default: 30 minutes), reading `HEARTBEAT.md` and notifying via channel if findings. See `src/workspace/README.md`.
|
||||
|
||||
## Debugging
|
||||
|
||||
```bash
|
||||
# Verbose logging
|
||||
RUST_LOG=ironclaw=trace cargo run
|
||||
|
||||
# Just the agent module
|
||||
RUST_LOG=ironclaw::agent=debug cargo run
|
||||
|
||||
# With HTTP request logging
|
||||
RUST_LOG=ironclaw=debug,tower_http=debug cargo run
|
||||
RUST_LOG=ironclaw=trace cargo run # verbose
|
||||
RUST_LOG=ironclaw::agent=debug cargo run # agent module only
|
||||
RUST_LOG=ironclaw=debug,tower_http=debug cargo run # + HTTP request logging
|
||||
```
|
||||
|
||||
## Module Specifications
|
||||
## Current Limitations
|
||||
|
||||
Some modules have a `README.md` that serves as the authoritative specification
|
||||
for that module's behavior. When modifying code in a module that has a spec:
|
||||
|
||||
1. **Read the spec first** before making changes
|
||||
2. **Code follows spec**: if the spec says X, the code must do X
|
||||
3. **Update both sides**: if you change behavior, update the spec to match;
|
||||
if you're implementing a spec change, update the code to match
|
||||
4. **Spec is the tiebreaker**: when code and spec disagree, the spec is correct
|
||||
(unless the spec is clearly outdated, in which case fix the spec first)
|
||||
|
||||
| Module | Spec File |
|
||||
|--------|-----------|
|
||||
| `src/setup/` | `src/setup/README.md` |
|
||||
| `src/workspace/` | `src/workspace/README.md` |
|
||||
| `src/tools/` | `src/tools/README.md` |
|
||||
| `src/agent/` | `src/agent/CLAUDE.md` |
|
||||
| `src/channels/web/` | `src/channels/web/CLAUDE.md` |
|
||||
| `src/db/` | `src/db/CLAUDE.md` |
|
||||
| `src/llm/` | `src/llm/CLAUDE.md` |
|
||||
| `tests/e2e/` | `tests/e2e/CLAUDE.md` |
|
||||
|
||||
## Workspace & Memory System
|
||||
|
||||
OpenClaw-inspired persistent memory with a flexible filesystem-like structure. Principle: "Memory is database, not RAM" -- if you want to remember something, write it explicitly. Uses hybrid search combining FTS (keyword) + vector (semantic) via Reciprocal Rank Fusion.
|
||||
|
||||
Four memory tools for LLM use: `memory_search` (hybrid search -- call before answering questions about prior work), `memory_write`, `memory_read`, `memory_tree`. Identity files (AGENTS.md, SOUL.md, USER.md, IDENTITY.md) are injected into the LLM system prompt.
|
||||
|
||||
The heartbeat system runs proactive periodic execution (default: 30 minutes), reading `HEARTBEAT.md` and notifying via channel if findings are detected.
|
||||
|
||||
See `src/workspace/README.md` for full API documentation, filesystem structure, hybrid search details, chunking strategy, and heartbeat system.
|
||||
1. Domain-specific tools (`marketplace.rs`, `restaurant.rs`, etc.) are stubs
|
||||
2. Integration tests need testcontainers for PostgreSQL
|
||||
3. MCP: no streaming support; stdio/HTTP/Unix transports all use request-response
|
||||
4. WIT bindgen: auto-extract tool schema from WASM is stubbed
|
||||
5. Built tools get empty capabilities; need UX for granting access
|
||||
6. No tool versioning or rollback
|
||||
7. Observability: only `log` and `noop` backends (no OpenTelemetry)
|
||||
|
||||
Generated
+707
-23
File diff suppressed because it is too large
Load Diff
+9
-1
@@ -56,7 +56,7 @@ rustls = { version = "0.23", optional = true, default-features = false }
|
||||
rustls-native-certs = { version = "0.8", optional = true }
|
||||
|
||||
# Database - libSQL/Turso (optional embedded database)
|
||||
libsql = { version = "0.6", optional = true, default-features = false, features = ["core", "replication"] }
|
||||
libsql = { version = "0.6", optional = true, default-features = false, features = ["core", "replication", "remote", "tls"] }
|
||||
|
||||
# Error handling
|
||||
thiserror = "2"
|
||||
@@ -73,6 +73,8 @@ toml = "0.8"
|
||||
# Core types
|
||||
uuid = { version = "1", features = ["v4", "v5", "serde"] }
|
||||
chrono = { version = "0.4", features = ["serde"] }
|
||||
chrono-tz = "0.10"
|
||||
iana-time-zone = "0.1"
|
||||
rust_decimal = { version = "1", features = ["serde", "serde-with-str", "maths"] }
|
||||
rust_decimal_macros = "1"
|
||||
|
||||
@@ -140,6 +142,11 @@ subtle = "2" # Constant-time comparisons for token validation
|
||||
# Multi-provider LLM support
|
||||
rig-core = "0.30"
|
||||
|
||||
# AWS Bedrock (native Converse API, opt-in via --features bedrock)
|
||||
aws-config = { version = "1", features = ["behavior-version-latest"], optional = true }
|
||||
aws-sdk-bedrockruntime = { version = "1", optional = true }
|
||||
aws-smithy-types = { version = "1", optional = true }
|
||||
|
||||
# Docker sandbox
|
||||
bollard = "0.18"
|
||||
|
||||
@@ -201,6 +208,7 @@ postgres = [
|
||||
libsql = ["dep:libsql"]
|
||||
integration = []
|
||||
html-to-markdown = ["dep:html-to-markdown-rs", "dep:readabilityrs"]
|
||||
bedrock = ["dep:aws-config", "dep:aws-sdk-bedrockruntime", "dep:aws-smithy-types"]
|
||||
|
||||
[[test]]
|
||||
name = "html_to_markdown"
|
||||
|
||||
+8
-4
@@ -39,7 +39,7 @@ This document tracks feature parity between IronClaw (Rust implementation) and O
|
||||
| Network modes (loopback/LAN/remote) | ✅ | 🚧 | HTTP only |
|
||||
| OpenAI-compatible HTTP API | ✅ | ✅ | /v1/chat/completions, per-request `model` override |
|
||||
| Canvas hosting | ✅ | ❌ | Agent-driven UI |
|
||||
| Gateway lock (PID-based) | ✅ | ❌ | |
|
||||
| Gateway lock (PID-based) | ✅ | ✅ | `fs4` flock-based, acquired in `main.rs` before agent startup |
|
||||
| launchd/systemd integration | ✅ | ❌ | |
|
||||
| Bonjour/mDNS discovery | ✅ | ❌ | |
|
||||
| Tailscale integration | ✅ | ❌ | |
|
||||
@@ -215,9 +215,13 @@ This document tracks feature parity between IronClaw (Rust implementation) and O
|
||||
| NEAR AI | ✅ | ✅ | - | Primary provider |
|
||||
| Anthropic (Claude) | ✅ | 🚧 | - | Via NEAR AI proxy; Opus 4.5, Sonnet 4, Sonnet 4.6 |
|
||||
| OpenAI | ✅ | 🚧 | - | Via NEAR AI proxy |
|
||||
| AWS Bedrock | ✅ | ❌ | P3 | |
|
||||
| Google Gemini | ✅ | ❌ | P3 | |
|
||||
| NVIDIA API | ✅ | ❌ | P3 | New provider |
|
||||
| AWS Bedrock | ✅ | ✅ | - | Native Converse API via aws-sdk-bedrockruntime (requires `--features bedrock`) |
|
||||
| Google Gemini | ✅ | ✅ | P3 | Via `gemini` adapter |
|
||||
| io.net | ✅ | ✅ | P3 | Via `ionet` adapter |
|
||||
| Mistral | ✅ | ✅ | P3 | Via `mistral` adapter |
|
||||
| Yandex AI Studio | ✅ | ✅ | P3 | Via `yandex` adapter |
|
||||
| Cloudflare Workers AI | ✅ | ✅ | P3 | Via `cloudflare` adapter |
|
||||
| NVIDIA API | ✅ | ✅ | P3 | Via `nvidia` adapter and `providers.json` |
|
||||
| OpenRouter | ✅ | ✅ | - | Via OpenAI-compatible provider (RigAdapter) |
|
||||
| Tinfoil | ❌ | ✅ | - | Private inference provider (IronClaw-only) |
|
||||
| OpenAI-compatible | ❌ | ✅ | - | Generic OpenAI-compatible endpoint (RigAdapter) |
|
||||
|
||||
@@ -14,6 +14,11 @@
|
||||
<a href="https://www.reddit.com/r/ironclawAI/"><img src="https://img.shields.io/badge/Reddit-r%2FironclawAI-FF4500?style=flat&logo=reddit&logoColor=white" alt="Reddit: r/ironclawAI" /></a>
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<a href="README.md">English</a> |
|
||||
<a href="README.zh-CN.md">简体中文</a>
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<a href="#philosophy">Philosophy</a> •
|
||||
<a href="#features">Features</a> •
|
||||
|
||||
+319
@@ -0,0 +1,319 @@
|
||||
<p align="center">
|
||||
<img src="ironclaw.png?v=2" alt="IronClaw" width="200"/>
|
||||
</p>
|
||||
|
||||
<h1 align="center">IronClaw</h1>
|
||||
|
||||
<p align="center">
|
||||
<strong>安全可靠的个人 AI 助手,始终站在你这边</strong>
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<a href="#license"><img src="https://img.shields.io/badge/license-MIT%20OR%20Apache%202.0-blue.svg" alt="License: MIT OR Apache-2.0" /></a>
|
||||
<a href="https://t.me/ironclawAI"><img src="https://img.shields.io/badge/Telegram-%40ironclawAI-26A5E4?style=flat&logo=telegram&logoColor=white" alt="Telegram: @ironclawAI" /></a>
|
||||
<a href="https://www.reddit.com/r/ironclawAI/"><img src="https://img.shields.io/badge/Reddit-r%2FironclawAI-FF4500?style=flat&logo=reddit&logoColor=white" alt="Reddit: r/ironclawAI" /></a>
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<a href="README.md">English</a> |
|
||||
<a href="README.zh-CN.md">简体中文</a>
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<a href="#设计理念">设计理念</a> •
|
||||
<a href="#功能特性">功能特性</a> •
|
||||
<a href="#安装">安装</a> •
|
||||
<a href="#配置">配置</a> •
|
||||
<a href="#安全机制">安全机制</a> •
|
||||
<a href="#系统架构">系统架构</a>
|
||||
</p>
|
||||
|
||||
---
|
||||
|
||||
## 设计理念
|
||||
|
||||
IronClaw 基于一个简单的原则:**你的 AI 助手应该为你服务,而不是与你为敌。**
|
||||
|
||||
在 AI 系统对数据处理日益不透明、与企业利益捆绑的今天,IronClaw 选择了一条不同的路:
|
||||
|
||||
- **数据归你所有** — 所有信息存储在本地,加密保护,始终在你掌控之下
|
||||
- **透明至上** — 完全开源,可审计,没有隐藏的遥测或数据收集
|
||||
- **自主扩展** — 随时构建新工具,无需等待供应商更新
|
||||
- **纵深防御** — 多层安全机制抵御提示注入和数据泄露
|
||||
|
||||
IronClaw 是一个你真正可以信赖的 AI 助手,无论是个人生活还是工作。
|
||||
|
||||
## 功能特性
|
||||
|
||||
### 安全优先
|
||||
|
||||
- **WASM 沙箱** — 不受信任的工具在隔离的 WebAssembly 容器中运行,采用基于能力的权限模型
|
||||
- **凭据保护** — 密钥永远不会暴露给工具;在宿主边界注入并进行泄露检测
|
||||
- **提示注入防御** — 模式检测、内容清理和策略执行
|
||||
- **端点白名单** — HTTP 请求仅限于明确批准的主机和路径
|
||||
|
||||
### 随时可用
|
||||
|
||||
- **多渠道接入** — REPL、HTTP webhook、WASM 渠道(Telegram、Slack)和 Web 网关
|
||||
- **Docker 沙箱** — 隔离的容器执行,支持每任务令牌和编排器/工作器模式
|
||||
- **Web 网关** — 浏览器 UI,支持实时 SSE/WebSocket 流式传输
|
||||
- **定时任务** — Cron 调度、事件触发器、Webhook 处理器,实现后台自动化
|
||||
- **心跳系统** — 主动后台执行,用于监控和维护任务
|
||||
- **并行任务** — 使用隔离上下文同时处理多个请求
|
||||
- **自修复** — 自动检测并恢复卡住的操作
|
||||
|
||||
### 自主扩展
|
||||
|
||||
- **动态工具构建** — 描述你的需求,IronClaw 会将其构建为 WASM 工具
|
||||
- **MCP 协议** — 连接模型上下文协议(Model Context Protocol)服务器以获取额外能力
|
||||
- **插件架构** — 无需重启即可加载新的 WASM 工具和渠道
|
||||
|
||||
### 持久记忆
|
||||
|
||||
- **混合搜索** — 全文搜索 + 向量搜索,采用倒数排名融合(Reciprocal Rank Fusion)
|
||||
- **工作空间文件系统** — 灵活的基于路径的存储,用于笔记、日志和上下文
|
||||
- **身份文件** — 跨会话保持一致的个性和偏好设置
|
||||
|
||||
## 安装
|
||||
|
||||
### 前置要求
|
||||
|
||||
- Rust 1.85+
|
||||
- PostgreSQL 15+,需安装 [pgvector](https://github.com/pgvector/pgvector) 扩展
|
||||
- NEAR AI 账户(通过设置向导进行身份验证)
|
||||
|
||||
## 下载或编译
|
||||
|
||||
访问 [Releases 页面](https://github.com/nearai/ironclaw/releases/) 查看最新版本。
|
||||
|
||||
<details>
|
||||
<summary>通过 Windows 安装程序安装 (Windows)</summary>
|
||||
|
||||
下载 [Windows 安装程序](https://github.com/nearai/ironclaw/releases/latest/download/ironclaw-x86_64-pc-windows-msvc.msi) 并运行。
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary>通过 PowerShell 脚本安装 (Windows)</summary>
|
||||
|
||||
```sh
|
||||
irm https://github.com/nearai/ironclaw/releases/latest/download/ironclaw-installer.ps1 | iex
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary>通过 Shell 脚本安装 (macOS、Linux、Windows/WSL)</summary>
|
||||
|
||||
```sh
|
||||
curl --proto '=https' --tlsv1.2 -LsSf https://github.com/nearai/ironclaw/releases/latest/download/ironclaw-installer.sh | sh
|
||||
```
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary>通过 Homebrew 安装 (macOS/Linux)</summary>
|
||||
|
||||
```sh
|
||||
brew install ironclaw
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary>从源码编译 (Windows、Linux、macOS 上使用 Cargo)</summary>
|
||||
|
||||
确保你已安装 [Rust](https://rustup.rs)。
|
||||
|
||||
```bash
|
||||
# 克隆仓库
|
||||
git clone https://github.com/nearai/ironclaw.git
|
||||
cd ironclaw
|
||||
|
||||
# 编译
|
||||
cargo build --release
|
||||
|
||||
# 运行测试
|
||||
cargo test
|
||||
```
|
||||
|
||||
如需进行**完整发布构建**(修改了渠道源码后),先运行 `./scripts/build-all.sh` 重新编译渠道。
|
||||
|
||||
</details>
|
||||
|
||||
### 数据库设置
|
||||
|
||||
```bash
|
||||
# 创建数据库
|
||||
createdb ironclaw
|
||||
|
||||
# 启用 pgvector 扩展
|
||||
psql ironclaw -c "CREATE EXTENSION IF NOT EXISTS vector;"
|
||||
```
|
||||
|
||||
## 配置
|
||||
|
||||
运行设置向导来配置 IronClaw:
|
||||
|
||||
```bash
|
||||
ironclaw onboard
|
||||
```
|
||||
|
||||
向导将引导你完成数据库连接、NEAR AI 身份验证(通过浏览器 OAuth)和密钥加密(使用系统钥匙串)。设置会保存在数据库中;引导变量(如 `DATABASE_URL`、`LLM_BACKEND`)写入 `~/.ironclaw/.env`,以便在数据库连接前可用。
|
||||
|
||||
### 替代 LLM 提供商
|
||||
|
||||
IronClaw 默认使用 NEAR AI,但兼容任何 OpenAI 兼容的端点。
|
||||
常用选项包括 **OpenRouter**(300+ 模型)、**Together AI**、**Fireworks AI**、**Ollama**(本地部署)以及自托管服务器如 **vLLM** 或 **LiteLLM**。
|
||||
|
||||
在向导中选择 *"OpenAI-compatible"*,或直接设置环境变量:
|
||||
|
||||
```env
|
||||
LLM_BACKEND=openai_compatible
|
||||
LLM_BASE_URL=https://openrouter.ai/api/v1
|
||||
LLM_API_KEY=sk-or-...
|
||||
LLM_MODEL=anthropic/claude-sonnet-4
|
||||
```
|
||||
|
||||
详见 [docs/LLM_PROVIDERS.md](docs/LLM_PROVIDERS.md) 获取完整的提供商指南。
|
||||
|
||||
## 安全机制
|
||||
|
||||
IronClaw 实现了纵深防御策略来保护你的数据并防止滥用。
|
||||
|
||||
### WASM 沙箱
|
||||
|
||||
所有不受信任的工具都在隔离的 WebAssembly 容器中运行:
|
||||
|
||||
- **基于能力的权限** — 明确授权 HTTP、密钥、工具调用等能力
|
||||
- **端点白名单** — HTTP 请求仅限已批准的主机和路径
|
||||
- **凭据注入** — 密钥在宿主边界注入,永远不会暴露给 WASM 代码
|
||||
- **泄露检测** — 扫描请求和响应以防止密钥外泄
|
||||
- **速率限制** — 每个工具独立的请求限制,防止滥用
|
||||
- **资源限制** — 内存、CPU 和执行时间约束
|
||||
|
||||
```
|
||||
WASM ──► 白名单 ──► 泄露扫描 ──► 凭据 ──► 执行 ──► 泄露扫描 ──► WASM
|
||||
验证器 (请求) 注入器 请求 (响应)
|
||||
```
|
||||
|
||||
### 提示注入防御
|
||||
|
||||
外部内容需通过多个安全层:
|
||||
|
||||
- 基于模式的注入尝试检测
|
||||
- 内容清理和转义
|
||||
- 带严重级别的策略规则(阻止/警告/审核/清理)
|
||||
- 工具输出包装,确保安全的 LLM 上下文注入
|
||||
|
||||
### 数据保护
|
||||
|
||||
- 所有数据存储在本地 PostgreSQL 数据库中
|
||||
- 密钥使用 AES-256-GCM 加密
|
||||
- 无遥测、无分析、无数据共享
|
||||
- 所有工具执行的完整审计日志
|
||||
|
||||
## 系统架构
|
||||
|
||||
```
|
||||
┌────────────────────────────────────────────────────────────────┐
|
||||
│ 渠道 │
|
||||
│ ┌──────┐ ┌──────┐ ┌─────────────┐ ┌─────────────┐ │
|
||||
│ │ REPL │ │ HTTP │ │ WASM 渠道 │ │ Web 网关 │ │
|
||||
│ └──┬───┘ └──┬───┘ └──────┬──────┘ │ (SSE + WS) │ │
|
||||
│ │ │ │ └──────┬──────┘ │
|
||||
│ └─────────┴──────────────┴────────────────┘ │
|
||||
│ │ │
|
||||
│ ┌─────────▼─────────┐ │
|
||||
│ │ 代理循环 │ 意图路由 │
|
||||
│ └────┬──────────┬───┘ │
|
||||
│ │ │ │
|
||||
│ ┌──────────▼────┐ ┌──▼───────────────┐ │
|
||||
│ │ 调度器 │ │ 定时任务引擎 │ │
|
||||
│ │ (并行任务) │ │(cron, 事件, wh) │ │
|
||||
│ └──────┬────────┘ └────────┬─────────┘ │
|
||||
│ │ │ │
|
||||
│ ┌─────────────┼────────────────────┘ │
|
||||
│ │ │ │
|
||||
│ ┌───▼─────┐ ┌────▼────────────────┐ │
|
||||
│ │ 本地 │ │ 编排器 │ │
|
||||
│ │ 工作器 │ │ ┌───────────────┐ │ │
|
||||
│ │(进程内) │ │ │ Docker 沙箱 │ │ │
|
||||
│ └───┬─────┘ │ │ 容器 │ │ │
|
||||
│ │ │ │ ┌───────────┐ │ │ │
|
||||
│ │ │ │ │工作器/CC │ │ │ │
|
||||
│ │ │ │ └───────────┘ │ │ │
|
||||
│ │ │ └───────────────┘ │ │
|
||||
│ │ └─────────┬───────────┘ │
|
||||
│ └──────────────────┤ │
|
||||
│ │ │
|
||||
│ ┌───────────▼──────────┐ │
|
||||
│ │ 工具注册表 │ │
|
||||
│ │ 内置、MCP、WASM │ │
|
||||
│ └──────────────────────┘ │
|
||||
└────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### 核心组件
|
||||
|
||||
| 组件 | 用途 |
|
||||
|------|------|
|
||||
| **代理循环** | 主消息处理和任务协调 |
|
||||
| **路由器** | 分类用户意图(命令、查询、任务) |
|
||||
| **调度器** | 管理带优先级的并行任务执行 |
|
||||
| **工作器** | 执行包含 LLM 推理和工具调用的任务 |
|
||||
| **编排器** | 容器生命周期、LLM 代理、每任务认证 |
|
||||
| **Web 网关** | 浏览器 UI,含聊天、记忆、任务、日志、扩展、定时任务 |
|
||||
| **定时任务引擎** | 定时(cron)和响应式(事件、webhook)后台任务 |
|
||||
| **工作空间** | 带混合搜索的持久记忆 |
|
||||
| **安全层** | 提示注入防御和内容清理 |
|
||||
|
||||
## 使用方式
|
||||
|
||||
```bash
|
||||
# 首次设置(配置数据库、认证等)
|
||||
ironclaw onboard
|
||||
|
||||
# 启动交互式 REPL
|
||||
cargo run
|
||||
|
||||
# 启用调试日志
|
||||
RUST_LOG=ironclaw=debug cargo run
|
||||
```
|
||||
|
||||
## 开发
|
||||
|
||||
```bash
|
||||
# 格式化代码
|
||||
cargo fmt
|
||||
|
||||
# 代码检查
|
||||
cargo clippy --all --benches --tests --examples --all-features
|
||||
|
||||
# 运行测试
|
||||
createdb ironclaw_test
|
||||
cargo test
|
||||
|
||||
# 运行指定测试
|
||||
cargo test test_name
|
||||
```
|
||||
|
||||
- **Telegram 渠道**:参见 [docs/TELEGRAM_SETUP.md](docs/TELEGRAM_SETUP.md) 了解设置和私信配对。
|
||||
- **修改渠道源码**:在 `cargo build` 之前运行 `./channels-src/telegram/build.sh` 以便打包更新后的 WASM。
|
||||
|
||||
## OpenClaw 传承
|
||||
|
||||
IronClaw 是受 [OpenClaw](https://github.com/openclaw/openclaw) 启发的 Rust 重新实现。参见 [FEATURE_PARITY.md](FEATURE_PARITY.md) 了解完整的功能追踪矩阵。
|
||||
|
||||
主要差异:
|
||||
|
||||
- **Rust vs TypeScript** — 原生性能、内存安全、单一二进制文件
|
||||
- **WASM 沙箱 vs Docker** — 轻量级、基于能力的安全机制
|
||||
- **PostgreSQL vs SQLite** — 生产级持久化存储
|
||||
- **安全优先设计** — 多层防御、凭据保护
|
||||
|
||||
## 许可证
|
||||
|
||||
可选择以下任一许可证:
|
||||
|
||||
- Apache License, Version 2.0 ([LICENSE-APACHE](LICENSE-APACHE))
|
||||
- MIT License ([LICENSE-MIT](LICENSE-MIT))
|
||||
Generated
+1
-1
@@ -267,7 +267,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "slack-channel"
|
||||
version = "0.1.0"
|
||||
version = "0.2.1"
|
||||
dependencies = [
|
||||
"hex",
|
||||
"hmac",
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
[package]
|
||||
name = "slack-channel"
|
||||
version = "0.2.0"
|
||||
version = "0.2.1"
|
||||
edition = "2021"
|
||||
description = "Slack Events API channel for IronClaw"
|
||||
license = "MIT OR Apache-2.0"
|
||||
|
||||
@@ -357,10 +357,108 @@ fn extract_slack_attachments(files: &Option<Vec<SlackFile>>) -> Vec<InboundAttac
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// Download a file from Slack using the url_private endpoint.
|
||||
///
|
||||
/// Slack file downloads require Bearer auth with the bot token, which is
|
||||
/// injected by the host credential system via `channel_host::http_request`.
|
||||
fn download_slack_file(url: &str) -> Result<Vec<u8>, String> {
|
||||
let headers = serde_json::json!({});
|
||||
|
||||
let result = channel_host::http_request("GET", url, &headers.to_string(), None, None);
|
||||
|
||||
let response = result.map_err(|e| format!("Slack file download failed: {}", e))?;
|
||||
|
||||
if response.status != 200 {
|
||||
let body_str = String::from_utf8_lossy(&response.body);
|
||||
return Err(format!(
|
||||
"Slack file download returned {}: {}",
|
||||
response.status, body_str
|
||||
));
|
||||
}
|
||||
|
||||
Ok(response.body)
|
||||
}
|
||||
|
||||
/// Download file bytes and store them via the host for processing.
|
||||
///
|
||||
/// Downloads all file types (images, documents, etc.) so the host-side
|
||||
/// middleware can process them (vision pipeline for images, text extraction
|
||||
/// for documents, transcription for audio, etc.).
|
||||
/// Maximum file size to download (20 MB). Files larger than this are skipped
|
||||
/// to avoid excessive memory use and slow downloads in the WASM runtime.
|
||||
const MAX_DOWNLOAD_SIZE_BYTES: u64 = 20 * 1024 * 1024;
|
||||
|
||||
fn download_and_store_slack_files(attachments: &[InboundAttachment]) {
|
||||
for att in attachments {
|
||||
let Some(ref url) = att.source_url else {
|
||||
continue;
|
||||
};
|
||||
|
||||
// Skip files that exceed the size limit
|
||||
if let Some(size) = att.size_bytes {
|
||||
if size > MAX_DOWNLOAD_SIZE_BYTES {
|
||||
channel_host::log(
|
||||
channel_host::LogLevel::Warn,
|
||||
&format!(
|
||||
"Skipping Slack file download: {} bytes exceeds {} MB limit (id={})",
|
||||
size,
|
||||
MAX_DOWNLOAD_SIZE_BYTES / (1024 * 1024),
|
||||
att.id
|
||||
),
|
||||
);
|
||||
continue;
|
||||
}
|
||||
}
|
||||
|
||||
match download_slack_file(url) {
|
||||
Ok(bytes) => {
|
||||
// Post-download size guard: metadata size_bytes is optional,
|
||||
// so a file with no size info could bypass the pre-download check.
|
||||
if bytes.len() as u64 > MAX_DOWNLOAD_SIZE_BYTES {
|
||||
channel_host::log(
|
||||
channel_host::LogLevel::Warn,
|
||||
&format!(
|
||||
"Discarding Slack file after download: {} bytes exceeds {} MB limit (id={})",
|
||||
bytes.len(),
|
||||
MAX_DOWNLOAD_SIZE_BYTES / (1024 * 1024),
|
||||
att.id
|
||||
),
|
||||
);
|
||||
continue;
|
||||
}
|
||||
|
||||
channel_host::log(
|
||||
channel_host::LogLevel::Info,
|
||||
&format!(
|
||||
"Downloaded Slack file: {} bytes, mime={}",
|
||||
bytes.len(),
|
||||
att.mime_type
|
||||
),
|
||||
);
|
||||
if let Err(e) = channel_host::store_attachment_data(&att.id, &bytes) {
|
||||
channel_host::log(
|
||||
channel_host::LogLevel::Error,
|
||||
&format!("Failed to store Slack file data: {}", e),
|
||||
);
|
||||
}
|
||||
}
|
||||
Err(e) => {
|
||||
channel_host::log(
|
||||
channel_host::LogLevel::Error,
|
||||
&format!("Failed to download Slack file: {}", e),
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Handle a Slack event and emit message if applicable.
|
||||
fn handle_slack_event(event: SlackEvent, team_id: Option<String>, _event_id: Option<String>) {
|
||||
let attachments = extract_slack_attachments(&event.files);
|
||||
|
||||
// Download and store file attachments for host-side processing
|
||||
download_and_store_slack_files(&attachments);
|
||||
|
||||
match event.event_type.as_str() {
|
||||
// Direct mention of the bot (always in a channel, not a DM)
|
||||
"app_mention" => {
|
||||
@@ -722,4 +820,10 @@ mod tests {
|
||||
let event: SlackEvent = serde_json::from_str(json).unwrap();
|
||||
assert!(event.files.is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_max_download_size_constant() {
|
||||
// Verify the constant is 20 MB
|
||||
assert_eq!(MAX_DOWNLOAD_SIZE_BYTES, 20 * 1024 * 1024);
|
||||
}
|
||||
}
|
||||
|
||||
Generated
+1
-1
@@ -212,7 +212,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "telegram-channel"
|
||||
version = "0.2.0"
|
||||
version = "0.2.1"
|
||||
dependencies = [
|
||||
"serde",
|
||||
"serde_json",
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
[package]
|
||||
name = "telegram-channel"
|
||||
version = "0.2.0"
|
||||
version = "0.2.1"
|
||||
edition = "2021"
|
||||
description = "Telegram Bot API channel for IronClaw"
|
||||
license = "MIT OR Apache-2.0"
|
||||
|
||||
@@ -878,10 +878,6 @@ fn send_message(
|
||||
// Voice File Download
|
||||
// ============================================================================
|
||||
|
||||
/// Download a voice file from Telegram by file_id.
|
||||
///
|
||||
/// 1. Call getFile to get the file_path.
|
||||
/// 2. Download the file bytes from /file/bot{TOKEN}/{file_path}.
|
||||
/// Percent-encode a string for safe use as a URL query parameter value.
|
||||
fn percent_encode(s: &str) -> String {
|
||||
let mut out = String::with_capacity(s.len());
|
||||
@@ -898,6 +894,10 @@ fn percent_encode(s: &str) -> String {
|
||||
out
|
||||
}
|
||||
|
||||
/// Maximum file size to download (20 MB). Files larger than this are discarded
|
||||
/// to avoid excessive memory use and slow downloads in the WASM runtime.
|
||||
const MAX_DOWNLOAD_SIZE_BYTES: u64 = 20 * 1024 * 1024;
|
||||
|
||||
fn download_telegram_file(file_id: &str) -> Result<Vec<u8>, String> {
|
||||
// Reject file_id containing curly braces to prevent credential placeholder injection
|
||||
if file_id.contains('{') || file_id.contains('}') {
|
||||
@@ -965,6 +965,16 @@ fn download_telegram_file(file_id: &str) -> Result<Vec<u8>, String> {
|
||||
));
|
||||
}
|
||||
|
||||
// Post-download size guard: Telegram metadata file_size is optional,
|
||||
// so enforce the limit on actual downloaded bytes.
|
||||
if response.body.len() as u64 > MAX_DOWNLOAD_SIZE_BYTES {
|
||||
return Err(format!(
|
||||
"Downloaded file exceeds {} MB limit ({} bytes)",
|
||||
MAX_DOWNLOAD_SIZE_BYTES / (1024 * 1024),
|
||||
response.body.len()
|
||||
));
|
||||
}
|
||||
|
||||
Ok(response.body)
|
||||
}
|
||||
|
||||
@@ -1535,6 +1545,39 @@ fn download_and_store_voice(attachments: &[InboundAttachment]) {
|
||||
}
|
||||
}
|
||||
|
||||
/// Download image file bytes and store them via the host for the vision pipeline.
|
||||
///
|
||||
/// Separated from `extract_attachments` so that function stays pure (no host
|
||||
/// calls) and remains testable in native unit tests.
|
||||
fn download_and_store_images(attachments: &[InboundAttachment]) {
|
||||
for att in attachments {
|
||||
if !att.mime_type.starts_with("image/") {
|
||||
continue;
|
||||
}
|
||||
|
||||
match download_telegram_file(&att.id) {
|
||||
Ok(bytes) => {
|
||||
channel_host::log(
|
||||
channel_host::LogLevel::Info,
|
||||
&format!("Downloaded image file: {} bytes", bytes.len()),
|
||||
);
|
||||
if let Err(e) = channel_host::store_attachment_data(&att.id, &bytes) {
|
||||
channel_host::log(
|
||||
channel_host::LogLevel::Error,
|
||||
&format!("Failed to store image data: {}", e),
|
||||
);
|
||||
}
|
||||
}
|
||||
Err(e) => {
|
||||
channel_host::log(
|
||||
channel_host::LogLevel::Error,
|
||||
&format!("Failed to download image file: {}", e),
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Returns true if the attachment should be downloaded for document text extraction.
|
||||
///
|
||||
/// Excludes voice (handled by transcription), image (vision pipeline),
|
||||
@@ -1608,6 +1651,9 @@ fn handle_message(message: TelegramMessage) {
|
||||
// Download and store voice attachments for host-side transcription
|
||||
download_and_store_voice(&attachments);
|
||||
|
||||
// Download and store image attachments for host-side vision pipeline
|
||||
download_and_store_images(&attachments);
|
||||
|
||||
// Download and store document attachments for host-side text extraction
|
||||
download_and_store_documents(&mut attachments);
|
||||
|
||||
@@ -1681,7 +1727,7 @@ fn handle_message(message: TelegramMessage) {
|
||||
let username_opt = from.username.as_deref();
|
||||
let is_allowed = allowed.contains(&"*".to_string())
|
||||
|| allowed.contains(&id_str)
|
||||
|| username_opt.map_or(false, |u| allowed.contains(&u.to_string()));
|
||||
|| username_opt.is_some_and(|u| allowed.contains(&u.to_string()));
|
||||
|
||||
if !is_allowed {
|
||||
if is_private && dm_policy == "pairing" {
|
||||
@@ -2605,4 +2651,10 @@ mod tests {
|
||||
assert!(!is_downloadable_document(&make("audio/mpeg", Some("song.mp3"))));
|
||||
assert!(!is_downloadable_document(&make("video/mp4", Some("clip.mp4"))));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_max_download_size_constant() {
|
||||
// Verify the constant is 20 MB, matching the Slack channel limit
|
||||
assert_eq!(MAX_DOWNLOAD_SIZE_BYTES, 20 * 1024 * 1024);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,9 +1,17 @@
|
||||
{
|
||||
"version": "0.2.0",
|
||||
"version": "0.2.2",
|
||||
"wit_version": "0.3.0",
|
||||
"type": "channel",
|
||||
"name": "telegram",
|
||||
"description": "Telegram Bot API channel for receiving and responding to Telegram messages",
|
||||
"auth": {
|
||||
"secret_name": "telegram_bot_token",
|
||||
"display_name": "Telegram",
|
||||
"instructions": "Get your bot token from @BotFather on Telegram (https://t.me/BotFather). Send /newbot or /token to get it.",
|
||||
"setup_url": "https://t.me/BotFather",
|
||||
"token_hint": "Looks like 123456789:AABBccDDeeFFgg...",
|
||||
"env_var": "TELEGRAM_BOT_TOKEN"
|
||||
},
|
||||
"setup": {
|
||||
"required_secrets": [
|
||||
{
|
||||
|
||||
+1
-1
@@ -3,7 +3,7 @@ services:
|
||||
postgres:
|
||||
image: pgvector/pgvector:pg16
|
||||
ports:
|
||||
- "5432:5432"
|
||||
- "127.0.0.1:5432:5432"
|
||||
environment:
|
||||
POSTGRES_DB: ironclaw
|
||||
POSTGRES_USER: ironclaw
|
||||
|
||||
@@ -11,7 +11,13 @@ configurations.
|
||||
| NEAR AI | `nearai` | OAuth (browser) | Default; multi-model |
|
||||
| Anthropic | `anthropic` | `ANTHROPIC_API_KEY` | Claude models |
|
||||
| OpenAI | `openai` | `OPENAI_API_KEY` | GPT models |
|
||||
| Google Gemini | `gemini` | `GEMINI_API_KEY` | Gemini models |
|
||||
| io.net | `ionet` | `IONET_API_KEY` | Intelligence API |
|
||||
| Mistral | `mistral` | `MISTRAL_API_KEY` | Mistral models |
|
||||
| Yandex AI Studio | `yandex` | `YANDEX_API_KEY` | YandexGPT models |
|
||||
| Cloudflare Workers AI | `cloudflare` | `CLOUDFLARE_API_KEY` | Access to Workers AI |
|
||||
| Ollama | `ollama` | No | Local inference |
|
||||
| AWS Bedrock | `bedrock` | AWS credentials | Native Converse API |
|
||||
| OpenRouter | `openai_compatible` | `LLM_API_KEY` | 300+ models |
|
||||
| Together AI | `openai_compatible` | `LLM_API_KEY` | Fast inference |
|
||||
| Fireworks AI | `openai_compatible` | `LLM_API_KEY` | Fast inference |
|
||||
@@ -68,6 +74,55 @@ Pull a model first: `ollama pull llama3.2`
|
||||
|
||||
---
|
||||
|
||||
## AWS Bedrock (requires `--features bedrock`)
|
||||
|
||||
Uses the native AWS Converse API via `aws-sdk-bedrockruntime`. Supports standard AWS
|
||||
authentication methods: IAM credentials, SSO profiles, and instance roles.
|
||||
|
||||
> **Build prerequisite:** The `aws-lc-sys` crate (transitive dependency via AWS SDK)
|
||||
> requires **CMake** to compile. Install it before building with `--features bedrock`:
|
||||
> - macOS: `brew install cmake`
|
||||
> - Ubuntu/Debian: `sudo apt install cmake`
|
||||
> - Fedora: `sudo dnf install cmake`
|
||||
|
||||
### With AWS credentials (IAM, SSO, instance roles)
|
||||
|
||||
```env
|
||||
LLM_BACKEND=bedrock
|
||||
BEDROCK_MODEL=anthropic.claude-opus-4-6-v1
|
||||
BEDROCK_REGION=us-east-1
|
||||
BEDROCK_CROSS_REGION=us
|
||||
# AWS_PROFILE=my-sso-profile # optional, for named profiles
|
||||
```
|
||||
|
||||
The AWS SDK credential chain automatically resolves credentials from environment
|
||||
variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`), shared credentials file
|
||||
(`~/.aws/credentials`), SSO profiles, and EC2/ECS instance roles.
|
||||
|
||||
### Cross-region inference
|
||||
|
||||
Set `BEDROCK_CROSS_REGION` to route requests across AWS regions for capacity:
|
||||
|
||||
| Prefix | Routing |
|
||||
|---|---|
|
||||
| `us` | US regions (us-east-1, us-east-2, us-west-2) |
|
||||
| `eu` | European regions |
|
||||
| `apac` | Asia-Pacific regions |
|
||||
| `global` | All commercial AWS regions |
|
||||
| _(unset)_ | Single-region only |
|
||||
|
||||
### Popular Bedrock model IDs
|
||||
|
||||
| Model | ID |
|
||||
|---|---|
|
||||
| Claude Opus 4.6 | `anthropic.claude-opus-4-6-v1` |
|
||||
| Claude Sonnet 4.5 | `anthropic.claude-sonnet-4-5-20250929-v1:0` |
|
||||
| Claude Haiku 4.5 | `anthropic.claude-haiku-4-5-20251001-v1:0` |
|
||||
| Amazon Nova Pro | `amazon.nova-pro-v1:0` |
|
||||
| Llama 4 Maverick | `meta.llama4-maverick-17b-instruct-v1:0` |
|
||||
|
||||
---
|
||||
|
||||
## OpenAI-Compatible Endpoints
|
||||
|
||||
All providers below use `LLM_BACKEND=openai_compatible`. Set `LLM_BASE_URL` to the
|
||||
|
||||
@@ -0,0 +1,13 @@
|
||||
-- Partial unique indexes to prevent duplicate singleton conversations.
|
||||
-- These guard against TOCTOU races in get_or_create_routine_conversation
|
||||
-- and get_or_create_heartbeat_conversation.
|
||||
|
||||
-- One routine conversation per user per routine_id.
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS uq_conv_routine
|
||||
ON conversations (user_id, (metadata->>'routine_id'))
|
||||
WHERE metadata->>'routine_id' IS NOT NULL;
|
||||
|
||||
-- One heartbeat conversation per user.
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS uq_conv_heartbeat
|
||||
ON conversations (user_id)
|
||||
WHERE metadata->>'thread_type' = 'heartbeat';
|
||||
+141
-11
@@ -1,7 +1,9 @@
|
||||
[
|
||||
{
|
||||
"id": "openai",
|
||||
"aliases": ["open_ai"],
|
||||
"aliases": [
|
||||
"open_ai"
|
||||
],
|
||||
"protocol": "open_ai_completions",
|
||||
"api_key_env": "OPENAI_API_KEY",
|
||||
"api_key_required": true,
|
||||
@@ -19,7 +21,9 @@
|
||||
},
|
||||
{
|
||||
"id": "anthropic",
|
||||
"aliases": ["claude"],
|
||||
"aliases": [
|
||||
"claude"
|
||||
],
|
||||
"protocol": "anthropic",
|
||||
"api_key_env": "ANTHROPIC_API_KEY",
|
||||
"api_key_required": true,
|
||||
@@ -52,7 +56,10 @@
|
||||
},
|
||||
{
|
||||
"id": "openai_compatible",
|
||||
"aliases": ["openai-compatible", "compatible"],
|
||||
"aliases": [
|
||||
"openai-compatible",
|
||||
"compatible"
|
||||
],
|
||||
"protocol": "open_ai_completions",
|
||||
"base_url_env": "LLM_BASE_URL",
|
||||
"base_url_required": true,
|
||||
@@ -89,7 +96,9 @@
|
||||
},
|
||||
{
|
||||
"id": "openrouter",
|
||||
"aliases": ["open_router"],
|
||||
"aliases": [
|
||||
"open_router"
|
||||
],
|
||||
"protocol": "open_ai_completions",
|
||||
"default_base_url": "https://openrouter.ai/api/v1",
|
||||
"api_key_env": "OPENROUTER_API_KEY",
|
||||
@@ -126,7 +135,10 @@
|
||||
},
|
||||
{
|
||||
"id": "nvidia",
|
||||
"aliases": ["nvidia_nim", "nim"],
|
||||
"aliases": [
|
||||
"nvidia_nim",
|
||||
"nim"
|
||||
],
|
||||
"protocol": "open_ai_completions",
|
||||
"default_base_url": "https://integrate.api.nvidia.com/v1",
|
||||
"api_key_env": "NVIDIA_API_KEY",
|
||||
@@ -144,7 +156,10 @@
|
||||
},
|
||||
{
|
||||
"id": "venice",
|
||||
"aliases": ["venice_ai", "veniceai"],
|
||||
"aliases": [
|
||||
"venice_ai",
|
||||
"veniceai"
|
||||
],
|
||||
"protocol": "open_ai_completions",
|
||||
"default_base_url": "https://api.venice.ai/api/v1",
|
||||
"api_key_env": "VENICE_API_KEY",
|
||||
@@ -162,7 +177,10 @@
|
||||
},
|
||||
{
|
||||
"id": "together",
|
||||
"aliases": ["together_ai", "togetherai"],
|
||||
"aliases": [
|
||||
"together_ai",
|
||||
"togetherai"
|
||||
],
|
||||
"protocol": "open_ai_completions",
|
||||
"default_base_url": "https://api.together.xyz/v1",
|
||||
"api_key_env": "TOGETHER_API_KEY",
|
||||
@@ -180,7 +198,9 @@
|
||||
},
|
||||
{
|
||||
"id": "fireworks",
|
||||
"aliases": ["fireworks_ai"],
|
||||
"aliases": [
|
||||
"fireworks_ai"
|
||||
],
|
||||
"protocol": "open_ai_completions",
|
||||
"default_base_url": "https://api.fireworks.ai/inference/v1",
|
||||
"api_key_env": "FIREWORKS_API_KEY",
|
||||
@@ -198,7 +218,9 @@
|
||||
},
|
||||
{
|
||||
"id": "deepseek",
|
||||
"aliases": ["deep_seek"],
|
||||
"aliases": [
|
||||
"deep_seek"
|
||||
],
|
||||
"protocol": "open_ai_completions",
|
||||
"default_base_url": "https://api.deepseek.com/v1",
|
||||
"api_key_env": "DEEPSEEK_API_KEY",
|
||||
@@ -234,7 +256,9 @@
|
||||
},
|
||||
{
|
||||
"id": "sambanova",
|
||||
"aliases": ["samba_nova"],
|
||||
"aliases": [
|
||||
"samba_nova"
|
||||
],
|
||||
"protocol": "open_ai_completions",
|
||||
"default_base_url": "https://api.sambanova.ai/v1",
|
||||
"api_key_env": "SAMBANOVA_API_KEY",
|
||||
@@ -249,5 +273,111 @@
|
||||
"display_name": "SambaNova",
|
||||
"can_list_models": false
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "gemini",
|
||||
"aliases": [
|
||||
"google_gemini",
|
||||
"google"
|
||||
],
|
||||
"protocol": "open_ai_completions",
|
||||
"default_base_url": "https://generativelanguage.googleapis.com/v1beta/openai",
|
||||
"api_key_env": "GEMINI_API_KEY",
|
||||
"api_key_required": true,
|
||||
"model_env": "GEMINI_MODEL",
|
||||
"default_model": "gemini-2.5-flash",
|
||||
"description": "Google Gemini (via OpenAI-compatible endpoint)",
|
||||
"setup": {
|
||||
"kind": "api_key",
|
||||
"secret_name": "llm_gemini_api_key",
|
||||
"key_url": "https://aistudio.google.com/app/apikey",
|
||||
"display_name": "Google Gemini",
|
||||
"can_list_models": true
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "ionet",
|
||||
"aliases": [
|
||||
"io_net",
|
||||
"io.net"
|
||||
],
|
||||
"protocol": "open_ai_completions",
|
||||
"default_base_url": "https://api.intelligence.io.solutions/api/v1",
|
||||
"api_key_env": "IONET_API_KEY",
|
||||
"api_key_required": true,
|
||||
"model_env": "IONET_MODEL",
|
||||
"default_model": "deepseek-coder-v2-instruct",
|
||||
"description": "io.net Intelligence API",
|
||||
"setup": {
|
||||
"kind": "api_key",
|
||||
"secret_name": "llm_ionet_api_key",
|
||||
"key_url": "https://cloud.io.net/intelligence",
|
||||
"display_name": "io.net",
|
||||
"can_list_models": true
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "mistral",
|
||||
"aliases": [
|
||||
"mistral_ai",
|
||||
"mistralai"
|
||||
],
|
||||
"protocol": "open_ai_completions",
|
||||
"default_base_url": "https://api.mistral.ai/v1",
|
||||
"api_key_env": "MISTRAL_API_KEY",
|
||||
"api_key_required": true,
|
||||
"model_env": "MISTRAL_MODEL",
|
||||
"default_model": "mistral-large-latest",
|
||||
"description": "Mistral AI API",
|
||||
"setup": {
|
||||
"kind": "api_key",
|
||||
"secret_name": "llm_mistral_api_key",
|
||||
"key_url": "https://console.mistral.ai/api-keys",
|
||||
"display_name": "Mistral",
|
||||
"can_list_models": true
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "yandex",
|
||||
"aliases": [
|
||||
"yandex_ai_studio",
|
||||
"yandexgpt",
|
||||
"yandex_gpt"
|
||||
],
|
||||
"protocol": "open_ai_completions",
|
||||
"default_base_url": "https://ai.api.cloud.yandex.net/v1",
|
||||
"api_key_env": "YANDEX_API_KEY",
|
||||
"api_key_required": true,
|
||||
"model_env": "YANDEX_MODEL",
|
||||
"extra_headers_env": "YANDEX_EXTRA_HEADERS",
|
||||
"default_model": "yandexgpt-lite",
|
||||
"description": "Yandex AI Studio (YandexGPT)",
|
||||
"setup": {
|
||||
"kind": "api_key",
|
||||
"secret_name": "llm_yandex_api_key",
|
||||
"key_url": "https://aistudio.yandex.ru/platform/folders/",
|
||||
"display_name": "Yandex AI Studio",
|
||||
"can_list_models": true
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "cloudflare",
|
||||
"aliases": [
|
||||
"cloudflare_ai",
|
||||
"cf_ai"
|
||||
],
|
||||
"protocol": "open_ai_completions",
|
||||
"api_key_env": "CLOUDFLARE_API_KEY",
|
||||
"api_key_required": true,
|
||||
"base_url_env": "CLOUDFLARE_BASE_URL",
|
||||
"model_env": "CLOUDFLARE_MODEL",
|
||||
"default_model": "@cf/meta/llama-3.3-70b-instruct-fp8-fast",
|
||||
"description": "Cloudflare Workers AI",
|
||||
"setup": {
|
||||
"kind": "open_ai_compatible",
|
||||
"secret_name": "llm_cloudflare_api_key",
|
||||
"display_name": "Cloudflare Workers AI",
|
||||
"can_list_models": false
|
||||
}
|
||||
}
|
||||
]
|
||||
]
|
||||
@@ -19,7 +19,7 @@
|
||||
"artifacts": {
|
||||
"wasm32-wasip2": {
|
||||
"url": "https://github.com/nearai/ironclaw/releases/latest/download/discord-wasm32-wasip2.tar.gz",
|
||||
"sha256": null
|
||||
"sha256": "030707431717bca3411a48f311c6ab5f92a45c747de26cafe4f6e3e23a8b3b2d"
|
||||
}
|
||||
},
|
||||
"auth_summary": {
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
"name": "slack",
|
||||
"display_name": "Slack Channel",
|
||||
"kind": "channel",
|
||||
"version": "0.2.0",
|
||||
"version": "0.2.1",
|
||||
"wit_version": "0.3.0",
|
||||
"description": "Talk to your agent in Slack",
|
||||
"keywords": [
|
||||
@@ -19,7 +19,7 @@
|
||||
"artifacts": {
|
||||
"wasm32-wasip2": {
|
||||
"url": "https://github.com/nearai/ironclaw/releases/latest/download/slack-wasm32-wasip2.tar.gz",
|
||||
"sha256": null
|
||||
"sha256": "6ed36077b67ac70a041f06f760f93ba79b33269885413c3c3f2c8c87ee60807e"
|
||||
}
|
||||
},
|
||||
"auth_summary": {
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
"name": "telegram",
|
||||
"display_name": "Telegram Channel",
|
||||
"kind": "channel",
|
||||
"version": "0.2.0",
|
||||
"version": "0.2.2",
|
||||
"wit_version": "0.3.0",
|
||||
"description": "Talk to your agent through a Telegram bot",
|
||||
"keywords": [
|
||||
@@ -19,7 +19,7 @@
|
||||
"artifacts": {
|
||||
"wasm32-wasip2": {
|
||||
"url": "https://github.com/nearai/ironclaw/releases/latest/download/telegram-wasm32-wasip2.tar.gz",
|
||||
"sha256": null
|
||||
"sha256": "98c86895a9c4b0a1e19fe8a47f1ccbfe7e972e112b05e584bc897130dc32283a"
|
||||
}
|
||||
},
|
||||
"auth_summary": {
|
||||
|
||||
@@ -19,7 +19,7 @@
|
||||
"artifacts": {
|
||||
"wasm32-wasip2": {
|
||||
"url": "https://github.com/nearai/ironclaw/releases/latest/download/whatsapp-wasm32-wasip2.tar.gz",
|
||||
"sha256": null
|
||||
"sha256": "bd35cad18d87292ea8d2f52db9b514ed9f814a414de910f59073d475c26c4c14"
|
||||
}
|
||||
},
|
||||
"auth_summary": {
|
||||
|
||||
@@ -20,7 +20,7 @@
|
||||
"artifacts": {
|
||||
"wasm32-wasip2": {
|
||||
"url": "https://github.com/nearai/ironclaw/releases/latest/download/github-wasm32-wasip2.tar.gz",
|
||||
"sha256": null
|
||||
"sha256": "6fcd32719a4ff15641a4b50fff8984686550f0c491dce60518f4126857d0c544"
|
||||
}
|
||||
},
|
||||
"auth_summary": {
|
||||
|
||||
@@ -19,7 +19,7 @@
|
||||
"artifacts": {
|
||||
"wasm32-wasip2": {
|
||||
"url": "https://github.com/nearai/ironclaw/releases/latest/download/gmail-wasm32-wasip2.tar.gz",
|
||||
"sha256": null
|
||||
"sha256": "023da7000b17568bf0e64b2e5013c8a042b2f323c85f1632339231c73d500e39"
|
||||
}
|
||||
},
|
||||
"auth_summary": {
|
||||
|
||||
@@ -19,7 +19,7 @@
|
||||
"artifacts": {
|
||||
"wasm32-wasip2": {
|
||||
"url": "https://github.com/nearai/ironclaw/releases/latest/download/google-calendar-wasm32-wasip2.tar.gz",
|
||||
"sha256": null
|
||||
"sha256": "fc42277b65881d6e9bcc5403dc54c7f5b3ddeaaaf04617fce2c5da05d76325f0"
|
||||
}
|
||||
},
|
||||
"auth_summary": {
|
||||
|
||||
@@ -19,7 +19,7 @@
|
||||
"artifacts": {
|
||||
"wasm32-wasip2": {
|
||||
"url": "https://github.com/nearai/ironclaw/releases/latest/download/google-docs-wasm32-wasip2.tar.gz",
|
||||
"sha256": null
|
||||
"sha256": "385c04abd1e6b8011ccc330e1f4bd7ce58577e488959b51594aa04eb26cbe7cc"
|
||||
}
|
||||
},
|
||||
"auth_summary": {
|
||||
|
||||
@@ -19,7 +19,7 @@
|
||||
"artifacts": {
|
||||
"wasm32-wasip2": {
|
||||
"url": "https://github.com/nearai/ironclaw/releases/latest/download/google-drive-wasm32-wasip2.tar.gz",
|
||||
"sha256": null
|
||||
"sha256": "1b107d575a5d52cc8c76d9a681802190f4373fb485f7f54f445533f097fa37c0"
|
||||
}
|
||||
},
|
||||
"auth_summary": {
|
||||
|
||||
@@ -19,7 +19,7 @@
|
||||
"artifacts": {
|
||||
"wasm32-wasip2": {
|
||||
"url": "https://github.com/nearai/ironclaw/releases/latest/download/google-sheets-wasm32-wasip2.tar.gz",
|
||||
"sha256": null
|
||||
"sha256": "c4f6b1e8c5126ac2c8a4b98e4283a3afa32223d2488fc3c3a609758c0c9beb90"
|
||||
}
|
||||
},
|
||||
"auth_summary": {
|
||||
|
||||
@@ -18,7 +18,7 @@
|
||||
"artifacts": {
|
||||
"wasm32-wasip2": {
|
||||
"url": "https://github.com/nearai/ironclaw/releases/latest/download/google-slides-wasm32-wasip2.tar.gz",
|
||||
"sha256": null
|
||||
"sha256": "7110b8565340c888e51f99e9c013bf4de8f8a7f7b33bace00eb8fc47831ff20b"
|
||||
}
|
||||
},
|
||||
"auth_summary": {
|
||||
|
||||
@@ -18,7 +18,7 @@
|
||||
"artifacts": {
|
||||
"wasm32-wasip2": {
|
||||
"url": "https://github.com/nearai/ironclaw/releases/latest/download/slack-tool-wasm32-wasip2.tar.gz",
|
||||
"sha256": null
|
||||
"sha256": "6ed36077b67ac70a041f06f760f93ba79b33269885413c3c3f2c8c87ee60807e"
|
||||
}
|
||||
},
|
||||
"auth_summary": {
|
||||
|
||||
@@ -19,7 +19,7 @@
|
||||
"artifacts": {
|
||||
"wasm32-wasip2": {
|
||||
"url": "https://github.com/nearai/ironclaw/releases/latest/download/telegram-mtproto-wasm32-wasip2.tar.gz",
|
||||
"sha256": null
|
||||
"sha256": "98c86895a9c4b0a1e19fe8a47f1ccbfe7e972e112b05e584bc897130dc32283a"
|
||||
}
|
||||
},
|
||||
"auth_summary": {
|
||||
|
||||
@@ -19,7 +19,7 @@
|
||||
"artifacts": {
|
||||
"wasm32-wasip2": {
|
||||
"url": "https://github.com/nearai/ironclaw/releases/latest/download/web-search-wasm32-wasip2.tar.gz",
|
||||
"sha256": null
|
||||
"sha256": "66cb2b9b00652385e9f30f17c74902b9222c17c53e9d3bd1ef42f5cab705bcf6"
|
||||
}
|
||||
},
|
||||
"auth_summary": {
|
||||
|
||||
@@ -209,6 +209,56 @@ else
|
||||
fi
|
||||
echo
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Check 6: LLM module isolation — no imports from other crate modules
|
||||
# --------------------------------------------------------------------------
|
||||
# src/llm/ should only import from:
|
||||
# - crate::llm (self-references)
|
||||
# - external crates (no crate:: prefix)
|
||||
# It must NOT import from crate::agent, crate::tools, crate::channels,
|
||||
# crate::safety, crate::config, crate::bootstrap, crate::cli, crate::db,
|
||||
# crate::workspace, crate::worker, crate::orchestrator, crate::skills,
|
||||
# crate::hooks, crate::setup, crate::context, etc.
|
||||
#
|
||||
# Test-only imports (crate::testing) are excluded since they don't affect
|
||||
# the runtime dependency graph and won't exist in the extracted crate.
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
echo "--- Check 6: LLM module isolation ---"
|
||||
|
||||
# Match any `crate::` reference (use-imports AND inline paths) that isn't
|
||||
# crate::llm or crate::testing. Filter out comments.
|
||||
# We strip inline comments (everything after //) with sed before checking,
|
||||
# so a line like `real_code(crate::foo); // crate::llm` is still caught.
|
||||
results=$(grep -rn 'crate::' src/llm/ \
|
||||
--include='*.rs' \
|
||||
| grep -v '^\s*//' \
|
||||
| sed 's|//.*||' \
|
||||
| grep 'crate::' \
|
||||
| grep -v 'crate::llm' \
|
||||
| grep -v 'crate::testing' \
|
||||
|| true)
|
||||
|
||||
if [ -n "$results" ]; then
|
||||
count=$(echo "$results" | wc -l | tr -d ' ')
|
||||
echo "WARNING: src/llm/ has $count reference(s) to modules outside crate::llm:"
|
||||
echo "$results"
|
||||
echo
|
||||
echo "(These are pre-existing; fix them before extracting the crate.)"
|
||||
echo "(New 'use crate::' imports are hard violations — see below.)"
|
||||
echo
|
||||
# Hard-fail only on new `use crate::` imports (easy to avoid in new code).
|
||||
use_imports=$(echo "$results" | grep '^[^:]*:.*use crate::' || true)
|
||||
if [ -n "$use_imports" ]; then
|
||||
echo "HARD VIOLATION: new 'use crate::' imports in src/llm/:"
|
||||
echo "$use_imports"
|
||||
violations=$((violations + 1))
|
||||
fi
|
||||
else
|
||||
echo "OK"
|
||||
fi
|
||||
echo
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Summary
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
@@ -51,9 +51,11 @@ echo "[6/6] Installing git hooks..."
|
||||
HOOKS_DIR=$(git rev-parse --git-path hooks 2>/dev/null) || true
|
||||
if [ -n "$HOOKS_DIR" ]; then
|
||||
mkdir -p "$HOOKS_DIR"
|
||||
SCRIPT_ABS="$(cd "$(dirname "$0")" && pwd)/commit-msg-regression.sh"
|
||||
ln -sf "$SCRIPT_ABS" "$HOOKS_DIR/commit-msg"
|
||||
SCRIPTS_ABS="$(cd "$(dirname "$0")" && pwd)"
|
||||
ln -sf "$SCRIPTS_ABS/commit-msg-regression.sh" "$HOOKS_DIR/commit-msg"
|
||||
echo " commit-msg hook installed (regression test enforcement)"
|
||||
ln -sf "$SCRIPTS_ABS/pre-commit-safety.sh" "$HOOKS_DIR/pre-commit"
|
||||
echo " pre-commit hook installed (UTF-8, case-sensitivity, /tmp, redaction checks)"
|
||||
else
|
||||
echo " Skipped: not a git repository"
|
||||
fi
|
||||
|
||||
Executable
+136
@@ -0,0 +1,136 @@
|
||||
#!/usr/bin/env bash
|
||||
# Pre-commit safety checks for common issues caught by AI code reviewers.
|
||||
#
|
||||
# Can be run standalone: bash scripts/pre-commit-safety.sh
|
||||
# Or installed as a git pre-commit hook via dev-setup.sh.
|
||||
#
|
||||
# Checks staged .rs files for:
|
||||
# 1. Unsafe UTF-8 byte slicing (panics on multi-byte chars)
|
||||
# 2. Case-sensitive file extension comparisons
|
||||
# 3. Hardcoded /tmp paths in tests (flaky in parallel runs)
|
||||
# 4. Tool parameters logged without redaction (secret leaks)
|
||||
# 5. Multi-step DB operations without transaction wrapping
|
||||
#
|
||||
# Suppress individual lines with an inline "// safety: <reason>" comment.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# Determine a suitable base ref for standalone diffs.
|
||||
resolve_base_ref() {
|
||||
local candidates=(
|
||||
"@{upstream}"
|
||||
"origin/HEAD"
|
||||
"origin/main"
|
||||
"origin/master"
|
||||
"main"
|
||||
"master"
|
||||
)
|
||||
|
||||
for ref in "${candidates[@]}"; do
|
||||
if git rev-parse --verify --quiet "$ref" >/dev/null 2>&1; then
|
||||
echo "$ref"
|
||||
return 0
|
||||
fi
|
||||
done
|
||||
|
||||
echo "pre-commit-safety: could not determine a base Git ref for diff (tried: ${candidates[*]})." >&2
|
||||
echo "pre-commit-safety: ensure your repository has an upstream or a local main/master branch." >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
# Support both pre-commit hook (staged files) and standalone (all changed vs base)
|
||||
if git diff --cached --quiet 2>/dev/null; then
|
||||
# No staged changes -- compare working tree against a resolved base ref
|
||||
BASE_REF="$(resolve_base_ref)"
|
||||
DIFF_OUTPUT=$(git diff "$BASE_REF" -- '*.rs' 2>/dev/null || true)
|
||||
else
|
||||
DIFF_OUTPUT=$(git diff --cached -U0 -- '*.rs' 2>/dev/null || true)
|
||||
fi
|
||||
|
||||
# Early exit if there are no relevant .rs changes
|
||||
if [ -z "$DIFF_OUTPUT" ]; then
|
||||
exit 0
|
||||
fi
|
||||
|
||||
WARNINGS=0
|
||||
|
||||
warn() {
|
||||
if [ "$WARNINGS" -eq 0 ]; then
|
||||
echo ""
|
||||
echo "=== Pre-commit Safety Checks ==="
|
||||
echo ""
|
||||
fi
|
||||
WARNINGS=$((WARNINGS + 1))
|
||||
echo " [$1] $2"
|
||||
}
|
||||
|
||||
# 1. Unsafe UTF-8 byte slicing: &s[..N] or &s[..some_var] on strings
|
||||
# Safe patterns: is_char_boundary, char_indices, // safety:
|
||||
if echo "$DIFF_OUTPUT" | grep -nE '^\+' | grep -E '\[\.\..*\]' | grep -vE 'is_char_boundary|char_indices|// safety:|as_bytes|Vec<|&\[u8\]|\[u8\]|bytes\(\)|&bytes' | head -3 | grep -q .; then
|
||||
warn "UTF8" "Possible unsafe byte-index string slicing. Use is_char_boundary() or char_indices()."
|
||||
echo "$DIFF_OUTPUT" | grep -nE '^\+' | grep -E '\[\.\..*\]' | grep -vE 'is_char_boundary|char_indices|// safety:|as_bytes|Vec<|&\[u8\]|\[u8\]|bytes\(\)|&bytes' | head -3 | sed 's/^/ /'
|
||||
fi
|
||||
|
||||
# 2. Case-sensitive file extension checks
|
||||
# Match: .ends_with(".png") without prior to_lowercase
|
||||
if echo "$DIFF_OUTPUT" | grep -nE '^\+.*ends_with\("\.([pP][nN][gG]|[jJ][pP][eE]?[gG]|[gG][iI][fF]|[wW][eE][bB][pP]|[mM][dD])"\)' | grep -vE 'to_lowercase|to_ascii_lowercase|// safety:' | head -3 | grep -q .; then
|
||||
warn "CASE" "Case-sensitive file extension comparison. Normalize to lowercase first."
|
||||
echo "$DIFF_OUTPUT" | grep -nE '^\+.*ends_with\("\.([pP][nN][gG]|[jJ][pP][eE]?[gG]|[gG][iI][fF]|[wW][eE][bB][pP]|[mM][dD])"\)' | grep -vE 'to_lowercase|to_ascii_lowercase|// safety:' | head -3 | sed 's/^/ /'
|
||||
fi
|
||||
|
||||
# 3. Hardcoded /tmp paths in test files
|
||||
if echo "$DIFF_OUTPUT" | grep -nE '^\+.*"/tmp/' | grep -vE 'tempfile|tempdir|// safety:' | head -3 | grep -q .; then
|
||||
warn "TMPDIR" "Hardcoded /tmp path. Use tempfile::tempdir() for parallel-safe tests."
|
||||
echo "$DIFF_OUTPUT" | grep -nE '^\+.*"/tmp/' | grep -vE 'tempfile|tempdir|// safety:' | head -3 | sed 's/^/ /'
|
||||
fi
|
||||
|
||||
# 4. Logging tool parameters without redaction
|
||||
if echo "$DIFF_OUTPUT" | grep -nE '^\+.*tracing::(info|debug|warn|error).*param' | grep -vE 'redact|// safety:' | head -3 | grep -q .; then
|
||||
warn "REDACT" "Logging tool parameters without redaction. Use redact_params() first."
|
||||
echo "$DIFF_OUTPUT" | grep -nE '^\+.*tracing::(info|debug|warn|error).*param' | grep -vE 'redact|// safety:' | head -3 | sed 's/^/ /'
|
||||
fi
|
||||
|
||||
# 5. Multi-step DB operations without transaction
|
||||
# Uses -W (function context) to reduce false positives from existing transactions.
|
||||
# Suppressible with "// safety:" in the hunk.
|
||||
DIFF_W_OUTPUT=$(git diff --cached -W -- '*.rs' 2>/dev/null || git diff "$(resolve_base_ref)" -W -- '*.rs' 2>/dev/null || true)
|
||||
if [ -n "$DIFF_W_OUTPUT" ]; then
|
||||
HUNK_COUNT=$(echo "$DIFF_W_OUTPUT" | awk '
|
||||
/^@@/ {
|
||||
if (count >= 2 && !has_tx && !has_safety) found++
|
||||
count=0; has_tx=0; has_safety=0
|
||||
}
|
||||
/^\+.*\.(execute|query)\(/ { count++ }
|
||||
/^\+.*(transaction|\.tx\.|\.begin\()/ { has_tx=1 }
|
||||
/ .*(transaction|\.tx\.|\.begin\()/ { has_tx=1 }
|
||||
/\/\/ safety:/ { has_safety=1 }
|
||||
END {
|
||||
if (count >= 2 && !has_tx && !has_safety) found++
|
||||
print found+0
|
||||
}
|
||||
')
|
||||
if [ "$HUNK_COUNT" -gt 0 ]; then
|
||||
warn "TX" "Multiple DB operations in same function without transaction. Wrap in a transaction for atomicity."
|
||||
echo "$DIFF_W_OUTPUT" | awk '
|
||||
/^@@/ {
|
||||
if (count >= 2 && !has_tx && !has_safety) { print buf }
|
||||
buf=""; count=0; has_tx=0; has_safety=0
|
||||
}
|
||||
/^\+.*\.(execute|query)\(/ { count++ }
|
||||
/^\+.*(transaction|\.tx\.|\.begin\()/ { has_tx=1 }
|
||||
/ .*(transaction|\.tx\.|\.begin\()/ { has_tx=1 }
|
||||
/\/\/ safety:/ { has_safety=1 }
|
||||
{ buf = buf "\n" $0 }
|
||||
END {
|
||||
if (count >= 2 && !has_tx && !has_safety) { print buf }
|
||||
}
|
||||
' | grep -E '^\+.*\.(execute|query)\(' | head -4 | sed 's/^/ /'
|
||||
fi
|
||||
fi
|
||||
|
||||
if [ "$WARNINGS" -gt 0 ]; then
|
||||
echo ""
|
||||
echo "Found $WARNINGS potential issue(s). Fix them or add '// safety: <reason>' to suppress."
|
||||
echo ""
|
||||
exit 1
|
||||
fi
|
||||
@@ -0,0 +1,54 @@
|
||||
---
|
||||
name: review-checklist
|
||||
version: 0.1.0
|
||||
description: Pre-merge review checklist based on recurring AI reviewer feedback patterns
|
||||
activation:
|
||||
patterns:
|
||||
- "review.*checklist"
|
||||
- "ready to merge"
|
||||
- "pre-merge check"
|
||||
- "check.*before.*merge"
|
||||
keywords:
|
||||
- review
|
||||
- checklist
|
||||
- merge
|
||||
- pre-merge
|
||||
max_context_tokens: 1500
|
||||
---
|
||||
|
||||
# Pre-Merge Review Checklist
|
||||
|
||||
Before merging, verify these items. They represent the most common issues caught by automated code reviewers (Copilot, Gemini) on IronClaw PRs.
|
||||
|
||||
## Database Operations
|
||||
- [ ] Multi-step DB operations are wrapped in transactions (INSERT+INSERT, UPDATE+DELETE, read-modify-write)
|
||||
- [ ] Both postgres AND libsql backends updated for any new Database trait methods
|
||||
- [ ] Migrations are atomic (SQL execution + version recording in same transaction)
|
||||
|
||||
## Security & Data Safety
|
||||
- [ ] Tool parameters are redacted via `redact_params()` before logging or SSE/WebSocket broadcast
|
||||
- [ ] URL validation resolves DNS before checking for private/loopback IPs (anti-SSRF via DNS rebinding)
|
||||
- [ ] Destructive tools have `requires_approval()` returning `Always` or `UnlessAutoApproved`
|
||||
- [ ] Data from worker containers is treated as untrusted (tool domain checks, server-side nesting depth)
|
||||
- [ ] No secrets or credentials in error messages, logs, or SSE events
|
||||
|
||||
## String Safety
|
||||
- [ ] No byte-index slicing (`&s[..n]`) on external/user strings -- use `is_char_boundary()` or `char_indices()`
|
||||
- [ ] File extension and media type comparisons are case-insensitive (`.to_ascii_lowercase()` before matching)
|
||||
- [ ] Path comparisons are case-insensitive where needed (macOS/Windows filesystems)
|
||||
|
||||
## Trait Wrappers & Decorator Chain
|
||||
- [ ] New `LlmProvider` trait methods are delegated in ALL wrapper types (grep `impl LlmProvider for`)
|
||||
- [ ] New trait methods are tested through the full decorator/provider chain, not just the base impl
|
||||
- [ ] Default trait method implementations are intentional -- wrappers that silently return defaults are bugs
|
||||
|
||||
## Tests
|
||||
- [ ] Temporary files/dirs use `tempfile` crate, no hardcoded `/tmp/` paths
|
||||
- [ ] Tests don't mutate global statics without synchronization (use per-test state or `serial_test`)
|
||||
- [ ] Tests don't make real network requests (use mocks, stubs, or RFC 5737 TEST-NET IPs like 192.0.2.1)
|
||||
- [ ] Test names and comments match actual test behavior and assertions
|
||||
|
||||
## Comments & Documentation
|
||||
- [ ] Code comments match actual behavior (especially route paths, tool names, function semantics)
|
||||
- [ ] Spec/README files updated if module behavior changed
|
||||
- [ ] Error messages are clear and non-redundant (don't nest tool name inside tool error that already contains it)
|
||||
+30
-2
@@ -96,6 +96,9 @@ pub struct Agent {
|
||||
pub(super) heartbeat_config: Option<HeartbeatConfig>,
|
||||
pub(super) hygiene_config: Option<crate::config::HygieneConfig>,
|
||||
pub(super) routine_config: Option<RoutineConfig>,
|
||||
/// Optional slot to expose the routine engine to the gateway for manual triggering.
|
||||
pub(super) routine_engine_slot:
|
||||
Option<Arc<tokio::sync::RwLock<Option<Arc<crate::agent::routine_engine::RoutineEngine>>>>>,
|
||||
}
|
||||
|
||||
impl Agent {
|
||||
@@ -148,9 +151,18 @@ impl Agent {
|
||||
heartbeat_config,
|
||||
hygiene_config,
|
||||
routine_config,
|
||||
routine_engine_slot: None,
|
||||
}
|
||||
}
|
||||
|
||||
/// Set the routine engine slot for exposing the engine to the gateway.
|
||||
pub fn set_routine_engine_slot(
|
||||
&mut self,
|
||||
slot: Arc<tokio::sync::RwLock<Option<Arc<crate::agent::routine_engine::RoutineEngine>>>>,
|
||||
) {
|
||||
self.routine_engine_slot = Some(slot);
|
||||
}
|
||||
|
||||
// Convenience accessors
|
||||
|
||||
/// Get the scheduler (for external wiring, e.g. CreateJobTool).
|
||||
@@ -342,8 +354,19 @@ impl Agent {
|
||||
let heartbeat_handle = if let Some(ref hb_config) = self.heartbeat_config {
|
||||
if hb_config.enabled {
|
||||
if let Some(workspace) = self.workspace() {
|
||||
let config = AgentHeartbeatConfig::default()
|
||||
let mut config = AgentHeartbeatConfig::default()
|
||||
.with_interval(std::time::Duration::from_secs(hb_config.interval_secs));
|
||||
config.quiet_hours_start = hb_config.quiet_hours_start;
|
||||
config.quiet_hours_end = hb_config.quiet_hours_end;
|
||||
config.timezone = hb_config
|
||||
.timezone
|
||||
.clone()
|
||||
.or_else(|| Some(self.config.default_timezone.clone()));
|
||||
if let (Some(user), Some(channel)) =
|
||||
(&hb_config.notify_user, &hb_config.notify_channel)
|
||||
{
|
||||
config = config.with_notify(user, channel);
|
||||
}
|
||||
|
||||
// Set up notification channel
|
||||
let (notify_tx, mut notify_rx) =
|
||||
@@ -394,8 +417,8 @@ impl Agent {
|
||||
hygiene,
|
||||
workspace.clone(),
|
||||
self.cheap_llm().clone(),
|
||||
self.safety().clone(),
|
||||
Some(notify_tx),
|
||||
self.store().map(Arc::clone),
|
||||
))
|
||||
} else {
|
||||
tracing::warn!("Heartbeat enabled but no workspace available");
|
||||
@@ -486,6 +509,11 @@ impl Agent {
|
||||
// SAFETY: self is consumed by run(), we can smuggle the engine in
|
||||
// via a local to use in the message loop below.
|
||||
|
||||
// Expose engine to gateway for manual triggering
|
||||
if let Some(ref slot) = self.routine_engine_slot {
|
||||
*slot.write().await = Some(Arc::clone(&engine));
|
||||
}
|
||||
|
||||
tracing::info!(
|
||||
"Routines enabled: cron ticker every {}s, max {} concurrent",
|
||||
rt_config.cron_check_interval_secs,
|
||||
|
||||
+48
-7
@@ -345,7 +345,6 @@ impl Agent {
|
||||
crate::workspace::hygiene::HygieneConfig::default(),
|
||||
workspace.clone(),
|
||||
self.llm().clone(),
|
||||
self.safety().clone(),
|
||||
);
|
||||
|
||||
match runner.check_heartbeat().await {
|
||||
@@ -406,7 +405,7 @@ impl Agent {
|
||||
.with_max_tokens(512)
|
||||
.with_temperature(0.3);
|
||||
|
||||
let reasoning = Reasoning::new(self.llm().clone(), self.safety().clone());
|
||||
let reasoning = Reasoning::new(self.llm().clone());
|
||||
match reasoning.complete(request).await {
|
||||
Ok((text, _usage)) => Ok(SubmissionResult::response(format!(
|
||||
"Thread Summary:\n\n{}",
|
||||
@@ -454,7 +453,7 @@ impl Agent {
|
||||
.with_max_tokens(512)
|
||||
.with_temperature(0.5);
|
||||
|
||||
let reasoning = Reasoning::new(self.llm().clone(), self.safety().clone());
|
||||
let reasoning = Reasoning::new(self.llm().clone());
|
||||
match reasoning.complete(request).await {
|
||||
Ok((text, _usage)) => Ok(SubmissionResult::response(format!(
|
||||
"Suggested Next Steps:\n\n{}",
|
||||
@@ -663,10 +662,14 @@ impl Agent {
|
||||
}
|
||||
|
||||
match self.llm().set_model(requested) {
|
||||
Ok(()) => Ok(SubmissionResult::response(format!(
|
||||
"Switched model to: {}",
|
||||
requested
|
||||
))),
|
||||
Ok(()) => {
|
||||
// Persist the model choice so it survives restarts.
|
||||
self.persist_selected_model(requested).await;
|
||||
Ok(SubmissionResult::response(format!(
|
||||
"Switched model to: {}",
|
||||
requested
|
||||
)))
|
||||
}
|
||||
Err(e) => Ok(SubmissionResult::error(format!(
|
||||
"Failed to switch model: {}",
|
||||
e
|
||||
@@ -822,4 +825,42 @@ impl Agent {
|
||||
_ => Ok(None),
|
||||
}
|
||||
}
|
||||
|
||||
/// Persist the selected model to the settings store (DB and/or TOML config).
|
||||
///
|
||||
/// Best-effort: logs warnings on failure but does not propagate errors,
|
||||
/// since the in-memory model switch already succeeded.
|
||||
async fn persist_selected_model(&self, model: &str) {
|
||||
// 1. Persist to DB if available.
|
||||
if let Some(store) = self.store() {
|
||||
let value = serde_json::Value::String(model.to_string());
|
||||
if let Err(e) = store.set_setting("default", "selected_model", &value).await {
|
||||
tracing::warn!("Failed to persist model to DB: {}", e);
|
||||
}
|
||||
}
|
||||
|
||||
// 2. Update TOML config file if it exists (sync I/O in spawn_blocking).
|
||||
let model_owned = model.to_string();
|
||||
if let Err(e) = tokio::task::spawn_blocking(move || {
|
||||
let toml_path = crate::settings::Settings::default_toml_path();
|
||||
match crate::settings::Settings::load_toml(&toml_path) {
|
||||
Ok(Some(mut settings)) => {
|
||||
settings.selected_model = Some(model_owned);
|
||||
if let Err(e) = settings.save_toml(&toml_path) {
|
||||
tracing::warn!("Failed to persist model to config.toml: {}", e);
|
||||
}
|
||||
}
|
||||
Ok(None) => {
|
||||
// No config file on disk; nothing to update.
|
||||
}
|
||||
Err(e) => {
|
||||
tracing::warn!("Failed to load config.toml for model persistence: {}", e);
|
||||
}
|
||||
}
|
||||
})
|
||||
.await
|
||||
{
|
||||
tracing::warn!("Model TOML persistence task failed: {}", e);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
+112
-37
@@ -13,7 +13,6 @@ use crate::agent::context_monitor::{CompactionStrategy, ContextBreakdown};
|
||||
use crate::agent::session::Thread;
|
||||
use crate::error::Error;
|
||||
use crate::llm::{ChatMessage, CompletionRequest, LlmProvider, Reasoning};
|
||||
use crate::safety::SafetyLayer;
|
||||
use crate::workspace::Workspace;
|
||||
|
||||
/// Result of a compaction operation.
|
||||
@@ -34,13 +33,12 @@ pub struct CompactionResult {
|
||||
/// Compacts conversation context to stay within limits.
|
||||
pub struct ContextCompactor {
|
||||
llm: Arc<dyn LlmProvider>,
|
||||
safety: Arc<SafetyLayer>,
|
||||
}
|
||||
|
||||
impl ContextCompactor {
|
||||
/// Create a new context compactor.
|
||||
pub fn new(llm: Arc<dyn LlmProvider>, safety: Arc<SafetyLayer>) -> Self {
|
||||
Self { llm, safety }
|
||||
pub fn new(llm: Arc<dyn LlmProvider>) -> Self {
|
||||
Self { llm }
|
||||
}
|
||||
|
||||
/// Compact a thread's context using the given strategy.
|
||||
@@ -105,27 +103,26 @@ impl ContextCompactor {
|
||||
// Generate summary
|
||||
let summary = self.generate_summary(&to_summarize).await?;
|
||||
|
||||
// Write to workspace if available
|
||||
let summary_written = if let Some(ws) = workspace {
|
||||
// Write to workspace if available.
|
||||
// If archival fails, preserve turns to avoid context loss.
|
||||
let (summary_written, turns_removed) = if let Some(ws) = workspace {
|
||||
match self.write_summary_to_workspace(ws, &summary).await {
|
||||
Ok(()) => true,
|
||||
Ok(()) => {
|
||||
thread.truncate_turns(keep_recent);
|
||||
(true, turns_to_remove)
|
||||
}
|
||||
Err(e) => {
|
||||
tracing::warn!(
|
||||
"Compaction summary write failed (turns will still be truncated): {}",
|
||||
e
|
||||
);
|
||||
false
|
||||
tracing::warn!("Compaction summary write failed (turns preserved): {}", e);
|
||||
(false, 0)
|
||||
}
|
||||
}
|
||||
} else {
|
||||
false
|
||||
thread.truncate_turns(keep_recent);
|
||||
(false, turns_to_remove)
|
||||
};
|
||||
|
||||
// Truncate thread
|
||||
thread.truncate_turns(keep_recent);
|
||||
|
||||
Ok(CompactionPartial {
|
||||
turns_removed: turns_to_remove,
|
||||
turns_removed,
|
||||
summary_written,
|
||||
summary: Some(summary),
|
||||
})
|
||||
@@ -167,23 +164,20 @@ impl ContextCompactor {
|
||||
// Format turns for storage
|
||||
let content = format_turns_for_storage(old_turns);
|
||||
|
||||
// Write to workspace
|
||||
let written = match self.write_context_to_workspace(ws, &content).await {
|
||||
Ok(()) => true,
|
||||
// Write to workspace. If archival fails, preserve turns.
|
||||
let (written, turns_removed) = match self.write_context_to_workspace(ws, &content).await {
|
||||
Ok(()) => {
|
||||
thread.truncate_turns(keep_recent);
|
||||
(true, turns_to_remove)
|
||||
}
|
||||
Err(e) => {
|
||||
tracing::warn!(
|
||||
"Compaction context write failed (turns will still be truncated): {}",
|
||||
e
|
||||
);
|
||||
false
|
||||
tracing::warn!("Compaction context write failed (turns preserved): {}", e);
|
||||
(false, 0)
|
||||
}
|
||||
};
|
||||
|
||||
// Truncate
|
||||
thread.truncate_turns(keep_recent);
|
||||
|
||||
Ok(CompactionPartial {
|
||||
turns_removed: turns_to_remove,
|
||||
turns_removed,
|
||||
summary_written: written,
|
||||
summary: None,
|
||||
})
|
||||
@@ -233,7 +227,7 @@ Be brief but capture all important details. Use bullet points."#,
|
||||
.with_max_tokens(1024)
|
||||
.with_temperature(0.3);
|
||||
|
||||
let reasoning = Reasoning::new(self.llm.clone(), self.safety.clone());
|
||||
let reasoning = Reasoning::new(self.llm.clone());
|
||||
let (text, _) = reasoning.complete(request).await?;
|
||||
Ok(text)
|
||||
}
|
||||
@@ -346,17 +340,11 @@ mod tests {
|
||||
// === QA Plan - Compaction strategy tests ===
|
||||
|
||||
use crate::agent::context_monitor::CompactionStrategy;
|
||||
use crate::config::SafetyConfig;
|
||||
use crate::safety::SafetyLayer;
|
||||
use crate::testing::StubLlm;
|
||||
|
||||
/// Helper: build a `ContextCompactor` with the given `StubLlm`.
|
||||
fn make_compactor(llm: Arc<StubLlm>) -> ContextCompactor {
|
||||
let safety = Arc::new(SafetyLayer::new(&SafetyConfig {
|
||||
max_output_length: 100_000,
|
||||
injection_check_enabled: false,
|
||||
}));
|
||||
ContextCompactor::new(llm, safety)
|
||||
ContextCompactor::new(llm)
|
||||
}
|
||||
|
||||
/// Helper: build a thread with `n` completed turns.
|
||||
@@ -370,6 +358,19 @@ mod tests {
|
||||
thread
|
||||
}
|
||||
|
||||
#[cfg(feature = "libsql")]
|
||||
async fn make_unmigrated_workspace() -> crate::workspace::Workspace {
|
||||
use crate::db::Database;
|
||||
use crate::db::libsql::LibSqlBackend;
|
||||
|
||||
// Intentionally skip migrations so workspace append operations fail.
|
||||
let backend = LibSqlBackend::new_memory()
|
||||
.await
|
||||
.expect("should create in-memory libsql backend");
|
||||
let db: Arc<dyn Database> = Arc::new(backend);
|
||||
crate::workspace::Workspace::new_with_db("compaction-test", db)
|
||||
}
|
||||
|
||||
// ------------------------------------------------------------------
|
||||
// 1. compact_truncate keeps last N turns
|
||||
// ------------------------------------------------------------------
|
||||
@@ -568,6 +569,43 @@ mod tests {
|
||||
assert_eq!(llm.calls(), 0);
|
||||
}
|
||||
|
||||
#[cfg(feature = "libsql")]
|
||||
#[tokio::test]
|
||||
async fn test_compact_with_summary_preserves_turns_when_workspace_write_fails() {
|
||||
let llm = Arc::new(StubLlm::new("summary"));
|
||||
let compactor = make_compactor(llm.clone());
|
||||
let mut thread = make_thread(8);
|
||||
let original_inputs: Vec<String> =
|
||||
thread.turns.iter().map(|t| t.user_input.clone()).collect();
|
||||
let workspace = make_unmigrated_workspace().await;
|
||||
|
||||
let result = compactor
|
||||
.compact(
|
||||
&mut thread,
|
||||
CompactionStrategy::Summarize { keep_recent: 3 },
|
||||
Some(&workspace),
|
||||
)
|
||||
.await
|
||||
.expect("compact should succeed even when workspace write fails");
|
||||
|
||||
// On archival failure, no turns should be removed.
|
||||
assert_eq!(thread.turns.len(), 8);
|
||||
assert_eq!(
|
||||
thread
|
||||
.turns
|
||||
.iter()
|
||||
.map(|t| t.user_input.as_str())
|
||||
.collect::<Vec<_>>(),
|
||||
original_inputs
|
||||
.iter()
|
||||
.map(|s| s.as_str())
|
||||
.collect::<Vec<_>>()
|
||||
);
|
||||
assert_eq!(result.turns_removed, 0);
|
||||
assert!(!result.summary_written);
|
||||
assert_eq!(llm.calls(), 1);
|
||||
}
|
||||
|
||||
// ------------------------------------------------------------------
|
||||
// 7. compact_to_workspace without workspace falls back to truncation
|
||||
// ------------------------------------------------------------------
|
||||
@@ -616,6 +654,43 @@ mod tests {
|
||||
assert_eq!(result.turns_removed, 0);
|
||||
}
|
||||
|
||||
#[cfg(feature = "libsql")]
|
||||
#[tokio::test]
|
||||
async fn test_compact_to_workspace_preserves_turns_when_workspace_write_fails() {
|
||||
let llm = Arc::new(StubLlm::new("unused"));
|
||||
let compactor = make_compactor(llm.clone());
|
||||
let mut thread = make_thread(20);
|
||||
let original_inputs: Vec<String> =
|
||||
thread.turns.iter().map(|t| t.user_input.clone()).collect();
|
||||
let workspace = make_unmigrated_workspace().await;
|
||||
|
||||
let result = compactor
|
||||
.compact(
|
||||
&mut thread,
|
||||
CompactionStrategy::MoveToWorkspace,
|
||||
Some(&workspace),
|
||||
)
|
||||
.await
|
||||
.expect("compact should succeed even when workspace write fails");
|
||||
|
||||
// On archival failure, no turns should be removed.
|
||||
assert_eq!(thread.turns.len(), 20);
|
||||
assert_eq!(
|
||||
thread
|
||||
.turns
|
||||
.iter()
|
||||
.map(|t| t.user_input.as_str())
|
||||
.collect::<Vec<_>>(),
|
||||
original_inputs
|
||||
.iter()
|
||||
.map(|s| s.as_str())
|
||||
.collect::<Vec<_>>()
|
||||
);
|
||||
assert_eq!(result.turns_removed, 0);
|
||||
assert!(!result.summary_written);
|
||||
assert_eq!(llm.calls(), 0);
|
||||
}
|
||||
|
||||
// ------------------------------------------------------------------
|
||||
// 9. format_turns_for_storage includes tool calls
|
||||
// ------------------------------------------------------------------
|
||||
|
||||
+42
-7
@@ -131,10 +131,12 @@ impl CostGuard {
|
||||
// Check hourly rate
|
||||
if let Some(limit) = self.config.max_actions_per_hour {
|
||||
let mut window = self.action_window.lock().await;
|
||||
let cutoff = Instant::now() - std::time::Duration::from_secs(3600);
|
||||
// Drain expired entries
|
||||
while window.front().is_some_and(|t| *t < cutoff) {
|
||||
window.pop_front();
|
||||
// checked_sub avoids panic when system uptime < 1 hour (Windows)
|
||||
if let Some(cutoff) = Instant::now().checked_sub(std::time::Duration::from_secs(3600)) {
|
||||
// Drain expired entries
|
||||
while window.front().is_some_and(|t| *t < cutoff) {
|
||||
window.pop_front();
|
||||
}
|
||||
}
|
||||
let count = window.len() as u64;
|
||||
if count >= limit {
|
||||
@@ -260,9 +262,11 @@ impl CostGuard {
|
||||
/// Number of actions in the current hourly window.
|
||||
pub async fn actions_this_hour(&self) -> u64 {
|
||||
let mut window = self.action_window.lock().await;
|
||||
let cutoff = Instant::now() - std::time::Duration::from_secs(3600);
|
||||
while window.front().is_some_and(|t| *t < cutoff) {
|
||||
window.pop_front();
|
||||
// checked_sub avoids panic when system uptime < 1 hour (Windows)
|
||||
if let Some(cutoff) = Instant::now().checked_sub(std::time::Duration::from_secs(3600)) {
|
||||
while window.front().is_some_and(|t| *t < cutoff) {
|
||||
window.pop_front();
|
||||
}
|
||||
}
|
||||
window.len() as u64
|
||||
}
|
||||
@@ -621,4 +625,35 @@ mod tests {
|
||||
"surcharge should be 100% of input cost for 1h cache writes"
|
||||
);
|
||||
}
|
||||
|
||||
/// Regression test for #657: Instant::now() - Duration panics on Windows
|
||||
/// when system uptime is less than the subtracted duration.
|
||||
#[tokio::test]
|
||||
async fn test_checked_sub_no_panic_on_fresh_guard() {
|
||||
// A fresh CostGuard with rate limits should not panic even if
|
||||
// checked_sub returns None (simulating short uptime).
|
||||
let guard = CostGuard::new(CostGuardConfig {
|
||||
max_cost_per_day_cents: None,
|
||||
max_actions_per_hour: Some(100),
|
||||
});
|
||||
|
||||
// These must not panic regardless of system uptime
|
||||
assert!(guard.check_allowed().await.is_ok());
|
||||
assert_eq!(guard.actions_this_hour().await, 0);
|
||||
|
||||
// Record some actions and verify again
|
||||
guard
|
||||
.record_llm_call("gpt-4o", 10, 10, 0, 0, Decimal::ONE, Decimal::ONE, None)
|
||||
.await;
|
||||
assert!(guard.check_allowed().await.is_ok());
|
||||
assert_eq!(guard.actions_this_hour().await, 1);
|
||||
}
|
||||
|
||||
/// Verify that checked_sub itself behaves as expected for the pattern we use.
|
||||
#[test]
|
||||
fn test_instant_checked_sub_returns_none_for_overflow() {
|
||||
// Duration::MAX will always exceed uptime, so checked_sub must return None
|
||||
let result = Instant::now().checked_sub(std::time::Duration::MAX);
|
||||
assert!(result.is_none());
|
||||
}
|
||||
}
|
||||
|
||||
+221
-31
@@ -50,8 +50,18 @@ impl Agent {
|
||||
|
||||
// Load workspace system prompt (identity files: AGENTS.md, SOUL.md, etc.)
|
||||
// In group chats, MEMORY.md is excluded to prevent leaking personal context.
|
||||
// Resolve the user's timezone
|
||||
let user_tz = crate::timezone::resolve_timezone(
|
||||
message.timezone.as_deref(),
|
||||
None, // user setting lookup can be added later
|
||||
&self.config.default_timezone,
|
||||
);
|
||||
|
||||
let system_prompt = if let Some(ws) = self.workspace() {
|
||||
match ws.system_prompt_for_context(is_group_chat).await {
|
||||
match ws
|
||||
.system_prompt_for_context_tz(is_group_chat, user_tz)
|
||||
.await
|
||||
{
|
||||
Ok(prompt) if !prompt.is_empty() => Some(prompt),
|
||||
Ok(_) => None,
|
||||
Err(e) => {
|
||||
@@ -103,7 +113,7 @@ impl Agent {
|
||||
None
|
||||
};
|
||||
|
||||
let mut reasoning = Reasoning::new(self.llm().clone(), self.safety().clone())
|
||||
let mut reasoning = Reasoning::new(self.llm().clone())
|
||||
.with_channel(message.channel.clone())
|
||||
.with_model_name(self.llm().active_model_name())
|
||||
.with_group_chat(is_group_chat);
|
||||
@@ -130,6 +140,7 @@ impl Agent {
|
||||
let mut job_ctx =
|
||||
JobContext::with_user(&message.user_id, "chat", "Interactive chat session");
|
||||
job_ctx.http_interceptor = self.deps.http_interceptor.clone();
|
||||
job_ctx.user_timezone = user_tz.name().to_string();
|
||||
|
||||
// Build system prompts once for this turn. Two variants: with tools
|
||||
// (normal iterations) and without (force_text final iteration).
|
||||
@@ -670,8 +681,53 @@ impl Agent {
|
||||
.into())
|
||||
});
|
||||
|
||||
// Send ToolResult preview
|
||||
if let Ok(ref output) = tool_result
|
||||
// Detect image generation sentinel in tool output
|
||||
// (only from image tools — avoids parsing all tool outputs)
|
||||
let is_image_sentinel = if let Ok(ref output) = tool_result
|
||||
&& matches!(tc.name.as_str(), "image_generate" | "image_edit")
|
||||
{
|
||||
if let Ok(sentinel) =
|
||||
serde_json::from_str::<serde_json::Value>(output)
|
||||
&& sentinel.get("type").and_then(|v| v.as_str())
|
||||
== Some("image_generated")
|
||||
{
|
||||
let data_url = sentinel
|
||||
.get("data")
|
||||
.and_then(|v| v.as_str())
|
||||
.unwrap_or_default()
|
||||
.to_string();
|
||||
let path = sentinel
|
||||
.get("path")
|
||||
.and_then(|v| v.as_str())
|
||||
.map(String::from);
|
||||
// Skip broadcasting if data_url is empty to avoid
|
||||
// sending a broken ImageGenerated SSE event.
|
||||
if data_url.is_empty() {
|
||||
tracing::warn!(
|
||||
"Image generation sentinel has empty data URL, skipping broadcast"
|
||||
);
|
||||
} else {
|
||||
let _ = self
|
||||
.channels
|
||||
.send_status(
|
||||
&message.channel,
|
||||
StatusUpdate::ImageGenerated { data_url, path },
|
||||
&message.metadata,
|
||||
)
|
||||
.await;
|
||||
}
|
||||
true
|
||||
} else {
|
||||
false
|
||||
}
|
||||
} else {
|
||||
false
|
||||
};
|
||||
|
||||
// Send ToolResult preview (skip for image sentinels to avoid
|
||||
// broadcasting multi-MB base64 data as a preview)
|
||||
if !is_image_sentinel
|
||||
&& let Ok(ref output) = tool_result
|
||||
&& !output.is_empty()
|
||||
{
|
||||
let _ = self
|
||||
@@ -687,23 +743,6 @@ impl Agent {
|
||||
.await;
|
||||
}
|
||||
|
||||
// Record result in thread
|
||||
{
|
||||
let mut sess = session.lock().await;
|
||||
if let Some(thread) = sess.threads.get_mut(&thread_id)
|
||||
&& let Some(turn) = thread.last_turn_mut()
|
||||
{
|
||||
match &tool_result {
|
||||
Ok(output) => {
|
||||
turn.record_tool_result(serde_json::json!(output));
|
||||
}
|
||||
Err(e) => {
|
||||
turn.record_tool_error(e.to_string());
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Check for auth awaiting — defer the return
|
||||
// until all results are recorded.
|
||||
if deferred_auth.is_none()
|
||||
@@ -743,6 +782,7 @@ impl Agent {
|
||||
}
|
||||
|
||||
// Sanitize and add tool result to context
|
||||
let is_tool_error = tool_result.is_err();
|
||||
let result_content = match tool_result {
|
||||
Ok(output) => {
|
||||
let sanitized =
|
||||
@@ -756,6 +796,23 @@ impl Agent {
|
||||
Err(e) => format!("Tool '{}' failed: {}", tc.name, e),
|
||||
};
|
||||
|
||||
// Record sanitized result in thread so messages()
|
||||
// and persist_tool_calls() use cleaned content.
|
||||
{
|
||||
let mut sess = session.lock().await;
|
||||
if let Some(thread) = sess.threads.get_mut(&thread_id)
|
||||
&& let Some(turn) = thread.last_turn_mut()
|
||||
{
|
||||
if is_tool_error {
|
||||
turn.record_tool_error(result_content.clone());
|
||||
} else {
|
||||
turn.record_tool_result(serde_json::json!(
|
||||
result_content
|
||||
));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
context_messages.push(ChatMessage::tool_result(
|
||||
&tc.id,
|
||||
&tc.name,
|
||||
@@ -785,6 +842,7 @@ impl Agent {
|
||||
tool_call_id: tc.id.clone(),
|
||||
context_messages: context_messages.clone(),
|
||||
deferred_tool_calls: tool_calls[approval_idx + 1..].to_vec(),
|
||||
user_timezone: Some(user_tz.name().to_string()),
|
||||
};
|
||||
|
||||
return Ok(AgenticLoopResult::NeedApproval { pending });
|
||||
@@ -1146,6 +1204,8 @@ mod tests {
|
||||
max_actions_per_hour: None,
|
||||
max_tool_iterations: 50,
|
||||
auto_approve_tools: false,
|
||||
default_timezone: "UTC".to_string(),
|
||||
max_tokens_per_job: 0,
|
||||
},
|
||||
deps,
|
||||
Arc::new(ChannelManager::new()),
|
||||
@@ -1204,6 +1264,96 @@ mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_always_approval_requirement_bypasses_session_auto_approve() {
|
||||
// Regression test: even if tool is auto-approved in session,
|
||||
// ApprovalRequirement::Always must still trigger approval.
|
||||
use crate::tools::ApprovalRequirement;
|
||||
|
||||
let mut session = Session::new("user-1");
|
||||
let tool_name = "tool_remove";
|
||||
|
||||
// Manually auto-approve tool_remove in this session
|
||||
session.auto_approve_tool(tool_name);
|
||||
assert!(
|
||||
session.is_tool_auto_approved(tool_name),
|
||||
"tool should be auto-approved"
|
||||
);
|
||||
|
||||
// However, ApprovalRequirement::Always should always require approval
|
||||
// This is verified by the dispatcher logic: Always => true (ignores session state)
|
||||
let always_req = ApprovalRequirement::Always;
|
||||
let requires_approval = match always_req {
|
||||
ApprovalRequirement::Never => false,
|
||||
ApprovalRequirement::UnlessAutoApproved => !session.is_tool_auto_approved(tool_name),
|
||||
ApprovalRequirement::Always => true,
|
||||
};
|
||||
|
||||
assert!(
|
||||
requires_approval,
|
||||
"ApprovalRequirement::Always must require approval even when tool is auto-approved"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_always_approval_requirement_vs_unless_auto_approved() {
|
||||
// Verify the two requirements behave differently
|
||||
use crate::tools::ApprovalRequirement;
|
||||
|
||||
let mut session = Session::new("user-2");
|
||||
let tool_name = "http";
|
||||
|
||||
// Scenario 1: Tool is auto-approved
|
||||
session.auto_approve_tool(tool_name);
|
||||
|
||||
// UnlessAutoApproved → doesn't require approval if auto-approved
|
||||
let unless_req = ApprovalRequirement::UnlessAutoApproved;
|
||||
let unless_needs = match unless_req {
|
||||
ApprovalRequirement::Never => false,
|
||||
ApprovalRequirement::UnlessAutoApproved => !session.is_tool_auto_approved(tool_name),
|
||||
ApprovalRequirement::Always => true,
|
||||
};
|
||||
assert!(
|
||||
!unless_needs,
|
||||
"UnlessAutoApproved should not need approval when auto-approved"
|
||||
);
|
||||
|
||||
// Always → always requires approval
|
||||
let always_req = ApprovalRequirement::Always;
|
||||
let always_needs = match always_req {
|
||||
ApprovalRequirement::Never => false,
|
||||
ApprovalRequirement::UnlessAutoApproved => !session.is_tool_auto_approved(tool_name),
|
||||
ApprovalRequirement::Always => true,
|
||||
};
|
||||
assert!(
|
||||
always_needs,
|
||||
"Always must always require approval, even when auto-approved"
|
||||
);
|
||||
|
||||
// Scenario 2: Tool is NOT auto-approved
|
||||
let new_tool = "new_tool";
|
||||
assert!(!session.is_tool_auto_approved(new_tool));
|
||||
|
||||
// UnlessAutoApproved → requires approval
|
||||
let unless_needs = match unless_req {
|
||||
ApprovalRequirement::Never => false,
|
||||
ApprovalRequirement::UnlessAutoApproved => !session.is_tool_auto_approved(new_tool),
|
||||
ApprovalRequirement::Always => true,
|
||||
};
|
||||
assert!(
|
||||
unless_needs,
|
||||
"UnlessAutoApproved should need approval when not auto-approved"
|
||||
);
|
||||
|
||||
// Always → always requires approval
|
||||
let always_needs = match always_req {
|
||||
ApprovalRequirement::Never => false,
|
||||
ApprovalRequirement::UnlessAutoApproved => !session.is_tool_auto_approved(new_tool),
|
||||
ApprovalRequirement::Always => true,
|
||||
};
|
||||
assert!(always_needs, "Always must always require approval");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_pending_approval_serialization_backcompat_without_deferred_calls() {
|
||||
// PendingApproval from before the deferred_tool_calls field was added
|
||||
@@ -1248,6 +1398,7 @@ mod tests {
|
||||
arguments: serde_json::json!({"message": "done"}),
|
||||
},
|
||||
],
|
||||
user_timezone: None,
|
||||
};
|
||||
|
||||
let json = serde_json::to_string(&pending).expect("serialize");
|
||||
@@ -1595,12 +1746,8 @@ mod tests {
|
||||
use crate::testing::StubLlm;
|
||||
|
||||
let stub = Arc::new(StubLlm::failing_non_transient("ctx-bomb"));
|
||||
let safety = Arc::new(SafetyLayer::new(&SafetyConfig {
|
||||
max_output_length: 100_000,
|
||||
injection_check_enabled: false,
|
||||
}));
|
||||
|
||||
let reasoning = Reasoning::new(stub.clone(), safety);
|
||||
let reasoning = Reasoning::new(stub.clone());
|
||||
|
||||
// Build a fat context with lots of history.
|
||||
let messages = vec![
|
||||
@@ -1710,11 +1857,7 @@ mod tests {
|
||||
use crate::llm::{Reasoning, ReasoningContext, RespondResult, ToolDefinition};
|
||||
|
||||
let provider = Arc::new(AlwaysToolCallProvider);
|
||||
let safety = Arc::new(SafetyLayer::new(&SafetyConfig {
|
||||
max_output_length: 100_000,
|
||||
injection_check_enabled: false,
|
||||
}));
|
||||
let reasoning = Reasoning::new(provider, safety);
|
||||
let reasoning = Reasoning::new(provider);
|
||||
|
||||
let tool_def = ToolDefinition {
|
||||
name: "echo".to_string(),
|
||||
@@ -1900,6 +2043,8 @@ mod tests {
|
||||
max_actions_per_hour: None,
|
||||
max_tool_iterations,
|
||||
auto_approve_tools: true,
|
||||
default_timezone: "UTC".to_string(),
|
||||
max_tokens_per_job: 0,
|
||||
},
|
||||
deps,
|
||||
Arc::new(ChannelManager::new()),
|
||||
@@ -2015,6 +2160,8 @@ mod tests {
|
||||
max_actions_per_hour: None,
|
||||
max_tool_iterations: max_iter,
|
||||
auto_approve_tools: true,
|
||||
default_timezone: "UTC".to_string(),
|
||||
max_tokens_per_job: 0,
|
||||
},
|
||||
deps,
|
||||
Arc::new(ChannelManager::new()),
|
||||
@@ -2116,4 +2263,47 @@ mod tests {
|
||||
"Error should include the underlying reason, got: {formatted}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_image_sentinel_empty_data_url_should_be_skipped() {
|
||||
// Regression: unwrap_or_default() on missing "data" field produces an empty
|
||||
// string. Broadcasting an empty data_url would send a broken SSE event.
|
||||
let sentinel = serde_json::json!({
|
||||
"type": "image_generated",
|
||||
"path": "/tmp/image.png"
|
||||
// "data" field is missing
|
||||
});
|
||||
|
||||
let data_url = sentinel
|
||||
.get("data")
|
||||
.and_then(|v| v.as_str())
|
||||
.unwrap_or_default()
|
||||
.to_string();
|
||||
|
||||
assert!(
|
||||
data_url.is_empty(),
|
||||
"Missing 'data' field should produce empty string"
|
||||
);
|
||||
// The fix: empty data_url means we skip broadcasting
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_image_sentinel_present_data_url_is_valid() {
|
||||
let sentinel = serde_json::json!({
|
||||
"type": "image_generated",
|
||||
"data": "data:image/png;base64,abc123",
|
||||
"path": "/tmp/image.png"
|
||||
});
|
||||
|
||||
let data_url = sentinel
|
||||
.get("data")
|
||||
.and_then(|v| v.as_str())
|
||||
.unwrap_or_default()
|
||||
.to_string();
|
||||
|
||||
assert!(
|
||||
!data_url.is_empty(),
|
||||
"Present 'data' field should produce non-empty string"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
+168
-8
@@ -29,8 +29,8 @@ use std::time::Duration;
|
||||
use tokio::sync::mpsc;
|
||||
|
||||
use crate::channels::OutgoingResponse;
|
||||
use crate::db::Database;
|
||||
use crate::llm::{ChatMessage, CompletionRequest, LlmProvider, Reasoning};
|
||||
use crate::safety::SafetyLayer;
|
||||
use crate::workspace::Workspace;
|
||||
use crate::workspace::hygiene::HygieneConfig;
|
||||
|
||||
@@ -47,6 +47,12 @@ pub struct HeartbeatConfig {
|
||||
pub notify_user_id: Option<String>,
|
||||
/// Channel to notify on heartbeat findings.
|
||||
pub notify_channel: Option<String>,
|
||||
/// Hour (0-23) when quiet hours start.
|
||||
pub quiet_hours_start: Option<u32>,
|
||||
/// Hour (0-23) when quiet hours end.
|
||||
pub quiet_hours_end: Option<u32>,
|
||||
/// Timezone for quiet hours evaluation (IANA name).
|
||||
pub timezone: Option<String>,
|
||||
}
|
||||
|
||||
impl Default for HeartbeatConfig {
|
||||
@@ -57,6 +63,9 @@ impl Default for HeartbeatConfig {
|
||||
max_failures: 3,
|
||||
notify_user_id: None,
|
||||
notify_channel: None,
|
||||
quiet_hours_start: None,
|
||||
quiet_hours_end: None,
|
||||
timezone: None,
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -74,6 +83,26 @@ impl HeartbeatConfig {
|
||||
self
|
||||
}
|
||||
|
||||
/// Check whether the current time falls within configured quiet hours.
|
||||
pub fn is_quiet_hours(&self) -> bool {
|
||||
use chrono::Timelike;
|
||||
let (Some(start), Some(end)) = (self.quiet_hours_start, self.quiet_hours_end) else {
|
||||
return false;
|
||||
};
|
||||
let tz = self
|
||||
.timezone
|
||||
.as_deref()
|
||||
.and_then(crate::timezone::parse_timezone)
|
||||
.unwrap_or(chrono_tz::UTC);
|
||||
let now_hour = crate::timezone::now_in_tz(tz).hour();
|
||||
if start <= end {
|
||||
now_hour >= start && now_hour < end
|
||||
} else {
|
||||
// Wraps midnight, e.g. 22..06
|
||||
now_hour >= start || now_hour < end
|
||||
}
|
||||
}
|
||||
|
||||
/// Set the notification target.
|
||||
pub fn with_notify(mut self, user_id: impl Into<String>, channel: impl Into<String>) -> Self {
|
||||
self.notify_user_id = Some(user_id.into());
|
||||
@@ -101,8 +130,8 @@ pub struct HeartbeatRunner {
|
||||
hygiene_config: HygieneConfig,
|
||||
workspace: Arc<Workspace>,
|
||||
llm: Arc<dyn LlmProvider>,
|
||||
safety: Arc<SafetyLayer>,
|
||||
response_tx: Option<mpsc::Sender<OutgoingResponse>>,
|
||||
store: Option<Arc<dyn Database>>,
|
||||
consecutive_failures: u32,
|
||||
}
|
||||
|
||||
@@ -113,15 +142,14 @@ impl HeartbeatRunner {
|
||||
hygiene_config: HygieneConfig,
|
||||
workspace: Arc<Workspace>,
|
||||
llm: Arc<dyn LlmProvider>,
|
||||
safety: Arc<SafetyLayer>,
|
||||
) -> Self {
|
||||
Self {
|
||||
config,
|
||||
hygiene_config,
|
||||
workspace,
|
||||
llm,
|
||||
safety,
|
||||
response_tx: None,
|
||||
store: None,
|
||||
consecutive_failures: 0,
|
||||
}
|
||||
}
|
||||
@@ -132,6 +160,12 @@ impl HeartbeatRunner {
|
||||
self
|
||||
}
|
||||
|
||||
/// Set the database store for persistent heartbeat conversations.
|
||||
pub fn with_store(mut self, store: Arc<dyn Database>) -> Self {
|
||||
self.store = Some(store);
|
||||
self
|
||||
}
|
||||
|
||||
/// Run the heartbeat loop.
|
||||
///
|
||||
/// This runs forever, checking periodically based on the configured interval.
|
||||
@@ -153,6 +187,12 @@ impl HeartbeatRunner {
|
||||
loop {
|
||||
interval.tick().await;
|
||||
|
||||
// Skip during quiet hours
|
||||
if self.config.is_quiet_hours() {
|
||||
tracing::debug!("Heartbeat skipped: quiet hours");
|
||||
continue;
|
||||
}
|
||||
|
||||
// Run memory hygiene in the background so it never delays the
|
||||
// heartbeat checklist. Failures are logged inside run_if_due.
|
||||
let hygiene_workspace = Arc::clone(&self.workspace);
|
||||
@@ -263,7 +303,7 @@ impl HeartbeatRunner {
|
||||
.with_max_tokens(max_tokens)
|
||||
.with_temperature(0.3);
|
||||
|
||||
let reasoning = Reasoning::new(self.llm.clone(), self.safety.clone());
|
||||
let reasoning = Reasoning::new(self.llm.clone());
|
||||
let (content, _usage) = match reasoning.complete(request).await {
|
||||
Ok(r) => r,
|
||||
Err(e) => return HeartbeatResult::Failed(format!("LLM call failed: {}", e)),
|
||||
@@ -292,9 +332,32 @@ impl HeartbeatRunner {
|
||||
return;
|
||||
};
|
||||
|
||||
let user_id = self.config.notify_user_id.as_deref().unwrap_or("default");
|
||||
|
||||
// Persist to heartbeat conversation and get thread_id
|
||||
let thread_id = if let Some(ref store) = self.store {
|
||||
match store.get_or_create_heartbeat_conversation(user_id).await {
|
||||
Ok(conv_id) => {
|
||||
if let Err(e) = store
|
||||
.add_conversation_message(conv_id, "assistant", message)
|
||||
.await
|
||||
{
|
||||
tracing::error!("Failed to persist heartbeat message: {}", e);
|
||||
}
|
||||
Some(conv_id.to_string())
|
||||
}
|
||||
Err(e) => {
|
||||
tracing::error!("Failed to get heartbeat conversation: {}", e);
|
||||
None
|
||||
}
|
||||
}
|
||||
} else {
|
||||
None
|
||||
};
|
||||
|
||||
let response = OutgoingResponse {
|
||||
content: format!("🔔 *Heartbeat Alert*\n\n{}", message),
|
||||
thread_id: None,
|
||||
thread_id,
|
||||
attachments: Vec::new(),
|
||||
metadata: serde_json::json!({
|
||||
"source": "heartbeat",
|
||||
@@ -354,13 +417,16 @@ pub fn spawn_heartbeat(
|
||||
hygiene_config: HygieneConfig,
|
||||
workspace: Arc<Workspace>,
|
||||
llm: Arc<dyn LlmProvider>,
|
||||
safety: Arc<SafetyLayer>,
|
||||
response_tx: Option<mpsc::Sender<OutgoingResponse>>,
|
||||
store: Option<Arc<dyn Database>>,
|
||||
) -> tokio::task::JoinHandle<()> {
|
||||
let mut runner = HeartbeatRunner::new(config, hygiene_config, workspace, llm, safety);
|
||||
let mut runner = HeartbeatRunner::new(config, hygiene_config, workspace, llm);
|
||||
if let Some(tx) = response_tx {
|
||||
runner = runner.with_response_channel(tx);
|
||||
}
|
||||
if let Some(s) = store {
|
||||
runner = runner.with_store(s);
|
||||
}
|
||||
|
||||
tokio::spawn(async move {
|
||||
runner.run().await;
|
||||
@@ -495,4 +561,98 @@ mod tests {
|
||||
let content = "<!-- comment -->\nActual task here";
|
||||
assert!(!is_effectively_empty(content));
|
||||
}
|
||||
|
||||
// ==================== quiet hours ====================
|
||||
|
||||
#[test]
|
||||
fn test_quiet_hours_inside() {
|
||||
use chrono::{Timelike, Utc};
|
||||
|
||||
let now_utc = Utc::now();
|
||||
let hour = now_utc.hour();
|
||||
let start = hour;
|
||||
let end = (hour + 1) % 24;
|
||||
|
||||
let config = HeartbeatConfig {
|
||||
quiet_hours_start: Some(start),
|
||||
quiet_hours_end: Some(end),
|
||||
timezone: Some("UTC".to_string()),
|
||||
..HeartbeatConfig::default()
|
||||
};
|
||||
// Current UTC hour is inside [start, end) by construction
|
||||
assert!(config.is_quiet_hours());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_quiet_hours_outside() {
|
||||
use chrono::{Timelike, Utc};
|
||||
|
||||
let now_utc = Utc::now();
|
||||
let hour = now_utc.hour();
|
||||
let start = (hour + 1) % 24;
|
||||
let end = (hour + 2) % 24;
|
||||
|
||||
let config = HeartbeatConfig {
|
||||
quiet_hours_start: Some(start),
|
||||
quiet_hours_end: Some(end),
|
||||
timezone: Some("UTC".to_string()),
|
||||
..HeartbeatConfig::default()
|
||||
};
|
||||
// Current UTC hour is outside [start, end) by construction
|
||||
assert!(!config.is_quiet_hours());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_quiet_hours_wraparound_excludes_now() {
|
||||
use chrono::{Timelike, Utc};
|
||||
|
||||
let now_utc = Utc::now();
|
||||
let hour = now_utc.hour();
|
||||
// Window covers all hours except the current one
|
||||
let start = (hour + 1) % 24;
|
||||
let end = hour;
|
||||
|
||||
let config = HeartbeatConfig {
|
||||
quiet_hours_start: Some(start),
|
||||
quiet_hours_end: Some(end),
|
||||
timezone: Some("UTC".to_string()),
|
||||
..HeartbeatConfig::default()
|
||||
};
|
||||
assert!(!config.is_quiet_hours());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_quiet_hours_none_configured() {
|
||||
let config = HeartbeatConfig::default();
|
||||
assert!(!config.is_quiet_hours());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_quiet_hours_same_start_end() {
|
||||
let config = HeartbeatConfig {
|
||||
quiet_hours_start: Some(10),
|
||||
quiet_hours_end: Some(10),
|
||||
timezone: Some("UTC".to_string()),
|
||||
..HeartbeatConfig::default()
|
||||
};
|
||||
// start == end means zero-width window, should be false
|
||||
assert!(!config.is_quiet_hours());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_spawn_heartbeat_accepts_store_param() {
|
||||
// Regression: spawn_heartbeat must accept an optional Database store
|
||||
// for persisting heartbeat notifications to a dedicated conversation.
|
||||
// Compile-time check: the 7th parameter is `Option<Arc<dyn Database>>`.
|
||||
#[allow(clippy::type_complexity)]
|
||||
let _fn_ptr: fn(
|
||||
HeartbeatConfig,
|
||||
HygieneConfig,
|
||||
Arc<crate::workspace::Workspace>,
|
||||
Arc<dyn crate::llm::LlmProvider>,
|
||||
Option<tokio::sync::mpsc::Sender<crate::channels::OutgoingResponse>>,
|
||||
Option<Arc<dyn crate::db::Database>>,
|
||||
) -> tokio::task::JoinHandle<()> = spawn_heartbeat;
|
||||
let _ = _fn_ptr;
|
||||
}
|
||||
}
|
||||
|
||||
+87
-9
@@ -57,7 +57,11 @@ pub struct Routine {
|
||||
#[serde(tag = "type", rename_all = "snake_case")]
|
||||
pub enum Trigger {
|
||||
/// Fire on a cron schedule (e.g. "0 9 * * MON-FRI" or "every 2h").
|
||||
Cron { schedule: String },
|
||||
Cron {
|
||||
schedule: String,
|
||||
#[serde(default)]
|
||||
timezone: Option<String>,
|
||||
},
|
||||
/// Fire when a channel message matches a pattern.
|
||||
Event {
|
||||
/// Optional channel filter (e.g. "telegram", "slack").
|
||||
@@ -99,7 +103,21 @@ impl Trigger {
|
||||
field: "schedule".into(),
|
||||
})?
|
||||
.to_string();
|
||||
Ok(Trigger::Cron { schedule })
|
||||
let timezone = config
|
||||
.get("timezone")
|
||||
.and_then(|v| v.as_str())
|
||||
.and_then(|tz| {
|
||||
if crate::timezone::parse_timezone(tz).is_some() {
|
||||
Some(tz.to_string())
|
||||
} else {
|
||||
tracing::warn!(
|
||||
"Ignoring invalid timezone '{}' from DB for cron trigger",
|
||||
tz
|
||||
);
|
||||
None
|
||||
}
|
||||
});
|
||||
Ok(Trigger::Cron { schedule, timezone })
|
||||
}
|
||||
"event" => {
|
||||
let pattern = config
|
||||
@@ -137,7 +155,10 @@ impl Trigger {
|
||||
/// Serialize trigger-specific config to JSON for DB storage.
|
||||
pub fn to_config_json(&self) -> serde_json::Value {
|
||||
match self {
|
||||
Trigger::Cron { schedule } => serde_json::json!({ "schedule": schedule }),
|
||||
Trigger::Cron { schedule, timezone } => serde_json::json!({
|
||||
"schedule": schedule,
|
||||
"timezone": timezone,
|
||||
}),
|
||||
Trigger::Event { channel, pattern } => serde_json::json!({
|
||||
"pattern": pattern,
|
||||
"channel": channel,
|
||||
@@ -415,12 +436,25 @@ pub fn content_hash(content: &str) -> u64 {
|
||||
}
|
||||
|
||||
/// Parse a cron expression and compute the next fire time from now.
|
||||
pub fn next_cron_fire(schedule: &str) -> Result<Option<DateTime<Utc>>, RoutineError> {
|
||||
///
|
||||
/// When `timezone` is provided and valid, the schedule is evaluated in that
|
||||
/// timezone and the result is converted back to UTC. Otherwise UTC is used.
|
||||
pub fn next_cron_fire(
|
||||
schedule: &str,
|
||||
timezone: Option<&str>,
|
||||
) -> Result<Option<DateTime<Utc>>, RoutineError> {
|
||||
let cron_schedule =
|
||||
cron::Schedule::from_str(schedule).map_err(|e| RoutineError::InvalidCron {
|
||||
reason: e.to_string(),
|
||||
})?;
|
||||
Ok(cron_schedule.upcoming(Utc).next())
|
||||
if let Some(tz) = timezone.and_then(crate::timezone::parse_timezone) {
|
||||
Ok(cron_schedule
|
||||
.upcoming(tz)
|
||||
.next()
|
||||
.map(|dt| dt.with_timezone(&Utc)))
|
||||
} else {
|
||||
Ok(cron_schedule.upcoming(Utc).next())
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
@@ -433,10 +467,11 @@ mod tests {
|
||||
fn test_trigger_roundtrip() {
|
||||
let trigger = Trigger::Cron {
|
||||
schedule: "0 9 * * MON-FRI".to_string(),
|
||||
timezone: None,
|
||||
};
|
||||
let json = trigger.to_config_json();
|
||||
let parsed = Trigger::from_db("cron", json).expect("parse cron");
|
||||
assert!(matches!(parsed, Trigger::Cron { schedule } if schedule == "0 9 * * MON-FRI"));
|
||||
assert!(matches!(parsed, Trigger::Cron { schedule, .. } if schedule == "0 9 * * MON-FRI"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
@@ -509,16 +544,58 @@ mod tests {
|
||||
#[test]
|
||||
fn test_next_cron_fire_valid() {
|
||||
// Every minute should always have a next fire
|
||||
let next = next_cron_fire("* * * * * *").expect("valid cron");
|
||||
let next = next_cron_fire("* * * * * *", None).expect("valid cron");
|
||||
assert!(next.is_some());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_next_cron_fire_invalid() {
|
||||
let result = next_cron_fire("not a cron");
|
||||
let result = next_cron_fire("not a cron", None);
|
||||
assert!(result.is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_trigger_cron_timezone_roundtrip() {
|
||||
let trigger = Trigger::Cron {
|
||||
schedule: "0 9 * * MON-FRI".to_string(),
|
||||
timezone: Some("America/New_York".to_string()),
|
||||
};
|
||||
let json = trigger.to_config_json();
|
||||
let parsed = Trigger::from_db("cron", json).expect("parse cron");
|
||||
assert!(matches!(parsed, Trigger::Cron { schedule, timezone }
|
||||
if schedule == "0 9 * * MON-FRI"
|
||||
&& timezone.as_deref() == Some("America/New_York")));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_trigger_cron_no_timezone_backward_compat() {
|
||||
let json = serde_json::json!({"schedule": "0 9 * * *"});
|
||||
let parsed = Trigger::from_db("cron", json).expect("parse cron");
|
||||
assert!(matches!(parsed, Trigger::Cron { timezone, .. } if timezone.is_none()));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_trigger_cron_invalid_timezone_coerced_to_none() {
|
||||
let json = serde_json::json!({"schedule": "0 9 * * *", "timezone": "Fake/Zone"});
|
||||
let parsed = Trigger::from_db("cron", json).expect("parse cron");
|
||||
assert!(
|
||||
matches!(parsed, Trigger::Cron { timezone, .. } if timezone.is_none()),
|
||||
"invalid timezone should be coerced to None"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_next_cron_fire_with_timezone() {
|
||||
let next_utc = next_cron_fire("0 0 9 * * * *", None)
|
||||
.expect("valid cron")
|
||||
.expect("has next");
|
||||
let next_est = next_cron_fire("0 0 9 * * * *", Some("America/New_York"))
|
||||
.expect("valid cron")
|
||||
.expect("has next");
|
||||
// EST is UTC-5 (or EDT UTC-4), so the UTC result should differ
|
||||
assert_ne!(next_utc, next_est, "timezone should shift the fire time");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_guardrails_default() {
|
||||
let g = RoutineGuardrails::default();
|
||||
@@ -531,7 +608,8 @@ mod tests {
|
||||
fn test_trigger_type_tag() {
|
||||
assert_eq!(
|
||||
Trigger::Cron {
|
||||
schedule: String::new()
|
||||
schedule: String::new(),
|
||||
timezone: None,
|
||||
}
|
||||
.type_tag(),
|
||||
"cron"
|
||||
|
||||
+60
-15
@@ -170,7 +170,7 @@ impl RoutineEngine {
|
||||
continue;
|
||||
}
|
||||
|
||||
let detail = if let Trigger::Cron { ref schedule } = routine.trigger {
|
||||
let detail = if let Trigger::Cron { ref schedule, .. } = routine.trigger {
|
||||
Some(schedule.clone())
|
||||
} else {
|
||||
None
|
||||
@@ -184,7 +184,11 @@ impl RoutineEngine {
|
||||
///
|
||||
/// Bypasses cooldown checks (those only apply to cron/event triggers).
|
||||
/// Still enforces enabled check and concurrent run limit.
|
||||
pub async fn fire_manual(&self, routine_id: Uuid) -> Result<Uuid, RoutineError> {
|
||||
pub async fn fire_manual(
|
||||
&self,
|
||||
routine_id: Uuid,
|
||||
user_id: Option<&str>,
|
||||
) -> Result<Uuid, RoutineError> {
|
||||
let routine = self
|
||||
.store
|
||||
.get_routine(routine_id)
|
||||
@@ -194,6 +198,13 @@ impl RoutineEngine {
|
||||
})?
|
||||
.ok_or(RoutineError::NotFound { id: routine_id })?;
|
||||
|
||||
// Enforce ownership when a user_id is provided (gateway calls).
|
||||
if let Some(uid) = user_id
|
||||
&& routine.user_id != uid
|
||||
{
|
||||
return Err(RoutineError::NotAuthorized { id: routine_id });
|
||||
}
|
||||
|
||||
if !routine.enabled {
|
||||
return Err(RoutineError::Disabled {
|
||||
name: routine.name.clone(),
|
||||
@@ -369,8 +380,12 @@ async fn execute_routine(ctx: EngineContext, routine: Routine, run: RoutineRun)
|
||||
|
||||
// Update routine runtime state
|
||||
let now = Utc::now();
|
||||
let next_fire = if let Trigger::Cron { ref schedule } = routine.trigger {
|
||||
next_cron_fire(schedule).unwrap_or(None)
|
||||
let next_fire = if let Trigger::Cron {
|
||||
ref schedule,
|
||||
ref timezone,
|
||||
} = routine.trigger
|
||||
{
|
||||
next_cron_fire(schedule, timezone.as_deref()).unwrap_or(None)
|
||||
} else {
|
||||
None
|
||||
};
|
||||
@@ -396,6 +411,39 @@ async fn execute_routine(ctx: EngineContext, routine: Routine, run: RoutineRun)
|
||||
tracing::error!(routine = %routine.name, "Failed to update runtime state: {}", e);
|
||||
}
|
||||
|
||||
// Persist routine result to its dedicated conversation thread
|
||||
let thread_id = match ctx
|
||||
.store
|
||||
.get_or_create_routine_conversation(routine.id, &routine.name, &routine.user_id)
|
||||
.await
|
||||
{
|
||||
Ok(conv_id) => {
|
||||
tracing::debug!(
|
||||
routine = %routine.name,
|
||||
routine_id = %routine.id,
|
||||
conversation_id = %conv_id,
|
||||
"Resolved routine conversation thread"
|
||||
);
|
||||
// Record the run result as a conversation message
|
||||
let msg = match (&summary, status) {
|
||||
(Some(s), _) => format!("[{}] {}: {}", run.trigger_type, status, s),
|
||||
(None, _) => format!("[{}] {}", run.trigger_type, status),
|
||||
};
|
||||
if let Err(e) = ctx
|
||||
.store
|
||||
.add_conversation_message(conv_id, "assistant", &msg)
|
||||
.await
|
||||
{
|
||||
tracing::error!(routine = %routine.name, "Failed to persist routine message: {}", e);
|
||||
}
|
||||
Some(conv_id.to_string())
|
||||
}
|
||||
Err(e) => {
|
||||
tracing::error!(routine = %routine.name, "Failed to get routine conversation: {}", e);
|
||||
None
|
||||
}
|
||||
};
|
||||
|
||||
// Send notifications based on config
|
||||
send_notification(
|
||||
&ctx.notify_tx,
|
||||
@@ -403,6 +451,7 @@ async fn execute_routine(ctx: EngineContext, routine: Routine, run: RoutineRun)
|
||||
&routine.name,
|
||||
status,
|
||||
summary.as_deref(),
|
||||
thread_id.as_deref(),
|
||||
)
|
||||
.await;
|
||||
}
|
||||
@@ -443,18 +492,13 @@ async fn execute_full_job(
|
||||
reason: "scheduler not available".to_string(),
|
||||
})?;
|
||||
|
||||
// Set the message tool's default channel/target from the routine's notify config
|
||||
// so the LLM can send results without triggering cross-channel approval.
|
||||
// TODO: This mutates shared global state and can race with concurrent jobs.
|
||||
// Move notify config into JobContext metadata and apply per-job instead.
|
||||
let mut metadata = serde_json::json!({ "max_iterations": max_iterations });
|
||||
// Carry the routine's notify config in job metadata so the message tool
|
||||
// can resolve channel/target per-job without global state mutation.
|
||||
if let Some(channel) = &routine.notify.channel {
|
||||
scheduler
|
||||
.tools()
|
||||
.set_message_tool_context(Some(channel.clone()), Some(routine.notify.user.clone()))
|
||||
.await;
|
||||
metadata["notify_channel"] = serde_json::json!(channel);
|
||||
}
|
||||
|
||||
let metadata = serde_json::json!({ "max_iterations": max_iterations });
|
||||
metadata["notify_user"] = serde_json::json!(&routine.notify.user);
|
||||
|
||||
// Build approval context: UnlessAutoApproved tools are auto-approved for routines;
|
||||
// Always tools require explicit listing in tool_permissions.
|
||||
@@ -611,6 +655,7 @@ async fn send_notification(
|
||||
routine_name: &str,
|
||||
status: RunStatus,
|
||||
summary: Option<&str>,
|
||||
thread_id: Option<&str>,
|
||||
) {
|
||||
let should_notify = match status {
|
||||
RunStatus::Ok => notify.on_success,
|
||||
@@ -637,7 +682,7 @@ async fn send_notification(
|
||||
|
||||
let response = OutgoingResponse {
|
||||
content: message,
|
||||
thread_id: None,
|
||||
thread_id: thread_id.map(String::from),
|
||||
attachments: Vec::new(),
|
||||
metadata: serde_json::json!({
|
||||
"source": "routine",
|
||||
|
||||
@@ -160,6 +160,13 @@ impl Scheduler {
|
||||
.create_job_for_user(user_id, title, description)
|
||||
.await?;
|
||||
|
||||
// Apply token budget from config, allowing per-job metadata override.
|
||||
let max_tokens = metadata
|
||||
.as_ref()
|
||||
.and_then(|m| m.get("max_tokens"))
|
||||
.and_then(|v| v.as_u64())
|
||||
.unwrap_or(self.config.max_tokens_per_job);
|
||||
|
||||
// Apply metadata if provided
|
||||
if let Some(meta) = metadata {
|
||||
self.context_manager
|
||||
@@ -169,6 +176,15 @@ impl Scheduler {
|
||||
.await?;
|
||||
}
|
||||
|
||||
// Set token budget (separate update to avoid overwriting metadata)
|
||||
if max_tokens > 0 {
|
||||
self.context_manager
|
||||
.update_context(job_id, |ctx| {
|
||||
ctx.max_tokens = max_tokens;
|
||||
})
|
||||
.await?;
|
||||
}
|
||||
|
||||
// Persist to DB before scheduling so the worker's FK references are valid
|
||||
if let Some(ref store) = self.store {
|
||||
let ctx = self.context_manager.get_context(job_id).await?;
|
||||
|
||||
+324
-11
@@ -16,6 +16,7 @@ use chrono::{DateTime, Utc};
|
||||
use serde::{Deserialize, Serialize};
|
||||
use uuid::Uuid;
|
||||
|
||||
use crate::channels::web::util::truncate_preview;
|
||||
use crate::llm::{ChatMessage, ToolCall};
|
||||
|
||||
/// A session containing one or more threads.
|
||||
@@ -164,6 +165,10 @@ pub struct PendingApproval {
|
||||
/// executed yet when approval was requested.
|
||||
#[serde(default)]
|
||||
pub deferred_tool_calls: Vec<ToolCall>,
|
||||
/// User timezone at the time the approval was requested, so it persists
|
||||
/// through the approval flow even if the approval message lacks timezone.
|
||||
#[serde(default)]
|
||||
pub user_timezone: Option<String>,
|
||||
}
|
||||
|
||||
/// A conversation thread within a session.
|
||||
@@ -316,7 +321,13 @@ impl Thread {
|
||||
}
|
||||
}
|
||||
|
||||
/// Get all messages for context building.
|
||||
/// Get all messages for context building, including tool call history.
|
||||
///
|
||||
/// Emits the full LLM-compatible message sequence per turn:
|
||||
/// `user → [assistant_with_tool_calls → tool_result*] → assistant`
|
||||
///
|
||||
/// This ensures the LLM sees prior tool executions and won't re-attempt
|
||||
/// completed actions in subsequent turns.
|
||||
pub fn messages(&self) -> Vec<ChatMessage> {
|
||||
let mut messages = Vec::new();
|
||||
for turn in &self.turns {
|
||||
@@ -328,6 +339,42 @@ impl Thread {
|
||||
turn.image_content_parts.clone(),
|
||||
));
|
||||
}
|
||||
|
||||
if !turn.tool_calls.is_empty() {
|
||||
// Build ToolCall objects with synthetic stable IDs
|
||||
let tool_calls: Vec<ToolCall> = turn
|
||||
.tool_calls
|
||||
.iter()
|
||||
.enumerate()
|
||||
.map(|(i, tc)| ToolCall {
|
||||
id: format!("turn{}_{}", turn.turn_number, i),
|
||||
name: tc.name.clone(),
|
||||
arguments: tc.parameters.clone(),
|
||||
})
|
||||
.collect();
|
||||
|
||||
// Assistant message declaring the tool calls (no text content)
|
||||
messages.push(ChatMessage::assistant_with_tool_calls(None, tool_calls));
|
||||
|
||||
// Individual tool result messages, truncated to limit context size.
|
||||
for (i, tc) in turn.tool_calls.iter().enumerate() {
|
||||
let call_id = format!("turn{}_{}", turn.turn_number, i);
|
||||
let content = if let Some(ref err) = tc.error {
|
||||
// .error already contains the full error text;
|
||||
// pass through without wrapping to avoid double-prefix.
|
||||
truncate_preview(err, 1000)
|
||||
} else if let Some(ref res) = tc.result {
|
||||
let raw = match res {
|
||||
serde_json::Value::String(s) => s.clone(),
|
||||
other => other.to_string(),
|
||||
};
|
||||
truncate_preview(&raw, 1000)
|
||||
} else {
|
||||
"OK".to_string()
|
||||
};
|
||||
messages.push(ChatMessage::tool_result(call_id, &tc.name, content));
|
||||
}
|
||||
}
|
||||
if let Some(ref response) = turn.response {
|
||||
messages.push(ChatMessage::assistant(response));
|
||||
}
|
||||
@@ -349,13 +396,16 @@ impl Thread {
|
||||
|
||||
/// Restore thread state from a checkpoint's messages.
|
||||
///
|
||||
/// Clears existing turns and rebuilds from message pairs.
|
||||
/// Messages should alternate: user, assistant, user, assistant...
|
||||
/// Clears existing turns and rebuilds from the message sequence.
|
||||
/// Handles the full message pattern including tool messages:
|
||||
/// `user → [assistant_with_tool_calls → tool_result*] → assistant`
|
||||
///
|
||||
/// Also supports the legacy pattern (user/assistant pairs only) for
|
||||
/// backward compatibility with old checkpoint data.
|
||||
pub fn restore_from_messages(&mut self, messages: Vec<ChatMessage>) {
|
||||
self.turns.clear();
|
||||
self.state = ThreadState::Idle;
|
||||
|
||||
// Messages alternate: user, assistant, user, assistant...
|
||||
let mut iter = messages.into_iter().peekable();
|
||||
let mut turn_number = 0;
|
||||
|
||||
@@ -363,18 +413,58 @@ impl Thread {
|
||||
if msg.role == crate::llm::Role::User {
|
||||
let mut turn = Turn::new(turn_number, &msg.content);
|
||||
|
||||
// Check if next is assistant response
|
||||
if let Some(next) = iter.peek()
|
||||
&& next.role == crate::llm::Role::Assistant
|
||||
{
|
||||
// iter.next() is guaranteed Some after a successful peek()
|
||||
if let Some(response) = iter.next() {
|
||||
turn.complete(&response.content);
|
||||
// Consume tool call sequences (assistant_with_tool_calls + tool_results).
|
||||
// A single turn may contain multiple rounds of tool calls, so we
|
||||
// track the cumulative base index into turn.tool_calls.
|
||||
while let Some(next) = iter.peek() {
|
||||
if next.role == crate::llm::Role::Assistant && next.tool_calls.is_some() {
|
||||
let call_base_idx = turn.tool_calls.len();
|
||||
|
||||
if let Some(assistant_msg) = iter.next()
|
||||
&& let Some(ref tcs) = assistant_msg.tool_calls
|
||||
{
|
||||
for tc in tcs {
|
||||
turn.record_tool_call(&tc.name, tc.arguments.clone());
|
||||
}
|
||||
}
|
||||
|
||||
// Consume the corresponding tool_result messages,
|
||||
// indexing relative to this batch's base offset.
|
||||
let mut pos = 0;
|
||||
while let Some(tr) = iter.peek() {
|
||||
if tr.role != crate::llm::Role::Tool {
|
||||
break;
|
||||
}
|
||||
if let Some(tool_msg) = iter.next() {
|
||||
let idx = call_base_idx + pos;
|
||||
if idx < turn.tool_calls.len() {
|
||||
// Store as result — the error/success distinction
|
||||
// is for the live turn only; restored context just
|
||||
// needs the content the LLM originally saw.
|
||||
turn.tool_calls[idx].result =
|
||||
Some(serde_json::Value::String(tool_msg.content.clone()));
|
||||
}
|
||||
}
|
||||
pos += 1;
|
||||
}
|
||||
} else {
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
// Check if next is the final assistant response for this turn
|
||||
let is_final_assistant = iter.peek().is_some_and(|n| {
|
||||
n.role == crate::llm::Role::Assistant && n.tool_calls.is_none()
|
||||
});
|
||||
if is_final_assistant && let Some(response) = iter.next() {
|
||||
turn.complete(&response.content);
|
||||
}
|
||||
|
||||
self.turns.push(turn);
|
||||
turn_number += 1;
|
||||
} else {
|
||||
// Skip non-user messages that aren't anchored to a turn
|
||||
continue;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -976,6 +1066,7 @@ mod tests {
|
||||
tool_call_id: "call_123".to_string(),
|
||||
context_messages: vec![ChatMessage::user("do it")],
|
||||
deferred_tool_calls: vec![],
|
||||
user_timezone: None,
|
||||
};
|
||||
|
||||
thread.await_approval(approval);
|
||||
@@ -1001,6 +1092,7 @@ mod tests {
|
||||
tool_call_id: "call_456".to_string(),
|
||||
context_messages: vec![],
|
||||
deferred_tool_calls: vec![],
|
||||
user_timezone: None,
|
||||
};
|
||||
|
||||
thread.await_approval(approval);
|
||||
@@ -1029,4 +1121,225 @@ mod tests {
|
||||
ThreadState::Processing
|
||||
);
|
||||
}
|
||||
|
||||
// Regression tests for #568: tool call history must survive hydration.
|
||||
|
||||
#[test]
|
||||
fn test_messages_includes_tool_calls() {
|
||||
let mut thread = Thread::new(Uuid::new_v4());
|
||||
|
||||
thread.start_turn("Search for X");
|
||||
{
|
||||
let turn = thread.turns.last_mut().unwrap();
|
||||
turn.record_tool_call("memory_search", serde_json::json!({"query": "X"}));
|
||||
turn.record_tool_result(serde_json::json!("Found X in doc.md"));
|
||||
}
|
||||
thread.complete_turn("I found X in doc.md.");
|
||||
|
||||
let messages = thread.messages();
|
||||
// user + assistant_with_tool_calls + tool_result + assistant = 4
|
||||
assert_eq!(messages.len(), 4);
|
||||
|
||||
assert_eq!(messages[0].role, crate::llm::Role::User);
|
||||
assert_eq!(messages[0].content, "Search for X");
|
||||
|
||||
assert_eq!(messages[1].role, crate::llm::Role::Assistant);
|
||||
assert!(messages[1].tool_calls.is_some());
|
||||
let tcs = messages[1].tool_calls.as_ref().unwrap();
|
||||
assert_eq!(tcs.len(), 1);
|
||||
assert_eq!(tcs[0].name, "memory_search");
|
||||
|
||||
assert_eq!(messages[2].role, crate::llm::Role::Tool);
|
||||
assert!(messages[2].content.contains("Found X"));
|
||||
|
||||
assert_eq!(messages[3].role, crate::llm::Role::Assistant);
|
||||
assert_eq!(messages[3].content, "I found X in doc.md.");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_messages_multiple_tool_calls_per_turn() {
|
||||
let mut thread = Thread::new(Uuid::new_v4());
|
||||
|
||||
thread.start_turn("Do two things");
|
||||
{
|
||||
let turn = thread.turns.last_mut().unwrap();
|
||||
turn.record_tool_call("echo", serde_json::json!({"msg": "a"}));
|
||||
turn.record_tool_result(serde_json::json!("a"));
|
||||
turn.record_tool_call("time", serde_json::json!({}));
|
||||
turn.record_tool_error("timeout");
|
||||
}
|
||||
thread.complete_turn("Done.");
|
||||
|
||||
let messages = thread.messages();
|
||||
// user + assistant_with_calls(2) + tool_result + tool_result + assistant = 5
|
||||
assert_eq!(messages.len(), 5);
|
||||
|
||||
let tcs = messages[1].tool_calls.as_ref().unwrap();
|
||||
assert_eq!(tcs.len(), 2);
|
||||
|
||||
// First tool: success
|
||||
assert_eq!(messages[2].content, "a");
|
||||
// Second tool: error (passed through directly, no wrapping)
|
||||
assert!(messages[3].content.contains("timeout"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_restore_from_messages_with_tool_calls() {
|
||||
let mut thread = Thread::new(Uuid::new_v4());
|
||||
|
||||
// Build a message sequence with tool calls
|
||||
let tc = ToolCall {
|
||||
id: "call_0".to_string(),
|
||||
name: "search".to_string(),
|
||||
arguments: serde_json::json!({"q": "test"}),
|
||||
};
|
||||
let messages = vec![
|
||||
ChatMessage::user("Find test"),
|
||||
ChatMessage::assistant_with_tool_calls(None, vec![tc]),
|
||||
ChatMessage::tool_result("call_0", "search", "result: found"),
|
||||
ChatMessage::assistant("Found it."),
|
||||
];
|
||||
|
||||
thread.restore_from_messages(messages);
|
||||
|
||||
assert_eq!(thread.turns.len(), 1);
|
||||
let turn = &thread.turns[0];
|
||||
assert_eq!(turn.user_input, "Find test");
|
||||
assert_eq!(turn.tool_calls.len(), 1);
|
||||
assert_eq!(turn.tool_calls[0].name, "search");
|
||||
assert_eq!(
|
||||
turn.tool_calls[0].result,
|
||||
Some(serde_json::Value::String("result: found".to_string()))
|
||||
);
|
||||
assert_eq!(turn.response, Some("Found it.".to_string()));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_restore_from_messages_with_tool_error() {
|
||||
let mut thread = Thread::new(Uuid::new_v4());
|
||||
|
||||
let tc = ToolCall {
|
||||
id: "call_0".to_string(),
|
||||
name: "http".to_string(),
|
||||
arguments: serde_json::json!({}),
|
||||
};
|
||||
let messages = vec![
|
||||
ChatMessage::user("Fetch URL"),
|
||||
ChatMessage::assistant_with_tool_calls(None, vec![tc]),
|
||||
ChatMessage::tool_result("call_0", "http", "Error: timeout"),
|
||||
ChatMessage::assistant("The request timed out."),
|
||||
];
|
||||
|
||||
thread.restore_from_messages(messages);
|
||||
|
||||
// restore_from_messages stores all tool content as result (not error),
|
||||
// because it can't reliably distinguish errors from results that happen
|
||||
// to start with "Error: ". The content is preserved for LLM context.
|
||||
let turn = &thread.turns[0];
|
||||
assert_eq!(
|
||||
turn.tool_calls[0].result,
|
||||
Some(serde_json::Value::String("Error: timeout".to_string()))
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_messages_round_trip_with_tools() {
|
||||
// Build a thread with tool calls, get messages(), restore, get messages() again
|
||||
// The two message sequences should be equivalent.
|
||||
let mut thread = Thread::new(Uuid::new_v4());
|
||||
|
||||
thread.start_turn("Do search");
|
||||
{
|
||||
let turn = thread.turns.last_mut().unwrap();
|
||||
turn.record_tool_call("search", serde_json::json!({"q": "test"}));
|
||||
turn.record_tool_result(serde_json::json!("found"));
|
||||
}
|
||||
thread.complete_turn("Here are results.");
|
||||
|
||||
let messages_original = thread.messages();
|
||||
|
||||
// Restore into a new thread
|
||||
let mut thread2 = Thread::new(Uuid::new_v4());
|
||||
thread2.restore_from_messages(messages_original.clone());
|
||||
|
||||
let messages_restored = thread2.messages();
|
||||
|
||||
// Same number of messages
|
||||
assert_eq!(messages_original.len(), messages_restored.len());
|
||||
|
||||
// Same roles
|
||||
for (orig, rest) in messages_original.iter().zip(messages_restored.iter()) {
|
||||
assert_eq!(orig.role, rest.role);
|
||||
}
|
||||
|
||||
// Same final response
|
||||
assert_eq!(
|
||||
messages_original.last().unwrap().content,
|
||||
messages_restored.last().unwrap().content
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_restore_multi_stage_tool_calls() {
|
||||
let mut thread = Thread::new(Uuid::new_v4());
|
||||
|
||||
let tc1 = ToolCall {
|
||||
id: "call_a".to_string(),
|
||||
name: "search".to_string(),
|
||||
arguments: serde_json::json!({"q": "data"}),
|
||||
};
|
||||
let tc2 = ToolCall {
|
||||
id: "call_b".to_string(),
|
||||
name: "write".to_string(),
|
||||
arguments: serde_json::json!({"path": "out.txt"}),
|
||||
};
|
||||
let messages = vec![
|
||||
ChatMessage::user("Find and save"),
|
||||
ChatMessage::assistant_with_tool_calls(None, vec![tc1]),
|
||||
ChatMessage::tool_result("call_a", "search", "found data"),
|
||||
ChatMessage::assistant_with_tool_calls(None, vec![tc2]),
|
||||
ChatMessage::tool_result("call_b", "write", "written"),
|
||||
ChatMessage::assistant("Done, saved to out.txt"),
|
||||
];
|
||||
|
||||
thread.restore_from_messages(messages);
|
||||
|
||||
assert_eq!(thread.turns.len(), 1);
|
||||
let turn = &thread.turns[0];
|
||||
assert_eq!(turn.tool_calls.len(), 2);
|
||||
assert_eq!(turn.tool_calls[0].name, "search");
|
||||
assert_eq!(turn.tool_calls[1].name, "write");
|
||||
assert_eq!(
|
||||
turn.tool_calls[0].result,
|
||||
Some(serde_json::Value::String("found data".to_string()))
|
||||
);
|
||||
assert_eq!(
|
||||
turn.tool_calls[1].result,
|
||||
Some(serde_json::Value::String("written".to_string()))
|
||||
);
|
||||
assert_eq!(turn.response, Some("Done, saved to out.txt".to_string()));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_messages_truncates_large_tool_results() {
|
||||
let mut thread = Thread::new(Uuid::new_v4());
|
||||
|
||||
thread.start_turn("Read big file");
|
||||
{
|
||||
let turn = thread.turns.last_mut().unwrap();
|
||||
turn.record_tool_call("read_file", serde_json::json!({"path": "big.txt"}));
|
||||
let big_result = "x".repeat(2000);
|
||||
turn.record_tool_result(serde_json::json!(big_result));
|
||||
}
|
||||
thread.complete_turn("Here's the file content.");
|
||||
|
||||
let messages = thread.messages();
|
||||
let tool_result_content = &messages[2].content;
|
||||
assert!(
|
||||
tool_result_content.len() <= 1010,
|
||||
"Tool result should be truncated, got {} chars",
|
||||
tool_result_content.len()
|
||||
);
|
||||
assert!(tool_result_content.ends_with("..."));
|
||||
}
|
||||
}
|
||||
|
||||
+305
-60
@@ -20,7 +20,7 @@ use crate::channels::web::util::truncate_preview;
|
||||
use crate::channels::{IncomingMessage, StatusUpdate};
|
||||
use crate::context::JobContext;
|
||||
use crate::error::Error;
|
||||
use crate::llm::ChatMessage;
|
||||
use crate::llm::{ChatMessage, ToolCall};
|
||||
use crate::tools::redact_params;
|
||||
|
||||
impl Agent {
|
||||
@@ -66,16 +66,7 @@ impl Agent {
|
||||
.await
|
||||
.unwrap_or_default();
|
||||
msg_count = db_messages.len();
|
||||
chat_messages = db_messages
|
||||
.iter()
|
||||
.filter_map(|m| match m.role.as_str() {
|
||||
"user" => Some(ChatMessage::user(&m.content)),
|
||||
"assistant" => Some(ChatMessage::assistant(&m.content)),
|
||||
// tool_calls rows are UI metadata (tool name + preview),
|
||||
// not part of the LLM conversation context.
|
||||
_ => None,
|
||||
})
|
||||
.collect();
|
||||
chat_messages = rebuild_chat_messages_from_db(&db_messages);
|
||||
} else {
|
||||
msg_count = 0;
|
||||
}
|
||||
@@ -230,7 +221,7 @@ impl Agent {
|
||||
)
|
||||
.await;
|
||||
|
||||
let compactor = ContextCompactor::new(self.llm().clone(), self.safety().clone());
|
||||
let compactor = ContextCompactor::new(self.llm().clone());
|
||||
if let Err(e) = compactor
|
||||
.compact(thread, strategy, self.workspace().map(|w| w.as_ref()))
|
||||
.await
|
||||
@@ -340,10 +331,10 @@ impl Agent {
|
||||
};
|
||||
|
||||
thread.complete_turn(&response);
|
||||
let tool_calls = thread
|
||||
let (turn_number, tool_calls) = thread
|
||||
.turns
|
||||
.last()
|
||||
.map(|t| t.tool_calls.clone())
|
||||
.map(|t| (t.turn_number, t.tool_calls.clone()))
|
||||
.unwrap_or_default();
|
||||
let _ = self
|
||||
.channels
|
||||
@@ -355,7 +346,7 @@ impl Agent {
|
||||
.await;
|
||||
|
||||
// Persist tool calls then assistant response (user message already persisted at turn start)
|
||||
self.persist_tool_calls(thread_id, &message.user_id, &tool_calls)
|
||||
self.persist_tool_calls(thread_id, &message.user_id, turn_number, &tool_calls)
|
||||
.await;
|
||||
self.persist_assistant_response(thread_id, &message.user_id, &response)
|
||||
.await;
|
||||
@@ -464,6 +455,7 @@ impl Agent {
|
||||
&self,
|
||||
thread_id: Uuid,
|
||||
user_id: &str,
|
||||
turn_number: usize,
|
||||
tool_calls: &[crate::agent::session::TurnToolCall],
|
||||
) {
|
||||
if tool_calls.is_empty() {
|
||||
@@ -477,14 +469,24 @@ impl Agent {
|
||||
|
||||
let summaries: Vec<serde_json::Value> = tool_calls
|
||||
.iter()
|
||||
.map(|tc| {
|
||||
let mut obj = serde_json::json!({ "name": tc.name });
|
||||
.enumerate()
|
||||
.map(|(i, tc)| {
|
||||
let mut obj = serde_json::json!({
|
||||
"name": tc.name,
|
||||
"call_id": format!("turn{}_{}", turn_number, i),
|
||||
});
|
||||
if let Some(ref result) = tc.result {
|
||||
let preview = match result {
|
||||
serde_json::Value::String(s) => truncate_preview(s, 500),
|
||||
other => truncate_preview(&other.to_string(), 500),
|
||||
};
|
||||
obj["result_preview"] = serde_json::Value::String(preview);
|
||||
// Store full result (truncated to ~1000 chars) for LLM context rebuild
|
||||
let full_result = match result {
|
||||
serde_json::Value::String(s) => truncate_preview(s, 1000),
|
||||
other => truncate_preview(&other.to_string(), 1000),
|
||||
};
|
||||
obj["result"] = serde_json::Value::String(full_result);
|
||||
}
|
||||
if let Some(ref error) = tc.error {
|
||||
obj["error"] = serde_json::Value::String(truncate_preview(error, 200));
|
||||
@@ -627,7 +629,7 @@ impl Agent {
|
||||
crate::agent::context_monitor::CompactionStrategy::Summarize { keep_recent: 5 },
|
||||
);
|
||||
|
||||
let compactor = ContextCompactor::new(self.llm().clone(), self.safety().clone());
|
||||
let compactor = ContextCompactor::new(self.llm().clone());
|
||||
match compactor
|
||||
.compact(thread, strategy, self.workspace().map(|w| w.as_ref()))
|
||||
.await
|
||||
@@ -746,6 +748,16 @@ impl Agent {
|
||||
let mut job_ctx =
|
||||
JobContext::with_user(&message.user_id, "chat", "Interactive chat session");
|
||||
job_ctx.http_interceptor = self.deps.http_interceptor.clone();
|
||||
// Prefer a valid timezone from the approval message, fall back to the
|
||||
// resolved timezone stored when the approval was originally requested.
|
||||
let tz_candidate = message
|
||||
.timezone
|
||||
.as_deref()
|
||||
.filter(|tz| crate::timezone::parse_timezone(tz).is_some())
|
||||
.or(pending.user_timezone.as_deref());
|
||||
if let Some(tz) = tz_candidate {
|
||||
job_ctx.user_timezone = tz.to_string();
|
||||
}
|
||||
|
||||
let _ = self
|
||||
.channels
|
||||
@@ -797,19 +809,33 @@ impl Agent {
|
||||
let mut context_messages = pending.context_messages;
|
||||
let deferred_tool_calls = pending.deferred_tool_calls;
|
||||
|
||||
// Record result in thread
|
||||
// Sanitize tool result, then record the cleaned version in the
|
||||
// thread. Must happen before auth intercept check which may return early.
|
||||
let is_tool_error = tool_result.is_err();
|
||||
let result_content = match &tool_result {
|
||||
Ok(output) => {
|
||||
let sanitized = self
|
||||
.safety()
|
||||
.sanitize_tool_output(&pending.tool_name, output);
|
||||
self.safety().wrap_for_llm(
|
||||
&pending.tool_name,
|
||||
&sanitized.content,
|
||||
sanitized.was_modified,
|
||||
)
|
||||
}
|
||||
Err(e) => format!("Error: {}", e),
|
||||
};
|
||||
|
||||
// Record sanitized result in thread
|
||||
{
|
||||
let mut sess = session.lock().await;
|
||||
if let Some(thread) = sess.threads.get_mut(&thread_id)
|
||||
&& let Some(turn) = thread.last_turn_mut()
|
||||
{
|
||||
match &tool_result {
|
||||
Ok(output) => {
|
||||
turn.record_tool_result(serde_json::json!(output));
|
||||
}
|
||||
Err(e) => {
|
||||
turn.record_tool_error(e.to_string());
|
||||
}
|
||||
if is_tool_error {
|
||||
turn.record_tool_error(result_content.clone());
|
||||
} else {
|
||||
turn.record_tool_result(serde_json::json!(result_content));
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -831,21 +857,6 @@ impl Agent {
|
||||
return Ok(SubmissionResult::response(instructions));
|
||||
}
|
||||
|
||||
// Add tool result to context
|
||||
let result_content = match tool_result {
|
||||
Ok(output) => {
|
||||
let sanitized = self
|
||||
.safety()
|
||||
.sanitize_tool_output(&pending.tool_name, &output);
|
||||
self.safety().wrap_for_llm(
|
||||
&pending.tool_name,
|
||||
&sanitized.content,
|
||||
sanitized.was_modified,
|
||||
)
|
||||
}
|
||||
Err(e) => format!("Error: {}", e),
|
||||
};
|
||||
|
||||
context_messages.push(ChatMessage::tool_result(
|
||||
&pending.tool_call_id,
|
||||
&pending.tool_name,
|
||||
@@ -1050,15 +1061,31 @@ impl Agent {
|
||||
.await;
|
||||
}
|
||||
|
||||
// Record in thread
|
||||
// Sanitize first, then record the cleaned version in thread.
|
||||
// Must happen before auth detection which may set deferred_auth.
|
||||
let is_deferred_error = deferred_result.is_err();
|
||||
let deferred_content = match &deferred_result {
|
||||
Ok(output) => {
|
||||
let sanitized = self.safety().sanitize_tool_output(&tc.name, output);
|
||||
self.safety().wrap_for_llm(
|
||||
&tc.name,
|
||||
&sanitized.content,
|
||||
sanitized.was_modified,
|
||||
)
|
||||
}
|
||||
Err(e) => format!("Error: {}", e),
|
||||
};
|
||||
|
||||
// Record sanitized result in thread
|
||||
{
|
||||
let mut sess = session.lock().await;
|
||||
if let Some(thread) = sess.threads.get_mut(&thread_id)
|
||||
&& let Some(turn) = thread.last_turn_mut()
|
||||
{
|
||||
match &deferred_result {
|
||||
Ok(output) => turn.record_tool_result(serde_json::json!(output)),
|
||||
Err(e) => turn.record_tool_error(e.to_string()),
|
||||
if is_deferred_error {
|
||||
turn.record_tool_error(deferred_content.clone());
|
||||
} else {
|
||||
turn.record_tool_result(serde_json::json!(deferred_content));
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1080,18 +1107,6 @@ impl Agent {
|
||||
deferred_auth = Some(instructions);
|
||||
}
|
||||
|
||||
let deferred_content = match deferred_result {
|
||||
Ok(output) => {
|
||||
let sanitized = self.safety().sanitize_tool_output(&tc.name, &output);
|
||||
self.safety().wrap_for_llm(
|
||||
&tc.name,
|
||||
&sanitized.content,
|
||||
sanitized.was_modified,
|
||||
)
|
||||
}
|
||||
Err(e) => format!("Error: {}", e),
|
||||
};
|
||||
|
||||
context_messages.push(ChatMessage::tool_result(&tc.id, &tc.name, deferred_content));
|
||||
}
|
||||
|
||||
@@ -1111,6 +1126,8 @@ impl Agent {
|
||||
tool_call_id: tc.id.clone(),
|
||||
context_messages: context_messages.clone(),
|
||||
deferred_tool_calls: deferred_tool_calls[approval_idx + 1..].to_vec(),
|
||||
// Carry forward the resolved timezone from the original pending approval
|
||||
user_timezone: pending.user_timezone.clone(),
|
||||
};
|
||||
|
||||
let request_id = new_pending.request_id;
|
||||
@@ -1157,13 +1174,13 @@ impl Agent {
|
||||
match result {
|
||||
Ok(AgenticLoopResult::Response(response)) => {
|
||||
thread.complete_turn(&response);
|
||||
let tool_calls = thread
|
||||
let (turn_number, tool_calls) = thread
|
||||
.turns
|
||||
.last()
|
||||
.map(|t| t.tool_calls.clone())
|
||||
.map(|t| (t.turn_number, t.tool_calls.clone()))
|
||||
.unwrap_or_default();
|
||||
// User message already persisted at turn start; save tool calls then assistant response
|
||||
self.persist_tool_calls(thread_id, &message.user_id, &tool_calls)
|
||||
self.persist_tool_calls(thread_id, &message.user_id, turn_number, &tool_calls)
|
||||
.await;
|
||||
self.persist_assistant_response(thread_id, &message.user_id, &response)
|
||||
.await;
|
||||
@@ -1478,3 +1495,231 @@ impl Agent {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Rebuild full LLM-compatible `ChatMessage` sequence from DB messages.
|
||||
///
|
||||
/// Parses `role="tool_calls"` rows to reconstruct `assistant_with_tool_calls`
|
||||
/// and `tool_result` messages so that the LLM sees the complete tool execution
|
||||
/// history on thread hydration. Falls back gracefully for legacy rows that
|
||||
/// lack the enriched fields (`call_id`, `parameters`, `result`).
|
||||
fn rebuild_chat_messages_from_db(
|
||||
db_messages: &[crate::history::ConversationMessage],
|
||||
) -> Vec<ChatMessage> {
|
||||
let mut result = Vec::new();
|
||||
|
||||
for msg in db_messages {
|
||||
match msg.role.as_str() {
|
||||
"user" => result.push(ChatMessage::user(&msg.content)),
|
||||
"assistant" => result.push(ChatMessage::assistant(&msg.content)),
|
||||
"tool_calls" => {
|
||||
// Try to parse the enriched JSON and rebuild tool messages.
|
||||
if let Ok(calls) = serde_json::from_str::<Vec<serde_json::Value>>(&msg.content) {
|
||||
if calls.is_empty() {
|
||||
continue;
|
||||
}
|
||||
|
||||
// Check if this is an enriched row (has call_id) or legacy
|
||||
let has_call_id = calls
|
||||
.first()
|
||||
.and_then(|c| c.get("call_id"))
|
||||
.and_then(|v| v.as_str())
|
||||
.is_some();
|
||||
|
||||
if has_call_id {
|
||||
// Build assistant_with_tool_calls + tool_result messages
|
||||
let tool_calls: Vec<ToolCall> = calls
|
||||
.iter()
|
||||
.map(|c| ToolCall {
|
||||
id: c["call_id"].as_str().unwrap_or("call_0").to_string(),
|
||||
name: c["name"].as_str().unwrap_or("unknown").to_string(),
|
||||
arguments: c
|
||||
.get("parameters")
|
||||
.cloned()
|
||||
.unwrap_or(serde_json::json!({})),
|
||||
})
|
||||
.collect();
|
||||
|
||||
// The assistant text for tool_calls is always None here;
|
||||
// the final assistant response comes as a separate
|
||||
// "assistant" row after this tool_calls row.
|
||||
result.push(ChatMessage::assistant_with_tool_calls(None, tool_calls));
|
||||
|
||||
// Emit tool_result messages for each call
|
||||
for c in &calls {
|
||||
let call_id = c["call_id"].as_str().unwrap_or("call_0").to_string();
|
||||
let name = c["name"].as_str().unwrap_or("unknown").to_string();
|
||||
let content = if let Some(err) = c.get("error").and_then(|v| v.as_str())
|
||||
{
|
||||
format!("Error: {}", err)
|
||||
} else if let Some(res) = c.get("result").and_then(|v| v.as_str()) {
|
||||
res.to_string()
|
||||
} else if let Some(preview) =
|
||||
c.get("result_preview").and_then(|v| v.as_str())
|
||||
{
|
||||
preview.to_string()
|
||||
} else {
|
||||
"OK".to_string()
|
||||
};
|
||||
result.push(ChatMessage::tool_result(call_id, name, content));
|
||||
}
|
||||
}
|
||||
// Legacy rows without call_id: skip (will appear as
|
||||
// simple user/assistant pairs, same as before this fix).
|
||||
}
|
||||
}
|
||||
_ => {} // Skip unknown roles
|
||||
}
|
||||
}
|
||||
|
||||
result
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn test_rebuild_chat_messages_user_assistant_only() {
|
||||
let messages = vec![
|
||||
make_db_msg("user", "Hello"),
|
||||
make_db_msg("assistant", "Hi there!"),
|
||||
];
|
||||
let result = rebuild_chat_messages_from_db(&messages);
|
||||
assert_eq!(result.len(), 2);
|
||||
assert_eq!(result[0].role, crate::llm::Role::User);
|
||||
assert_eq!(result[1].role, crate::llm::Role::Assistant);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_rebuild_chat_messages_with_enriched_tool_calls() {
|
||||
let tool_json = serde_json::json!([
|
||||
{
|
||||
"name": "memory_search",
|
||||
"call_id": "call_0",
|
||||
"parameters": {"query": "test"},
|
||||
"result": "Found 3 results",
|
||||
"result_preview": "Found 3 re..."
|
||||
},
|
||||
{
|
||||
"name": "echo",
|
||||
"call_id": "call_1",
|
||||
"parameters": {"message": "hi"},
|
||||
"error": "timeout"
|
||||
}
|
||||
]);
|
||||
let messages = vec![
|
||||
make_db_msg("user", "Search for test"),
|
||||
make_db_msg("tool_calls", &tool_json.to_string()),
|
||||
make_db_msg("assistant", "I found some results."),
|
||||
];
|
||||
let result = rebuild_chat_messages_from_db(&messages);
|
||||
|
||||
// user + assistant_with_tool_calls + tool_result*2 + assistant
|
||||
assert_eq!(result.len(), 5);
|
||||
|
||||
// user
|
||||
assert_eq!(result[0].role, crate::llm::Role::User);
|
||||
|
||||
// assistant with tool_calls
|
||||
assert_eq!(result[1].role, crate::llm::Role::Assistant);
|
||||
assert!(result[1].tool_calls.is_some());
|
||||
let tcs = result[1].tool_calls.as_ref().unwrap();
|
||||
assert_eq!(tcs.len(), 2);
|
||||
assert_eq!(tcs[0].name, "memory_search");
|
||||
assert_eq!(tcs[0].id, "call_0");
|
||||
assert_eq!(tcs[1].name, "echo");
|
||||
|
||||
// tool results
|
||||
assert_eq!(result[2].role, crate::llm::Role::Tool);
|
||||
assert_eq!(result[2].tool_call_id, Some("call_0".to_string()));
|
||||
assert!(result[2].content.contains("Found 3 results"));
|
||||
|
||||
assert_eq!(result[3].role, crate::llm::Role::Tool);
|
||||
assert_eq!(result[3].tool_call_id, Some("call_1".to_string()));
|
||||
assert!(result[3].content.contains("Error: timeout"));
|
||||
|
||||
// final assistant
|
||||
assert_eq!(result[4].role, crate::llm::Role::Assistant);
|
||||
assert_eq!(result[4].content, "I found some results.");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_rebuild_chat_messages_legacy_tool_calls_skipped() {
|
||||
// Legacy format: no call_id field
|
||||
let tool_json = serde_json::json!([
|
||||
{"name": "echo", "result_preview": "hello"}
|
||||
]);
|
||||
let messages = vec![
|
||||
make_db_msg("user", "Hi"),
|
||||
make_db_msg("tool_calls", &tool_json.to_string()),
|
||||
make_db_msg("assistant", "Done"),
|
||||
];
|
||||
let result = rebuild_chat_messages_from_db(&messages);
|
||||
|
||||
// Legacy rows are skipped, only user + assistant
|
||||
assert_eq!(result.len(), 2);
|
||||
assert_eq!(result[0].role, crate::llm::Role::User);
|
||||
assert_eq!(result[1].role, crate::llm::Role::Assistant);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_rebuild_chat_messages_empty() {
|
||||
let result = rebuild_chat_messages_from_db(&[]);
|
||||
assert!(result.is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_rebuild_chat_messages_malformed_tool_calls_json() {
|
||||
let messages = vec![
|
||||
make_db_msg("user", "Hi"),
|
||||
make_db_msg("tool_calls", "not valid json"),
|
||||
make_db_msg("assistant", "Done"),
|
||||
];
|
||||
let result = rebuild_chat_messages_from_db(&messages);
|
||||
// Malformed JSON is silently skipped
|
||||
assert_eq!(result.len(), 2);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_rebuild_chat_messages_multi_turn_with_tools() {
|
||||
let tool_json_1 = serde_json::json!([
|
||||
{"name": "search", "call_id": "call_0", "parameters": {}, "result": "found it"}
|
||||
]);
|
||||
let tool_json_2 = serde_json::json!([
|
||||
{"name": "write", "call_id": "call_0", "parameters": {"path": "a.txt"}, "result": "ok"}
|
||||
]);
|
||||
let messages = vec![
|
||||
make_db_msg("user", "Find X"),
|
||||
make_db_msg("tool_calls", &tool_json_1.to_string()),
|
||||
make_db_msg("assistant", "Found X"),
|
||||
make_db_msg("user", "Write it"),
|
||||
make_db_msg("tool_calls", &tool_json_2.to_string()),
|
||||
make_db_msg("assistant", "Written"),
|
||||
];
|
||||
let result = rebuild_chat_messages_from_db(&messages);
|
||||
|
||||
// Turn 1: user + assistant_with_calls + tool_result + assistant = 4
|
||||
// Turn 2: user + assistant_with_calls + tool_result + assistant = 4
|
||||
assert_eq!(result.len(), 8);
|
||||
|
||||
// Verify turn boundaries
|
||||
assert_eq!(result[0].content, "Find X");
|
||||
assert!(result[1].tool_calls.is_some());
|
||||
assert_eq!(result[2].role, crate::llm::Role::Tool);
|
||||
assert_eq!(result[3].content, "Found X");
|
||||
|
||||
assert_eq!(result[4].content, "Write it");
|
||||
assert!(result[5].tool_calls.is_some());
|
||||
assert_eq!(result[6].role, crate::llm::Role::Tool);
|
||||
assert_eq!(result[7].content, "Written");
|
||||
}
|
||||
|
||||
fn make_db_msg(role: &str, content: &str) -> crate::history::ConversationMessage {
|
||||
crate::history::ConversationMessage {
|
||||
id: uuid::Uuid::new_v4(),
|
||||
role: role.to_string(),
|
||||
content: content.to_string(),
|
||||
created_at: chrono::Utc::now(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
+165
-37
@@ -15,7 +15,8 @@ use crate::db::Database;
|
||||
use crate::error::Error;
|
||||
use crate::hooks::HookRegistry;
|
||||
use crate::llm::{
|
||||
ActionPlan, ChatMessage, LlmProvider, Reasoning, ReasoningContext, RespondResult, ToolSelection,
|
||||
ActionPlan, ChatMessage, LlmProvider, Reasoning, ReasoningContext, RespondResult, ToolCall,
|
||||
ToolSelection,
|
||||
};
|
||||
use crate::safety::SafetyLayer;
|
||||
use crate::tools::rate_limiter::RateLimitResult;
|
||||
@@ -211,7 +212,7 @@ impl Worker {
|
||||
let job_ctx = self.context_manager().get_context(self.job_id).await?;
|
||||
|
||||
// Create reasoning engine
|
||||
let reasoning = Reasoning::new(self.llm().clone(), self.safety().clone());
|
||||
let reasoning = Reasoning::new(self.llm().clone());
|
||||
|
||||
// Build initial reasoning context (tool definitions refreshed each iteration in execution_loop)
|
||||
let mut reason_ctx = ReasoningContext::new().with_job(&job_ctx.description);
|
||||
@@ -416,7 +417,8 @@ Report when the job is complete or if you encounter issues you cannot resolve."#
|
||||
|
||||
iteration += 1;
|
||||
if iteration > max_iterations {
|
||||
self.mark_stuck("Maximum iterations exceeded").await?;
|
||||
self.mark_failed("Maximum iterations exceeded: job hit the iteration cap")
|
||||
.await?;
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
@@ -436,7 +438,8 @@ Report when the job is complete or if you encounter issues you cannot resolve."#
|
||||
"LLM rate limited during tool selection, backing off"
|
||||
);
|
||||
if consecutive_rate_limits >= MAX_CONSECUTIVE_RATE_LIMITS {
|
||||
self.mark_stuck("Persistent rate limiting").await?;
|
||||
self.mark_failed("Persistent rate limiting: exceeded retry limit")
|
||||
.await?;
|
||||
return Ok(());
|
||||
}
|
||||
self.log_event(
|
||||
@@ -466,7 +469,8 @@ Report when the job is complete or if you encounter issues you cannot resolve."#
|
||||
"LLM rate limited during respond_with_tools, backing off"
|
||||
);
|
||||
if consecutive_rate_limits >= MAX_CONSECUTIVE_RATE_LIMITS {
|
||||
self.mark_stuck("Persistent rate limiting").await?;
|
||||
self.mark_failed("Persistent rate limiting: exceeded retry limit")
|
||||
.await?;
|
||||
return Ok(());
|
||||
}
|
||||
self.log_event(
|
||||
@@ -482,6 +486,20 @@ Report when the job is complete or if you encounter issues you cannot resolve."#
|
||||
Err(e) => return Err(e.into()),
|
||||
};
|
||||
|
||||
// Track token usage from LLM call against the job budget.
|
||||
// NOTE: select_tools() also makes LLM calls but doesn't expose
|
||||
// TokenUsage; only respond_with_tools() usage is tracked here.
|
||||
let total_tokens = respond_output.usage.total() as u64;
|
||||
if total_tokens > 0
|
||||
&& let Err(msg) = self
|
||||
.context_manager()
|
||||
.update_context(self.job_id, |ctx| ctx.add_tokens(total_tokens))
|
||||
.await?
|
||||
{
|
||||
self.mark_failed(&msg).await?;
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
match respond_output.result {
|
||||
RespondResult::Text(response) => {
|
||||
// Check for explicit completion phrases. Use word-boundary
|
||||
@@ -576,37 +594,54 @@ Report when the job is complete or if you encounter issues you cannot resolve."#
|
||||
}
|
||||
}
|
||||
}
|
||||
} else if selections.len() == 1 {
|
||||
consecutive_tool_intent_nudges = 0;
|
||||
// Single tool: execute directly
|
||||
let selection = &selections[0];
|
||||
tracing::debug!(
|
||||
"Job {} selecting tool: {} - {}",
|
||||
self.job_id,
|
||||
selection.tool_name,
|
||||
selection.reasoning
|
||||
);
|
||||
|
||||
let result = self
|
||||
.execute_tool(&selection.tool_name, &selection.parameters)
|
||||
.await;
|
||||
|
||||
self.process_tool_result(reason_ctx, selection, result)
|
||||
.await?;
|
||||
} else {
|
||||
// Multiple tools: execute in parallel
|
||||
tracing::debug!(
|
||||
"Job {} executing {} tools in parallel",
|
||||
self.job_id,
|
||||
selections.len()
|
||||
);
|
||||
consecutive_tool_intent_nudges = 0;
|
||||
|
||||
let results = self.execute_tools_parallel(&selections).await;
|
||||
// Record the assistant tool_calls message so that tool_result
|
||||
// messages have a matching parent (prevents orphaned rewrites).
|
||||
let tool_calls: Vec<ToolCall> = selections
|
||||
.iter()
|
||||
.map(|s| ToolCall {
|
||||
id: s.tool_call_id.clone(),
|
||||
name: s.tool_name.clone(),
|
||||
arguments: s.parameters.clone(),
|
||||
})
|
||||
.collect();
|
||||
reason_ctx
|
||||
.messages
|
||||
.push(ChatMessage::assistant_with_tool_calls(None, tool_calls));
|
||||
|
||||
// Process all results
|
||||
for (selection, result) in selections.iter().zip(results) {
|
||||
self.process_tool_result(reason_ctx, selection, result.result)
|
||||
if selections.len() == 1 {
|
||||
// Single tool: execute directly
|
||||
let selection = &selections[0];
|
||||
tracing::debug!(
|
||||
"Job {} selecting tool: {} - {}",
|
||||
self.job_id,
|
||||
selection.tool_name,
|
||||
selection.reasoning
|
||||
);
|
||||
|
||||
let result = self
|
||||
.execute_tool(&selection.tool_name, &selection.parameters)
|
||||
.await;
|
||||
|
||||
self.process_tool_result(reason_ctx, selection, result)
|
||||
.await?;
|
||||
} else {
|
||||
// Multiple tools: execute in parallel
|
||||
tracing::debug!(
|
||||
"Job {} executing {} tools in parallel",
|
||||
self.job_id,
|
||||
selections.len()
|
||||
);
|
||||
|
||||
let results = self.execute_tools_parallel(&selections).await;
|
||||
|
||||
// Process all results
|
||||
for (selection, result) in selections.iter().zip(results) {
|
||||
self.process_tool_result(reason_ctx, selection, result.result)
|
||||
.await?;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1087,11 +1122,6 @@ Report when the job is complete or if you encounter issues you cannot resolve."#
|
||||
action.reasoning
|
||||
);
|
||||
|
||||
// Execute the planned tool
|
||||
let result = self
|
||||
.execute_tool(&action.tool_name, &action.parameters)
|
||||
.await;
|
||||
|
||||
// Create a synthetic ToolSelection for process_tool_result.
|
||||
// Plan actions don't originate from an LLM tool_call response so
|
||||
// there is no real tool_call_id; generate a unique one.
|
||||
@@ -1103,6 +1133,24 @@ Report when the job is complete or if you encounter issues you cannot resolve."#
|
||||
tool_call_id: format!("plan_{}_{}", self.job_id, i),
|
||||
};
|
||||
|
||||
// Record the assistant tool_calls message so that the tool_result
|
||||
// has a matching parent (prevents orphaned rewrites).
|
||||
reason_ctx
|
||||
.messages
|
||||
.push(ChatMessage::assistant_with_tool_calls(
|
||||
None,
|
||||
vec![ToolCall {
|
||||
id: selection.tool_call_id.clone(),
|
||||
name: selection.tool_name.clone(),
|
||||
arguments: selection.parameters.clone(),
|
||||
}],
|
||||
));
|
||||
|
||||
// Execute the planned tool
|
||||
let result = self
|
||||
.execute_tool(&action.tool_name, &action.parameters)
|
||||
.await;
|
||||
|
||||
// Process the result
|
||||
let completed = self
|
||||
.process_tool_result(reason_ctx, &selection, result)
|
||||
@@ -1731,4 +1779,84 @@ mod tests {
|
||||
"Always tool should be allowed with permission"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_token_budget_exceeded_fails_job() {
|
||||
let worker = make_worker(vec![]).await;
|
||||
|
||||
// Transition to InProgress (required for mark_failed)
|
||||
worker
|
||||
.context_manager()
|
||||
.update_context(worker.job_id, |ctx| {
|
||||
ctx.transition_to(JobState::InProgress, None)
|
||||
})
|
||||
.await
|
||||
.unwrap()
|
||||
.unwrap();
|
||||
|
||||
// Set a token budget
|
||||
worker
|
||||
.context_manager()
|
||||
.update_context(worker.job_id, |ctx| {
|
||||
ctx.max_tokens = 100;
|
||||
})
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
// Simulate adding tokens that exceed the budget
|
||||
let budget_result = worker
|
||||
.context_manager()
|
||||
.update_context(worker.job_id, |ctx| ctx.add_tokens(200))
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
assert!(
|
||||
budget_result.is_err(),
|
||||
"Should return error when token budget exceeded"
|
||||
);
|
||||
|
||||
// Verify that mark_failed transitions job to Failed
|
||||
worker
|
||||
.mark_failed(&budget_result.unwrap_err())
|
||||
.await
|
||||
.unwrap();
|
||||
let ctx = worker
|
||||
.context_manager()
|
||||
.get_context(worker.job_id)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(ctx.state, JobState::Failed);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_iteration_cap_marks_failed_not_stuck() {
|
||||
let worker = make_worker(vec![]).await;
|
||||
|
||||
// Transition to InProgress (required for mark_failed)
|
||||
worker
|
||||
.context_manager()
|
||||
.update_context(worker.job_id, |ctx| {
|
||||
ctx.transition_to(JobState::InProgress, None)
|
||||
})
|
||||
.await
|
||||
.unwrap()
|
||||
.unwrap();
|
||||
|
||||
// Simulate what the execution loop does when max_iterations is exceeded
|
||||
worker
|
||||
.mark_failed("Maximum iterations exceeded: job hit the iteration cap")
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
let ctx = worker
|
||||
.context_manager()
|
||||
.get_context(worker.job_id)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(
|
||||
ctx.state,
|
||||
JobState::Failed,
|
||||
"Iteration cap should transition to Failed, not Stuck"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
+247
-93
@@ -21,7 +21,7 @@ use crate::secrets::SecretsStore;
|
||||
use crate::skills::SkillRegistry;
|
||||
use crate::skills::catalog::SkillCatalog;
|
||||
use crate::tools::ToolRegistry;
|
||||
use crate::tools::mcp::McpSessionManager;
|
||||
use crate::tools::mcp::{McpProcessManager, McpSessionManager};
|
||||
use crate::tools::wasm::SharedCredentialRegistry;
|
||||
use crate::tools::wasm::WasmToolRuntime;
|
||||
use crate::workspace::{EmbeddingProvider, Workspace};
|
||||
@@ -41,6 +41,7 @@ pub struct AppComponents {
|
||||
pub workspace: Option<Arc<Workspace>>,
|
||||
pub extension_manager: Option<Arc<ExtensionManager>>,
|
||||
pub mcp_session_manager: Arc<McpSessionManager>,
|
||||
pub mcp_process_manager: Arc<McpProcessManager>,
|
||||
pub wasm_tool_runtime: Option<Arc<WasmToolRuntime>>,
|
||||
pub log_broadcaster: Arc<LogBroadcaster>,
|
||||
pub context_manager: Arc<ContextManager>,
|
||||
@@ -244,11 +245,31 @@ impl AppBuilder {
|
||||
let master_key = match self.config.secrets.master_key() {
|
||||
Some(k) => k,
|
||||
None => {
|
||||
// No secrets DB available, but we can still load tokens from
|
||||
// OS credential stores (e.g., Anthropic OAuth via Claude Code's
|
||||
// macOS Keychain / Linux ~/.claude/.credentials.json).
|
||||
crate::config::inject_os_credentials();
|
||||
|
||||
// Consume unused handles
|
||||
#[cfg(feature = "libsql")]
|
||||
{
|
||||
self.libsql_db.take();
|
||||
}
|
||||
|
||||
// Re-resolve only the LLM config with OS credentials.
|
||||
let store: Option<&(dyn crate::db::SettingsStore + Sync)> =
|
||||
self.db.as_ref().map(|db| db.as_ref() as _);
|
||||
let toml_path = self.toml_path.as_deref();
|
||||
if let Err(e) = self
|
||||
.config
|
||||
.re_resolve_llm(store, "default", toml_path)
|
||||
.await
|
||||
{
|
||||
tracing::warn!(
|
||||
"Failed to re-resolve LLM config after OS credential injection: {e}"
|
||||
);
|
||||
}
|
||||
|
||||
return Ok(());
|
||||
}
|
||||
};
|
||||
@@ -291,18 +312,16 @@ impl AppBuilder {
|
||||
// Inject LLM API keys from encrypted storage
|
||||
crate::config::inject_llm_keys_from_secrets(secrets.as_ref(), "default").await;
|
||||
|
||||
// Re-resolve config with newly available keys
|
||||
if let Some(ref db) = self.db {
|
||||
let toml_path = self.toml_path.as_deref();
|
||||
match Config::from_db_with_toml(db.as_ref(), "default", toml_path).await {
|
||||
Ok(refreshed) => {
|
||||
self.config = refreshed;
|
||||
tracing::debug!("LlmConfig re-resolved after secret injection");
|
||||
}
|
||||
Err(e) => {
|
||||
tracing::warn!("Failed to re-resolve config after secret injection: {}", e);
|
||||
}
|
||||
}
|
||||
// Re-resolve only the LLM config with newly available keys.
|
||||
let store: Option<&(dyn crate::db::SettingsStore + Sync)> =
|
||||
self.db.as_ref().map(|db| db.as_ref() as _);
|
||||
let toml_path = self.toml_path.as_deref();
|
||||
if let Err(e) = self
|
||||
.config
|
||||
.re_resolve_llm(store, "default", toml_path)
|
||||
.await
|
||||
{
|
||||
tracing::warn!("Failed to re-resolve LLM config after secret injection: {e}");
|
||||
}
|
||||
}
|
||||
|
||||
@@ -315,7 +334,7 @@ impl AppBuilder {
|
||||
/// Delegates to `build_provider_chain` which applies all decorators
|
||||
/// (retry, smart routing, failover, circuit breaker, response cache).
|
||||
#[allow(clippy::type_complexity)]
|
||||
pub fn init_llm(
|
||||
pub async fn init_llm(
|
||||
&self,
|
||||
) -> Result<
|
||||
(
|
||||
@@ -326,7 +345,7 @@ impl AppBuilder {
|
||||
anyhow::Error,
|
||||
> {
|
||||
let (llm, cheap_llm, recording_handle) =
|
||||
crate::llm::build_provider_chain(&self.config.llm, self.session.clone())?;
|
||||
crate::llm::build_provider_chain(&self.config.llm, self.session.clone()).await?;
|
||||
Ok((llm, cheap_llm, recording_handle))
|
||||
}
|
||||
|
||||
@@ -381,16 +400,55 @@ impl AppBuilder {
|
||||
None
|
||||
};
|
||||
|
||||
// Register image/vision tools if we have a workspace and LLM API credentials
|
||||
if workspace.is_some() {
|
||||
let (api_base, api_key_opt) = if let Some(ref provider) = self.config.llm.provider {
|
||||
(
|
||||
provider.base_url.clone(),
|
||||
provider.api_key.as_ref().map(|s| {
|
||||
use secrecy::ExposeSecret;
|
||||
s.expose_secret().to_string()
|
||||
}),
|
||||
)
|
||||
} else {
|
||||
(
|
||||
self.config.llm.nearai.base_url.clone(),
|
||||
self.config.llm.nearai.api_key.as_ref().map(|s| {
|
||||
use secrecy::ExposeSecret;
|
||||
s.expose_secret().to_string()
|
||||
}),
|
||||
)
|
||||
};
|
||||
|
||||
if let Some(api_key) = api_key_opt {
|
||||
// Check for image generation models
|
||||
let model_name = self
|
||||
.config
|
||||
.llm
|
||||
.provider
|
||||
.as_ref()
|
||||
.map(|p| p.model.clone())
|
||||
.unwrap_or_else(|| self.config.llm.nearai.model.clone());
|
||||
let models = vec![model_name.clone()];
|
||||
let gen_model = crate::llm::image_models::suggest_image_model(&models)
|
||||
.unwrap_or("flux-1.1-pro")
|
||||
.to_string();
|
||||
tools.register_image_tools(api_base.clone(), api_key.clone(), gen_model, None);
|
||||
|
||||
// Check for vision models
|
||||
let vision_model = crate::llm::vision_models::suggest_vision_model(&models)
|
||||
.unwrap_or(&model_name)
|
||||
.to_string();
|
||||
tools.register_vision_tools(api_base, api_key, vision_model, None);
|
||||
}
|
||||
}
|
||||
|
||||
// Register builder tool if enabled
|
||||
if self.config.builder.enabled
|
||||
&& (self.config.agent.allow_local_tools || !self.config.sandbox.enabled)
|
||||
{
|
||||
tools
|
||||
.register_builder_tool(
|
||||
llm.clone(),
|
||||
safety.clone(),
|
||||
Some(self.config.builder.to_builder_config()),
|
||||
)
|
||||
.register_builder_tool(llm.clone(), Some(self.config.builder.to_builder_config()))
|
||||
.await;
|
||||
tracing::info!("Builder mode enabled");
|
||||
}
|
||||
@@ -406,6 +464,7 @@ impl AppBuilder {
|
||||
) -> Result<
|
||||
(
|
||||
Arc<McpSessionManager>,
|
||||
Arc<McpProcessManager>,
|
||||
Option<Arc<WasmToolRuntime>>,
|
||||
Option<Arc<ExtensionManager>>,
|
||||
Vec<crate::extensions::RegistryEntry>,
|
||||
@@ -413,10 +472,13 @@ impl AppBuilder {
|
||||
),
|
||||
anyhow::Error,
|
||||
> {
|
||||
use crate::tools::mcp::{McpClient, config::load_mcp_servers_from_db, is_authenticated};
|
||||
use crate::tools::mcp::{
|
||||
McpClient, McpTransport, config::load_mcp_servers_from_db, is_authenticated,
|
||||
};
|
||||
use crate::tools::wasm::{WasmToolLoader, load_dev_tools};
|
||||
|
||||
let mcp_session_manager = Arc::new(McpSessionManager::new());
|
||||
let mcp_process_manager = Arc::new(McpProcessManager::new());
|
||||
|
||||
// Create WASM tool runtime eagerly so extensions installed after startup
|
||||
// (e.g. via the web UI) can still be activated. The tools directory is only
|
||||
@@ -492,97 +554,175 @@ impl AppBuilder {
|
||||
let db = self.db.clone();
|
||||
let tools = Arc::clone(tools);
|
||||
let mcp_sm = Arc::clone(&mcp_session_manager);
|
||||
let pm = Arc::clone(&mcp_process_manager);
|
||||
async move {
|
||||
if let Some(ref secrets) = secrets_store {
|
||||
let servers_result = if let Some(ref d) = db {
|
||||
load_mcp_servers_from_db(d.as_ref(), "default").await
|
||||
} else {
|
||||
crate::tools::mcp::config::load_mcp_servers().await
|
||||
};
|
||||
match servers_result {
|
||||
Ok(servers) => {
|
||||
let enabled: Vec<_> = servers.enabled_servers().cloned().collect();
|
||||
if !enabled.is_empty() {
|
||||
tracing::info!(
|
||||
"Loading {} configured MCP server(s)...",
|
||||
enabled.len()
|
||||
);
|
||||
}
|
||||
let servers_result = if let Some(ref d) = db {
|
||||
load_mcp_servers_from_db(d.as_ref(), "default").await
|
||||
} else {
|
||||
crate::tools::mcp::config::load_mcp_servers().await
|
||||
};
|
||||
match servers_result {
|
||||
Ok(servers) => {
|
||||
let enabled: Vec<_> = servers.enabled_servers().cloned().collect();
|
||||
if !enabled.is_empty() {
|
||||
tracing::info!("Loading {} configured MCP server(s)...", enabled.len());
|
||||
}
|
||||
|
||||
let mut join_set = tokio::task::JoinSet::new();
|
||||
for server in enabled {
|
||||
let mcp_sm = Arc::clone(&mcp_sm);
|
||||
let secrets = Arc::clone(secrets);
|
||||
let tools = Arc::clone(&tools);
|
||||
let mut join_set = tokio::task::JoinSet::new();
|
||||
for server in enabled {
|
||||
let mcp_sm = Arc::clone(&mcp_sm);
|
||||
let secrets = secrets_store.clone();
|
||||
let tools = Arc::clone(&tools);
|
||||
let pm = Arc::clone(&pm);
|
||||
|
||||
join_set.spawn(async move {
|
||||
let server_name = server.name.clone();
|
||||
let has_tokens =
|
||||
is_authenticated(&server, &secrets, "default").await;
|
||||
join_set.spawn(async move {
|
||||
let server_name = server.name.clone();
|
||||
|
||||
let client = if has_tokens || server.requires_auth() {
|
||||
McpClient::new_authenticated(
|
||||
server, mcp_sm, secrets, "default",
|
||||
)
|
||||
} else {
|
||||
McpClient::new_with_name(&server_name, &server.url)
|
||||
};
|
||||
|
||||
match client.list_tools().await {
|
||||
Ok(mcp_tools) => {
|
||||
let tool_count = mcp_tools.len();
|
||||
match client.create_tools().await {
|
||||
Ok(tool_impls) => {
|
||||
for tool in tool_impls {
|
||||
tools.register(tool).await;
|
||||
}
|
||||
tracing::info!(
|
||||
"Loaded {} tools from MCP server '{}'",
|
||||
tool_count,
|
||||
server_name
|
||||
);
|
||||
}
|
||||
Err(e) => {
|
||||
tracing::warn!(
|
||||
"Failed to create tools from MCP server '{}': {}",
|
||||
server_name,
|
||||
e
|
||||
);
|
||||
}
|
||||
let client: McpClient = match server.effective_transport() {
|
||||
crate::tools::mcp::config::EffectiveTransport::Stdio {
|
||||
command,
|
||||
args,
|
||||
env,
|
||||
} => {
|
||||
match pm
|
||||
.spawn_stdio(
|
||||
&server_name,
|
||||
command,
|
||||
args.to_vec(),
|
||||
env.clone(),
|
||||
)
|
||||
.await
|
||||
{
|
||||
Ok(transport) => McpClient::new_with_transport(
|
||||
&server_name,
|
||||
transport as Arc<dyn McpTransport>,
|
||||
None,
|
||||
secrets,
|
||||
"default",
|
||||
Some(server),
|
||||
),
|
||||
Err(e) => {
|
||||
tracing::warn!(
|
||||
"Failed to spawn stdio MCP server '{}': {}",
|
||||
server_name,
|
||||
e
|
||||
);
|
||||
return;
|
||||
}
|
||||
}
|
||||
Err(e) => {
|
||||
let err_str = e.to_string();
|
||||
if err_str.contains("401")
|
||||
|| err_str.contains("authentication")
|
||||
{
|
||||
}
|
||||
#[cfg(unix)]
|
||||
crate::tools::mcp::config::EffectiveTransport::Unix {
|
||||
socket_path,
|
||||
} => {
|
||||
match crate::tools::mcp::unix_transport::UnixMcpTransport::connect(
|
||||
&server_name,
|
||||
socket_path,
|
||||
)
|
||||
.await
|
||||
{
|
||||
Ok(transport) => McpClient::new_with_transport(
|
||||
&server_name,
|
||||
Arc::new(transport) as Arc<dyn McpTransport>,
|
||||
None,
|
||||
secrets,
|
||||
"default",
|
||||
Some(server),
|
||||
),
|
||||
Err(e) => {
|
||||
tracing::warn!(
|
||||
"MCP server '{}' requires authentication. \
|
||||
Run: ironclaw mcp auth {}",
|
||||
"Failed to connect to Unix MCP server '{}': {}",
|
||||
server_name,
|
||||
e
|
||||
);
|
||||
return;
|
||||
}
|
||||
}
|
||||
}
|
||||
#[cfg(not(unix))]
|
||||
crate::tools::mcp::config::EffectiveTransport::Unix { .. } => {
|
||||
tracing::warn!(
|
||||
"Unix socket transport is not supported on this platform (server '{}')",
|
||||
server_name
|
||||
);
|
||||
return;
|
||||
}
|
||||
crate::tools::mcp::config::EffectiveTransport::Http => {
|
||||
if let Some(ref secrets) = secrets {
|
||||
let has_tokens =
|
||||
is_authenticated(&server, secrets, "default")
|
||||
.await;
|
||||
|
||||
if has_tokens || server.requires_auth() {
|
||||
McpClient::new_authenticated(
|
||||
server,
|
||||
Arc::clone(&mcp_sm),
|
||||
Arc::clone(secrets),
|
||||
"default",
|
||||
)
|
||||
} else {
|
||||
McpClient::new_with_config(server)
|
||||
}
|
||||
} else {
|
||||
McpClient::new_with_config(server)
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
match client.list_tools().await {
|
||||
Ok(mcp_tools) => {
|
||||
let tool_count = mcp_tools.len();
|
||||
match client.create_tools().await {
|
||||
Ok(tool_impls) => {
|
||||
for tool in tool_impls {
|
||||
tools.register(tool).await;
|
||||
}
|
||||
tracing::info!(
|
||||
"Loaded {} tools from MCP server '{}'",
|
||||
tool_count,
|
||||
server_name
|
||||
);
|
||||
} else {
|
||||
}
|
||||
Err(e) => {
|
||||
tracing::warn!(
|
||||
"Failed to connect to MCP server '{}': {}",
|
||||
"Failed to create tools from MCP server '{}': {}",
|
||||
server_name,
|
||||
e
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
while let Some(result) = join_set.join_next().await {
|
||||
if let Err(e) = result {
|
||||
tracing::warn!("MCP server loading task panicked: {}", e);
|
||||
Err(e) => {
|
||||
let err_str = e.to_string();
|
||||
if err_str.contains("401")
|
||||
|| err_str.contains("authentication")
|
||||
{
|
||||
tracing::warn!(
|
||||
"MCP server '{}' requires authentication. \
|
||||
Run: ironclaw mcp auth {}",
|
||||
server_name,
|
||||
server_name
|
||||
);
|
||||
} else {
|
||||
tracing::warn!(
|
||||
"Failed to connect to MCP server '{}': {}",
|
||||
server_name,
|
||||
e
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
while let Some(result) = join_set.join_next().await {
|
||||
if let Err(e) = result {
|
||||
tracing::warn!("MCP server loading task panicked: {}", e);
|
||||
}
|
||||
}
|
||||
Err(e) => {
|
||||
tracing::debug!("No MCP servers configured ({})", e);
|
||||
}
|
||||
}
|
||||
Err(e) => {
|
||||
tracing::debug!("No MCP servers configured ({})", e);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -653,6 +793,7 @@ impl AppBuilder {
|
||||
|
||||
Ok((
|
||||
mcp_session_manager,
|
||||
mcp_process_manager,
|
||||
wasm_tool_runtime,
|
||||
extension_manager,
|
||||
catalog_entries,
|
||||
@@ -665,10 +806,21 @@ impl AppBuilder {
|
||||
self.init_database().await?;
|
||||
self.init_secrets().await?;
|
||||
|
||||
// Post-init validation: if a non-nearai backend was selected but
|
||||
// credentials were never resolved (deferred resolution found no keys),
|
||||
// fail early with a clear error instead of a confusing runtime failure.
|
||||
if self.config.llm.backend != "nearai" && self.config.llm.provider.is_none() {
|
||||
let backend = &self.config.llm.backend;
|
||||
anyhow::bail!(
|
||||
"LLM_BACKEND={backend} is configured but no credentials were found. \
|
||||
Set the appropriate API key environment variable or run the setup wizard."
|
||||
);
|
||||
}
|
||||
|
||||
let (llm, cheap_llm, recording_handle) = if let Some(llm) = self.llm_override.take() {
|
||||
(llm, None, None)
|
||||
} else {
|
||||
self.init_llm()?
|
||||
self.init_llm().await?
|
||||
};
|
||||
let (safety, tools, embeddings, workspace) = self.init_tools(&llm).await?;
|
||||
|
||||
@@ -677,6 +829,7 @@ impl AppBuilder {
|
||||
|
||||
let (
|
||||
mcp_session_manager,
|
||||
mcp_process_manager,
|
||||
wasm_tool_runtime,
|
||||
extension_manager,
|
||||
catalog_entries,
|
||||
@@ -774,6 +927,7 @@ impl AppBuilder {
|
||||
workspace,
|
||||
extension_manager,
|
||||
mcp_session_manager,
|
||||
mcp_process_manager,
|
||||
wasm_tool_runtime,
|
||||
log_broadcaster: self.log_broadcaster,
|
||||
context_manager,
|
||||
|
||||
@@ -414,10 +414,103 @@ pub enum MigrationError {
|
||||
Io(String),
|
||||
}
|
||||
|
||||
// ── PID Lock ──────────────────────────────────────────────────────────────
|
||||
|
||||
/// Path to the PID lock file: `~/.ironclaw/ironclaw.pid`.
|
||||
pub fn pid_lock_path() -> PathBuf {
|
||||
ironclaw_base_dir().join("ironclaw.pid")
|
||||
}
|
||||
|
||||
/// A PID-based lock that prevents multiple IronClaw instances from running
|
||||
/// simultaneously.
|
||||
///
|
||||
/// Uses `fs4::try_lock_exclusive()` for atomic locking (no TOCTOU race),
|
||||
/// then writes the current PID into the locked file for diagnostics.
|
||||
/// The OS-level lock is held for the lifetime of this struct and
|
||||
/// automatically released on drop (along with the PID file cleanup).
|
||||
#[derive(Debug)]
|
||||
pub struct PidLock {
|
||||
path: PathBuf,
|
||||
/// Held open to maintain the OS-level exclusive lock.
|
||||
_file: std::fs::File,
|
||||
}
|
||||
|
||||
/// Errors from PID lock acquisition.
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum PidLockError {
|
||||
#[error("Another IronClaw instance is already running (PID {pid})")]
|
||||
AlreadyRunning { pid: u32 },
|
||||
#[error("Failed to acquire PID lock: {0}")]
|
||||
Io(#[from] std::io::Error),
|
||||
}
|
||||
|
||||
impl PidLock {
|
||||
/// Try to acquire the PID lock.
|
||||
///
|
||||
/// Uses an exclusive file lock (`flock`/`LockFileEx`) so that two
|
||||
/// concurrent processes cannot both acquire the lock — no TOCTOU race.
|
||||
/// If the lock file exists but the holding process is gone (stale),
|
||||
/// the lock is reclaimed automatically by the OS.
|
||||
pub fn acquire() -> Result<Self, PidLockError> {
|
||||
Self::acquire_at(pid_lock_path())
|
||||
}
|
||||
|
||||
/// Acquire at a specific path (for testing).
|
||||
fn acquire_at(path: PathBuf) -> Result<Self, PidLockError> {
|
||||
use fs4::FileExt;
|
||||
use std::fs::OpenOptions;
|
||||
use std::io::Write;
|
||||
|
||||
// Ensure parent directory exists
|
||||
if let Some(parent) = path.parent() {
|
||||
std::fs::create_dir_all(parent)?;
|
||||
}
|
||||
|
||||
// Open (or create) the lock file
|
||||
let mut file = OpenOptions::new()
|
||||
.read(true)
|
||||
.write(true)
|
||||
.create(true)
|
||||
.truncate(false)
|
||||
.open(&path)?;
|
||||
|
||||
// Try non-blocking exclusive lock — if another process holds it,
|
||||
// this fails immediately instead of blocking.
|
||||
if let Err(e) = file.try_lock_exclusive() {
|
||||
if e.kind() == std::io::ErrorKind::WouldBlock {
|
||||
// Lock held by another process — read its PID for the error message
|
||||
let pid = std::fs::read_to_string(&path)
|
||||
.ok()
|
||||
.and_then(|s| s.trim().parse::<u32>().ok())
|
||||
.unwrap_or(0);
|
||||
return Err(PidLockError::AlreadyRunning { pid });
|
||||
}
|
||||
// Other errors (permissions, unsupported filesystem, etc.)
|
||||
return Err(PidLockError::Io(e));
|
||||
}
|
||||
|
||||
// We hold the exclusive lock — write our PID
|
||||
file.set_len(0)?; // truncate
|
||||
write!(file, "{}", std::process::id())?;
|
||||
|
||||
Ok(PidLock { path, _file: file })
|
||||
}
|
||||
}
|
||||
|
||||
impl Drop for PidLock {
|
||||
fn drop(&mut self) {
|
||||
// Remove the PID file; the OS-level lock is released when _file is dropped.
|
||||
let _ = std::fs::remove_file(&self.path);
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use std::process::Command;
|
||||
use std::sync::Mutex;
|
||||
use std::thread;
|
||||
use std::time::{Duration, Instant};
|
||||
use tempfile::tempdir;
|
||||
|
||||
static ENV_MUTEX: Mutex<()> = Mutex::new(());
|
||||
@@ -986,4 +1079,162 @@ INJECTED="pwned"#;
|
||||
unsafe { std::env::remove_var("IRONCLAW_BASE_DIR") };
|
||||
}
|
||||
}
|
||||
|
||||
// ── PID Lock tests ───────────────────────────────────────────────
|
||||
|
||||
#[test]
|
||||
fn test_pid_lock_acquire_and_drop() {
|
||||
let dir = tempdir().unwrap();
|
||||
let pid_path = dir.path().join("ironclaw.pid");
|
||||
|
||||
// Acquire lock
|
||||
let lock = PidLock::acquire_at(pid_path.clone()).unwrap();
|
||||
assert!(pid_path.exists());
|
||||
|
||||
// PID file should contain our PID
|
||||
let contents = std::fs::read_to_string(&pid_path).unwrap();
|
||||
assert_eq!(contents.trim().parse::<u32>().unwrap(), std::process::id());
|
||||
|
||||
// Drop should remove the file
|
||||
drop(lock);
|
||||
assert!(!pid_path.exists());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_pid_lock_rejects_second_acquire() {
|
||||
let dir = tempdir().unwrap();
|
||||
let pid_path = dir.path().join("ironclaw.pid");
|
||||
|
||||
// First lock succeeds
|
||||
let _lock1 = PidLock::acquire_at(pid_path.clone()).unwrap();
|
||||
|
||||
// Second acquire on same file must fail (exclusive flock held)
|
||||
let result = PidLock::acquire_at(pid_path.clone());
|
||||
assert!(result.is_err());
|
||||
match result.unwrap_err() {
|
||||
PidLockError::AlreadyRunning { pid } => {
|
||||
assert_eq!(pid, std::process::id());
|
||||
}
|
||||
other => panic!("expected AlreadyRunning, got: {}", other),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_pid_lock_reclaims_after_drop() {
|
||||
let dir = tempdir().unwrap();
|
||||
let pid_path = dir.path().join("ironclaw.pid");
|
||||
|
||||
// Acquire and release
|
||||
let lock = PidLock::acquire_at(pid_path.clone()).unwrap();
|
||||
drop(lock);
|
||||
|
||||
// Should succeed — OS lock was released on drop
|
||||
let lock2 = PidLock::acquire_at(pid_path).unwrap();
|
||||
drop(lock2);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_pid_lock_reclaims_stale_file_without_flock() {
|
||||
let dir = tempdir().unwrap();
|
||||
let pid_path = dir.path().join("ironclaw.pid");
|
||||
|
||||
// Write a stale PID file manually (no flock held)
|
||||
std::fs::write(&pid_path, "4294967294").unwrap();
|
||||
|
||||
// Should succeed because no OS lock is held on the file
|
||||
let lock = PidLock::acquire_at(pid_path.clone()).unwrap();
|
||||
let contents = std::fs::read_to_string(&pid_path).unwrap();
|
||||
assert_eq!(contents.trim().parse::<u32>().unwrap(), std::process::id());
|
||||
drop(lock);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_pid_lock_handles_corrupt_pid_file() {
|
||||
let dir = tempdir().unwrap();
|
||||
let pid_path = dir.path().join("ironclaw.pid");
|
||||
|
||||
// Write garbage (no flock held)
|
||||
std::fs::write(&pid_path, "not-a-number").unwrap();
|
||||
|
||||
// Should succeed — no OS lock held, file is reclaimed
|
||||
let lock = PidLock::acquire_at(pid_path).unwrap();
|
||||
drop(lock);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_pid_lock_creates_parent_dirs() {
|
||||
let dir = tempdir().unwrap();
|
||||
let pid_path = dir.path().join("nested").join("deep").join("ironclaw.pid");
|
||||
|
||||
let lock = PidLock::acquire_at(pid_path.clone()).unwrap();
|
||||
assert!(pid_path.exists());
|
||||
drop(lock);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_pid_lock_child_helper_holds_lock() {
|
||||
if std::env::var("IRONCLAW_PID_LOCK_CHILD").ok().as_deref() != Some("1") {
|
||||
return;
|
||||
}
|
||||
|
||||
let pid_path = PathBuf::from(
|
||||
std::env::var("IRONCLAW_PID_LOCK_PATH").expect("IRONCLAW_PID_LOCK_PATH missing"),
|
||||
);
|
||||
let hold_ms = std::env::var("IRONCLAW_PID_LOCK_HOLD_MS")
|
||||
.ok()
|
||||
.and_then(|s| s.parse::<u64>().ok())
|
||||
.unwrap_or(3000);
|
||||
|
||||
let _lock = PidLock::acquire_at(pid_path).expect("child failed to acquire pid lock");
|
||||
thread::sleep(Duration::from_millis(hold_ms));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_pid_lock_rejects_lock_held_by_other_process() {
|
||||
let dir = tempdir().unwrap();
|
||||
let pid_path = dir.path().join("ironclaw.pid");
|
||||
|
||||
let current_exe = std::env::current_exe().unwrap();
|
||||
let mut child = Command::new(current_exe)
|
||||
.args([
|
||||
"--exact",
|
||||
"bootstrap::tests::test_pid_lock_child_helper_holds_lock",
|
||||
"--nocapture",
|
||||
"--test-threads=1",
|
||||
])
|
||||
.env("IRONCLAW_PID_LOCK_CHILD", "1")
|
||||
.env("IRONCLAW_PID_LOCK_PATH", pid_path.display().to_string())
|
||||
.env("IRONCLAW_PID_LOCK_HOLD_MS", "3000")
|
||||
.spawn()
|
||||
.unwrap();
|
||||
|
||||
let started = Instant::now();
|
||||
while started.elapsed() < Duration::from_secs(2) {
|
||||
if pid_path.exists() {
|
||||
break;
|
||||
}
|
||||
if let Some(status) = child.try_wait().unwrap() {
|
||||
panic!("child exited before acquiring lock: {}", status);
|
||||
}
|
||||
thread::sleep(Duration::from_millis(20));
|
||||
}
|
||||
assert!(
|
||||
pid_path.exists(),
|
||||
"child did not create lock file in time: {}",
|
||||
pid_path.display()
|
||||
);
|
||||
|
||||
let result = PidLock::acquire_at(pid_path.clone());
|
||||
match result.unwrap_err() {
|
||||
PidLockError::AlreadyRunning { .. } => {}
|
||||
other => panic!("expected AlreadyRunning, got: {}", other),
|
||||
}
|
||||
|
||||
let status = child.wait().unwrap();
|
||||
assert!(status.success(), "child process failed: {}", status);
|
||||
|
||||
// After the child exits, lock should be released and reacquirable.
|
||||
let lock = PidLock::acquire_at(pid_path).unwrap();
|
||||
drop(lock);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -79,6 +79,8 @@ pub struct IncomingMessage {
|
||||
pub received_at: DateTime<Utc>,
|
||||
/// Channel-specific metadata.
|
||||
pub metadata: serde_json::Value,
|
||||
/// IANA timezone string from the client (e.g. "America/New_York").
|
||||
pub timezone: Option<String>,
|
||||
/// File or media attachments on this message.
|
||||
pub attachments: Vec<IncomingAttachment>,
|
||||
}
|
||||
@@ -99,6 +101,7 @@ impl IncomingMessage {
|
||||
thread_id: None,
|
||||
received_at: Utc::now(),
|
||||
metadata: serde_json::Value::Null,
|
||||
timezone: None,
|
||||
attachments: Vec::new(),
|
||||
}
|
||||
}
|
||||
@@ -121,6 +124,12 @@ impl IncomingMessage {
|
||||
self
|
||||
}
|
||||
|
||||
/// Set the client timezone.
|
||||
pub fn with_timezone(mut self, tz: impl Into<String>) -> Self {
|
||||
self.timezone = Some(tz.into());
|
||||
self
|
||||
}
|
||||
|
||||
/// Set attachments.
|
||||
pub fn with_attachments(mut self, attachments: Vec<IncomingAttachment>) -> Self {
|
||||
self.attachments = attachments;
|
||||
@@ -222,6 +231,13 @@ pub enum StatusUpdate {
|
||||
success: bool,
|
||||
message: String,
|
||||
},
|
||||
/// An image was generated by a tool.
|
||||
ImageGenerated {
|
||||
/// Base64 data URL of the generated image.
|
||||
data_url: String,
|
||||
/// Optional workspace path where the image was saved.
|
||||
path: Option<String>,
|
||||
},
|
||||
}
|
||||
|
||||
impl StatusUpdate {
|
||||
@@ -454,4 +470,10 @@ mod tests {
|
||||
panic!("expected ToolCompleted variant");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_incoming_message_with_timezone() {
|
||||
let msg = IncomingMessage::new("test", "user1", "hello").with_timezone("America/New_York");
|
||||
assert_eq!(msg.timezone.as_deref(), Some("America/New_York"));
|
||||
}
|
||||
}
|
||||
|
||||
+126
-6
@@ -17,7 +17,9 @@ use tokio::sync::{RwLock, mpsc, oneshot};
|
||||
use tokio_stream::wrappers::ReceiverStream;
|
||||
use uuid::Uuid;
|
||||
|
||||
use crate::channels::{Channel, IncomingMessage, MessageStream, OutgoingResponse};
|
||||
use crate::channels::{
|
||||
AttachmentKind, Channel, IncomingAttachment, IncomingMessage, MessageStream, OutgoingResponse,
|
||||
};
|
||||
use crate::config::HttpConfig;
|
||||
use crate::error::ChannelError;
|
||||
|
||||
@@ -46,8 +48,9 @@ struct RateLimitState {
|
||||
request_count: u32,
|
||||
}
|
||||
|
||||
/// Maximum JSON body size for webhook requests (64 KB).
|
||||
const MAX_BODY_BYTES: usize = 64 * 1024;
|
||||
/// Maximum JSON body size for webhook requests (15 MB, to support base64 image attachments
|
||||
/// with ~33% overhead from base64 encoding).
|
||||
const MAX_BODY_BYTES: usize = 15 * 1024 * 1024;
|
||||
|
||||
/// Maximum number of pending wait-for-response requests.
|
||||
const MAX_PENDING_RESPONSES: usize = 100;
|
||||
@@ -115,8 +118,34 @@ struct WebhookRequest {
|
||||
/// Whether to wait for a synchronous response.
|
||||
#[serde(default)]
|
||||
wait_for_response: bool,
|
||||
/// Optional file attachments (base64-encoded).
|
||||
#[serde(default)]
|
||||
attachments: Vec<AttachmentData>,
|
||||
}
|
||||
|
||||
/// A file attachment in a webhook request.
|
||||
#[derive(Debug, Deserialize)]
|
||||
struct AttachmentData {
|
||||
/// MIME type (e.g. "image/png", "application/pdf").
|
||||
mime_type: String,
|
||||
/// Optional filename.
|
||||
#[serde(default)]
|
||||
filename: Option<String>,
|
||||
/// Base64-encoded file data.
|
||||
#[serde(default)]
|
||||
data_base64: Option<String>,
|
||||
/// URL to fetch the file from (not downloaded server-side for SSRF prevention).
|
||||
#[serde(default)]
|
||||
url: Option<String>,
|
||||
}
|
||||
|
||||
/// Maximum size per attachment (5 MB decoded).
|
||||
const MAX_ATTACHMENT_BYTES: usize = 5 * 1024 * 1024;
|
||||
/// Maximum total attachment size (10 MB decoded).
|
||||
const MAX_TOTAL_ATTACHMENT_BYTES: usize = 10 * 1024 * 1024;
|
||||
/// Maximum number of attachments per request.
|
||||
const MAX_ATTACHMENTS: usize = 5;
|
||||
|
||||
#[derive(Debug, Serialize)]
|
||||
struct WebhookResponse {
|
||||
/// Message ID assigned to this request.
|
||||
@@ -211,15 +240,106 @@ async fn webhook_handler(
|
||||
);
|
||||
}
|
||||
|
||||
let msg = IncomingMessage::new("http", &state.user_id, &req.content).with_metadata(
|
||||
// Validate and decode attachments
|
||||
let attachments = if !req.attachments.is_empty() {
|
||||
if req.attachments.len() > MAX_ATTACHMENTS {
|
||||
return (
|
||||
StatusCode::BAD_REQUEST,
|
||||
Json(WebhookResponse {
|
||||
message_id: Uuid::nil(),
|
||||
status: "error".to_string(),
|
||||
response: Some(format!("Too many attachments (max {})", MAX_ATTACHMENTS)),
|
||||
}),
|
||||
);
|
||||
}
|
||||
|
||||
let mut decoded_attachments = Vec::new();
|
||||
let mut total_bytes: usize = 0;
|
||||
for att in &req.attachments {
|
||||
if let Some(ref b64) = att.data_base64 {
|
||||
use base64::Engine;
|
||||
let data = match base64::engine::general_purpose::STANDARD.decode(b64) {
|
||||
Ok(d) => d,
|
||||
Err(_) => {
|
||||
return (
|
||||
StatusCode::BAD_REQUEST,
|
||||
Json(WebhookResponse {
|
||||
message_id: Uuid::nil(),
|
||||
status: "error".to_string(),
|
||||
response: Some("Invalid base64 in attachment".to_string()),
|
||||
}),
|
||||
);
|
||||
}
|
||||
};
|
||||
if data.len() > MAX_ATTACHMENT_BYTES {
|
||||
return (
|
||||
StatusCode::PAYLOAD_TOO_LARGE,
|
||||
Json(WebhookResponse {
|
||||
message_id: Uuid::nil(),
|
||||
status: "error".to_string(),
|
||||
response: Some(format!(
|
||||
"Attachment too large (max {} bytes)",
|
||||
MAX_ATTACHMENT_BYTES
|
||||
)),
|
||||
}),
|
||||
);
|
||||
}
|
||||
total_bytes += data.len();
|
||||
if total_bytes > MAX_TOTAL_ATTACHMENT_BYTES {
|
||||
return (
|
||||
StatusCode::PAYLOAD_TOO_LARGE,
|
||||
Json(WebhookResponse {
|
||||
message_id: Uuid::nil(),
|
||||
status: "error".to_string(),
|
||||
response: Some("Total attachment size exceeds limit".to_string()),
|
||||
}),
|
||||
);
|
||||
}
|
||||
decoded_attachments.push(IncomingAttachment {
|
||||
id: Uuid::new_v4().to_string(),
|
||||
kind: AttachmentKind::from_mime_type(&att.mime_type),
|
||||
mime_type: att.mime_type.clone(),
|
||||
filename: att.filename.clone(),
|
||||
size_bytes: Some(data.len() as u64),
|
||||
source_url: None,
|
||||
storage_key: None,
|
||||
extracted_text: None,
|
||||
data,
|
||||
duration_secs: None,
|
||||
});
|
||||
} else if let Some(ref url) = att.url {
|
||||
// URL-only attachment: set source_url but don't download (SSRF prevention)
|
||||
decoded_attachments.push(IncomingAttachment {
|
||||
id: Uuid::new_v4().to_string(),
|
||||
kind: AttachmentKind::from_mime_type(&att.mime_type),
|
||||
mime_type: att.mime_type.clone(),
|
||||
filename: att.filename.clone(),
|
||||
size_bytes: None,
|
||||
source_url: Some(url.clone()),
|
||||
storage_key: None,
|
||||
extracted_text: None,
|
||||
data: Vec::new(),
|
||||
duration_secs: None,
|
||||
});
|
||||
}
|
||||
}
|
||||
decoded_attachments
|
||||
} else {
|
||||
Vec::new()
|
||||
};
|
||||
|
||||
let mut msg = IncomingMessage::new("http", &state.user_id, &req.content).with_metadata(
|
||||
serde_json::json!({
|
||||
"wait_for_response": req.wait_for_response,
|
||||
}),
|
||||
);
|
||||
|
||||
if !attachments.is_empty() {
|
||||
msg = msg.with_attachments(attachments);
|
||||
}
|
||||
|
||||
if let Some(thread_id) = &req.thread_id {
|
||||
let msg = msg.with_thread(thread_id);
|
||||
return process_message(state, msg, req.wait_for_response).await;
|
||||
msg = msg.with_thread(thread_id);
|
||||
}
|
||||
|
||||
process_message(state, msg, req.wait_for_response).await
|
||||
|
||||
+57
-9
@@ -18,7 +18,7 @@
|
||||
//! - `Esc` - Interrupt current operation
|
||||
|
||||
use std::borrow::Cow;
|
||||
use std::io::{self, Write};
|
||||
use std::io::{self, IsTerminal, Write};
|
||||
use std::sync::Arc;
|
||||
use std::sync::atomic::{AtomicBool, Ordering};
|
||||
|
||||
@@ -297,10 +297,15 @@ impl Channel for ReplChannel {
|
||||
let esc_interrupt_triggered_for_thread = Arc::new(AtomicBool::new(false));
|
||||
|
||||
std::thread::spawn(move || {
|
||||
let sys_tz = crate::timezone::detect_system_timezone().name().to_string();
|
||||
|
||||
// Single message mode: send it and return
|
||||
if let Some(msg) = single_message {
|
||||
let incoming = IncomingMessage::new("repl", "default", &msg);
|
||||
let incoming = IncomingMessage::new("repl", "default", &msg).with_timezone(&sys_tz);
|
||||
let _ = tx.blocking_send(incoming);
|
||||
// Ensure the agent exits after handling exactly one turn in -m mode,
|
||||
// even when other channels (gateway/http) are enabled.
|
||||
let _ = tx.blocking_send(IncomingMessage::new("repl", "default", "/quit"));
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -361,7 +366,8 @@ impl Channel for ReplChannel {
|
||||
"/quit" | "/exit" => {
|
||||
// Forward shutdown command so the agent loop exits even
|
||||
// when other channels (e.g. web gateway) are still active.
|
||||
let msg = IncomingMessage::new("repl", "default", "/quit");
|
||||
let msg = IncomingMessage::new("repl", "default", "/quit")
|
||||
.with_timezone(&sys_tz);
|
||||
let _ = tx.blocking_send(msg);
|
||||
break;
|
||||
}
|
||||
@@ -382,7 +388,8 @@ impl Channel for ReplChannel {
|
||||
_ => {}
|
||||
}
|
||||
|
||||
let msg = IncomingMessage::new("repl", "default", line);
|
||||
let msg =
|
||||
IncomingMessage::new("repl", "default", line).with_timezone(&sys_tz);
|
||||
if tx.blocking_send(msg).is_err() {
|
||||
break;
|
||||
}
|
||||
@@ -390,21 +397,29 @@ impl Channel for ReplChannel {
|
||||
Err(ReadlineError::Interrupted) => {
|
||||
if esc_interrupt_triggered_for_thread.swap(false, Ordering::Relaxed) {
|
||||
// Esc: interrupt current operation and keep REPL open.
|
||||
let msg = IncomingMessage::new("repl", "default", "/interrupt");
|
||||
let msg = IncomingMessage::new("repl", "default", "/interrupt")
|
||||
.with_timezone(&sys_tz);
|
||||
if tx.blocking_send(msg).is_err() {
|
||||
break;
|
||||
}
|
||||
} else {
|
||||
// Ctrl+C (VINTR): request graceful shutdown.
|
||||
let msg = IncomingMessage::new("repl", "default", "/quit");
|
||||
let msg = IncomingMessage::new("repl", "default", "/quit")
|
||||
.with_timezone(&sys_tz);
|
||||
let _ = tx.blocking_send(msg);
|
||||
break;
|
||||
}
|
||||
}
|
||||
Err(ReadlineError::Eof) => {
|
||||
// Ctrl+D: send /quit so the agent loop runs graceful shutdown
|
||||
let msg = IncomingMessage::new("repl", "default", "/quit");
|
||||
let _ = tx.blocking_send(msg);
|
||||
// Ctrl+D in interactive mode: graceful shutdown.
|
||||
// In daemon mode (stdin = /dev/null, no TTY), EOF arrives
|
||||
// immediately — just drop the REPL thread silently so other
|
||||
// channels (gateway, telegram, …) keep running.
|
||||
if std::io::stdin().is_terminal() {
|
||||
let msg = IncomingMessage::new("repl", "default", "/quit")
|
||||
.with_timezone(&sys_tz);
|
||||
let _ = tx.blocking_send(msg);
|
||||
}
|
||||
break;
|
||||
}
|
||||
Err(e) => {
|
||||
@@ -585,6 +600,13 @@ impl Channel for ReplChannel {
|
||||
eprintln!("\x1b[31m {extension_name}: {message}\x1b[0m");
|
||||
}
|
||||
}
|
||||
StatusUpdate::ImageGenerated { path, .. } => {
|
||||
if let Some(ref p) = path {
|
||||
eprintln!("\x1b[36m [image] {p}\x1b[0m");
|
||||
} else {
|
||||
eprintln!("\x1b[36m [image generated]\x1b[0m");
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
@@ -614,3 +636,29 @@ impl Channel for ReplChannel {
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use futures::StreamExt;
|
||||
|
||||
use super::*;
|
||||
|
||||
#[tokio::test]
|
||||
async fn single_message_mode_sends_message_then_quit() {
|
||||
let repl = ReplChannel::with_message("hi".to_string());
|
||||
let mut stream = repl.start().await.expect("repl start should succeed");
|
||||
|
||||
let first = stream.next().await.expect("first message missing");
|
||||
assert_eq!(first.channel, "repl");
|
||||
assert_eq!(first.content, "hi");
|
||||
|
||||
let second = stream.next().await.expect("quit message missing");
|
||||
assert_eq!(second.channel, "repl");
|
||||
assert_eq!(second.content, "/quit");
|
||||
|
||||
assert!(
|
||||
stream.next().await.is_none(),
|
||||
"stream should end after /quit"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -2809,6 +2809,14 @@ fn status_to_wit(status: &StatusUpdate, metadata: &serde_json::Value) -> wit_cha
|
||||
),
|
||||
metadata_json,
|
||||
},
|
||||
StatusUpdate::ImageGenerated { path, .. } => wit_channel::StatusUpdate {
|
||||
status: wit_channel::StatusType::Status,
|
||||
message: match path {
|
||||
Some(p) => format!("[image] {}", p),
|
||||
None => "[image generated]".to_string(),
|
||||
},
|
||||
metadata_json,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -426,7 +426,7 @@ pub async fn chat_threads_handler(
|
||||
.map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, e.to_string()))?;
|
||||
|
||||
if let Ok(summaries) = store
|
||||
.list_conversations_with_preview(&state.user_id, "gateway", 50)
|
||||
.list_conversations_all_channels(&state.user_id, 50)
|
||||
.await
|
||||
{
|
||||
let mut assistant_thread = None;
|
||||
@@ -441,6 +441,7 @@ pub async fn chat_threads_handler(
|
||||
updated_at: s.last_activity.to_rfc3339(),
|
||||
title: s.title.clone(),
|
||||
thread_type: s.thread_type.clone(),
|
||||
channel: Some(s.channel.clone()),
|
||||
};
|
||||
|
||||
if s.id == assistant_id {
|
||||
@@ -460,6 +461,7 @@ pub async fn chat_threads_handler(
|
||||
updated_at: chrono::Utc::now().to_rfc3339(),
|
||||
title: None,
|
||||
thread_type: Some("assistant".to_string()),
|
||||
channel: Some("gateway".to_string()),
|
||||
});
|
||||
}
|
||||
|
||||
@@ -472,9 +474,10 @@ pub async fn chat_threads_handler(
|
||||
}
|
||||
|
||||
// Fallback: in-memory only (no assistant thread without DB)
|
||||
let threads: Vec<ThreadInfo> = sess
|
||||
.threads
|
||||
.values()
|
||||
let mut sorted_threads: Vec<_> = sess.threads.values().collect();
|
||||
sorted_threads.sort_by(|a, b| b.updated_at.cmp(&a.updated_at));
|
||||
let threads: Vec<ThreadInfo> = sorted_threads
|
||||
.into_iter()
|
||||
.map(|t| ThreadInfo {
|
||||
id: t.id,
|
||||
state: format!("{:?}", t.state),
|
||||
@@ -483,6 +486,7 @@ pub async fn chat_threads_handler(
|
||||
updated_at: t.updated_at.to_rfc3339(),
|
||||
title: None,
|
||||
thread_type: None,
|
||||
channel: Some("gateway".to_string()),
|
||||
})
|
||||
.collect();
|
||||
|
||||
@@ -502,38 +506,39 @@ pub async fn chat_new_thread_handler(
|
||||
))?;
|
||||
|
||||
let session = session_manager.get_or_create_session(&state.user_id).await;
|
||||
let mut sess = session.lock().await;
|
||||
let thread = sess.create_thread();
|
||||
let thread_id = thread.id;
|
||||
let info = ThreadInfo {
|
||||
id: thread.id,
|
||||
state: format!("{:?}", thread.state),
|
||||
turn_count: thread.turns.len(),
|
||||
created_at: thread.created_at.to_rfc3339(),
|
||||
updated_at: thread.updated_at.to_rfc3339(),
|
||||
title: None,
|
||||
thread_type: Some("thread".to_string()),
|
||||
let (thread_id, info) = {
|
||||
let mut sess = session.lock().await;
|
||||
let thread = sess.create_thread();
|
||||
let id = thread.id;
|
||||
let info = ThreadInfo {
|
||||
id: thread.id,
|
||||
state: format!("{:?}", thread.state),
|
||||
turn_count: thread.turns.len(),
|
||||
created_at: thread.created_at.to_rfc3339(),
|
||||
updated_at: thread.updated_at.to_rfc3339(),
|
||||
title: None,
|
||||
thread_type: Some("thread".to_string()),
|
||||
channel: Some("gateway".to_string()),
|
||||
};
|
||||
(id, info)
|
||||
};
|
||||
|
||||
// Persist the empty conversation row with thread_type metadata
|
||||
// Persist the empty conversation row with thread_type metadata synchronously
|
||||
// so that the subsequent loadThreads() call from the frontend sees it.
|
||||
if let Some(ref store) = state.store {
|
||||
let store = Arc::clone(store);
|
||||
let user_id = state.user_id.clone();
|
||||
tokio::spawn(async move {
|
||||
if let Err(e) = store
|
||||
.ensure_conversation(thread_id, "gateway", &user_id, None)
|
||||
.await
|
||||
{
|
||||
tracing::warn!("Failed to persist new thread: {}", e);
|
||||
}
|
||||
let metadata_val = serde_json::json!("thread");
|
||||
if let Err(e) = store
|
||||
.update_conversation_metadata_field(thread_id, "thread_type", &metadata_val)
|
||||
.await
|
||||
{
|
||||
tracing::warn!("Failed to set thread_type metadata: {}", e);
|
||||
}
|
||||
});
|
||||
if let Err(e) = store
|
||||
.ensure_conversation(thread_id, "gateway", &state.user_id, None)
|
||||
.await
|
||||
{
|
||||
tracing::warn!("Failed to persist new thread: {}", e);
|
||||
}
|
||||
let metadata_val = serde_json::json!("thread");
|
||||
if let Err(e) = store
|
||||
.update_conversation_metadata_field(thread_id, "thread_type", &metadata_val)
|
||||
.await
|
||||
{
|
||||
tracing::warn!("Failed to set thread_type metadata: {}", e);
|
||||
}
|
||||
}
|
||||
|
||||
Ok(Json(info))
|
||||
|
||||
@@ -276,11 +276,25 @@ pub async fn jobs_cancel_handler(
|
||||
})));
|
||||
}
|
||||
|
||||
// Fall back to agent job cancellation via DB status update.
|
||||
// Fall back to agent job cancellation: stop the worker via the scheduler
|
||||
// (which updates the in-memory ContextManager AND aborts the task handle),
|
||||
// then persist the status to the DB as a fallback.
|
||||
if let Some(ref store) = state.store
|
||||
&& let Ok(Some(job)) = store.get_job(job_id).await
|
||||
{
|
||||
if job.state.is_active() {
|
||||
// Try to stop via scheduler (aborts the worker task + updates
|
||||
// in-memory ContextManager). This is best-effort — the job may
|
||||
// not be in the scheduler map if it already finished.
|
||||
if let Some(ref slot) = state.scheduler
|
||||
&& let Some(ref scheduler) = *slot.read().await
|
||||
{
|
||||
let _ = scheduler.stop(job_id).await;
|
||||
}
|
||||
|
||||
// Always persist cancellation to the DB so the state is
|
||||
// consistent even if the scheduler wasn't available or the
|
||||
// job wasn't in its in-memory map.
|
||||
store
|
||||
.update_job_status(
|
||||
job_id,
|
||||
|
||||
@@ -10,9 +10,9 @@ use axum::{
|
||||
use serde::Deserialize;
|
||||
use uuid::Uuid;
|
||||
|
||||
use crate::channels::IncomingMessage;
|
||||
use crate::channels::web::server::GatewayState;
|
||||
use crate::channels::web::types::*;
|
||||
use crate::error::RoutineError;
|
||||
|
||||
pub async fn routines_list_handler(
|
||||
State(state): State<Arc<GatewayState>>,
|
||||
@@ -108,6 +108,7 @@ pub async fn routines_detail_handler(
|
||||
status: format!("{:?}", run.status),
|
||||
result_summary: run.result_summary.clone(),
|
||||
tokens_used: run.tokens_used,
|
||||
job_id: run.job_id,
|
||||
})
|
||||
.collect();
|
||||
|
||||
@@ -133,56 +134,27 @@ pub async fn routines_trigger_handler(
|
||||
State(state): State<Arc<GatewayState>>,
|
||||
Path(id): Path<String>,
|
||||
) -> Result<Json<serde_json::Value>, (StatusCode, String)> {
|
||||
let store = state.store.as_ref().ok_or((
|
||||
StatusCode::SERVICE_UNAVAILABLE,
|
||||
"Database not available".to_string(),
|
||||
))?;
|
||||
// Clone the Arc out of the lock to avoid holding the RwLock across .await.
|
||||
let engine = {
|
||||
let guard = state.routine_engine.read().await;
|
||||
guard.as_ref().cloned().ok_or((
|
||||
StatusCode::SERVICE_UNAVAILABLE,
|
||||
"Routine engine not available".to_string(),
|
||||
))?
|
||||
};
|
||||
|
||||
let routine_id = Uuid::parse_str(&id)
|
||||
.map_err(|_| (StatusCode::BAD_REQUEST, "Invalid routine ID".to_string()))?;
|
||||
|
||||
let routine = store
|
||||
.get_routine(routine_id)
|
||||
let run_id = engine
|
||||
.fire_manual(routine_id, Some(&state.user_id))
|
||||
.await
|
||||
.map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, e.to_string()))?
|
||||
.ok_or((StatusCode::NOT_FOUND, "Routine not found".to_string()))?;
|
||||
|
||||
if routine.user_id != state.user_id {
|
||||
return Err((StatusCode::FORBIDDEN, "Access denied".to_string()));
|
||||
}
|
||||
|
||||
// Send the routine prompt through the message pipeline as a manual trigger.
|
||||
let prompt = match &routine.action {
|
||||
crate::agent::routine::RoutineAction::Lightweight { prompt, .. } => prompt.clone(),
|
||||
crate::agent::routine::RoutineAction::FullJob {
|
||||
title, description, ..
|
||||
} => format!("{}: {}", title, description),
|
||||
};
|
||||
|
||||
let content = format!("[routine:{}] {}", routine.name, prompt);
|
||||
let thread_id = format!(
|
||||
"routine-{}-{}",
|
||||
routine_id,
|
||||
chrono::Utc::now().timestamp_millis()
|
||||
);
|
||||
let msg = IncomingMessage::new("gateway", &state.user_id, content).with_thread(thread_id);
|
||||
|
||||
let tx_guard = state.msg_tx.read().await;
|
||||
let tx = tx_guard.as_ref().ok_or((
|
||||
StatusCode::SERVICE_UNAVAILABLE,
|
||||
"Channel not started".to_string(),
|
||||
))?;
|
||||
|
||||
tx.send(msg).await.map_err(|_| {
|
||||
(
|
||||
StatusCode::INTERNAL_SERVER_ERROR,
|
||||
"Channel closed".to_string(),
|
||||
)
|
||||
})?;
|
||||
.map_err(|e| (routine_error_status(&e), e.to_string()))?;
|
||||
|
||||
Ok(Json(serde_json::json!({
|
||||
"status": "triggered",
|
||||
"routine_id": routine_id,
|
||||
"run_id": run_id,
|
||||
})))
|
||||
}
|
||||
|
||||
@@ -281,6 +253,7 @@ pub async fn routines_runs_handler(
|
||||
status: format!("{:?}", run.status),
|
||||
result_summary: run.result_summary.clone(),
|
||||
tokens_used: run.tokens_used,
|
||||
job_id: run.job_id,
|
||||
})
|
||||
.collect();
|
||||
|
||||
@@ -293,7 +266,7 @@ pub async fn routines_runs_handler(
|
||||
/// Convert a Routine to the trimmed RoutineInfo for list display.
|
||||
fn routine_to_info(r: &crate::agent::routine::Routine) -> RoutineInfo {
|
||||
let (trigger_type, trigger_summary) = match &r.trigger {
|
||||
crate::agent::routine::Trigger::Cron { schedule } => {
|
||||
crate::agent::routine::Trigger::Cron { schedule, .. } => {
|
||||
("cron".to_string(), format!("cron: {}", schedule))
|
||||
}
|
||||
crate::agent::routine::Trigger::Event {
|
||||
@@ -337,3 +310,13 @@ fn routine_to_info(r: &crate::agent::routine::Routine) -> RoutineInfo {
|
||||
status: status.to_string(),
|
||||
}
|
||||
}
|
||||
|
||||
/// Map `RoutineError` variants to appropriate HTTP status codes.
|
||||
fn routine_error_status(err: &RoutineError) -> StatusCode {
|
||||
match err {
|
||||
RoutineError::NotFound { .. } => StatusCode::NOT_FOUND,
|
||||
RoutineError::NotAuthorized { .. } => StatusCode::FORBIDDEN,
|
||||
RoutineError::Disabled { .. } | RoutineError::MaxConcurrent { .. } => StatusCode::CONFLICT,
|
||||
_ => StatusCode::INTERNAL_SERVER_ERROR,
|
||||
}
|
||||
}
|
||||
|
||||
+26
-2
@@ -99,6 +99,7 @@ impl GatewayChannel {
|
||||
chat_rate_limiter: server::RateLimiter::new(30, 60),
|
||||
registry_entries: Vec::new(),
|
||||
cost_guard: None,
|
||||
routine_engine: Arc::new(tokio::sync::RwLock::new(None)),
|
||||
startup_time: std::time::Instant::now(),
|
||||
});
|
||||
|
||||
@@ -134,6 +135,7 @@ impl GatewayChannel {
|
||||
chat_rate_limiter: server::RateLimiter::new(30, 60),
|
||||
registry_entries: self.state.registry_entries.clone(),
|
||||
cost_guard: self.state.cost_guard.clone(),
|
||||
routine_engine: Arc::clone(&self.state.routine_engine),
|
||||
startup_time: self.state.startup_time,
|
||||
};
|
||||
mutate(&mut new_state);
|
||||
@@ -281,7 +283,15 @@ impl Channel for GatewayChannel {
|
||||
msg: &IncomingMessage,
|
||||
response: OutgoingResponse,
|
||||
) -> Result<(), ChannelError> {
|
||||
let thread_id = msg.thread_id.clone().unwrap_or_default();
|
||||
let thread_id = match &msg.thread_id {
|
||||
Some(tid) => tid.clone(),
|
||||
None => {
|
||||
tracing::warn!(
|
||||
"Gateway respond with no thread_id — skipping (clients would drop it)"
|
||||
);
|
||||
return Ok(());
|
||||
}
|
||||
};
|
||||
|
||||
self.state.sse.broadcast(SseEvent::Response {
|
||||
content: response.content,
|
||||
@@ -376,6 +386,11 @@ impl Channel for GatewayChannel {
|
||||
success,
|
||||
message,
|
||||
},
|
||||
StatusUpdate::ImageGenerated { data_url, path } => SseEvent::ImageGenerated {
|
||||
data_url,
|
||||
path,
|
||||
thread_id,
|
||||
},
|
||||
};
|
||||
|
||||
self.state.sse.broadcast(event);
|
||||
@@ -387,9 +402,18 @@ impl Channel for GatewayChannel {
|
||||
_user_id: &str,
|
||||
response: OutgoingResponse,
|
||||
) -> Result<(), ChannelError> {
|
||||
let thread_id = match response.thread_id {
|
||||
Some(tid) => tid,
|
||||
None => {
|
||||
tracing::warn!(
|
||||
"Gateway broadcast with no thread_id — skipping (clients would drop it)"
|
||||
);
|
||||
return Ok(());
|
||||
}
|
||||
};
|
||||
self.state.sse.broadcast(SseEvent::Response {
|
||||
content: response.content,
|
||||
thread_id: String::new(),
|
||||
thread_id,
|
||||
});
|
||||
Ok(())
|
||||
}
|
||||
|
||||
+143
-71
@@ -57,6 +57,10 @@ pub type PromptQueue = Arc<
|
||||
>,
|
||||
>;
|
||||
|
||||
/// Slot for the routine engine, filled at runtime after the agent starts.
|
||||
pub type RoutineEngineSlot =
|
||||
Arc<tokio::sync::RwLock<Option<Arc<crate::agent::routine_engine::RoutineEngine>>>>;
|
||||
|
||||
/// Simple sliding-window rate limiter.
|
||||
///
|
||||
/// Tracks the number of requests in the current window. Resets when the window expires.
|
||||
@@ -165,6 +169,8 @@ pub struct GatewayState {
|
||||
pub registry_entries: Vec<crate::extensions::RegistryEntry>,
|
||||
/// Cost guard for token/cost tracking.
|
||||
pub cost_guard: Option<Arc<crate::agent::cost_guard::CostGuard>>,
|
||||
/// Routine engine slot for manual routine triggering (filled at runtime).
|
||||
pub routine_engine: RoutineEngineSlot,
|
||||
/// Server startup time for uptime calculation.
|
||||
pub startup_time: std::time::Instant,
|
||||
}
|
||||
@@ -345,7 +351,7 @@ pub async fn start_server(
|
||||
.merge(statics)
|
||||
.merge(projects)
|
||||
.merge(protected)
|
||||
.layer(DefaultBodyLimit::max(1024 * 1024)) // 1 MB max request body
|
||||
.layer(DefaultBodyLimit::max(10 * 1024 * 1024)) // 10 MB max request body (image uploads)
|
||||
.layer(cors)
|
||||
.layer(SetResponseHeaderLayer::if_not_present(
|
||||
header::X_CONTENT_TYPE_OPTIONS,
|
||||
@@ -602,8 +608,59 @@ async fn oauth_callback_handler(
|
||||
|
||||
// --- Chat handlers ---
|
||||
|
||||
/// Convert web gateway `ImageData` to `IncomingAttachment` objects.
|
||||
pub(crate) fn images_to_attachments(
|
||||
images: &[ImageData],
|
||||
) -> Vec<crate::channels::IncomingAttachment> {
|
||||
use base64::Engine;
|
||||
images
|
||||
.iter()
|
||||
.enumerate()
|
||||
.filter_map(|(i, img)| {
|
||||
if !img.media_type.starts_with("image/") {
|
||||
tracing::warn!(
|
||||
"Skipping image {i}: invalid media type '{}' (must start with 'image/')",
|
||||
img.media_type
|
||||
);
|
||||
return None;
|
||||
}
|
||||
let data = match base64::engine::general_purpose::STANDARD.decode(&img.data) {
|
||||
Ok(d) => d,
|
||||
Err(e) => {
|
||||
tracing::warn!("Skipping image {i}: invalid base64 data: {e}");
|
||||
return None;
|
||||
}
|
||||
};
|
||||
Some(crate::channels::IncomingAttachment {
|
||||
id: format!("web-image-{i}"),
|
||||
kind: crate::channels::AttachmentKind::Image,
|
||||
mime_type: img.media_type.clone(),
|
||||
filename: Some(format!("image-{i}.{}", mime_to_ext(&img.media_type))),
|
||||
size_bytes: Some(data.len() as u64),
|
||||
source_url: None,
|
||||
storage_key: None,
|
||||
extracted_text: None,
|
||||
data,
|
||||
duration_secs: None,
|
||||
})
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// Map MIME type to file extension.
|
||||
fn mime_to_ext(mime: &str) -> &str {
|
||||
match mime {
|
||||
"image/png" => "png",
|
||||
"image/gif" => "gif",
|
||||
"image/webp" => "webp",
|
||||
"image/svg+xml" => "svg",
|
||||
_ => "jpg",
|
||||
}
|
||||
}
|
||||
|
||||
async fn chat_send_handler(
|
||||
State(state): State<Arc<GatewayState>>,
|
||||
headers: axum::http::HeaderMap,
|
||||
Json(req): Json<SendMessageRequest>,
|
||||
) -> Result<(StatusCode, Json<SendMessageResponse>), (StatusCode, String)> {
|
||||
tracing::debug!(
|
||||
@@ -620,17 +677,32 @@ async fn chat_send_handler(
|
||||
}
|
||||
|
||||
let mut msg = IncomingMessage::new("gateway", &state.user_id, &req.content);
|
||||
// Prefer timezone from JSON body, fall back to X-Timezone header
|
||||
let tz = req
|
||||
.timezone
|
||||
.as_deref()
|
||||
.or_else(|| headers.get("X-Timezone").and_then(|v| v.to_str().ok()));
|
||||
if let Some(tz) = tz {
|
||||
msg = msg.with_timezone(tz);
|
||||
}
|
||||
|
||||
if let Some(ref thread_id) = req.thread_id {
|
||||
msg = msg.with_thread(thread_id);
|
||||
msg = msg.with_metadata(serde_json::json!({"thread_id": thread_id}));
|
||||
}
|
||||
|
||||
// Convert uploaded images to IncomingAttachments
|
||||
if !req.images.is_empty() {
|
||||
let attachments = images_to_attachments(&req.images);
|
||||
msg = msg.with_attachments(attachments);
|
||||
}
|
||||
|
||||
let msg_id = msg.id;
|
||||
tracing::debug!(
|
||||
"[chat_send_handler] Created message id={}, content={:?}",
|
||||
"[chat_send_handler] Created message id={}, content={:?}, images={}",
|
||||
msg_id,
|
||||
req.content
|
||||
req.content,
|
||||
req.images.len()
|
||||
);
|
||||
|
||||
let tx_guard = state.msg_tx.read().await;
|
||||
@@ -1037,7 +1109,7 @@ async fn chat_threads_handler(
|
||||
.map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, e.to_string()))?;
|
||||
|
||||
if let Ok(summaries) = store
|
||||
.list_conversations_with_preview(&state.user_id, "gateway", 50)
|
||||
.list_conversations_all_channels(&state.user_id, 50)
|
||||
.await
|
||||
{
|
||||
let mut assistant_thread = None;
|
||||
@@ -1052,6 +1124,7 @@ async fn chat_threads_handler(
|
||||
updated_at: s.last_activity.to_rfc3339(),
|
||||
title: s.title.clone(),
|
||||
thread_type: s.thread_type.clone(),
|
||||
channel: Some(s.channel.clone()),
|
||||
};
|
||||
|
||||
if s.id == assistant_id {
|
||||
@@ -1071,6 +1144,7 @@ async fn chat_threads_handler(
|
||||
updated_at: chrono::Utc::now().to_rfc3339(),
|
||||
title: None,
|
||||
thread_type: Some("assistant".to_string()),
|
||||
channel: Some("gateway".to_string()),
|
||||
});
|
||||
}
|
||||
|
||||
@@ -1083,9 +1157,10 @@ async fn chat_threads_handler(
|
||||
}
|
||||
|
||||
// Fallback: in-memory only (no assistant thread without DB)
|
||||
let threads: Vec<ThreadInfo> = sess
|
||||
.threads
|
||||
.values()
|
||||
let mut sorted_threads: Vec<_> = sess.threads.values().collect();
|
||||
sorted_threads.sort_by(|a, b| b.updated_at.cmp(&a.updated_at));
|
||||
let threads: Vec<ThreadInfo> = sorted_threads
|
||||
.into_iter()
|
||||
.map(|t| ThreadInfo {
|
||||
id: t.id,
|
||||
state: format!("{:?}", t.state),
|
||||
@@ -1094,6 +1169,7 @@ async fn chat_threads_handler(
|
||||
updated_at: t.updated_at.to_rfc3339(),
|
||||
title: None,
|
||||
thread_type: None,
|
||||
channel: Some("gateway".to_string()),
|
||||
})
|
||||
.collect();
|
||||
|
||||
@@ -1113,38 +1189,39 @@ async fn chat_new_thread_handler(
|
||||
))?;
|
||||
|
||||
let session = session_manager.get_or_create_session(&state.user_id).await;
|
||||
let mut sess = session.lock().await;
|
||||
let thread = sess.create_thread();
|
||||
let thread_id = thread.id;
|
||||
let info = ThreadInfo {
|
||||
id: thread.id,
|
||||
state: format!("{:?}", thread.state),
|
||||
turn_count: thread.turns.len(),
|
||||
created_at: thread.created_at.to_rfc3339(),
|
||||
updated_at: thread.updated_at.to_rfc3339(),
|
||||
title: None,
|
||||
thread_type: Some("thread".to_string()),
|
||||
let (thread_id, info) = {
|
||||
let mut sess = session.lock().await;
|
||||
let thread = sess.create_thread();
|
||||
let id = thread.id;
|
||||
let info = ThreadInfo {
|
||||
id: thread.id,
|
||||
state: format!("{:?}", thread.state),
|
||||
turn_count: thread.turns.len(),
|
||||
created_at: thread.created_at.to_rfc3339(),
|
||||
updated_at: thread.updated_at.to_rfc3339(),
|
||||
title: None,
|
||||
thread_type: Some("thread".to_string()),
|
||||
channel: Some("gateway".to_string()),
|
||||
};
|
||||
(id, info)
|
||||
};
|
||||
|
||||
// Persist the empty conversation row with thread_type metadata
|
||||
// Persist the empty conversation row with thread_type metadata synchronously
|
||||
// so that the subsequent loadThreads() call from the frontend sees it.
|
||||
if let Some(ref store) = state.store {
|
||||
let store = Arc::clone(store);
|
||||
let user_id = state.user_id.clone();
|
||||
tokio::spawn(async move {
|
||||
if let Err(e) = store
|
||||
.ensure_conversation(thread_id, "gateway", &user_id, None)
|
||||
.await
|
||||
{
|
||||
tracing::warn!("Failed to persist new thread: {}", e);
|
||||
}
|
||||
let metadata_val = serde_json::json!("thread");
|
||||
if let Err(e) = store
|
||||
.update_conversation_metadata_field(thread_id, "thread_type", &metadata_val)
|
||||
.await
|
||||
{
|
||||
tracing::warn!("Failed to set thread_type metadata: {}", e);
|
||||
}
|
||||
});
|
||||
if let Err(e) = store
|
||||
.ensure_conversation(thread_id, "gateway", &state.user_id, None)
|
||||
.await
|
||||
{
|
||||
tracing::warn!("Failed to persist new thread: {}", e);
|
||||
}
|
||||
let metadata_val = serde_json::json!("thread");
|
||||
if let Err(e) = store
|
||||
.update_conversation_metadata_field(thread_id, "thread_type", &metadata_val)
|
||||
.await
|
||||
{
|
||||
tracing::warn!("Failed to set thread_type metadata: {}", e);
|
||||
}
|
||||
}
|
||||
|
||||
Ok(Json(info))
|
||||
@@ -1940,6 +2017,7 @@ async fn routines_detail_handler(
|
||||
status: format!("{:?}", run.status),
|
||||
result_summary: run.result_summary.clone(),
|
||||
tokens_used: run.tokens_used,
|
||||
job_id: run.job_id,
|
||||
})
|
||||
.collect();
|
||||
|
||||
@@ -1965,47 +2043,35 @@ async fn routines_trigger_handler(
|
||||
State(state): State<Arc<GatewayState>>,
|
||||
Path(id): Path<String>,
|
||||
) -> Result<Json<serde_json::Value>, (StatusCode, String)> {
|
||||
let store = state.store.as_ref().ok_or((
|
||||
StatusCode::SERVICE_UNAVAILABLE,
|
||||
"Database not available".to_string(),
|
||||
))?;
|
||||
let engine = {
|
||||
let guard = state.routine_engine.read().await;
|
||||
guard.as_ref().cloned().ok_or((
|
||||
StatusCode::SERVICE_UNAVAILABLE,
|
||||
"Routine engine not available".to_string(),
|
||||
))?
|
||||
};
|
||||
|
||||
let routine_id = Uuid::parse_str(&id)
|
||||
.map_err(|_| (StatusCode::BAD_REQUEST, "Invalid routine ID".to_string()))?;
|
||||
|
||||
let routine = store
|
||||
.get_routine(routine_id)
|
||||
let run_id = engine
|
||||
.fire_manual(routine_id, Some(&state.user_id))
|
||||
.await
|
||||
.map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, e.to_string()))?
|
||||
.ok_or((StatusCode::NOT_FOUND, "Routine not found".to_string()))?;
|
||||
|
||||
// Send the routine prompt through the message pipeline as a manual trigger.
|
||||
let prompt = match &routine.action {
|
||||
crate::agent::routine::RoutineAction::Lightweight { prompt, .. } => prompt.clone(),
|
||||
crate::agent::routine::RoutineAction::FullJob {
|
||||
title, description, ..
|
||||
} => format!("{}: {}", title, description),
|
||||
};
|
||||
|
||||
let content = format!("[routine:{}] {}", routine.name, prompt);
|
||||
let msg = IncomingMessage::new("gateway", &state.user_id, content);
|
||||
|
||||
let tx_guard = state.msg_tx.read().await;
|
||||
let tx = tx_guard.as_ref().ok_or((
|
||||
StatusCode::SERVICE_UNAVAILABLE,
|
||||
"Channel not started".to_string(),
|
||||
))?;
|
||||
|
||||
tx.send(msg).await.map_err(|_| {
|
||||
(
|
||||
StatusCode::INTERNAL_SERVER_ERROR,
|
||||
"Channel closed".to_string(),
|
||||
)
|
||||
})?;
|
||||
.map_err(|e| {
|
||||
let status = match &e {
|
||||
crate::error::RoutineError::NotFound { .. } => StatusCode::NOT_FOUND,
|
||||
crate::error::RoutineError::NotAuthorized { .. } => StatusCode::FORBIDDEN,
|
||||
crate::error::RoutineError::Disabled { .. }
|
||||
| crate::error::RoutineError::MaxConcurrent { .. } => StatusCode::CONFLICT,
|
||||
_ => StatusCode::INTERNAL_SERVER_ERROR,
|
||||
};
|
||||
(status, e.to_string())
|
||||
})?;
|
||||
|
||||
Ok(Json(serde_json::json!({
|
||||
"status": "triggered",
|
||||
"routine_id": routine_id,
|
||||
"run_id": run_id,
|
||||
})))
|
||||
}
|
||||
|
||||
@@ -2104,6 +2170,7 @@ async fn routines_runs_handler(
|
||||
status: format!("{:?}", run.status),
|
||||
result_summary: run.result_summary.clone(),
|
||||
tokens_used: run.tokens_used,
|
||||
job_id: run.job_id,
|
||||
})
|
||||
.collect();
|
||||
|
||||
@@ -2116,7 +2183,7 @@ async fn routines_runs_handler(
|
||||
/// Convert a Routine to the trimmed RoutineInfo for list display.
|
||||
fn routine_to_info(r: &crate::agent::routine::Routine) -> RoutineInfo {
|
||||
let (trigger_type, trigger_summary) = match &r.trigger {
|
||||
crate::agent::routine::Trigger::Cron { schedule } => {
|
||||
crate::agent::routine::Trigger::Cron { schedule, .. } => {
|
||||
("cron".to_string(), format!("cron: {}", schedule))
|
||||
}
|
||||
crate::agent::routine::Trigger::Event {
|
||||
@@ -2463,6 +2530,7 @@ mod tests {
|
||||
chat_rate_limiter: RateLimiter::new(30, 60),
|
||||
registry_entries: vec![],
|
||||
cost_guard: None,
|
||||
routine_engine: Arc::new(tokio::sync::RwLock::new(None)),
|
||||
startup_time: std::time::Instant::now(),
|
||||
})
|
||||
}
|
||||
@@ -2620,7 +2688,9 @@ mod tests {
|
||||
secrets,
|
||||
sse_sender: None,
|
||||
gateway_token: None,
|
||||
created_at: std::time::Instant::now() - std::time::Duration::from_secs(600),
|
||||
created_at: std::time::Instant::now()
|
||||
.checked_sub(std::time::Duration::from_secs(600))
|
||||
.expect("System uptime is too low to run expired flow test"),
|
||||
};
|
||||
|
||||
ext_mgr
|
||||
@@ -2727,7 +2797,9 @@ mod tests {
|
||||
sse_sender: None,
|
||||
gateway_token: None,
|
||||
// Expired — handler will reject after lookup (no network I/O)
|
||||
created_at: std::time::Instant::now() - std::time::Duration::from_secs(600),
|
||||
created_at: std::time::Instant::now()
|
||||
.checked_sub(std::time::Duration::from_secs(600))
|
||||
.expect("System uptime is too low to run expired flow test"),
|
||||
};
|
||||
|
||||
ext_mgr
|
||||
|
||||
@@ -142,6 +142,7 @@ impl SseManager {
|
||||
SseEvent::JobStatus { .. } => "job_status",
|
||||
SseEvent::JobResult { .. } => "job_result",
|
||||
SseEvent::Heartbeat => "heartbeat",
|
||||
SseEvent::ImageGenerated { .. } => "image_generated",
|
||||
SseEvent::ExtensionStatus { .. } => "extension_status",
|
||||
};
|
||||
Ok(Event::default().event(event_type).data(data))
|
||||
|
||||
+270
-49
@@ -5,6 +5,7 @@ let eventSource = null;
|
||||
let logEventSource = null;
|
||||
let currentTab = 'chat';
|
||||
let currentThreadId = null;
|
||||
let currentThreadIsReadOnly = false;
|
||||
let assistantThreadId = null;
|
||||
let hasMore = false;
|
||||
let oldestTimestamp = null;
|
||||
@@ -13,8 +14,11 @@ let sseHasConnectedBefore = false;
|
||||
let jobEvents = new Map(); // job_id -> Array of events
|
||||
let jobListRefreshTimer = null;
|
||||
let pairingPollInterval = null;
|
||||
let unreadThreads = new Map(); // thread_id -> unread count
|
||||
let _loadThreadsTimer = null;
|
||||
const JOB_EVENTS_CAP = 500;
|
||||
const MEMORY_SEARCH_QUERY_MAX_LENGTH = 100;
|
||||
let stagedImages = [];
|
||||
|
||||
// --- Slash Commands ---
|
||||
|
||||
@@ -178,6 +182,7 @@ function confirmRestart() {
|
||||
body: {
|
||||
content: '/restart',
|
||||
thread_id: currentThreadId,
|
||||
timezone: Intl.DateTimeFormat().resolvedOptions().timeZone,
|
||||
},
|
||||
})
|
||||
.then((response) => {
|
||||
@@ -220,23 +225,6 @@ function updateRestartButtonVisibility() {
|
||||
}
|
||||
}
|
||||
|
||||
function startGatewayStatusPolling() {
|
||||
fetchGatewayStatus();
|
||||
// Poll every 5 seconds
|
||||
setInterval(fetchGatewayStatus, 5000);
|
||||
}
|
||||
|
||||
function fetchGatewayStatus() {
|
||||
apiFetch('/api/gateway/status')
|
||||
.then((data) => {
|
||||
restartEnabled = data.restart_enabled || false;
|
||||
updateRestartButtonVisibility();
|
||||
})
|
||||
.catch((err) => {
|
||||
console.warn('[gateway status] Failed to fetch:', err);
|
||||
});
|
||||
}
|
||||
|
||||
// --- SSE ---
|
||||
|
||||
function connectSSE() {
|
||||
@@ -273,7 +261,13 @@ function connectSSE() {
|
||||
|
||||
eventSource.addEventListener('response', (e) => {
|
||||
const data = JSON.parse(e.data);
|
||||
if (!isCurrentThread(data.thread_id)) return;
|
||||
if (!isCurrentThread(data.thread_id)) {
|
||||
if (data.thread_id) {
|
||||
unreadThreads.set(data.thread_id, (unreadThreads.get(data.thread_id) || 0) + 1);
|
||||
debouncedLoadThreads();
|
||||
}
|
||||
return;
|
||||
}
|
||||
finalizeActivityGroup();
|
||||
addMessage('assistant', data.content);
|
||||
enableChatInput();
|
||||
@@ -288,7 +282,10 @@ function connectSSE() {
|
||||
|
||||
eventSource.addEventListener('thinking', (e) => {
|
||||
const data = JSON.parse(e.data);
|
||||
if (!isCurrentThread(data.thread_id)) return;
|
||||
if (!isCurrentThread(data.thread_id)) {
|
||||
if (data.thread_id) debouncedLoadThreads();
|
||||
return;
|
||||
}
|
||||
showActivityThinking(data.message);
|
||||
});
|
||||
|
||||
@@ -324,7 +321,10 @@ function connectSSE() {
|
||||
|
||||
eventSource.addEventListener('status', (e) => {
|
||||
const data = JSON.parse(e.data);
|
||||
if (!isCurrentThread(data.thread_id)) return;
|
||||
if (!isCurrentThread(data.thread_id)) {
|
||||
if (data.thread_id) debouncedLoadThreads();
|
||||
return;
|
||||
}
|
||||
// "Done" and "Awaiting approval" are terminal signals from the agent:
|
||||
// the agentic loop finished, so re-enable input as a safety net in case
|
||||
// the response SSE event is empty or lost.
|
||||
@@ -373,6 +373,12 @@ function connectSSE() {
|
||||
if (currentTab === 'extensions') loadExtensions();
|
||||
});
|
||||
|
||||
eventSource.addEventListener('image_generated', (e) => {
|
||||
const data = JSON.parse(e.data);
|
||||
if (!isCurrentThread(data.thread_id)) return;
|
||||
addGeneratedImage(data.data_url, data.path);
|
||||
});
|
||||
|
||||
eventSource.addEventListener('error', (e) => {
|
||||
if (e.data) {
|
||||
const data = JSON.parse(e.data);
|
||||
@@ -414,9 +420,9 @@ function connectSSE() {
|
||||
}
|
||||
|
||||
// Check if an SSE event belongs to the currently viewed thread.
|
||||
// Events without a thread_id (legacy) are always shown.
|
||||
// Events without a thread_id are dropped (prevents notification leaking).
|
||||
function isCurrentThread(threadId) {
|
||||
if (!threadId) return true;
|
||||
if (!threadId) return false;
|
||||
if (!currentThreadId) return true;
|
||||
return threadId === currentThreadId;
|
||||
}
|
||||
@@ -430,23 +436,135 @@ function sendMessage() {
|
||||
return;
|
||||
}
|
||||
const content = input.value.trim();
|
||||
if (!content) return;
|
||||
if (!content && stagedImages.length === 0) return;
|
||||
|
||||
addMessage('user', content);
|
||||
addMessage('user', content || '(images attached)');
|
||||
input.value = '';
|
||||
autoResizeTextarea(input);
|
||||
input.focus();
|
||||
|
||||
const body = { content, thread_id: currentThreadId || undefined, timezone: Intl.DateTimeFormat().resolvedOptions().timeZone };
|
||||
if (stagedImages.length > 0) {
|
||||
body.images = stagedImages.map(img => ({ media_type: img.media_type, data: img.data }));
|
||||
stagedImages = [];
|
||||
renderImagePreviews();
|
||||
}
|
||||
|
||||
apiFetch('/api/chat/send', {
|
||||
method: 'POST',
|
||||
body: { content, thread_id: currentThreadId || undefined },
|
||||
body: body,
|
||||
}).catch((err) => {
|
||||
addMessage('system', 'Failed to send: ' + err.message);
|
||||
});
|
||||
}
|
||||
|
||||
function enableChatInput() {
|
||||
// no-op: input and send button are always enabled
|
||||
if (currentThreadIsReadOnly) return;
|
||||
const input = document.getElementById('chat-input');
|
||||
const btn = document.getElementById('send-btn');
|
||||
if (input) {
|
||||
input.disabled = false;
|
||||
input.placeholder = 'Message or / for commands...';
|
||||
}
|
||||
if (btn) btn.disabled = false;
|
||||
}
|
||||
|
||||
// --- Image Upload ---
|
||||
|
||||
function renderImagePreviews() {
|
||||
const strip = document.getElementById('image-preview-strip');
|
||||
strip.innerHTML = '';
|
||||
stagedImages.forEach((img, idx) => {
|
||||
const container = document.createElement('div');
|
||||
container.className = 'image-preview-container';
|
||||
|
||||
const preview = document.createElement('img');
|
||||
preview.className = 'image-preview';
|
||||
preview.src = img.dataUrl;
|
||||
preview.alt = 'Attached image';
|
||||
|
||||
const removeBtn = document.createElement('button');
|
||||
removeBtn.className = 'image-preview-remove';
|
||||
removeBtn.textContent = '\u00d7';
|
||||
removeBtn.addEventListener('click', () => {
|
||||
stagedImages.splice(idx, 1);
|
||||
renderImagePreviews();
|
||||
});
|
||||
|
||||
container.appendChild(preview);
|
||||
container.appendChild(removeBtn);
|
||||
strip.appendChild(container);
|
||||
});
|
||||
}
|
||||
|
||||
const MAX_IMAGE_SIZE_BYTES = 5 * 1024 * 1024; // 5 MB per image
|
||||
const MAX_STAGED_IMAGES = 5;
|
||||
|
||||
function handleImageFiles(files) {
|
||||
Array.from(files).forEach(file => {
|
||||
if (!file.type.startsWith('image/')) return;
|
||||
if (file.size > MAX_IMAGE_SIZE_BYTES) {
|
||||
alert(`Image "${file.name}" exceeds 5 MB limit (${(file.size / 1024 / 1024).toFixed(1)} MB)`);
|
||||
return;
|
||||
}
|
||||
if (stagedImages.length >= MAX_STAGED_IMAGES) {
|
||||
alert(`Maximum ${MAX_STAGED_IMAGES} images allowed per message`);
|
||||
return;
|
||||
}
|
||||
const reader = new FileReader();
|
||||
reader.onload = function(e) {
|
||||
const dataUrl = e.target.result;
|
||||
const commaIdx = dataUrl.indexOf(',');
|
||||
const meta = dataUrl.substring(0, commaIdx); // e.g. "data:image/png;base64"
|
||||
const base64 = dataUrl.substring(commaIdx + 1);
|
||||
const mediaType = meta.replace('data:', '').replace(';base64', '');
|
||||
stagedImages.push({ media_type: mediaType, data: base64, dataUrl: dataUrl });
|
||||
renderImagePreviews();
|
||||
};
|
||||
reader.readAsDataURL(file);
|
||||
});
|
||||
}
|
||||
|
||||
document.getElementById('attach-btn').addEventListener('click', () => {
|
||||
document.getElementById('image-file-input').click();
|
||||
});
|
||||
|
||||
document.getElementById('image-file-input').addEventListener('change', (e) => {
|
||||
handleImageFiles(e.target.files);
|
||||
e.target.value = '';
|
||||
});
|
||||
|
||||
document.getElementById('chat-input').addEventListener('paste', (e) => {
|
||||
const items = (e.clipboardData || e.originalEvent.clipboardData).items;
|
||||
for (let i = 0; i < items.length; i++) {
|
||||
if (items[i].kind === 'file' && items[i].type.startsWith('image/')) {
|
||||
const file = items[i].getAsFile();
|
||||
if (file) handleImageFiles([file]);
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
function addGeneratedImage(dataUrl, path) {
|
||||
const container = document.getElementById('chat-messages');
|
||||
const card = document.createElement('div');
|
||||
card.className = 'generated-image-card';
|
||||
|
||||
const img = document.createElement('img');
|
||||
img.className = 'generated-image';
|
||||
img.src = dataUrl;
|
||||
img.alt = 'Generated image';
|
||||
|
||||
card.appendChild(img);
|
||||
|
||||
if (path) {
|
||||
const pathLabel = document.createElement('div');
|
||||
pathLabel.className = 'generated-image-path';
|
||||
pathLabel.textContent = path;
|
||||
card.appendChild(pathLabel);
|
||||
}
|
||||
|
||||
container.appendChild(card);
|
||||
container.scrollTop = container.scrollHeight;
|
||||
}
|
||||
|
||||
// --- Slash Autocomplete ---
|
||||
@@ -541,6 +659,13 @@ function sendApprovalAction(requestId, action) {
|
||||
|
||||
function renderMarkdown(text) {
|
||||
if (typeof marked !== 'undefined') {
|
||||
// Escape raw HTML error pages instead of rendering them as markup.
|
||||
// Only triggers when the text *starts with* a doctype or <html> tag
|
||||
// (after optional whitespace), so normal messages that mention HTML
|
||||
// tags in prose or code fences are not affected. See #263.
|
||||
if (/^\s*<!doctype\s/i.test(text) || /^\s*<html[\s>]/i.test(text)) {
|
||||
return escapeHtml(text);
|
||||
}
|
||||
let html = marked.parse(text);
|
||||
// Sanitize HTML output to prevent XSS from tool output or LLM responses.
|
||||
html = sanitizeRenderedHtml(html);
|
||||
@@ -1134,7 +1259,9 @@ function loadHistory(before) {
|
||||
// Fresh load: clear and render
|
||||
container.innerHTML = '';
|
||||
for (const turn of data.turns) {
|
||||
addMessage('user', turn.user_input);
|
||||
if (turn.user_input) {
|
||||
addMessage('user', turn.user_input);
|
||||
}
|
||||
if (turn.tool_calls && turn.tool_calls.length > 0) {
|
||||
addToolCallsSummary(turn.tool_calls);
|
||||
}
|
||||
@@ -1156,8 +1283,10 @@ function loadHistory(before) {
|
||||
const savedHeight = container.scrollHeight;
|
||||
const fragment = document.createDocumentFragment();
|
||||
for (const turn of data.turns) {
|
||||
const userDiv = createMessageElement('user', turn.user_input);
|
||||
fragment.appendChild(userDiv);
|
||||
if (turn.user_input) {
|
||||
const userDiv = createMessageElement('user', turn.user_input);
|
||||
fragment.appendChild(userDiv);
|
||||
}
|
||||
if (turn.tool_calls && turn.tool_calls.length > 0) {
|
||||
fragment.appendChild(createToolCallsSummaryElement(turn.tool_calls));
|
||||
}
|
||||
@@ -1256,6 +1385,37 @@ function removeScrollSpinner() {
|
||||
|
||||
// --- Threads ---
|
||||
|
||||
function threadTitle(thread) {
|
||||
if (thread.title) return thread.title;
|
||||
const ch = thread.channel || 'gateway';
|
||||
if (thread.thread_type === 'heartbeat') return 'Heartbeat Alerts';
|
||||
if (thread.thread_type === 'routine') return 'Routine';
|
||||
if (ch !== 'gateway') return ch.charAt(0).toUpperCase() + ch.slice(1);
|
||||
if (thread.turn_count === 0) return 'New chat';
|
||||
return thread.id.substring(0, 8);
|
||||
}
|
||||
|
||||
function relativeTime(isoStr) {
|
||||
if (!isoStr) return '';
|
||||
const diff = Date.now() - new Date(isoStr).getTime();
|
||||
const mins = Math.floor(diff / 60000);
|
||||
if (mins < 1) return 'now';
|
||||
if (mins < 60) return mins + 'm ago';
|
||||
const hrs = Math.floor(mins / 60);
|
||||
if (hrs < 24) return hrs + 'h ago';
|
||||
const days = Math.floor(hrs / 24);
|
||||
return days + 'd ago';
|
||||
}
|
||||
|
||||
function isReadOnlyChannel(channel) {
|
||||
return channel && channel !== 'gateway' && channel !== 'routine' && channel !== 'heartbeat';
|
||||
}
|
||||
|
||||
function debouncedLoadThreads() {
|
||||
if (_loadThreadsTimer) clearTimeout(_loadThreadsTimer);
|
||||
_loadThreadsTimer = setTimeout(() => { _loadThreadsTimer = null; loadThreads(); }, 500);
|
||||
}
|
||||
|
||||
function loadThreads() {
|
||||
apiFetch('/api/chat/threads').then((data) => {
|
||||
// Pinned assistant thread
|
||||
@@ -1264,9 +1424,13 @@ function loadThreads() {
|
||||
const el = document.getElementById('assistant-thread');
|
||||
const isActive = currentThreadId === assistantThreadId;
|
||||
el.className = 'assistant-item' + (isActive ? ' active' : '');
|
||||
const labelEl = document.getElementById('assistant-label');
|
||||
if (labelEl) {
|
||||
const at = data.assistant_thread;
|
||||
labelEl.textContent = 'Assistant';
|
||||
}
|
||||
const meta = document.getElementById('assistant-meta');
|
||||
const count = data.assistant_thread.turn_count || 0;
|
||||
meta.textContent = count > 0 ? count + ' turns' : '';
|
||||
meta.textContent = relativeTime(data.assistant_thread.updated_at);
|
||||
}
|
||||
|
||||
// Regular threads
|
||||
@@ -1275,16 +1439,38 @@ function loadThreads() {
|
||||
const threads = data.threads || [];
|
||||
for (const thread of threads) {
|
||||
const item = document.createElement('div');
|
||||
item.className = 'thread-item' + (thread.id === currentThreadId ? ' active' : '');
|
||||
const isActive = thread.id === currentThreadId;
|
||||
item.className = 'thread-item' + (isActive ? ' active' : '');
|
||||
|
||||
// Channel badge for non-gateway threads
|
||||
const ch = thread.channel || 'gateway';
|
||||
if (ch !== 'gateway') {
|
||||
const badge = document.createElement('span');
|
||||
badge.className = 'thread-badge thread-badge-' + ch;
|
||||
badge.textContent = ch;
|
||||
item.appendChild(badge);
|
||||
}
|
||||
|
||||
const label = document.createElement('span');
|
||||
label.className = 'thread-label';
|
||||
label.textContent = thread.title || thread.id.substring(0, 8);
|
||||
label.title = thread.title ? thread.title + ' (' + thread.id + ')' : thread.id;
|
||||
label.textContent = threadTitle(thread);
|
||||
label.title = (thread.title || '') + ' (' + thread.id + ')';
|
||||
item.appendChild(label);
|
||||
|
||||
const meta = document.createElement('span');
|
||||
meta.className = 'thread-meta';
|
||||
meta.textContent = (thread.turn_count || 0) + ' turns';
|
||||
meta.textContent = relativeTime(thread.updated_at);
|
||||
item.appendChild(meta);
|
||||
|
||||
// Unread dot
|
||||
const unread = unreadThreads.get(thread.id) || 0;
|
||||
if (unread > 0 && !isActive) {
|
||||
const dot = document.createElement('span');
|
||||
dot.className = 'thread-unread';
|
||||
dot.textContent = unread > 9 ? '9+' : String(unread);
|
||||
item.appendChild(dot);
|
||||
}
|
||||
|
||||
item.addEventListener('click', () => switchThread(thread.id));
|
||||
list.appendChild(item);
|
||||
}
|
||||
@@ -1294,17 +1480,36 @@ function loadThreads() {
|
||||
switchToAssistant();
|
||||
}
|
||||
|
||||
// Enable chat input once a thread is available
|
||||
// Enable/disable chat input based on channel type
|
||||
if (currentThreadId) {
|
||||
enableChatInput();
|
||||
const currentThread = threads.find(t => t.id === currentThreadId);
|
||||
const ch = currentThread ? currentThread.channel : 'gateway';
|
||||
currentThreadIsReadOnly = isReadOnlyChannel(ch);
|
||||
if (currentThreadIsReadOnly) {
|
||||
disableChatInputReadOnly();
|
||||
} else {
|
||||
enableChatInput();
|
||||
}
|
||||
}
|
||||
}).catch(() => {});
|
||||
}
|
||||
|
||||
function disableChatInputReadOnly() {
|
||||
const input = document.getElementById('chat-input');
|
||||
const btn = document.getElementById('send-btn');
|
||||
if (input) {
|
||||
input.disabled = true;
|
||||
input.placeholder = 'Read-only thread (external channel)';
|
||||
}
|
||||
if (btn) btn.disabled = true;
|
||||
}
|
||||
|
||||
function switchToAssistant() {
|
||||
if (!assistantThreadId) return;
|
||||
finalizeActivityGroup();
|
||||
currentThreadId = assistantThreadId;
|
||||
currentThreadIsReadOnly = false;
|
||||
unreadThreads.delete(assistantThreadId);
|
||||
hasMore = false;
|
||||
oldestTimestamp = null;
|
||||
loadHistory();
|
||||
@@ -1314,6 +1519,7 @@ function switchToAssistant() {
|
||||
function switchThread(threadId) {
|
||||
finalizeActivityGroup();
|
||||
currentThreadId = threadId;
|
||||
unreadThreads.delete(threadId);
|
||||
hasMore = false;
|
||||
oldestTimestamp = null;
|
||||
loadHistory();
|
||||
@@ -1370,7 +1576,7 @@ chatInput.addEventListener('keydown', (e) => {
|
||||
}
|
||||
}
|
||||
|
||||
if (e.key === 'Enter' && !e.shiftKey) {
|
||||
if (e.key === 'Enter' && !e.shiftKey && !e.isComposing) {
|
||||
e.preventDefault();
|
||||
hideSlashAutocomplete();
|
||||
sendMessage();
|
||||
@@ -3305,6 +3511,10 @@ function shortModelName(model) {
|
||||
|
||||
function fetchGatewayStatus() {
|
||||
apiFetch('/api/gateway/status').then(function(data) {
|
||||
// Update restart button visibility
|
||||
restartEnabled = data.restart_enabled || false;
|
||||
updateRestartButtonVisibility();
|
||||
|
||||
var popover = document.getElementById('gateway-popover');
|
||||
var html = '';
|
||||
|
||||
@@ -3370,10 +3580,15 @@ let teeReportCache = null;
|
||||
let teeReportLoading = false;
|
||||
|
||||
function teeApiBase() {
|
||||
var parts = window.location.hostname.split('.');
|
||||
if (parts.length < 2) return null;
|
||||
var domain = parts.slice(1).join('.');
|
||||
return window.location.protocol + '//api.' + domain;
|
||||
var hostname = window.location.hostname;
|
||||
// Skip IP addresses (IPv4 and IPv6) and localhost
|
||||
if (hostname === "localhost" || /^(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)$/.test(hostname) || hostname.indexOf(":") !== -1) {
|
||||
return null;
|
||||
}
|
||||
var parts = hostname.split(".");
|
||||
if (parts.length < 2) return null;
|
||||
var domain = parts.slice(1).join(".");
|
||||
return window.location.protocol + "//api." + domain;
|
||||
}
|
||||
|
||||
function teeInstanceName() {
|
||||
@@ -3384,13 +3599,19 @@ function checkTeeStatus() {
|
||||
var base = teeApiBase();
|
||||
if (!base) return;
|
||||
var name = teeInstanceName();
|
||||
fetch(base + '/instances/' + encodeURIComponent(name) + '/attestation').then(function(res) {
|
||||
if (!res.ok) throw new Error(res.status);
|
||||
return res.json();
|
||||
}).then(function(data) {
|
||||
teeInfo = data;
|
||||
document.getElementById('tee-shield').style.display = 'flex';
|
||||
}).catch(function() {});
|
||||
try {
|
||||
fetch(base + '/instances/' + encodeURIComponent(name) + '/attestation').then(function(res) {
|
||||
if (!res.ok) throw new Error(res.status);
|
||||
return res.json();
|
||||
}).then(function(data) {
|
||||
teeInfo = data;
|
||||
document.getElementById('tee-shield').style.display = 'flex';
|
||||
}).catch(function(err) {
|
||||
console.warn('Failed to fetch TEE attestation:', err);
|
||||
});
|
||||
} catch (e) {
|
||||
console.warn("Failed to check TEE status:", e);
|
||||
}
|
||||
}
|
||||
|
||||
function fetchTeeReport() {
|
||||
|
||||
@@ -99,7 +99,7 @@
|
||||
<span id="sse-status">Connected</span>
|
||||
<div class="gateway-popover" id="gateway-popover"></div>
|
||||
</div>
|
||||
<button class="restart-btn" id="restart-btn" onclick="triggerRestart()" title="Gracefully restart the process">
|
||||
<button class="restart-btn" id="restart-btn" onclick="triggerRestart()" title="Gracefully restart the process" style="display: none;">
|
||||
<svg id="restart-icon" width="13" height="13" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">
|
||||
<path d="M23 4v6h-6"></path>
|
||||
<path d="M1 20v-6h6"></path>
|
||||
@@ -113,12 +113,12 @@
|
||||
<div class="tab-panel active" id="tab-chat">
|
||||
<div class="thread-sidebar" id="thread-sidebar">
|
||||
<div class="thread-sidebar-header">
|
||||
<span>Threads</span>
|
||||
<button class="thread-new-btn" onclick="createNewThread()" title="New thread (Ctrl/Cmd+N)">+</button>
|
||||
<div class="spacer"></div>
|
||||
<button class="thread-toggle-btn" id="thread-toggle-btn" onclick="toggleThreadSidebar()" title="Toggle sidebar">«</button>
|
||||
</div>
|
||||
<div class="assistant-item" id="assistant-thread" onclick="switchToAssistant()">
|
||||
<span class="assistant-label">Assistant</span>
|
||||
<span class="assistant-label" id="assistant-label">Assistant</span>
|
||||
<span class="assistant-meta" id="assistant-meta"></span>
|
||||
</div>
|
||||
<div class="threads-section-header">
|
||||
@@ -130,7 +130,10 @@
|
||||
<div class="chat-messages" id="chat-messages"></div>
|
||||
<div id="slash-autocomplete" class="slash-autocomplete" style="display:none"></div>
|
||||
<div class="chat-input">
|
||||
<div id="image-preview-strip" class="image-preview-strip"></div>
|
||||
<textarea id="chat-input" placeholder="Message or / for commands..." rows="1"></textarea>
|
||||
<input type="file" id="image-file-input" accept="image/*" multiple style="display:none">
|
||||
<button id="attach-btn" class="attach-btn" title="Attach images" aria-label="Attach images">📎</button>
|
||||
<button id="send-btn" onclick="sendMessage()">Send</button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
@@ -159,7 +159,7 @@ body {
|
||||
flex-shrink: 0;
|
||||
}
|
||||
|
||||
.tab-bar button:not(.status-logs-btn) {
|
||||
.tab-bar button:not(.status-logs-btn):not(.restart-btn) {
|
||||
padding: 10px 20px;
|
||||
background: none;
|
||||
border: none;
|
||||
@@ -171,11 +171,11 @@ body {
|
||||
transition: color 0.2s, border-color 0.2s;
|
||||
}
|
||||
|
||||
.tab-bar button:not(.status-logs-btn):hover {
|
||||
.tab-bar button:not(.status-logs-btn):not(.restart-btn):hover {
|
||||
color: var(--text);
|
||||
}
|
||||
|
||||
.tab-bar button:not(.status-logs-btn).active {
|
||||
.tab-bar button:not(.status-logs-btn):not(.restart-btn).active {
|
||||
color: var(--accent);
|
||||
border-bottom-color: var(--accent);
|
||||
}
|
||||
@@ -260,42 +260,42 @@ body {
|
||||
}
|
||||
|
||||
/* Restart Button */
|
||||
.restart-btn {
|
||||
.tab-bar .restart-btn {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 0.375rem;
|
||||
margin: 0.375rem;
|
||||
padding: 0.25rem 0.75rem;
|
||||
border-radius: 0.5rem;
|
||||
font-size: 0.8rem;
|
||||
border: 1px solid;
|
||||
border-color: #00d894;
|
||||
border: 1px solid #00d894;
|
||||
color: #00d894;
|
||||
background-color: transparent;
|
||||
cursor: pointer;
|
||||
transition: color 150ms, background-color 150ms, border-color 150ms;
|
||||
}
|
||||
|
||||
.restart-btn:hover:not(:disabled) {
|
||||
.tab-bar .restart-btn:hover:not(:disabled) {
|
||||
background-color: rgba(0, 216, 148, 0.1);
|
||||
}
|
||||
|
||||
.restart-btn:disabled {
|
||||
.tab-bar .restart-btn:disabled {
|
||||
border-color: #333;
|
||||
color: #666;
|
||||
cursor: not-allowed;
|
||||
}
|
||||
|
||||
.restart-btn:disabled:hover {
|
||||
.tab-bar .restart-btn:disabled:hover {
|
||||
background-color: transparent;
|
||||
}
|
||||
|
||||
.restart-btn svg {
|
||||
.tab-bar .restart-btn svg {
|
||||
flex-shrink: 0;
|
||||
width: 13px;
|
||||
height: 13px;
|
||||
}
|
||||
|
||||
.restart-btn svg.spinning {
|
||||
.tab-bar .restart-btn svg.spinning {
|
||||
animation: spin-icon 1s linear infinite;
|
||||
}
|
||||
|
||||
@@ -1272,6 +1272,7 @@ body {
|
||||
/* Chat input */
|
||||
.chat-input {
|
||||
display: flex;
|
||||
flex-wrap: wrap;
|
||||
padding: 12px 16px max(12px, env(safe-area-inset-bottom)) 16px;
|
||||
gap: 8px;
|
||||
background: var(--bg-secondary);
|
||||
@@ -3074,7 +3075,7 @@ mark {
|
||||
}
|
||||
|
||||
.thread-sidebar {
|
||||
width: 200px;
|
||||
width: 240px;
|
||||
background: var(--bg-secondary);
|
||||
border-right: 1px solid var(--border);
|
||||
display: flex;
|
||||
@@ -3082,6 +3083,8 @@ mark {
|
||||
flex-shrink: 0;
|
||||
transition: width 0.2s ease;
|
||||
overflow: hidden;
|
||||
padding: 6px;
|
||||
gap: 2px;
|
||||
}
|
||||
|
||||
.thread-sidebar.collapsed {
|
||||
@@ -3099,8 +3102,7 @@ mark {
|
||||
.thread-sidebar-header {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
padding: 10px 12px;
|
||||
border-bottom: 1px solid var(--border);
|
||||
padding: 10px 10px;
|
||||
font-size: 13px;
|
||||
font-weight: 600;
|
||||
gap: 8px;
|
||||
@@ -3134,21 +3136,22 @@ mark {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: space-between;
|
||||
padding: 10px 12px;
|
||||
padding: 12px 14px;
|
||||
cursor: pointer;
|
||||
font-size: 13px;
|
||||
font-weight: 600;
|
||||
color: var(--text);
|
||||
border-bottom: 1px solid var(--border);
|
||||
background: var(--bg-secondary);
|
||||
background: var(--bg-tertiary);
|
||||
border-radius: var(--radius);
|
||||
margin-bottom: 2px;
|
||||
}
|
||||
|
||||
.assistant-item:hover {
|
||||
background: var(--bg-tertiary);
|
||||
background: rgba(255, 255, 255, 0.06);
|
||||
}
|
||||
|
||||
.assistant-item.active {
|
||||
background: rgba(52, 211, 153, 0.08);
|
||||
background: rgba(52, 211, 153, 0.1);
|
||||
color: var(--accent);
|
||||
border-left: 2px solid var(--accent);
|
||||
}
|
||||
@@ -3166,7 +3169,7 @@ mark {
|
||||
}
|
||||
|
||||
.threads-section-header {
|
||||
padding: 8px 12px 4px;
|
||||
padding: 10px 10px 4px;
|
||||
font-size: 11px;
|
||||
font-weight: 500;
|
||||
text-transform: uppercase;
|
||||
@@ -3196,11 +3199,11 @@ mark {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: space-between;
|
||||
padding: 8px 12px;
|
||||
padding: 10px 14px;
|
||||
cursor: pointer;
|
||||
font-size: 13px;
|
||||
color: var(--text-secondary);
|
||||
border-bottom: 1px solid rgba(255, 255, 255, 0.03);
|
||||
border-radius: var(--radius);
|
||||
}
|
||||
|
||||
.thread-item:hover {
|
||||
@@ -3222,6 +3225,43 @@ mark {
|
||||
.thread-meta {
|
||||
font-size: 11px;
|
||||
color: var(--text-secondary);
|
||||
flex-shrink: 0;
|
||||
}
|
||||
|
||||
.thread-badge {
|
||||
display: inline-block;
|
||||
font-size: 9px;
|
||||
font-weight: 600;
|
||||
text-transform: uppercase;
|
||||
letter-spacing: 0.5px;
|
||||
padding: 1px 5px;
|
||||
border-radius: 3px;
|
||||
background: rgba(255, 255, 255, 0.08);
|
||||
color: var(--text-secondary);
|
||||
margin-right: 6px;
|
||||
flex-shrink: 0;
|
||||
}
|
||||
|
||||
.thread-badge-routine { background: rgba(52, 211, 153, 0.15); color: var(--accent); }
|
||||
.thread-badge-heartbeat { background: rgba(245, 166, 35, 0.15); color: var(--warning); }
|
||||
.thread-badge-telegram { background: rgba(0, 136, 204, 0.15); color: #0088cc; }
|
||||
.thread-badge-signal { background: rgba(59, 118, 240, 0.15); color: #3b76f0; }
|
||||
.thread-badge-slack { background: rgba(74, 21, 75, 0.15); color: #e01e5a; }
|
||||
|
||||
.thread-unread {
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
min-width: 16px;
|
||||
height: 16px;
|
||||
font-size: 10px;
|
||||
font-weight: 700;
|
||||
background: var(--accent);
|
||||
color: var(--bg);
|
||||
border-radius: 8px;
|
||||
padding: 0 4px;
|
||||
margin-left: auto;
|
||||
flex-shrink: 0;
|
||||
}
|
||||
|
||||
/* --- Memory editing --- */
|
||||
@@ -3620,7 +3660,7 @@ mark {
|
||||
left: 0;
|
||||
top: 0;
|
||||
bottom: 0;
|
||||
width: 200px;
|
||||
width: 240px;
|
||||
z-index: 50;
|
||||
}
|
||||
|
||||
@@ -3722,3 +3762,93 @@ mark {
|
||||
text-overflow: ellipsis;
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
/* Image Upload */
|
||||
.attach-btn {
|
||||
background: none;
|
||||
border: none;
|
||||
cursor: pointer;
|
||||
font-size: 1.2em;
|
||||
padding: 8px;
|
||||
align-self: flex-end;
|
||||
color: var(--text-secondary);
|
||||
transition: color 0.2s;
|
||||
min-height: 40px;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
}
|
||||
|
||||
.attach-btn:hover {
|
||||
color: var(--text);
|
||||
}
|
||||
|
||||
.image-preview-strip {
|
||||
display: flex;
|
||||
flex-direction: row;
|
||||
gap: 8px;
|
||||
padding: 4px;
|
||||
overflow-x: auto;
|
||||
min-height: 0;
|
||||
width: 100%;
|
||||
}
|
||||
|
||||
.image-preview-strip:empty {
|
||||
display: none;
|
||||
}
|
||||
|
||||
.image-preview-container {
|
||||
position: relative;
|
||||
display: inline-block;
|
||||
flex-shrink: 0;
|
||||
}
|
||||
|
||||
.image-preview {
|
||||
width: 60px;
|
||||
height: 60px;
|
||||
border-radius: 6px;
|
||||
object-fit: cover;
|
||||
display: block;
|
||||
}
|
||||
|
||||
.image-preview-remove {
|
||||
position: absolute;
|
||||
top: -6px;
|
||||
right: -6px;
|
||||
width: 18px;
|
||||
height: 18px;
|
||||
border-radius: 50%;
|
||||
background: var(--danger);
|
||||
color: #fff;
|
||||
border: none;
|
||||
font-size: 12px;
|
||||
line-height: 18px;
|
||||
text-align: center;
|
||||
cursor: pointer;
|
||||
padding: 0;
|
||||
}
|
||||
|
||||
.image-preview-remove:hover {
|
||||
background: #c33;
|
||||
}
|
||||
|
||||
/* Generated Image */
|
||||
.generated-image-card {
|
||||
max-width: 512px;
|
||||
margin: 8px 0;
|
||||
border-radius: 8px;
|
||||
overflow: hidden;
|
||||
border: 1px solid var(--border);
|
||||
}
|
||||
|
||||
.generated-image {
|
||||
max-width: 100%;
|
||||
display: block;
|
||||
}
|
||||
|
||||
.generated-image-path {
|
||||
font-size: 12px;
|
||||
color: var(--text-secondary);
|
||||
padding: 4px 8px;
|
||||
background: var(--bg-secondary);
|
||||
}
|
||||
|
||||
@@ -84,6 +84,7 @@ impl TestGatewayBuilder {
|
||||
chat_rate_limiter: RateLimiter::new(30, 60),
|
||||
registry_entries: Vec::new(),
|
||||
cost_guard: None,
|
||||
routine_engine: Arc::new(tokio::sync::RwLock::new(None)),
|
||||
startup_time: std::time::Instant::now(),
|
||||
})
|
||||
}
|
||||
|
||||
@@ -5,10 +5,23 @@ use uuid::Uuid;
|
||||
|
||||
// --- Chat ---
|
||||
|
||||
/// Base64-encoded image data sent from the web frontend.
|
||||
#[derive(Debug, Clone, Deserialize)]
|
||||
pub struct ImageData {
|
||||
/// MIME type (e.g., "image/png", "image/jpeg").
|
||||
pub media_type: String,
|
||||
/// Base64-encoded image data (without data: URL prefix).
|
||||
pub data: String,
|
||||
}
|
||||
|
||||
#[derive(Debug, Deserialize)]
|
||||
pub struct SendMessageRequest {
|
||||
pub content: String,
|
||||
pub thread_id: Option<String>,
|
||||
pub timezone: Option<String>,
|
||||
/// Optional images attached to the message.
|
||||
#[serde(default)]
|
||||
pub images: Vec<ImageData>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Serialize)]
|
||||
@@ -28,6 +41,8 @@ pub struct ThreadInfo {
|
||||
pub title: Option<String>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub thread_type: Option<String>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub channel: Option<String>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Serialize)]
|
||||
@@ -217,6 +232,16 @@ pub enum SseEvent {
|
||||
session_id: Option<String>,
|
||||
},
|
||||
|
||||
/// An image was generated by a tool.
|
||||
#[serde(rename = "image_generated")]
|
||||
ImageGenerated {
|
||||
data_url: String,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
path: Option<String>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
thread_id: Option<String>,
|
||||
},
|
||||
|
||||
/// Extension activation status change (WASM channels).
|
||||
#[serde(rename = "extension_status")]
|
||||
ExtensionStatus {
|
||||
@@ -611,6 +636,10 @@ pub enum WsClientMessage {
|
||||
Message {
|
||||
content: String,
|
||||
thread_id: Option<String>,
|
||||
timezone: Option<String>,
|
||||
/// Optional images attached to the message.
|
||||
#[serde(default)]
|
||||
images: Vec<ImageData>,
|
||||
},
|
||||
/// Approve or deny a pending tool execution.
|
||||
#[serde(rename = "approval")]
|
||||
@@ -677,6 +706,7 @@ impl WsServerMessage {
|
||||
SseEvent::JobToolResult { .. } => "job_tool_result",
|
||||
SseEvent::JobStatus { .. } => "job_status",
|
||||
SseEvent::JobResult { .. } => "job_result",
|
||||
SseEvent::ImageGenerated { .. } => "image_generated",
|
||||
SseEvent::ExtensionStatus { .. } => "extension_status",
|
||||
};
|
||||
let data = serde_json::to_value(event).unwrap_or(serde_json::Value::Null);
|
||||
@@ -746,6 +776,7 @@ pub struct RoutineRunInfo {
|
||||
pub status: String,
|
||||
pub result_summary: Option<String>,
|
||||
pub tokens_used: Option<i32>,
|
||||
pub job_id: Option<Uuid>,
|
||||
}
|
||||
|
||||
// --- Settings ---
|
||||
@@ -796,7 +827,9 @@ mod tests {
|
||||
let json = r#"{"type":"message","content":"hello","thread_id":"t1"}"#;
|
||||
let msg: WsClientMessage = serde_json::from_str(json).unwrap();
|
||||
match msg {
|
||||
WsClientMessage::Message { content, thread_id } => {
|
||||
WsClientMessage::Message {
|
||||
content, thread_id, ..
|
||||
} => {
|
||||
assert_eq!(content, "hello");
|
||||
assert_eq!(thread_id.as_deref(), Some("t1"));
|
||||
}
|
||||
@@ -809,7 +842,9 @@ mod tests {
|
||||
let json = r#"{"type":"message","content":"hi"}"#;
|
||||
let msg: WsClientMessage = serde_json::from_str(json).unwrap();
|
||||
match msg {
|
||||
WsClientMessage::Message { content, thread_id } => {
|
||||
WsClientMessage::Message {
|
||||
content, thread_id, ..
|
||||
} => {
|
||||
assert_eq!(content, "hi");
|
||||
assert!(thread_id.is_none());
|
||||
}
|
||||
@@ -1063,4 +1098,40 @@ mod tests {
|
||||
let req: AuthCancelRequest = serde_json::from_str(json).unwrap();
|
||||
assert_eq!(req.extension_name, "telegram");
|
||||
}
|
||||
|
||||
// ---- ThreadInfo channel field tests ----
|
||||
|
||||
#[test]
|
||||
fn test_thread_info_channel_serialized() {
|
||||
let info = ThreadInfo {
|
||||
id: Uuid::nil(),
|
||||
state: "Idle".to_string(),
|
||||
turn_count: 0,
|
||||
created_at: "2026-01-01T00:00:00Z".to_string(),
|
||||
updated_at: "2026-01-01T00:00:00Z".to_string(),
|
||||
title: None,
|
||||
thread_type: None,
|
||||
channel: Some("telegram".to_string()),
|
||||
};
|
||||
let json = serde_json::to_string(&info).unwrap();
|
||||
let parsed: serde_json::Value = serde_json::from_str(&json).unwrap();
|
||||
assert_eq!(parsed["channel"], "telegram");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_thread_info_channel_omitted_when_none() {
|
||||
let info = ThreadInfo {
|
||||
id: Uuid::nil(),
|
||||
state: "Idle".to_string(),
|
||||
turn_count: 0,
|
||||
created_at: "2026-01-01T00:00:00Z".to_string(),
|
||||
updated_at: "2026-01-01T00:00:00Z".to_string(),
|
||||
title: None,
|
||||
thread_type: None,
|
||||
channel: None,
|
||||
};
|
||||
let json = serde_json::to_string(&info).unwrap();
|
||||
let parsed: serde_json::Value = serde_json::from_str(&json).unwrap();
|
||||
assert!(parsed.get("channel").is_none());
|
||||
}
|
||||
}
|
||||
|
||||
@@ -3,6 +3,10 @@
|
||||
use crate::channels::web::types::{ToolCallInfo, TurnInfo};
|
||||
|
||||
/// Truncate a string to at most `max_bytes` bytes at a char boundary, appending "...".
|
||||
///
|
||||
/// If the input is wrapped in `<tool_output …>…</tool_output>` and truncation
|
||||
/// removes the closing tag, the tag is re-appended so downstream XML parsers
|
||||
/// never see an unclosed element.
|
||||
pub fn truncate_preview(s: &str, max_bytes: usize) -> String {
|
||||
if s.len() <= max_bytes {
|
||||
return s.to_string();
|
||||
@@ -12,7 +16,14 @@ pub fn truncate_preview(s: &str, max_bytes: usize) -> String {
|
||||
while end > 0 && !s.is_char_boundary(end) {
|
||||
end -= 1;
|
||||
}
|
||||
format!("{}...", &s[..end])
|
||||
let mut result = format!("{}...", &s[..end]);
|
||||
|
||||
// Re-close <tool_output> if truncation cut through the closing tag.
|
||||
if s.starts_with("<tool_output") && !result.ends_with("</tool_output>") {
|
||||
result.push_str("\n</tool_output>");
|
||||
}
|
||||
|
||||
result
|
||||
}
|
||||
|
||||
/// Build TurnInfo pairs from flat DB messages (user/tool_calls/assistant triples).
|
||||
@@ -83,6 +94,19 @@ pub fn build_turns_from_db_messages(
|
||||
|
||||
turns.push(turn);
|
||||
turn_number += 1;
|
||||
} else if msg.role == "assistant" {
|
||||
// Standalone assistant message (e.g. routine output, heartbeat)
|
||||
// with no preceding user message — render as a turn with empty input.
|
||||
turns.push(TurnInfo {
|
||||
turn_number,
|
||||
user_input: String::new(),
|
||||
response: Some(msg.content.clone()),
|
||||
state: "Completed".to_string(),
|
||||
started_at: msg.created_at.to_rfc3339(),
|
||||
completed_at: Some(msg.created_at.to_rfc3339()),
|
||||
tool_calls: Vec::new(),
|
||||
});
|
||||
turn_number += 1;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -149,6 +173,33 @@ mod tests {
|
||||
assert_eq!(truncate_preview("hello", 0), "...");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_truncate_preview_closes_tool_output_tag() {
|
||||
let s = "<tool_output name=\"search\" sanitized=\"true\">\nSome very long content here\n</tool_output>";
|
||||
// Truncate so it cuts before the closing tag
|
||||
let result = truncate_preview(s, 60);
|
||||
assert!(result.ends_with("</tool_output>"));
|
||||
assert!(result.contains("..."));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_truncate_preview_no_extra_close_when_intact() {
|
||||
let s = "<tool_output name=\"echo\" sanitized=\"false\">\nshort\n</tool_output>";
|
||||
// The string is short enough not to be truncated
|
||||
let result = truncate_preview(s, 500);
|
||||
assert_eq!(result, s);
|
||||
// Should not have a duplicate closing tag
|
||||
assert_eq!(result.matches("</tool_output>").count(), 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_truncate_preview_non_xml_unaffected() {
|
||||
let s = "Just a plain long string that gets truncated";
|
||||
let result = truncate_preview(s, 10);
|
||||
assert_eq!(result, "Just a pla...");
|
||||
assert!(!result.contains("</tool_output>"));
|
||||
}
|
||||
|
||||
// ---- build_turns_from_db_messages tests ----
|
||||
|
||||
fn make_msg(role: &str, content: &str, offset_ms: i64) -> crate::history::ConversationMessage {
|
||||
@@ -220,6 +271,29 @@ mod tests {
|
||||
assert_eq!(turns[0].response.as_deref(), Some("Done"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_build_turns_standalone_assistant_messages() {
|
||||
// Routine conversations only have assistant messages (no user messages).
|
||||
let messages = vec![
|
||||
make_msg("assistant", "Routine executed: all checks passed", 0),
|
||||
make_msg("assistant", "Routine executed: found 2 issues", 5000),
|
||||
];
|
||||
let turns = build_turns_from_db_messages(&messages);
|
||||
assert_eq!(turns.len(), 2);
|
||||
// Standalone assistant messages should have empty user_input
|
||||
assert_eq!(turns[0].user_input, "");
|
||||
assert_eq!(
|
||||
turns[0].response.as_deref(),
|
||||
Some("Routine executed: all checks passed")
|
||||
);
|
||||
assert_eq!(turns[0].state, "Completed");
|
||||
assert_eq!(turns[1].user_input, "");
|
||||
assert_eq!(
|
||||
turns[1].response.as_deref(),
|
||||
Some("Routine executed: found 2 issues")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_build_turns_backward_compatible() {
|
||||
let messages = vec![
|
||||
|
||||
+20
-1
@@ -156,12 +156,26 @@ async fn handle_client_message(
|
||||
direct_tx: &mpsc::Sender<WsServerMessage>,
|
||||
) {
|
||||
match msg {
|
||||
WsClientMessage::Message { content, thread_id } => {
|
||||
WsClientMessage::Message {
|
||||
content,
|
||||
thread_id,
|
||||
timezone,
|
||||
images,
|
||||
} => {
|
||||
let mut incoming = IncomingMessage::new("gateway", user_id, &content);
|
||||
if let Some(ref tz) = timezone {
|
||||
incoming = incoming.with_timezone(tz);
|
||||
}
|
||||
if let Some(ref tid) = thread_id {
|
||||
incoming = incoming.with_thread(tid);
|
||||
}
|
||||
|
||||
// Convert uploaded images to IncomingAttachments
|
||||
if !images.is_empty() {
|
||||
let attachments = crate::channels::web::server::images_to_attachments(&images);
|
||||
incoming = incoming.with_attachments(attachments);
|
||||
}
|
||||
|
||||
let tx_guard = state.msg_tx.read().await;
|
||||
if let Some(ref tx) = *tx_guard {
|
||||
if tx.send(incoming).await.is_err() {
|
||||
@@ -349,6 +363,8 @@ mod tests {
|
||||
WsClientMessage::Message {
|
||||
content: "hello agent".to_string(),
|
||||
thread_id: Some("t1".to_string()),
|
||||
timezone: None,
|
||||
images: Vec::new(),
|
||||
},
|
||||
&state,
|
||||
"user1",
|
||||
@@ -373,6 +389,8 @@ mod tests {
|
||||
WsClientMessage::Message {
|
||||
content: "hello".to_string(),
|
||||
thread_id: None,
|
||||
timezone: None,
|
||||
images: Vec::new(),
|
||||
},
|
||||
&state,
|
||||
"user1",
|
||||
@@ -493,6 +511,7 @@ mod tests {
|
||||
chat_rate_limiter: crate::channels::web::server::RateLimiter::new(30, 60),
|
||||
registry_entries: Vec::new(),
|
||||
cost_guard: None,
|
||||
routine_engine: Arc::new(tokio::sync::RwLock::new(None)),
|
||||
startup_time: std::time::Instant::now(),
|
||||
}
|
||||
}
|
||||
|
||||
+1
-1
@@ -107,7 +107,7 @@ enum CheckResult {
|
||||
|
||||
async fn check_nearai_session() -> CheckResult {
|
||||
// Check if session file exists
|
||||
let session_path = crate::llm::session::default_session_path();
|
||||
let session_path = crate::config::llm::default_session_path();
|
||||
if !session_path.exists() {
|
||||
// Check for API key mode
|
||||
if std::env::var("NEARAI_API_KEY").is_ok() {
|
||||
|
||||
+243
-121
@@ -2,52 +2,79 @@
|
||||
//!
|
||||
//! Commands for adding, removing, authenticating, and testing MCP servers.
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::io::Write;
|
||||
use std::sync::Arc;
|
||||
|
||||
use clap::Subcommand;
|
||||
use clap::{Args, Subcommand};
|
||||
|
||||
use crate::config::Config;
|
||||
use crate::db::Database;
|
||||
#[cfg(feature = "postgres")]
|
||||
use crate::secrets::PostgresSecretsStore;
|
||||
use crate::secrets::{SecretsCrypto, SecretsStore};
|
||||
use crate::tools::mcp::{
|
||||
McpClient, McpServerConfig, McpSessionManager, OAuthConfig,
|
||||
auth::{authorize_mcp_server, is_authenticated},
|
||||
config::{self, McpServersFile},
|
||||
config::{self, EffectiveTransport, McpServersFile},
|
||||
};
|
||||
|
||||
/// Arguments for the `mcp add` subcommand.
|
||||
#[derive(Args, Debug, Clone)]
|
||||
pub struct McpAddArgs {
|
||||
/// Server name (e.g., "notion", "github")
|
||||
pub name: String,
|
||||
|
||||
/// Server URL (e.g., "https://mcp.notion.com") -- required for http transport
|
||||
pub url: Option<String>,
|
||||
|
||||
/// Transport type: http (default), stdio, unix
|
||||
#[arg(long, default_value = "http")]
|
||||
pub transport: String,
|
||||
|
||||
/// Command to run (stdio transport)
|
||||
#[arg(long)]
|
||||
pub command: Option<String>,
|
||||
|
||||
/// Command arguments (stdio transport, can be repeated)
|
||||
#[arg(long = "arg", num_args = 1..)]
|
||||
pub cmd_args: Vec<String>,
|
||||
|
||||
/// Environment variables (stdio transport, KEY=VALUE format, can be repeated)
|
||||
#[arg(long = "env", value_parser = parse_env_var)]
|
||||
pub env: Vec<(String, String)>,
|
||||
|
||||
/// Unix socket path (unix transport)
|
||||
#[arg(long)]
|
||||
pub socket: Option<String>,
|
||||
|
||||
/// Custom HTTP headers (KEY:VALUE format, can be repeated)
|
||||
#[arg(long = "header", value_parser = parse_header)]
|
||||
pub headers: Vec<(String, String)>,
|
||||
|
||||
/// OAuth client ID (if authentication is required)
|
||||
#[arg(long)]
|
||||
pub client_id: Option<String>,
|
||||
|
||||
/// OAuth authorization URL (optional, can be discovered)
|
||||
#[arg(long)]
|
||||
pub auth_url: Option<String>,
|
||||
|
||||
/// OAuth token URL (optional, can be discovered)
|
||||
#[arg(long)]
|
||||
pub token_url: Option<String>,
|
||||
|
||||
/// Scopes to request (comma-separated)
|
||||
#[arg(long)]
|
||||
pub scopes: Option<String>,
|
||||
|
||||
/// Server description
|
||||
#[arg(long)]
|
||||
pub description: Option<String>,
|
||||
}
|
||||
|
||||
#[derive(Subcommand, Debug, Clone)]
|
||||
pub enum McpCommand {
|
||||
/// Add an MCP server
|
||||
Add {
|
||||
/// Server name (e.g., "notion", "github")
|
||||
name: String,
|
||||
|
||||
/// Server URL (e.g., "https://mcp.notion.com")
|
||||
url: String,
|
||||
|
||||
/// OAuth client ID (if authentication is required)
|
||||
#[arg(long)]
|
||||
client_id: Option<String>,
|
||||
|
||||
/// OAuth authorization URL (optional, can be discovered)
|
||||
#[arg(long)]
|
||||
auth_url: Option<String>,
|
||||
|
||||
/// OAuth token URL (optional, can be discovered)
|
||||
#[arg(long)]
|
||||
token_url: Option<String>,
|
||||
|
||||
/// Scopes to request (comma-separated)
|
||||
#[arg(long)]
|
||||
scopes: Option<String>,
|
||||
|
||||
/// Server description
|
||||
#[arg(long)]
|
||||
description: Option<String>,
|
||||
},
|
||||
Add(Box<McpAddArgs>),
|
||||
|
||||
/// Remove an MCP server
|
||||
Remove {
|
||||
@@ -97,29 +124,24 @@ pub enum McpCommand {
|
||||
},
|
||||
}
|
||||
|
||||
fn parse_header(s: &str) -> Result<(String, String), String> {
|
||||
let pos = s
|
||||
.find(':')
|
||||
.ok_or_else(|| format!("invalid header format '{}', expected KEY:VALUE", s))?;
|
||||
Ok((s[..pos].trim().to_string(), s[pos + 1..].trim().to_string()))
|
||||
}
|
||||
|
||||
fn parse_env_var(s: &str) -> Result<(String, String), String> {
|
||||
let pos = s
|
||||
.find('=')
|
||||
.ok_or_else(|| format!("invalid env var format '{}', expected KEY=VALUE", s))?;
|
||||
Ok((s[..pos].to_string(), s[pos + 1..].to_string()))
|
||||
}
|
||||
|
||||
/// Run an MCP command.
|
||||
pub async fn run_mcp_command(cmd: McpCommand) -> anyhow::Result<()> {
|
||||
match cmd {
|
||||
McpCommand::Add {
|
||||
name,
|
||||
url,
|
||||
client_id,
|
||||
auth_url,
|
||||
token_url,
|
||||
scopes,
|
||||
description,
|
||||
} => {
|
||||
add_server(
|
||||
name,
|
||||
url,
|
||||
client_id,
|
||||
auth_url,
|
||||
token_url,
|
||||
scopes,
|
||||
description,
|
||||
)
|
||||
.await
|
||||
}
|
||||
McpCommand::Add(args) => add_server(*args).await,
|
||||
McpCommand::Remove { name } => remove_server(name).await,
|
||||
McpCommand::List { verbose } => list_servers(verbose).await,
|
||||
McpCommand::Auth { name, user } => auth_server(name, user).await,
|
||||
@@ -133,16 +155,58 @@ pub async fn run_mcp_command(cmd: McpCommand) -> anyhow::Result<()> {
|
||||
}
|
||||
|
||||
/// Add a new MCP server.
|
||||
async fn add_server(
|
||||
name: String,
|
||||
url: String,
|
||||
client_id: Option<String>,
|
||||
auth_url: Option<String>,
|
||||
token_url: Option<String>,
|
||||
scopes: Option<String>,
|
||||
description: Option<String>,
|
||||
) -> anyhow::Result<()> {
|
||||
let mut config = McpServerConfig::new(&name, &url);
|
||||
async fn add_server(args: McpAddArgs) -> anyhow::Result<()> {
|
||||
let McpAddArgs {
|
||||
name,
|
||||
url,
|
||||
transport,
|
||||
command,
|
||||
cmd_args,
|
||||
env,
|
||||
socket,
|
||||
headers,
|
||||
client_id,
|
||||
auth_url,
|
||||
token_url,
|
||||
scopes,
|
||||
description,
|
||||
} = args;
|
||||
|
||||
let transport_lower = transport.to_lowercase();
|
||||
|
||||
let mut config = match transport_lower.as_str() {
|
||||
"stdio" => {
|
||||
let cmd = command
|
||||
.clone()
|
||||
.ok_or_else(|| anyhow::anyhow!("--command is required for stdio transport"))?;
|
||||
let env_map: HashMap<String, String> = env.into_iter().collect();
|
||||
McpServerConfig::new_stdio(&name, &cmd, cmd_args.clone(), env_map)
|
||||
}
|
||||
"unix" => {
|
||||
let socket_path = socket
|
||||
.clone()
|
||||
.ok_or_else(|| anyhow::anyhow!("--socket is required for unix transport"))?;
|
||||
McpServerConfig::new_unix(&name, &socket_path)
|
||||
}
|
||||
"http" => {
|
||||
let url_val = url
|
||||
.as_deref()
|
||||
.ok_or_else(|| anyhow::anyhow!("URL is required for http transport"))?;
|
||||
McpServerConfig::new(&name, url_val)
|
||||
}
|
||||
other => {
|
||||
anyhow::bail!(
|
||||
"Unknown transport type '{}'. Supported: http, stdio, unix",
|
||||
other
|
||||
);
|
||||
}
|
||||
};
|
||||
|
||||
// Apply headers if any
|
||||
if !headers.is_empty() {
|
||||
let headers_map: HashMap<String, String> = headers.into_iter().collect();
|
||||
config = config.with_headers(headers_map);
|
||||
}
|
||||
|
||||
if let Some(desc) = description {
|
||||
config = config.with_description(desc);
|
||||
@@ -151,8 +215,12 @@ async fn add_server(
|
||||
// Track if auth is required
|
||||
let requires_auth = client_id.is_some();
|
||||
|
||||
// Set up OAuth if client_id is provided
|
||||
// Set up OAuth if client_id is provided (HTTP transport only)
|
||||
if let Some(client_id) = client_id {
|
||||
if transport_lower != "http" {
|
||||
anyhow::bail!("OAuth authentication is only supported with http transport");
|
||||
}
|
||||
|
||||
let mut oauth = OAuthConfig::new(client_id);
|
||||
|
||||
if let (Some(auth), Some(token)) = (auth_url, token_url) {
|
||||
@@ -181,7 +249,24 @@ async fn add_server(
|
||||
|
||||
println!();
|
||||
println!(" ✓ Added MCP server '{}'", name);
|
||||
println!(" URL: {}", url);
|
||||
|
||||
match transport_lower.as_str() {
|
||||
"stdio" => {
|
||||
println!(
|
||||
" Transport: stdio (command: {})",
|
||||
command.as_deref().unwrap_or("")
|
||||
);
|
||||
}
|
||||
"unix" => {
|
||||
println!(
|
||||
" Transport: unix (socket: {})",
|
||||
socket.as_deref().unwrap_or("")
|
||||
);
|
||||
}
|
||||
_ => {
|
||||
println!(" URL: {}", url.as_deref().unwrap_or(""));
|
||||
}
|
||||
}
|
||||
|
||||
if requires_auth {
|
||||
println!();
|
||||
@@ -236,9 +321,40 @@ async fn list_servers(verbose: bool) -> anyhow::Result<()> {
|
||||
""
|
||||
};
|
||||
|
||||
let effective = server.effective_transport();
|
||||
|
||||
let transport_label = match &effective {
|
||||
EffectiveTransport::Http => "http".to_string(),
|
||||
EffectiveTransport::Stdio { command, .. } => {
|
||||
format!("stdio ({})", command)
|
||||
}
|
||||
EffectiveTransport::Unix { socket_path } => {
|
||||
format!("unix ({})", socket_path)
|
||||
}
|
||||
};
|
||||
|
||||
if verbose {
|
||||
println!(" {} {}{}", status, server.name, auth_status);
|
||||
println!(" URL: {}", server.url);
|
||||
println!(" Transport: {}", transport_label);
|
||||
match &effective {
|
||||
EffectiveTransport::Http => {
|
||||
println!(" URL: {}", server.url);
|
||||
}
|
||||
EffectiveTransport::Stdio { command, args, env } => {
|
||||
println!(" Command: {}", command);
|
||||
if !args.is_empty() {
|
||||
println!(" Args: {}", args.join(", "));
|
||||
}
|
||||
if !env.is_empty() {
|
||||
// Only print env var names, not values (may contain secrets).
|
||||
let env_keys: Vec<&str> = env.keys().map(|k| k.as_str()).collect();
|
||||
println!(" Env: {}", env_keys.join(", "));
|
||||
}
|
||||
}
|
||||
EffectiveTransport::Unix { socket_path } => {
|
||||
println!(" Socket: {}", socket_path);
|
||||
}
|
||||
}
|
||||
if let Some(ref desc) = server.description {
|
||||
println!(" Description: {}", desc);
|
||||
}
|
||||
@@ -248,11 +364,27 @@ async fn list_servers(verbose: bool) -> anyhow::Result<()> {
|
||||
println!(" Scopes: {}", oauth.scopes.join(", "));
|
||||
}
|
||||
}
|
||||
if !server.headers.is_empty() {
|
||||
let header_keys: Vec<&String> = server.headers.keys().collect();
|
||||
println!(
|
||||
" Headers: {}",
|
||||
header_keys
|
||||
.iter()
|
||||
.map(|k| k.as_str())
|
||||
.collect::<Vec<_>>()
|
||||
.join(", ")
|
||||
);
|
||||
}
|
||||
println!();
|
||||
} else {
|
||||
let display = match &effective {
|
||||
EffectiveTransport::Http => server.url.clone(),
|
||||
EffectiveTransport::Stdio { command, .. } => command.to_string(),
|
||||
EffectiveTransport::Unix { socket_path } => socket_path.to_string(),
|
||||
};
|
||||
println!(
|
||||
" {} {} - {}{}",
|
||||
status, server.name, server.url, auth_status
|
||||
" {} {} - {} [{}]{}",
|
||||
status, server.name, display, transport_label, auth_status
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -374,7 +506,7 @@ async fn test_server(name: String, user_id: String) -> anyhow::Result<()> {
|
||||
return Ok(());
|
||||
} else {
|
||||
// No OAuth and no tokens - try unauthenticated
|
||||
McpClient::new_with_name(&server.name, &server.url)
|
||||
McpClient::new_with_config(server.clone())
|
||||
};
|
||||
|
||||
// Test connection
|
||||
@@ -504,61 +636,9 @@ async fn get_secrets_store() -> anyhow::Result<Arc<dyn SecretsStore + Send + Syn
|
||||
)
|
||||
})?;
|
||||
|
||||
let crypto = SecretsCrypto::new(master_key.clone())?;
|
||||
let crypto = Arc::new(SecretsCrypto::new(master_key.clone())?);
|
||||
|
||||
#[cfg(feature = "postgres")]
|
||||
{
|
||||
let store = crate::history::Store::new(&config.database).await?;
|
||||
store.run_migrations().await?;
|
||||
Ok(Arc::new(PostgresSecretsStore::new(
|
||||
store.pool(),
|
||||
Arc::new(crypto),
|
||||
)))
|
||||
}
|
||||
|
||||
#[cfg(all(feature = "libsql", not(feature = "postgres")))]
|
||||
{
|
||||
use crate::db::Database as _;
|
||||
use crate::db::libsql::LibSqlBackend;
|
||||
use secrecy::ExposeSecret as _;
|
||||
|
||||
let default_path = crate::config::default_libsql_path();
|
||||
let db_path = config
|
||||
.database
|
||||
.libsql_path
|
||||
.as_deref()
|
||||
.unwrap_or(&default_path);
|
||||
|
||||
let backend = if let Some(ref url) = config.database.libsql_url {
|
||||
let token = config.database.libsql_auth_token.as_ref().ok_or_else(|| {
|
||||
anyhow::anyhow!("LIBSQL_AUTH_TOKEN is required when LIBSQL_URL is set")
|
||||
})?;
|
||||
LibSqlBackend::new_remote_replica(db_path, url, token.expose_secret())
|
||||
.await
|
||||
.map_err(|e| anyhow::anyhow!("{}", e))?
|
||||
} else {
|
||||
LibSqlBackend::new_local(db_path)
|
||||
.await
|
||||
.map_err(|e| anyhow::anyhow!("{}", e))?
|
||||
};
|
||||
backend
|
||||
.run_migrations()
|
||||
.await
|
||||
.map_err(|e| anyhow::anyhow!("{}", e))?;
|
||||
|
||||
Ok(Arc::new(crate::secrets::LibSqlSecretsStore::new(
|
||||
backend.shared_db(),
|
||||
Arc::new(crypto),
|
||||
)))
|
||||
}
|
||||
|
||||
#[cfg(not(any(feature = "postgres", feature = "libsql")))]
|
||||
{
|
||||
let _ = crypto;
|
||||
anyhow::bail!(
|
||||
"No database backend available for secrets. Enable 'postgres' or 'libsql' feature."
|
||||
);
|
||||
}
|
||||
Ok(crate::db::create_secrets_store(&config.database, crypto).await?)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
@@ -579,4 +659,46 @@ mod tests {
|
||||
|
||||
TestCli::command().debug_assert();
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_parse_header_valid() {
|
||||
let result = parse_header("Authorization: Bearer token123").unwrap();
|
||||
assert_eq!(result.0, "Authorization");
|
||||
assert_eq!(result.1, "Bearer token123");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_parse_header_no_spaces() {
|
||||
let result = parse_header("X-Api-Key:abc123").unwrap();
|
||||
assert_eq!(result.0, "X-Api-Key");
|
||||
assert_eq!(result.1, "abc123");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_parse_header_invalid() {
|
||||
let result = parse_header("no-colon-here");
|
||||
assert!(result.is_err());
|
||||
assert!(result.unwrap_err().contains("invalid header format"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_parse_env_var_valid() {
|
||||
let result = parse_env_var("NODE_ENV=production").unwrap();
|
||||
assert_eq!(result.0, "NODE_ENV");
|
||||
assert_eq!(result.1, "production");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_parse_env_var_with_equals_in_value() {
|
||||
let result = parse_env_var("KEY=value=with=equals").unwrap();
|
||||
assert_eq!(result.0, "KEY");
|
||||
assert_eq!(result.1, "value=with=equals");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_parse_env_var_invalid() {
|
||||
let result = parse_env_var("no-equals-here");
|
||||
assert!(result.is_err());
|
||||
assert!(result.unwrap_err().contains("invalid env var format"));
|
||||
}
|
||||
}
|
||||
|
||||
+1
-1
@@ -132,7 +132,7 @@ pub enum Command {
|
||||
about = "Manage MCP servers",
|
||||
long_about = "Add, auth, list, or test MCP servers.\nExample: ironclaw mcp add notion https://mcp.notion.com"
|
||||
)]
|
||||
Mcp(McpCommand),
|
||||
Mcp(Box<McpCommand>),
|
||||
|
||||
/// Query and manage workspace memory
|
||||
#[command(
|
||||
|
||||
+6
-342
@@ -24,8 +24,6 @@ use std::time::Duration;
|
||||
use base64::{Engine, engine::general_purpose::URL_SAFE_NO_PAD};
|
||||
use rand::RngCore;
|
||||
use sha2::{Digest, Sha256};
|
||||
use tokio::io::{AsyncBufReadExt, AsyncWriteExt, BufReader};
|
||||
use tokio::net::TcpListener;
|
||||
use tokio::sync::RwLock;
|
||||
|
||||
use crate::secrets::{CreateSecretParams, SecretsStore};
|
||||
@@ -64,259 +62,12 @@ pub fn builtin_credentials(secret_name: &str) -> Option<OAuthCredentials> {
|
||||
|
||||
// ── Shared callback server ──────────────────────────────────────────────
|
||||
|
||||
/// Fixed port for all OAuth callbacks.
|
||||
///
|
||||
/// Every redirect URI registered with providers must use this port:
|
||||
/// `http://localhost:9876/callback` (or `/auth/callback` for NEAR AI).
|
||||
pub const OAUTH_CALLBACK_PORT: u16 = 9876;
|
||||
|
||||
/// Returns the OAuth callback base URL.
|
||||
///
|
||||
/// Checks `IRONCLAW_OAUTH_CALLBACK_URL` env var first (useful for remote/VPS
|
||||
/// deployments where `127.0.0.1` is unreachable from the user's browser),
|
||||
/// then falls back to `http://{callback_host()}:{OAUTH_CALLBACK_PORT}`.
|
||||
pub fn callback_url() -> String {
|
||||
std::env::var("IRONCLAW_OAUTH_CALLBACK_URL")
|
||||
.ok()
|
||||
.filter(|v| !v.is_empty())
|
||||
.unwrap_or_else(|| format!("http://{}:{}", callback_host(), OAUTH_CALLBACK_PORT))
|
||||
}
|
||||
|
||||
/// Returns the hostname used in OAuth callback URLs.
|
||||
///
|
||||
/// Reads `OAUTH_CALLBACK_HOST` from the environment (default: `127.0.0.1`).
|
||||
///
|
||||
/// **Remote server usage:** set `OAUTH_CALLBACK_HOST` to the network interface
|
||||
/// address you want to listen on (e.g. the server's LAN IP or `0.0.0.0`).
|
||||
/// The callback listener will bind to that specific address instead of the
|
||||
/// loopback interface, so the OAuth redirect can reach an external browser.
|
||||
/// Note: this transmits the session token over plain HTTP — prefer SSH port
|
||||
/// forwarding (`ssh -L 9876:127.0.0.1:9876 user@host`) when possible.
|
||||
///
|
||||
/// # Example
|
||||
///
|
||||
/// ```bash
|
||||
/// export OAUTH_CALLBACK_HOST=203.0.113.10
|
||||
/// ironclaw login
|
||||
/// # Opens: http://203.0.113.10:9876/auth/callback
|
||||
/// ```
|
||||
pub fn callback_host() -> String {
|
||||
std::env::var("OAUTH_CALLBACK_HOST").unwrap_or_else(|_| "127.0.0.1".to_string())
|
||||
}
|
||||
|
||||
/// Returns `true` if `host` is a loopback address that only accepts local connections.
|
||||
///
|
||||
/// Covers `localhost` (case-insensitive), the full `127.0.0.0/8` IPv4 loopback
|
||||
/// range, and `::1` for IPv6.
|
||||
pub fn is_loopback_host(host: &str) -> bool {
|
||||
if host.eq_ignore_ascii_case("localhost") {
|
||||
return true;
|
||||
}
|
||||
host.parse::<std::net::IpAddr>()
|
||||
.map(|ip| ip.is_loopback())
|
||||
.unwrap_or(false)
|
||||
}
|
||||
|
||||
/// Error from the OAuth callback listener.
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum OAuthCallbackError {
|
||||
#[error("Port {0} is in use (another auth flow running?): {1}")]
|
||||
PortInUse(u16, String),
|
||||
|
||||
#[error("Authorization denied by user")]
|
||||
Denied,
|
||||
|
||||
#[error("Timed out waiting for authorization")]
|
||||
Timeout,
|
||||
|
||||
#[error("CSRF state mismatch: expected {expected}, got {actual}")]
|
||||
StateMismatch { expected: String, actual: String },
|
||||
|
||||
#[error("IO error: {0}")]
|
||||
Io(String),
|
||||
}
|
||||
|
||||
/// Map a `std::io::Error` from a bind attempt to an `OAuthCallbackError`.
|
||||
fn bind_error(e: std::io::Error) -> OAuthCallbackError {
|
||||
if e.kind() == std::io::ErrorKind::AddrInUse {
|
||||
OAuthCallbackError::PortInUse(OAUTH_CALLBACK_PORT, e.to_string())
|
||||
} else {
|
||||
OAuthCallbackError::Io(e.to_string())
|
||||
}
|
||||
}
|
||||
|
||||
/// Bind the OAuth callback listener on the fixed port.
|
||||
///
|
||||
/// When `OAUTH_CALLBACK_HOST` is a loopback address (the default `127.0.0.1`),
|
||||
/// binds to `127.0.0.1` first and falls back to `[::1]` so local-only auth
|
||||
/// flows remain restricted to the local machine.
|
||||
///
|
||||
/// When `OAUTH_CALLBACK_HOST` is set to a remote address, binds to that
|
||||
/// specific address so only connections directed to it are accepted.
|
||||
pub async fn bind_callback_listener() -> Result<TcpListener, OAuthCallbackError> {
|
||||
let host = callback_host();
|
||||
|
||||
if is_loopback_host(&host) {
|
||||
// Local mode: prefer IPv4 loopback, fall back to IPv6.
|
||||
let ipv4_addr = format!("127.0.0.1:{}", OAUTH_CALLBACK_PORT);
|
||||
match TcpListener::bind(&ipv4_addr).await {
|
||||
Ok(listener) => return Ok(listener),
|
||||
Err(e) if e.kind() == std::io::ErrorKind::AddrInUse => {
|
||||
return Err(OAuthCallbackError::PortInUse(
|
||||
OAUTH_CALLBACK_PORT,
|
||||
e.to_string(),
|
||||
));
|
||||
}
|
||||
Err(_) => {
|
||||
// IPv4 not available, fall back to IPv6
|
||||
}
|
||||
}
|
||||
TcpListener::bind(format!("[::1]:{}", OAUTH_CALLBACK_PORT))
|
||||
.await
|
||||
.map_err(bind_error)
|
||||
} else {
|
||||
// Remote mode: bind to the specific configured host address only,
|
||||
// not 0.0.0.0, to limit exposure to the intended interface.
|
||||
let addr = format!("{}:{}", host, OAUTH_CALLBACK_PORT);
|
||||
TcpListener::bind(&addr).await.map_err(bind_error)
|
||||
}
|
||||
}
|
||||
|
||||
/// Wait for an OAuth callback and extract a query parameter value.
|
||||
///
|
||||
/// Listens for a GET request matching `path_prefix` (e.g., "/callback" or "/auth/callback"),
|
||||
/// extracts the value of `param_name` (e.g., "code" or "token"), and shows a branded
|
||||
/// landing page using `display_name` (e.g., "Google", "Notion", "NEAR AI").
|
||||
///
|
||||
/// When `expected_state` is `Some`, the callback's `state` query parameter is validated
|
||||
/// against it to prevent CSRF attacks. If the state doesn't match, the callback is
|
||||
/// rejected with an error page.
|
||||
///
|
||||
/// Times out after 5 minutes.
|
||||
pub async fn wait_for_callback(
|
||||
listener: TcpListener,
|
||||
path_prefix: &str,
|
||||
param_name: &str,
|
||||
display_name: &str,
|
||||
expected_state: Option<&str>,
|
||||
) -> Result<String, OAuthCallbackError> {
|
||||
let path_prefix = path_prefix.to_string();
|
||||
let param_name = param_name.to_string();
|
||||
let display_name = display_name.to_string();
|
||||
let expected_state = expected_state.map(String::from);
|
||||
|
||||
tokio::time::timeout(Duration::from_secs(300), async move {
|
||||
loop {
|
||||
let (mut socket, _) = listener
|
||||
.accept()
|
||||
.await
|
||||
.map_err(|e| OAuthCallbackError::Io(e.to_string()))?;
|
||||
|
||||
let mut reader = BufReader::new(&mut socket);
|
||||
let mut request_line = String::new();
|
||||
reader
|
||||
.read_line(&mut request_line)
|
||||
.await
|
||||
.map_err(|e| OAuthCallbackError::Io(e.to_string()))?;
|
||||
|
||||
if let Some(path) = request_line.split_whitespace().nth(1)
|
||||
&& path.starts_with(&path_prefix)
|
||||
&& let Some(query) = path.split('?').nth(1)
|
||||
{
|
||||
// Check for error first
|
||||
if query.contains("error=") {
|
||||
let html = landing_html(&display_name, false);
|
||||
let response = format!(
|
||||
"HTTP/1.1 400 Bad Request\r\n\
|
||||
Content-Type: text/html; charset=utf-8\r\n\
|
||||
Connection: close\r\n\
|
||||
\r\n\
|
||||
{}",
|
||||
html
|
||||
);
|
||||
let _ = socket.write_all(response.as_bytes()).await;
|
||||
return Err(OAuthCallbackError::Denied);
|
||||
}
|
||||
|
||||
// Parse all query params into a map for validation
|
||||
let params: HashMap<&str, String> = query
|
||||
.split('&')
|
||||
.filter_map(|p| {
|
||||
let mut parts = p.splitn(2, '=');
|
||||
let key = parts.next()?;
|
||||
let val = parts.next().unwrap_or("");
|
||||
Some((
|
||||
key,
|
||||
urlencoding::decode(val)
|
||||
.unwrap_or_else(|_| val.into())
|
||||
.into_owned(),
|
||||
))
|
||||
})
|
||||
.collect();
|
||||
|
||||
// Validate CSRF state parameter
|
||||
if let Some(ref expected) = expected_state {
|
||||
let actual = params.get("state").cloned().unwrap_or_default();
|
||||
if actual != *expected {
|
||||
let html = landing_html(&display_name, false);
|
||||
let response = format!(
|
||||
"HTTP/1.1 403 Forbidden\r\n\
|
||||
Content-Type: text/html; charset=utf-8\r\n\
|
||||
Connection: close\r\n\
|
||||
\r\n\
|
||||
{}",
|
||||
html
|
||||
);
|
||||
let _ = socket.write_all(response.as_bytes()).await;
|
||||
return Err(OAuthCallbackError::StateMismatch {
|
||||
expected: expected.clone(),
|
||||
actual,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
// Look for the target parameter
|
||||
if let Some(value) = params.get(param_name.as_str()) {
|
||||
let html = landing_html(&display_name, true);
|
||||
let response = format!(
|
||||
"HTTP/1.1 200 OK\r\n\
|
||||
Content-Type: text/html; charset=utf-8\r\n\
|
||||
Connection: close\r\n\
|
||||
\r\n\
|
||||
{}",
|
||||
html
|
||||
);
|
||||
let _ = socket.write_all(response.as_bytes()).await;
|
||||
let _ = socket.shutdown().await;
|
||||
|
||||
return Ok(value.clone());
|
||||
}
|
||||
}
|
||||
|
||||
// Not the callback we're looking for
|
||||
let response = "HTTP/1.1 404 Not Found\r\nConnection: close\r\n\r\n";
|
||||
let _ = socket.write_all(response.as_bytes()).await;
|
||||
}
|
||||
})
|
||||
.await
|
||||
.map_err(|_| OAuthCallbackError::Timeout)?
|
||||
}
|
||||
|
||||
/// Escape a string for safe interpolation into HTML content.
|
||||
fn html_escape(s: &str) -> String {
|
||||
let mut out = String::with_capacity(s.len());
|
||||
for c in s.chars() {
|
||||
match c {
|
||||
'&' => out.push_str("&"),
|
||||
'<' => out.push_str("<"),
|
||||
'>' => out.push_str(">"),
|
||||
'"' => out.push_str("""),
|
||||
'\'' => out.push_str("'"),
|
||||
_ => out.push(c),
|
||||
}
|
||||
}
|
||||
out
|
||||
}
|
||||
// Core OAuth callback infrastructure is defined in `crate::llm::oauth_helpers`
|
||||
// and re-exported here for backward compatibility.
|
||||
pub use crate::llm::oauth_helpers::{
|
||||
OAUTH_CALLBACK_PORT, OAuthCallbackError, bind_callback_listener, callback_host, callback_url,
|
||||
is_loopback_host, landing_html, wait_for_callback,
|
||||
};
|
||||
|
||||
// ── Shared OAuth flow steps ─────────────────────────────────────────
|
||||
|
||||
@@ -598,93 +349,6 @@ pub async fn validate_oauth_token(
|
||||
}
|
||||
}
|
||||
|
||||
// ── Landing pages ───────────────────────────────────────────────────
|
||||
|
||||
pub fn landing_html(provider_name: &str, success: bool) -> String {
|
||||
let safe_name = html_escape(provider_name);
|
||||
let (icon, heading, subtitle, accent) = if success {
|
||||
(
|
||||
r##"<div style="width:64px;height:64px;border-radius:50%;background:#22c55e;display:flex;align-items:center;justify-content:center;margin:0 auto 24px">
|
||||
<svg width="32" height="32" viewBox="0 0 24 24" fill="none" stroke="#fff" stroke-width="3" stroke-linecap="round" stroke-linejoin="round"><polyline points="20 6 9 17 4 12"/></svg>
|
||||
</div>"##,
|
||||
format!("{} Connected", safe_name),
|
||||
"You can close this window and return to your terminal.",
|
||||
"#22c55e",
|
||||
)
|
||||
} else {
|
||||
(
|
||||
r##"<div style="width:64px;height:64px;border-radius:50%;background:#ef4444;display:flex;align-items:center;justify-content:center;margin:0 auto 24px">
|
||||
<svg width="32" height="32" viewBox="0 0 24 24" fill="none" stroke="#fff" stroke-width="3" stroke-linecap="round" stroke-linejoin="round"><line x1="18" y1="6" x2="6" y2="18"/><line x1="6" y1="6" x2="18" y2="18"/></svg>
|
||||
</div>"##,
|
||||
"Authorization Failed".to_string(),
|
||||
"The request was denied. You can close this window and try again.",
|
||||
"#ef4444",
|
||||
)
|
||||
};
|
||||
|
||||
format!(
|
||||
r#"<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width,initial-scale=1">
|
||||
<title>IronClaw - {heading}</title>
|
||||
<style>
|
||||
* {{ margin:0; padding:0; box-sizing:border-box }}
|
||||
body {{
|
||||
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif;
|
||||
background: #0a0a0a;
|
||||
color: #e5e5e5;
|
||||
display: flex;
|
||||
justify-content: center;
|
||||
align-items: center;
|
||||
min-height: 100vh;
|
||||
}}
|
||||
.card {{
|
||||
text-align: center;
|
||||
padding: 48px 40px;
|
||||
max-width: 420px;
|
||||
border: 1px solid #262626;
|
||||
border-radius: 16px;
|
||||
background: #141414;
|
||||
}}
|
||||
h1 {{
|
||||
font-size: 22px;
|
||||
font-weight: 600;
|
||||
margin-bottom: 8px;
|
||||
color: #fafafa;
|
||||
}}
|
||||
p {{
|
||||
font-size: 14px;
|
||||
color: #a3a3a3;
|
||||
line-height: 1.5;
|
||||
}}
|
||||
.accent {{ color: {accent}; }}
|
||||
.brand {{
|
||||
margin-top: 32px;
|
||||
font-size: 12px;
|
||||
color: #525252;
|
||||
letter-spacing: 0.5px;
|
||||
text-transform: uppercase;
|
||||
}}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="card">
|
||||
{icon}
|
||||
<h1>{heading}</h1>
|
||||
<p>{subtitle}</p>
|
||||
<div class="brand">IronClaw</div>
|
||||
</div>
|
||||
</body>
|
||||
</html>"#,
|
||||
heading = heading,
|
||||
icon = icon,
|
||||
subtitle = subtitle,
|
||||
accent = accent,
|
||||
)
|
||||
}
|
||||
|
||||
// ── Gateway callback support ─────────────────────────────────────────
|
||||
|
||||
/// State for an in-progress OAuth flow, keyed by CSRF `state` parameter.
|
||||
|
||||
+124
-2
@@ -8,9 +8,35 @@ use std::path::PathBuf;
|
||||
use crate::bootstrap::ironclaw_base_dir;
|
||||
use crate::settings::Settings;
|
||||
|
||||
/// Load settings from JSON and TOML config files, matching the runtime
|
||||
/// priority: TOML overlay > settings.json > defaults.
|
||||
///
|
||||
/// This mirrors the loading chain in `Config::from_env_with_toml()` but
|
||||
/// without resolving the full `Config` (which requires async + secrets).
|
||||
fn load_settings() -> Settings {
|
||||
load_settings_from(&Settings::default_path(), &Settings::default_toml_path())
|
||||
}
|
||||
|
||||
/// Inner implementation with injectable paths (testable).
|
||||
fn load_settings_from(json_path: &std::path::Path, toml_path: &std::path::Path) -> Settings {
|
||||
let mut settings = Settings::load_from(json_path);
|
||||
|
||||
match Settings::load_toml(toml_path) {
|
||||
Ok(Some(toml_settings)) => {
|
||||
settings.merge_from(&toml_settings);
|
||||
}
|
||||
Ok(None) => {} // File not found — fine for default path
|
||||
Err(e) => {
|
||||
eprintln!("Warning: failed to parse {}: {}", toml_path.display(), e);
|
||||
}
|
||||
}
|
||||
|
||||
settings
|
||||
}
|
||||
|
||||
/// Run the status command, printing system health info.
|
||||
pub async fn run_status_command() -> anyhow::Result<()> {
|
||||
let settings = Settings::default();
|
||||
let settings = load_settings();
|
||||
|
||||
println!("IronClaw Status");
|
||||
println!("===============\n");
|
||||
@@ -57,7 +83,7 @@ pub async fn run_status_command() -> anyhow::Result<()> {
|
||||
|
||||
// Session / Auth
|
||||
print!(" Session: ");
|
||||
let session_path = crate::llm::session::default_session_path();
|
||||
let session_path = crate::config::llm::default_session_path();
|
||||
if session_path.exists() {
|
||||
println!("found ({})", session_path.display());
|
||||
} else {
|
||||
@@ -209,3 +235,99 @@ fn default_tools_dir() -> PathBuf {
|
||||
fn default_channels_dir() -> PathBuf {
|
||||
ironclaw_base_dir().join("channels")
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::load_settings_from;
|
||||
|
||||
/// Regression test for #354: load_settings_from must read config.toml.
|
||||
#[test]
|
||||
fn reads_toml_heartbeat_enabled() {
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
let json_path = dir.path().join("settings.json");
|
||||
let toml_path = dir.path().join("config.toml");
|
||||
|
||||
// No JSON file — only TOML
|
||||
std::fs::write(
|
||||
&toml_path,
|
||||
"[heartbeat]\nenabled = true\ninterval_secs = 600",
|
||||
)
|
||||
.expect("write toml");
|
||||
|
||||
let settings = load_settings_from(&json_path, &toml_path);
|
||||
assert!(settings.heartbeat.enabled);
|
||||
assert_eq!(settings.heartbeat.interval_secs, 600);
|
||||
}
|
||||
|
||||
/// Without any config files, defaults are returned.
|
||||
#[test]
|
||||
fn defaults_without_config_files() {
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
let settings = load_settings_from(
|
||||
&dir.path().join("nonexistent.json"),
|
||||
&dir.path().join("nonexistent.toml"),
|
||||
);
|
||||
assert!(!settings.heartbeat.enabled);
|
||||
}
|
||||
|
||||
/// settings.json is respected.
|
||||
#[test]
|
||||
fn reads_json_heartbeat_enabled() {
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
let json_path = dir.path().join("settings.json");
|
||||
let toml_path = dir.path().join("nonexistent.toml");
|
||||
|
||||
std::fs::write(
|
||||
&json_path,
|
||||
r#"{"heartbeat":{"enabled":true,"interval_secs":900}}"#,
|
||||
)
|
||||
.expect("write json");
|
||||
|
||||
let settings = load_settings_from(&json_path, &toml_path);
|
||||
assert!(settings.heartbeat.enabled);
|
||||
assert_eq!(settings.heartbeat.interval_secs, 900);
|
||||
}
|
||||
|
||||
/// TOML overlay wins over JSON settings.
|
||||
#[test]
|
||||
fn toml_overlay_wins_over_json() {
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
let json_path = dir.path().join("settings.json");
|
||||
let toml_path = dir.path().join("config.toml");
|
||||
|
||||
std::fs::write(
|
||||
&json_path,
|
||||
r#"{"heartbeat":{"enabled":false,"interval_secs":100}}"#,
|
||||
)
|
||||
.expect("write json");
|
||||
std::fs::write(
|
||||
&toml_path,
|
||||
"[heartbeat]\nenabled = true\ninterval_secs = 200",
|
||||
)
|
||||
.expect("write toml");
|
||||
|
||||
let settings = load_settings_from(&json_path, &toml_path);
|
||||
assert!(settings.heartbeat.enabled);
|
||||
assert_eq!(settings.heartbeat.interval_secs, 200);
|
||||
}
|
||||
|
||||
/// Invalid TOML is warned but doesn't crash; falls back to JSON/defaults.
|
||||
#[test]
|
||||
fn invalid_toml_falls_back_gracefully() {
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
let json_path = dir.path().join("settings.json");
|
||||
let toml_path = dir.path().join("config.toml");
|
||||
|
||||
std::fs::write(
|
||||
&json_path,
|
||||
r#"{"heartbeat":{"enabled":true,"interval_secs":500}}"#,
|
||||
)
|
||||
.expect("write json");
|
||||
std::fs::write(&toml_path, "this is not valid toml [[[").expect("write bad toml");
|
||||
|
||||
let settings = load_settings_from(&json_path, &toml_path);
|
||||
// Should fall back to JSON values, not crash
|
||||
assert!(settings.heartbeat.enabled);
|
||||
assert_eq!(settings.heartbeat.interval_secs, 500);
|
||||
}
|
||||
}
|
||||
|
||||
+2
-56
@@ -11,10 +11,6 @@ use tokio::fs;
|
||||
|
||||
use crate::bootstrap::ironclaw_base_dir;
|
||||
use crate::config::Config;
|
||||
#[allow(unused_imports)]
|
||||
use crate::db::Database;
|
||||
#[cfg(feature = "postgres")]
|
||||
use crate::secrets::PostgresSecretsStore;
|
||||
use crate::secrets::{CreateSecretParams, SecretsCrypto, SecretsStore};
|
||||
use crate::tools::wasm::{CapabilitiesFile, compute_binary_hash};
|
||||
|
||||
@@ -563,59 +559,9 @@ async fn init_secrets_store() -> anyhow::Result<Arc<dyn SecretsStore + Send + Sy
|
||||
)
|
||||
})?;
|
||||
|
||||
let crypto = SecretsCrypto::new(master_key.clone())?;
|
||||
let crypto = Arc::new(SecretsCrypto::new(master_key.clone())?);
|
||||
|
||||
let store: Arc<dyn SecretsStore + Send + Sync> = {
|
||||
#[cfg(feature = "postgres")]
|
||||
{
|
||||
let store = crate::history::Store::new(&config.database).await?;
|
||||
store.run_migrations().await?;
|
||||
Arc::new(PostgresSecretsStore::new(store.pool(), Arc::new(crypto)))
|
||||
}
|
||||
#[cfg(all(feature = "libsql", not(feature = "postgres")))]
|
||||
{
|
||||
use crate::db::Database as _;
|
||||
use crate::db::libsql::LibSqlBackend;
|
||||
use secrecy::ExposeSecret as _;
|
||||
|
||||
let default_path = crate::config::default_libsql_path();
|
||||
let db_path = config
|
||||
.database
|
||||
.libsql_path
|
||||
.as_deref()
|
||||
.unwrap_or(&default_path);
|
||||
|
||||
let backend = if let Some(ref url) = config.database.libsql_url {
|
||||
let token = config.database.libsql_auth_token.as_ref().ok_or_else(|| {
|
||||
anyhow::anyhow!("LIBSQL_AUTH_TOKEN is required when LIBSQL_URL is set")
|
||||
})?;
|
||||
LibSqlBackend::new_remote_replica(db_path, url, token.expose_secret())
|
||||
.await
|
||||
.map_err(|e| anyhow::anyhow!("{}", e))?
|
||||
} else {
|
||||
LibSqlBackend::new_local(db_path)
|
||||
.await
|
||||
.map_err(|e| anyhow::anyhow!("{}", e))?
|
||||
};
|
||||
backend
|
||||
.run_migrations()
|
||||
.await
|
||||
.map_err(|e| anyhow::anyhow!("{}", e))?;
|
||||
|
||||
Arc::new(crate::secrets::LibSqlSecretsStore::new(
|
||||
backend.shared_db(),
|
||||
Arc::new(crypto),
|
||||
))
|
||||
}
|
||||
#[cfg(not(any(feature = "postgres", feature = "libsql")))]
|
||||
{
|
||||
let _ = crypto;
|
||||
anyhow::bail!(
|
||||
"No database backend available for secrets. Enable 'postgres' or 'libsql' feature."
|
||||
);
|
||||
}
|
||||
};
|
||||
Ok(store)
|
||||
Ok(crate::db::create_secrets_store(&config.database, crypto).await?)
|
||||
}
|
||||
|
||||
/// Configure authentication for a tool.
|
||||
|
||||
@@ -27,6 +27,10 @@ pub struct AgentConfig {
|
||||
pub max_tool_iterations: usize,
|
||||
/// When true, skip tool approval checks entirely. For benchmarks/CI.
|
||||
pub auto_approve_tools: bool,
|
||||
/// Default timezone for new sessions (IANA name, e.g. "America/New_York").
|
||||
pub default_timezone: String,
|
||||
/// Maximum tokens per job (0 = unlimited).
|
||||
pub max_tokens_per_job: u64,
|
||||
}
|
||||
|
||||
impl AgentConfig {
|
||||
@@ -47,6 +51,8 @@ impl AgentConfig {
|
||||
max_actions_per_hour: None,
|
||||
max_tool_iterations: 10,
|
||||
auto_approve_tools: true,
|
||||
default_timezone: "UTC".to_string(),
|
||||
max_tokens_per_job: 0,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -89,6 +95,44 @@ impl AgentConfig {
|
||||
"AGENT_AUTO_APPROVE_TOOLS",
|
||||
settings.agent.auto_approve_tools,
|
||||
)?,
|
||||
default_timezone: {
|
||||
let tz: String = parse_optional_env(
|
||||
"DEFAULT_TIMEZONE",
|
||||
settings.agent.default_timezone.clone(),
|
||||
)?;
|
||||
if crate::timezone::parse_timezone(&tz).is_none() {
|
||||
return Err(ConfigError::InvalidValue {
|
||||
key: "DEFAULT_TIMEZONE".into(),
|
||||
message: format!("invalid IANA timezone: '{tz}'"),
|
||||
});
|
||||
}
|
||||
tz
|
||||
},
|
||||
max_tokens_per_job: parse_optional_env(
|
||||
"AGENT_MAX_TOKENS_PER_JOB",
|
||||
settings.agent.max_tokens_per_job,
|
||||
)?,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn test_default_timezone_rejects_invalid() {
|
||||
let mut settings = Settings::default();
|
||||
settings.agent.default_timezone = "Fake/Zone".to_string();
|
||||
|
||||
let result = AgentConfig::resolve(&settings);
|
||||
assert!(result.is_err(), "invalid IANA timezone should be rejected");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_default_timezone_accepts_valid() {
|
||||
let settings = Settings::default(); // default is "UTC"
|
||||
let config = AgentConfig::resolve(&settings).expect("resolve");
|
||||
assert_eq!(config.default_timezone, "UTC");
|
||||
}
|
||||
}
|
||||
|
||||
+102
-1
@@ -1,4 +1,4 @@
|
||||
use crate::config::helpers::{optional_env, parse_bool_env, parse_optional_env};
|
||||
use crate::config::helpers::{optional_env, parse_bool_env, parse_option_env, parse_optional_env};
|
||||
use crate::error::ConfigError;
|
||||
use crate::settings::Settings;
|
||||
|
||||
@@ -13,6 +13,12 @@ pub struct HeartbeatConfig {
|
||||
pub notify_channel: Option<String>,
|
||||
/// User ID to notify on heartbeat findings.
|
||||
pub notify_user: Option<String>,
|
||||
/// Hour (0-23) when quiet hours start.
|
||||
pub quiet_hours_start: Option<u32>,
|
||||
/// Hour (0-23) when quiet hours end.
|
||||
pub quiet_hours_end: Option<u32>,
|
||||
/// Timezone for quiet hours evaluation (IANA name).
|
||||
pub timezone: Option<String>,
|
||||
}
|
||||
|
||||
impl Default for HeartbeatConfig {
|
||||
@@ -22,6 +28,9 @@ impl Default for HeartbeatConfig {
|
||||
interval_secs: 1800, // 30 minutes
|
||||
notify_channel: None,
|
||||
notify_user: None,
|
||||
quiet_hours_start: None,
|
||||
quiet_hours_end: None,
|
||||
timezone: None,
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -38,6 +47,98 @@ impl HeartbeatConfig {
|
||||
.or_else(|| settings.heartbeat.notify_channel.clone()),
|
||||
notify_user: optional_env("HEARTBEAT_NOTIFY_USER")?
|
||||
.or_else(|| settings.heartbeat.notify_user.clone()),
|
||||
quiet_hours_start: parse_option_env::<u32>("HEARTBEAT_QUIET_START")?
|
||||
.or(settings.heartbeat.quiet_hours_start)
|
||||
.map(|h| {
|
||||
if h > 23 {
|
||||
return Err(ConfigError::InvalidValue {
|
||||
key: "HEARTBEAT_QUIET_START".into(),
|
||||
message: "must be 0-23".into(),
|
||||
});
|
||||
}
|
||||
Ok(h)
|
||||
})
|
||||
.transpose()?,
|
||||
quiet_hours_end: parse_option_env::<u32>("HEARTBEAT_QUIET_END")?
|
||||
.or(settings.heartbeat.quiet_hours_end)
|
||||
.map(|h| {
|
||||
if h > 23 {
|
||||
return Err(ConfigError::InvalidValue {
|
||||
key: "HEARTBEAT_QUIET_END".into(),
|
||||
message: "must be 0-23".into(),
|
||||
});
|
||||
}
|
||||
Ok(h)
|
||||
})
|
||||
.transpose()?,
|
||||
timezone: {
|
||||
let tz = optional_env("HEARTBEAT_TIMEZONE")?
|
||||
.or_else(|| settings.heartbeat.timezone.clone());
|
||||
if let Some(ref tz_str) = tz
|
||||
&& crate::timezone::parse_timezone(tz_str).is_none()
|
||||
{
|
||||
return Err(ConfigError::InvalidValue {
|
||||
key: "HEARTBEAT_TIMEZONE".into(),
|
||||
message: format!("invalid IANA timezone: '{tz_str}'"),
|
||||
});
|
||||
}
|
||||
tz
|
||||
},
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn test_quiet_hours_settings_fallback() {
|
||||
// When env vars are not set, settings values should be used
|
||||
let mut settings = Settings::default();
|
||||
settings.heartbeat.quiet_hours_start = Some(22);
|
||||
settings.heartbeat.quiet_hours_end = Some(6);
|
||||
|
||||
let config = HeartbeatConfig::resolve(&settings).expect("resolve");
|
||||
assert_eq!(config.quiet_hours_start, Some(22));
|
||||
assert_eq!(config.quiet_hours_end, Some(6));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_quiet_hours_rejects_invalid_hour() {
|
||||
let mut settings = Settings::default();
|
||||
settings.heartbeat.quiet_hours_start = Some(24);
|
||||
|
||||
let result = HeartbeatConfig::resolve(&settings);
|
||||
assert!(result.is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_quiet_hours_accepts_boundary_values() {
|
||||
let mut settings = Settings::default();
|
||||
settings.heartbeat.quiet_hours_start = Some(0);
|
||||
settings.heartbeat.quiet_hours_end = Some(23);
|
||||
|
||||
let config = HeartbeatConfig::resolve(&settings).expect("resolve");
|
||||
assert_eq!(config.quiet_hours_start, Some(0));
|
||||
assert_eq!(config.quiet_hours_end, Some(23));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_heartbeat_timezone_rejects_invalid() {
|
||||
let mut settings = Settings::default();
|
||||
settings.heartbeat.timezone = Some("Fake/Zone".to_string());
|
||||
|
||||
let result = HeartbeatConfig::resolve(&settings);
|
||||
assert!(result.is_err(), "invalid IANA timezone should be rejected");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_heartbeat_timezone_accepts_valid() {
|
||||
let mut settings = Settings::default();
|
||||
settings.heartbeat.timezone = Some("America/New_York".to_string());
|
||||
|
||||
let config = HeartbeatConfig::resolve(&settings).expect("resolve");
|
||||
assert_eq!(config.timezone.as_deref(), Some("America/New_York"));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -25,8 +25,13 @@ pub(crate) fn optional_env(key: &str) -> Result<Option<String>, ConfigError> {
|
||||
}
|
||||
|
||||
// Fall back to thread-safe overlay (secrets injected from DB)
|
||||
if let Some(val) = INJECTED_VARS.get().and_then(|map| map.get(key)) {
|
||||
return Ok(Some(val.clone()));
|
||||
if let Some(val) = INJECTED_VARS
|
||||
.lock()
|
||||
.unwrap_or_else(|p| p.into_inner())
|
||||
.get(key)
|
||||
.cloned()
|
||||
{
|
||||
return Ok(Some(val));
|
||||
}
|
||||
|
||||
Ok(None)
|
||||
|
||||
+226
-128
@@ -5,128 +5,11 @@ use secrecy::SecretString;
|
||||
use crate::bootstrap::ironclaw_base_dir;
|
||||
use crate::config::helpers::{optional_env, parse_optional_env};
|
||||
use crate::error::ConfigError;
|
||||
use crate::llm::config::*;
|
||||
use crate::llm::registry::{ProviderProtocol, ProviderRegistry};
|
||||
use crate::llm::session::SessionConfig;
|
||||
use crate::settings::Settings;
|
||||
|
||||
/// Prompt cache retention policy for Anthropic.
|
||||
///
|
||||
/// Controls Anthropic's automatic prompt caching via a top-level
|
||||
/// `cache_control` field injected through rig-core's `additional_params`.
|
||||
/// - `None` — caching disabled, no `cache_control` injected.
|
||||
/// - `Short` — 5-minute TTL (default), `{"type": "ephemeral"}`, 1.25× write surcharge.
|
||||
/// - `Long` — 1-hour TTL, `{"type": "ephemeral", "ttl": "1h"}`, 2× write surcharge.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
|
||||
pub enum CacheRetention {
|
||||
/// No prompt caching.
|
||||
None,
|
||||
/// 5-minute TTL (default). Write cost: 1.25× base input.
|
||||
#[default]
|
||||
Short,
|
||||
/// 1-hour TTL. Write cost: 2× base input.
|
||||
Long,
|
||||
}
|
||||
|
||||
impl std::str::FromStr for CacheRetention {
|
||||
type Err = String;
|
||||
|
||||
fn from_str(s: &str) -> Result<Self, Self::Err> {
|
||||
match s.to_lowercase().as_str() {
|
||||
"none" | "off" | "disabled" => Ok(Self::None),
|
||||
"short" | "5m" | "ephemeral" => Ok(Self::Short),
|
||||
"long" | "1h" => Ok(Self::Long),
|
||||
_ => Err(format!(
|
||||
"invalid cache retention '{}', expected one of: none, short, long",
|
||||
s
|
||||
)),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl std::fmt::Display for CacheRetention {
|
||||
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
match self {
|
||||
Self::None => write!(f, "none"),
|
||||
Self::Short => write!(f, "short"),
|
||||
Self::Long => write!(f, "long"),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Resolved configuration for a registry-based provider.
|
||||
///
|
||||
/// This single struct replaces what used to be five separate config types
|
||||
/// (`OpenAiDirectConfig`, `AnthropicDirectConfig`, `OllamaConfig`,
|
||||
/// `OpenAiCompatibleConfig`, `TinfoilConfig`). The `protocol` field
|
||||
/// determines which rig-core client constructor to use.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct RegistryProviderConfig {
|
||||
/// Which API protocol to use (determines the rig-core client).
|
||||
pub protocol: ProviderProtocol,
|
||||
/// Provider identifier (e.g., "groq", "openai", "tinfoil").
|
||||
pub provider_id: String,
|
||||
/// API key (optional for some providers like Ollama).
|
||||
pub api_key: Option<SecretString>,
|
||||
/// Base URL for the API endpoint.
|
||||
pub base_url: String,
|
||||
/// Model identifier.
|
||||
pub model: String,
|
||||
/// Extra HTTP headers injected into every request.
|
||||
pub extra_headers: Vec<(String, String)>,
|
||||
}
|
||||
|
||||
/// LLM provider configuration.
|
||||
///
|
||||
/// NearAI remains the default backend with its own config struct (session auth).
|
||||
/// All other providers are resolved through the provider registry, producing
|
||||
/// a generic `RegistryProviderConfig`.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct LlmConfig {
|
||||
/// Backend identifier (e.g., "nearai", "openai", "groq", "tinfoil").
|
||||
pub backend: String,
|
||||
/// Session manager configuration (auth URL, token persistence path).
|
||||
/// Used by the NearAI provider for OAuth/session-token auth.
|
||||
pub session: SessionConfig,
|
||||
/// NEAR AI config (always populated, also used for embeddings).
|
||||
pub nearai: NearAiConfig,
|
||||
/// Resolved provider config for registry-based providers.
|
||||
/// `None` when backend is "nearai".
|
||||
pub provider: Option<RegistryProviderConfig>,
|
||||
}
|
||||
|
||||
/// NEAR AI configuration.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct NearAiConfig {
|
||||
/// Model to use (e.g., "claude-3-5-sonnet-20241022", "gpt-4o")
|
||||
pub model: String,
|
||||
/// Cheap/fast model for lightweight tasks (heartbeat, routing, evaluation).
|
||||
pub cheap_model: Option<String>,
|
||||
/// Base URL for the NEAR AI API.
|
||||
pub base_url: String,
|
||||
/// API key for NEAR AI Cloud.
|
||||
pub api_key: Option<SecretString>,
|
||||
/// Optional fallback model for failover.
|
||||
pub fallback_model: Option<String>,
|
||||
/// Maximum number of retries for transient errors (default: 3).
|
||||
pub max_retries: u32,
|
||||
/// Consecutive failures before circuit breaker opens. None = disabled.
|
||||
pub circuit_breaker_threshold: Option<u32>,
|
||||
/// Seconds the circuit stays open before probing (default: 30).
|
||||
pub circuit_breaker_recovery_secs: u64,
|
||||
/// Enable in-memory response caching. Default: false.
|
||||
pub response_cache_enabled: bool,
|
||||
/// TTL in seconds for cached responses (default: 3600).
|
||||
pub response_cache_ttl_secs: u64,
|
||||
/// Max cached responses before LRU eviction (default: 1000).
|
||||
pub response_cache_max_entries: usize,
|
||||
/// Cooldown duration in seconds for failover (default: 300).
|
||||
pub failover_cooldown_secs: u64,
|
||||
/// Consecutive failures before failover cooldown (default: 3).
|
||||
pub failover_cooldown_threshold: u32,
|
||||
/// Enable cascade mode for smart routing. Default: true.
|
||||
pub smart_routing_cascade: bool,
|
||||
}
|
||||
|
||||
impl LlmConfig {
|
||||
/// Create a test-friendly config without reading env vars.
|
||||
#[cfg(feature = "libsql")]
|
||||
@@ -154,6 +37,8 @@ impl LlmConfig {
|
||||
smart_routing_cascade: false,
|
||||
},
|
||||
provider: None,
|
||||
bedrock: None,
|
||||
request_timeout_secs: 120,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -184,8 +69,10 @@ impl LlmConfig {
|
||||
let backend_lower = backend.to_lowercase();
|
||||
let is_nearai =
|
||||
backend_lower == "nearai" || backend_lower == "near_ai" || backend_lower == "near";
|
||||
let is_bedrock =
|
||||
backend_lower == "bedrock" || backend_lower == "aws_bedrock" || backend_lower == "aws";
|
||||
|
||||
if !is_nearai && registry.find(&backend_lower).is_none() {
|
||||
if !is_nearai && !is_bedrock && registry.find(&backend_lower).is_none() {
|
||||
tracing::warn!(
|
||||
"Unknown LLM backend '{}'. Will attempt as openai_compatible fallback.",
|
||||
backend
|
||||
@@ -232,8 +119,8 @@ impl LlmConfig {
|
||||
smart_routing_cascade: parse_optional_env("SMART_ROUTING_CASCADE", true)?,
|
||||
};
|
||||
|
||||
// Resolve registry provider config (for non-NearAI backends)
|
||||
let provider = if is_nearai {
|
||||
// Resolve registry provider config (for non-NearAI, non-Bedrock backends)
|
||||
let provider = if is_nearai || is_bedrock {
|
||||
None
|
||||
} else {
|
||||
Some(Self::resolve_registry_provider(
|
||||
@@ -243,9 +130,50 @@ impl LlmConfig {
|
||||
)?)
|
||||
};
|
||||
|
||||
let bedrock = if is_bedrock {
|
||||
let explicit_region =
|
||||
optional_env("BEDROCK_REGION")?.or_else(|| settings.bedrock_region.clone());
|
||||
if explicit_region.is_none() {
|
||||
tracing::info!("BEDROCK_REGION not set, defaulting to us-east-1");
|
||||
}
|
||||
let region = explicit_region.unwrap_or_else(|| "us-east-1".to_string());
|
||||
let model = optional_env("BEDROCK_MODEL")?
|
||||
.or_else(|| settings.selected_model.clone())
|
||||
.ok_or_else(|| ConfigError::MissingRequired {
|
||||
key: "BEDROCK_MODEL".to_string(),
|
||||
hint: "Set BEDROCK_MODEL when LLM_BACKEND=bedrock".to_string(),
|
||||
})?;
|
||||
let cross_region = optional_env("BEDROCK_CROSS_REGION")?
|
||||
.or_else(|| settings.bedrock_cross_region.clone());
|
||||
if let Some(ref cr) = cross_region
|
||||
&& !matches!(cr.as_str(), "us" | "eu" | "apac" | "global")
|
||||
{
|
||||
return Err(ConfigError::InvalidValue {
|
||||
key: "BEDROCK_CROSS_REGION".to_string(),
|
||||
message: format!(
|
||||
"'{}' is not valid, expected one of: us, eu, apac, global",
|
||||
cr
|
||||
),
|
||||
});
|
||||
}
|
||||
let profile = optional_env("AWS_PROFILE")?.or_else(|| settings.bedrock_profile.clone());
|
||||
Some(BedrockConfig {
|
||||
region,
|
||||
model,
|
||||
cross_region,
|
||||
profile,
|
||||
})
|
||||
} else {
|
||||
None
|
||||
};
|
||||
|
||||
let request_timeout_secs = parse_optional_env("LLM_REQUEST_TIMEOUT_SECS", 120)?;
|
||||
|
||||
Ok(Self {
|
||||
backend: if is_nearai {
|
||||
"nearai".to_string()
|
||||
} else if is_bedrock {
|
||||
"bedrock".to_string()
|
||||
} else if let Some(ref p) = provider {
|
||||
p.provider_id.clone()
|
||||
} else {
|
||||
@@ -254,6 +182,8 @@ impl LlmConfig {
|
||||
session,
|
||||
nearai,
|
||||
provider,
|
||||
bedrock,
|
||||
request_timeout_secs,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -366,6 +296,39 @@ impl LlmConfig {
|
||||
Vec::new()
|
||||
};
|
||||
|
||||
// Resolve OAuth token (Anthropic-specific: `claude login` flow).
|
||||
// Only check for OAuth token when the provider is actually Anthropic.
|
||||
let oauth_token = if canonical_id == "anthropic" {
|
||||
optional_env("ANTHROPIC_OAUTH_TOKEN")?.map(SecretString::from)
|
||||
} else {
|
||||
None
|
||||
};
|
||||
let api_key = if api_key.is_none() && oauth_token.is_some() {
|
||||
// OAuth token present but no API key: use a placeholder so the
|
||||
// config block is populated. The provider factory will route to
|
||||
// the OAuth provider instead of rig-core's x-api-key client.
|
||||
Some(SecretString::from(OAUTH_PLACEHOLDER.to_string()))
|
||||
} else {
|
||||
api_key
|
||||
};
|
||||
|
||||
// Resolve Anthropic prompt cache retention from env (default: Short).
|
||||
let cache_retention: CacheRetention = if canonical_id == "anthropic" {
|
||||
optional_env("ANTHROPIC_CACHE_RETENTION")?
|
||||
.and_then(|val| match val.parse::<CacheRetention>() {
|
||||
Ok(r) => Some(r),
|
||||
Err(e) => {
|
||||
tracing::warn!(
|
||||
"Invalid ANTHROPIC_CACHE_RETENTION: {e}; defaulting to short"
|
||||
);
|
||||
None
|
||||
}
|
||||
})
|
||||
.unwrap_or_default()
|
||||
} else {
|
||||
CacheRetention::default()
|
||||
};
|
||||
|
||||
Ok(RegistryProviderConfig {
|
||||
protocol,
|
||||
provider_id: canonical_id.to_string(),
|
||||
@@ -373,6 +336,8 @@ impl LlmConfig {
|
||||
base_url,
|
||||
model,
|
||||
extra_headers,
|
||||
oauth_token,
|
||||
cache_retention,
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -411,7 +376,7 @@ fn parse_extra_headers(val: &str) -> Result<Vec<(String, String)>, ConfigError>
|
||||
}
|
||||
|
||||
/// Get the default session file path (~/.ironclaw/session.json).
|
||||
fn default_session_path() -> PathBuf {
|
||||
pub fn default_session_path() -> PathBuf {
|
||||
ironclaw_base_dir().join("session.json")
|
||||
}
|
||||
|
||||
@@ -677,8 +642,6 @@ mod tests {
|
||||
|
||||
#[test]
|
||||
fn backend_alias_normalized_to_canonical_id() {
|
||||
// When the user sets LLM_BACKEND to an alias (e.g., "open_ai"),
|
||||
// LlmConfig.backend should resolve to the canonical ID ("openai").
|
||||
let _guard = ENV_MUTEX.lock().expect("env mutex poisoned");
|
||||
clear_openai_compatible_env();
|
||||
// SAFETY: Under ENV_MUTEX.
|
||||
@@ -705,8 +668,6 @@ mod tests {
|
||||
|
||||
#[test]
|
||||
fn unknown_backend_falls_back_to_openai_compatible() {
|
||||
// An unrecognized LLM_BACKEND should fall back to the openai_compatible
|
||||
// provider definition instead of erroring.
|
||||
let _guard = ENV_MUTEX.lock().expect("env mutex poisoned");
|
||||
clear_openai_compatible_env();
|
||||
// SAFETY: Under ENV_MUTEX.
|
||||
@@ -717,7 +678,6 @@ mod tests {
|
||||
|
||||
let settings = Settings::default();
|
||||
let cfg = LlmConfig::resolve(&settings).expect("resolve should succeed");
|
||||
// Falls back to openai_compatible since "some_custom_provider" is unknown
|
||||
assert_eq!(cfg.backend, "openai_compatible");
|
||||
let provider = cfg.provider.expect("should have provider config");
|
||||
assert_eq!(provider.provider_id, "openai_compatible");
|
||||
@@ -759,7 +719,6 @@ mod tests {
|
||||
|
||||
#[test]
|
||||
fn base_url_resolution_priority() {
|
||||
// Env var > settings > registry default
|
||||
let _guard = ENV_MUTEX.lock().expect("env mutex poisoned");
|
||||
clear_openai_compatible_env();
|
||||
|
||||
@@ -800,6 +759,119 @@ mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
// ── OAuth resolution tests ──────────────────────────────────────
|
||||
|
||||
/// Clear all Anthropic-related env vars.
|
||||
fn clear_anthropic_env() {
|
||||
// SAFETY: Only called under ENV_MUTEX in tests.
|
||||
unsafe {
|
||||
std::env::remove_var("LLM_BACKEND");
|
||||
std::env::remove_var("ANTHROPIC_API_KEY");
|
||||
std::env::remove_var("ANTHROPIC_OAUTH_TOKEN");
|
||||
std::env::remove_var("ANTHROPIC_MODEL");
|
||||
std::env::remove_var("ANTHROPIC_BASE_URL");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn anthropic_oauth_token_sets_placeholder_api_key() {
|
||||
use secrecy::ExposeSecret;
|
||||
|
||||
let _guard = ENV_MUTEX.lock().expect("env mutex poisoned");
|
||||
clear_anthropic_env();
|
||||
// SAFETY: Under ENV_MUTEX.
|
||||
unsafe {
|
||||
std::env::set_var("ANTHROPIC_OAUTH_TOKEN", "sk-ant-oat01-test-token");
|
||||
}
|
||||
|
||||
let settings = Settings {
|
||||
llm_backend: Some("anthropic".to_string()),
|
||||
..Default::default()
|
||||
};
|
||||
let cfg = LlmConfig::resolve(&settings).expect("resolve should succeed");
|
||||
let provider = cfg.provider.expect("provider config should be present");
|
||||
|
||||
assert_eq!(
|
||||
provider
|
||||
.api_key
|
||||
.as_ref()
|
||||
.map(|k| k.expose_secret().to_string()),
|
||||
Some(OAUTH_PLACEHOLDER.to_string()),
|
||||
"api_key should be the OAuth placeholder when only OAuth token is set"
|
||||
);
|
||||
assert!(
|
||||
provider.oauth_token.is_some(),
|
||||
"oauth_token should be populated"
|
||||
);
|
||||
assert_eq!(
|
||||
provider.oauth_token.as_ref().unwrap().expose_secret(),
|
||||
"sk-ant-oat01-test-token"
|
||||
);
|
||||
|
||||
clear_anthropic_env();
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn anthropic_api_key_takes_priority_over_oauth() {
|
||||
use secrecy::ExposeSecret;
|
||||
|
||||
let _guard = ENV_MUTEX.lock().expect("env mutex poisoned");
|
||||
clear_anthropic_env();
|
||||
// SAFETY: Under ENV_MUTEX.
|
||||
unsafe {
|
||||
std::env::set_var("ANTHROPIC_API_KEY", "sk-ant-real-key");
|
||||
std::env::set_var("ANTHROPIC_OAUTH_TOKEN", "sk-ant-oat01-test-token");
|
||||
}
|
||||
|
||||
let settings = Settings {
|
||||
llm_backend: Some("anthropic".to_string()),
|
||||
..Default::default()
|
||||
};
|
||||
let cfg = LlmConfig::resolve(&settings).expect("resolve should succeed");
|
||||
let provider = cfg.provider.expect("provider config should be present");
|
||||
|
||||
assert_eq!(
|
||||
provider
|
||||
.api_key
|
||||
.as_ref()
|
||||
.map(|k| k.expose_secret().to_string()),
|
||||
Some("sk-ant-real-key".to_string()),
|
||||
"real API key should take priority over OAuth placeholder"
|
||||
);
|
||||
assert!(
|
||||
provider.oauth_token.is_some(),
|
||||
"oauth_token should still be populated"
|
||||
);
|
||||
|
||||
clear_anthropic_env();
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn non_anthropic_provider_has_no_oauth_token() {
|
||||
let _guard = ENV_MUTEX.lock().expect("env mutex poisoned");
|
||||
clear_anthropic_env();
|
||||
// SAFETY: Under ENV_MUTEX.
|
||||
unsafe {
|
||||
std::env::set_var("ANTHROPIC_OAUTH_TOKEN", "sk-ant-oat01-test-token");
|
||||
}
|
||||
|
||||
let settings = Settings {
|
||||
llm_backend: Some("openai".to_string()),
|
||||
..Default::default()
|
||||
};
|
||||
let cfg = LlmConfig::resolve(&settings).expect("resolve should succeed");
|
||||
let provider = cfg.provider.expect("provider config should be present");
|
||||
|
||||
assert!(
|
||||
provider.oauth_token.is_none(),
|
||||
"non-Anthropic providers should not pick up ANTHROPIC_OAUTH_TOKEN"
|
||||
);
|
||||
|
||||
clear_anthropic_env();
|
||||
}
|
||||
|
||||
// ── Cache retention tests ───────────────────────────────────────
|
||||
|
||||
#[test]
|
||||
fn cache_retention_from_str_primary_values() {
|
||||
assert_eq!(
|
||||
@@ -881,4 +953,30 @@ mod tests {
|
||||
assert_eq!(parsed, variant, "round-trip failed for {s}");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_request_timeout_defaults_to_120() {
|
||||
let _guard = ENV_MUTEX.lock().expect("env mutex poisoned");
|
||||
// SAFETY: Under ENV_MUTEX.
|
||||
unsafe {
|
||||
std::env::remove_var("LLM_REQUEST_TIMEOUT_SECS");
|
||||
}
|
||||
let config = LlmConfig::resolve(&Settings::default()).expect("resolve");
|
||||
assert_eq!(config.request_timeout_secs, 120);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_request_timeout_configurable() {
|
||||
let _guard = ENV_MUTEX.lock().expect("env mutex poisoned");
|
||||
// SAFETY: Under ENV_MUTEX.
|
||||
unsafe {
|
||||
std::env::set_var("LLM_REQUEST_TIMEOUT_SECS", "300");
|
||||
}
|
||||
let config = LlmConfig::resolve(&Settings::default()).expect("resolve");
|
||||
assert_eq!(config.request_timeout_secs, 300);
|
||||
// SAFETY: Cleanup
|
||||
unsafe {
|
||||
std::env::remove_var("LLM_REQUEST_TIMEOUT_SECS");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
+104
-6
@@ -13,7 +13,7 @@ mod embeddings;
|
||||
mod heartbeat;
|
||||
pub(crate) mod helpers;
|
||||
mod hygiene;
|
||||
mod llm;
|
||||
pub(crate) mod llm;
|
||||
mod routines;
|
||||
mod safety;
|
||||
mod sandbox;
|
||||
@@ -24,7 +24,7 @@ mod tunnel;
|
||||
mod wasm;
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::sync::OnceLock;
|
||||
use std::sync::{LazyLock, Mutex};
|
||||
|
||||
use crate::error::ConfigError;
|
||||
use crate::settings::Settings;
|
||||
@@ -37,7 +37,7 @@ pub use self::database::{DatabaseBackend, DatabaseConfig, SslMode, default_libsq
|
||||
pub use self::embeddings::EmbeddingsConfig;
|
||||
pub use self::heartbeat::HeartbeatConfig;
|
||||
pub use self::hygiene::HygieneConfig;
|
||||
pub use self::llm::{CacheRetention, LlmConfig, NearAiConfig, RegistryProviderConfig};
|
||||
pub use self::llm::default_session_path;
|
||||
pub use self::routines::RoutineConfig;
|
||||
pub use self::safety::SafetyConfig;
|
||||
pub use self::sandbox::{ClaudeCodeConfig, SandboxModeConfig};
|
||||
@@ -46,6 +46,10 @@ pub use self::skills::SkillsConfig;
|
||||
pub use self::transcription::TranscriptionConfig;
|
||||
pub use self::tunnel::TunnelConfig;
|
||||
pub use self::wasm::WasmConfig;
|
||||
pub use crate::llm::config::{
|
||||
BedrockConfig, CacheRetention, LlmConfig, NearAiConfig, OAUTH_PLACEHOLDER,
|
||||
RegistryProviderConfig,
|
||||
};
|
||||
pub use crate::llm::session::SessionConfig;
|
||||
|
||||
/// Thread-safe overlay for injected env vars (secrets loaded from DB).
|
||||
@@ -53,7 +57,12 @@ pub use crate::llm::session::SessionConfig;
|
||||
/// Used by `inject_llm_keys_from_secrets()` to make API keys available to
|
||||
/// `optional_env()` without unsafe `set_var` calls. `optional_env()` checks
|
||||
/// real env vars first, then falls back to this overlay.
|
||||
static INJECTED_VARS: OnceLock<HashMap<String, String>> = OnceLock::new();
|
||||
///
|
||||
/// Uses `Mutex<HashMap>` instead of `OnceLock` so that both
|
||||
/// `inject_os_credentials()` and `inject_llm_keys_from_secrets()` can merge
|
||||
/// their data. Whichever runs first initialises the map; the second merges in.
|
||||
static INJECTED_VARS: LazyLock<Mutex<HashMap<String, String>>> =
|
||||
LazyLock::new(|| Mutex::new(HashMap::new()));
|
||||
|
||||
/// Main configuration for the agent.
|
||||
#[derive(Debug, Clone)]
|
||||
@@ -252,6 +261,32 @@ impl Config {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Re-resolve only the LLM config after credential injection.
|
||||
///
|
||||
/// Called by `AppBuilder::init_secrets()` after injecting API keys into
|
||||
/// the env overlay. Only rebuilds `self.llm` — all other config fields
|
||||
/// are unaffected, preserving values from the initial config load (or
|
||||
/// from `Config::for_testing()` in test mode).
|
||||
pub async fn re_resolve_llm(
|
||||
&mut self,
|
||||
store: Option<&(dyn crate::db::SettingsStore + Sync)>,
|
||||
user_id: &str,
|
||||
toml_path: Option<&std::path::Path>,
|
||||
) -> Result<(), ConfigError> {
|
||||
let settings = if let Some(store) = store {
|
||||
let mut s = match store.get_all_settings(user_id).await {
|
||||
Ok(map) => Settings::from_db_map(&map),
|
||||
Err(_) => Settings::default(),
|
||||
};
|
||||
Self::apply_toml_overlay(&mut s, toml_path)?;
|
||||
s
|
||||
} else {
|
||||
Settings::default()
|
||||
};
|
||||
self.llm = LlmConfig::resolve(&settings)?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Build config from settings (shared by from_env and from_db).
|
||||
async fn build(settings: &Settings) -> Result<Self, ConfigError> {
|
||||
Ok(Self {
|
||||
@@ -285,6 +320,9 @@ impl Config {
|
||||
/// env-var-first resolution in `LlmConfig::resolve()`. Keys in the overlay
|
||||
/// are read by `optional_env()` before falling back to `std::env::var()`,
|
||||
/// so explicit env vars always win.
|
||||
///
|
||||
/// Also loads tokens from OS credential stores (macOS Keychain, Linux
|
||||
/// credentials files) which don't require the secrets DB.
|
||||
pub async fn inject_llm_keys_from_secrets(
|
||||
secrets: &dyn crate::secrets::SecretsStore,
|
||||
user_id: &str,
|
||||
@@ -292,7 +330,10 @@ pub async fn inject_llm_keys_from_secrets(
|
||||
// Static mappings for well-known providers.
|
||||
// The registry's setup hints define secret_name -> env_var mappings,
|
||||
// so new providers added to providers.json get injection automatically.
|
||||
let mut mappings: Vec<(&str, &str)> = vec![("llm_nearai_api_key", "NEARAI_API_KEY")];
|
||||
let mut mappings: Vec<(&str, &str)> = vec![
|
||||
("llm_nearai_api_key", "NEARAI_API_KEY"),
|
||||
("llm_anthropic_oauth_token", "ANTHROPIC_OAUTH_TOKEN"),
|
||||
];
|
||||
|
||||
// Dynamically discover secret->env mappings from the provider registry.
|
||||
// Uses selectable() which deduplicates user overrides correctly.
|
||||
@@ -331,5 +372,62 @@ pub async fn inject_llm_keys_from_secrets(
|
||||
}
|
||||
}
|
||||
|
||||
let _ = INJECTED_VARS.set(injected);
|
||||
inject_os_credential_store_tokens(&mut injected);
|
||||
|
||||
merge_injected_vars(injected);
|
||||
}
|
||||
|
||||
/// Load tokens from OS credential stores (no DB required).
|
||||
///
|
||||
/// Called unconditionally during startup — even when the encrypted secrets DB
|
||||
/// is unavailable (no master key, no DB connection). This ensures OAuth tokens
|
||||
/// from `claude login` (macOS Keychain / Linux credentials.json)
|
||||
/// are available for config resolution.
|
||||
pub fn inject_os_credentials() {
|
||||
let mut injected = HashMap::new();
|
||||
inject_os_credential_store_tokens(&mut injected);
|
||||
merge_injected_vars(injected);
|
||||
}
|
||||
|
||||
/// Merge new entries into the global injected-vars overlay.
|
||||
///
|
||||
/// New keys are inserted; existing keys are overwritten (later callers win,
|
||||
/// e.g. fresh OS credential store tokens override stale DB copies).
|
||||
fn merge_injected_vars(new_entries: HashMap<String, String>) {
|
||||
if new_entries.is_empty() {
|
||||
return;
|
||||
}
|
||||
match INJECTED_VARS.lock() {
|
||||
Ok(mut map) => map.extend(new_entries),
|
||||
Err(poisoned) => poisoned.into_inner().extend(new_entries),
|
||||
}
|
||||
}
|
||||
|
||||
/// Inject a single key-value pair into the overlay.
|
||||
///
|
||||
/// Used by the setup wizard to make credentials available to `optional_env()`
|
||||
/// without calling `unsafe { std::env::set_var }`.
|
||||
pub fn inject_single_var(key: &str, value: &str) {
|
||||
match INJECTED_VARS.lock() {
|
||||
Ok(mut map) => {
|
||||
map.insert(key.to_string(), value.to_string());
|
||||
}
|
||||
Err(poisoned) => {
|
||||
poisoned
|
||||
.into_inner()
|
||||
.insert(key.to_string(), value.to_string());
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Shared helper: extract tokens from OS credential stores into the overlay map.
|
||||
fn inject_os_credential_store_tokens(injected: &mut HashMap<String, String>) {
|
||||
// Try the OS credential store for a fresh Anthropic OAuth token.
|
||||
// Tokens from `claude login` expire in 8-12h, so the DB copy may be stale.
|
||||
// A fresh extraction from macOS Keychain / Linux credentials.json wins
|
||||
// over the (possibly expired) copy stored in the encrypted secrets DB.
|
||||
if let Some(fresh) = crate::config::ClaudeCodeConfig::extract_oauth_token() {
|
||||
injected.insert("ANTHROPIC_OAUTH_TOKEN".to_string(), fresh);
|
||||
tracing::debug!("Refreshed ANTHROPIC_OAUTH_TOKEN from OS credential store");
|
||||
}
|
||||
}
|
||||
|
||||
+46
-5
@@ -20,6 +20,10 @@ pub struct SandboxModeConfig {
|
||||
pub auto_pull_image: bool,
|
||||
/// Additional domains to allow through the network proxy.
|
||||
pub extra_allowed_domains: Vec<String>,
|
||||
/// How often the reaper scans for orphaned containers (seconds). Default: 300 (5 min).
|
||||
pub reaper_interval_secs: u64,
|
||||
/// Containers older than this with no active job are reaped (seconds). Default: 600 (10 min).
|
||||
pub orphan_threshold_secs: u64,
|
||||
}
|
||||
|
||||
impl Default for SandboxModeConfig {
|
||||
@@ -33,6 +37,8 @@ impl Default for SandboxModeConfig {
|
||||
image: "ironclaw-worker:latest".to_string(),
|
||||
auto_pull_image: true,
|
||||
extra_allowed_domains: Vec::new(),
|
||||
reaper_interval_secs: 300,
|
||||
orphan_threshold_secs: 600,
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -43,6 +49,24 @@ impl SandboxModeConfig {
|
||||
.map(|s| s.split(',').map(|d| d.trim().to_string()).collect())
|
||||
.unwrap_or_default();
|
||||
|
||||
let reaper_interval_secs: u64 = parse_optional_env("SANDBOX_REAPER_INTERVAL_SECS", 300)?;
|
||||
let orphan_threshold_secs: u64 = parse_optional_env("SANDBOX_ORPHAN_THRESHOLD_SECS", 600)?;
|
||||
|
||||
// Validate that reaper timings are non-zero to prevent tokio::time::interval panics
|
||||
if reaper_interval_secs == 0 {
|
||||
return Err(ConfigError::InvalidValue {
|
||||
key: "SANDBOX_REAPER_INTERVAL_SECS".to_string(),
|
||||
message: "must be greater than 0".to_string(),
|
||||
});
|
||||
}
|
||||
|
||||
if orphan_threshold_secs == 0 {
|
||||
return Err(ConfigError::InvalidValue {
|
||||
key: "SANDBOX_ORPHAN_THRESHOLD_SECS".to_string(),
|
||||
message: "must be greater than 0".to_string(),
|
||||
});
|
||||
}
|
||||
|
||||
Ok(Self {
|
||||
enabled: parse_bool_env("SANDBOX_ENABLED", true)?,
|
||||
policy: parse_string_env("SANDBOX_POLICY", "readonly")?,
|
||||
@@ -52,6 +76,8 @@ impl SandboxModeConfig {
|
||||
image: parse_string_env("SANDBOX_IMAGE", "ironclaw-worker:latest")?,
|
||||
auto_pull_image: parse_bool_env("SANDBOX_AUTO_PULL", true)?,
|
||||
extra_allowed_domains: extra_domains,
|
||||
reaper_interval_secs,
|
||||
orphan_threshold_secs,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -233,9 +259,14 @@ impl ClaudeCodeConfig {
|
||||
/// Expected shape: `{"claudeAiOauth": {"accessToken": "sk-ant-oat01-..."}}`
|
||||
fn parse_oauth_access_token(json: &str) -> Option<String> {
|
||||
let creds: serde_json::Value = serde_json::from_str(json).ok()?;
|
||||
creds["claudeAiOauth"]["accessToken"]
|
||||
.as_str()
|
||||
.map(String::from)
|
||||
let token = creds["claudeAiOauth"]["accessToken"].as_str()?;
|
||||
// Validate that the token looks like a real OAuth token before using it.
|
||||
// Claude CLI tokens start with "sk-ant-oat".
|
||||
if !token.starts_with("sk-ant-oat") {
|
||||
tracing::debug!("Ignoring credential store token with unexpected prefix");
|
||||
return None;
|
||||
}
|
||||
Some(token.to_string())
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
@@ -268,6 +299,8 @@ mod tests {
|
||||
image: "custom-worker:v2".to_string(),
|
||||
auto_pull_image: false,
|
||||
extra_allowed_domains: vec!["example.com".to_string()],
|
||||
reaper_interval_secs: 300,
|
||||
orphan_threshold_secs: 600,
|
||||
};
|
||||
assert!(!cfg.enabled);
|
||||
assert_eq!(cfg.policy, "full_access");
|
||||
@@ -290,6 +323,8 @@ mod tests {
|
||||
image: "test:latest".to_string(),
|
||||
auto_pull_image: false,
|
||||
extra_allowed_domains: vec!["custom.example.com".to_string()],
|
||||
reaper_interval_secs: 300,
|
||||
orphan_threshold_secs: 600,
|
||||
};
|
||||
let sc = mode.to_sandbox_config();
|
||||
assert!(sc.enabled);
|
||||
@@ -401,14 +436,14 @@ mod tests {
|
||||
fn parse_oauth_token_nested_extra_fields() {
|
||||
let json = r#"{
|
||||
"claudeAiOauth": {
|
||||
"accessToken": "sk-ant-real-token",
|
||||
"accessToken": "sk-ant-oat01-real-token",
|
||||
"refreshToken": "rt-abc",
|
||||
"expiresAt": 1700000000
|
||||
}
|
||||
}"#;
|
||||
assert_eq!(
|
||||
parse_oauth_access_token(json),
|
||||
Some("sk-ant-real-token".to_string())
|
||||
Some("sk-ant-oat01-real-token".to_string())
|
||||
);
|
||||
}
|
||||
|
||||
@@ -418,6 +453,12 @@ mod tests {
|
||||
assert_eq!(parse_oauth_access_token(json), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parse_oauth_token_rejects_invalid_prefix() {
|
||||
let json = r#"{"claudeAiOauth": {"accessToken": "not-an-oauth-token"}}"#;
|
||||
assert_eq!(parse_oauth_access_token(json), None);
|
||||
}
|
||||
|
||||
// ── default_claude_code_allowed_tools ───────────────────────────
|
||||
|
||||
#[test]
|
||||
|
||||
@@ -164,6 +164,8 @@ pub struct JobContext {
|
||||
/// previous results by ID via `$tool_call_id` parameter syntax.
|
||||
#[serde(skip)]
|
||||
pub tool_output_stash: Arc<tokio::sync::RwLock<HashMap<String, String>>>,
|
||||
/// User's preferred timezone (IANA name, e.g. "America/New_York"). Defaults to "UTC".
|
||||
pub user_timezone: String,
|
||||
}
|
||||
|
||||
impl JobContext {
|
||||
@@ -203,9 +205,16 @@ impl JobContext {
|
||||
http_interceptor: None,
|
||||
metadata: serde_json::Value::Null,
|
||||
tool_output_stash: Arc::new(tokio::sync::RwLock::new(HashMap::new())),
|
||||
user_timezone: "UTC".to_string(),
|
||||
}
|
||||
}
|
||||
|
||||
/// Set the user timezone on this context.
|
||||
pub fn with_timezone(mut self, tz: impl Into<String>) -> Self {
|
||||
self.user_timezone = tz.into();
|
||||
self
|
||||
}
|
||||
|
||||
/// Transition to a new state.
|
||||
pub fn transition_to(
|
||||
&mut self,
|
||||
|
||||
+346
-11
@@ -20,9 +20,10 @@ impl ConversationStore for LibSqlBackend {
|
||||
) -> Result<Uuid, DatabaseError> {
|
||||
let conn = self.connect().await?;
|
||||
let id = Uuid::new_v4();
|
||||
let now = fmt_ts(&Utc::now());
|
||||
conn.execute(
|
||||
"INSERT INTO conversations (id, channel, user_id, thread_id) VALUES (?1, ?2, ?3, ?4)",
|
||||
params![id.to_string(), channel, user_id, opt_text(thread_id)],
|
||||
"INSERT INTO conversations (id, channel, user_id, thread_id, started_at, last_activity) VALUES (?1, ?2, ?3, ?4, ?5, ?5)",
|
||||
params![id.to_string(), channel, user_id, opt_text(thread_id), now],
|
||||
)
|
||||
.await
|
||||
.map_err(|e| DatabaseError::Query(e.to_string()))?;
|
||||
@@ -71,8 +72,8 @@ impl ConversationStore for LibSqlBackend {
|
||||
let now = fmt_ts(&Utc::now());
|
||||
conn.execute(
|
||||
r#"
|
||||
INSERT INTO conversations (id, channel, user_id, thread_id)
|
||||
VALUES (?1, ?2, ?3, ?4)
|
||||
INSERT INTO conversations (id, channel, user_id, thread_id, started_at, last_activity)
|
||||
VALUES (?1, ?2, ?3, ?4, ?5, ?5)
|
||||
ON CONFLICT (id) DO UPDATE SET last_activity = ?5
|
||||
"#,
|
||||
params![id.to_string(), channel, user_id, opt_text(thread_id), now],
|
||||
@@ -97,6 +98,7 @@ impl ConversationStore for LibSqlBackend {
|
||||
c.started_at,
|
||||
c.last_activity,
|
||||
c.metadata,
|
||||
c.channel,
|
||||
(SELECT COUNT(*) FROM conversation_messages m WHERE m.conversation_id = c.id AND m.role = 'user') AS message_count,
|
||||
(SELECT substr(m2.content, 1, 100)
|
||||
FROM conversation_messages m2
|
||||
@@ -106,7 +108,7 @@ impl ConversationStore for LibSqlBackend {
|
||||
) AS title
|
||||
FROM conversations c
|
||||
WHERE c.user_id = ?1 AND c.channel = ?2
|
||||
ORDER BY c.last_activity DESC
|
||||
ORDER BY datetime(c.last_activity) DESC
|
||||
LIMIT ?3
|
||||
"#,
|
||||
params![user_id, channel, limit],
|
||||
@@ -125,6 +127,13 @@ impl ConversationStore for LibSqlBackend {
|
||||
.get("thread_type")
|
||||
.and_then(|v| v.as_str())
|
||||
.map(String::from);
|
||||
let sql_title = get_opt_text(&row, 6);
|
||||
let title = sql_title.or_else(|| {
|
||||
metadata
|
||||
.get("routine_name")
|
||||
.and_then(|v| v.as_str())
|
||||
.map(String::from)
|
||||
});
|
||||
results.push(ConversationSummary {
|
||||
id: row
|
||||
.get::<String>(0)
|
||||
@@ -133,14 +142,213 @@ impl ConversationStore for LibSqlBackend {
|
||||
.unwrap_or_default(),
|
||||
started_at: get_ts(&row, 1),
|
||||
last_activity: get_ts(&row, 2),
|
||||
message_count: get_i64(&row, 4),
|
||||
title: get_opt_text(&row, 5),
|
||||
message_count: get_i64(&row, 5),
|
||||
title,
|
||||
thread_type,
|
||||
channel: get_text(&row, 4),
|
||||
});
|
||||
}
|
||||
Ok(results)
|
||||
}
|
||||
|
||||
async fn list_conversations_all_channels(
|
||||
&self,
|
||||
user_id: &str,
|
||||
limit: i64,
|
||||
) -> Result<Vec<ConversationSummary>, DatabaseError> {
|
||||
let conn = self.connect().await?;
|
||||
let mut rows = conn
|
||||
.query(
|
||||
r#"
|
||||
SELECT
|
||||
c.id,
|
||||
c.started_at,
|
||||
c.last_activity,
|
||||
c.metadata,
|
||||
c.channel,
|
||||
(SELECT COUNT(*) FROM conversation_messages m WHERE m.conversation_id = c.id AND m.role = 'user') AS message_count,
|
||||
(SELECT substr(m2.content, 1, 100)
|
||||
FROM conversation_messages m2
|
||||
WHERE m2.conversation_id = c.id AND m2.role = 'user'
|
||||
ORDER BY m2.created_at ASC, m2.rowid ASC
|
||||
LIMIT 1
|
||||
) AS title
|
||||
FROM conversations c
|
||||
WHERE c.user_id = ?1
|
||||
ORDER BY datetime(c.last_activity) DESC
|
||||
LIMIT ?2
|
||||
"#,
|
||||
params![user_id, limit],
|
||||
)
|
||||
.await
|
||||
.map_err(|e| DatabaseError::Query(e.to_string()))?;
|
||||
|
||||
let mut results = Vec::new();
|
||||
while let Some(row) = rows
|
||||
.next()
|
||||
.await
|
||||
.map_err(|e| DatabaseError::Query(e.to_string()))?
|
||||
{
|
||||
let metadata = get_json(&row, 3);
|
||||
let thread_type = metadata
|
||||
.get("thread_type")
|
||||
.and_then(|v| v.as_str())
|
||||
.map(String::from);
|
||||
let sql_title = get_opt_text(&row, 6);
|
||||
let title = sql_title.or_else(|| {
|
||||
metadata
|
||||
.get("routine_name")
|
||||
.and_then(|v| v.as_str())
|
||||
.map(String::from)
|
||||
});
|
||||
results.push(ConversationSummary {
|
||||
id: row
|
||||
.get::<String>(0)
|
||||
.unwrap_or_default()
|
||||
.parse()
|
||||
.unwrap_or_default(),
|
||||
started_at: get_ts(&row, 1),
|
||||
last_activity: get_ts(&row, 2),
|
||||
message_count: get_i64(&row, 5),
|
||||
title,
|
||||
thread_type,
|
||||
channel: get_text(&row, 4),
|
||||
});
|
||||
}
|
||||
Ok(results)
|
||||
}
|
||||
|
||||
/// Uses BEGIN IMMEDIATE to serialize concurrent writers and prevent
|
||||
/// duplicate routine conversations (TOCTOU race).
|
||||
async fn get_or_create_routine_conversation(
|
||||
&self,
|
||||
routine_id: Uuid,
|
||||
routine_name: &str,
|
||||
user_id: &str,
|
||||
) -> Result<Uuid, DatabaseError> {
|
||||
let conn = self.connect().await?;
|
||||
let rid = routine_id.to_string();
|
||||
|
||||
conn.execute("BEGIN IMMEDIATE", params![])
|
||||
.await
|
||||
.map_err(|e| DatabaseError::Query(e.to_string()))?;
|
||||
|
||||
let result: Result<Uuid, DatabaseError> = async {
|
||||
let mut rows = conn
|
||||
.query(
|
||||
r#"
|
||||
SELECT id FROM conversations
|
||||
WHERE user_id = ?1 AND json_extract(metadata, '$.routine_id') = ?2
|
||||
LIMIT 1
|
||||
"#,
|
||||
params![user_id, rid],
|
||||
)
|
||||
.await
|
||||
.map_err(|e| DatabaseError::Query(e.to_string()))?;
|
||||
|
||||
if let Some(row) = rows
|
||||
.next()
|
||||
.await
|
||||
.map_err(|e| DatabaseError::Query(e.to_string()))?
|
||||
{
|
||||
let id_str: String = row.get(0).unwrap_or_default();
|
||||
return id_str
|
||||
.parse()
|
||||
.map_err(|_| DatabaseError::Serialization("Invalid UUID".to_string()));
|
||||
}
|
||||
|
||||
let id = Uuid::new_v4();
|
||||
let now = fmt_ts(&Utc::now());
|
||||
let metadata = serde_json::json!({
|
||||
"thread_type": "routine",
|
||||
"routine_id": routine_id.to_string(),
|
||||
"routine_name": routine_name,
|
||||
});
|
||||
conn.execute(
|
||||
"INSERT INTO conversations (id, channel, user_id, metadata, started_at, last_activity) VALUES (?1, ?2, ?3, ?4, ?5, ?5)",
|
||||
params![id.to_string(), "routine", user_id, metadata.to_string(), now],
|
||||
)
|
||||
.await
|
||||
.map_err(|e| DatabaseError::Query(e.to_string()))?;
|
||||
Ok(id)
|
||||
}
|
||||
.await;
|
||||
|
||||
match &result {
|
||||
Ok(_) => {
|
||||
conn.execute("COMMIT", params![])
|
||||
.await
|
||||
.map_err(|e| DatabaseError::Query(e.to_string()))?;
|
||||
}
|
||||
Err(_) => {
|
||||
let _ = conn.execute("ROLLBACK", params![]).await;
|
||||
}
|
||||
}
|
||||
result
|
||||
}
|
||||
|
||||
/// Uses BEGIN IMMEDIATE to serialize concurrent writers and prevent
|
||||
/// duplicate heartbeat conversations (TOCTOU race).
|
||||
async fn get_or_create_heartbeat_conversation(
|
||||
&self,
|
||||
user_id: &str,
|
||||
) -> Result<Uuid, DatabaseError> {
|
||||
let conn = self.connect().await?;
|
||||
|
||||
conn.execute("BEGIN IMMEDIATE", params![])
|
||||
.await
|
||||
.map_err(|e| DatabaseError::Query(e.to_string()))?;
|
||||
|
||||
let result: Result<Uuid, DatabaseError> = async {
|
||||
let mut rows = conn
|
||||
.query(
|
||||
r#"
|
||||
SELECT id FROM conversations
|
||||
WHERE user_id = ?1 AND json_extract(metadata, '$.thread_type') = 'heartbeat'
|
||||
LIMIT 1
|
||||
"#,
|
||||
params![user_id],
|
||||
)
|
||||
.await
|
||||
.map_err(|e| DatabaseError::Query(e.to_string()))?;
|
||||
|
||||
if let Some(row) = rows
|
||||
.next()
|
||||
.await
|
||||
.map_err(|e| DatabaseError::Query(e.to_string()))?
|
||||
{
|
||||
let id_str: String = row.get(0).unwrap_or_default();
|
||||
return id_str
|
||||
.parse()
|
||||
.map_err(|_| DatabaseError::Serialization("Invalid UUID".to_string()));
|
||||
}
|
||||
|
||||
let id = Uuid::new_v4();
|
||||
let now = fmt_ts(&Utc::now());
|
||||
let metadata = serde_json::json!({ "thread_type": "heartbeat" });
|
||||
conn.execute(
|
||||
"INSERT INTO conversations (id, channel, user_id, metadata, started_at, last_activity) VALUES (?1, ?2, ?3, ?4, ?5, ?5)",
|
||||
params![id.to_string(), "heartbeat", user_id, metadata.to_string(), now],
|
||||
)
|
||||
.await
|
||||
.map_err(|e| DatabaseError::Query(e.to_string()))?;
|
||||
Ok(id)
|
||||
}
|
||||
.await;
|
||||
|
||||
match &result {
|
||||
Ok(_) => {
|
||||
conn.execute("COMMIT", params![])
|
||||
.await
|
||||
.map_err(|e| DatabaseError::Query(e.to_string()))?;
|
||||
}
|
||||
Err(_) => {
|
||||
let _ = conn.execute("ROLLBACK", params![]).await;
|
||||
}
|
||||
}
|
||||
result
|
||||
}
|
||||
|
||||
async fn get_or_create_assistant_conversation(
|
||||
&self,
|
||||
user_id: &str,
|
||||
@@ -174,10 +382,11 @@ impl ConversationStore for LibSqlBackend {
|
||||
|
||||
// Create new
|
||||
let id = Uuid::new_v4();
|
||||
let now = fmt_ts(&Utc::now());
|
||||
let metadata = serde_json::json!({"thread_type": "assistant", "title": "Assistant"});
|
||||
conn.execute(
|
||||
"INSERT INTO conversations (id, channel, user_id, metadata) VALUES (?1, ?2, ?3, ?4)",
|
||||
params![id.to_string(), channel, user_id, metadata.to_string()],
|
||||
"INSERT INTO conversations (id, channel, user_id, metadata, started_at, last_activity) VALUES (?1, ?2, ?3, ?4, ?5, ?5)",
|
||||
params![id.to_string(), channel, user_id, metadata.to_string(), now],
|
||||
)
|
||||
.await
|
||||
.map_err(|e| DatabaseError::Query(e.to_string()))?;
|
||||
@@ -192,9 +401,10 @@ impl ConversationStore for LibSqlBackend {
|
||||
) -> Result<Uuid, DatabaseError> {
|
||||
let conn = self.connect().await?;
|
||||
let id = Uuid::new_v4();
|
||||
let now = fmt_ts(&Utc::now());
|
||||
conn.execute(
|
||||
"INSERT INTO conversations (id, channel, user_id, metadata) VALUES (?1, ?2, ?3, ?4)",
|
||||
params![id.to_string(), channel, user_id, metadata.to_string()],
|
||||
"INSERT INTO conversations (id, channel, user_id, metadata, started_at, last_activity) VALUES (?1, ?2, ?3, ?4, ?5, ?5)",
|
||||
params![id.to_string(), channel, user_id, metadata.to_string(), now],
|
||||
)
|
||||
.await
|
||||
.map_err(|e| DatabaseError::Query(e.to_string()))?;
|
||||
@@ -353,3 +563,128 @@ impl ConversationStore for LibSqlBackend {
|
||||
Ok(found.is_some())
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::db::Database;
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_get_or_create_routine_conversation_is_idempotent() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let db_path = dir.path().join("test_routine_conv.db");
|
||||
let backend = LibSqlBackend::new_local(&db_path).await.unwrap();
|
||||
backend.run_migrations().await.unwrap();
|
||||
|
||||
let routine_id = Uuid::new_v4();
|
||||
let user_id = "test_user";
|
||||
|
||||
// First call — creates the conversation
|
||||
let id1 = backend
|
||||
.get_or_create_routine_conversation(routine_id, "my-routine", user_id)
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
// Second call — should return the SAME conversation
|
||||
let id2 = backend
|
||||
.get_or_create_routine_conversation(routine_id, "my-routine", user_id)
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
assert_eq!(id1, id2, "Expected same conversation ID on repeated calls");
|
||||
|
||||
// Third call — still the same
|
||||
let id3 = backend
|
||||
.get_or_create_routine_conversation(routine_id, "my-routine", user_id)
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
assert_eq!(id1, id3);
|
||||
|
||||
// Different routine_id should get a different conversation
|
||||
let other_routine_id = Uuid::new_v4();
|
||||
let id4 = backend
|
||||
.get_or_create_routine_conversation(other_routine_id, "other-routine", user_id)
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
assert_ne!(
|
||||
id1, id4,
|
||||
"Different routines should get different conversations"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_routine_conversation_persists_across_messages() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let db_path = dir.path().join("test_routine_persist.db");
|
||||
let backend = LibSqlBackend::new_local(&db_path).await.unwrap();
|
||||
backend.run_migrations().await.unwrap();
|
||||
|
||||
let routine_id = Uuid::new_v4();
|
||||
let user_id = "test_user";
|
||||
|
||||
// First invocation: create conversation and add a message
|
||||
let id1 = backend
|
||||
.get_or_create_routine_conversation(routine_id, "my-routine", user_id)
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
backend
|
||||
.add_conversation_message(id1, "assistant", "[cron] Completed: all good")
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
// Second invocation: should find existing conversation
|
||||
let id2 = backend
|
||||
.get_or_create_routine_conversation(routine_id, "my-routine", user_id)
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
assert_eq!(id1, id2, "Second invocation should reuse same conversation");
|
||||
|
||||
backend
|
||||
.add_conversation_message(id2, "assistant", "[cron] Completed: still good")
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
// Verify only one routine conversation exists (not two)
|
||||
let convs = backend
|
||||
.list_conversations_all_channels(user_id, 50)
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
let routine_convs: Vec<_> = convs.iter().filter(|c| c.channel == "routine").collect();
|
||||
assert_eq!(
|
||||
routine_convs.len(),
|
||||
1,
|
||||
"Should have exactly 1 routine conversation, found {}",
|
||||
routine_convs.len()
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_get_or_create_heartbeat_conversation_is_idempotent() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let db_path = dir.path().join("test_heartbeat_conv.db");
|
||||
let backend = LibSqlBackend::new_local(&db_path).await.unwrap();
|
||||
backend.run_migrations().await.unwrap();
|
||||
|
||||
let user_id = "test_user";
|
||||
|
||||
let id1 = backend
|
||||
.get_or_create_heartbeat_conversation(user_id)
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
let id2 = backend
|
||||
.get_or_create_heartbeat_conversation(user_id)
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
assert_eq!(
|
||||
id1, id2,
|
||||
"Expected same heartbeat conversation on repeated calls"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -28,14 +28,16 @@ impl JobStore for LibSqlBackend {
|
||||
r#"
|
||||
INSERT INTO agent_jobs (
|
||||
id, conversation_id, title, description, category, status, source,
|
||||
user_id,
|
||||
budget_amount, budget_token, bid_amount, estimated_cost, estimated_time_secs,
|
||||
actual_cost, repair_attempts, created_at, started_at, completed_at
|
||||
) VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10, ?11, ?12, ?13, ?14, ?15, ?16, ?17)
|
||||
) VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10, ?11, ?12, ?13, ?14, ?15, ?16, ?17, ?18)
|
||||
ON CONFLICT (id) DO UPDATE SET
|
||||
title = excluded.title,
|
||||
description = excluded.description,
|
||||
category = excluded.category,
|
||||
status = excluded.status,
|
||||
user_id = excluded.user_id,
|
||||
estimated_cost = excluded.estimated_cost,
|
||||
estimated_time_secs = excluded.estimated_time_secs,
|
||||
actual_cost = excluded.actual_cost,
|
||||
@@ -51,6 +53,7 @@ impl JobStore for LibSqlBackend {
|
||||
opt_text(ctx.category.as_deref()),
|
||||
status,
|
||||
"direct",
|
||||
ctx.user_id.as_str(),
|
||||
opt_text_owned(ctx.budget.map(|d| d.to_string())),
|
||||
opt_text(ctx.budget_token.as_deref()),
|
||||
opt_text_owned(ctx.bid_amount.map(|d| d.to_string())),
|
||||
@@ -121,6 +124,9 @@ impl JobStore for LibSqlBackend {
|
||||
tool_output_stash: std::sync::Arc::new(tokio::sync::RwLock::new(
|
||||
std::collections::HashMap::new(),
|
||||
)),
|
||||
// TODO(#661): persist user_timezone in agent_jobs table so
|
||||
// background/routine jobs retain the session's timezone context.
|
||||
user_timezone: "UTC".to_string(),
|
||||
}))
|
||||
}
|
||||
None => Ok(None),
|
||||
|
||||
+125
-9
@@ -118,15 +118,37 @@ impl LibSqlBackend {
|
||||
/// Sets `PRAGMA busy_timeout = 5000` on every connection so concurrent
|
||||
/// writers wait up to 5 seconds instead of failing instantly with
|
||||
/// "database is locked".
|
||||
///
|
||||
/// Retries up to 3 times with exponential backoff to handle transient
|
||||
/// "unable to open database file" errors from concurrent connection
|
||||
/// creation (e.g. cron ticker vs main thread).
|
||||
pub async fn connect(&self) -> Result<Connection, DatabaseError> {
|
||||
let conn = self
|
||||
.db
|
||||
.connect()
|
||||
.map_err(|e| DatabaseError::Pool(format!("Failed to create connection: {}", e)))?;
|
||||
conn.query("PRAGMA busy_timeout = 5000", ())
|
||||
.await
|
||||
.map_err(|e| DatabaseError::Pool(format!("Failed to set busy_timeout: {}", e)))?;
|
||||
Ok(conn)
|
||||
let mut last_err = None;
|
||||
for attempt in 0..3u32 {
|
||||
match self.db.connect() {
|
||||
Ok(conn) => {
|
||||
conn.query("PRAGMA busy_timeout = 5000", ())
|
||||
.await
|
||||
.map_err(|e| {
|
||||
DatabaseError::Pool(format!("Failed to set busy_timeout: {}", e))
|
||||
})?;
|
||||
return Ok(conn);
|
||||
}
|
||||
Err(e) => {
|
||||
last_err = Some(e);
|
||||
if attempt < 2 {
|
||||
tokio::time::sleep(std::time::Duration::from_millis(
|
||||
50 * 2u64.pow(attempt),
|
||||
))
|
||||
.await;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
Err(DatabaseError::Pool(format!(
|
||||
"Failed to create connection after 3 attempts: {}",
|
||||
last_err.map(|e| e.to_string()).unwrap_or_default()
|
||||
)))
|
||||
}
|
||||
}
|
||||
|
||||
@@ -147,10 +169,18 @@ pub(crate) fn parse_timestamp(s: &str) -> Result<DateTime<Utc>, String> {
|
||||
}
|
||||
// Naive with fractional seconds (legacy or SQLite datetime() output)
|
||||
if let Ok(ndt) = NaiveDateTime::parse_from_str(s, "%Y-%m-%d %H:%M:%S%.f") {
|
||||
tracing::warn!(
|
||||
timestamp = %s,
|
||||
"parsed naive timestamp without timezone; assuming UTC for backward compatibility"
|
||||
);
|
||||
return Ok(ndt.and_utc());
|
||||
}
|
||||
// Naive without fractional seconds (legacy format)
|
||||
if let Ok(ndt) = NaiveDateTime::parse_from_str(s, "%Y-%m-%d %H:%M:%S") {
|
||||
tracing::warn!(
|
||||
timestamp = %s,
|
||||
"parsed naive timestamp without timezone; assuming UTC for backward compatibility"
|
||||
);
|
||||
return Ok(ndt.and_utc());
|
||||
}
|
||||
Err(format!("unparseable timestamp: {:?}", s))
|
||||
@@ -380,8 +410,47 @@ pub(crate) fn row_to_routine_run_libsql(row: &libsql::Row) -> Result<RoutineRun,
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use chrono::{TimeZone, Utc};
|
||||
|
||||
use crate::db::Database;
|
||||
use crate::db::libsql::LibSqlBackend;
|
||||
use crate::db::libsql::{LibSqlBackend, parse_timestamp};
|
||||
|
||||
#[test]
|
||||
fn test_parse_timestamp_accepts_rfc3339_and_legacy_naive_formats() {
|
||||
let expected = Utc.with_ymd_and_hms(2026, 3, 7, 12, 34, 56).unwrap();
|
||||
|
||||
let with_millis = parse_timestamp("2026-03-07T12:34:56.789Z").unwrap();
|
||||
assert_eq!(with_millis, expected + chrono::Duration::milliseconds(789));
|
||||
|
||||
let naive_with_millis = parse_timestamp("2026-03-07 12:34:56.789").unwrap();
|
||||
assert_eq!(
|
||||
naive_with_millis,
|
||||
expected + chrono::Duration::milliseconds(789)
|
||||
);
|
||||
|
||||
let naive_without_millis = parse_timestamp("2026-03-07 12:34:56").unwrap();
|
||||
assert_eq!(naive_without_millis, expected);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_libsql_now_format_is_rfc3339_and_parseable() {
|
||||
let backend = LibSqlBackend::new_memory().await.unwrap();
|
||||
backend.run_migrations().await.unwrap();
|
||||
|
||||
let conn = backend.connect().await.unwrap();
|
||||
let mut rows = conn
|
||||
.query("SELECT strftime('%Y-%m-%dT%H:%M:%fZ', 'now')", ())
|
||||
.await
|
||||
.unwrap();
|
||||
let row = rows.next().await.unwrap().unwrap();
|
||||
let ts: String = row.get(0).unwrap();
|
||||
|
||||
let parsed = parse_timestamp(&ts).unwrap();
|
||||
assert_eq!(
|
||||
ts,
|
||||
parsed.to_rfc3339_opts(chrono::SecondsFormat::Millis, true)
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_wal_mode_after_migrations() {
|
||||
@@ -413,6 +482,24 @@ mod tests {
|
||||
assert_eq!(timeout, 5000);
|
||||
}
|
||||
|
||||
/// Regression test: save_job must persist user_id and get_job must return it.
|
||||
#[tokio::test]
|
||||
async fn test_save_job_persists_user_id() {
|
||||
use crate::context::JobContext;
|
||||
use crate::db::JobStore;
|
||||
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let db_path = dir.path().join("test_user_id.db");
|
||||
let backend = LibSqlBackend::new_local(&db_path).await.unwrap();
|
||||
backend.run_migrations().await.unwrap();
|
||||
|
||||
let ctx = JobContext::with_user("test-user-42", "Test Job", "A test job");
|
||||
backend.save_job(&ctx).await.unwrap();
|
||||
|
||||
let loaded = backend.get_job(ctx.job_id).await.unwrap().unwrap();
|
||||
assert_eq!(loaded.user_id, "test-user-42");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_concurrent_writes_succeed() {
|
||||
// Use a temp file so connections share state (in-memory DBs are connection-local)
|
||||
@@ -459,4 +546,33 @@ mod tests {
|
||||
let count: i64 = row.get(0).unwrap();
|
||||
assert_eq!(count, 20);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_connect_retry_succeeds_on_valid_db() {
|
||||
// Verify connect() works with retry logic on a file-backed DB
|
||||
// (exercises the retry path even though transient failures are hard
|
||||
// to reproduce deterministically).
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let db_path = dir.path().join("test_retry.db");
|
||||
let backend = LibSqlBackend::new_local(&db_path).await.unwrap();
|
||||
backend.run_migrations().await.unwrap();
|
||||
|
||||
// Multiple concurrent connect() calls should all succeed
|
||||
let mut handles = Vec::new();
|
||||
for _ in 0..10 {
|
||||
let b = LibSqlBackend {
|
||||
db: backend.shared_db(),
|
||||
};
|
||||
handles.push(tokio::spawn(async move { b.connect().await }));
|
||||
}
|
||||
|
||||
for handle in handles {
|
||||
let result = handle.await.unwrap();
|
||||
assert!(
|
||||
result.is_ok(),
|
||||
"concurrent connect failed: {:?}",
|
||||
result.err()
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
+64
-55
@@ -26,7 +26,7 @@ pub const SCHEMA: &str = r#"
|
||||
CREATE TABLE IF NOT EXISTS _migrations (
|
||||
version INTEGER PRIMARY KEY,
|
||||
name TEXT NOT NULL,
|
||||
applied_at TEXT NOT NULL DEFAULT (datetime('now'))
|
||||
applied_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now'))
|
||||
);
|
||||
|
||||
-- ==================== Conversations ====================
|
||||
@@ -36,8 +36,8 @@ CREATE TABLE IF NOT EXISTS conversations (
|
||||
channel TEXT NOT NULL,
|
||||
user_id TEXT NOT NULL,
|
||||
thread_id TEXT,
|
||||
started_at TEXT NOT NULL DEFAULT (datetime('now')),
|
||||
last_activity TEXT NOT NULL DEFAULT (datetime('now')),
|
||||
started_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now')),
|
||||
last_activity TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now')),
|
||||
metadata TEXT NOT NULL DEFAULT '{}'
|
||||
);
|
||||
|
||||
@@ -45,12 +45,21 @@ CREATE INDEX IF NOT EXISTS idx_conversations_channel ON conversations(channel);
|
||||
CREATE INDEX IF NOT EXISTS idx_conversations_user ON conversations(user_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_conversations_last_activity ON conversations(last_activity);
|
||||
|
||||
-- Partial unique indexes to prevent duplicate singleton conversations.
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS uq_conv_routine
|
||||
ON conversations (user_id, json_extract(metadata, '$.routine_id'))
|
||||
WHERE json_extract(metadata, '$.routine_id') IS NOT NULL;
|
||||
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS uq_conv_heartbeat
|
||||
ON conversations (user_id)
|
||||
WHERE json_extract(metadata, '$.thread_type') = 'heartbeat';
|
||||
|
||||
CREATE TABLE IF NOT EXISTS conversation_messages (
|
||||
id TEXT PRIMARY KEY,
|
||||
conversation_id TEXT NOT NULL REFERENCES conversations(id) ON DELETE CASCADE,
|
||||
role TEXT NOT NULL,
|
||||
content TEXT NOT NULL,
|
||||
created_at TEXT NOT NULL DEFAULT (datetime('now'))
|
||||
created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now'))
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_conversation_messages_conversation
|
||||
@@ -82,7 +91,7 @@ CREATE TABLE IF NOT EXISTS agent_jobs (
|
||||
failure_reason TEXT,
|
||||
stuck_since TEXT,
|
||||
repair_attempts INTEGER NOT NULL DEFAULT 0,
|
||||
created_at TEXT NOT NULL DEFAULT (datetime('now')),
|
||||
created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now')),
|
||||
started_at TEXT,
|
||||
completed_at TEXT
|
||||
);
|
||||
@@ -107,7 +116,7 @@ CREATE TABLE IF NOT EXISTS job_actions (
|
||||
duration_ms INTEGER,
|
||||
success INTEGER NOT NULL,
|
||||
error_message TEXT,
|
||||
created_at TEXT NOT NULL DEFAULT (datetime('now')),
|
||||
created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now')),
|
||||
UNIQUE(job_id, sequence_num)
|
||||
);
|
||||
|
||||
@@ -128,8 +137,8 @@ CREATE TABLE IF NOT EXISTS dynamic_tools (
|
||||
failure_count INTEGER NOT NULL DEFAULT 0,
|
||||
last_error TEXT,
|
||||
status TEXT NOT NULL DEFAULT 'active',
|
||||
created_at TEXT NOT NULL DEFAULT (datetime('now')),
|
||||
updated_at TEXT NOT NULL DEFAULT (datetime('now'))
|
||||
created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now')),
|
||||
updated_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now'))
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_dynamic_tools_status ON dynamic_tools(status);
|
||||
@@ -147,7 +156,7 @@ CREATE TABLE IF NOT EXISTS llm_calls (
|
||||
output_tokens INTEGER NOT NULL,
|
||||
cost TEXT NOT NULL,
|
||||
purpose TEXT,
|
||||
created_at TEXT NOT NULL DEFAULT (datetime('now'))
|
||||
created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now'))
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_llm_calls_job ON llm_calls(job_id);
|
||||
@@ -167,7 +176,7 @@ CREATE TABLE IF NOT EXISTS estimation_snapshots (
|
||||
actual_time_secs INTEGER,
|
||||
estimated_value TEXT NOT NULL,
|
||||
actual_value TEXT,
|
||||
created_at TEXT NOT NULL DEFAULT (datetime('now'))
|
||||
created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now'))
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_estimation_category ON estimation_snapshots(category);
|
||||
@@ -183,7 +192,7 @@ CREATE TABLE IF NOT EXISTS repair_attempts (
|
||||
action_taken TEXT NOT NULL,
|
||||
success INTEGER NOT NULL,
|
||||
error_message TEXT,
|
||||
created_at TEXT NOT NULL DEFAULT (datetime('now'))
|
||||
created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now'))
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_repair_attempts_target ON repair_attempts(target_type, target_id);
|
||||
@@ -197,8 +206,8 @@ CREATE TABLE IF NOT EXISTS memory_documents (
|
||||
agent_id TEXT,
|
||||
path TEXT NOT NULL,
|
||||
content TEXT NOT NULL,
|
||||
created_at TEXT NOT NULL DEFAULT (datetime('now')),
|
||||
updated_at TEXT NOT NULL DEFAULT (datetime('now')),
|
||||
created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now')),
|
||||
updated_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now')),
|
||||
metadata TEXT NOT NULL DEFAULT '{}',
|
||||
UNIQUE (user_id, agent_id, path)
|
||||
);
|
||||
@@ -213,7 +222,7 @@ CREATE TRIGGER IF NOT EXISTS update_memory_documents_updated_at
|
||||
FOR EACH ROW
|
||||
WHEN NEW.updated_at = OLD.updated_at
|
||||
BEGIN
|
||||
UPDATE memory_documents SET updated_at = datetime('now') WHERE id = NEW.id;
|
||||
UPDATE memory_documents SET updated_at = strftime('%Y-%m-%dT%H:%M:%fZ', 'now') WHERE id = NEW.id;
|
||||
END;
|
||||
|
||||
-- ==================== Workspace: Memory Chunks ====================
|
||||
@@ -225,7 +234,7 @@ CREATE TABLE IF NOT EXISTS memory_chunks (
|
||||
chunk_index INTEGER NOT NULL,
|
||||
content TEXT NOT NULL,
|
||||
embedding BLOB,
|
||||
created_at TEXT NOT NULL DEFAULT (datetime('now')),
|
||||
created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now')),
|
||||
UNIQUE (document_id, chunk_index)
|
||||
);
|
||||
|
||||
@@ -287,8 +296,8 @@ CREATE TABLE IF NOT EXISTS secrets (
|
||||
expires_at TEXT,
|
||||
last_used_at TEXT,
|
||||
usage_count INTEGER NOT NULL DEFAULT 0,
|
||||
created_at TEXT NOT NULL DEFAULT (datetime('now')),
|
||||
updated_at TEXT NOT NULL DEFAULT (datetime('now')),
|
||||
created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now')),
|
||||
updated_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now')),
|
||||
UNIQUE (user_id, name)
|
||||
);
|
||||
|
||||
@@ -309,8 +318,8 @@ CREATE TABLE IF NOT EXISTS wasm_tools (
|
||||
source_url TEXT,
|
||||
trust_level TEXT NOT NULL DEFAULT 'user',
|
||||
status TEXT NOT NULL DEFAULT 'active',
|
||||
created_at TEXT NOT NULL DEFAULT (datetime('now')),
|
||||
updated_at TEXT NOT NULL DEFAULT (datetime('now')),
|
||||
created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now')),
|
||||
updated_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now')),
|
||||
UNIQUE (user_id, name, version)
|
||||
);
|
||||
|
||||
@@ -331,8 +340,8 @@ CREATE TABLE IF NOT EXISTS wasm_channels (
|
||||
binary_hash BLOB NOT NULL,
|
||||
capabilities_json TEXT NOT NULL DEFAULT '{}',
|
||||
status TEXT NOT NULL DEFAULT 'active',
|
||||
created_at TEXT NOT NULL DEFAULT (datetime('now')),
|
||||
updated_at TEXT NOT NULL DEFAULT (datetime('now')),
|
||||
created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now')),
|
||||
updated_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now')),
|
||||
UNIQUE (user_id, name)
|
||||
);
|
||||
|
||||
@@ -350,8 +359,8 @@ CREATE TABLE IF NOT EXISTS tool_capabilities (
|
||||
max_response_body_bytes INTEGER NOT NULL DEFAULT 10485760,
|
||||
workspace_read_prefixes TEXT NOT NULL DEFAULT '[]',
|
||||
http_timeout_secs INTEGER NOT NULL DEFAULT 30,
|
||||
created_at TEXT NOT NULL DEFAULT (datetime('now')),
|
||||
updated_at TEXT NOT NULL DEFAULT (datetime('now')),
|
||||
created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now')),
|
||||
updated_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now')),
|
||||
UNIQUE (wasm_tool_id)
|
||||
);
|
||||
|
||||
@@ -364,7 +373,7 @@ CREATE TABLE IF NOT EXISTS leak_detection_patterns (
|
||||
severity TEXT NOT NULL DEFAULT 'high',
|
||||
action TEXT NOT NULL DEFAULT 'block',
|
||||
enabled INTEGER NOT NULL DEFAULT 1,
|
||||
created_at TEXT NOT NULL DEFAULT (datetime('now'))
|
||||
created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now'))
|
||||
);
|
||||
|
||||
-- ==================== Rate Limit State ====================
|
||||
@@ -373,9 +382,9 @@ CREATE TABLE IF NOT EXISTS tool_rate_limit_state (
|
||||
id TEXT PRIMARY KEY,
|
||||
wasm_tool_id TEXT NOT NULL REFERENCES wasm_tools(id) ON DELETE CASCADE,
|
||||
user_id TEXT NOT NULL,
|
||||
minute_window_start TEXT NOT NULL DEFAULT (datetime('now')),
|
||||
minute_window_start TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now')),
|
||||
minute_count INTEGER NOT NULL DEFAULT 0,
|
||||
hour_window_start TEXT NOT NULL DEFAULT (datetime('now')),
|
||||
hour_window_start TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now')),
|
||||
hour_count INTEGER NOT NULL DEFAULT 0,
|
||||
UNIQUE (wasm_tool_id, user_id)
|
||||
);
|
||||
@@ -391,7 +400,7 @@ CREATE TABLE IF NOT EXISTS secret_usage_log (
|
||||
target_path TEXT,
|
||||
success INTEGER NOT NULL,
|
||||
error_message TEXT,
|
||||
created_at TEXT NOT NULL DEFAULT (datetime('now'))
|
||||
created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now'))
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_secret_usage_user ON secret_usage_log(user_id);
|
||||
@@ -406,7 +415,7 @@ CREATE TABLE IF NOT EXISTS leak_detection_events (
|
||||
source TEXT NOT NULL,
|
||||
action_taken TEXT NOT NULL,
|
||||
context_preview TEXT,
|
||||
created_at TEXT NOT NULL DEFAULT (datetime('now'))
|
||||
created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now'))
|
||||
);
|
||||
|
||||
-- ==================== Tool Failures ====================
|
||||
@@ -416,8 +425,8 @@ CREATE TABLE IF NOT EXISTS tool_failures (
|
||||
tool_name TEXT NOT NULL UNIQUE,
|
||||
error_message TEXT,
|
||||
error_count INTEGER DEFAULT 1,
|
||||
first_failure TEXT DEFAULT (datetime('now')),
|
||||
last_failure TEXT DEFAULT (datetime('now')),
|
||||
first_failure TEXT DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now')),
|
||||
last_failure TEXT DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now')),
|
||||
last_build_result TEXT,
|
||||
repaired_at TEXT,
|
||||
repair_attempts INTEGER DEFAULT 0
|
||||
@@ -432,7 +441,7 @@ CREATE TABLE IF NOT EXISTS job_events (
|
||||
job_id TEXT NOT NULL REFERENCES agent_jobs(id),
|
||||
event_type TEXT NOT NULL,
|
||||
data TEXT NOT NULL,
|
||||
created_at TEXT NOT NULL DEFAULT (datetime('now'))
|
||||
created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now'))
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_job_events_job ON job_events(job_id, id);
|
||||
@@ -462,8 +471,8 @@ CREATE TABLE IF NOT EXISTS routines (
|
||||
next_fire_at TEXT,
|
||||
run_count INTEGER NOT NULL DEFAULT 0,
|
||||
consecutive_failures INTEGER NOT NULL DEFAULT 0,
|
||||
created_at TEXT NOT NULL DEFAULT (datetime('now')),
|
||||
updated_at TEXT NOT NULL DEFAULT (datetime('now')),
|
||||
created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now')),
|
||||
updated_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now')),
|
||||
UNIQUE (user_id, name)
|
||||
);
|
||||
|
||||
@@ -476,13 +485,13 @@ CREATE TABLE IF NOT EXISTS routine_runs (
|
||||
routine_id TEXT NOT NULL REFERENCES routines(id) ON DELETE CASCADE,
|
||||
trigger_type TEXT NOT NULL,
|
||||
trigger_detail TEXT,
|
||||
started_at TEXT NOT NULL DEFAULT (datetime('now')),
|
||||
started_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now')),
|
||||
completed_at TEXT,
|
||||
status TEXT NOT NULL DEFAULT 'running',
|
||||
result_summary TEXT,
|
||||
tokens_used INTEGER,
|
||||
job_id TEXT REFERENCES agent_jobs(id),
|
||||
created_at TEXT NOT NULL DEFAULT (datetime('now'))
|
||||
created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now'))
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_routine_runs_routine ON routine_runs(routine_id);
|
||||
@@ -493,7 +502,7 @@ CREATE TABLE IF NOT EXISTS settings (
|
||||
user_id TEXT NOT NULL,
|
||||
key TEXT NOT NULL,
|
||||
value TEXT NOT NULL,
|
||||
updated_at TEXT NOT NULL DEFAULT (datetime('now')),
|
||||
updated_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now')),
|
||||
PRIMARY KEY (user_id, key)
|
||||
);
|
||||
|
||||
@@ -549,24 +558,24 @@ CREATE INDEX IF NOT EXISTS idx_heartbeat_next_run ON heartbeat_state(next_run);
|
||||
|
||||
-- Pre-populate leak detection patterns (matches PostgreSQL V2 migration).
|
||||
INSERT OR IGNORE INTO leak_detection_patterns (id, name, pattern, severity, action, enabled, created_at) VALUES
|
||||
('550e8400-e29b-41d4-a716-446655440001', 'openai_api_key', 'sk-(?:proj-)?[a-zA-Z0-9]{20,}(?:T3BlbkFJ[a-zA-Z0-9_-]*)?', 'critical', 'block', 1, datetime('now')),
|
||||
('550e8400-e29b-41d4-a716-446655440002', 'anthropic_api_key', 'sk-ant-api[a-zA-Z0-9_-]{90,}', 'critical', 'block', 1, datetime('now')),
|
||||
('550e8400-e29b-41d4-a716-446655440003', 'aws_access_key', 'AKIA[0-9A-Z]{16}', 'critical', 'block', 1, datetime('now')),
|
||||
('550e8400-e29b-41d4-a716-446655440004', 'aws_secret_key', '(?<![A-Za-z0-9/+=])[A-Za-z0-9/+=]{40}(?![A-Za-z0-9/+=])', 'high', 'block', 1, datetime('now')),
|
||||
('550e8400-e29b-41d4-a716-446655440005', 'github_token', 'gh[pousr]_[A-Za-z0-9_]{36,}', 'critical', 'block', 1, datetime('now')),
|
||||
('550e8400-e29b-41d4-a716-446655440006', 'github_fine_grained_pat', 'github_pat_[a-zA-Z0-9]{22}_[a-zA-Z0-9]{59}', 'critical', 'block', 1, datetime('now')),
|
||||
('550e8400-e29b-41d4-a716-446655440007', 'stripe_api_key', 'sk_(?:live|test)_[a-zA-Z0-9]{24,}', 'critical', 'block', 1, datetime('now')),
|
||||
('550e8400-e29b-41d4-a716-446655440008', 'nearai_session', 'sess_[a-zA-Z0-9]{32,}', 'critical', 'block', 1, datetime('now')),
|
||||
('550e8400-e29b-41d4-a716-446655440009', 'bearer_token', 'Bearer\s+[a-zA-Z0-9_-]{20,}', 'high', 'redact', 1, datetime('now')),
|
||||
('550e8400-e29b-41d4-a716-44665544000a', 'pem_private_key', '-----BEGIN\s+(?:RSA\s+)?PRIVATE\s+KEY-----', 'critical', 'block', 1, datetime('now')),
|
||||
('550e8400-e29b-41d4-a716-44665544000b', 'ssh_private_key', '-----BEGIN\s+(?:OPENSSH|EC|DSA)\s+PRIVATE\s+KEY-----', 'critical', 'block', 1, datetime('now')),
|
||||
('550e8400-e29b-41d4-a716-44665544000c', 'google_api_key', 'AIza[0-9A-Za-z_-]{35}', 'high', 'block', 1, datetime('now')),
|
||||
('550e8400-e29b-41d4-a716-44665544000d', 'slack_token', 'xox[baprs]-[0-9a-zA-Z-]{10,}', 'high', 'block', 1, datetime('now')),
|
||||
('550e8400-e29b-41d4-a716-44665544000e', 'discord_token', '[MN][A-Za-z\d]{23,}\.[\w-]{6}\.[\w-]{27}', 'high', 'block', 1, datetime('now')),
|
||||
('550e8400-e29b-41d4-a716-44665544000f', 'twilio_api_key', 'SK[a-fA-F0-9]{32}', 'high', 'block', 1, datetime('now')),
|
||||
('550e8400-e29b-41d4-a716-446655440010', 'sendgrid_api_key', 'SG\.[a-zA-Z0-9_-]{22}\.[a-zA-Z0-9_-]{43}', 'high', 'block', 1, datetime('now')),
|
||||
('550e8400-e29b-41d4-a716-446655440011', 'mailchimp_api_key', '[a-f0-9]{32}-us[0-9]{1,2}', 'medium', 'block', 1, datetime('now')),
|
||||
('550e8400-e29b-41d4-a716-446655440012', 'high_entropy_hex', '(?<![a-fA-F0-9])[a-fA-F0-9]{64}(?![a-fA-F0-9])', 'medium', 'warn', 1, datetime('now'));
|
||||
('550e8400-e29b-41d4-a716-446655440001', 'openai_api_key', 'sk-(?:proj-)?[a-zA-Z0-9]{20,}(?:T3BlbkFJ[a-zA-Z0-9_-]*)?', 'critical', 'block', 1, strftime('%Y-%m-%dT%H:%M:%fZ', 'now')),
|
||||
('550e8400-e29b-41d4-a716-446655440002', 'anthropic_api_key', 'sk-ant-api[a-zA-Z0-9_-]{90,}', 'critical', 'block', 1, strftime('%Y-%m-%dT%H:%M:%fZ', 'now')),
|
||||
('550e8400-e29b-41d4-a716-446655440003', 'aws_access_key', 'AKIA[0-9A-Z]{16}', 'critical', 'block', 1, strftime('%Y-%m-%dT%H:%M:%fZ', 'now')),
|
||||
('550e8400-e29b-41d4-a716-446655440004', 'aws_secret_key', '(?<![A-Za-z0-9/+=])[A-Za-z0-9/+=]{40}(?![A-Za-z0-9/+=])', 'high', 'block', 1, strftime('%Y-%m-%dT%H:%M:%fZ', 'now')),
|
||||
('550e8400-e29b-41d4-a716-446655440005', 'github_token', 'gh[pousr]_[A-Za-z0-9_]{36,}', 'critical', 'block', 1, strftime('%Y-%m-%dT%H:%M:%fZ', 'now')),
|
||||
('550e8400-e29b-41d4-a716-446655440006', 'github_fine_grained_pat', 'github_pat_[a-zA-Z0-9]{22}_[a-zA-Z0-9]{59}', 'critical', 'block', 1, strftime('%Y-%m-%dT%H:%M:%fZ', 'now')),
|
||||
('550e8400-e29b-41d4-a716-446655440007', 'stripe_api_key', 'sk_(?:live|test)_[a-zA-Z0-9]{24,}', 'critical', 'block', 1, strftime('%Y-%m-%dT%H:%M:%fZ', 'now')),
|
||||
('550e8400-e29b-41d4-a716-446655440008', 'nearai_session', 'sess_[a-zA-Z0-9]{32,}', 'critical', 'block', 1, strftime('%Y-%m-%dT%H:%M:%fZ', 'now')),
|
||||
('550e8400-e29b-41d4-a716-446655440009', 'bearer_token', 'Bearer\s+[a-zA-Z0-9_-]{20,}', 'high', 'redact', 1, strftime('%Y-%m-%dT%H:%M:%fZ', 'now')),
|
||||
('550e8400-e29b-41d4-a716-44665544000a', 'pem_private_key', '-----BEGIN\s+(?:RSA\s+)?PRIVATE\s+KEY-----', 'critical', 'block', 1, strftime('%Y-%m-%dT%H:%M:%fZ', 'now')),
|
||||
('550e8400-e29b-41d4-a716-44665544000b', 'ssh_private_key', '-----BEGIN\s+(?:OPENSSH|EC|DSA)\s+PRIVATE\s+KEY-----', 'critical', 'block', 1, strftime('%Y-%m-%dT%H:%M:%fZ', 'now')),
|
||||
('550e8400-e29b-41d4-a716-44665544000c', 'google_api_key', 'AIza[0-9A-Za-z_-]{35}', 'high', 'block', 1, strftime('%Y-%m-%dT%H:%M:%fZ', 'now')),
|
||||
('550e8400-e29b-41d4-a716-44665544000d', 'slack_token', 'xox[baprs]-[0-9a-zA-Z-]{10,}', 'high', 'block', 1, strftime('%Y-%m-%dT%H:%M:%fZ', 'now')),
|
||||
('550e8400-e29b-41d4-a716-44665544000e', 'discord_token', '[MN][A-Za-z\d]{23,}\.[\w-]{6}\.[\w-]{27}', 'high', 'block', 1, strftime('%Y-%m-%dT%H:%M:%fZ', 'now')),
|
||||
('550e8400-e29b-41d4-a716-44665544000f', 'twilio_api_key', 'SK[a-fA-F0-9]{32}', 'high', 'block', 1, strftime('%Y-%m-%dT%H:%M:%fZ', 'now')),
|
||||
('550e8400-e29b-41d4-a716-446655440010', 'sendgrid_api_key', 'SG\.[a-zA-Z0-9_-]{22}\.[a-zA-Z0-9_-]{43}', 'high', 'block', 1, strftime('%Y-%m-%dT%H:%M:%fZ', 'now')),
|
||||
('550e8400-e29b-41d4-a716-446655440011', 'mailchimp_api_key', '[a-f0-9]{32}-us[0-9]{1,2}', 'medium', 'block', 1, strftime('%Y-%m-%dT%H:%M:%fZ', 'now')),
|
||||
('550e8400-e29b-41d4-a716-446655440012', 'high_entropy_hex', '(?<![a-fA-F0-9])[a-fA-F0-9]{64}(?![a-fA-F0-9])', 'medium', 'warn', 1, strftime('%Y-%m-%dT%H:%M:%fZ', 'now'));
|
||||
|
||||
"#;
|
||||
|
||||
@@ -604,7 +613,7 @@ CREATE TABLE IF NOT EXISTS memory_chunks_new (
|
||||
chunk_index INTEGER NOT NULL,
|
||||
content TEXT NOT NULL,
|
||||
embedding BLOB,
|
||||
created_at TEXT NOT NULL DEFAULT (datetime('now')),
|
||||
created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now')),
|
||||
UNIQUE (document_id, chunk_index)
|
||||
);
|
||||
|
||||
|
||||
+116
@@ -91,6 +91,64 @@ pub async fn connect_from_config(
|
||||
}
|
||||
}
|
||||
|
||||
/// Create a secrets store from database and secrets configuration.
|
||||
///
|
||||
/// This is the shared factory for CLI commands and other call sites that need
|
||||
/// a `SecretsStore` without going through the full `AppBuilder`. Mirrors the
|
||||
/// pattern of [`connect_from_config`] but returns a secrets-specific store.
|
||||
pub async fn create_secrets_store(
|
||||
config: &crate::config::DatabaseConfig,
|
||||
crypto: Arc<crate::secrets::SecretsCrypto>,
|
||||
) -> Result<Arc<dyn crate::secrets::SecretsStore + Send + Sync>, DatabaseError> {
|
||||
match config.backend {
|
||||
#[cfg(feature = "libsql")]
|
||||
crate::config::DatabaseBackend::LibSql => {
|
||||
use secrecy::ExposeSecret as _;
|
||||
|
||||
let default_path = crate::config::default_libsql_path();
|
||||
let db_path = config.libsql_path.as_deref().unwrap_or(&default_path);
|
||||
|
||||
let backend = if let Some(ref url) = config.libsql_url {
|
||||
let token = config.libsql_auth_token.as_ref().ok_or_else(|| {
|
||||
DatabaseError::Pool(
|
||||
"LIBSQL_AUTH_TOKEN required when LIBSQL_URL is set".to_string(),
|
||||
)
|
||||
})?;
|
||||
libsql::LibSqlBackend::new_remote_replica(db_path, url, token.expose_secret())
|
||||
.await
|
||||
.map_err(|e| DatabaseError::Pool(e.to_string()))?
|
||||
} else {
|
||||
libsql::LibSqlBackend::new_local(db_path)
|
||||
.await
|
||||
.map_err(|e| DatabaseError::Pool(e.to_string()))?
|
||||
};
|
||||
backend.run_migrations().await?;
|
||||
|
||||
Ok(Arc::new(crate::secrets::LibSqlSecretsStore::new(
|
||||
backend.shared_db(),
|
||||
crypto,
|
||||
)))
|
||||
}
|
||||
#[cfg(feature = "postgres")]
|
||||
_ => {
|
||||
let pg = postgres::PgBackend::new(config)
|
||||
.await
|
||||
.map_err(|e| DatabaseError::Pool(e.to_string()))?;
|
||||
pg.run_migrations().await?;
|
||||
|
||||
Ok(Arc::new(crate::secrets::PostgresSecretsStore::new(
|
||||
pg.pool(),
|
||||
crypto,
|
||||
)))
|
||||
}
|
||||
#[cfg(not(feature = "postgres"))]
|
||||
_ => Err(DatabaseError::Pool(
|
||||
"No database backend available for secrets. Enable 'postgres' or 'libsql' feature."
|
||||
.to_string(),
|
||||
)),
|
||||
}
|
||||
}
|
||||
|
||||
// ==================== Sub-traits ====================
|
||||
//
|
||||
// Each sub-trait groups related persistence methods. The `Database` supertrait
|
||||
@@ -125,6 +183,21 @@ pub trait ConversationStore: Send + Sync {
|
||||
channel: &str,
|
||||
limit: i64,
|
||||
) -> Result<Vec<ConversationSummary>, DatabaseError>;
|
||||
async fn list_conversations_all_channels(
|
||||
&self,
|
||||
user_id: &str,
|
||||
limit: i64,
|
||||
) -> Result<Vec<ConversationSummary>, DatabaseError>;
|
||||
async fn get_or_create_routine_conversation(
|
||||
&self,
|
||||
routine_id: Uuid,
|
||||
routine_name: &str,
|
||||
user_id: &str,
|
||||
) -> Result<Uuid, DatabaseError>;
|
||||
async fn get_or_create_heartbeat_conversation(
|
||||
&self,
|
||||
user_id: &str,
|
||||
) -> Result<Uuid, DatabaseError>;
|
||||
async fn get_or_create_assistant_conversation(
|
||||
&self,
|
||||
user_id: &str,
|
||||
@@ -420,3 +493,46 @@ pub trait Database:
|
||||
/// Run schema migrations for this backend.
|
||||
async fn run_migrations(&self) -> Result<(), DatabaseError>;
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
/// Regression test: `create_secrets_store` selects the correct backend at
|
||||
/// runtime based on `DatabaseConfig`, not at compile time. Previously the
|
||||
/// CLI duplicated this logic with compile-time `#[cfg]` gates that always
|
||||
/// chose postgres when both features were enabled (PR #209).
|
||||
#[cfg(feature = "libsql")]
|
||||
#[tokio::test]
|
||||
async fn test_create_secrets_store_libsql_backend() {
|
||||
use secrecy::SecretString;
|
||||
|
||||
let tmp = tempfile::tempdir().unwrap();
|
||||
let db_path = tmp.path().join("test.db");
|
||||
|
||||
let config = crate::config::DatabaseConfig {
|
||||
backend: crate::config::DatabaseBackend::LibSql,
|
||||
libsql_path: Some(db_path),
|
||||
libsql_url: None,
|
||||
libsql_auth_token: None,
|
||||
url: SecretString::from("unused://libsql".to_string()),
|
||||
pool_size: 1,
|
||||
ssl_mode: crate::config::SslMode::default(),
|
||||
};
|
||||
|
||||
let master_key = SecretString::from("a]".repeat(16));
|
||||
let crypto = Arc::new(crate::secrets::SecretsCrypto::new(master_key).unwrap());
|
||||
|
||||
let store = create_secrets_store(&config, crypto).await;
|
||||
assert!(
|
||||
store.is_ok(),
|
||||
"create_secrets_store should succeed for libsql backend"
|
||||
);
|
||||
|
||||
// Verify basic operation works
|
||||
let store = store.unwrap();
|
||||
let exists = store.exists("test_user", "nonexistent_secret").await;
|
||||
assert!(exists.is_ok());
|
||||
assert!(!exists.unwrap());
|
||||
}
|
||||
}
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user