* feat: structured fallback deliverables for failed/stuck jobs (#221)
When a job fails or gets stuck, build a FallbackDeliverable that captures
partial results, action statistics, cost, timing, and repair attempts.
This replaces opaque error strings with structured data users can act on.
- Add FallbackDeliverable, LastAction, ActionStats types in context/fallback.rs
- Store fallback in JobContext.metadata["fallback_deliverable"] on failure
- Surface fallback in job_status tool output and SSE job_result events
- Update mark_failed() and mark_stuck() in worker to build fallback
- 8 unit tests covering zero/mixed actions, truncation, timing, serialization
Co-Authored-By: Claude Opus 4.6 <[email protected]>
* fix: address review comments on fallback deliverables
- Fix doc comment: "200 chars" -> "200 bytes (UTF-8 safe)" since
truncate_str operates on byte length, not character count.
- Add code comment documenting that SSE fallback_deliverable is
currently always None (forward-compatible infrastructure).
Co-Authored-By: Claude Opus 4.6 <[email protected]>
* refactor: take Option<&FallbackDeliverable> instead of &Option<…>
Addresses Gemini review feedback: idiomatic Rust prefers
Option<&T> over &Option<T> for borrowed optional values.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
* fix: guard against non-object metadata and add fallback test
- store_fallback_in_metadata now resets metadata to {} when it's any
non-object type (string, array, number), not just null. Prevents
panic on index assignment.
- Add test_job_status_includes_fallback_deliverable to verify the
fallback field is surfaced in job_status tool output.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
* fix: use sanitized output in fallback preview + add integration tests
Security fix: FallbackDeliverable::build() now uses output_sanitized
instead of output_raw, preventing secrets/PII from leaking through
the job_status tool and SSE job_result events.
Also adds:
- test_fallback_uses_sanitized_output: proves raw secrets don't leak
- test_store_fallback_in_metadata_roundtrip: full serialize/deserialize
- test_store_fallback_handles_non_object_metadata: edge case coverage
- test_store_fallback_none_is_noop: None input is safe
Addresses serrrfirat review feedback on PR #236.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
* fix: harden fallback deliverables against review findings
- Truncate failure_reason to 1000 bytes to prevent metadata bloat
- Add tracing::warn on fallback serialization failure (was silently discarded)
- Fix module/struct docs to cover stuck jobs, remove stale SSE claim
- Fix job.rs test to use real FallbackDeliverable field names
- Add tests for failure_reason truncation and completed_at=None elapsed time
- Fix pre-existing clippy warning in settings.rs (field_reassign_with_default)
Co-Authored-By: Claude Opus 4.6 <[email protected]>
* fix: address Copilot review findings on fallback deliverables
- Fix output_raw/output_sanitized field swap in ActionRecord::succeed()
so sanitized data actually goes into the sanitized field (security)
- Return None instead of empty Memory when get_memory fails in
build_fallback, with tracing::warn for observability
- Replace manual elapsed calculation with ctx.elapsed() which already
clamps negative durations
Co-Authored-By: Claude Opus 4.6 <[email protected]>
* fix: resolve rebase conflicts and update tests for parameter swap
- Add fallback field to SseEvent::JobResult in job_monitor
- Fix type annotation in fallback deliverable test
- Update test_action_record_succeed_sets_fields for new parameter order
- Use create_job_for_user in test (API changed on main)
Co-Authored-By: Claude Opus 4.6 <[email protected]>
* chore: trigger CI re-check after rebase
* fix: fall back to error message for failed action output_preview
When the last action is a failed tool call, output_sanitized is None,
leaving output_preview empty. Now falls back to the action's error
message so users see what went wrong.
[skip-regression-check]
* ci: add safety comments to test code for no-panics check
The CI no-panics grep check cannot distinguish test code inside
src/ files from production code. Add // safety: test annotations
to .unwrap(), .expect(), and assert!() calls in #[cfg(test)] modules.
* fix: clarify succeed() doc and avoid clone in output_preview
- Fix doc comment: output_raw is stored as pretty-printed JSON string,
not a raw JSON value
- Borrow string slice directly in fallback preview to avoid cloning
potentially large sanitized outputs before truncation
* refactor: reuse floor_char_boundary in truncate_str
Replace hand-rolled UTF-8 boundary logic with existing
crate::util::floor_char_boundary to reduce duplication.
* fix: rename SSE fallback field to fallback_deliverable for consistency
The SSE JobResult field was named `fallback` while everywhere else
(metadata key, job_status tool) uses `fallback_deliverable`. Align
the SSE wire format to avoid forcing clients to handle two names.
---------
Co-authored-by: Claude Opus 4.6 <[email protected]>
* feat: LRU embedding cache for workspace search (#165)
Add CachedEmbeddingProvider that wraps any EmbeddingProvider with an
in-memory LRU cache keyed by SHA-256(model_name + text). This avoids
redundant HTTP calls when the same text is embedded multiple times
(common during reindexing and repeated searches).
- Cache uses HashMap + last_accessed tracking with manual LRU eviction
(same pattern as llm::response_cache::CachedProvider)
- Lock is never held during HTTP calls to prevent blocking
- embed_batch() partitions into hits/misses and only fetches misses
- Default 10,000 entries (~58 MB for 1536-dim vectors)
- Configurable via EMBEDDING_CACHE_SIZE env var
- Workspace.with_embeddings() auto-wraps; with_embeddings_uncached()
available for tests
Co-Authored-By: Claude Opus 4.6 <[email protected]>
* fix: address review comments on embedding cache
- Validate embed_batch return count matches expected miss count
- Replace unwrap_or_default() with proper error propagation
- Fix batch eviction: run final eviction pass after insert to enforce cap
- Fix test: use different-length inputs to verify ordering correctness
- Reject EMBEDDING_CACHE_SIZE=0 in config validation (minimum is 1)
Co-Authored-By: Claude Opus 4.6 <[email protected]>
* fix: replace .expect() with proper error handling in embed_batch
The all-cache-hits early-return path used .expect("all cache hits") which
violates the project convention of no .unwrap()/.expect() in production
code. Replaced with the same ok_or_else pattern used in the normal path.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
* fix: clarify memory sizing docs and use saturating_add for eviction
- Update memory comments in embedding_cache.rs, config/embeddings.rs,
and workspace/mod.rs to note the ~58 MB figure is payload-only
(actual memory is higher due to HashMap/key/allocation overhead)
- Use saturating_add(1) instead of + 1 for eviction threshold to
prevent overflow if max_entries is usize::MAX
Co-Authored-By: Claude Opus 4.6 <[email protected]>
* fix: address Copilot review on embedding cache
- Avoid double-clone per miss in embed_batch: move embedding into
results, clone only for the cache entry
- Evict per-insert instead of after all inserts to keep peak memory
bounded during large batches
- Clamp max_entries to at least 1 in constructor to prevent unexpected
eviction behavior when set to 0 via the public API
Co-Authored-By: Claude Opus 4.6 <[email protected]>
* fix: reduce embedding_cache module visibility to private
Types are already re-exported via `pub use`, so the module itself
doesn't need to be public. Reduces unnecessary API surface.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
* fix: address serrrfirat review feedback on embedding cache
- Add TODO comment for O(n) LRU eviction scalability
- Add thundering herd note at lock release in embed()
- Warn when cache max_entries exceeds 100k
- Use with_embeddings_uncached() in integration test
- Add tests: error_does_not_pollute_cache, embed_batch_empty_input
- Update README with cache-aware with_embeddings() docs
Co-Authored-By: Claude Opus 4.6 <[email protected]>
* fix: prevent u32 wrapping in FailThenSucceedMock failure counter
fetch_sub(1) wraps to u32::MAX when called past zero, silently
breaking the mock for 3+ calls. Switch to load-then-store to avoid
the wrapping bug in both embed() and embed_batch().
Co-Authored-By: Claude Opus 4.6 <[email protected]>
* fix: address Copilot and serrrfirat review findings on embedding cache
- Switch tokio::sync::Mutex to std::sync::Mutex (lock never held across
.await — cheaper synchronous lock)
- Extract DEFAULT_EMBEDDING_CACHE_SIZE constant to avoid 10_000 duplication
between EmbeddingCacheConfig and EmbeddingsConfig
Co-Authored-By: Claude Opus 4.6 <[email protected]>
* test: add all-misses batch test for embedding cache
Adds embed_batch_all_misses test covering the case where every text in a
batch is a cache miss — fulfilling the commitment from serrrfirat's review.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
* chore: trigger CI re-check after rebase
* fix: use raw [u8;32] cache keys and pre-allocate HashMap capacity
Address Copilot review findings:
- cache_key() now returns [u8; 32] instead of hex String, avoiding a
64-byte allocation per lookup
- HashMap::with_capacity(max_entries) avoids incremental reallocation
- Fix pre-existing staging compilation error in cli/routines.rs
(missing max_tool_rounds/use_tools fields)
[skip-regression-check]
* fix: make cache accessors sync and update doc for [u8;32] keys
Address Copilot review:
- len(), is_empty(), clear() are now sync since they only take a
std::sync::Mutex lock with no .await points
- Update cache_size doc comment to reflect [u8;32] keys instead of
String keys
[skip-regression-check]
* fix: remove clone_on_copy for [u8; 32] cache keys
[skip-regression-check]
* ci: add safety comments to test code for no-panics check
The CI no-panics grep check cannot distinguish test code inside
src/ files from production code. Add // safety: test annotations
to .unwrap(), .expect(), and assert!() calls in #[cfg(test)] modules.
* fix: correct cache doc and demote hit/miss logs to trace
- Fix misleading "String keys" in memory comment (cache uses [u8; 32])
- Demote per-request hit/miss logs from debug to trace to reduce noise
on hot paths (batch summary stays at trace too)
* docs: add missing Arc import in workspace README example
* perf: batch eviction in embed_batch to avoid O(n×m) cost
Replace per-insert evict_lru call with a single evict_k_oldest pass
that computes eviction count upfront and removes the k oldest entries
in one O(n) scan. Avoids O(n×m) HashMap iterations while holding the
mutex during batch inserts.
* fix: cap batch cache inserts at max_entries and use O(n) selection
- evict_k_oldest now uses select_nth_unstable_by_key for O(n) average
partial selection instead of O(n log n) full sort
- embed_batch caps cached entries at max_entries when misses exceed
capacity, preventing the cache from growing unbounded
- Added test: batch_exceeding_capacity_respects_max_entries
* fix: flatten test assert for fmt compatibility
Shorten assert message to fit single line so cargo fmt doesn't
split the safety annotation onto a separate line.
* fix: address review feedback and improve embedding cache (takeover #235)
- Fix merge conflict: add missing allow_always field in PendingApproval
- Thread EmbeddingCacheConfig through CLI memory commands so they respect
EMBEDDING_CACHE_SIZE instead of silently using default (fixes#235 review)
- Cap HashMap pre-allocation at min(max_entries, 1024) to avoid upfront
memory waste at large cache sizes
- Fix FailThenSucceedMock race: replace load+store with atomic fetch_update
- Remove noisy '// safety: test' comments (40+ lines of diff noise)
- Fix collapsed lines from comment removal
- Simplify redundant Ok(...collect()?) to just collect()
Co-Authored-By: ztsalexey <[email protected]>
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* fix(embedding-cache): skip eviction on concurrent duplicate insert
When the lock is released for the HTTP call, another caller may insert
the same key. Re-check under lock and just update the existing entry
without evicting, avoiding unnecessary cache churn under concurrency.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
---------
Co-authored-by: ztsalexey <[email protected]>
Co-authored-by: Claude Opus 4.6 <[email protected]>
Co-authored-by: ztsalexey <[email protected]>
* feat: receive relay events via webhook callbacks instead of SSE
Replace the SSE pull model with push-based webhook callbacks from
channel-relay. Eliminates the reconnect loop, stream token auth,
and SSE parser — events arrive via HTTP POST to /relay/events.
- Add webhook handler with HMAC signature verification
- Simplify RelayChannel to use mpsc from webhook handler
- Remove SSE connect/reconnect/parse logic from RelayClient
- Add register_callback() to RelayClient for callback URL registration
- Update activation flow to create event channel and register callback
- Wire relay webhook endpoint into web gateway
* fix: address review feedback on webhook callback PR
- Return 503 when relay event channel is full/closed (enables retry)
- Reject malformed timestamps with 400 instead of proceeding
- Allow relay activation without settings store (no-store/ephemeral mode)
- Check installed_relay_extensions set in is_relay_channel for no-db mode
- Fix staging test constructors for new RelayChannel signature
* security: adapt relay client to new channel-relay auth model
Adapts the relay integration to the hardened channel-relay security model:
- Switch from X-API-Key header to Authorization: Bearer sk-agent-*
for all relay API calls (chat-api token verification)
- Remove register_callback() — PUT /callbacks endpoint removed
- Remove event_callback_url from initiate_oauth() — parameter removed
- Make signing_secret a required field in RelayConfig (new env var:
CHANNEL_RELAY_SIGNING_SECRET)
- Update integration tests for Bearer auth and removed endpoints
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* security: use server-side approval tokens, remove caller-supplied routing
- Approval flow now calls POST /approvals to register server-side
record, then embeds only the opaque approval_token in button value
- Remove instance_id parameter from proxy_provider() — channel-relay
no longer accepts it (uses verified identity)
- Remove instance_id and user_id from initiate_oauth() — channel-relay
derives them from the Bearer token
- Add create_approval() to RelayClient
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* fix: pass webhook_url during OAuth so callback_url is set on connection
The channel-relay OAuth flow now accepts webhook_url to set the
callback_url during connection creation. IronClaw computes its webhook
URL from callback_base + webhook_path and passes it during initiate_oauth.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* security: remove webhook_url from OAuth initiation
Channel-relay now derives the callback URL from chat-api's instance_url.
IronClaw no longer supplies webhook_url during OAuth — the relay is the
authority on where events get delivered.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* chore: cargo fmt
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* security: remove all URL params from OAuth initiation
IronClaw no longer supplies any URLs to channel-relay. The relay
derives all URLs from the trusted instance_url in chat-api.
initiate_oauth() takes no parameters.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* fix: restore CSRF nonce for OAuth callback validation
Re-add nonce generation and secret storage in auth_channel_relay.
The nonce is passed to channel-relay as state_nonce param (not a URL).
Channel-relay embeds it in the signed state and appends it to the
redirect URL so IronClaw's callback handler can validate and activate.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* security: per-instance callback signing secrets
relay_signing_secret() now prefers OPENCLAW_GATEWAY_TOKEN (per-instance)
over the shared CHANNEL_RELAY_SIGNING_SECRET. A compromised instance
can no longer forge callbacks to other instances on the same relay.
CHANNEL_RELAY_SIGNING_SECRET is now optional in RelayConfig.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* security: clean per-instance callback secrets, no shared secrets, no fallbacks
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* fix: pass team_id to get_signing_secret for workspace-scoped lookup
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* security: remove sender_id from create_approval — relay derives it
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* fix: remove stale relay sender_id validation
* fix: harden relay webhook activation lifecycle
---------
Co-authored-by: Pierre <[email protected]>
Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>
The HTTP tool returned `ApprovalRequirement::Always` for requests with
credentials, but `Always` is hardcoded to ignore the session auto-approve
set. This meant users who clicked "always" were re-prompted on every
subsequent HTTP call — the UI offered "always" but the backend ignored it.
Two fixes:
1. HTTP credentialed requests now return `UnlessAutoApproved` instead of
`Always`, so the session auto-approve set is respected.
2. `StatusUpdate::ApprovalNeeded` now carries `allow_always: bool`. All
channel UIs (Telegram, Slack, Signal, REPL, Web) conditionally hide
the "always" option when a tool truly requires per-invocation approval
(`ApprovalRequirement::Always`, e.g. destructive shell commands).
Also boxes `PendingApproval` in `AgenticLoopResult::NeedApproval` to fix
a pre-existing clippy `large_enum_variant` warning.
Regression tests included (test_credentialed_requests_respect_auto_approve,
test_allow_always_matches_approval_requirement) but CI heuristic cannot
detect them in cross-fork PR diffs.
[skip-regression-check]
Co-authored-by: Tyler <[email protected]>
* fix(feishu): parse flat token response from tenant_access_token API
The Feishu /auth/v3/tenant_access_token/internal endpoint returns a flat
JSON response with tenant_access_token and expire at the top level, not
nested under a "data" field. The previous code used FeishuApiResponse<T>
which expects a "data" wrapper, causing all token exchanges to fail with
"Token response missing data" despite receiving a valid HTTP 200 response.
- Replace TenantAccessTokenData with TenantAccessTokenResponse that includes
code/msg/tenant_access_token/expire at the top level
- Deserialize token response directly instead of via FeishuApiResponse<T> wrapper
- Add empty-token guard to catch malformed responses
- No changes to FeishuApiResponse<T> or other API call paths
Fixes#1391
* fix(feishu): address review feedback on token response parsing
- Remove #[serde(default)] from tenant_access_token and expire fields
so deserialization fails explicitly when critical fields are missing
- Add expire > 0 validation guard to prevent refresh loops or overflow
- Use saturating_add/saturating_mul for expiry calculation
- Add 5 regression tests for TenantAccessTokenResponse deserialization
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
---------
Co-authored-by: reidliu <[email protected]>
Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>
* fix: skip NEAR AI session check when backend is not nearai
When a user configures a non-NEAR AI backend (e.g. Anthropic), the
doctor command was incorrectly failing with "session file not found"
even though no NEAR AI session is needed. The check now skips with a
descriptive message when LLM_BACKEND is not nearai/near_ai/near.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
* fix(ci): avoid holding sync MutexGuard across await in doctor test
Convert check_nearai_session_skips_for_non_nearai_backend from
#[tokio::test] to #[test] with block_on, matching the pattern used by
all other ENV_MUTEX tests. Fixes clippy::await_holding_lock error.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
---------
Co-authored-by: Kristian Glass <[email protected]>
Co-authored-by: Claude Sonnet 4.6 <[email protected]>
Bump registry version to pass check-version-bumps.sh after
channels-src/telegram/ changes.
Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>
* fix: navigate telegram E2E tests to channels subtab
wasm_channel extensions (like telegram) are now rendered in the
Settings → Channels subtab, not the Extensions subtab. Update
test_telegram_hot_activation to navigate there and use the correct
card selector. Also mock /api/gateway/status which loadChannelsStatus
fetches.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* fix: select telegram card by name, not first card in channels subtab
Built-in channel cards (Web Gateway, HTTP, etc.) render first in the
channels subtab content, so .first matches them instead of the
telegram extension card. Select by has_text="Telegram" to target
the correct card.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* refactor: make gateway_status_handler parameterizable in mock helper
Address review feedback: extract default gateway status handler and
accept an optional gateway_status_handler kwarg in mock_extension_lists
for test flexibility.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>
- Add `builder: None` to AgentDeps initializer in e2e_telegram_message_routing
test (field added in #712 but test not updated)
- Update go_to_extensions() in test_telegram_hot_activation to navigate via
settings tab -> extensions subtab (extensions tab was moved to settings)
Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>
* feat(self-repair): wire stuck_threshold, store, and builder (#647)
Wire the previously dead-code fields in DefaultSelfRepair:
- stuck_threshold: detect_stuck_jobs() now filters by duration, only
reporting jobs stuck longer than the configured threshold
- with_store(): wired in agent_loop.rs from AgentDeps.store for
tool failure tracking via Database trait
- with_builder(): wired from register_builder_tool() return value
through AppComponents and AgentDeps for automatic tool rebuilding
- tools: passed alongside builder for hot-reload logging
Remove all #[allow(dead_code)] annotations. Add regression tests for
threshold-based filtering (both above and below threshold).
Co-Authored-By: Claude Opus 4.6 <[email protected]>
* fix: add missing `builder` field to AgentDeps in gateway workflow harness
After rebase onto staging, AgentDeps gained a `builder` field for
self-repair tool rebuilding. The gateway workflow test harness was
missing this field, causing CI compilation failure.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
* ci: retrigger CI
* fix: force CI refresh after path_routing_tests dedup
* test: add E2E test for stuck job repair and tool rebuild cycle
Tests the full self-repair flow requested in review:
1. Job transitions Pending -> InProgress -> Stuck
2. detect_stuck_jobs() finds it (zero threshold)
3. repair_stuck_job() recovers it back to InProgress
4. A broken tool is repaired via MockBuilder
5. Verify builder was invoked and repair succeeded
Uses a MockBuilder (impl SoftwareBuilder) that returns successful
BuildResult without requiring an LLM or filesystem. Uses libsql
test database for the store (increment_repair_attempts, mark_tool_repaired).
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* fix(self-repair): measure stuck_duration from Stuck transition, not started_at
- Use ctx.transitions to find the most recent Stuck transition timestamp
instead of ctx.started_at (which reflects job start, not stuck time)
- Fix StuckJob.last_activity to use stuck transition timestamp
- Remove misleading "hot-reloaded into registry" log
- Remove stray "// ci fix" comment in memory.rs
- Add regression test: backdated started_at must not inflate stuck_duration
Co-Authored-By: Claude Opus 4.6 <[email protected]>
* ci: re-trigger CI with latest changes
Co-Authored-By: Claude Opus 4.6 <[email protected]>
* fix: add type annotation to Ok(()) in test to resolve E0282
Co-Authored-By: Claude Opus 4.6 <[email protected]>
---------
Co-authored-by: Claude Opus 4.6 <[email protected]>
* feat(testing): add FaultInjector framework for StubLlm (#1220)
Adds a configurable fault injection framework for testing retry, failover,
and circuit breaker behavior. The FaultInjector attaches to StubLlm and
provides per-call control over failure type, timing, and sequencing.
Components:
- FaultType: maps to LlmError variants (RequestFailed, RateLimited,
AuthFailed, InvalidResponse, IoError, ContextLengthExceeded, SessionExpired)
- FaultAction: Succeed, Fail(FaultType), Delay(Duration)
- FaultMode: SequenceOnce (play then succeed), SequenceLoop (repeat forever),
Random (seeded xorshift64 PRNG for reproducibility)
- FaultInjector: thread-safe (AtomicU32 counter + Mutex RNG)
Integration:
- StubLlm gains optional fault_injector field via with_fault_injector()
- When set, takes precedence over should_fail/error_kind
- Backward compatible: existing StubLlm usage unchanged
Closes#1220
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* refactor(testing): address review feedback on FaultInjector
- Remove redundant .abs() in random fault comparison
- Extract check_faults() helper to DRY up StubLlm methods
- Guard xorshift seed=0 (fixed point) by mapping to 1
- Add StubLlm integration test (stub_llm_fault_injector_sequence)
- Remove dead seed field from FaultMode::Random
- Move pub mod fault_injection to top of mod.rs
- Add Debug impl for FaultInjector
- Add empty_sequence_always_succeeds test
- Add random_seed_zero_does_not_always_fail test
* fix(testing): address #1233 review -- seed-0 bug, reset(), Debug derive
- Store seed in FaultMode::Random so reset() can re-init the RNG
- Add reset() method for test reproducibility (re-seeds RNG, zeros counter)
- Strengthen seed=0 regression test to 100 iterations with stricter assertion
- Add reset_restores_random_rng_from_stored_seed test
- Debug impl and empty_sequence test were already present from prior commit
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* ci: re-trigger CI with latest changes
Co-Authored-By: Claude Opus 4.6 <[email protected]>
* ci: trigger new run with skip-regression-check label
Co-Authored-By: Claude Opus 4.6 <[email protected]>
* fix(testing): address PR #1233 review -- error_rate validation and edge cases
- Validate error_rate is in 0.0..=1.0 and not NaN (panics on invalid input)
- Fix error_rate==1.0 edge case: use <= instead of < so 1.0 always fails
- Add regression tests for error_rate validation (NaN, negative, >1.0)
- Add tests for error_rate boundary values (0.0 never fails, 1.0 always fails)
- Add delay action test using tokio::time::pause() for deterministic timing
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>
* feat(telegram): support auto split large message
* fix(telegram): strengthen split_message test assertion
Replace word-by-word contains check with assert_eq! on rejoined chunks,
ensuring split_message preserves content exactly.
send_response is still used (lines 745, 753) so it is intentionally kept.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* fix(telegram): add missing split_message tests and document limitations
- Add test for sentence-boundary splitting
- Add test for hard-cut on pathological input (no spaces)
- Add test for multi-byte character safety (emoji)
- Document CJK sentence punctuation limitation
- Document trim behavior at chunk boundaries
Co-Authored-By: Claude Opus 4.6 <[email protected]>
* ci: re-trigger CI with latest changes
Co-Authored-By: Claude Opus 4.6 <[email protected]>
---------
Co-authored-by: Hans <[email protected]>
Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>
* fix: remove debug_assert guards that panic on valid error paths (#1312)
Two debug_assert! calls added in #1312 fire on expected runtime error
paths (not programmer bugs), turning graceful error returns into panics
in debug/test builds:
- state.rs: Completed→Cancelled is a user-facing error handled by
transition_to() returning Err — not a bug
- execute.rs: empty tool_name from malformed LLM output is handled by
ToolError::NotFound — not a bug
Removes both asserts; keeps the circuit-breaker assert (genuinely guards
a caller invariant).
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* fix: tighten empty tool name test to assert ToolError::NotFound variant
Address review feedback: assert the specific error variant instead of
just is_err() so the regression test actually enforces the expected
error path.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* style: cargo fmt
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>
* feat(gateway): full settings page polish with all tiers
- Backend: add ActiveConfigSnapshot to expose resolved LLM backend,
model, and enabled channels via /api/gateway/status
- Add missing Agent settings (daily cost cap, actions/hour, local tools)
- Add Sandbox, Routines, Safety, Skills, and Search setting groups
- Settings import/export (JSON download + file upload)
- Active env defaults shown as placeholders in Inference settings
- Styled confirmation modals replace window.confirm() for remove actions
- Global restart banner persists across settings subtab switches
- Client-side validation with min/max constraints on number inputs
- Accessibility: aria-label on inputs, role=status on save indicators
- Settings search filters rows across current subtab
- Smooth CSS transitions for conditional field visibility (showWhen)
- Tunnel settings in Channels subtab
- Mobile responsive settings layout at 768px breakpoint
- i18n keys for toolbar, search, and import/export in en + zh-CN
Co-Authored-By: Claude Opus 4.6 <[email protected]>
* feat(gateway): polish settings page and remove registered tools debug section
Remove the "Registered Tools" table from the extensions tab (debug info
not useful to end users), clean up associated CSS/i18n/JS. Additional
settings page UI polish: extension card state styling, layout refinements.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* fix(gateway): address PR review feedback [skip-regression-check]
- Use refreshCurrentSettingsTab() in SSE event handlers to reduce duplication
- Remove unused formatGroupName/formatSettingLabel helpers
- Use i18n keys for MCP Configure/Reconfigure buttons
- Add data-i18n-placeholder to settings search input
- Remove data-i18n from confirm modal button (set dynamically by showConfirmModal)
- Fix cargo fmt in main.rs
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* fix(e2e): update tests for unified settings tab layout [skip-regression-check]
- Update TABS list: replace extensions/skills with settings
- Add settings_subtab/settings_subpanel selectors to helpers
- Update test_connection, test_skills, test_extensions, test_wasm_lifecycle
to navigate via Settings > subtab instead of top-level tabs
- Move MCP card tests to use go_to_mcp() helper (MCP is now a separate subtab)
- Remove tools table tests and mock_ext_apis tools= parameter
- Fix CSP violation: replace inline onclick on confirm modal cancel button
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* fix(gateway): address second round of PR review feedback [skip-regression-check]
- Use I18n.t() for MCP empty state, export/import toasts, confirm modal
- Fix CLI channel card using wrong channel key ('repl' -> 'cli')
- Fix settings search counting hidden rows as visible
- Add aria-label i18n for settings search input
- Add common.loadFailed i18n key (en + zh-CN)
- Update E2E tests: WASM channel tests use Channels subtab,
remove tests use custom confirm modal instead of window.confirm
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* fix(e2e): fix WASM channel card selector and skills remove confirm [skip-regression-check]
- WASM channel tests: filter by display name to avoid matching built-in
channel cards in the Channels subtab
- Skills remove test: click confirm modal button instead of using
window.confirm (skill removal now uses custom confirm modal)
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* fix(gateway): address third round of PR review feedback [skip-regression-check]
- approval_needed SSE: refresh any active settings subtab, not just
Extensions — approvals can surface from Channels/MCP setup flows too
- renderCardsSkeleton: remove nested .extensions-list wrapper that
caused skeleton cards to render constrained inside grid cells
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* fix(e2e): fix auth_completed reload test race condition [skip-regression-check]
Use expect_response to deterministically wait for the /api/extensions
reload triggered by handleAuthCompleted → refreshCurrentSettingsTab,
instead of a fixed 600ms sleep that was too short under CI load.
Also remove stale /api/extensions/tools route handler.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* fix(e2e): debug auth_completed reload test with function counter [skip-regression-check]
Inject a counter wrapper around refreshCurrentSettingsTab to verify it's
actually called, and wait for the async fetch to complete before
asserting the reload count.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* feat(gateway): localize all settings labels, descriptions, and channel cards [skip-regression-check]
Move 120+ hardcoded strings in settings definitions (INFERENCE_SETTINGS,
AGENT_SETTINGS, NETWORKING_SETTINGS) and channel card labels to i18n
keys. Render functions now resolve labels via I18n.t() so the settings
page translates when switching locales.
Covers: group titles, setting labels/descriptions, built-in channel
names/descriptions, and the "No settings found" empty state.
Both en.js and zh-CN.js updated with all new cfg.* and channels.* keys.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* fix(gateway): localize remaining hardcoded UI strings [skip-regression-check]
- Fix export error toast using wrong i18n key (importFailed → exportFailed)
- Replace "Failed to load settings:" with I18n.t('common.loadFailed')
- Localize renderBuiltinChannelCard: "Built-in", "Active", "Inactive"
- Localize settings placeholders: "env: ", "env default", "use env default"
- Localize "✓ Saved" indicator
- Add new i18n keys to both en.js and zh-CN.js
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* fix(gateway): confirm modal a11y, Esc/click-outside, search guard [skip-regression-check]
- Add role="dialog", aria-modal="true", aria-labelledby to confirm modal
- Focus confirm button when modal opens
- Close modal on Escape key or overlay click
- Skip settings search on non-settings panels (Extensions/MCP/Skills/Channels)
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* fix(gateway): boolean tri-state, search reset on subtab switch, stale model suggestions [skip-regression-check]
Address PR review feedback:
- Boolean settings now use a tri-state select (env default / On / Off)
instead of a checkbox, matching the pattern used by other select settings
and allowing users to revert to the env default
- Clear search input when switching settings subtabs so stale filters
don't carry over to the new panel
- Always assign model suggestions (even empty array) so stale IDs from a
previous successful /v1/models fetch don't persist when the endpoint
later returns empty
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* fix(gateway): auth_completed handler, bedrock_cross_region select, integer-only number inputs [skip-regression-check]
Address PR review feedback:
- auth_completed SSE listener now delegates to handleAuthCompleted(data)
instead of inlining logic with a bare closeConfigureModal() call, so
only the matching extension's modal is dismissed
- bedrock_cross_region changed from free text to select with the four
valid values (us/eu/apac/global), matching backend validation
- Number settings now use step=1 and parseInt() instead of parseFloat(),
preventing fractional values that the backend (u32/u64) would reject
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
---------
Co-authored-by: Claude Opus 4.6 <[email protected]>
* fix: full_job routine runs stay running until linked job completion (#1317)
Previously, execute_full_job() returned RunStatus::Ok immediately after
dispatching the job, causing routine runs to be marked as completed before
the linked worker job had actually finished. This meant failure notifications
were never sent and max_concurrent guardrails stopped applying once the run
was prematurely finalized.
Changes:
- execute_full_job() now returns RunStatus::Running instead of Ok
- execute_routine() skips finalization for Running status (leaves run open)
- New sync_dispatched_runs() polls on each cron tick, checks linked job
state, and finalizes runs when jobs reach terminal states
- New list_dispatched_routine_runs() DB method on both backends
- Deferred notifications are sent when the run is actually finalized
- consecutive_failures is preserved (not reset) while outcome is unknown
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* fix: address PR review feedback (watcher predicate, running_count safety)
- FullJobWatcher: use is_parallel_blocking() instead of is_active() so
the watcher exits when a job reaches Completed (not terminal but
finished executing). Fixes infinite-poll for routine jobs.
- Remove running_count decrement from sync_dispatched_runs() — in normal
flow execute_routine() handles it; sync only runs for crash recovery
where the counter is already 0.
- Update PR description to match actual FullJobWatcher behavior.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* fix: sync only at startup to prevent double-completion race
- Move sync_dispatched_runs() out of cron loop into startup-only path.
During normal operation FullJobWatcher handles finalization inline;
running sync on every tick would race with the watcher.
- Update complete_dispatched_run() to properly advance runtime fields
(last_run_at, next_fire_at, run_count) for crash recovery — in that
scenario execute_routine() never reached its runtime update.
- Fix stale doc comment on complete_dispatched_run().
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* fix: use boot_time filter for safe periodic sync of orphaned runs
- Add boot_time field to RoutineEngine, set to Utc::now() at creation.
- sync_dispatched_runs() now filters runs by started_at < boot_time,
so it only processes orphans from a previous process — never races
with FullJobWatcher instances from the current process.
- Move sync back into the cron loop (safe with boot_time filter) and
run it BEFORE check_cron_triggers to avoid picking up freshly
dispatched runs.
- Fix doc comments to match actual behavior.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>
* fix: add FullJobWatcher to track full_job lifecycle for concurrency (#1318)
full_job routines previously bypassed max_concurrent and global concurrency
limits because execute_full_job() returned RunStatus::Ok immediately after
dispatch. This meant running_count was decremented and the routine_run row
was finalized before the actual job completed.
Introduce FullJobWatcher struct that polls store.get_job() every 5s until
the linked job reaches a non-active state, then maps the final JobState to
RunStatus. execute_full_job now creates and awaits the watcher, keeping both
the DB-level running row and the in-memory running_count elevated for the
full job duration.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* test: full_job concurrency regression tests (issue #1318)
Add two integration tests verifying full_job routine concurrency:
1. full_job_max_concurrent_blocks_second_fire_while_first_active:
Inserts a Running routine_run (simulating an in-flight full_job) and
verifies fire_manual returns MaxConcurrent error for max_concurrent=1.
2. global_concurrency_counts_live_full_job_runs:
Elevates running_count to simulate a live full_job holding the global
slot, verifies check_cron_triggers skips due routines, then releases
the slot and verifies the routine fires.
Also makes running_count_for_test() unconditionally public so integration
tests (separate crate) can access it.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* style: fmt and clippy fixes for full_job concurrency tests
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* fix: address PR review feedback on FullJobWatcher
- Add #[doc(hidden)] to running_count_for_test() to hide from public API
- Derive MAX_POLLS from POLL_INTERVAL to keep constants coupled
- Check job state before first sleep to finalize promptly for fast jobs
- Update execute_full_job doc comment to reflect blocking behavior
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>
One flaky test (test_builtin_echo_tool timeout) was stopping the entire
e2e coverage suite via -x, preventing 118+ remaining tests from running
and generating coverage data.
Tests are independent (each gets a fresh browser context via the
function-scoped page fixture), so removing -x is safe.
Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>
* fix: consume matched event routine messages
* style: run rustfmt for event routine fix
* fix: preserve preprocessing for routine-triggered messages
* fix: match routines against rewritten input
* refactor: narrow check_event_triggers API and simplify routine_engine_slot
Address Copilot review feedback:
- Change check_event_triggers to accept (user_id, channel, content) instead
of &IncomingMessage, eliminating the need to clone the full message
(including attachments) when hooks rewrite content.
- Remove routine_trigger_message and the Cow<IncomingMessage> indirection;
the event-trigger check now inlines the is_internal + UserInput guard and
passes the post-hook content string directly.
- Make routine_engine_slot non-optional since Agent::new() always
initializes it. Removes the redundant Option wrapper and simplifies
accessor/setter methods.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>
* fix: add debug_assert invariant guards to critical code paths (closes#1215)
Add three debug_assert! calls to catch impossible-in-correct-code states
early in debug builds without affecting release performance:
- execute_tool_with_safety: assert tool_name is non-empty at entry
- JobContext::transition_to: assert state machine transition is valid
- CircuitBreakerProvider::record_success: assert circuit is not Open
(check_allowed() must gate all calls before record_success())
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
* test: add regression test for empty tool name invariant guard
Covers the debug_assert!(!tool_name.is_empty()) added in execute_tool_with_safety.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
---------
Co-authored-by: Claude Sonnet 4.6 <[email protected]>
* feat: upgrade MiniMax default model to M2.7
- Add MiniMax-M2.7 and MiniMax-M2.7-highspeed to model list
- Set MiniMax-M2.7 as default model
- Keep all previous models as alternatives
- Update related tests
* fix: use canonical model name in test per review
Use MiniMax-M2.7-highspeed (canonical casing) in the reasoning
models test for consistency with the documentation and provider
configuration.
[skip-regression-check]
These tests guard against catastrophic regex backtracking (seconds/minutes),
not 12ms differences. CI runners with coverage instrumentation (cargo-llvm-cov)
consistently exceed the 100ms threshold due to overhead, causing flaky failures.
500ms still catches real regressions while tolerating CI variability.
[skip-regression-check]
Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>
* fix: Rate limiter returns retry after None instead of a duration
linter fix
* review fixes
* fix: rate limiter returns None for retry_after duration
Add regression test to src/llm/retry.rs that verifies RateLimited errors
always have a fallback duration (never None) due to the 60-second fallback
applied in all rate limit error creation sites (nearai_chat.rs,
anthropic_oauth.rs, embeddings.rs).
The production code fix adds `.or(Some(Duration::from_secs(60)))` to ensure
the error message never displays "retry after None" to the user.
[skip-regression-check]
Co-Authored-By: Claude Haiku 4.5 <[email protected]>
---------
Co-authored-by: Claude Haiku 4.5 <[email protected]>
