* feat: structured fallback deliverables for failed/stuck jobs (#221)
When a job fails or gets stuck, build a FallbackDeliverable that captures
partial results, action statistics, cost, timing, and repair attempts.
This replaces opaque error strings with structured data users can act on.
- Add FallbackDeliverable, LastAction, ActionStats types in context/fallback.rs
- Store fallback in JobContext.metadata["fallback_deliverable"] on failure
- Surface fallback in job_status tool output and SSE job_result events
- Update mark_failed() and mark_stuck() in worker to build fallback
- 8 unit tests covering zero/mixed actions, truncation, timing, serialization
Co-Authored-By: Claude Opus 4.6 <[email protected]>
* fix: address review comments on fallback deliverables
- Fix doc comment: "200 chars" -> "200 bytes (UTF-8 safe)" since
truncate_str operates on byte length, not character count.
- Add code comment documenting that SSE fallback_deliverable is
currently always None (forward-compatible infrastructure).
Co-Authored-By: Claude Opus 4.6 <[email protected]>
* refactor: take Option<&FallbackDeliverable> instead of &Option<…>
Addresses Gemini review feedback: idiomatic Rust prefers
Option<&T> over &Option<T> for borrowed optional values.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
* fix: guard against non-object metadata and add fallback test
- store_fallback_in_metadata now resets metadata to {} when it's any
non-object type (string, array, number), not just null. Prevents
panic on index assignment.
- Add test_job_status_includes_fallback_deliverable to verify the
fallback field is surfaced in job_status tool output.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
* fix: use sanitized output in fallback preview + add integration tests
Security fix: FallbackDeliverable::build() now uses output_sanitized
instead of output_raw, preventing secrets/PII from leaking through
the job_status tool and SSE job_result events.
Also adds:
- test_fallback_uses_sanitized_output: proves raw secrets don't leak
- test_store_fallback_in_metadata_roundtrip: full serialize/deserialize
- test_store_fallback_handles_non_object_metadata: edge case coverage
- test_store_fallback_none_is_noop: None input is safe
Addresses serrrfirat review feedback on PR #236.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
* fix: harden fallback deliverables against review findings
- Truncate failure_reason to 1000 bytes to prevent metadata bloat
- Add tracing::warn on fallback serialization failure (was silently discarded)
- Fix module/struct docs to cover stuck jobs, remove stale SSE claim
- Fix job.rs test to use real FallbackDeliverable field names
- Add tests for failure_reason truncation and completed_at=None elapsed time
- Fix pre-existing clippy warning in settings.rs (field_reassign_with_default)
Co-Authored-By: Claude Opus 4.6 <[email protected]>
* fix: address Copilot review findings on fallback deliverables
- Fix output_raw/output_sanitized field swap in ActionRecord::succeed()
so sanitized data actually goes into the sanitized field (security)
- Return None instead of empty Memory when get_memory fails in
build_fallback, with tracing::warn for observability
- Replace manual elapsed calculation with ctx.elapsed() which already
clamps negative durations
Co-Authored-By: Claude Opus 4.6 <[email protected]>
* fix: resolve rebase conflicts and update tests for parameter swap
- Add fallback field to SseEvent::JobResult in job_monitor
- Fix type annotation in fallback deliverable test
- Update test_action_record_succeed_sets_fields for new parameter order
- Use create_job_for_user in test (API changed on main)
Co-Authored-By: Claude Opus 4.6 <[email protected]>
* chore: trigger CI re-check after rebase
* fix: fall back to error message for failed action output_preview
When the last action is a failed tool call, output_sanitized is None,
leaving output_preview empty. Now falls back to the action's error
message so users see what went wrong.
[skip-regression-check]
* ci: add safety comments to test code for no-panics check
The CI no-panics grep check cannot distinguish test code inside
src/ files from production code. Add // safety: test annotations
to .unwrap(), .expect(), and assert!() calls in #[cfg(test)] modules.
* fix: clarify succeed() doc and avoid clone in output_preview
- Fix doc comment: output_raw is stored as pretty-printed JSON string,
not a raw JSON value
- Borrow string slice directly in fallback preview to avoid cloning
potentially large sanitized outputs before truncation
* refactor: reuse floor_char_boundary in truncate_str
Replace hand-rolled UTF-8 boundary logic with existing
crate::util::floor_char_boundary to reduce duplication.
* fix: rename SSE fallback field to fallback_deliverable for consistency
The SSE JobResult field was named `fallback` while everywhere else
(metadata key, job_status tool) uses `fallback_deliverable`. Align
the SSE wire format to avoid forcing clients to handle two names.
---------
Co-authored-by: Claude Opus 4.6 <[email protected]>
* feat: configurable hybrid search fusion strategy (#169)
Add WeightedScore fusion as an alternative to the default RRF algorithm.
Users can now tune search behavior via env vars (SEARCH_FUSION_STRATEGY,
SEARCH_FTS_WEIGHT, SEARCH_VECTOR_WEIGHT, SEARCH_RRF_K) or by passing
SearchConfig with the new fields. Default behavior (RRF, k=60) is
unchanged.
- Add FusionStrategy enum (Rrf/WeightedScore) to workspace::search
- Add weighted_score_fusion() and fuse_results() dispatcher
- Add config/search.rs with WorkspaceSearchConfig from env vars
- Wire search defaults through Workspace struct
- Update both postgres and libsql backends to use fuse_results()
- Add 7 new tests (4 fusion + 3 config)
Co-Authored-By: Claude Opus 4.6 <[email protected]>
* fix: swap default search weights to match issue #169 spec (0.7 vector / 0.3 FTS)
The issue spec says "0.7/0.3 (vector/keyword) for weighted mode" but
our defaults had fts_weight=0.7, vector_weight=0.3 (inverted). Also
fixes the misleading docstring on weighted_score_fusion that claimed
1/rank normalizes to [0,1].
Co-Authored-By: Claude Opus 4.6 <[email protected]>
* fix: validate weight inputs and update stale doc comments
- Reject NaN, infinite, and negative values for SEARCH_FTS_WEIGHT and
SEARCH_VECTOR_WEIGHT with a clear ConfigError
- Fix module-level docs that incorrectly claimed WeightedScore
"normalizes per-method scores to [0,1]"
- Update SearchResult.score doc from "Combined RRF score" to
strategy-agnostic "Combined fusion score"
Co-Authored-By: Claude Opus 4.6 <[email protected]>
* fix: validate weight setters against NaN/inf/negative values
with_fts_weight() and with_vector_weight() now silently ignore
non-finite (NaN, ±inf) and negative values, matching the env var
validation already in place for SEARCH_FTS_WEIGHT / SEARCH_VECTOR_WEIGHT.
Values > 1.0 remain valid since weights are normalized internally.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
* fix: use crate-wide ENV_MUTEX in search config tests
Replace the module-local `ENV_MUTEX` in `search.rs` with a shared
`crate::config::helpers::ENV_MUTEX` to prevent cross-module env races
when `cargo test` runs tests in parallel.
Addresses copilot review comment. Tracked in #245.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
* fix: per-strategy weight defaults to match issue #169 spec
RRF mode now defaults to 0.5/0.5 (fts/vector) and WeightedScore
defaults to 0.3/0.7, matching the acceptance criteria in #169.
Previously both modes used 0.3/0.7 uniformly.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
* fix: reject both weights=0 in weighted fusion mode
When both SEARCH_FTS_WEIGHT and SEARCH_VECTOR_WEIGHT are 0.0 under
WeightedScore strategy, all scores would be 0.0, producing arbitrary
ordering. RRF mode is unaffected since it ignores weights entirely.
Addresses Copilot review comment. The other comment (rrf_k=0 division
by zero) is a false positive — ranks are 1-based, so k=0 just gives
inverse-rank scoring with no infinity.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
* fix: clarify weight doc comments and error key
- SearchConfig field docs: clarify that Default always uses 0.5,
per-strategy defaults only apply via WorkspaceSearchConfig::resolve()
- WorkspaceSearchConfig field docs: same clarification
- Error key for both-weights-zero now references both env vars
Co-Authored-By: Claude Opus 4.6 <[email protected]>
* fix: remove broken intra-doc links to pub(crate) resolve()
WorkspaceSearchConfig::resolve is pub(crate), so linking to it from
public field docs triggers rustdoc private_intra_doc_links warnings.
Switch to plain-text references.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
* fix: add document_path to weighted_score_fusion results
The weighted_score_fusion function was missing the document_path field
added in a recent main branch commit, causing a compile error after rebase.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
* chore: trigger CI re-check after rebase
* fix: resolve pre-existing staging fmt and clippy issues
- Fix import ordering in cli/mod.rs (cargo fmt)
- Fix line wrapping in tools/mcp/auth.rs (cargo fmt)
- Move path_routing_tests before MemoryTreeTool to fix
clippy::items_after_test_module
[skip-regression-check]
* fix: remove duplicate path_routing_tests module after rebase
[skip-regression-check]
---------
Co-authored-by: Claude Opus 4.6 <[email protected]>
* feat: add OpenRouter preset to setup wizard
Add OpenRouter as a top-level provider option in the onboarding wizard
(Step 3). Selecting it pre-fills the base URL (https://openrouter.ai/api/v1)
and prompts for an API key, avoiding manual URL entry. Under the hood it
uses the existing openai_compatible backend.
Inlines the key collection flow (rather than delegating to
setup_api_key_provider) so success messages consistently say "OpenRouter"
instead of "openai_compatible", including the early-return env-key path.
Closes#178
Co-Authored-By: Claude Opus 4.6 <[email protected]>
* fix: address serrrfirat review comments on OpenRouter wizard preset
- Re-run path now recognizes OpenRouter: display shows "OpenRouter"
and keep-current routes to setup_openrouter() when base URL contains
openrouter.ai
- Refactor setup_openrouter() to delegate to setup_api_key_provider()
with a display_name override, eliminating ~40 lines of duplication
- Update README: remove false claim about model fetching from
OpenRouter API, add footnote explaining shared secret/env var
between OpenRouter and OpenAI-compatible
- Fix pre-existing clippy warning in settings.rs (field_reassign_with_default)
Co-Authored-By: Claude Opus 4.6 <[email protected]>
---------
Co-authored-by: Claude Opus 4.6 <[email protected]>
Add LLM_EXTRA_HEADERS env var (format: Key:Value,Key2:Value2) to inject
custom HTTP headers into every request to OpenAI-compatible endpoints.
This enables OpenRouter attribution headers (HTTP-Referer, X-Title)
and other service-specific headers without code changes.
Closes#179
Co-authored-by: Claude Opus 4.6 <[email protected]>
Co-authored-by: Illia Polosukhin <[email protected]>
Each config test module (llm.rs, embeddings.rs) defined its own
ENV_MUTEX, which doesn't prevent cross-module env races since
cargo test runs in parallel. Move to a single shared mutex in
config/helpers.rs so all unsafe set_var/remove_var calls are
serialized crate-wide.
Closes#245
Co-authored-by: Claude Opus 4.6 <[email protected]>
* fix: remove .expect() calls in FailoverProvider::try_providers (#155)
Replace two .expect() calls with proper error propagation to comply
with the project no-panic convention. Both were logically unreachable
but would panic if invariants were broken by a future refactor.
Closes#155
Co-Authored-By: Claude Opus 4.6 <[email protected]>
* Apply suggestions from code review
Co-authored-by: Copilot <[email protected]>
---------
Co-authored-by: Claude Opus 4.6 <[email protected]>
Co-authored-by: Illia Polosukhin <[email protected]>
Co-authored-by: Copilot <[email protected]>
ProviderCooldown used 0 as both the "not in cooldown" sentinel and a
valid timestamp from now_nanos(), so activate_cooldown(0) would silently
fail to activate. Store max(now_nanos, 1) to keep 0 reserved.
Closes#125
Co-authored-by: Claude Opus 4.6 <[email protected]>
Track per-provider failure state with lock-free atomics and temporarily
skip providers that have repeatedly failed with retryable errors. This
reduces latency when a provider is known to be down, instead of
wasting time on every request trying all providers sequentially.
- Add CooldownConfig (duration + threshold) and ProviderCooldown (atomics)
- Rewrite try_providers() to skip cooled-down providers, with a safety
net that always tries the oldest-cooled provider if all are down
- Add 2 env vars: LLM_FAILOVER_COOLDOWN_SECS, LLM_FAILOVER_THRESHOLD
- Add MultiCallMockProvider and 7 new test cases
- Mark "Cooldown management" as complete in FEATURE_PARITY.md
Co-authored-by: Claude Opus 4.6 <[email protected]>
* feat: add multi-provider LLM failover
Add FailoverProvider that wraps multiple LlmProvider instances and
tries each in sequence on transient failures. Non-retryable errors
(auth, context length, model not available) propagate immediately.
- New `FailoverProvider` with generic `try_providers` helper
- `is_retryable()` classifies transient errors (request failed,
rate limited, invalid response, session renewal, HTTP, IO)
- Configurable via `NEARAI_FALLBACK_MODEL` env var
- Returns `Result` from constructor (no panics in production)
- Updates FEATURE_PARITY.md: failover chains ✅, cooldown ❌
Co-Authored-By: Claude Opus 4.6 <[email protected]>
* fix: track last-used provider for accurate cost/model reporting
After failover, model_name() and cost_per_token() now reflect the
provider that actually handled the request, not always the primary.
Also corrects is_retryable() docs to list ModelNotAvailable as retryable.
Addresses PR #28 review comments.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
* feat: add retry with exponential backoff for LLM providers
Add retry logic with exponential backoff and jitter to both NearAiProvider
and NearAiChatProvider for transient errors (HTTP 429, 500, 502, 503, 504).
Extract shared retry helpers (is_retryable_status, retry_backoff_delay)
into src/llm/retry.rs so both providers reuse the same logic.
Configurable via NEARAI_MAX_RETRIES env var (default: 3).
* docs: clarify max_retries means N retries, not N total attempts
* warn when fallback model equals primary model
* fix: saturating_mul in backoff delay, dedupe to_lowercase allocation
---------
Co-authored-by: Claude Opus 4.6 <[email protected]>
