mirror of
https://github.com/outbackdingo/optimclaw.git
synced 2026-08-31 16:49:34 +00:00
Add automated QA: schema validator, CI matrix, Docker build, and P1 test coverage (#353)
* Add automated QA: tool schema validator, feature-flag CI matrix, Docker build P0 items from the automated QA plan (#352): - Add validate_tool_schema() that checks OpenAI strict-mode rules (type: object, required keys in properties, nested object/array recursion) with 10 unit tests and 6 integration tests covering all core built-in tools - CI test matrix now runs with --all-features, default features, and --no-default-features --features libsql to catch dead code behind wrong cfg gates - CI clippy now runs the same 3-feature matrix with --all flags - Docker build job added to catch missing files in Dockerfile Co-Authored-By: Claude Opus 4.6 <[email protected]> * Add P1 automated QA tests and fix LeakDetector prefix shadowing bug P1 test coverage: config round-trip (settings + bootstrap), shell tool arg handling, safety adversarial tests (sanitizer, leak detector, allowlist), turn persistence (conversations, metadata, pagination, jobs), and a clippy fix for libsql-only builds. Fixed a real bug where AhoCorasick non-overlapping prefix iteration caused shorter prefixes (e.g. "sk-") to shadow longer ones (e.g. "sk-ant-api"), preventing Anthropic API key and SSH private key detection. Co-Authored-By: Claude Opus 4.6 <[email protected]> * Add P2 automated QA tests: chaos, lifecycle, collision, and recovery Cover all P2 items from the automated QA plan: - Circuit breaker chaos tests (hanging provider, rapid cycles, mixed errors) - Failover chaos tests (hanging failover, all-fail, tools path, single provider) - Value estimator boundary tests (negative cost, zero price, zero earnings) - Context length recovery test (ContextLengthExceeded -> compact -> retry) - WASM channel lifecycle tests (write/commit/read round-trip, namespace isolation) - Extension registry collision tests (same-name different-kind coexistence) - Extension filesystem collision tests (separate dirs, detect_kind priority) Co-Authored-By: Claude Opus 4.6 <[email protected]> * Add P3 concurrent stress tests for ContextManager and SessionManager Tests verify thread safety of double-checked locking, TOCTOU prevention, and RwLock-based concurrent access patterns under load. Co-Authored-By: Claude Opus 4.6 <[email protected]> * Add dispatcher loop guard and self-repair stuck job tests Dispatcher: test force_text mechanism prevents infinite tool call loops, verify iteration bound arithmetic guarantees termination for all configs. Self-repair: test stuck job detection, recovery within attempt limits, manual escalation when limit exceeded, graceful degradation without store/builder dependencies. Co-Authored-By: Claude Opus 4.6 <[email protected]> * Add E2E testing infrastructure design doc Python + Playwright framework with mock LLM server for deterministic browser-level testing of the web gateway. Covers connection/auth, chat round-trip with SSE streaming, and skills lifecycle scenarios. Co-Authored-By: Claude Opus 4.6 <[email protected]> * Add E2E testing infrastructure implementation plan 10-task plan covering: scaffolding, mock LLM server, helpers, conftest fixtures, connection/chat/skills test scenarios, CI workflow, README, and integration run. Co-Authored-By: Claude Opus 4.6 <[email protected]> * scaffold: E2E test project with pyproject.toml Co-Authored-By: Claude Opus 4.6 <[email protected]> * feat: E2E helpers with DOM selectors and port discovery Co-Authored-By: Claude Opus 4.6 <[email protected]> * feat: mock OpenAI-compat LLM server for E2E tests Co-Authored-By: Claude Opus 4.6 <[email protected]> * feat: E2E conftest with session fixtures for mock LLM and ironclaw Co-Authored-By: Claude Opus 4.6 <[email protected]> * feat: E2E scenario 1 -- connection and tab navigation tests Co-Authored-By: Claude Opus 4.6 <[email protected]> * feat: E2E scenario 2 -- chat message round-trip tests Co-Authored-By: Claude Opus 4.6 <[email protected]> * feat: E2E scenario 3 -- skills search, install, remove tests Co-Authored-By: Claude Opus 4.6 <[email protected]> * ci: add weekly E2E test workflow with Playwright Co-Authored-By: Claude Opus 4.6 <[email protected]> * docs: E2E test README with setup and usage instructions Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: E2E test integration fixes from first run - Use temp file DB instead of :memory: (libSQL :memory: doesn't persist tables across execute_batch) - Fix installed skills selector: #skills-list not #installed-skills - Add pytest-timeout to dependencies - Improve skills install/remove test with wait_for instead of fixed sleeps 8 passed, 1 skipped (skills install depends on ClawHub availability) Co-Authored-By: Claude Opus 4.6 <[email protected]> * test: add OpenAI strict-mode schema validator for all built-in tools (QA 1.1) Add src/tools/schema_validator.rs with validate_strict_schema() that checks tool parameter schemas against OpenAI function calling strict-mode rules: type object at top level, required keys in properties, enum type consistency, array items definitions, nested object recursion, and additionalProperties. 17 tests validate all 34+ built-in tool schemas across 5 test groups: - 9 simple tools (echo, time, json, http, shell, file read/write/list/patch) - 4 job tools (create, list, status, cancel) - 4 skill tools (list, search, install, remove) - 13 inline schemas for extension, routine, and complex job tools - 4 memory tool schemas Co-Authored-By: Claude Opus 4.6 <[email protected]> * test: add E2E scenarios for SSE reconnect, HTML injection, and tool approval (QA 3.3/5/6) Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: E2E test reliability for HTML injection and SSE reconnect - HTML injection: test sanitization directly via JS injection instead of depending on full LLM round-trip (avoids intermittent 404 from mock) - SSE reconnect: increase wait times for DB persistence and relax assertion to check total message count after history reload Co-Authored-By: Claude Opus 4.6 <[email protected]> * style: cargo fmt formatting Co-Authored-By: Claude Opus 4.6 <[email protected]> * test: add WASM and MCP tool schema validation tests (QA 1.1) Extends the schema validator with representative WASM tool schemas (weather, HTTP client, batch processor, status), MCP tool schemas (default, file read, SQL query, strict mode), and defect detection tests for common external schema issues (missing type, typo in required, array without items, enum type mismatch). Co-Authored-By: Claude Opus 4.6 <[email protected]> * test: add auth middleware and compaction module tests Auth middleware (8 new tests): valid/invalid bearer tokens, query param fallback, case sensitivity, empty tokens, whitespace handling. Compaction module (16 new tests): truncation strategy, summarize strategy with mock LLM, workspace fallback, format_turns helper, sequential compactions, coherence after compaction, token decrease verification. Co-Authored-By: Claude Opus 4.6 <[email protected]> * test: add config round-trip integration tests (QA 1.2) Test the full bootstrap .env lifecycle: write via the same format as save_bootstrap_env/upsert_bootstrap_var, read back via dotenvy, and assert values match. Covers LLM backend selection, embedding disable flag, onboard completion flag, session token keys, multi-key preservation across upsert, and special characters (spaces, equals, quotes, backslashes, hashes). Co-Authored-By: Claude Opus 4.6 <[email protected]> * test: add value estimator boundary tests and dispatcher loop guard (QA 4.3/4.4) Value estimator (14 new tests): zero/negative prices, large values, negative cost, exact margin boundaries, custom margin configuration. Dispatcher loop guard (2 new tests): verifies the dispatch loop terminates when all tool calls fail (regression guard for PR #252 infinite loop) and when max iterations are reached. Co-Authored-By: Claude Opus 4.6 <[email protected]> * test: add failover edge cases and provider chaos tests (QA 2.6/4.1) Failover edge cases (4 new tests): cooldown at zero nanos, half-open failure reopens circuit, all providers fail gracefully (no panic), single failing provider with cooldown. Provider chaos tests (15 new tests): flakey provider with retries, hanging provider with timeout, garbage provider, circuit breaker trip/recover, failover chain cascading, non-transient error stops chain, full stack integration (retry + failover + circuit breaker). Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: address PR review feedback on QA tests - Fix Bearer auth case-sensitivity per RFC 6750 (auth.rs) - Refactor bootstrap.rs to expose path-parameterized variants so config_round_trip tests call real code instead of reimplementations - Remove deprecated event_loop fixture, use dynamic ports, minimal env, session-scoped browser, and wire HEADED=1 in E2E conftest - Add cross-referencing doc comments between schema validators - Simplify array validation logic in tool.rs - Bump e2e.yml checkout@v4 to @v6 Co-Authored-By: Claude Opus 4.6 <[email protected]> * style: cargo fmt and fix clippy warning in signal.rs Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: improve E2E fixture error reporting and prevent stdin blocking - Add --no-onboard flag to prevent wizard from blocking in CI - Pipe /dev/null to stdin to prevent any stdin reads from hanging - Add RUST_BACKTRACE=1 for crash diagnostics - On server startup timeout, dump stderr to pytest output so CI logs show why the server failed to start Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: set session-scoped event loop for E2E async fixtures pytest-asyncio 1.3.0 defaults asyncio_default_fixture_loop_scope to None (function scope), causing session-scoped async fixtures to be re-evaluated per test function with independent event loops. Each test then independently attempts to start the ironclaw server, times out at 120s, and wastes ~24 minutes of CI before the job is cancelled. Setting asyncio_default_fixture_loop_scope = "session" ensures all session-scoped async fixtures share a single event loop, so the server starts once and is reused across all tests. Also adds -x flag to pytest in CI to stop on first failure instead of running all 19 tests when the fixture is broken. Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: set test loop scope to session to match fixture loop scope With asyncio_default_fixture_loop_scope=session but asyncio_default_test_loop_scope=function (the default), tests run on a per-function event loop while fixtures produce objects (Playwright pages, browser contexts) on the session event loop. This event loop mismatch causes the test to hang indefinitely awaiting Playwright operations that are bound to the wrong loop. Setting both scopes to "session" ensures a single event loop is shared across all fixtures and tests, eliminating the deadlock. Co-Authored-By: Claude Opus 4.6 <[email protected]> * ci: add roll-up jobs to match branch protection required checks Branch protection expects "Code Style (fmt + clippy)" and "Run Tests" status checks, but only individual job names were reported. Add roll-up jobs that aggregate results and report the expected names. Co-Authored-By: Claude Opus 4.6 <[email protected]> --------- Co-authored-by: Claude Opus 4.6 <[email protected]>
This commit is contained in:
Zaki Manian
co-authored by
Claude Opus 4.6
Claude Opus 4.6
parent
e8eb4ca0bd
commit
a24fd3e8a3
@@ -567,4 +567,205 @@ mod tests {
|
||||
assert_eq!(cb.cost_per_token(), (Decimal::ZERO, Decimal::ZERO));
|
||||
assert_eq!(cb.calculate_cost(100, 50), Decimal::ZERO);
|
||||
}
|
||||
|
||||
// === QA Plan P2 - 4.1: Provider chaos tests ===
|
||||
|
||||
/// Provider that hangs forever (tests timeout handling at the caller).
|
||||
struct HangingProvider;
|
||||
|
||||
#[async_trait]
|
||||
impl LlmProvider for HangingProvider {
|
||||
fn model_name(&self) -> &str {
|
||||
"hanging"
|
||||
}
|
||||
fn cost_per_token(&self) -> (Decimal, Decimal) {
|
||||
(Decimal::ZERO, Decimal::ZERO)
|
||||
}
|
||||
async fn complete(
|
||||
&self,
|
||||
_request: CompletionRequest,
|
||||
) -> Result<CompletionResponse, LlmError> {
|
||||
// Hang forever
|
||||
std::future::pending().await
|
||||
}
|
||||
async fn complete_with_tools(
|
||||
&self,
|
||||
_request: ToolCompletionRequest,
|
||||
) -> Result<ToolCompletionResponse, LlmError> {
|
||||
std::future::pending().await
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn hanging_provider_behind_breaker_can_be_timed_out() {
|
||||
let hanging: Arc<dyn LlmProvider> = Arc::new(HangingProvider);
|
||||
let cb = CircuitBreakerProvider::new(hanging, fast_config(1));
|
||||
|
||||
// The caller should be able to timeout the request.
|
||||
let result =
|
||||
tokio::time::timeout(Duration::from_millis(100), cb.complete(make_request())).await;
|
||||
|
||||
// Should timeout, not hang forever.
|
||||
assert!(result.is_err(), "should timeout, not hang");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn rapid_open_close_cycles_do_not_corrupt_state() {
|
||||
let stub = Arc::new(StubLlm::failing("test"));
|
||||
let cb = CircuitBreakerProvider::new(
|
||||
stub.clone(),
|
||||
CircuitBreakerConfig {
|
||||
failure_threshold: 1,
|
||||
recovery_timeout: Duration::from_millis(10),
|
||||
half_open_successes_needed: 1,
|
||||
},
|
||||
);
|
||||
|
||||
// Cycle through open/half-open/open several times.
|
||||
for _ in 0..5 {
|
||||
// Trip to open.
|
||||
let _ = cb.complete(make_request()).await;
|
||||
assert_eq!(cb.circuit_state().await, CircuitState::Open);
|
||||
|
||||
// Wait for recovery.
|
||||
tokio::time::sleep(Duration::from_millis(15)).await;
|
||||
|
||||
// Probe fails (stub still failing) → back to Open.
|
||||
let _ = cb.complete(make_request()).await;
|
||||
assert_eq!(cb.circuit_state().await, CircuitState::Open);
|
||||
}
|
||||
|
||||
// Now flip to succeeding and verify recovery still works.
|
||||
tokio::time::sleep(Duration::from_millis(15)).await;
|
||||
stub.set_failing(false);
|
||||
let result = cb.complete(make_request()).await;
|
||||
assert!(result.is_ok());
|
||||
assert_eq!(cb.circuit_state().await, CircuitState::Closed);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn mixed_error_types_only_transient_counts() {
|
||||
// Non-transient errors should never trip the breaker, even after many attempts.
|
||||
let non_transient = Arc::new(StubLlm::failing_non_transient("test"));
|
||||
let cb_nt = CircuitBreakerProvider::new(non_transient, fast_config(3));
|
||||
|
||||
// 100 non-transient errors should not trip the breaker.
|
||||
for _ in 0..100 {
|
||||
let _ = cb_nt.complete(make_request()).await;
|
||||
}
|
||||
assert_eq!(cb_nt.circuit_state().await, CircuitState::Closed);
|
||||
assert_eq!(cb_nt.consecutive_failures().await, 0);
|
||||
}
|
||||
|
||||
// === QA Plan 2.6: Edge case tests ===
|
||||
|
||||
/// With a recovery_timeout of zero, the circuit should transition from
|
||||
/// Open to HalfOpen immediately on the next call (the elapsed time
|
||||
/// always >= Duration::ZERO). This verifies that zero-duration timeouts
|
||||
/// are not treated as a special "disabled" sentinel.
|
||||
#[tokio::test]
|
||||
async fn test_cooldown_at_zero_nanos() {
|
||||
let stub = Arc::new(StubLlm::failing("test"));
|
||||
let cb = CircuitBreakerProvider::new(
|
||||
stub.clone(),
|
||||
CircuitBreakerConfig {
|
||||
failure_threshold: 1,
|
||||
recovery_timeout: Duration::ZERO,
|
||||
half_open_successes_needed: 1,
|
||||
},
|
||||
);
|
||||
|
||||
// Trip the breaker with one failure.
|
||||
let _ = cb.complete(make_request()).await;
|
||||
assert_eq!(cb.circuit_state().await, CircuitState::Open);
|
||||
|
||||
// With recovery_timeout = 0, the very next call should transition
|
||||
// from Open -> HalfOpen immediately (no sleep needed).
|
||||
// Since the stub is still failing, the probe will fail, sending
|
||||
// it back to Open. But the key assertion is that the transition
|
||||
// to HalfOpen actually happened (not stuck in Open forever).
|
||||
stub.set_failing(false);
|
||||
let result = cb.complete(make_request()).await;
|
||||
assert!(
|
||||
result.is_ok(),
|
||||
"zero recovery_timeout should allow immediate probe"
|
||||
);
|
||||
assert_eq!(
|
||||
cb.circuit_state().await,
|
||||
CircuitState::Closed,
|
||||
"successful probe after zero-timeout should close the circuit"
|
||||
);
|
||||
|
||||
// Verify it also works when the probe fails: should re-open, not
|
||||
// get stuck in some intermediate state.
|
||||
stub.set_failing(true);
|
||||
// Trip again.
|
||||
let _ = cb.complete(make_request()).await;
|
||||
assert_eq!(cb.circuit_state().await, CircuitState::Open);
|
||||
// Next call: Open -> HalfOpen (zero timeout), probe fails -> Open.
|
||||
let _ = cb.complete(make_request()).await;
|
||||
assert_eq!(
|
||||
cb.circuit_state().await,
|
||||
CircuitState::Open,
|
||||
"failed probe should re-open circuit even with zero timeout"
|
||||
);
|
||||
}
|
||||
|
||||
/// When in half-open state, a single failure should immediately
|
||||
/// re-open the circuit (not close it or leave it in half-open).
|
||||
/// Also verifies that any accumulated half_open_successes are reset.
|
||||
#[tokio::test]
|
||||
async fn test_circuit_breaker_half_open_failure_reopens() {
|
||||
let stub = Arc::new(StubLlm::failing("test"));
|
||||
let cb = CircuitBreakerProvider::new(
|
||||
stub.clone(),
|
||||
CircuitBreakerConfig {
|
||||
failure_threshold: 1,
|
||||
recovery_timeout: Duration::from_millis(20),
|
||||
half_open_successes_needed: 3, // require multiple successes
|
||||
},
|
||||
);
|
||||
|
||||
// Trip the breaker.
|
||||
let _ = cb.complete(make_request()).await;
|
||||
assert_eq!(cb.circuit_state().await, CircuitState::Open);
|
||||
|
||||
// Wait for recovery, then succeed once to accumulate 1 half-open success.
|
||||
tokio::time::sleep(Duration::from_millis(30)).await;
|
||||
stub.set_failing(false);
|
||||
let _ = cb.complete(make_request()).await;
|
||||
// Still in half-open (need 3 successes, got 1).
|
||||
assert_eq!(cb.circuit_state().await, CircuitState::HalfOpen);
|
||||
|
||||
// Now fail: should immediately re-open, discarding the 1 accumulated success.
|
||||
stub.set_failing(true);
|
||||
let _ = cb.complete(make_request()).await;
|
||||
assert_eq!(
|
||||
cb.circuit_state().await,
|
||||
CircuitState::Open,
|
||||
"failure in half-open should immediately re-open the circuit"
|
||||
);
|
||||
|
||||
// After re-opening, wait for recovery and verify that the half-open
|
||||
// success counter was reset (need 3 fresh successes, not 2).
|
||||
tokio::time::sleep(Duration::from_millis(30)).await;
|
||||
stub.set_failing(false);
|
||||
|
||||
// First success: half-open, count=1.
|
||||
let _ = cb.complete(make_request()).await;
|
||||
assert_eq!(cb.circuit_state().await, CircuitState::HalfOpen);
|
||||
|
||||
// Second success: half-open, count=2.
|
||||
let _ = cb.complete(make_request()).await;
|
||||
assert_eq!(cb.circuit_state().await, CircuitState::HalfOpen);
|
||||
|
||||
// Third success: closes the circuit.
|
||||
let _ = cb.complete(make_request()).await;
|
||||
assert_eq!(
|
||||
cb.circuit_state().await,
|
||||
CircuitState::Closed,
|
||||
"3 fresh successes needed after re-open, not 2"
|
||||
);
|
||||
assert_eq!(cb.consecutive_failures().await, 0);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1154,4 +1154,170 @@ mod tests {
|
||||
// FailoverProvider itself should report the new model.
|
||||
assert_eq!(failover.active_model_name(), "new-model");
|
||||
}
|
||||
|
||||
// === QA Plan P2 - 4.1: Provider chaos tests ===
|
||||
|
||||
#[tokio::test]
|
||||
async fn hanging_provider_failover_to_healthy_one() {
|
||||
// When primary hangs, caller can timeout and the secondary should be reachable
|
||||
// on a fresh request. The failover itself doesn't timeout individual providers
|
||||
// (that's the HTTP client's job), but after the first provider enters cooldown
|
||||
// from repeated failures, the failover skips it.
|
||||
let p1 = Arc::new(MultiCallMockProvider::always_fail("p1-broken"));
|
||||
let p2 = Arc::new(MultiCallMockProvider::always_ok("p2-healthy"));
|
||||
|
||||
let config = CooldownConfig {
|
||||
cooldown_duration: Duration::from_secs(60),
|
||||
failure_threshold: 1,
|
||||
};
|
||||
let failover =
|
||||
FailoverProvider::with_cooldown(vec![p1.clone(), p2.clone()], config).unwrap();
|
||||
|
||||
// First request: p1 fails → cooldown, p2 succeeds.
|
||||
let r = failover.complete(make_request()).await.unwrap();
|
||||
assert_eq!(r.content, "p2-healthy ok");
|
||||
|
||||
// Second request: p1 skipped (in cooldown), p2 serves directly.
|
||||
let prev_p1 = p1.call_count();
|
||||
let r = failover.complete(make_request()).await.unwrap();
|
||||
assert_eq!(r.content, "p2-healthy ok");
|
||||
assert_eq!(p1.call_count(), prev_p1, "p1 should be skipped in cooldown");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn all_providers_fail_returns_error_not_panic() {
|
||||
let p1 = Arc::new(MultiCallMockProvider::always_fail("p1"));
|
||||
let p2 = Arc::new(MultiCallMockProvider::always_fail("p2"));
|
||||
let p3 = Arc::new(MultiCallMockProvider::always_fail("p3"));
|
||||
|
||||
let failover = FailoverProvider::new(vec![p1 as Arc<dyn LlmProvider>, p2, p3]).unwrap();
|
||||
|
||||
// Should return an error, not panic.
|
||||
let result = failover.complete(make_request()).await;
|
||||
assert!(result.is_err());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn failover_with_tools_follows_same_path() {
|
||||
let p1 = Arc::new(MultiCallMockProvider::always_fail("p1"));
|
||||
let p2 = Arc::new(MultiCallMockProvider::always_ok("p2"));
|
||||
|
||||
let failover = FailoverProvider::new(vec![p1 as Arc<dyn LlmProvider>, p2]).unwrap();
|
||||
|
||||
let result = failover.complete_with_tools(make_tool_request()).await;
|
||||
assert!(result.is_ok());
|
||||
assert_eq!(result.unwrap().content.unwrap(), "p2 ok");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn single_provider_failover_still_works() {
|
||||
let p1 = Arc::new(MultiCallMockProvider::always_ok("solo"));
|
||||
let failover = FailoverProvider::new(vec![p1 as Arc<dyn LlmProvider>]).unwrap();
|
||||
|
||||
let result = failover.complete(make_request()).await;
|
||||
assert!(result.is_ok());
|
||||
assert_eq!(result.unwrap().content, "solo ok");
|
||||
}
|
||||
|
||||
// === QA Plan 2.6: Failover edge case tests ===
|
||||
|
||||
/// When all providers fail with retryable errors, the failover must
|
||||
/// return a graceful error (not panic via .unwrap()/.expect()). Verify
|
||||
/// the error content includes the last provider's identity.
|
||||
#[tokio::test]
|
||||
async fn test_failover_all_providers_fail_no_panic() {
|
||||
let p1 = Arc::new(MultiCallMockProvider::always_fail("alpha"));
|
||||
let p2 = Arc::new(MultiCallMockProvider::always_fail("beta"));
|
||||
let p3 = Arc::new(MultiCallMockProvider::always_fail("gamma"));
|
||||
|
||||
let failover = FailoverProvider::new(vec![
|
||||
p1 as Arc<dyn LlmProvider>,
|
||||
p2 as Arc<dyn LlmProvider>,
|
||||
p3 as Arc<dyn LlmProvider>,
|
||||
])
|
||||
.unwrap();
|
||||
|
||||
// All three providers fail. Must return Err, not panic.
|
||||
let result = failover.complete(make_request()).await;
|
||||
assert!(result.is_err(), "should return error, not panic");
|
||||
let err = result.unwrap_err();
|
||||
match &err {
|
||||
LlmError::RequestFailed { provider, reason } => {
|
||||
// The last error should come from the last provider tried.
|
||||
assert_eq!(
|
||||
provider, "gamma",
|
||||
"error should identify the last provider tried"
|
||||
);
|
||||
assert!(
|
||||
reason.contains("failed"),
|
||||
"error reason should describe the failure: {}",
|
||||
reason
|
||||
);
|
||||
}
|
||||
other => panic!("expected RequestFailed, got: {:?}", other),
|
||||
}
|
||||
|
||||
// Also test complete_with_tools follows the same graceful path.
|
||||
let p4 = Arc::new(MultiCallMockProvider::always_fail("delta"));
|
||||
let p5 = Arc::new(MultiCallMockProvider::always_fail("epsilon"));
|
||||
let failover2 =
|
||||
FailoverProvider::new(vec![p4 as Arc<dyn LlmProvider>, p5 as Arc<dyn LlmProvider>])
|
||||
.unwrap();
|
||||
|
||||
let result = failover2.complete_with_tools(make_tool_request()).await;
|
||||
assert!(
|
||||
result.is_err(),
|
||||
"complete_with_tools should also return error, not panic"
|
||||
);
|
||||
}
|
||||
|
||||
/// A single provider that always fails with no fallback available.
|
||||
/// Verifies the failover returns the error from that provider and
|
||||
/// does not panic or produce an "unreachable" invariant violation.
|
||||
#[tokio::test]
|
||||
async fn test_failover_with_single_provider_failing() {
|
||||
let solo = Arc::new(MultiCallMockProvider::always_fail("solo-broken"));
|
||||
let failover = FailoverProvider::new(vec![solo.clone() as Arc<dyn LlmProvider>]).unwrap();
|
||||
|
||||
// First call: should return error from the solo provider.
|
||||
let result = failover.complete(make_request()).await;
|
||||
assert!(result.is_err());
|
||||
match result.unwrap_err() {
|
||||
LlmError::RequestFailed { provider, .. } => {
|
||||
assert_eq!(provider, "solo-broken");
|
||||
}
|
||||
other => panic!("expected RequestFailed, got: {:?}", other),
|
||||
}
|
||||
|
||||
// After repeated failures, the single provider enters cooldown.
|
||||
// But since it's the only provider, the "never skip all" logic
|
||||
// should still try it (as the oldest-cooled provider).
|
||||
let config = CooldownConfig {
|
||||
cooldown_duration: Duration::from_secs(300),
|
||||
failure_threshold: 1,
|
||||
};
|
||||
let solo2 = Arc::new(MultiCallMockProvider::always_fail("solo-cd"));
|
||||
let failover2 =
|
||||
FailoverProvider::with_cooldown(vec![solo2.clone() as Arc<dyn LlmProvider>], config)
|
||||
.unwrap();
|
||||
|
||||
// First call: fails, enters cooldown (threshold=1).
|
||||
let _ = failover2.complete(make_request()).await;
|
||||
assert_eq!(solo2.call_count(), 1);
|
||||
|
||||
// Second call: provider is in cooldown, but it's the only one,
|
||||
// so "never skip all" should try it anyway.
|
||||
let result = failover2.complete(make_request()).await;
|
||||
assert!(result.is_err(), "should still fail but not panic");
|
||||
assert_eq!(
|
||||
solo2.call_count(),
|
||||
2,
|
||||
"sole provider should be retried despite cooldown"
|
||||
);
|
||||
|
||||
// Third call: same behavior, no state corruption.
|
||||
let result = failover2.complete(make_request()).await;
|
||||
assert!(result.is_err());
|
||||
assert_eq!(solo2.call_count(), 3);
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user