Fix(llm): complete response cache — set_model invalidation, stats logging, sync mutex (#290)

* fix(llm): complete response cache — set_model invalidation, stats logging, sync mutex

# Conflicts:
#	src/llm/response_cache.rs

* fix(llm): address response cache review comments

- Add total_hit_count AtomicU64 that is never decremented on eviction;
  maybe_log_stats now uses this counter so hit_rate_pct stays accurate
  under high eviction pressure
- Log cache stats before returning on provider error so milestone
  intervals (every 100 requests) are never silently skipped
- Add tracing-test dev-dep and three new tests: total_hits_survives_eviction,
  stats_logged_at_request_100, stats_logged_on_provider_error_at_interval
- Update PR description to reflect actual set_model() behavior (key
  isolation, not cache clear)

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

---------

Co-authored-by: Claude Sonnet 4.6 <[email protected]>
This commit is contained in:
Nitanshu Lokhande
2026-03-06 08:36:42 +00:00
committed by GitHub
co-authored by Claude Sonnet 4.6
parent 26d274ac79
commit 7806273aa6
3 changed files with 356 additions and 33 deletions
Generated
+22
View File
@@ -2902,6 +2902,7 @@ dependencies = [
"tower-http 0.6.8",
"tracing",
"tracing-subscriber",
"tracing-test",
"url",
"urlencoding",
"uuid",
@@ -6229,6 +6230,27 @@ dependencies = [
"tracing-serde",
]
[[package]]
name = "tracing-test"
version = "0.2.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "19a4c448db514d4f24c5ddb9f73f2ee71bfb24c526cf0c570ba142d1119e0051"
dependencies = [
"tracing-core",
"tracing-subscriber",
"tracing-test-macro",
]
[[package]]
name = "tracing-test-macro"
version = "0.2.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ad06847b7afb65c7866a36664b75c40b895e318cea4f71299f013fb22965329d"
dependencies = [
"quote",
"syn 2.0.114",
]
[[package]]
name = "try-lock"
version = "0.2.5"