feat: enable Anthropic prompt caching via automatic cache_control injection (#660)

* feat(llm): add Anthropic prompt caching and cache token tracking - Inject cache_control via additional_params for Claude models in rig_adapter - Add cache_read_input_tokens and cache_creation_input_tokens to CompletionResponse and ToolCompletionResponse - Extract cached_input_tokens from rig-core unified Usage - Add is_anthropic_model() detection helper with provider prefix support - Log prompt cache hits at debug level (consistent with response_cache) - Add 7 unit tests for cache injection and model detection - Update all mock providers and test fixtures with new fields * feat(cost): apply 90% cache discount to prompt-cached tokens in CostGuard - Add cache_read_input_tokens to TokenUsage so cache counts flow from CompletionResponse through the reasoning layer to the dispatcher - Update CostGuard::record_llm_call() to accept cache_read_input_tokens: cached tokens are billed at 10% of the normal input rate - Thread cache_read_input_tokens from dispatcher into CostGuard - Add test_cache_discount_reduces_cost verifying exact savings match 90% of input cost for fully-cached requests - Update all existing test callers with zero-cache parameter * refactor(cache): scope cache_control to Anthropic backend and validate model support - Replace model-name-based is_anthropic_model() with explicit enable_prompt_cache flag on RigAdapter, set only for the direct Anthropic backend via with_prompt_cache(true) - Add supports_prompt_cache() to validate model names per Anthropic docs: only Claude 3+ models support caching; claude-2 and claude-instant are excluded to prevent 400 errors - Warn when caching is enabled but model does not support it - Replace is_anthropic_model tests with flag-based and model validation tests * fix(cache): validate model at construction and propagate cache metrics through proxy - Move supports_prompt_cache() check into with_prompt_cache() so unsupported models are detected once at construction, not per request - Add cache_read_input_tokens and cache_creation_input_tokens to ProxyCompletionResponse and ProxyToolCompletionResponse with serde(default) for backward compatibility - Pass cache metrics through orchestrator proxy instead of zeroing - Use claude-opus-4-6 in cache discount test to match Anthropic semantics * feat(llm): add configurable cache retention with write surcharge - Add CacheRetention enum (none/short/long) to AnthropicDirectConfig - Parse ANTHROPIC_CACHE_RETENTION env var (default: short) - Inject TTL-aware cache_control (short=5m ephemeral, long=1h) - Extract cache_creation_input_tokens from raw Anthropic response - Add cache_write_multiplier() to LlmProvider trait (1.25x short, 2.0x long) - Pipe dynamic write multiplier through dispatcher to CostGuard - Add TokenUsage.cache_creation_input_tokens field - Add tests for Long TTL injection, 5m and 1h write surcharges - Document ANTHROPIC_CACHE_RETENTION in .env.example * docs: fix stale cache_retention field comment * fix: resolve CI failures after upstream merge - Add missing cost_per_token arg to cache test callsites - Apply cargo fmt to long lines in tests and tracing macros * fix: address Copilot review feedback - Use saturating_add for cache token sum to prevent u32 overflow - Tighten supports_prompt_cache to explicitly match claude-3+/claude-4+ and named families (claude-sonnet/claude-opus/claude-haiku) * fix: adapt prompt caching to registry architecture and add missing cache fields - Resolve merge conflicts: adapt CacheRetention and cache injection to the declarative provider registry (RegistryProviderConfig replaces AnthropicDirectConfig) - Parse ANTHROPIC_CACHE_RETENTION env var in create_anthropic_from_registry() - Use Anthropic automatic caching via top-level cache_control in additional_params (rig-core #[serde(flatten)] places it at request root) - Add cache_read/creation_input_tokens fields to all mock LlmProviders added on main after PR #291 branched (response_cache, dispatcher, provider_chaos, trace_llm) - Suppress clippy::too_many_arguments on record_llm_call and build_rig_request - Add regression tests for cache injection (short/long/none) and cache_write_multiplier values Co-Authored-By: Canvinus <[email protected]> * fix: delegate cache_write_multiplier through provider wrappers and make cache_read_discount configurable The 6 decorator providers (Retry, CircuitBreaker, Failover, SmartRouting, CachedProvider, RecordingLlm) did not delegate cache_write_multiplier() to their inner provider, causing it to always return 1.0 instead of the actual 1.25x/2.0x from RigAdapter. This fix adds delegation for both cache_write_multiplier() and the new cache_read_discount() method. Also makes the cache read discount per-provider instead of hardcoding Anthropic's 90% discount (÷10). OpenAI uses 50% (÷2), so the discount is now returned by each provider via the LlmProvider trait. Addresses review feedback on PR #660. Co-Authored-By: Claude Opus 4.6 <[email protected]> * style: cargo fmt Co-Authored-By: Claude Opus 4.6 <[email protected]> * test: add CacheRetention FromStr/Display unit tests Tests cover primary values, aliases (off/disabled/5m/ephemeral/1h), case-insensitivity, invalid input error, and Display round-trip. Addresses Copilot review feedback on PR #660. Co-Authored-By: Claude Opus 4.6 <[email protected]> --------- Co-authored-by: Andrey <[email protected]> Co-authored-by: Andrey Gruzdev <[email protected]> Co-authored-by: Claude Opus 4.6 <[email protected]>
2026-08-31 16:49:34 +00:00 · 2026-03-07 09:10:05 +00:00
parent 633b234e44
commit 424a0366a9
22 changed files with 831 additions and 18 deletions
@@ -9,6 +9,50 @@ use crate::llm::registry::{ProviderProtocol, ProviderRegistry};
 use crate::llm::session::SessionConfig;
 use crate::settings::Settings;

+/// Prompt cache retention policy for Anthropic.
+///
+/// Controls Anthropic's automatic prompt caching via a top-level
+/// `cache_control` field injected through rig-core's `additional_params`.
+/// - `None` — caching disabled, no `cache_control` injected.
+/// - `Short` — 5-minute TTL (default), `{"type": "ephemeral"}`, 1.25× write surcharge.
+/// - `Long` — 1-hour TTL, `{"type": "ephemeral", "ttl": "1h"}`, 2× write surcharge.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum CacheRetention {
+    /// No prompt caching.
+    None,
+    /// 5-minute TTL (default). Write cost: 1.25× base input.
+    #[default]
+    Short,
+    /// 1-hour TTL. Write cost: 2× base input.
+    Long,
+}
+
+impl std::str::FromStr for CacheRetention {
+    type Err = String;
+
+    fn from_str(s: &str) -> Result<Self, Self::Err> {
+        match s.to_lowercase().as_str() {
+            "none" | "off" | "disabled" => Ok(Self::None),
+            "short" | "5m" | "ephemeral" => Ok(Self::Short),
+            "long" | "1h" => Ok(Self::Long),
+            _ => Err(format!(
+                "invalid cache retention '{}', expected one of: none, short, long",
+                s
+            )),
+        }
+    }
+}
+
+impl std::fmt::Display for CacheRetention {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        match self {
+            Self::None => write!(f, "none"),
+            Self::Short => write!(f, "short"),
+            Self::Long => write!(f, "long"),
+        }
+    }
+}
+
 /// Resolved configuration for a registry-based provider.
 ///
 /// This single struct replaces what used to be five separate config types
@@ -755,4 +799,86 @@ mod tests {
            std::env::remove_var("LLM_BACKEND");
        }
    }
+
+    #[test]
+    fn cache_retention_from_str_primary_values() {
+        assert_eq!(
+            "none".parse::<CacheRetention>().unwrap(),
+            CacheRetention::None
+        );
+        assert_eq!(
+            "short".parse::<CacheRetention>().unwrap(),
+            CacheRetention::Short
+        );
+        assert_eq!(
+            "long".parse::<CacheRetention>().unwrap(),
+            CacheRetention::Long
+        );
+    }
+
+    #[test]
+    fn cache_retention_from_str_aliases() {
+        assert_eq!(
+            "off".parse::<CacheRetention>().unwrap(),
+            CacheRetention::None
+        );
+        assert_eq!(
+            "disabled".parse::<CacheRetention>().unwrap(),
+            CacheRetention::None
+        );
+        assert_eq!(
+            "5m".parse::<CacheRetention>().unwrap(),
+            CacheRetention::Short
+        );
+        assert_eq!(
+            "ephemeral".parse::<CacheRetention>().unwrap(),
+            CacheRetention::Short
+        );
+        assert_eq!(
+            "1h".parse::<CacheRetention>().unwrap(),
+            CacheRetention::Long
+        );
+    }
+
+    #[test]
+    fn cache_retention_from_str_case_insensitive() {
+        assert_eq!(
+            "NONE".parse::<CacheRetention>().unwrap(),
+            CacheRetention::None
+        );
+        assert_eq!(
+            "Short".parse::<CacheRetention>().unwrap(),
+            CacheRetention::Short
+        );
+        assert_eq!(
+            "LONG".parse::<CacheRetention>().unwrap(),
+            CacheRetention::Long
+        );
+        assert_eq!(
+            "Ephemeral".parse::<CacheRetention>().unwrap(),
+            CacheRetention::Short
+        );
+    }
+
+    #[test]
+    fn cache_retention_from_str_invalid() {
+        let err = "bogus".parse::<CacheRetention>().unwrap_err();
+        assert!(
+            err.contains("bogus"),
+            "error should mention the invalid value"
+        );
+    }
+
+    #[test]
+    fn cache_retention_display_round_trip() {
+        for variant in [
+            CacheRetention::None,
+            CacheRetention::Short,
+            CacheRetention::Long,
+        ] {
+            let s = variant.to_string();
+            let parsed: CacheRetention = s.parse().unwrap();
+            assert_eq!(parsed, variant, "round-trip failed for {s}");
+        }
+    }
 }
@@ -36,7 +36,7 @@ pub use self::database::{DatabaseBackend, DatabaseConfig, SslMode, default_libsq
 pub use self::embeddings::EmbeddingsConfig;
 pub use self::heartbeat::HeartbeatConfig;
 pub use self::hygiene::HygieneConfig;
-pub use self::llm::{LlmConfig, NearAiConfig, RegistryProviderConfig};
+pub use self::llm::{CacheRetention, LlmConfig, NearAiConfig, RegistryProviderConfig};
 pub use self::routines::RoutineConfig;
 pub use self::safety::SafetyConfig;
 pub use self::sandbox::{ClaudeCodeConfig, SandboxModeConfig};