mirror of
https://github.com/outbackdingo/optimclaw.git
synced 2026-09-01 00:59:33 +00:00
feat: enable Anthropic prompt caching via automatic cache_control injection (#660)
* feat(llm): add Anthropic prompt caching and cache token tracking - Inject cache_control via additional_params for Claude models in rig_adapter - Add cache_read_input_tokens and cache_creation_input_tokens to CompletionResponse and ToolCompletionResponse - Extract cached_input_tokens from rig-core unified Usage - Add is_anthropic_model() detection helper with provider prefix support - Log prompt cache hits at debug level (consistent with response_cache) - Add 7 unit tests for cache injection and model detection - Update all mock providers and test fixtures with new fields * feat(cost): apply 90% cache discount to prompt-cached tokens in CostGuard - Add cache_read_input_tokens to TokenUsage so cache counts flow from CompletionResponse through the reasoning layer to the dispatcher - Update CostGuard::record_llm_call() to accept cache_read_input_tokens: cached tokens are billed at 10% of the normal input rate - Thread cache_read_input_tokens from dispatcher into CostGuard - Add test_cache_discount_reduces_cost verifying exact savings match 90% of input cost for fully-cached requests - Update all existing test callers with zero-cache parameter * refactor(cache): scope cache_control to Anthropic backend and validate model support - Replace model-name-based is_anthropic_model() with explicit enable_prompt_cache flag on RigAdapter, set only for the direct Anthropic backend via with_prompt_cache(true) - Add supports_prompt_cache() to validate model names per Anthropic docs: only Claude 3+ models support caching; claude-2 and claude-instant are excluded to prevent 400 errors - Warn when caching is enabled but model does not support it - Replace is_anthropic_model tests with flag-based and model validation tests * fix(cache): validate model at construction and propagate cache metrics through proxy - Move supports_prompt_cache() check into with_prompt_cache() so unsupported models are detected once at construction, not per request - Add cache_read_input_tokens and cache_creation_input_tokens to ProxyCompletionResponse and ProxyToolCompletionResponse with serde(default) for backward compatibility - Pass cache metrics through orchestrator proxy instead of zeroing - Use claude-opus-4-6 in cache discount test to match Anthropic semantics * feat(llm): add configurable cache retention with write surcharge - Add CacheRetention enum (none/short/long) to AnthropicDirectConfig - Parse ANTHROPIC_CACHE_RETENTION env var (default: short) - Inject TTL-aware cache_control (short=5m ephemeral, long=1h) - Extract cache_creation_input_tokens from raw Anthropic response - Add cache_write_multiplier() to LlmProvider trait (1.25x short, 2.0x long) - Pipe dynamic write multiplier through dispatcher to CostGuard - Add TokenUsage.cache_creation_input_tokens field - Add tests for Long TTL injection, 5m and 1h write surcharges - Document ANTHROPIC_CACHE_RETENTION in .env.example * docs: fix stale cache_retention field comment * fix: resolve CI failures after upstream merge - Add missing cost_per_token arg to cache test callsites - Apply cargo fmt to long lines in tests and tracing macros * fix: address Copilot review feedback - Use saturating_add for cache token sum to prevent u32 overflow - Tighten supports_prompt_cache to explicitly match claude-3+/claude-4+ and named families (claude-sonnet/claude-opus/claude-haiku) * fix: adapt prompt caching to registry architecture and add missing cache fields - Resolve merge conflicts: adapt CacheRetention and cache injection to the declarative provider registry (RegistryProviderConfig replaces AnthropicDirectConfig) - Parse ANTHROPIC_CACHE_RETENTION env var in create_anthropic_from_registry() - Use Anthropic automatic caching via top-level cache_control in additional_params (rig-core #[serde(flatten)] places it at request root) - Add cache_read/creation_input_tokens fields to all mock LlmProviders added on main after PR #291 branched (response_cache, dispatcher, provider_chaos, trace_llm) - Suppress clippy::too_many_arguments on record_llm_call and build_rig_request - Add regression tests for cache injection (short/long/none) and cache_write_multiplier values Co-Authored-By: Canvinus <[email protected]> * fix: delegate cache_write_multiplier through provider wrappers and make cache_read_discount configurable The 6 decorator providers (Retry, CircuitBreaker, Failover, SmartRouting, CachedProvider, RecordingLlm) did not delegate cache_write_multiplier() to their inner provider, causing it to always return 1.0 instead of the actual 1.25x/2.0x from RigAdapter. This fix adds delegation for both cache_write_multiplier() and the new cache_read_discount() method. Also makes the cache read discount per-provider instead of hardcoding Anthropic's 90% discount (÷10). OpenAI uses 50% (÷2), so the discount is now returned by each provider via the LlmProvider trait. Addresses review feedback on PR #660. Co-Authored-By: Claude Opus 4.6 <[email protected]> * style: cargo fmt Co-Authored-By: Claude Opus 4.6 <[email protected]> * test: add CacheRetention FromStr/Display unit tests Tests cover primary values, aliases (off/disabled/5m/ephemeral/1h), case-insensitivity, invalid input error, and Display round-trip. Addresses Copilot review feedback on PR #660. Co-Authored-By: Claude Opus 4.6 <[email protected]> --------- Co-authored-by: Andrey <[email protected]> Co-authored-by: Andrey Gruzdev <[email protected]> Co-authored-by: Claude Opus 4.6 <[email protected]>
This commit is contained in:
co-authored by
Andrey
Andrey Gruzdev
Claude Opus 4.6
parent
633b234e44
commit
424a0366a9
@@ -245,6 +245,14 @@ impl LlmProvider for CircuitBreakerProvider {
|
||||
self.inner.cost_per_token()
|
||||
}
|
||||
|
||||
fn cache_write_multiplier(&self) -> Decimal {
|
||||
self.inner.cache_write_multiplier()
|
||||
}
|
||||
|
||||
fn cache_read_discount(&self) -> Decimal {
|
||||
self.inner.cache_read_discount()
|
||||
}
|
||||
|
||||
async fn complete(&self, request: CompletionRequest) -> Result<CompletionResponse, LlmError> {
|
||||
self.check_allowed().await?;
|
||||
match self.inner.complete(request).await {
|
||||
|
||||
@@ -296,6 +296,14 @@ impl LlmProvider for FailoverProvider {
|
||||
self.providers[self.last_used.load(Ordering::Relaxed)].cost_per_token()
|
||||
}
|
||||
|
||||
fn cache_write_multiplier(&self) -> Decimal {
|
||||
self.providers[self.last_used.load(Ordering::Relaxed)].cache_write_multiplier()
|
||||
}
|
||||
|
||||
fn cache_read_discount(&self) -> Decimal {
|
||||
self.providers[self.last_used.load(Ordering::Relaxed)].cache_read_discount()
|
||||
}
|
||||
|
||||
async fn complete(&self, request: CompletionRequest) -> Result<CompletionResponse, LlmError> {
|
||||
let (provider_idx, response) = self
|
||||
.try_providers(|provider| {
|
||||
@@ -404,6 +412,8 @@ mod tests {
|
||||
input_tokens: 10,
|
||||
output_tokens: 5,
|
||||
finish_reason: FinishReason::Stop,
|
||||
cache_read_input_tokens: 0,
|
||||
cache_creation_input_tokens: 0,
|
||||
}))),
|
||||
tool_complete_result: Mutex::new(Some(Ok(ToolCompletionResponse {
|
||||
content: Some(content.to_string()),
|
||||
@@ -411,6 +421,8 @@ mod tests {
|
||||
input_tokens: 10,
|
||||
output_tokens: 5,
|
||||
finish_reason: FinishReason::Stop,
|
||||
cache_read_input_tokens: 0,
|
||||
cache_creation_input_tokens: 0,
|
||||
}))),
|
||||
}
|
||||
}
|
||||
@@ -792,6 +804,8 @@ mod tests {
|
||||
input_tokens: 10,
|
||||
output_tokens: 5,
|
||||
finish_reason: FinishReason::Stop,
|
||||
cache_read_input_tokens: 0,
|
||||
cache_creation_input_tokens: 0,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -817,6 +831,8 @@ mod tests {
|
||||
input_tokens: 10,
|
||||
output_tokens: 5,
|
||||
finish_reason: FinishReason::Stop,
|
||||
cache_read_input_tokens: 0,
|
||||
cache_creation_input_tokens: 0,
|
||||
})
|
||||
}
|
||||
|
||||
|
||||
+29
-1
@@ -177,6 +177,8 @@ fn create_openai_compat_from_registry(
|
||||
fn create_anthropic_from_registry(
|
||||
config: &RegistryProviderConfig,
|
||||
) -> Result<Arc<dyn LlmProvider>, LlmError> {
|
||||
use crate::config::CacheRetention;
|
||||
use crate::config::helpers::optional_env;
|
||||
use rig::providers::anthropic;
|
||||
|
||||
let api_key = config
|
||||
@@ -200,8 +202,32 @@ fn create_anthropic_from_registry(
|
||||
reason: format!("Failed to create Anthropic client: {e}"),
|
||||
})?;
|
||||
|
||||
// Resolve prompt cache retention from env (default: Short).
|
||||
// Injects top-level cache_control via additional_params for Anthropic
|
||||
// automatic caching (the API auto-places the breakpoint at the last
|
||||
// cacheable block).
|
||||
let cache_retention: CacheRetention = optional_env("ANTHROPIC_CACHE_RETENTION")
|
||||
.ok()
|
||||
.flatten()
|
||||
.and_then(|val| match val.parse::<CacheRetention>() {
|
||||
Ok(r) => Some(r),
|
||||
Err(e) => {
|
||||
tracing::warn!("Invalid ANTHROPIC_CACHE_RETENTION: {e}; defaulting to short");
|
||||
None
|
||||
}
|
||||
})
|
||||
.unwrap_or_default();
|
||||
|
||||
let model = client.completion_model(&config.model);
|
||||
|
||||
if cache_retention != CacheRetention::None {
|
||||
tracing::info!(
|
||||
model = %config.model,
|
||||
retention = %cache_retention,
|
||||
"Anthropic automatic prompt caching enabled"
|
||||
);
|
||||
}
|
||||
|
||||
tracing::info!(
|
||||
provider = %config.provider_id,
|
||||
model = %config.model,
|
||||
@@ -209,7 +235,9 @@ fn create_anthropic_from_registry(
|
||||
"Using Anthropic provider"
|
||||
);
|
||||
|
||||
Ok(Arc::new(RigAdapter::new(model, &config.model)))
|
||||
Ok(Arc::new(
|
||||
RigAdapter::new(model, &config.model).with_cache_retention(cache_retention),
|
||||
))
|
||||
}
|
||||
|
||||
fn create_ollama_from_registry(
|
||||
|
||||
@@ -499,6 +499,8 @@ impl LlmProvider for NearAiChatProvider {
|
||||
finish_reason,
|
||||
input_tokens,
|
||||
output_tokens,
|
||||
cache_read_input_tokens: 0,
|
||||
cache_creation_input_tokens: 0,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -604,6 +606,8 @@ impl LlmProvider for NearAiChatProvider {
|
||||
finish_reason,
|
||||
input_tokens,
|
||||
output_tokens,
|
||||
cache_read_input_tokens: 0,
|
||||
cache_creation_input_tokens: 0,
|
||||
})
|
||||
}
|
||||
|
||||
|
||||
@@ -153,6 +153,12 @@ pub struct CompletionResponse {
|
||||
pub input_tokens: u32,
|
||||
pub output_tokens: u32,
|
||||
pub finish_reason: FinishReason,
|
||||
/// Tokens read from the provider's server-side prompt cache (Anthropic).
|
||||
/// Zero when caching is not supported or on a cache miss.
|
||||
pub cache_read_input_tokens: u32,
|
||||
/// Tokens written to the provider's server-side prompt cache (Anthropic).
|
||||
/// Zero when caching is not supported or no new prefix was cached.
|
||||
pub cache_creation_input_tokens: u32,
|
||||
}
|
||||
|
||||
/// Why the completion finished.
|
||||
@@ -254,6 +260,10 @@ pub struct ToolCompletionResponse {
|
||||
pub input_tokens: u32,
|
||||
pub output_tokens: u32,
|
||||
pub finish_reason: FinishReason,
|
||||
/// Tokens read from the provider's server-side prompt cache (Anthropic).
|
||||
pub cache_read_input_tokens: u32,
|
||||
/// Tokens written to the provider's server-side prompt cache (Anthropic).
|
||||
pub cache_creation_input_tokens: u32,
|
||||
}
|
||||
|
||||
/// Metadata about a model returned by the provider's API.
|
||||
@@ -328,6 +338,23 @@ pub trait LlmProvider: Send + Sync {
|
||||
let (input_cost, output_cost) = self.cost_per_token();
|
||||
input_cost * Decimal::from(input_tokens) + output_cost * Decimal::from(output_tokens)
|
||||
}
|
||||
|
||||
/// Cost multiplier for cache-creation tokens (Anthropic prompt caching).
|
||||
///
|
||||
/// Returns `1.0` by default (no surcharge). Anthropic providers return
|
||||
/// `1.25` for 5-minute TTL or `2.0` for 1-hour TTL.
|
||||
fn cache_write_multiplier(&self) -> Decimal {
|
||||
Decimal::ONE
|
||||
}
|
||||
|
||||
/// Discount divisor for cache-read tokens.
|
||||
///
|
||||
/// Cached-read cost = `input_rate / cache_read_discount()`.
|
||||
/// Returns `1` by default (no discount). Anthropic returns `10` (90% off),
|
||||
/// OpenAI would return `2` (50% off).
|
||||
fn cache_read_discount(&self) -> Decimal {
|
||||
Decimal::ONE
|
||||
}
|
||||
}
|
||||
|
||||
/// Sanitize a message list to ensure tool_use / tool_result integrity.
|
||||
|
||||
@@ -292,6 +292,10 @@ pub struct ToolSelection {
|
||||
pub struct TokenUsage {
|
||||
pub input_tokens: u32,
|
||||
pub output_tokens: u32,
|
||||
/// Tokens served from the provider's server-side prompt cache (Anthropic).
|
||||
pub cache_read_input_tokens: u32,
|
||||
/// Tokens written to the provider's prompt cache (Anthropic).
|
||||
pub cache_creation_input_tokens: u32,
|
||||
}
|
||||
|
||||
impl TokenUsage {
|
||||
@@ -434,6 +438,8 @@ impl Reasoning {
|
||||
let usage = TokenUsage {
|
||||
input_tokens: response.input_tokens,
|
||||
output_tokens: response.output_tokens,
|
||||
cache_read_input_tokens: response.cache_read_input_tokens,
|
||||
cache_creation_input_tokens: response.cache_creation_input_tokens,
|
||||
};
|
||||
Ok((clean_response(&response.content), usage))
|
||||
}
|
||||
@@ -612,6 +618,8 @@ Respond in JSON format:
|
||||
let usage = TokenUsage {
|
||||
input_tokens: response.input_tokens,
|
||||
output_tokens: response.output_tokens,
|
||||
cache_read_input_tokens: response.cache_read_input_tokens,
|
||||
cache_creation_input_tokens: response.cache_creation_input_tokens,
|
||||
};
|
||||
|
||||
// If there were tool calls, return them for execution
|
||||
@@ -690,6 +698,8 @@ Respond in JSON format:
|
||||
usage: TokenUsage {
|
||||
input_tokens: response.input_tokens,
|
||||
output_tokens: response.output_tokens,
|
||||
cache_read_input_tokens: response.cache_read_input_tokens,
|
||||
cache_creation_input_tokens: response.cache_creation_input_tokens,
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
@@ -461,6 +461,14 @@ impl LlmProvider for RecordingLlm {
|
||||
self.inner.cost_per_token()
|
||||
}
|
||||
|
||||
fn cache_write_multiplier(&self) -> Decimal {
|
||||
self.inner.cache_write_multiplier()
|
||||
}
|
||||
|
||||
fn cache_read_discount(&self) -> Decimal {
|
||||
self.inner.cache_read_discount()
|
||||
}
|
||||
|
||||
async fn complete(&self, request: CompletionRequest) -> Result<CompletionResponse, LlmError> {
|
||||
let (hint, tool_results) = self.capture_new_messages(&request.messages).await;
|
||||
let response = self.inner.complete(request).await?;
|
||||
|
||||
@@ -181,6 +181,14 @@ impl LlmProvider for CachedProvider {
|
||||
self.inner.cost_per_token()
|
||||
}
|
||||
|
||||
fn cache_write_multiplier(&self) -> Decimal {
|
||||
self.inner.cache_write_multiplier()
|
||||
}
|
||||
|
||||
fn cache_read_discount(&self) -> Decimal {
|
||||
self.inner.cache_read_discount()
|
||||
}
|
||||
|
||||
async fn complete(&self, request: CompletionRequest) -> Result<CompletionResponse, LlmError> {
|
||||
let effective_model = self.inner.effective_model_name(request.model.as_deref());
|
||||
let key = cache_key(&effective_model, &request);
|
||||
@@ -352,6 +360,8 @@ mod tests {
|
||||
input_tokens: 1,
|
||||
output_tokens: 1,
|
||||
finish_reason: FinishReason::Stop,
|
||||
cache_read_input_tokens: 0,
|
||||
cache_creation_input_tokens: 0,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -365,6 +375,8 @@ mod tests {
|
||||
input_tokens: 1,
|
||||
output_tokens: 1,
|
||||
finish_reason: FinishReason::Stop,
|
||||
cache_read_input_tokens: 0,
|
||||
cache_creation_input_tokens: 0,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
@@ -109,6 +109,14 @@ impl LlmProvider for RetryProvider {
|
||||
self.inner.cost_per_token()
|
||||
}
|
||||
|
||||
fn cache_write_multiplier(&self) -> Decimal {
|
||||
self.inner.cache_write_multiplier()
|
||||
}
|
||||
|
||||
fn cache_read_discount(&self) -> Decimal {
|
||||
self.inner.cache_read_discount()
|
||||
}
|
||||
|
||||
async fn complete(&self, request: CompletionRequest) -> Result<CompletionResponse, LlmError> {
|
||||
let mut last_error: Option<LlmError> = None;
|
||||
|
||||
|
||||
+255
-5
@@ -3,6 +3,7 @@
|
||||
//! This lets us use any rig-core provider (OpenAI, Anthropic, Ollama, etc.) as an
|
||||
//! `Arc<dyn LlmProvider>` without changing any of the agent, reasoning, or tool code.
|
||||
|
||||
use crate::config::CacheRetention;
|
||||
use async_trait::async_trait;
|
||||
use rig::OneOrMany;
|
||||
use rig::completion::{
|
||||
@@ -14,6 +15,7 @@ use rig::message::{
|
||||
ToolResultContent, UserContent,
|
||||
};
|
||||
use rust_decimal::Decimal;
|
||||
use rust_decimal_macros::dec;
|
||||
use serde::Serialize;
|
||||
use serde::de::DeserializeOwned;
|
||||
use serde_json::Value as JsonValue;
|
||||
@@ -34,6 +36,11 @@ pub struct RigAdapter<M: CompletionModel> {
|
||||
model_name: String,
|
||||
input_cost: Decimal,
|
||||
output_cost: Decimal,
|
||||
/// Prompt cache retention policy (Anthropic only).
|
||||
/// When not `CacheRetention::None`, injects top-level `cache_control`
|
||||
/// via `additional_params` for Anthropic automatic caching. Also controls
|
||||
/// the cost multiplier for cache-creation tokens.
|
||||
cache_retention: CacheRetention,
|
||||
}
|
||||
|
||||
impl<M: CompletionModel> RigAdapter<M> {
|
||||
@@ -47,8 +54,35 @@ impl<M: CompletionModel> RigAdapter<M> {
|
||||
model_name: name,
|
||||
input_cost,
|
||||
output_cost,
|
||||
cache_retention: CacheRetention::None,
|
||||
}
|
||||
}
|
||||
|
||||
/// Set Anthropic prompt cache retention policy.
|
||||
///
|
||||
/// Controls both cache injection and cost tracking:
|
||||
/// - `None` — no caching, no surcharge (1.0×).
|
||||
/// - `Short` — 5-minute TTL via `{"type": "ephemeral"}`, 1.25× write surcharge.
|
||||
/// - `Long` — 1-hour TTL via `{"type": "ephemeral", "ttl": "1h"}`, 2.0× write surcharge.
|
||||
///
|
||||
/// Cache injection uses Anthropic's **automatic caching** — a top-level
|
||||
/// `cache_control` field in `additional_params` that gets `#[serde(flatten)]`'d
|
||||
/// into the request body by rig-core.
|
||||
///
|
||||
/// If the configured model does not support caching (e.g. claude-2),
|
||||
/// a warning is logged once at construction and caching is disabled.
|
||||
pub fn with_cache_retention(mut self, retention: CacheRetention) -> Self {
|
||||
if retention != CacheRetention::None && !supports_prompt_cache(&self.model_name) {
|
||||
tracing::warn!(
|
||||
model = %self.model_name,
|
||||
"Prompt caching requested but model does not support it; disabling"
|
||||
);
|
||||
self.cache_retention = CacheRetention::None;
|
||||
} else {
|
||||
self.cache_retention = retention;
|
||||
}
|
||||
self
|
||||
}
|
||||
}
|
||||
|
||||
// -- Type conversion helpers --
|
||||
@@ -360,7 +394,44 @@ fn saturate_u32(val: u64) -> u32 {
|
||||
val.min(u32::MAX as u64) as u32
|
||||
}
|
||||
|
||||
/// Returns `true` if the model supports Anthropic prompt caching.
|
||||
///
|
||||
/// Per Anthropic docs, only Claude 3+ models support prompt caching.
|
||||
/// Unsupported: claude-2, claude-2.1, claude-instant-*.
|
||||
fn supports_prompt_cache(name: &str) -> bool {
|
||||
let lower = name.to_lowercase();
|
||||
// Strip optional provider prefix (e.g. "anthropic/claude-...")
|
||||
let model = lower.strip_prefix("anthropic/").unwrap_or(&lower);
|
||||
// Only Claude 3+ families support prompt caching
|
||||
model.starts_with("claude-3")
|
||||
|| model.starts_with("claude-4")
|
||||
|| model.starts_with("claude-sonnet")
|
||||
|| model.starts_with("claude-opus")
|
||||
|| model.starts_with("claude-haiku")
|
||||
}
|
||||
|
||||
/// Extract `cache_creation_input_tokens` from the raw provider response.
|
||||
///
|
||||
/// Rig-core's unified `Usage` does not surface this field, but Anthropic's raw
|
||||
/// response includes it at `usage.cache_creation_input_tokens`. We serialize the
|
||||
/// raw response to JSON and attempt to read the value.
|
||||
fn extract_cache_creation<T: Serialize>(raw: &T) -> u32 {
|
||||
serde_json::to_value(raw)
|
||||
.ok()
|
||||
.and_then(|v| v.get("usage")?.get("cache_creation_input_tokens")?.as_u64())
|
||||
.map(|n| n.min(u32::MAX as u64) as u32)
|
||||
.unwrap_or(0)
|
||||
}
|
||||
|
||||
/// Build a rig-core CompletionRequest from our internal types.
|
||||
///
|
||||
/// When `cache_retention` is not `None`, injects a top-level `cache_control`
|
||||
/// field via `additional_params`. Rig-core's `AnthropicCompletionRequest`
|
||||
/// uses `#[serde(flatten)]` on `additional_params`, so the field lands at
|
||||
/// the request root — which is exactly what Anthropic's **automatic caching**
|
||||
/// expects. The API auto-places the cache breakpoint at the last cacheable
|
||||
/// block and moves it forward as conversations grow.
|
||||
#[allow(clippy::too_many_arguments)]
|
||||
fn build_rig_request(
|
||||
preamble: Option<String>,
|
||||
mut history: Vec<RigMessage>,
|
||||
@@ -368,6 +439,7 @@ fn build_rig_request(
|
||||
tool_choice: Option<RigToolChoice>,
|
||||
temperature: Option<f32>,
|
||||
max_tokens: Option<u32>,
|
||||
cache_retention: CacheRetention,
|
||||
) -> Result<RigRequest, LlmError> {
|
||||
// rig-core requires at least one message in chat_history
|
||||
if history.is_empty() {
|
||||
@@ -379,6 +451,17 @@ fn build_rig_request(
|
||||
reason: format!("Failed to build chat history: {}", e),
|
||||
})?;
|
||||
|
||||
// Inject top-level cache_control for Anthropic automatic prompt caching.
|
||||
let additional_params = match cache_retention {
|
||||
CacheRetention::None => None,
|
||||
CacheRetention::Short => Some(serde_json::json!({
|
||||
"cache_control": {"type": "ephemeral"}
|
||||
})),
|
||||
CacheRetention::Long => Some(serde_json::json!({
|
||||
"cache_control": {"type": "ephemeral", "ttl": "1h"}
|
||||
})),
|
||||
};
|
||||
|
||||
Ok(RigRequest {
|
||||
preamble,
|
||||
chat_history,
|
||||
@@ -387,7 +470,7 @@ fn build_rig_request(
|
||||
temperature: temperature.map(|t| t as f64),
|
||||
max_tokens: max_tokens.map(|t| t as u64),
|
||||
tool_choice,
|
||||
additional_params: None,
|
||||
additional_params,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -405,6 +488,22 @@ where
|
||||
(self.input_cost, self.output_cost)
|
||||
}
|
||||
|
||||
fn cache_write_multiplier(&self) -> Decimal {
|
||||
match self.cache_retention {
|
||||
CacheRetention::None => Decimal::ONE,
|
||||
CacheRetention::Short => Decimal::new(125, 2), // 1.25× (125% of input rate)
|
||||
CacheRetention::Long => Decimal::TWO, // 2.0× (200% of input rate)
|
||||
}
|
||||
}
|
||||
|
||||
fn cache_read_discount(&self) -> Decimal {
|
||||
if self.cache_retention != CacheRetention::None {
|
||||
dec!(10) // Anthropic: 90% discount (cost = input_rate / 10)
|
||||
} else {
|
||||
Decimal::ONE
|
||||
}
|
||||
}
|
||||
|
||||
async fn complete(&self, request: CompletionRequest) -> Result<CompletionResponse, LlmError> {
|
||||
if let Some(requested_model) = request.model.as_deref()
|
||||
&& requested_model != self.model_name.as_str()
|
||||
@@ -427,6 +526,7 @@ where
|
||||
None,
|
||||
request.temperature,
|
||||
request.max_tokens,
|
||||
self.cache_retention,
|
||||
)?;
|
||||
|
||||
let response =
|
||||
@@ -440,12 +540,26 @@ where
|
||||
|
||||
let (text, _tool_calls, finish) = extract_response(&response.choice, &response.usage);
|
||||
|
||||
Ok(CompletionResponse {
|
||||
let resp = CompletionResponse {
|
||||
content: text.unwrap_or_default(),
|
||||
input_tokens: saturate_u32(response.usage.input_tokens),
|
||||
output_tokens: saturate_u32(response.usage.output_tokens),
|
||||
finish_reason: finish,
|
||||
})
|
||||
cache_read_input_tokens: saturate_u32(response.usage.cached_input_tokens),
|
||||
cache_creation_input_tokens: extract_cache_creation(&response.raw_response),
|
||||
};
|
||||
|
||||
if resp.cache_read_input_tokens > 0 {
|
||||
tracing::debug!(
|
||||
model = %self.model_name,
|
||||
input = resp.input_tokens,
|
||||
output = resp.output_tokens,
|
||||
cache_read = resp.cache_read_input_tokens,
|
||||
"prompt cache hit",
|
||||
);
|
||||
}
|
||||
|
||||
Ok(resp)
|
||||
}
|
||||
|
||||
async fn complete_with_tools(
|
||||
@@ -478,6 +592,7 @@ where
|
||||
tool_choice,
|
||||
request.temperature,
|
||||
request.max_tokens,
|
||||
self.cache_retention,
|
||||
)?;
|
||||
|
||||
let response =
|
||||
@@ -504,13 +619,27 @@ where
|
||||
}
|
||||
}
|
||||
|
||||
Ok(ToolCompletionResponse {
|
||||
let resp = ToolCompletionResponse {
|
||||
content: text,
|
||||
tool_calls,
|
||||
input_tokens: saturate_u32(response.usage.input_tokens),
|
||||
output_tokens: saturate_u32(response.usage.output_tokens),
|
||||
finish_reason: finish,
|
||||
})
|
||||
cache_read_input_tokens: saturate_u32(response.usage.cached_input_tokens),
|
||||
cache_creation_input_tokens: extract_cache_creation(&response.raw_response),
|
||||
};
|
||||
|
||||
if resp.cache_read_input_tokens > 0 {
|
||||
tracing::debug!(
|
||||
model = %self.model_name,
|
||||
input = resp.input_tokens,
|
||||
output = resp.output_tokens,
|
||||
cache_read = resp.cache_read_input_tokens,
|
||||
"prompt cache hit",
|
||||
);
|
||||
}
|
||||
|
||||
Ok(resp)
|
||||
}
|
||||
|
||||
fn active_model_name(&self) -> String {
|
||||
@@ -869,4 +998,125 @@ mod tests {
|
||||
let known = HashSet::from(["echo".to_string()]);
|
||||
assert_eq!(normalize_tool_name("other_tool", &known), "other_tool");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_build_rig_request_injects_cache_control_short() {
|
||||
let req = build_rig_request(
|
||||
Some("You are helpful.".to_string()),
|
||||
vec![RigMessage::user("Hello")],
|
||||
Vec::new(),
|
||||
None,
|
||||
None,
|
||||
None,
|
||||
CacheRetention::Short,
|
||||
)
|
||||
.unwrap();
|
||||
|
||||
let params = req
|
||||
.additional_params
|
||||
.expect("should have additional_params for Short retention");
|
||||
assert_eq!(params["cache_control"]["type"], "ephemeral");
|
||||
assert!(
|
||||
params["cache_control"].get("ttl").is_none(),
|
||||
"Short retention should not include ttl"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_build_rig_request_injects_cache_control_long() {
|
||||
let req = build_rig_request(
|
||||
Some("You are helpful.".to_string()),
|
||||
vec![RigMessage::user("Hello")],
|
||||
Vec::new(),
|
||||
None,
|
||||
None,
|
||||
None,
|
||||
CacheRetention::Long,
|
||||
)
|
||||
.unwrap();
|
||||
|
||||
let params = req
|
||||
.additional_params
|
||||
.expect("should have additional_params for Long retention");
|
||||
assert_eq!(params["cache_control"]["type"], "ephemeral");
|
||||
assert_eq!(params["cache_control"]["ttl"], "1h");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_build_rig_request_no_cache_control_when_none() {
|
||||
let req = build_rig_request(
|
||||
Some("You are helpful.".to_string()),
|
||||
vec![RigMessage::user("Hello")],
|
||||
Vec::new(),
|
||||
None,
|
||||
None,
|
||||
None,
|
||||
CacheRetention::None,
|
||||
)
|
||||
.unwrap();
|
||||
|
||||
assert!(
|
||||
req.additional_params.is_none(),
|
||||
"additional_params should be None when cache is disabled"
|
||||
);
|
||||
}
|
||||
|
||||
/// Verify that the multiplier match arms in `RigAdapter::cache_write_multiplier`
|
||||
/// produce the expected values. We use a standalone helper because constructing
|
||||
/// a real `RigAdapter` requires a rig `Model` (which needs network/provider setup).
|
||||
/// The helper mirrors the same match expression — if the impl drifts, the
|
||||
/// `test_build_rig_request_*` tests will still catch regressions end-to-end.
|
||||
#[test]
|
||||
fn test_cache_write_multiplier_values() {
|
||||
use rust_decimal::Decimal;
|
||||
// None → 1.0× (no surcharge)
|
||||
assert_eq!(
|
||||
cache_write_multiplier_for(CacheRetention::None),
|
||||
Decimal::ONE
|
||||
);
|
||||
// Short → 1.25× (25% surcharge)
|
||||
assert_eq!(
|
||||
cache_write_multiplier_for(CacheRetention::Short),
|
||||
Decimal::new(125, 2)
|
||||
);
|
||||
// Long → 2.0× (100% surcharge)
|
||||
assert_eq!(
|
||||
cache_write_multiplier_for(CacheRetention::Long),
|
||||
Decimal::TWO
|
||||
);
|
||||
}
|
||||
|
||||
fn cache_write_multiplier_for(retention: CacheRetention) -> rust_decimal::Decimal {
|
||||
match retention {
|
||||
CacheRetention::None => rust_decimal::Decimal::ONE,
|
||||
CacheRetention::Short => rust_decimal::Decimal::new(125, 2),
|
||||
CacheRetention::Long => rust_decimal::Decimal::TWO,
|
||||
}
|
||||
}
|
||||
|
||||
// -- supports_prompt_cache tests --
|
||||
|
||||
#[test]
|
||||
fn test_supports_prompt_cache_supported_models() {
|
||||
// All Claude 3+ models per Anthropic docs
|
||||
assert!(supports_prompt_cache("claude-opus-4-6"));
|
||||
assert!(supports_prompt_cache("claude-sonnet-4-6"));
|
||||
assert!(supports_prompt_cache("claude-sonnet-4"));
|
||||
assert!(supports_prompt_cache("claude-haiku-4-5"));
|
||||
assert!(supports_prompt_cache("claude-3-5-sonnet-20241022"));
|
||||
assert!(supports_prompt_cache("claude-haiku-3"));
|
||||
assert!(supports_prompt_cache("Claude-Opus-4-5")); // case-insensitive
|
||||
assert!(supports_prompt_cache("anthropic/claude-sonnet-4-6")); // provider prefix
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_supports_prompt_cache_unsupported_models() {
|
||||
// Legacy Claude models that predate caching
|
||||
assert!(!supports_prompt_cache("claude-2"));
|
||||
assert!(!supports_prompt_cache("claude-2.1"));
|
||||
assert!(!supports_prompt_cache("claude-instant-1.2"));
|
||||
// Non-Claude models
|
||||
assert!(!supports_prompt_cache("gpt-4o"));
|
||||
assert!(!supports_prompt_cache("llama3"));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -857,6 +857,14 @@ impl LlmProvider for SmartRoutingProvider {
|
||||
self.primary.cost_per_token()
|
||||
}
|
||||
|
||||
fn cache_write_multiplier(&self) -> Decimal {
|
||||
self.primary.cache_write_multiplier()
|
||||
}
|
||||
|
||||
fn cache_read_discount(&self) -> Decimal {
|
||||
self.primary.cache_read_discount()
|
||||
}
|
||||
|
||||
async fn complete(&self, request: CompletionRequest) -> Result<CompletionResponse, LlmError> {
|
||||
self.stats.total_requests.fetch_add(1, Ordering::Relaxed);
|
||||
|
||||
@@ -1471,6 +1479,8 @@ mod tests {
|
||||
input_tokens: 10,
|
||||
output_tokens: 5,
|
||||
finish_reason: crate::llm::FinishReason::Stop,
|
||||
cache_read_input_tokens: 0,
|
||||
cache_creation_input_tokens: 0,
|
||||
};
|
||||
assert!(SmartRoutingProvider::response_is_uncertain(&response));
|
||||
}
|
||||
@@ -1482,6 +1492,8 @@ mod tests {
|
||||
input_tokens: 10,
|
||||
output_tokens: 0,
|
||||
finish_reason: crate::llm::FinishReason::Stop,
|
||||
cache_read_input_tokens: 0,
|
||||
cache_creation_input_tokens: 0,
|
||||
};
|
||||
assert!(SmartRoutingProvider::response_is_uncertain(&response));
|
||||
}
|
||||
@@ -1493,6 +1505,8 @@ mod tests {
|
||||
input_tokens: 10,
|
||||
output_tokens: 1,
|
||||
finish_reason: crate::llm::FinishReason::Stop,
|
||||
cache_read_input_tokens: 0,
|
||||
cache_creation_input_tokens: 0,
|
||||
};
|
||||
assert!(!SmartRoutingProvider::response_is_uncertain(&response));
|
||||
}
|
||||
@@ -1505,6 +1519,8 @@ mod tests {
|
||||
input_tokens: 10,
|
||||
output_tokens: 20,
|
||||
finish_reason: crate::llm::FinishReason::Stop,
|
||||
cache_read_input_tokens: 0,
|
||||
cache_creation_input_tokens: 0,
|
||||
};
|
||||
assert!(!SmartRoutingProvider::response_is_uncertain(&response));
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user