Files
optimclaw/src/worker/api.rs
T
424a0366a9 feat: enable Anthropic prompt caching via automatic cache_control injection (#660)
* feat(llm): add Anthropic prompt caching and cache token tracking

- Inject cache_control via additional_params for Claude models in rig_adapter
- Add cache_read_input_tokens and cache_creation_input_tokens to
  CompletionResponse and ToolCompletionResponse
- Extract cached_input_tokens from rig-core unified Usage
- Add is_anthropic_model() detection helper with provider prefix support
- Log prompt cache hits at debug level (consistent with response_cache)
- Add 7 unit tests for cache injection and model detection
- Update all mock providers and test fixtures with new fields

* feat(cost): apply 90% cache discount to prompt-cached tokens in CostGuard

- Add cache_read_input_tokens to TokenUsage so cache counts flow from
  CompletionResponse through the reasoning layer to the dispatcher
- Update CostGuard::record_llm_call() to accept cache_read_input_tokens:
  cached tokens are billed at 10% of the normal input rate
- Thread cache_read_input_tokens from dispatcher into CostGuard
- Add test_cache_discount_reduces_cost verifying exact savings match
  90% of input cost for fully-cached requests
- Update all existing test callers with zero-cache parameter

* refactor(cache): scope cache_control to Anthropic backend and validate model support

- Replace model-name-based is_anthropic_model() with explicit
  enable_prompt_cache flag on RigAdapter, set only for the direct
  Anthropic backend via with_prompt_cache(true)
- Add supports_prompt_cache() to validate model names per Anthropic
  docs: only Claude 3+ models support caching; claude-2 and
  claude-instant are excluded to prevent 400 errors
- Warn when caching is enabled but model does not support it
- Replace is_anthropic_model tests with flag-based and model
  validation tests

* fix(cache): validate model at construction and propagate cache metrics through proxy

- Move supports_prompt_cache() check into with_prompt_cache() so
  unsupported models are detected once at construction, not per request
- Add cache_read_input_tokens and cache_creation_input_tokens to
  ProxyCompletionResponse and ProxyToolCompletionResponse with
  serde(default) for backward compatibility
- Pass cache metrics through orchestrator proxy instead of zeroing
- Use claude-opus-4-6 in cache discount test to match Anthropic
  semantics

* feat(llm): add configurable cache retention with write surcharge

- Add CacheRetention enum (none/short/long) to AnthropicDirectConfig
- Parse ANTHROPIC_CACHE_RETENTION env var (default: short)
- Inject TTL-aware cache_control (short=5m ephemeral, long=1h)
- Extract cache_creation_input_tokens from raw Anthropic response
- Add cache_write_multiplier() to LlmProvider trait (1.25x short, 2.0x long)
- Pipe dynamic write multiplier through dispatcher to CostGuard
- Add TokenUsage.cache_creation_input_tokens field
- Add tests for Long TTL injection, 5m and 1h write surcharges
- Document ANTHROPIC_CACHE_RETENTION in .env.example

* docs: fix stale cache_retention field comment

* fix: resolve CI failures after upstream merge

- Add missing cost_per_token arg to cache test callsites
- Apply cargo fmt to long lines in tests and tracing macros

* fix: address Copilot review feedback

- Use saturating_add for cache token sum to prevent u32 overflow
- Tighten supports_prompt_cache to explicitly match claude-3+/claude-4+
  and named families (claude-sonnet/claude-opus/claude-haiku)

* fix: adapt prompt caching to registry architecture and add missing cache fields

- Resolve merge conflicts: adapt CacheRetention and cache injection to
  the declarative provider registry (RegistryProviderConfig replaces
  AnthropicDirectConfig)
- Parse ANTHROPIC_CACHE_RETENTION env var in create_anthropic_from_registry()
- Use Anthropic automatic caching via top-level cache_control in
  additional_params (rig-core #[serde(flatten)] places it at request root)
- Add cache_read/creation_input_tokens fields to all mock LlmProviders
  added on main after PR #291 branched (response_cache, dispatcher,
  provider_chaos, trace_llm)
- Suppress clippy::too_many_arguments on record_llm_call and
  build_rig_request
- Add regression tests for cache injection (short/long/none) and
  cache_write_multiplier values

Co-Authored-By: Canvinus <[email protected]>

* fix: delegate cache_write_multiplier through provider wrappers and make cache_read_discount configurable

The 6 decorator providers (Retry, CircuitBreaker, Failover, SmartRouting,
CachedProvider, RecordingLlm) did not delegate cache_write_multiplier()
to their inner provider, causing it to always return 1.0 instead of the
actual 1.25x/2.0x from RigAdapter. This fix adds delegation for both
cache_write_multiplier() and the new cache_read_discount() method.

Also makes the cache read discount per-provider instead of hardcoding
Anthropic's 90% discount (÷10). OpenAI uses 50% (÷2), so the discount
is now returned by each provider via the LlmProvider trait.

Addresses review feedback on PR #660.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* style: cargo fmt

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* test: add CacheRetention FromStr/Display unit tests

Tests cover primary values, aliases (off/disabled/5m/ephemeral/1h),
case-insensitivity, invalid input error, and Display round-trip.

Addresses Copilot review feedback on PR #660.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

---------

Co-authored-by: Andrey <[email protected]>
Co-authored-by: Andrey Gruzdev <[email protected]>
Co-authored-by: Claude Opus 4.6 <[email protected]>
2026-03-07 09:10:05 +00:00

473 lines
15 KiB
Rust

//! HTTP client for worker-to-orchestrator communication.
//!
//! Every request includes a bearer token from `IRONCLAW_WORKER_TOKEN` env var.
//! The orchestrator validates this token is scoped to the correct job.
use serde::{Deserialize, Serialize};
use uuid::Uuid;
use crate::error::WorkerError;
use crate::llm::{
ChatMessage, CompletionRequest, CompletionResponse, FinishReason, ToolCall,
ToolCompletionRequest, ToolCompletionResponse, ToolDefinition,
};
/// HTTP client that a container worker uses to talk to the orchestrator.
pub struct WorkerHttpClient {
client: reqwest::Client,
orchestrator_url: String,
job_id: Uuid,
token: String,
}
/// Status update sent from worker to orchestrator.
#[derive(Debug, Serialize, Deserialize)]
pub struct StatusUpdate {
pub state: String,
pub message: Option<String>,
pub iteration: u32,
}
/// Job description fetched from orchestrator.
#[derive(Debug, Serialize, Deserialize)]
pub struct JobDescription {
pub title: String,
pub description: String,
pub project_dir: Option<String>,
}
/// Completion result from the orchestrator (proxied from the real LLM).
#[derive(Debug, Serialize, Deserialize)]
pub struct ProxyCompletionRequest {
pub messages: Vec<ChatMessage>,
pub model: Option<String>,
pub max_tokens: Option<u32>,
pub temperature: Option<f32>,
pub stop_sequences: Option<Vec<String>>,
}
#[derive(Debug, Serialize, Deserialize)]
pub struct ProxyCompletionResponse {
pub content: String,
pub input_tokens: u32,
pub output_tokens: u32,
pub finish_reason: String,
#[serde(default)]
pub cache_read_input_tokens: u32,
#[serde(default)]
pub cache_creation_input_tokens: u32,
}
#[derive(Debug, Serialize, Deserialize)]
pub struct ProxyToolCompletionRequest {
pub messages: Vec<ChatMessage>,
pub tools: Vec<ToolDefinition>,
pub model: Option<String>,
pub max_tokens: Option<u32>,
pub temperature: Option<f32>,
pub tool_choice: Option<String>,
}
#[derive(Debug, Serialize, Deserialize)]
pub struct ProxyToolCompletionResponse {
pub content: Option<String>,
pub tool_calls: Vec<ToolCall>,
pub input_tokens: u32,
pub output_tokens: u32,
pub finish_reason: String,
#[serde(default)]
pub cache_read_input_tokens: u32,
#[serde(default)]
pub cache_creation_input_tokens: u32,
}
/// Completion result for the worker to report when done.
#[derive(Debug, Serialize, Deserialize)]
pub struct CompletionReport {
pub success: bool,
pub message: Option<String>,
pub iterations: u32,
}
/// Payload sent to the orchestrator for each job event (shared by worker and Claude Code bridge).
#[derive(Debug, Serialize, Deserialize)]
pub struct JobEventPayload {
pub event_type: String,
pub data: serde_json::Value,
}
/// Response from the prompt polling endpoint.
#[derive(Debug, Deserialize)]
pub struct PromptResponse {
pub content: String,
#[serde(default)]
pub done: bool,
}
/// A single credential delivered from the orchestrator to a container worker.
///
/// Shared between the orchestrator endpoint and the worker client.
#[derive(Debug, Serialize, Deserialize)]
pub struct CredentialResponse {
pub env_var: String,
pub value: String,
}
impl WorkerHttpClient {
/// Create a new client from environment.
///
/// Reads `IRONCLAW_WORKER_TOKEN` from the environment.
pub fn from_env(orchestrator_url: String, job_id: Uuid) -> Result<Self, WorkerError> {
let token =
std::env::var("IRONCLAW_WORKER_TOKEN").map_err(|_| WorkerError::MissingToken)?;
Ok(Self {
client: reqwest::Client::new(),
orchestrator_url: orchestrator_url.trim_end_matches('/').to_string(),
job_id,
token,
})
}
/// Create with an explicit token (for testing).
pub fn new(orchestrator_url: String, job_id: Uuid, token: String) -> Self {
Self {
client: reqwest::Client::new(),
orchestrator_url: orchestrator_url.trim_end_matches('/').to_string(),
job_id,
token,
}
}
/// Get the base orchestrator URL.
pub fn orchestrator_url(&self) -> &str {
&self.orchestrator_url
}
fn url(&self, path: &str) -> String {
format!("{}/worker/{}/{}", self.orchestrator_url, self.job_id, path)
}
/// Send a GET request, check the status, and deserialize the JSON body.
async fn get_json<T: serde::de::DeserializeOwned>(
&self,
path: &str,
context: &str,
) -> Result<T, WorkerError> {
let resp = self
.client
.get(self.url(path))
.bearer_auth(&self.token)
.send()
.await
.map_err(|e| WorkerError::ConnectionFailed {
url: self.orchestrator_url.clone(),
reason: e.to_string(),
})?;
if !resp.status().is_success() {
return Err(WorkerError::OrchestratorRejected {
job_id: self.job_id,
reason: format!("{} returned {}", context, resp.status()),
});
}
resp.json().await.map_err(|e| WorkerError::LlmProxyFailed {
reason: format!("{}: failed to parse response: {}", context, e),
})
}
/// Send a POST request with a JSON body, check the status, and deserialize the response.
async fn post_json<B: Serialize, T: serde::de::DeserializeOwned>(
&self,
path: &str,
body: &B,
context: &str,
) -> Result<T, WorkerError> {
let resp = self
.client
.post(self.url(path))
.bearer_auth(&self.token)
.json(body)
.send()
.await
.map_err(|e| WorkerError::LlmProxyFailed {
reason: format!("{}: {}", context, e),
})?;
if !resp.status().is_success() {
let status = resp.status();
let body = resp.text().await.unwrap_or_default();
return Err(WorkerError::LlmProxyFailed {
reason: format!("{}: orchestrator returned {}: {}", context, status, body),
});
}
resp.json().await.map_err(|e| WorkerError::LlmProxyFailed {
reason: format!("{}: failed to parse response: {}", context, e),
})
}
/// Fetch the job description from the orchestrator.
pub async fn get_job(&self) -> Result<JobDescription, WorkerError> {
self.get_json("job", "GET /job").await
}
/// Proxy an LLM completion request through the orchestrator.
pub async fn llm_complete(
&self,
request: &CompletionRequest,
) -> Result<CompletionResponse, WorkerError> {
let proxy_req = ProxyCompletionRequest {
messages: request.messages.clone(),
model: request.model.clone(),
max_tokens: request.max_tokens,
temperature: request.temperature,
stop_sequences: request.stop_sequences.clone(),
};
let proxy_resp: ProxyCompletionResponse = self
.post_json("llm/complete", &proxy_req, "LLM complete")
.await?;
Ok(CompletionResponse {
content: proxy_resp.content,
input_tokens: proxy_resp.input_tokens,
output_tokens: proxy_resp.output_tokens,
finish_reason: parse_finish_reason(&proxy_resp.finish_reason),
cache_read_input_tokens: proxy_resp.cache_read_input_tokens,
cache_creation_input_tokens: proxy_resp.cache_creation_input_tokens,
})
}
/// Proxy an LLM tool completion request through the orchestrator.
pub async fn llm_complete_with_tools(
&self,
request: &ToolCompletionRequest,
) -> Result<ToolCompletionResponse, WorkerError> {
let proxy_req = ProxyToolCompletionRequest {
messages: request.messages.clone(),
tools: request.tools.clone(),
model: request.model.clone(),
max_tokens: request.max_tokens,
temperature: request.temperature,
tool_choice: request.tool_choice.clone(),
};
let proxy_resp: ProxyToolCompletionResponse = self
.post_json("llm/complete_with_tools", &proxy_req, "LLM tool complete")
.await?;
Ok(ToolCompletionResponse {
content: proxy_resp.content,
tool_calls: proxy_resp.tool_calls,
input_tokens: proxy_resp.input_tokens,
output_tokens: proxy_resp.output_tokens,
finish_reason: parse_finish_reason(&proxy_resp.finish_reason),
cache_read_input_tokens: proxy_resp.cache_read_input_tokens,
cache_creation_input_tokens: proxy_resp.cache_creation_input_tokens,
})
}
/// Report status to the orchestrator.
pub async fn report_status(&self, update: &StatusUpdate) -> Result<(), WorkerError> {
let resp = self
.client
.post(self.url("status"))
.bearer_auth(&self.token)
.json(update)
.send()
.await
.map_err(|e| WorkerError::ConnectionFailed {
url: self.orchestrator_url.clone(),
reason: e.to_string(),
})?;
if !resp.status().is_success() {
tracing::warn!(
"Status report failed with {}: {}",
resp.status(),
resp.text().await.unwrap_or_default()
);
}
Ok(())
}
/// Post a job event to the orchestrator (fire-and-forget style, logs on failure).
pub async fn post_event(&self, payload: &JobEventPayload) {
let resp = self
.client
.post(self.url("event"))
.bearer_auth(&self.token)
.json(payload)
.send()
.await;
match resp {
Ok(r) if !r.status().is_success() => {
tracing::debug!(
job_id = %self.job_id,
event_type = %payload.event_type,
status = %r.status(),
"Job event POST rejected"
);
}
Err(e) => {
tracing::debug!(
job_id = %self.job_id,
event_type = %payload.event_type,
"Job event POST failed: {}", e
);
}
_ => {}
}
}
/// Poll the orchestrator for a follow-up prompt.
///
/// Returns `None` if no prompt is available (204 No Content).
pub async fn poll_prompt(&self) -> Result<Option<PromptResponse>, WorkerError> {
let resp = self
.client
.get(self.url("prompt"))
.bearer_auth(&self.token)
.send()
.await
.map_err(|e| WorkerError::ConnectionFailed {
url: self.orchestrator_url.clone(),
reason: e.to_string(),
})?;
if resp.status() == reqwest::StatusCode::NO_CONTENT {
return Ok(None);
}
if !resp.status().is_success() {
return Err(WorkerError::OrchestratorRejected {
job_id: self.job_id,
reason: format!("prompt endpoint returned {}", resp.status()),
});
}
let prompt: PromptResponse =
resp.json().await.map_err(|e| WorkerError::LlmProxyFailed {
reason: format!("failed to parse prompt response: {}", e),
})?;
Ok(Some(prompt))
}
/// Fetch credentials granted to this job from the orchestrator.
///
/// Returns an empty vec if no credentials are granted (204 No Content)
/// or if the endpoint returns 404. The caller should set each credential
/// as an environment variable before starting the execution loop.
pub async fn fetch_credentials(&self) -> Result<Vec<CredentialResponse>, WorkerError> {
let resp = self
.client
.get(self.url("credentials"))
.bearer_auth(&self.token)
.send()
.await
.map_err(|e| WorkerError::ConnectionFailed {
url: self.orchestrator_url.clone(),
reason: e.to_string(),
})?;
// 204 or 404 means no credentials granted, not an error
if resp.status() == reqwest::StatusCode::NO_CONTENT
|| resp.status() == reqwest::StatusCode::NOT_FOUND
{
return Ok(vec![]);
}
if !resp.status().is_success() {
return Err(WorkerError::SecretResolveFailed {
secret_name: "(all)".to_string(),
reason: format!("credentials endpoint returned {}", resp.status()),
});
}
resp.json()
.await
.map_err(|e| WorkerError::SecretResolveFailed {
secret_name: "(all)".to_string(),
reason: format!("failed to parse credentials response: {}", e),
})
}
/// Signal job completion to the orchestrator.
pub async fn report_complete(&self, report: &CompletionReport) -> Result<(), WorkerError> {
let _: serde_json::Value = self
.post_json("complete", report, "report complete")
.await?;
Ok(())
}
}
fn parse_finish_reason(s: &str) -> FinishReason {
match s {
"stop" => FinishReason::Stop,
"length" => FinishReason::Length,
"tool_use" | "tool_calls" => FinishReason::ToolUse,
"content_filter" => FinishReason::ContentFilter,
_ => FinishReason::Unknown,
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_url_construction() {
let client = WorkerHttpClient::new(
"http://host.docker.internal:50051".to_string(),
Uuid::nil(),
"test-token".to_string(),
);
assert_eq!(
client.url("llm/complete"),
format!(
"http://host.docker.internal:50051/worker/{}/llm/complete",
Uuid::nil()
)
);
}
#[test]
fn test_parse_finish_reason() {
assert_eq!(parse_finish_reason("stop"), FinishReason::Stop);
assert_eq!(parse_finish_reason("tool_use"), FinishReason::ToolUse);
assert_eq!(parse_finish_reason("unknown"), FinishReason::Unknown);
}
#[test]
fn test_credentials_url_construction() {
let client = WorkerHttpClient::new(
"http://host.docker.internal:50051".to_string(),
Uuid::nil(),
"test-token".to_string(),
);
assert_eq!(
client.url("credentials"),
format!(
"http://host.docker.internal:50051/worker/{}/credentials",
Uuid::nil()
)
);
}
#[test]
fn test_job_description_deserialization() {
let json = r#"{"title":"Test","description":"desc","project_dir":null}"#;
let job: JobDescription = serde_json::from_str(json).unwrap();
assert_eq!(job.title, "Test");
assert_eq!(job.description, "desc");
assert!(job.project_dir.is_none());
}
}