mirror of
https://github.com/outbackdingo/optimclaw.git
synced 2026-08-25 14:53:34 +00:00
* feat(llm): add Anthropic prompt caching and cache token tracking - Inject cache_control via additional_params for Claude models in rig_adapter - Add cache_read_input_tokens and cache_creation_input_tokens to CompletionResponse and ToolCompletionResponse - Extract cached_input_tokens from rig-core unified Usage - Add is_anthropic_model() detection helper with provider prefix support - Log prompt cache hits at debug level (consistent with response_cache) - Add 7 unit tests for cache injection and model detection - Update all mock providers and test fixtures with new fields * feat(cost): apply 90% cache discount to prompt-cached tokens in CostGuard - Add cache_read_input_tokens to TokenUsage so cache counts flow from CompletionResponse through the reasoning layer to the dispatcher - Update CostGuard::record_llm_call() to accept cache_read_input_tokens: cached tokens are billed at 10% of the normal input rate - Thread cache_read_input_tokens from dispatcher into CostGuard - Add test_cache_discount_reduces_cost verifying exact savings match 90% of input cost for fully-cached requests - Update all existing test callers with zero-cache parameter * refactor(cache): scope cache_control to Anthropic backend and validate model support - Replace model-name-based is_anthropic_model() with explicit enable_prompt_cache flag on RigAdapter, set only for the direct Anthropic backend via with_prompt_cache(true) - Add supports_prompt_cache() to validate model names per Anthropic docs: only Claude 3+ models support caching; claude-2 and claude-instant are excluded to prevent 400 errors - Warn when caching is enabled but model does not support it - Replace is_anthropic_model tests with flag-based and model validation tests * fix(cache): validate model at construction and propagate cache metrics through proxy - Move supports_prompt_cache() check into with_prompt_cache() so unsupported models are detected once at construction, not per request - Add cache_read_input_tokens and cache_creation_input_tokens to ProxyCompletionResponse and ProxyToolCompletionResponse with serde(default) for backward compatibility - Pass cache metrics through orchestrator proxy instead of zeroing - Use claude-opus-4-6 in cache discount test to match Anthropic semantics * feat(llm): add configurable cache retention with write surcharge - Add CacheRetention enum (none/short/long) to AnthropicDirectConfig - Parse ANTHROPIC_CACHE_RETENTION env var (default: short) - Inject TTL-aware cache_control (short=5m ephemeral, long=1h) - Extract cache_creation_input_tokens from raw Anthropic response - Add cache_write_multiplier() to LlmProvider trait (1.25x short, 2.0x long) - Pipe dynamic write multiplier through dispatcher to CostGuard - Add TokenUsage.cache_creation_input_tokens field - Add tests for Long TTL injection, 5m and 1h write surcharges - Document ANTHROPIC_CACHE_RETENTION in .env.example * docs: fix stale cache_retention field comment * fix: resolve CI failures after upstream merge - Add missing cost_per_token arg to cache test callsites - Apply cargo fmt to long lines in tests and tracing macros * fix: address Copilot review feedback - Use saturating_add for cache token sum to prevent u32 overflow - Tighten supports_prompt_cache to explicitly match claude-3+/claude-4+ and named families (claude-sonnet/claude-opus/claude-haiku) * fix: adapt prompt caching to registry architecture and add missing cache fields - Resolve merge conflicts: adapt CacheRetention and cache injection to the declarative provider registry (RegistryProviderConfig replaces AnthropicDirectConfig) - Parse ANTHROPIC_CACHE_RETENTION env var in create_anthropic_from_registry() - Use Anthropic automatic caching via top-level cache_control in additional_params (rig-core #[serde(flatten)] places it at request root) - Add cache_read/creation_input_tokens fields to all mock LlmProviders added on main after PR #291 branched (response_cache, dispatcher, provider_chaos, trace_llm) - Suppress clippy::too_many_arguments on record_llm_call and build_rig_request - Add regression tests for cache injection (short/long/none) and cache_write_multiplier values Co-Authored-By: Canvinus <[email protected]> * fix: delegate cache_write_multiplier through provider wrappers and make cache_read_discount configurable The 6 decorator providers (Retry, CircuitBreaker, Failover, SmartRouting, CachedProvider, RecordingLlm) did not delegate cache_write_multiplier() to their inner provider, causing it to always return 1.0 instead of the actual 1.25x/2.0x from RigAdapter. This fix adds delegation for both cache_write_multiplier() and the new cache_read_discount() method. Also makes the cache read discount per-provider instead of hardcoding Anthropic's 90% discount (÷10). OpenAI uses 50% (÷2), so the discount is now returned by each provider via the LlmProvider trait. Addresses review feedback on PR #660. Co-Authored-By: Claude Opus 4.6 <[email protected]> * style: cargo fmt Co-Authored-By: Claude Opus 4.6 <[email protected]> * test: add CacheRetention FromStr/Display unit tests Tests cover primary values, aliases (off/disabled/5m/ephemeral/1h), case-insensitivity, invalid input error, and Display round-trip. Addresses Copilot review feedback on PR #660. Co-Authored-By: Claude Opus 4.6 <[email protected]> --------- Co-authored-by: Andrey <[email protected]> Co-authored-by: Andrey Gruzdev <[email protected]> Co-authored-by: Claude Opus 4.6 <[email protected]>
694 lines
20 KiB
Rust
694 lines
20 KiB
Rust
//! Integration tests for the OpenAI-compatible API endpoints.
|
|
//!
|
|
//! Uses a mock LLM provider so no real API key is needed.
|
|
|
|
use std::net::SocketAddr;
|
|
use std::sync::Arc;
|
|
use std::time::Duration;
|
|
|
|
use async_trait::async_trait;
|
|
use rust_decimal::Decimal;
|
|
|
|
use ironclaw::channels::web::server::GatewayState;
|
|
use ironclaw::channels::web::test_helpers::TestGatewayBuilder;
|
|
use ironclaw::error::LlmError;
|
|
use ironclaw::llm::{
|
|
CompletionRequest, CompletionResponse, FinishReason, LlmProvider, ToolCompletionRequest,
|
|
ToolCompletionResponse,
|
|
};
|
|
|
|
const AUTH_TOKEN: &str = "test-openai-token";
|
|
|
|
// ---------------------------------------------------------------------------
|
|
// Mock LLM provider
|
|
// ---------------------------------------------------------------------------
|
|
|
|
#[derive(Default)]
|
|
struct MockLlmState {
|
|
completion_models: tokio::sync::Mutex<Vec<Option<String>>>,
|
|
tool_completion_models: tokio::sync::Mutex<Vec<Option<String>>>,
|
|
}
|
|
|
|
struct MockLlmProvider {
|
|
state: Arc<MockLlmState>,
|
|
}
|
|
|
|
impl MockLlmProvider {
|
|
fn new(state: Arc<MockLlmState>) -> Self {
|
|
Self { state }
|
|
}
|
|
}
|
|
|
|
#[async_trait]
|
|
impl LlmProvider for MockLlmProvider {
|
|
fn model_name(&self) -> &str {
|
|
"mock-model-v1"
|
|
}
|
|
|
|
fn cost_per_token(&self) -> (Decimal, Decimal) {
|
|
(Decimal::ZERO, Decimal::ZERO)
|
|
}
|
|
|
|
async fn complete(&self, req: CompletionRequest) -> Result<CompletionResponse, LlmError> {
|
|
self.state
|
|
.completion_models
|
|
.lock()
|
|
.await
|
|
.push(req.model.clone());
|
|
|
|
// Echo the last user message back
|
|
let user_msg = req
|
|
.messages
|
|
.iter()
|
|
.rev()
|
|
.find(|m| m.role == ironclaw::llm::Role::User)
|
|
.map(|m| m.content.clone())
|
|
.unwrap_or_else(|| "no user message".to_string());
|
|
|
|
Ok(CompletionResponse {
|
|
content: format!("Mock response to: {}", user_msg),
|
|
input_tokens: 10,
|
|
output_tokens: 5,
|
|
finish_reason: FinishReason::Stop,
|
|
cache_read_input_tokens: 0,
|
|
cache_creation_input_tokens: 0,
|
|
})
|
|
}
|
|
|
|
async fn complete_with_tools(
|
|
&self,
|
|
req: ToolCompletionRequest,
|
|
) -> Result<ToolCompletionResponse, LlmError> {
|
|
self.state
|
|
.tool_completion_models
|
|
.lock()
|
|
.await
|
|
.push(req.model.clone());
|
|
|
|
// If tools are provided, return a tool call
|
|
if let Some(tool) = req.tools.first() {
|
|
Ok(ToolCompletionResponse {
|
|
content: None,
|
|
tool_calls: vec![ironclaw::llm::ToolCall {
|
|
id: "call_mock_001".to_string(),
|
|
name: tool.name.clone(),
|
|
arguments: serde_json::json!({"test": true}),
|
|
}],
|
|
input_tokens: 15,
|
|
output_tokens: 8,
|
|
finish_reason: FinishReason::ToolUse,
|
|
cache_read_input_tokens: 0,
|
|
cache_creation_input_tokens: 0,
|
|
})
|
|
} else {
|
|
Ok(ToolCompletionResponse {
|
|
content: Some("No tools available".to_string()),
|
|
tool_calls: vec![],
|
|
input_tokens: 10,
|
|
output_tokens: 4,
|
|
finish_reason: FinishReason::Stop,
|
|
cache_read_input_tokens: 0,
|
|
cache_creation_input_tokens: 0,
|
|
})
|
|
}
|
|
}
|
|
|
|
async fn list_models(&self) -> Result<Vec<String>, LlmError> {
|
|
Ok(vec![
|
|
"mock-model-v1".to_string(),
|
|
"mock-model-v2".to_string(),
|
|
])
|
|
}
|
|
}
|
|
|
|
struct FixedModelProvider {
|
|
model: &'static str,
|
|
}
|
|
|
|
impl FixedModelProvider {
|
|
fn new(model: &'static str) -> Self {
|
|
Self { model }
|
|
}
|
|
}
|
|
|
|
#[async_trait]
|
|
impl LlmProvider for FixedModelProvider {
|
|
fn model_name(&self) -> &str {
|
|
self.model
|
|
}
|
|
|
|
fn cost_per_token(&self) -> (Decimal, Decimal) {
|
|
(Decimal::ZERO, Decimal::ZERO)
|
|
}
|
|
|
|
async fn complete(&self, _req: CompletionRequest) -> Result<CompletionResponse, LlmError> {
|
|
Ok(CompletionResponse {
|
|
content: "fixed response".to_string(),
|
|
input_tokens: 10,
|
|
output_tokens: 5,
|
|
finish_reason: FinishReason::Stop,
|
|
cache_read_input_tokens: 0,
|
|
cache_creation_input_tokens: 0,
|
|
})
|
|
}
|
|
|
|
async fn complete_with_tools(
|
|
&self,
|
|
_req: ToolCompletionRequest,
|
|
) -> Result<ToolCompletionResponse, LlmError> {
|
|
Ok(ToolCompletionResponse {
|
|
content: Some("fixed response".to_string()),
|
|
tool_calls: vec![],
|
|
input_tokens: 10,
|
|
output_tokens: 5,
|
|
finish_reason: FinishReason::Stop,
|
|
cache_read_input_tokens: 0,
|
|
cache_creation_input_tokens: 0,
|
|
})
|
|
}
|
|
|
|
fn effective_model_name(&self, _requested_model: Option<&str>) -> String {
|
|
self.model.to_string()
|
|
}
|
|
}
|
|
|
|
// ---------------------------------------------------------------------------
|
|
// Test helpers
|
|
// ---------------------------------------------------------------------------
|
|
|
|
async fn start_test_server() -> (SocketAddr, Arc<GatewayState>, Arc<MockLlmState>) {
|
|
let mock_state = Arc::new(MockLlmState::default());
|
|
|
|
let llm_provider: Arc<dyn LlmProvider> = Arc::new(MockLlmProvider::new(mock_state.clone()));
|
|
let (bound_addr, state) = start_test_server_with_provider(llm_provider).await;
|
|
|
|
(bound_addr, state, mock_state)
|
|
}
|
|
|
|
async fn start_test_server_with_provider(
|
|
llm_provider: Arc<dyn LlmProvider>,
|
|
) -> (SocketAddr, Arc<GatewayState>) {
|
|
TestGatewayBuilder::new()
|
|
.llm_provider(llm_provider)
|
|
.start(AUTH_TOKEN)
|
|
.await
|
|
.expect("Failed to start test server")
|
|
}
|
|
|
|
fn client() -> reqwest::Client {
|
|
reqwest::Client::builder()
|
|
.timeout(Duration::from_secs(10))
|
|
.build()
|
|
.unwrap()
|
|
}
|
|
|
|
// ---------------------------------------------------------------------------
|
|
// Tests
|
|
// ---------------------------------------------------------------------------
|
|
|
|
#[tokio::test]
|
|
async fn test_chat_completions_basic() {
|
|
let (addr, _state, mock_state) = start_test_server().await;
|
|
let url = format!("http://{}/v1/chat/completions", addr);
|
|
|
|
let resp = client()
|
|
.post(&url)
|
|
.bearer_auth(AUTH_TOKEN)
|
|
.json(&serde_json::json!({
|
|
"model": "mock-model-v1",
|
|
"messages": [
|
|
{"role": "user", "content": "Hello world"}
|
|
]
|
|
}))
|
|
.send()
|
|
.await
|
|
.unwrap();
|
|
|
|
assert_eq!(resp.status(), 200);
|
|
|
|
let body: serde_json::Value = resp.json().await.unwrap();
|
|
assert_eq!(body["object"], "chat.completion");
|
|
assert_eq!(body["model"], "mock-model-v1");
|
|
assert_eq!(body["choices"][0]["finish_reason"], "stop");
|
|
|
|
let content = body["choices"][0]["message"]["content"].as_str().unwrap();
|
|
assert!(
|
|
content.contains("Hello world"),
|
|
"Expected echo, got: {}",
|
|
content
|
|
);
|
|
|
|
// Check usage
|
|
assert_eq!(body["usage"]["prompt_tokens"], 10);
|
|
assert_eq!(body["usage"]["completion_tokens"], 5);
|
|
assert_eq!(body["usage"]["total_tokens"], 15);
|
|
|
|
let models = mock_state.completion_models.lock().await;
|
|
assert_eq!(*models, vec![Some("mock-model-v1".to_string())]);
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn test_chat_completions_with_system_message() {
|
|
let (addr, _state, _mock_state) = start_test_server().await;
|
|
let url = format!("http://{}/v1/chat/completions", addr);
|
|
|
|
let resp = client()
|
|
.post(&url)
|
|
.bearer_auth(AUTH_TOKEN)
|
|
.json(&serde_json::json!({
|
|
"model": "mock-model-v1",
|
|
"messages": [
|
|
{"role": "system", "content": "You are helpful."},
|
|
{"role": "user", "content": "What is 2+2?"}
|
|
],
|
|
"temperature": 0.5,
|
|
"max_tokens": 100
|
|
}))
|
|
.send()
|
|
.await
|
|
.unwrap();
|
|
|
|
assert_eq!(resp.status(), 200);
|
|
let body: serde_json::Value = resp.json().await.unwrap();
|
|
let content = body["choices"][0]["message"]["content"].as_str().unwrap();
|
|
assert!(content.contains("2+2"));
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn test_chat_completions_with_tools() {
|
|
let (addr, _state, mock_state) = start_test_server().await;
|
|
let url = format!("http://{}/v1/chat/completions", addr);
|
|
|
|
let resp = client()
|
|
.post(&url)
|
|
.bearer_auth(AUTH_TOKEN)
|
|
.json(&serde_json::json!({
|
|
"model": "mock-model-v1",
|
|
"messages": [
|
|
{"role": "user", "content": "What's the weather?"}
|
|
],
|
|
"tools": [{
|
|
"type": "function",
|
|
"function": {
|
|
"name": "get_weather",
|
|
"description": "Get the weather",
|
|
"parameters": {
|
|
"type": "object",
|
|
"properties": {
|
|
"location": {"type": "string"}
|
|
}
|
|
}
|
|
}
|
|
}]
|
|
}))
|
|
.send()
|
|
.await
|
|
.unwrap();
|
|
|
|
assert_eq!(resp.status(), 200);
|
|
let body: serde_json::Value = resp.json().await.unwrap();
|
|
|
|
assert_eq!(body["choices"][0]["finish_reason"], "tool_calls");
|
|
|
|
let tool_calls = &body["choices"][0]["message"]["tool_calls"];
|
|
assert!(tool_calls.is_array());
|
|
assert_eq!(tool_calls[0]["id"], "call_mock_001");
|
|
assert_eq!(tool_calls[0]["type"], "function");
|
|
assert_eq!(tool_calls[0]["function"]["name"], "get_weather");
|
|
|
|
let models = mock_state.tool_completion_models.lock().await;
|
|
assert_eq!(*models, vec![Some("mock-model-v1".to_string())]);
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn test_chat_completions_streaming() {
|
|
let (addr, _state, mock_state) = start_test_server().await;
|
|
let url = format!("http://{}/v1/chat/completions", addr);
|
|
|
|
let resp = client()
|
|
.post(&url)
|
|
.bearer_auth(AUTH_TOKEN)
|
|
.json(&serde_json::json!({
|
|
"model": "mock-model-v1",
|
|
"messages": [
|
|
{"role": "user", "content": "Stream test"}
|
|
],
|
|
"stream": true
|
|
}))
|
|
.send()
|
|
.await
|
|
.unwrap();
|
|
|
|
assert_eq!(resp.status(), 200);
|
|
|
|
// Check simulated streaming header
|
|
assert_eq!(
|
|
resp.headers()
|
|
.get("x-ironclaw-streaming")
|
|
.and_then(|v| v.to_str().ok()),
|
|
Some("simulated"),
|
|
"Expected x-ironclaw-streaming: simulated header"
|
|
);
|
|
|
|
let text = resp.text().await.unwrap();
|
|
|
|
// Should contain SSE data lines
|
|
assert!(
|
|
text.contains("data:"),
|
|
"Expected SSE data lines, got: {}",
|
|
text
|
|
);
|
|
// Should end with [DONE]
|
|
assert!(
|
|
text.contains("[DONE]"),
|
|
"Expected [DONE] sentinel, got: {}",
|
|
text
|
|
);
|
|
// Should contain the role chunk
|
|
assert!(
|
|
text.contains("\"role\":\"assistant\""),
|
|
"Expected role chunk, got: {}",
|
|
text
|
|
);
|
|
|
|
// Collect all content from the chunks
|
|
let mut full_content = String::new();
|
|
for line in text.lines() {
|
|
if let Some(data) = line.strip_prefix("data:") {
|
|
let data = data.trim();
|
|
if data == "[DONE]" {
|
|
continue;
|
|
}
|
|
if let Ok(chunk) = serde_json::from_str::<serde_json::Value>(data)
|
|
&& let Some(content) = chunk["choices"][0]["delta"]["content"].as_str()
|
|
{
|
|
full_content.push_str(content);
|
|
}
|
|
}
|
|
}
|
|
assert!(
|
|
full_content.contains("Stream test"),
|
|
"Expected reassembled content to contain 'Stream test', got: '{}'",
|
|
full_content
|
|
);
|
|
|
|
let models = mock_state.completion_models.lock().await;
|
|
assert_eq!(*models, vec![Some("mock-model-v1".to_string())]);
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn test_chat_completions_empty_messages() {
|
|
let (addr, _state, _mock_state) = start_test_server().await;
|
|
let url = format!("http://{}/v1/chat/completions", addr);
|
|
|
|
let resp = client()
|
|
.post(&url)
|
|
.bearer_auth(AUTH_TOKEN)
|
|
.json(&serde_json::json!({
|
|
"model": "mock-model-v1",
|
|
"messages": []
|
|
}))
|
|
.send()
|
|
.await
|
|
.unwrap();
|
|
|
|
assert_eq!(resp.status(), 400);
|
|
let body: serde_json::Value = resp.json().await.unwrap();
|
|
assert!(body["error"]["message"].as_str().unwrap().contains("empty"));
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn test_chat_completions_model_override() {
|
|
let (addr, _state, mock_state) = start_test_server().await;
|
|
let url = format!("http://{}/v1/chat/completions", addr);
|
|
|
|
let resp = client()
|
|
.post(&url)
|
|
.bearer_auth(AUTH_TOKEN)
|
|
.json(&serde_json::json!({
|
|
"model": "gpt-4",
|
|
"messages": [{"role": "user", "content": "Hi"}]
|
|
}))
|
|
.send()
|
|
.await
|
|
.unwrap();
|
|
|
|
assert_eq!(resp.status(), 200);
|
|
let body: serde_json::Value = resp.json().await.unwrap();
|
|
assert_eq!(body["model"], "gpt-4");
|
|
|
|
let models = mock_state.completion_models.lock().await;
|
|
assert_eq!(*models, vec![Some("gpt-4".to_string())]);
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn test_chat_completions_uses_effective_model_when_override_ignored() {
|
|
let provider: Arc<dyn LlmProvider> = Arc::new(FixedModelProvider::new("configured-model"));
|
|
let (addr, _state) = start_test_server_with_provider(provider).await;
|
|
let url = format!("http://{}/v1/chat/completions", addr);
|
|
|
|
let resp = client()
|
|
.post(&url)
|
|
.bearer_auth(AUTH_TOKEN)
|
|
.json(&serde_json::json!({
|
|
"model": "gpt-4",
|
|
"messages": [{"role": "user", "content": "Hi"}]
|
|
}))
|
|
.send()
|
|
.await
|
|
.unwrap();
|
|
|
|
assert_eq!(resp.status(), 200);
|
|
let body: serde_json::Value = resp.json().await.unwrap();
|
|
assert_eq!(body["model"], "configured-model");
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn test_chat_completions_streaming_uses_effective_model_when_override_ignored() {
|
|
let provider: Arc<dyn LlmProvider> = Arc::new(FixedModelProvider::new("configured-model"));
|
|
let (addr, _state) = start_test_server_with_provider(provider).await;
|
|
let url = format!("http://{}/v1/chat/completions", addr);
|
|
|
|
let resp = client()
|
|
.post(&url)
|
|
.bearer_auth(AUTH_TOKEN)
|
|
.json(&serde_json::json!({
|
|
"model": "gpt-4",
|
|
"messages": [{"role": "user", "content": "Hi"}],
|
|
"stream": true
|
|
}))
|
|
.send()
|
|
.await
|
|
.unwrap();
|
|
|
|
assert_eq!(resp.status(), 200);
|
|
let text = resp.text().await.unwrap();
|
|
assert!(
|
|
text.contains("\"model\":\"configured-model\""),
|
|
"Expected streaming chunks to report configured model, got: {}",
|
|
text
|
|
);
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn test_chat_completions_model_too_long() {
|
|
let (addr, _state, mock_state) = start_test_server().await;
|
|
let url = format!("http://{}/v1/chat/completions", addr);
|
|
|
|
let resp = client()
|
|
.post(&url)
|
|
.bearer_auth(AUTH_TOKEN)
|
|
.json(&serde_json::json!({
|
|
"model": "m".repeat(300),
|
|
"messages": [{"role": "user", "content": "Hi"}]
|
|
}))
|
|
.send()
|
|
.await
|
|
.unwrap();
|
|
|
|
assert_eq!(resp.status(), 400);
|
|
let body: serde_json::Value = resp.json().await.unwrap();
|
|
assert!(
|
|
body["error"]["message"]
|
|
.as_str()
|
|
.unwrap_or("")
|
|
.contains("model"),
|
|
"Expected model validation error, got: {}",
|
|
body
|
|
);
|
|
|
|
// Validation should fail before provider invocation.
|
|
let models = mock_state.completion_models.lock().await;
|
|
assert!(
|
|
models.is_empty(),
|
|
"provider should not be called: {:?}",
|
|
*models
|
|
);
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn test_chat_completions_model_with_control_chars() {
|
|
let (addr, _state, mock_state) = start_test_server().await;
|
|
let url = format!("http://{}/v1/chat/completions", addr);
|
|
|
|
let resp = client()
|
|
.post(&url)
|
|
.bearer_auth(AUTH_TOKEN)
|
|
.json(&serde_json::json!({
|
|
"model": "gpt-4\noops",
|
|
"messages": [{"role": "user", "content": "Hi"}]
|
|
}))
|
|
.send()
|
|
.await
|
|
.unwrap();
|
|
|
|
assert_eq!(resp.status(), 400);
|
|
let body: serde_json::Value = resp.json().await.unwrap();
|
|
assert!(
|
|
body["error"]["message"]
|
|
.as_str()
|
|
.unwrap_or("")
|
|
.contains("control"),
|
|
"Expected model validation error, got: {}",
|
|
body
|
|
);
|
|
|
|
// Validation should fail before provider invocation.
|
|
let models = mock_state.completion_models.lock().await;
|
|
assert!(
|
|
models.is_empty(),
|
|
"provider should not be called: {:?}",
|
|
*models
|
|
);
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn test_chat_completions_model_with_surrounding_whitespace() {
|
|
let (addr, _state, mock_state) = start_test_server().await;
|
|
let url = format!("http://{}/v1/chat/completions", addr);
|
|
|
|
let resp = client()
|
|
.post(&url)
|
|
.bearer_auth(AUTH_TOKEN)
|
|
.json(&serde_json::json!({
|
|
"model": " gpt-4 ",
|
|
"messages": [{"role": "user", "content": "Hi"}]
|
|
}))
|
|
.send()
|
|
.await
|
|
.unwrap();
|
|
|
|
assert_eq!(resp.status(), 400);
|
|
let body: serde_json::Value = resp.json().await.unwrap();
|
|
assert!(
|
|
body["error"]["message"]
|
|
.as_str()
|
|
.unwrap_or("")
|
|
.contains("leading or trailing whitespace"),
|
|
"Expected model validation error, got: {}",
|
|
body
|
|
);
|
|
|
|
let models = mock_state.completion_models.lock().await;
|
|
assert!(
|
|
models.is_empty(),
|
|
"provider should not be called: {:?}",
|
|
*models
|
|
);
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn test_chat_completions_no_auth() {
|
|
let (addr, _state, _mock_state) = start_test_server().await;
|
|
let url = format!("http://{}/v1/chat/completions", addr);
|
|
|
|
let resp = client()
|
|
.post(&url)
|
|
// No auth header
|
|
.json(&serde_json::json!({
|
|
"model": "mock-model-v1",
|
|
"messages": [{"role": "user", "content": "Hi"}]
|
|
}))
|
|
.send()
|
|
.await
|
|
.unwrap();
|
|
|
|
assert_eq!(resp.status(), 401);
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn test_models_endpoint() {
|
|
let (addr, _state, _mock_state) = start_test_server().await;
|
|
let url = format!("http://{}/v1/models", addr);
|
|
|
|
let resp = client()
|
|
.get(&url)
|
|
.bearer_auth(AUTH_TOKEN)
|
|
.send()
|
|
.await
|
|
.unwrap();
|
|
|
|
assert_eq!(resp.status(), 200);
|
|
let body: serde_json::Value = resp.json().await.unwrap();
|
|
|
|
assert_eq!(body["object"], "list");
|
|
let data = body["data"].as_array().unwrap();
|
|
assert_eq!(data.len(), 2);
|
|
assert_eq!(data[0]["id"], "mock-model-v1");
|
|
assert_eq!(data[1]["id"], "mock-model-v2");
|
|
assert_eq!(data[0]["object"], "model");
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn test_models_no_auth() {
|
|
let (addr, _state, _mock_state) = start_test_server().await;
|
|
let url = format!("http://{}/v1/models", addr);
|
|
|
|
let resp = client().get(&url).send().await.unwrap();
|
|
assert_eq!(resp.status(), 401);
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn test_no_llm_provider_returns_503() {
|
|
// Create state WITHOUT llm_provider
|
|
let (bound_addr, _state) = TestGatewayBuilder::new()
|
|
.start(AUTH_TOKEN)
|
|
.await
|
|
.expect("Failed to start test server");
|
|
|
|
let url = format!("http://{}/v1/chat/completions", bound_addr);
|
|
let resp = client()
|
|
.post(&url)
|
|
.bearer_auth(AUTH_TOKEN)
|
|
.json(&serde_json::json!({
|
|
"model": "mock-model-v1",
|
|
"messages": [{"role": "user", "content": "Hi"}]
|
|
}))
|
|
.send()
|
|
.await
|
|
.unwrap();
|
|
|
|
assert_eq!(resp.status(), 503);
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn test_chat_completions_body_too_large() {
|
|
let (addr, _state, _mock_state) = start_test_server().await;
|
|
let url = format!("http://{}/v1/chat/completions", addr);
|
|
|
|
// Build a payload over 1 MB (the gateway's DefaultBodyLimit)
|
|
let big_content = "x".repeat(2 * 1024 * 1024);
|
|
let resp = client()
|
|
.post(&url)
|
|
.bearer_auth(AUTH_TOKEN)
|
|
.json(&serde_json::json!({
|
|
"model": "mock-model-v1",
|
|
"messages": [{"role": "user", "content": big_content}]
|
|
}))
|
|
.send()
|
|
.await
|
|
.unwrap();
|
|
|
|
assert_eq!(resp.status(), 413);
|
|
}
|