Improve test infrastructure: StubChannel, gateway helpers, security tests, search edge cases (#623)

* feat(testing): add StubChannel test double for Channel trait

Adds StubChannel to src/testing.rs alongside StubLlm. Supports message
injection via mpsc sender, response/status capture, and configurable
health check toggling. Includes handle methods for use after ownership
transfer to ChannelManager.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* feat(testing): wire StubChannel into TestHarnessBuilder

Add with_stub_channel() builder method that creates a StubChannel
pre-registered in a ChannelManager. Tests can inject messages via
the sender and verify routing through the manager. The channel field
on TestHarness is Optional, defaulting to None for backward compat.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* test: gate external-service tests behind integration feature flag

Replace silent try_connect() skip pattern with explicit feature gating.
cargo test now runs only self-contained tests.
cargo test --features integration runs tests requiring PostgreSQL.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* test(channels): add ChannelManager unit tests using StubChannel

Cover add/start_all stream merging, respond routing, unknown channel
errors, health_check_all with mixed health, empty-channels error path,
and injection channel merging -- all via StubChannel test double.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* docs: document test tier separation (unit/integration/live)

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* ci: add architecture boundary check script

Grep-based checks for three architecture boundaries:
- Direct database driver usage (tokio_postgres/libsql) outside src/db/
- .unwrap()/.expect() in production code (warning only)
- Direct std::env::var reads outside config layer (warning only)

The DB driver check is a hard violation; the other two are warnings
for gradual cleanup. Run with: bash scripts/check-boundaries.sh

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* test(search): add RRF edge case tests for empty inputs, limits, and config modes

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* test(security): add regression tests for skill installer ZIP and SSRF protections

Add 11 regression tests covering the security controls in skill_tools:

ZIP extraction safety:
- Valid SKILL.md extraction works correctly
- Non-SKILL.md entries are ignored (returns error)
- Path traversal entries (../../SKILL.md) do not match
- Nested path entries (subdir/SKILL.md) do not match
- Oversized entries (>1MB uncompressed) are rejected

SSRF prevention:
- Loopback addresses (127.0.0.1) are blocked
- Private ranges (10.x, 172.16.x, 192.168.x) are blocked
- Link-local addresses (169.254.x) are blocked
- Public IPs (8.8.8.8, 1.1.1.1) are allowed
- IPv4-mapped IPv6 unwrapping logic works correctly
- Metadata endpoints and .internal/.local hostnames are blocked
- Normal hostnames (github.com, clawhub.dev) are allowed

Also documents a known gap: url::Url::host_str() returns bracketed
IPv6 addresses that std::net::IpAddr cannot parse, so IPv4-mapped
IPv6 URLs currently bypass IP-based checks in validate_fetch_url.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* refactor(testing): extract TestGatewayBuilder to eliminate gateway test duplication

Both ws_gateway_integration.rs and openai_compat_integration.rs manually
constructed GatewayState with 19+ fields. Extracted to a shared builder in
src/channels/web/test_helpers.rs that provides sensible defaults and lets
tests override only what they need.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* docs: add implementation plans for testing batches 1 and 2

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix(security): close IPv6 SSRF bypass in validate_fetch_url

validate_fetch_url used host_str() which returns bracketed IPv6
(e.g. "[::ffff:7f00:1]") that IpAddr::parse() cannot handle,
silently skipping IP-based SSRF checks for all IPv6 URLs.

Switch to url::Host enum matching to extract proper IpAddr values
without string parsing. IPv4-mapped IPv6 addresses like
::ffff:127.0.0.1 are now correctly unwrapped and blocked.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* test(skills): add activation criteria limits enforcement tests

Adds test_activation_criteria_enforce_limits to verify that
enforce_limits() correctly trims excess patterns (>5), keywords (>20),
and tags (>10), and filters out short keywords/tags (<3 chars).

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* test(wasm): add security regression tests for WASM tool loader

Add 6 tests covering: tool name path separator rejection, empty name
rejection, nonexistent file handling, invalid WASM bytes rejection,
dotfile discovery behavior, and subdirectory non-recursion.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* refactor: address PR review feedback

- Remove plan files from repo (ilblackdragon review)
- Replace CLAUDE.md test tier rules with pointer to check-boundaries.sh
- Add Check 4 to check-boundaries.sh: enforces integration tests are
  gated behind the 'integration' feature flag

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* ci: add try_connect silent-skip pattern check to check-boundaries.sh

Check 5 catches try_connect() and similar silent-skip patterns in
integration tests. Tests should use feature gates to fail loudly
when prerequisites are missing, not silently return.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix(security): harden skill fetch SSRF checks

* fix(scripts): use bash arrays in check-boundaries.sh tier violation check

Refactor Check 4 in check-boundaries.sh to use bash arrays and printf
instead of string concatenation with echo -e. This is more robust with
special characters in filenames and avoids portability concerns with
echo -e. [skip-regression-check]

Co-Authored-By: Claude Opus 4.6 <[email protected]>

---------

Co-authored-by: Claude Opus 4.6 <[email protected]>
This commit is contained in:
Zaki Manian
2026-03-07 08:30:47 +00:00
committed by GitHub
co-authored by Claude Opus 4.6
parent cf96a3253c
commit 45ec691f4c
15 changed files with 1479 additions and 166 deletions
+214 -1
View File
@@ -2,6 +2,7 @@
//!
//! Provides:
//! - [`StubLlm`]: A configurable LLM provider that returns a fixed response
//! - [`StubChannel`]: A configurable channel stub with message injection and response capture
//! - [`TestHarnessBuilder`]: Builder for wiring `AgentDeps` with defaults
//! - [`TestHarness`]: The assembled components ready for use in tests
//!
@@ -18,14 +19,19 @@
//! ```
use std::sync::Arc;
use std::sync::Mutex;
use std::sync::atomic::{AtomicBool, AtomicU32, Ordering};
use async_trait::async_trait;
use rust_decimal::Decimal;
use tokio::sync::mpsc;
use crate::agent::AgentDeps;
use crate::channels::{
Channel, ChannelManager, IncomingMessage, MessageStream, OutgoingResponse, StatusUpdate,
};
use crate::db::Database;
use crate::error::LlmError;
use crate::error::{ChannelError, LlmError};
use crate::llm::{
CompletionRequest, CompletionResponse, FinishReason, LlmProvider, ToolCompletionRequest,
ToolCompletionResponse,
@@ -189,12 +195,138 @@ impl LlmProvider for StubLlm {
}
}
/// A configurable channel stub for tests.
///
/// Supports:
/// - Message injection via the returned `mpsc::Sender`
/// - Response capture for assertion
/// - Status update capture
/// - Configurable health check failure
///
/// # Usage
///
/// ```rust,no_run
/// let (channel, sender) = StubChannel::new("test");
/// sender.send(IncomingMessage::new("test", "user1", "hello")).await.unwrap();
/// // ... run agent logic that calls channel.respond() ...
/// let responses = channel.captured_responses();
/// ```
pub struct StubChannel {
name: String,
rx: tokio::sync::Mutex<Option<mpsc::Receiver<IncomingMessage>>>,
responses: Arc<Mutex<Vec<(IncomingMessage, OutgoingResponse)>>>,
statuses: Arc<Mutex<Vec<StatusUpdate>>>,
healthy: AtomicBool,
}
impl StubChannel {
/// Create a new stub channel and its message sender.
///
/// The sender is used by tests to inject messages into the channel's stream.
/// The channel captures all responses and status updates for later assertion.
pub fn new(name: impl Into<String>) -> (Self, mpsc::Sender<IncomingMessage>) {
let (tx, rx) = mpsc::channel(64);
let channel = Self {
name: name.into(),
rx: tokio::sync::Mutex::new(Some(rx)),
responses: Arc::new(Mutex::new(Vec::new())),
statuses: Arc::new(Mutex::new(Vec::new())),
healthy: AtomicBool::new(true),
};
(channel, tx)
}
/// Get all captured (message, response) pairs.
pub fn captured_responses(&self) -> Vec<(IncomingMessage, OutgoingResponse)> {
self.responses.lock().expect("poisoned").clone()
}
/// Get a shared handle to the response capture list.
///
/// Call this *before* moving the channel into a `ChannelManager`,
/// since `add()` takes ownership.
pub fn captured_responses_handle(
&self,
) -> Arc<Mutex<Vec<(IncomingMessage, OutgoingResponse)>>> {
Arc::clone(&self.responses)
}
/// Get all captured status updates.
pub fn captured_statuses(&self) -> Vec<StatusUpdate> {
self.statuses.lock().expect("poisoned").clone()
}
/// Get a shared handle to the status capture list.
pub fn captured_statuses_handle(&self) -> Arc<Mutex<Vec<StatusUpdate>>> {
Arc::clone(&self.statuses)
}
/// Set whether `health_check()` succeeds or fails.
pub fn set_healthy(&self, healthy: bool) {
self.healthy.store(healthy, Ordering::Relaxed);
}
}
#[async_trait]
impl Channel for StubChannel {
fn name(&self) -> &str {
&self.name
}
async fn start(&self) -> Result<MessageStream, ChannelError> {
let rx = self
.rx
.lock()
.await
.take()
.ok_or_else(|| ChannelError::StartupFailed {
name: self.name.clone(),
reason: "start() already called".to_string(),
})?;
let stream = tokio_stream::wrappers::ReceiverStream::new(rx);
Ok(Box::pin(stream))
}
async fn respond(
&self,
msg: &IncomingMessage,
response: OutgoingResponse,
) -> Result<(), ChannelError> {
self.responses
.lock()
.expect("poisoned")
.push((msg.clone(), response));
Ok(())
}
async fn send_status(
&self,
status: StatusUpdate,
_metadata: &serde_json::Value,
) -> Result<(), ChannelError> {
self.statuses.lock().expect("poisoned").push(status);
Ok(())
}
async fn health_check(&self) -> Result<(), ChannelError> {
if self.healthy.load(Ordering::Relaxed) {
Ok(())
} else {
Err(ChannelError::HealthCheckFailed {
name: self.name.clone(),
})
}
}
}
/// Assembled test components.
pub struct TestHarness {
/// The agent dependencies, ready for use.
pub deps: AgentDeps,
/// Direct reference to the database (as `Arc<dyn Database>`).
pub db: Arc<dyn Database>,
/// Stub channel sender + manager, present if `with_stub_channel()` was called.
pub channel: Option<(mpsc::Sender<IncomingMessage>, ChannelManager)>,
/// Temp directory guard — keeps the test database alive. Dropped
/// automatically when the harness goes out of scope.
#[cfg(feature = "libsql")]
@@ -214,6 +346,7 @@ pub struct TestHarnessBuilder {
db: Option<Arc<dyn Database>>,
llm: Option<Arc<dyn LlmProvider>>,
tools: Option<Arc<ToolRegistry>>,
stub_channel: bool,
}
impl TestHarnessBuilder {
@@ -223,6 +356,7 @@ impl TestHarnessBuilder {
db: None,
llm: None,
tools: None,
stub_channel: false,
}
}
@@ -244,6 +378,15 @@ impl TestHarnessBuilder {
self
}
/// Include a `StubChannel` wired into a `ChannelManager`.
///
/// The harness will expose the sender (for injecting messages) and
/// the manager (for routing responses) via [`TestHarness::channel`].
pub fn with_stub_channel(mut self) -> Self {
self.stub_channel = true;
self
}
/// Build the harness with defaults applied.
#[cfg(feature = "libsql")]
pub async fn build(self) -> TestHarness {
@@ -280,6 +423,15 @@ impl TestHarnessBuilder {
max_actions_per_hour: None,
}));
let channel = if self.stub_channel {
let (stub, sender) = StubChannel::new("stub");
let manager = ChannelManager::new();
manager.add(Box::new(stub)).await;
Some((sender, manager))
} else {
None
};
let deps = AgentDeps {
store: Some(Arc::clone(&db)),
llm,
@@ -300,6 +452,7 @@ impl TestHarnessBuilder {
TestHarness {
deps,
db,
channel,
_temp_dir: temp_dir,
}
}
@@ -653,6 +806,48 @@ mod tests {
assert_eq!(response.finish_reason, FinishReason::Stop);
}
#[tokio::test]
async fn test_stub_channel_inject_and_capture() {
use futures::StreamExt;
let (channel, sender) = StubChannel::new("test-channel");
// Start the channel to get the message stream
let mut stream = channel.start().await.expect("start failed");
// Inject a message
sender
.send(IncomingMessage::new("test-channel", "user1", "hello"))
.await
.expect("send failed");
// Read it from the stream
let msg = stream.next().await.expect("stream ended");
assert_eq!(msg.content, "hello");
assert_eq!(msg.user_id, "user1");
assert_eq!(msg.channel, "test-channel");
// Send a response and verify it was captured
let response = OutgoingResponse::text("world");
channel
.respond(&msg, response)
.await
.expect("respond failed");
let captured = channel.captured_responses();
assert_eq!(captured.len(), 1);
assert_eq!(captured[0].1.content, "world");
}
#[tokio::test]
async fn test_stub_channel_health_check() {
let (channel, _sender) = StubChannel::new("healthy");
channel.health_check().await.expect("health check failed");
channel.set_healthy(false);
assert!(channel.health_check().await.is_err());
}
// === Database CRUD coverage for untested trait methods ===
#[cfg(feature = "libsql")]
@@ -705,6 +900,24 @@ mod tests {
assert!(!deleted);
}
#[tokio::test]
async fn test_harness_with_channel() {
let harness = TestHarnessBuilder::new().with_stub_channel().build().await;
let (sender, channel_manager) =
harness.channel.as_ref().expect("channel should be present");
// Inject a message via sender
sender
.send(IncomingMessage::new("stub", "user1", "test message"))
.await
.expect("send failed");
// Verify channel is registered in the manager
let names = channel_manager.channel_names().await;
assert!(names.contains(&"stub".to_string()));
}
#[cfg(feature = "libsql")]
#[tokio::test]
async fn test_settings_bulk_operations() {