Files
optimclaw/tests/e2e_telegram_message_routing.rs
T
4c043bf057 feat: complete multi-tenant isolation — phases 2–4 (#1614)
* feat: complete multi-tenant isolation — per-user budgets, model selection, heartbeat cycling

Finishes the remaining isolation work from phases 2–4 of #59:

Phase 2 (DB scoping): Fix /status and /list commands to use _for_user
DB variants instead of global queries that leaked cross-user job data.

Phase 3 (Runtime isolation): Per-user workspace in routine engine's
spawn_fire so lightweight routines run in the correct user context.
Per-user daily cost tracking in CostGuard with configurable budget via
MAX_COST_PER_USER_PER_DAY_CENTS. Multi-user heartbeat that cycles
through all users with routines, auto-detected from GATEWAY_USER_TOKENS.

Phase 4 (Provider/tools): Per-user model selection via preferred_model
setting — looked up from SettingsStore on first iteration, threaded
through ReasoningContext.model_override to CompletionRequest. Works
with providers that support per-request model overrides (NearAI).

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* fix: use selected_model setting key to match /model command persistence

The dispatcher was reading "preferred_model" but the /model command
(merged from staging) persists to "selected_model". Since set_setting
is already per-user scoped, using the same key makes /model work as
the per-user model override in multi-tenant mode.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* fix: heartbeat hygiene, /model multi-tenant guard, RigAdapter model override

Three follow-up fixes for multi-tenant isolation:

1. Multi-user heartbeat now runs memory hygiene per user before each
   heartbeat check, matching single-user heartbeat behavior.

2. /model command in multi-tenant mode only persists to per-user
   settings (selected_model) without calling set_model() on the shared
   LlmProvider. The per-request model_override in the dispatcher reads
   from the same setting. Added multi_tenant flag to AgentConfig
   (auto-detected from GATEWAY_USER_TOKENS).

3. RigAdapter now supports per-request model overrides by injecting the
   model name into rig-core's additional_params. OpenAI/Anthropic/Ollama
   API servers use last-key-wins for duplicate JSON keys, so the override
   takes effect via serde's flatten serialization order.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* fix: address PR review — cost model attribution, heartbeat concurrency, pruning

Fixes from review comments on #1614:

- Cost tracking now uses the override model name (not active_model_name)
  when a per-user model override is active, for accurate attribution.
- Multi-user heartbeat runs per-user checks concurrently via JoinSet
  instead of sequentially, preventing one slow user from blocking others.
- Per-user failure counts tracked independently; users exceeding
  max_failures are skipped (matching single-user semantics).
- per_user_daily_cost HashMap pruned on day rollover to prevent
  unbounded growth in long-lived deployments.
- Doc comment fixed: says "routines" not "active routines".

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* fix: /status ownership, model persistence scoping, heartbeat robustness

Addresses second round of PR review on #1614:

- /status <job_id> DB path now validates job.user_id == requesting user
  before returning data (was missing ownership check, security fix).

- persist_selected_model takes user_id param instead of owner_id, and
  skips .env/TOML writes in multi-tenant mode (these are shared global
  files). handle_system_command now receives user_id from caller.

- JoinSet collection handles Err(JoinError) explicitly instead of
  silently dropping panicked tasks.

- Notification forwarder extracts owner_id from response metadata in
  multi-tenant mode for per-user routing instead of broadcasting to
  the agent owner.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* fix: cost pricing, fire_manual workspace, heartbeat concurrency cap

Round 3 review fixes:

- Cost tracking passes None for cost_per_token when model override is
  active, letting CostGuard look up pricing by model name instead of
  using the default provider's rates (serrrfirat).

- fire_manual() now uses per-user workspace, matching spawn_fire()
  pattern (serrrfirat).

- Removed MULTI_TENANT env var — multi-tenant mode is auto-detected
  solely from GATEWAY_USER_TOKENS presence (serrrfirat + Copilot).

- Multi-user heartbeat capped at 8 concurrent tasks to avoid flooding
  the LLM provider (serrrfirat + Copilot).

- Fixed inject_model_override doc comment accuracy (Copilot).

- Added comment explaining multi-tenant notification routing priority
  (Copilot).

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* feat: user-scoped webhook endpoint for multi-tenant isolation

Adds POST /api/webhooks/u/{user_id}/{path} — a user-scoped webhook
endpoint that filters the routine lookup by user_id, preventing
cross-user webhook triggering when paths collide.

The existing /api/webhooks/{path} endpoint remains unchanged for
backward compatibility in single-user deployments.

Changes:
- get_webhook_routine_by_path gains user_id: Option<&str> param
- Both postgres and libsql implementations add AND user_id = ? filter
  when user_id is provided
- New webhook_trigger_user_scoped_handler extracts (user_id, path)
  from URL and passes to shared fire_webhook_inner logic
- Route registered on public router (webhooks are called by external
  services that can't send bearer tokens)

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* feat: add TenantCtx for compile-time tenant isolation

Implements zmanian's architectural proposal from #1614 review:
two-tier scoped database access (TenantScope/AdminScope) so handler
code cannot accidentally bypass tenant scoping.

TenantScope (default): wraps user_id + Arc<dyn Database>, auto-binds
user_id on every operation. ID-based lookups return None for cross-
tenant resources. No escape hatch — forgetting to scope is a compile
error.

AdminScope (explicit opt-in): cross-tenant access for system-level
components (heartbeat, routine engine, self-repair, scheduler, worker).

TenantCtx bundles TenantScope + workspace + cost guard + per-user
rate limiting. Constructed once per request in handle_message, threaded
through all command handlers and ChatDelegate.

Key changes:
- New src/tenant.rs (~920 lines): TenantScope, AdminScope, TenantCtx,
  TenantRateState, TenantRateRegistry
- All command handlers: user_id: &str → ctx: &TenantCtx
- ChatDelegate: cost check/record/settings via self.tenant
- System components: store field changed to AdminScope
- Config: TENANT_MAX_LLM_CONCURRENT, TENANT_MAX_JOBS_CONCURRENT env vars
- Fixes bug: /status <job_id> cross-tenant leak (now auto-filtered)

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>
2026-03-25 17:24:48 -07:00

358 lines
12 KiB
Rust

//! E2E tests for Telegram message routing through the real agent + message tool.
#[cfg(feature = "libsql")]
mod support;
#[cfg(feature = "libsql")]
mod tests {
use std::sync::Arc;
use std::time::Duration;
use async_trait::async_trait;
use futures::StreamExt;
use ironclaw::agent::{Agent, AgentDeps};
use ironclaw::app::{AppBuilder, AppBuilderFlags};
use ironclaw::channels::web::log_layer::LogBroadcaster;
use ironclaw::channels::{
Channel, ChannelManager, IncomingMessage, MessageStream, OutgoingResponse, StatusUpdate,
};
use ironclaw::config::Config;
use ironclaw::db::{Database, libsql::LibSqlBackend};
use ironclaw::error::ChannelError;
use ironclaw::llm::{LlmProvider, SessionConfig, SessionManager};
use tokio::sync::{Mutex, mpsc};
use tokio_stream::wrappers::ReceiverStream;
use crate::support::test_channel::{TestChannel, TestChannelHandle};
use crate::support::trace_llm::{LlmTrace, TraceLlm, TraceResponse, TraceStep, TraceToolCall};
type TelegramCaptures = Arc<Mutex<Vec<(String, OutgoingResponse)>>>;
struct RecordingTelegramChannel {
captures: TelegramCaptures,
}
impl RecordingTelegramChannel {
fn new() -> (Self, TelegramCaptures) {
let captures = Arc::new(Mutex::new(Vec::new()));
(
Self {
captures: Arc::clone(&captures),
},
captures,
)
}
}
#[async_trait]
impl Channel for RecordingTelegramChannel {
fn name(&self) -> &str {
"telegram"
}
async fn start(&self) -> Result<MessageStream, ChannelError> {
let (_tx, rx) = mpsc::channel::<IncomingMessage>(1);
Ok(ReceiverStream::new(rx).boxed())
}
async fn respond(
&self,
_msg: &IncomingMessage,
response: OutgoingResponse,
) -> Result<(), ChannelError> {
self.captures
.lock()
.await
.push(("respond".to_string(), response));
Ok(())
}
async fn send_status(
&self,
_status: StatusUpdate,
_metadata: &serde_json::Value,
) -> Result<(), ChannelError> {
Ok(())
}
async fn broadcast(
&self,
user_id: &str,
response: OutgoingResponse,
) -> Result<(), ChannelError> {
self.captures
.lock()
.await
.push((user_id.to_string(), response));
Ok(())
}
async fn health_check(&self) -> Result<(), ChannelError> {
Ok(())
}
}
struct Harness {
gateway: Arc<TestChannel>,
telegram_captures: Arc<Mutex<Vec<(String, OutgoingResponse)>>>,
db: Arc<dyn Database>,
owner_id: String,
_temp_dir: tempfile::TempDir,
agent_handle: Option<tokio::task::JoinHandle<()>>,
}
impl Harness {
async fn store_telegram_owner_binding(&self, owner_id: i64) {
for scope in [&self.owner_id, "test-user"] {
self.db
.set_setting(
scope,
"channels.wasm_channel_owner_ids.telegram",
&serde_json::json!(owner_id),
)
.await
.expect("failed to store telegram owner binding");
}
}
async fn wait_for_telegram_broadcasts(
&self,
expected: usize,
timeout: Duration,
) -> Vec<(String, OutgoingResponse)> {
let deadline = tokio::time::Instant::now() + timeout;
loop {
let snapshot = self.telegram_captures.lock().await.clone();
if snapshot.len() >= expected || tokio::time::Instant::now() >= deadline {
return snapshot;
}
tokio::time::sleep(Duration::from_millis(50)).await;
}
}
}
impl Drop for Harness {
fn drop(&mut self) {
self.gateway.signal_shutdown();
if let Some(handle) = self.agent_handle.take() {
handle.abort();
}
}
}
async fn build_harness(trace: LlmTrace) -> Harness {
let temp_dir = tempfile::tempdir().expect("failed to create temp dir");
let db_path = temp_dir.path().join("telegram_message_routing.db");
let backend = LibSqlBackend::new_local(&db_path)
.await
.expect("failed to create test LibSqlBackend");
backend
.run_migrations()
.await
.expect("failed to run migrations");
let db: Arc<dyn Database> = Arc::new(backend);
let skills_dir = temp_dir.path().join("skills");
let installed_skills_dir = temp_dir.path().join("installed_skills");
let _ = std::fs::create_dir_all(&skills_dir);
let _ = std::fs::create_dir_all(&installed_skills_dir);
let mut config = Config::for_testing(db_path, skills_dir, installed_skills_dir);
config.agent.auto_approve_tools = true;
let session = Arc::new(SessionManager::new(SessionConfig::default()));
let log_broadcaster = Arc::new(LogBroadcaster::new());
let llm: Arc<dyn LlmProvider> = Arc::new(TraceLlm::from_trace(trace));
let mut builder = AppBuilder::new(
config,
AppBuilderFlags::default(),
None,
session,
log_broadcaster,
);
builder.with_database(Arc::clone(&db));
builder.with_llm(llm);
let mut components = builder
.build_all()
.await
.expect("AppBuilder::build_all() failed");
components.config.agent.auto_approve_tools = true;
components.config.agent.allow_local_tools = true;
let deps = AgentDeps {
owner_id: components.config.owner_id.clone(),
store: components.db.clone(),
llm: components.llm.clone(),
cheap_llm: components.cheap_llm.clone(),
safety: components.safety.clone(),
tools: components.tools.clone(),
workspace: components.workspace.clone(),
extension_manager: components.extension_manager.clone(),
skill_registry: components.skill_registry.clone(),
skill_catalog: components.skill_catalog.clone(),
skills_config: components.config.skills.clone(),
hooks: components.hooks.clone(),
cost_guard: components.cost_guard.clone(),
sse_tx: None,
http_interceptor: None,
transcription: None,
document_extraction: None,
sandbox_readiness: ironclaw::agent::SandboxReadiness::DisabledByConfig,
builder: None,
llm_backend: "nearai".to_string(),
tenant_rates: std::sync::Arc::new(ironclaw::tenant::TenantRateRegistry::new(4, 3)),
};
let gateway = Arc::new(TestChannel::new());
let gateway_handle = TestChannelHandle::new(Arc::clone(&gateway));
let (telegram_channel, telegram_captures) = RecordingTelegramChannel::new();
let channel_manager = ChannelManager::new();
channel_manager.add(Box::new(gateway_handle)).await;
channel_manager.add(Box::new(telegram_channel)).await;
let channels = Arc::new(channel_manager);
deps.tools
.register_message_tools(Arc::clone(&channels), deps.extension_manager.clone())
.await;
let agent = Agent::new(
components.config.agent.clone(),
deps,
channels,
None,
None,
None,
Some(Arc::clone(&components.context_manager)),
None,
);
let agent_handle = tokio::spawn(async move {
if let Err(err) = agent.run().await {
eprintln!("[telegram routing e2e] Agent exited with error: {err}");
}
});
if let Some(rx) = gateway.take_ready_rx().await {
let _ = tokio::time::timeout(Duration::from_secs(5), rx).await;
}
Harness {
gateway,
telegram_captures,
db,
owner_id: components.config.owner_id.clone(),
_temp_dir: temp_dir,
agent_handle: Some(agent_handle),
}
}
fn single_message_trace(arguments: serde_json::Value, final_text: &str) -> LlmTrace {
LlmTrace::single_turn(
"telegram-message-routing",
"send a reminder",
vec![
TraceStep {
request_hint: None,
response: TraceResponse::ToolCalls {
tool_calls: vec![TraceToolCall {
id: "call_message_1".to_string(),
name: "message".to_string(),
arguments,
}],
input_tokens: 32,
output_tokens: 12,
},
expected_tool_results: Vec::new(),
},
TraceStep {
request_hint: None,
response: TraceResponse::Text {
content: final_text.to_string(),
input_tokens: 24,
output_tokens: 8,
},
expected_tool_results: Vec::new(),
},
],
)
}
#[tokio::test]
async fn telegram_message_tool_uses_bound_owner_target_when_target_omitted() {
let harness = build_harness(single_message_trace(
serde_json::json!({
"content": "Walk Conan",
"channel": "telegram",
}),
"Sent on Telegram.",
))
.await;
harness.store_telegram_owner_binding(424242).await;
harness
.gateway
.send_message("remind me to walk conan")
.await;
let responses = harness
.gateway
.wait_for_responses(1, Duration::from_secs(10))
.await;
assert!(
responses
.iter()
.any(|response| response.content.contains("Sent on Telegram")),
"expected assistant confirmation, got: {:?}",
responses
.iter()
.map(|response| &response.content)
.collect::<Vec<_>>()
);
let broadcasts = harness
.wait_for_telegram_broadcasts(1, Duration::from_secs(10))
.await;
assert_eq!(
broadcasts.len(),
1,
"expected exactly one telegram broadcast"
);
assert_eq!(broadcasts[0].0, "424242");
assert_eq!(broadcasts[0].1.content, "Walk Conan");
}
#[tokio::test]
async fn telegram_message_tool_prefers_explicit_target_over_bound_owner_target() {
let harness = build_harness(single_message_trace(
serde_json::json!({
"content": "Walk Conan",
"channel": "telegram",
"target": "999999",
}),
"Sent on Telegram.",
))
.await;
harness.store_telegram_owner_binding(424242).await;
harness.gateway.send_message("send the reminder").await;
let _ = harness
.gateway
.wait_for_responses(1, Duration::from_secs(10))
.await;
let broadcasts = harness
.wait_for_telegram_broadcasts(1, Duration::from_secs(10))
.await;
assert_eq!(
broadcasts.len(),
1,
"expected exactly one telegram broadcast"
);
assert_eq!(broadcasts[0].0, "999999");
assert_eq!(broadcasts[0].1.content, "Walk Conan");
}
}