feat: Sandbox jobs (#4)

* Orchestrating jobs and running them in sandboxes

* Fix heartbeat: dynamic max_tokens, empty content guard, notification fallback

- Query /v1/models API for context_length and set max_tokens to half
  (floor 4096) instead of hardcoded 1024; reasoning models like GLM-4.7
  need much larger budgets
- Guard against empty LLM content (reasoning models can burn all tokens
  on chain-of-thought and return content: null)
- Simplify notification routing: try configured channel first, fall back
  to broadcast_all so heartbeat alerts always reach someone
- Add ModelMetadata struct and model_metadata() to LlmProvider trait
- Refactor NearAiChatProvider::list_models into shared fetch_models()
- Add standalone test_heartbeat example for isolated debugging

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* Add job detail view with drill-down from jobs list

Click a job row to see full details across four sub-tabs:
Overview (metadata grid, description, state transitions timeline),
Actions (expandable tool call cards with input/output JSON),
Thinking (conversation messages styled by role), and
Files (embedded workspace tree browser).

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* Strip model-internal XML tags from LLM responses, fix Telegram parse_mode 400

Some models (GLM-4.7, etc.) emit <tool_call>tool_list</tool_call> in the
content field instead of using the OpenAI tool_calls array. This XML leaks
through to channels as text, and Telegram's Markdown parser chokes on the
underscores, returning 400 "can't parse entities".

Two fixes:
- Generalize clean_response() to strip <tool_call>, <function_call>,
  <tool_calls>, and pipe-delimited variants (<|tool_call|>) alongside
  the existing <thinking> tag stripping
- Add Telegram send_message helper with parse_mode fallback: try Markdown
  first, retry as plain text on "can't parse entities" 400 errors

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* Add SystemCommand submission type for thread-state-independent commands

System commands (/help, /model, /version, /tools, /ping, /debug) now
bypass thread-state checks and safety validation via a dedicated
Submission::SystemCommand variant. Previously these flowed through
process_user_input() which blocked them during Processing/AwaitingApproval
/Completed states.

- Add /model [name] for runtime model switching with provider validation
- Add active_model_name()/set_model() to LlmProvider trait with RwLock
  hot-swap in both NEAR AI providers
- Rewrite /help with aligned columns grouped by category
- Expand REPL tab-completion from 10 to 23 slash commands
- Remove REPL-local /help interception (now handled by agent)

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* Add per-tool execution timeouts, auto-create sandbox project dirs, serve built files

The sandbox e2e pipeline (agent -> container -> built website -> browsable URL)
was broken by three gaps: hardcoded 60s timeouts killed sandbox jobs that need
minutes, no auto-created project directory meant container output vanished, and
no HTTP route to browse the built files.

- Add `execution_timeout()` to the `Tool` trait (default 60s), replace all four
  hardcoded `Duration::from_secs(60)` call sites (agent_loop, worker, scheduler,
  worker/runtime) with the per-tool value
- Override to 660s in `RunInSandboxTool` (10 min polling + 60s buffer)
- Auto-create `~/.ironclaw/projects/{uuid}/` when no `project_dir` is specified,
  so every sandbox job gets a persistent bind mount
- Include `project_dir` and `browse_url` in sandbox tool output JSON
- Add `/projects/{id}` and `/projects/{id}/{path}` static file serving routes
  to the web gateway with path traversal protection and MIME type detection
- Add `mime_guess` dependency for content-type detection

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* Apply cargo fmt to wizard.rs after merge

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* Persist sandbox jobs in DB, fix web UI, unify job model

Sandbox container jobs were invisible to the web UI because they lived
only in ContainerJobManager's in-memory HashMap while the API queried
ContextManager. This persists them to the agent_jobs table and fixes
all six front-end bugs (empty job list, broken back button, empty
actions/thinking tabs, wrong files tab, stuck status, no persistence).

Key changes:
- V4 migration adds project_dir and user_id columns to agent_jobs
- Embedded migrations via refinery (no external CLI needed)
- SandboxJobRecord CRUD in Store with fire-and-forget DB writes
- Unified job_id: sandbox tool generates UUID, passes to ContainerJobManager
- Web API queries DB for sandbox jobs, merges with ContextManager direct jobs
- New endpoints: restart, project file list/read with path traversal protection
- Front-end: rebuild DOM on back navigation, sandbox-aware tabs, job cards in
  chat stream, source badges, restart button for failed/interrupted jobs
- Gateway defaults to enabled, prints Web UI URL on startup
- Stale jobs marked "interrupted" on restart for visibility and restartability

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* Secure in-chat auth: tokens never touch the LLM or chat history

Remove the token parameter from tool_auth so the LLM cannot pass raw
API keys. Add dedicated REST (POST /api/chat/auth-token) and WebSocket
(auth_token) endpoints that route tokens directly to ext_mgr.auth(),
completely bypassing the message pipeline, turns, history, and compaction.

Web UI shows an auth card (password input + OAuth button) when the agent
enters auth mode, submitted via the dedicated endpoint. CLI auth mode
interception is unchanged (already secure).

New StatusUpdate::AuthRequired/AuthCompleted variants propagate through
all channels (SSE, WebSocket, REPL, WASM).

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* feat: Add Claude Code mode for sandbox jobs

Run Claude Code CLI inside Docker containers as an alternative to the
standard worker mode. The bridge spawns `claude -p` with stream-json
output, posts events to the orchestrator, and supports follow-up
prompts via `--resume`.

Key additions:
- `claude-bridge` CLI subcommand and ClaudeBridgeRuntime
- JobMode enum (Worker vs ClaudeCode) with per-mode container config
- Orchestrator endpoints for Claude events and prompt polling
- SSE event variants for real-time Claude Code streaming to frontend
- Claude Code sub-tab in web UI with terminal-style output and input bar
- Database migration for job_mode column and claude_code_events table
- ClaudeCodeConfig with env var support (CLAUDE_CODE_ENABLED, etc.)
- Mode parameter on run_in_sandbox tool schema

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: Skip create_job tool when sandbox is enabled to prevent duplicate jobs

When sandbox mode is on, the LLM would call create_job (creating a
pending "direct" entry) then run_in_sandbox (creating a second "sandbox"
entry), producing two jobs in the list for a single user request.

Now register_job_tools() skips create_job when sandbox is enabled since
run_in_sandbox already creates tracked jobs. Also improved the
run_in_sandbox description to guide the LLM to use it directly and to
mention wait=false for async execution.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* feat: Web gateway UI quality-of-life improvements

Phase 1: Send button disabled state to prevent double-sends, copy button
on code blocks, confirm() guards on destructive actions, SSE-driven job
list auto-refresh, log filters re-applied on tab switch, jobEvents memory
leak fix (cap at 500, cleanup after 60s).

Phase 2: Toast notification system replacing chat-based system messages,
memory search highlighting with centered snippets, keyboard shortcuts
(Ctrl+1-5 tabs, Ctrl+K focus, Ctrl+N new thread, Escape close/blur),
activity tab toolbar with event type filter and auto-scroll toggle.

Phase 3: Thread sidebar with load/switch/create, thread_id passed with
messages, collapsible to hamburger. Memory inline editing with textarea,
Save/Cancel, POST to /api/memory/write.

Phase 4: Gateway status popover on hover (polls every 30s), extension
install form (name/URL/kind), markdown rendering in memory viewer for
.md files, mobile responsive layout at 768px breakpoint.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* feat: Add routines system, remove non-sandbox job mode from web UI

Routines: scheduled & reactive job system with cron and event triggers,
lightweight (single LLM call) and full-job execution modes, guardrails
(cooldown, max concurrent, dedup), and LLM-facing tools for CRUD.

Web UI: remove ContextManager-backed "direct" job mode entirely. Jobs
are now exclusively sandbox-backed (DB + container). Simplify job detail
response, drop dead types (ActionInfo, MessageInfo, MessageToolCallInfo),
fix Browse Files CSS loading (trailing-slash redirect), fix Activity tab
event rendering.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: Re-enable chat input on agent completion, auto-auth on tool_activate, recover tool calls from content XML

Three fixes:

1. Chat input stays disabled after agent finishes: the "Done" status
   SSE event now calls enableChatInput() as a safety net when the
   response event is empty or lost. Same for auth_completed and
   cancelAuth().

2. tool_activate never triggers auth: when activation fails due to
   missing authentication, it now auto-initiates the auth flow
   (same pattern as the web API handler). detect_auth_awaiting()
   also matches tool_activate results now.

3. Models like GLM-4.7 emit tool calls as XML tags in content
   (<tool_call>tool_list</tool_call>) instead of using the structured
   tool_calls array. recover_tool_calls_from_content() extracts and
   validates these before falling back to plain text.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* feat: Add routines web UI tab, update docs for sandbox-jobs branch

Add full routines management to the web gateway (list, detail, trigger,
toggle, delete) with 7 new API endpoints, response types, and frontend
(HTML, JS, CSS). Update FEATURE_PARITY.md (~23 rows), CLAUDE.md (new
subsystems, config, TODOs), and README.md (architecture diagram,
features, components, fix onboard command).

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: Bind Telegram bot to owner account during setup

Without owner binding, anyone who discovers the bot can send it messages.
The setup wizard now prompts the user to message their bot, captures their
Telegram user ID via getUpdates, and persists it as telegram_owner_id in
settings. On startup, the owner_id is injected into the WASM channel config
so the existing owner restriction logic drops messages from non-owners.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* feat: Move settings from disk to PostgreSQL database

Settings previously lived in three JSON files on disk (settings.json,
mcp-servers.json, session.json). This made them inaccessible from the
web UI and caused redundant disk reads (Settings::load() called 8+
times during startup).

Now all settings live in a `settings` table (user_id + key -> JSONB)
with only 4 bootstrap fields remaining on disk (database_url, pool
size, secrets key source, onboard_completed) since they're needed
before the DB connection exists.

- Add V8 migration for settings table
- Add BootstrapConfig (thin disk file) and Settings DB round-trip
- Add Store CRUD methods for settings (get/set/delete/list/bulk)
- Refactor Config to load from DB (env > DB > default cascade)
- Add SessionManager DB persistence for session tokens
- Add DB-backed MCP server config load/save functions
- Add 6 settings web API endpoints (list/get/set/delete/export/import)
- Add one-time disk-to-DB migration on first boot
- Make CLI config commands async with DB access (disk fallback)

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* feat: Seed workspace on boot, fix gateway duplicate logs and URL auto-auth

- Add Workspace::seed_if_empty() to create core identity files (README,
  MEMORY, IDENTITY, SOUL, AGENTS, USER, HEARTBEAT) when missing, called
  on every boot without overwriting existing user edits
- Remove duplicate gateway log lines from web/mod.rs (main.rs has the
  useful clickable ?token= URL)
- Auto-authenticate from ?token= URL parameter in the web UI and strip
  the token from the address bar after successful auth

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: Harden sandbox security (path traversal + orchestrator auth)

Two vulnerabilities fixed:

1. project_dir path traversal: The create_job tool let the LLM specify
   arbitrary host paths for Docker bind mounts. Removed project_dir from
   the tool schema entirely, and added canonicalization + prefix validation
   at both resolve_project_dir() and the job_manager bind mount point.

2. Orchestrator API auth bypass: worker_auth_middleware was defined but
   never applied. Each handler manually called validate_token(), so any
   new endpoint that forgot would be publicly accessible. Applied the
   middleware as route_layer on all /worker/ routes, removed manual auth
   from all 7 handlers. Bind to 127.0.0.1 on macOS/Windows (Linux keeps
   0.0.0.0 since containers reach host via docker bridge, not loopback).

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* feat: Rework gateway chat with pinned assistant, pagination, and NEAR AI response chaining

Implements the 4-phase plan for overhauling the web gateway chat:

- Phase 1: Pinned "Assistant" thread at top of sidebar, regular threads below
- Phase 2: Cursor-based history pagination with infinite scroll
- Phase 3: NEAR AI previous_response_id chaining (delta-only messages),
  with fallback to full history on chain errors, and DB persistence of
  chain state across restarts
- Phase 4: SSE thread isolation (events filtered by thread_id)

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: Add per-request HTTP timeout to WASM host, redact credentials in errors

Three fixes for WASM channel reliability:

1. Per-request timeout: Add optional timeout-ms parameter to http-request
   in both channel and tool WIT interfaces. Telegram long-poll now specifies
   35s (outliving the 30s server-side hold), while regular API calls use
   the 30s default. Fixes the triple-30s timeout race that caused polling
   failures.

2. Credential redaction: reqwest::Error includes the full URL (with injected
   bot tokens) in its Display output. Scrub credential values from error
   messages before logging or returning to WASM.

3. Webhook route registration: Remove tunnel URL gate so webhook routes are
   always available when webhook channels exist, not only when TUNNEL_URL
   is configured.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* chore: Fix clippy warnings in WASM tools and channels

- slack channel: allow dead_code on signing_secret_name (forward compat field)
- gmail tool: use div_ceil() instead of manual (n+2)/3
- google-calendar tool: extract CreateEventParams/UpdateEventParams structs
  to fix too-many-arguments warnings

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* Fix approval flow

* fix: Rebuild bundled telegram.wasm with updated WIT interface

The bundled WASM binary must match the host's WIT definition.
Previous binary was compiled against the old 4-arg http-request;
this rebuild includes the new timeout-ms parameter.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* refactor: Load WASM channels from disk instead of bundling in binary

Remove include_bytes! embedding of telegram.wasm. Channels are now
loaded from their build output directories (channels-src/<name>/target/)
during onboarding, then from ~/.ironclaw/channels/ at runtime.

- bundled.rs: locate_channel_artifacts() finds WASM + capabilities from
  build output; IRONCLAW_CHANNELS_SRC env var overrides the default path
- available_channel_names(): only lists channels with build artifacts
- bundled_channel_names(): lists all known channels (manifest)
- Setup wizard uses available_channel_names() to offer installable channels
- Add *.wasm to .gitignore, remove tracked telegram.wasm

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: Persist gateway auth token, fix thread hydration race, polish auth screen

Three web gateway UX fixes:

1. Token persistence: Store auth token in sessionStorage so refreshing
   the page doesn't force re-authentication. Hide the auth screen
   immediately when a saved token exists to prevent flash.

2. Thread hydration: Remove the !msgs.is_empty() bail-out in
   maybe_hydrate_thread so that even brand-new (empty) assistant threads
   get hydrated with their correct DB UUID. Previously resolve_thread
   would mint a fresh UUID, causing messages to land in the wrong
   conversation and duplicate threads to appear.

3. Auth screen: Redesign as a centered card with brand, tagline, labeled
   input, and hint text.

Also adds 34 new tests covering session/thread lifecycle, thread
resolution isolation (user, channel, external ID), hydration edge cases,
serialization round-trips, approval flows, and stale mapping recovery.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix: Use bindgen! for WASM tool wrapper, add dev tool loading

Three changes:

1. Rewrite src/tools/wasm/wrapper.rs to use wasmtime::component::bindgen!
   instead of manual linker.root().func_wrap(). This fixes the
   "component imports instance 'near:agent/host', but a matching
   implementation was not found in the linker" error. All 6 host functions
   (log, now-millis, workspace-read, http-request, secret-exists,
   tool-invoke) are now properly registered under the near:agent/host
   namespace. Also adds WASI support, credential injection, and leak
   detection for HTTP requests made by WASM tools.

2. Add dev tool loading to src/tools/wasm/loader.rs. During startup, the
   loader now also scans tools-src/*/target/wasm32-wasip2/release/ for
   build artifacts that are newer than installed copies. This means during
   development you just rebuild the WASM and restart the host; no manual
   copy step needed. Set IRONCLAW_TOOLS_SRC to override the source dir.

3. Wire up load_dev_tools() in main.rs alongside the existing
   load_from_dir() call.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* feat: Wire main startup and CLI to use DB-backed settings

main.rs now reloads Config from the database after connecting,
attaches the store to the session manager for dual-write tokens,
and loads MCP servers from DB instead of disk. ExtensionManager
and MCP CLI commands use DB when available with disk fallback.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

---------

Co-authored-by: Claude Opus 4.6 <[email protected]>
This commit is contained in:
Illia Polosukhin
2026-02-11 08:31:25 +00:00
committed by GitHub
co-authored by Claude Opus 4.6
parent 202665a55c
commit ced83d5b4d
99 changed files with 17095 additions and 1162 deletions
+640 -116
View File
@@ -9,13 +9,14 @@ use uuid::Uuid;
use crate::agent::compaction::ContextCompactor;
use crate::agent::context_monitor::ContextMonitor;
use crate::agent::heartbeat::spawn_heartbeat;
use crate::agent::routine_engine::{RoutineEngine, spawn_cron_ticker};
use crate::agent::self_repair::{DefaultSelfRepair, RepairResult, SelfRepair};
use crate::agent::session::{PendingApproval, Session, ThreadState};
use crate::agent::session_manager::SessionManager;
use crate::agent::submission::{Submission, SubmissionParser, SubmissionResult};
use crate::agent::{HeartbeatConfig as AgentHeartbeatConfig, MessageIntent, Router, Scheduler};
use crate::channels::{ChannelManager, IncomingMessage, OutgoingResponse, StatusUpdate};
use crate::config::{AgentConfig, HeartbeatConfig};
use crate::config::{AgentConfig, HeartbeatConfig, RoutineConfig};
use crate::context::ContextManager;
use crate::context::JobContext;
use crate::error::Error;
@@ -77,6 +78,7 @@ pub struct Agent {
session_manager: Arc<SessionManager>,
context_monitor: ContextMonitor,
heartbeat_config: Option<HeartbeatConfig>,
routine_config: Option<RoutineConfig>,
}
impl Agent {
@@ -89,6 +91,7 @@ impl Agent {
deps: AgentDeps,
channels: ChannelManager,
heartbeat_config: Option<HeartbeatConfig>,
routine_config: Option<RoutineConfig>,
context_manager: Option<Arc<ContextManager>>,
session_manager: Option<Arc<SessionManager>>,
) -> Self {
@@ -116,6 +119,7 @@ impl Agent {
session_manager,
context_monitor: ContextMonitor::new(),
heartbeat_config,
routine_config,
}
}
@@ -256,53 +260,28 @@ impl Agent {
let channels = self.channels.clone();
tokio::spawn(async move {
while let Some(response) = notify_rx.recv().await {
// Route notification to configured channel/user, or broadcast to all
match (&notify_channel, &notify_user) {
(Some(channel), Some(user)) => {
// Send to specific channel and user
if let Err(e) =
channels.broadcast(channel, user, response.clone()).await
{
let user = notify_user.as_deref().unwrap_or("default");
// Try the configured channel first, fall back to
// broadcasting on all channels.
let targeted_ok = if let Some(ref channel) = notify_channel {
channels
.broadcast(channel, user, response.clone())
.await
.is_ok()
} else {
false
};
if !targeted_ok {
let results = channels.broadcast_all(user, response).await;
for (ch, result) in results {
if let Err(e) = result {
tracing::warn!(
"Failed to send heartbeat to {}/{}: {}",
channel,
user,
"Failed to broadcast heartbeat to {}: {}",
ch,
e
);
} else {
tracing::debug!(
"Heartbeat notification sent to {}/{}",
channel,
user
);
}
}
(None, Some(user)) => {
// Broadcast to all channels for this user
let results = channels.broadcast_all(user, response).await;
for (ch, result) in results {
if let Err(e) = result {
tracing::warn!(
"Failed to broadcast heartbeat to {}: {}",
ch,
e
);
}
}
}
_ => {
// No explicit target, broadcast to all channels
// for the default user so notifications actually
// reach someone instead of vanishing into logs.
let results = channels.broadcast_all("default", response).await;
for (ch, result) in results {
if let Err(e) = result {
tracing::warn!(
"Failed to broadcast heartbeat to {}: {}",
ch,
e
);
}
}
}
}
@@ -330,6 +309,85 @@ impl Agent {
None
};
// Spawn routine engine if enabled
let routine_handle = if let Some(ref rt_config) = self.routine_config {
if rt_config.enabled {
if let (Some(store), Some(workspace)) = (self.store(), self.workspace()) {
// Set up notification channel (same pattern as heartbeat)
let (notify_tx, mut notify_rx) =
tokio::sync::mpsc::channel::<OutgoingResponse>(32);
let engine = Arc::new(RoutineEngine::new(
rt_config.clone(),
Arc::clone(store),
self.llm().clone(),
Arc::clone(workspace),
notify_tx,
));
// Register routine tools
self.deps
.tools
.register_routine_tools(Arc::clone(store), Arc::clone(&engine));
// Load initial event cache
engine.refresh_event_cache().await;
// Spawn notification forwarder
let channels = self.channels.clone();
tokio::spawn(async move {
while let Some(response) = notify_rx.recv().await {
let user = response
.metadata
.get("notify_user")
.and_then(|v| v.as_str())
.unwrap_or("default")
.to_string();
let results = channels.broadcast_all(&user, response).await;
for (ch, result) in results {
if let Err(e) = result {
tracing::warn!(
"Failed to broadcast routine notification to {}: {}",
ch,
e
);
}
}
}
});
// Spawn cron ticker
let cron_interval =
std::time::Duration::from_secs(rt_config.cron_check_interval_secs);
let cron_handle = spawn_cron_ticker(Arc::clone(&engine), cron_interval);
// Store engine reference for event trigger checking
// Safety: we're in run() which takes self, no other reference exists
let engine_ref = Arc::clone(&engine);
// SAFETY: self is consumed by run(), we can smuggle the engine in
// via a local to use in the message loop below.
tracing::info!(
"Routines enabled: cron ticker every {}s, max {} concurrent",
rt_config.cron_check_interval_secs,
rt_config.max_concurrent_routines
);
Some((cron_handle, engine_ref))
} else {
tracing::warn!("Routines enabled but store/workspace not available");
None
}
} else {
None
}
} else {
None
};
// Extract engine ref for use in message loop
let routine_engine_for_loop = routine_handle.as_ref().map(|(_, e)| Arc::clone(e));
// Main message loop
tracing::info!("Agent {} ready and listening", self.config.name);
@@ -374,6 +432,14 @@ impl Agent {
.await;
}
}
// Check event triggers (cheap in-memory regex, fires async if matched)
if let Some(ref engine) = routine_engine_for_loop {
let fired = engine.check_event_triggers(&message).await;
if fired > 0 {
tracing::debug!("Fired {} event-triggered routines", fired);
}
}
}
// Cleanup
@@ -383,6 +449,9 @@ impl Agent {
if let Some(handle) = heartbeat_handle {
handle.abort();
}
if let Some((cron_handle, _)) = routine_handle {
cron_handle.abort();
}
self.scheduler.stop_all().await;
self.channels.shutdown_all().await?;
@@ -393,6 +462,11 @@ impl Agent {
// Parse submission type first
let submission = SubmissionParser::parse(&message.content);
// Hydrate thread from DB if it's a historical thread not in memory
if let Some(ref external_thread_id) = message.thread_id {
self.maybe_hydrate_thread(message, external_thread_id).await;
}
// Resolve session and thread
let (session, thread_id) = self
.session_manager
@@ -444,6 +518,9 @@ impl Agent {
self.process_user_input(message, session, thread_id, &content)
.await
}
Submission::SystemCommand { command, args } => {
self.handle_system_command(&command, &args).await
}
Submission::Undo => self.process_undo(session, thread_id).await,
Submission::Redo => self.process_redo(session, thread_id).await,
Submission::Interrupt => self.process_interrupt(session, thread_id).await,
@@ -515,6 +592,107 @@ impl Agent {
}
}
/// Hydrate a historical thread from DB into memory if not already present.
///
/// Called before `resolve_thread` so that the session manager finds the
/// thread on lookup instead of creating a new one.
///
/// Creates an in-memory thread with the exact UUID the frontend sent,
/// even when the conversation has zero messages (e.g. a brand-new
/// assistant thread). Without this, `resolve_thread` would mint a
/// fresh UUID and all messages would land in the wrong conversation.
async fn maybe_hydrate_thread(&self, message: &IncomingMessage, external_thread_id: &str) {
// Only hydrate UUID-shaped thread IDs (web gateway uses UUIDs)
let thread_uuid = match Uuid::parse_str(external_thread_id) {
Ok(id) => id,
Err(_) => return,
};
// Check if already in memory
let session = self
.session_manager
.get_or_create_session(&message.user_id)
.await;
{
let sess = session.lock().await;
if sess.threads.contains_key(&thread_uuid) {
return;
}
}
// Load history from DB (may be empty for a newly created thread).
let mut chat_messages: Vec<ChatMessage> = Vec::new();
let msg_count;
if let Some(store) = self.store() {
let db_messages = store
.list_conversation_messages(thread_uuid)
.await
.unwrap_or_default();
msg_count = db_messages.len();
chat_messages = db_messages
.iter()
.filter_map(|m| match m.role.as_str() {
"user" => Some(ChatMessage::user(&m.content)),
"assistant" => Some(ChatMessage::assistant(&m.content)),
_ => None,
})
.collect();
} else {
msg_count = 0;
}
// Create thread with the historical ID and restore messages
let session_id = {
let sess = session.lock().await;
sess.id
};
let mut thread = crate::agent::session::Thread::with_id(thread_uuid, session_id);
if !chat_messages.is_empty() {
thread.restore_from_messages(chat_messages);
}
// Restore response chain from conversation metadata
if let Some(store) = self.store() {
if let Ok(Some(metadata)) = store.get_conversation_metadata(thread_uuid).await {
if let Some(rid) = metadata
.get("last_response_id")
.and_then(|v| v.as_str())
.map(String::from)
{
thread.last_response_id = Some(rid.clone());
self.llm()
.seed_response_chain(&thread_uuid.to_string(), rid);
tracing::debug!("Restored response chain for thread {}", thread_uuid);
}
}
}
// Insert into session and register with session manager
{
let mut sess = session.lock().await;
sess.threads.insert(thread_uuid, thread);
sess.active_thread = Some(thread_uuid);
sess.last_active_at = chrono::Utc::now();
}
self.session_manager
.register_thread(
&message.user_id,
&message.channel,
thread_uuid,
Arc::clone(&session),
)
.await;
tracing::debug!(
"Hydrated thread {} from DB ({} messages)",
thread_uuid,
msg_count
);
}
async fn process_user_input(
&self,
message: &IncomingMessage,
@@ -694,6 +872,7 @@ impl Agent {
match result {
Ok(AgenticLoopResult::Response(response)) => {
thread.complete_turn(&response);
self.persist_response_chain(thread);
let _ = self
.channels
.send_status(
@@ -702,6 +881,10 @@ impl Agent {
&message.metadata,
)
.await;
// Fire-and-forget: persist turn to DB
self.persist_turn(thread_id, &message.user_id, content, Some(&response));
Ok(SubmissionResult::response(response))
}
Ok(AgenticLoopResult::NeedApproval { pending }) => {
@@ -728,11 +911,95 @@ impl Agent {
}
Err(e) => {
thread.fail_turn(e.to_string());
// Persist the user message even on failure
self.persist_turn(thread_id, &message.user_id, content, None);
Ok(SubmissionResult::error(e.to_string()))
}
}
}
/// Fire-and-forget: persist a turn (user message + optional assistant response) to the DB.
fn persist_turn(
&self,
thread_id: Uuid,
user_id: &str,
user_input: &str,
response: Option<&str>,
) {
let store = match self.store() {
Some(s) => Arc::clone(s),
None => return,
};
let user_id = user_id.to_string();
let user_input = user_input.to_string();
let response = response.map(String::from);
tokio::spawn(async move {
if let Err(e) = store
.ensure_conversation(thread_id, "gateway", &user_id, None)
.await
{
tracing::warn!("Failed to ensure conversation {}: {}", thread_id, e);
return;
}
if let Err(e) = store
.add_conversation_message(thread_id, "user", &user_input)
.await
{
tracing::warn!("Failed to persist user message: {}", e);
return;
}
if let Some(ref resp) = response {
if let Err(e) = store
.add_conversation_message(thread_id, "assistant", resp)
.await
{
tracing::warn!("Failed to persist assistant message: {}", e);
}
}
});
}
/// Sync the provider's response chain ID to the thread and DB metadata.
///
/// Call after a successful agentic loop to persist the latest
/// `previous_response_id` so chaining survives restarts.
fn persist_response_chain(&self, thread: &mut crate::agent::session::Thread) {
let tid = thread.id.to_string();
let response_id = match self.llm().get_response_chain_id(&tid) {
Some(rid) => rid,
None => return,
};
// Update in-memory thread
thread.last_response_id = Some(response_id.clone());
// Fire-and-forget DB write
let store = match self.store() {
Some(s) => Arc::clone(s),
None => return,
};
let thread_id = thread.id;
tokio::spawn(async move {
let val = serde_json::json!(response_id);
if let Err(e) = store
.update_conversation_metadata_field(thread_id, "last_response_id", &val)
.await
{
tracing::warn!(
"Failed to persist response chain for thread {}: {}",
thread_id,
e
);
}
});
}
/// Run the agentic loop: call LLM, execute tools, repeat until text response.
///
/// Returns `AgenticLoopResult::Response` on completion, or
@@ -808,7 +1075,12 @@ impl Agent {
// Call LLM with current context
let context = ReasoningContext::new()
.with_messages(context_messages.clone())
.with_tools(tool_defs);
.with_tools(tool_defs)
.with_metadata({
let mut m = std::collections::HashMap::new();
m.insert("thread_id".to_string(), thread_id.to_string());
m
});
let result = reasoning.respond_with_tools(&context).await?;
@@ -832,13 +1104,16 @@ impl Agent {
// Tools have been executed or we've tried multiple times, return response
return Ok(AgenticLoopResult::Response(text));
}
RespondResult::ToolCalls(tool_calls) => {
RespondResult::ToolCalls {
tool_calls,
content,
} => {
tools_executed = true;
// Add the assistant message with tool_calls to context.
// OpenAI-compatible APIs require this before tool-result messages.
// OpenAI protocol requires this before tool-result messages.
context_messages.push(ChatMessage::assistant_with_tool_calls(
"",
content,
tool_calls.clone(),
));
@@ -960,10 +1235,26 @@ impl Agent {
if let Some((ext_name, instructions)) =
detect_auth_awaiting(&tc.name, &tool_result)
{
let mut sess = session.lock().await;
if let Some(thread) = sess.threads.get_mut(&thread_id) {
thread.enter_auth_mode(ext_name);
let auth_data = parse_auth_result(&tool_result);
{
let mut sess = session.lock().await;
if let Some(thread) = sess.threads.get_mut(&thread_id) {
thread.enter_auth_mode(ext_name.clone());
}
}
let _ = self
.channels
.send_status(
&message.channel,
StatusUpdate::AuthRequired {
extension_name: ext_name,
instructions: Some(instructions.clone()),
auth_url: auth_data.auth_url,
setup_url: auth_data.setup_url,
},
&message.metadata,
)
.await;
return Ok(AgenticLoopResult::Response(instructions));
}
@@ -1024,19 +1315,59 @@ impl Agent {
.into());
}
// Execute with timeout
let result = tokio::time::timeout(std::time::Duration::from_secs(60), async {
tracing::debug!(
tool = %tool_name,
params = %params,
"Tool call started"
);
// Execute with per-tool timeout
let timeout = tool.execution_timeout();
let start = std::time::Instant::now();
let result = tokio::time::timeout(timeout, async {
tool.execute(params.clone(), job_ctx).await
})
.await
.map_err(|_| crate::error::ToolError::Timeout {
name: tool_name.to_string(),
timeout: std::time::Duration::from_secs(60),
})?
.map_err(|e| crate::error::ToolError::ExecutionFailed {
name: tool_name.to_string(),
reason: e.to_string(),
})?;
.await;
let elapsed = start.elapsed();
match &result {
Ok(Ok(output)) => {
let result_str = serde_json::to_string(&output.result)
.unwrap_or_else(|_| "<serialize error>".to_string());
tracing::debug!(
tool = %tool_name,
elapsed_ms = elapsed.as_millis() as u64,
result = %result_str,
"Tool call succeeded"
);
}
Ok(Err(e)) => {
tracing::debug!(
tool = %tool_name,
elapsed_ms = elapsed.as_millis() as u64,
error = %e,
"Tool call failed"
);
}
Err(_) => {
tracing::debug!(
tool = %tool_name,
elapsed_ms = elapsed.as_millis() as u64,
timeout_secs = timeout.as_secs(),
"Tool call timed out"
);
}
}
let result = result
.map_err(|_| crate::error::ToolError::Timeout {
name: tool_name.to_string(),
timeout,
})?
.map_err(|e| crate::error::ToolError::ExecutionFailed {
name: tool_name.to_string(),
reason: e.to_string(),
})?;
// Convert result to string
serde_json::to_string_pretty(&result.result).map_err(|e| {
@@ -1380,10 +1711,11 @@ impl Agent {
if let Some((ext_name, instructions)) =
detect_auth_awaiting(&pending.tool_name, &tool_result)
{
let auth_data = parse_auth_result(&tool_result);
{
let mut sess = session.lock().await;
if let Some(thread) = sess.threads.get_mut(&thread_id) {
thread.enter_auth_mode(ext_name);
thread.enter_auth_mode(ext_name.clone());
thread.complete_turn(&instructions);
}
}
@@ -1391,7 +1723,12 @@ impl Agent {
.channels
.send_status(
&message.channel,
StatusUpdate::Status("Awaiting token".into()),
StatusUpdate::AuthRequired {
extension_name: ext_name,
instructions: Some(instructions.clone()),
auth_url: auth_data.auth_url,
setup_url: auth_data.setup_url,
},
&message.metadata,
)
.await;
@@ -1434,6 +1771,7 @@ impl Agent {
match result {
Ok(AgenticLoopResult::Response(response)) => {
thread.complete_turn(&response);
self.persist_response_chain(thread);
let _ = self
.channels
.send_status(
@@ -1532,16 +1870,6 @@ impl Agent {
pending.extension_name
);
// Notify via channel status so the response doesn't echo the token
let _ = self
.channels
.send_status(
&message.channel,
StatusUpdate::Status("Authenticated, loading tools...".into()),
&message.metadata,
)
.await;
// Auto-activate so tools are available immediately after auth
match ext_mgr.activate(&pending.extension_name).await {
Ok(activate_result) => {
@@ -1551,10 +1879,23 @@ impl Agent {
} else {
format!("\n\nTools: {}", activate_result.tools_loaded.join(", "))
};
Ok(Some(format!(
let msg = format!(
"{} authenticated and activated ({} tools loaded).{}",
pending.extension_name, tool_count, tool_list
)))
);
let _ = self
.channels
.send_status(
&message.channel,
StatusUpdate::AuthCompleted {
extension_name: pending.extension_name.clone(),
success: true,
message: msg.clone(),
},
&message.metadata,
)
.await;
Ok(Some(msg))
}
Err(e) => {
tracing::warn!(
@@ -1562,16 +1903,29 @@ impl Agent {
pending.extension_name,
e
);
Ok(Some(format!(
let msg = format!(
"{} authenticated successfully, but activation failed: {}. \
Try activating manually.",
pending.extension_name, e
)))
);
let _ = self
.channels
.send_status(
&message.channel,
StatusUpdate::AuthCompleted {
extension_name: pending.extension_name.clone(),
success: true,
message: msg.clone(),
},
&message.metadata,
)
.await;
Ok(Some(msg))
}
}
}
Ok(result) => {
// Unexpected state, re-enter auth mode
// Invalid token, re-enter auth mode
{
let mut sess = session.lock().await;
if let Some(thread) = sess.threads.get_mut(&thread_id) {
@@ -1580,13 +1934,43 @@ impl Agent {
}
let msg = result
.instructions
.clone()
.unwrap_or_else(|| "Invalid token. Please try again.".to_string());
// Re-emit AuthRequired so web UI re-shows the card
let _ = self
.channels
.send_status(
&message.channel,
StatusUpdate::AuthRequired {
extension_name: pending.extension_name.clone(),
instructions: Some(msg.clone()),
auth_url: result.auth_url,
setup_url: result.setup_url,
},
&message.metadata,
)
.await;
Ok(Some(msg))
}
Err(e) => {
let msg = format!(
"Authentication failed for {}: {}",
pending.extension_name, e
);
let _ = self
.channels
.send_status(
&message.channel,
StatusUpdate::AuthCompleted {
extension_name: pending.extension_name.clone(),
success: false,
message: msg.clone(),
},
&message.metadata,
)
.await;
Ok(Some(msg))
}
Err(e) => Ok(Some(format!(
"Authentication failed for {}: {}",
pending.extension_name, e
))),
}
}
@@ -1937,40 +2321,49 @@ impl Agent {
}
}
async fn handle_command(
/// Handle system commands that bypass thread-state checks entirely.
async fn handle_system_command(
&self,
command: &str,
_args: &[String],
) -> Result<Option<String>, Error> {
args: &[String],
) -> Result<SubmissionResult, Error> {
match command {
"help" => Ok(Some(
r#"Commands:
/job <desc> - Create a job
/status [id] - Check job status
/cancel <id> - Cancel a job
/list - List all jobs
/help <job_id> - Help a stuck job
"help" => Ok(SubmissionResult::response(concat!(
"System:\n",
" /help Show this help\n",
" /model [name] Show or switch the active model\n",
" /version Show version info\n",
" /tools List available tools\n",
" /debug Toggle debug mode\n",
" /ping Connectivity check\n",
"\n",
"Jobs:\n",
" /job <desc> Create a new job\n",
" /status [id] Check job status\n",
" /cancel <id> Cancel a job\n",
" /list List all jobs\n",
"\n",
"Session:\n",
" /undo Undo last turn\n",
" /redo Redo undone turn\n",
" /compact Compress context window\n",
" /clear Clear current thread\n",
" /interrupt Stop current operation\n",
" /new New conversation thread\n",
" /thread <id> Switch to thread\n",
" /resume <id> Resume from checkpoint\n",
"\n",
"Agent:\n",
" /heartbeat Run heartbeat check\n",
" /summarize Summarize current thread\n",
" /suggest Suggest next steps\n",
"\n",
" /quit Exit",
))),
/undo - Undo last turn
/redo - Redo undone turn
/compact - Compress context
/clear - Clear thread
/interrupt - Stop current turn
/thread new - New thread
/thread <id> - Switch thread
/resume <id> - Resume checkpoint
"ping" => Ok(SubmissionResult::response("pong!")),
/heartbeat - Run heartbeat check now
/summarize - Summarize current thread
/suggest - Suggest next steps
/quit - Exit"#
.to_string(),
)),
"ping" => Ok(Some("pong!".to_string())),
"version" => Ok(Some(format!(
"version" => Ok(SubmissionResult::response(format!(
"{} v{}",
env!("CARGO_PKG_NAME"),
env!("CARGO_PKG_VERSION")
@@ -1978,12 +2371,113 @@ impl Agent {
"tools" => {
let tools = self.tools().list().await;
Ok(Some(format!("Available tools: {}", tools.join(", "))))
Ok(SubmissionResult::response(format!(
"Available tools: {}",
tools.join(", ")
)))
}
_ => Ok(Some(format!("Unknown command: {}. Try /help", command))),
"debug" => {
// Debug toggle is handled client-side in the REPL.
// For non-REPL channels, just acknowledge.
Ok(SubmissionResult::ok_with_message(
"Debug toggle is handled by your client.",
))
}
"model" => {
if args.is_empty() {
// Show current model
let name = self.llm().active_model_name();
Ok(SubmissionResult::response(format!(
"Active model: {}",
name
)))
} else {
let requested = &args[0];
// Validate the model exists
match self.llm().list_models().await {
Ok(models) if !models.is_empty() => {
if !models.iter().any(|m| m == requested) {
return Ok(SubmissionResult::error(format!(
"Unknown model: {}. Available models:\n {}",
requested,
models.join("\n ")
)));
}
}
Ok(_) => {
// Empty model list, can't validate but try anyway
}
Err(e) => {
tracing::warn!("Could not fetch model list for validation: {}", e);
// Proceed anyway, the provider will error on the next call if invalid
}
}
match self.llm().set_model(requested) {
Ok(()) => Ok(SubmissionResult::response(format!(
"Switched model to: {}",
requested
))),
Err(e) => Ok(SubmissionResult::error(format!(
"Failed to switch model: {}",
e
))),
}
}
}
_ => Ok(SubmissionResult::error(format!(
"Unknown command: {}. Try /help",
command
))),
}
}
/// Handle legacy command routing from the Router (job commands that go through
/// process_user_input -> router -> handle_job_or_command -> here).
async fn handle_command(
&self,
command: &str,
args: &[String],
) -> Result<Option<String>, Error> {
// System commands are now handled directly via Submission::SystemCommand,
// but the router may still send us unknown /commands.
match self.handle_system_command(command, args).await? {
SubmissionResult::Response { content } => Ok(Some(content)),
SubmissionResult::Ok { message } => Ok(message),
SubmissionResult::Error { message } => Ok(Some(format!("Error: {}", message))),
_ => Ok(None),
}
}
}
/// Parsed auth result fields for emitting StatusUpdate::AuthRequired.
struct ParsedAuthData {
auth_url: Option<String>,
setup_url: Option<String>,
}
/// Extract auth_url and setup_url from a tool_auth result JSON string.
fn parse_auth_result(result: &Result<String, Error>) -> ParsedAuthData {
let parsed = result
.as_ref()
.ok()
.and_then(|s| serde_json::from_str::<serde_json::Value>(s).ok());
ParsedAuthData {
auth_url: parsed
.as_ref()
.and_then(|v| v.get("auth_url"))
.and_then(|v| v.as_str())
.map(|s| s.to_string()),
setup_url: parsed
.as_ref()
.and_then(|v| v.get("setup_url"))
.and_then(|v| v.as_str())
.map(|s| s.to_string()),
}
}
/// Check if a tool_auth result indicates the extension is awaiting a token.
@@ -1994,7 +2488,7 @@ fn detect_auth_awaiting(
tool_name: &str,
result: &Result<String, Error>,
) -> Option<(String, String)> {
if tool_name != "tool_auth" {
if tool_name != "tool_auth" && tool_name != "tool_activate" {
return None;
}
let output = result.as_ref().ok()?;
@@ -2078,4 +2572,34 @@ mod tests {
let (_, instructions) = detect_auth_awaiting("tool_auth", &result).unwrap();
assert_eq!(instructions, "Please provide your API token/key.");
}
#[test]
fn test_detect_auth_awaiting_tool_activate() {
let result: Result<String, Error> = Ok(serde_json::json!({
"name": "slack",
"kind": "McpServer",
"awaiting_token": true,
"status": "awaiting_token",
"instructions": "Provide your Slack Bot token."
})
.to_string());
let detected = detect_auth_awaiting("tool_activate", &result);
assert!(detected.is_some());
let (name, instructions) = detected.unwrap();
assert_eq!(name, "slack");
assert!(instructions.contains("Slack Bot"));
}
#[test]
fn test_detect_auth_awaiting_tool_activate_not_awaiting() {
let result: Result<String, Error> = Ok(serde_json::json!({
"name": "slack",
"tools_loaded": ["slack_post_message"],
"message": "Activated"
})
.to_string());
assert!(detect_auth_awaiting("tool_activate", &result).is_none());
}
}