* feat: complete multi-tenant isolation — per-user budgets, model selection, heartbeat cycling
Finishes the remaining isolation work from phases 2–4 of #59:
Phase 2 (DB scoping): Fix /status and /list commands to use _for_user
DB variants instead of global queries that leaked cross-user job data.
Phase 3 (Runtime isolation): Per-user workspace in routine engine's
spawn_fire so lightweight routines run in the correct user context.
Per-user daily cost tracking in CostGuard with configurable budget via
MAX_COST_PER_USER_PER_DAY_CENTS. Multi-user heartbeat that cycles
through all users with routines, auto-detected from GATEWAY_USER_TOKENS.
Phase 4 (Provider/tools): Per-user model selection via preferred_model
setting — looked up from SettingsStore on first iteration, threaded
through ReasoningContext.model_override to CompletionRequest. Works
with providers that support per-request model overrides (NearAI).
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* fix: use selected_model setting key to match /model command persistence
The dispatcher was reading "preferred_model" but the /model command
(merged from staging) persists to "selected_model". Since set_setting
is already per-user scoped, using the same key makes /model work as
the per-user model override in multi-tenant mode.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* fix: heartbeat hygiene, /model multi-tenant guard, RigAdapter model override
Three follow-up fixes for multi-tenant isolation:
1. Multi-user heartbeat now runs memory hygiene per user before each
heartbeat check, matching single-user heartbeat behavior.
2. /model command in multi-tenant mode only persists to per-user
settings (selected_model) without calling set_model() on the shared
LlmProvider. The per-request model_override in the dispatcher reads
from the same setting. Added multi_tenant flag to AgentConfig
(auto-detected from GATEWAY_USER_TOKENS).
3. RigAdapter now supports per-request model overrides by injecting the
model name into rig-core's additional_params. OpenAI/Anthropic/Ollama
API servers use last-key-wins for duplicate JSON keys, so the override
takes effect via serde's flatten serialization order.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* fix: address PR review — cost model attribution, heartbeat concurrency, pruning
Fixes from review comments on #1614:
- Cost tracking now uses the override model name (not active_model_name)
when a per-user model override is active, for accurate attribution.
- Multi-user heartbeat runs per-user checks concurrently via JoinSet
instead of sequentially, preventing one slow user from blocking others.
- Per-user failure counts tracked independently; users exceeding
max_failures are skipped (matching single-user semantics).
- per_user_daily_cost HashMap pruned on day rollover to prevent
unbounded growth in long-lived deployments.
- Doc comment fixed: says "routines" not "active routines".
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* fix: /status ownership, model persistence scoping, heartbeat robustness
Addresses second round of PR review on #1614:
- /status <job_id> DB path now validates job.user_id == requesting user
before returning data (was missing ownership check, security fix).
- persist_selected_model takes user_id param instead of owner_id, and
skips .env/TOML writes in multi-tenant mode (these are shared global
files). handle_system_command now receives user_id from caller.
- JoinSet collection handles Err(JoinError) explicitly instead of
silently dropping panicked tasks.
- Notification forwarder extracts owner_id from response metadata in
multi-tenant mode for per-user routing instead of broadcasting to
the agent owner.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* fix: cost pricing, fire_manual workspace, heartbeat concurrency cap
Round 3 review fixes:
- Cost tracking passes None for cost_per_token when model override is
active, letting CostGuard look up pricing by model name instead of
using the default provider's rates (serrrfirat).
- fire_manual() now uses per-user workspace, matching spawn_fire()
pattern (serrrfirat).
- Removed MULTI_TENANT env var — multi-tenant mode is auto-detected
solely from GATEWAY_USER_TOKENS presence (serrrfirat + Copilot).
- Multi-user heartbeat capped at 8 concurrent tasks to avoid flooding
the LLM provider (serrrfirat + Copilot).
- Fixed inject_model_override doc comment accuracy (Copilot).
- Added comment explaining multi-tenant notification routing priority
(Copilot).
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* feat: user-scoped webhook endpoint for multi-tenant isolation
Adds POST /api/webhooks/u/{user_id}/{path} — a user-scoped webhook
endpoint that filters the routine lookup by user_id, preventing
cross-user webhook triggering when paths collide.
The existing /api/webhooks/{path} endpoint remains unchanged for
backward compatibility in single-user deployments.
Changes:
- get_webhook_routine_by_path gains user_id: Option<&str> param
- Both postgres and libsql implementations add AND user_id = ? filter
when user_id is provided
- New webhook_trigger_user_scoped_handler extracts (user_id, path)
from URL and passes to shared fire_webhook_inner logic
- Route registered on public router (webhooks are called by external
services that can't send bearer tokens)
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* feat: add TenantCtx for compile-time tenant isolation
Implements zmanian's architectural proposal from #1614 review:
two-tier scoped database access (TenantScope/AdminScope) so handler
code cannot accidentally bypass tenant scoping.
TenantScope (default): wraps user_id + Arc<dyn Database>, auto-binds
user_id on every operation. ID-based lookups return None for cross-
tenant resources. No escape hatch — forgetting to scope is a compile
error.
AdminScope (explicit opt-in): cross-tenant access for system-level
components (heartbeat, routine engine, self-repair, scheduler, worker).
TenantCtx bundles TenantScope + workspace + cost guard + per-user
rate limiting. Constructed once per request in handle_message, threaded
through all command handlers and ChatDelegate.
Key changes:
- New src/tenant.rs (~920 lines): TenantScope, AdminScope, TenantCtx,
TenantRateState, TenantRateRegistry
- All command handlers: user_id: &str → ctx: &TenantCtx
- ChatDelegate: cost check/record/settings via self.tenant
- System components: store field changed to AdminScope
- Config: TENANT_MAX_LLM_CONCURRENT, TENANT_MAX_JOBS_CONCURRENT env vars
- Fixes bug: /status <job_id> cross-tenant leak (now auto-filtered)
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>
* Fix REPL single-message hang and cap CI test duration
* Fix Clippy nested-if lint in REPL startup
* Fix single-message approval flow
* Handle empty single-message REPL exits
* Wait for one-shot event routines before exit
* Fix MCP lifecycle trace user scope
* Fix REPL single-message hang and cap CI test duration
* Fix Clippy nested-if lint in REPL startup
* Fix single-message approval flow
* Handle empty single-message REPL exits
* Wait for one-shot event routines before exit
* feat(agent): thread per-tool reasoning from LLM through to REPL, HTTP, SSE, and DB
Add end-to-end agent reasoning summaries so users can see *why* the
agent chose specific tools, not just what it did.
- Add `reasoning: Option<String>` to `ToolCall` (all providers)
- Populate from LLM response content in `Reasoning::respond_with_tools`
and `select_tools`, with per-tool override when providers supply it
- Extend `Turn` with `narrative` and `TurnToolCall` with `rationale` +
`tool_call_id` for identity-based result matching
- Persist reasoning in DB via existing tool_calls JSON (no migration)
- Add `StatusUpdate::ReasoningUpdate` and `SseEvent::ReasoningUpdate` +
`SseEvent::JobReasoning` for real-time streaming
- Emit reasoning events in both chat dispatcher and worker job path
- Add `/reasoning [N|all]` command for inspecting turn reasoning
- Surface `narrative` and `rationale` in HTTP `/api/chat/history`
Based on the design from #361 and #456, reconstructed cleanly with
Option<String> to minimize blast radius (vs mandatory String that broke
compilation in #456).
Closes#456
Co-Authored-By: panosAthDBX <[email protected]>
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* fix: address PR review feedback from Gemini and Copilot
- Fix `_ => Ok(None)` in agent_loop.rs to avoid accidental shutdown
- Fix fallback in record_tool_result_for/record_tool_error_for to use
first pending call instead of last_mut (parallel execution safety)
- Include per-tool decisions in WASM channel reasoning messages
- Apply truncate_at_tool_tags + clean_response to shared_reasoning in
select_tools (parity with respond_with_tools)
- Persist turn-level narrative to DB in tool_calls JSON wrapper
- Parse both old (array) and new (object) tool_calls formats in
build_turns_from_db_messages for backward compatibility
- Populate reasoning from action.reasoning in execute_plan ToolCalls
[skip-regression-check]
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* fix: address second round of review comments + merge fixes
- Add reasoning: None to new github_copilot.rs ToolCall sites (from staging merge)
- Run cargo fmt on 4 files with formatting diffs
- Truncate narrative to 1000 chars before DB persistence
- Clone turn data and drop session lock in /reasoning command
- Extract ToolDecisionDto::from_json_array shared helper (deduplicate
worker/job.rs and orchestrator/api.rs)
- Add unit tests for wrapped tool_calls JSON format with narrative
[skip-regression-check]
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* fix: address third round of review comments (Copilot + serrrfirat)
- Reword ToolCall.reasoning docstring to reflect provider-supplied or
fallback contract
- Sanitize narrative through SafetyLayer before storage/emission
- Clean per-tool reasoning via truncate_at_tool_tags + clean_response
in select_tools (parity with shared reasoning)
- Convert 4 approval-path recording sites in thread_ops.rs to
identity-based record_tool_result_for/record_tool_error_for
- Preserve tool_call_id and reasoning through restore_from_messages
- Fix has_result/has_error to reject JSON null values
- Truncate tool_call_id to 128 chars before DB persistence
- Add 4 unit tests for record_tool_result_for/error_for edge cases
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* fix: address zmanian review — sanitize JobDelegate reasoning + warn on dropped results
- Sanitize narrative and per-tool rationale through SafetyLayer in
JobDelegate reasoning events (parity with ChatDelegate)
- Add tracing::warn when record_tool_result_for/error_for drops a
result because no matching or pending tool call exists
- Add 3 unit tests for reasoning normalization (thinking tags,
tool tags, empty-after-cleaning)
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* fix: address 4 remaining unreplied review comments
- Clean per-tool reasoning in respond_with_tools via truncate_at_tool_tags
+ clean_response (parity with select_tools)
- Handle wrapped JSON format in rebuild_chat_messages_from_db so cold
hydration works after persist_tool_calls format change
- Update persist_tool_calls doc comment to describe new JSON shape
- Sanitize per-tool rationale through SafetyLayer in ChatDelegate before
emission and storage (parity with JobDelegate)
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* fix: address zmanian review round 2
- Add tracing::debug on fallback-to-pending path in record_tool_result_for
and record_tool_error_for (item 1)
- Add comment explaining why /reasoning is special-cased in agent_loop.rs
(item 4)
- Items 2 (narrative persistence), 3 (rationale sanitization), and 5
(catch-all fix) were already addressed in prior commits
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
---------
Co-authored-by: panosAthDBX <[email protected]>
Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>
* fix: ensure LLM calls always end with user message (closes#763)
Claude 4.6 models (claude-sonnet-4-6, claude-opus-4-6) no longer support
assistant message prefill — any LLM call where the conversation ends on an
assistant message is rejected with HTTP 400 "This model does not support
assistant message prefill".
The same root cause also triggers NEAR AI's "No user query found in messages"
400 error for the routine engine path.
Two fixes:
1. src/worker/container.rs — before_llm_call()
After poll_and_inject_prompt(), if no user follow-up arrived and
handle_text_response() left an assistant message at the end of the
conversation, inject a sentinel "Continue." user message before
the next LLM call.
2. src/agent/routine_engine.rs — execute_lightweight_with_tools()
Before the force_text final completion call, ensure messages end
with a user-role message. Tool result messages (Role::Tool) satisfy
Anthropic but not NEAR AI; assistant messages satisfy neither.
Also updates the worker system prompt to instruct the agent to include
the phrase "The job is complete" in its final message, so the agentic
loop can detect termination reliably.
Tested with claude-sonnet-4-6 and claude-opus-4-6.
Workaround: ANTHROPIC_MODEL=claude-sonnet-4-20250514 (still supports prefill).
* fix: broaden sentinel guard to any non-user message (per review)
Gemini suggested the Role::Assistant check in before_llm_call() is too
specific. Changed to !Role::User to match the routine_engine.rs fix and
cover tool results too.
* fix: address zmanian review — JobDelegate sentinel, shared helper, NearAI complete() flattening
- Extract ensure_ends_with_user_message() to src/util.rs with 4 unit tests
(empty list, after assistant, after tool result, no-op when already user)
- Add sentinel guard to JobDelegate::before_llm_call() in src/worker/job.rs
so scheduler jobs (CreateJob / /job path) no longer hit Claude 4.6 / NEAR AI 400s
- Replace inline guards in ContainerDelegate and routine_engine.rs with the
shared helper — all 3 call sites now use one implementation
- Fix complete() in nearai_chat.rs to apply flatten_tool_messages when
flatten_tool_messages=true — previously only complete_with_tools() flattened,
so force_text paths could still send role:"tool" messages to NEAR AI
- Update stale comment in container.rs: "assistant message" → "non-user message"
- Add flatten tests in nearai_chat.rs covering the complete() path
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
* ci: fix fmt and tar advisory
---------
Co-authored-by: Jacob Lasky <[email protected]>
Co-authored-by: Claude Sonnet 4.6 <[email protected]>
Co-authored-by: Illia Polosukhin <[email protected]>
Co-authored-by: firat.sertgoz <[email protected]>
* refactor: extract AppEvent to crates/ironclaw_common
SseEvent was defined in src/channels/web/types.rs but imported by 12+
modules across agent, orchestrator, worker, tools, and extensions — it
had become the application-wide event protocol, not a web transport
concern.
Create crates/ironclaw_common as a shared workspace crate and move the
enum there as AppEvent. Also move the truncate_preview utility which
was similarly leaked from the web gateway into agent modules.
- New crate: crates/ironclaw_common (AppEvent, truncate_preview)
- Rename SseEvent → AppEvent, from_sse_event → from_app_event
- web/types.rs re-exports AppEvent for internal gateway use
- web/util.rs re-exports truncate_preview
- Wire format unchanged (serde renames are on variants, not the enum)
Aligned with the event bus direction on refactor/architectural-hardening
where DomainEvent (≡ AppEvent) is wrapped in a SystemEvent envelope.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* refactor: add AppEvent::event_type() helper, deduplicate match blocks
Address Gemini review: extract the variant→string match into a single
method on AppEvent, replacing the duplicated 22-arm matches in sse.rs
and types.rs.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* refactor: rename leftover sse vars/tests to match AppEvent rename
Address Copilot review: rename sse_event vars to app_event in
orchestrator/api.rs and ws.rs, rename test functions from
test_ws_server_from_sse_* to test_ws_server_from_app_event_*, and
update stale SSE comments.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* refactor: add Deserialize to AppEvent, round-trip test, fix stale comments
Address zmanian review:
- Add Deserialize derive to AppEvent so downstream consumers can
deserialize incoming events
- Add event_type_matches_serde_type_field test that round-trips every
variant through serde and asserts event_type() matches the serialized
"type" field — catches drift between serde renames and the manual match
- Add round_trip_deserialize test for basic Serialize/Deserialize parity
- Update remaining "SSE" references in comments across server.rs,
manager.rs, ws_gateway_integration.rs, and worker/job.rs
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>
Henry Park