* feat: complete multi-tenant isolation — per-user budgets, model selection, heartbeat cycling
Finishes the remaining isolation work from phases 2–4 of #59:
Phase 2 (DB scoping): Fix /status and /list commands to use _for_user
DB variants instead of global queries that leaked cross-user job data.
Phase 3 (Runtime isolation): Per-user workspace in routine engine's
spawn_fire so lightweight routines run in the correct user context.
Per-user daily cost tracking in CostGuard with configurable budget via
MAX_COST_PER_USER_PER_DAY_CENTS. Multi-user heartbeat that cycles
through all users with routines, auto-detected from GATEWAY_USER_TOKENS.
Phase 4 (Provider/tools): Per-user model selection via preferred_model
setting — looked up from SettingsStore on first iteration, threaded
through ReasoningContext.model_override to CompletionRequest. Works
with providers that support per-request model overrides (NearAI).
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* fix: use selected_model setting key to match /model command persistence
The dispatcher was reading "preferred_model" but the /model command
(merged from staging) persists to "selected_model". Since set_setting
is already per-user scoped, using the same key makes /model work as
the per-user model override in multi-tenant mode.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* fix: heartbeat hygiene, /model multi-tenant guard, RigAdapter model override
Three follow-up fixes for multi-tenant isolation:
1. Multi-user heartbeat now runs memory hygiene per user before each
heartbeat check, matching single-user heartbeat behavior.
2. /model command in multi-tenant mode only persists to per-user
settings (selected_model) without calling set_model() on the shared
LlmProvider. The per-request model_override in the dispatcher reads
from the same setting. Added multi_tenant flag to AgentConfig
(auto-detected from GATEWAY_USER_TOKENS).
3. RigAdapter now supports per-request model overrides by injecting the
model name into rig-core's additional_params. OpenAI/Anthropic/Ollama
API servers use last-key-wins for duplicate JSON keys, so the override
takes effect via serde's flatten serialization order.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* fix: address PR review — cost model attribution, heartbeat concurrency, pruning
Fixes from review comments on #1614:
- Cost tracking now uses the override model name (not active_model_name)
when a per-user model override is active, for accurate attribution.
- Multi-user heartbeat runs per-user checks concurrently via JoinSet
instead of sequentially, preventing one slow user from blocking others.
- Per-user failure counts tracked independently; users exceeding
max_failures are skipped (matching single-user semantics).
- per_user_daily_cost HashMap pruned on day rollover to prevent
unbounded growth in long-lived deployments.
- Doc comment fixed: says "routines" not "active routines".
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* fix: /status ownership, model persistence scoping, heartbeat robustness
Addresses second round of PR review on #1614:
- /status <job_id> DB path now validates job.user_id == requesting user
before returning data (was missing ownership check, security fix).
- persist_selected_model takes user_id param instead of owner_id, and
skips .env/TOML writes in multi-tenant mode (these are shared global
files). handle_system_command now receives user_id from caller.
- JoinSet collection handles Err(JoinError) explicitly instead of
silently dropping panicked tasks.
- Notification forwarder extracts owner_id from response metadata in
multi-tenant mode for per-user routing instead of broadcasting to
the agent owner.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* fix: cost pricing, fire_manual workspace, heartbeat concurrency cap
Round 3 review fixes:
- Cost tracking passes None for cost_per_token when model override is
active, letting CostGuard look up pricing by model name instead of
using the default provider's rates (serrrfirat).
- fire_manual() now uses per-user workspace, matching spawn_fire()
pattern (serrrfirat).
- Removed MULTI_TENANT env var — multi-tenant mode is auto-detected
solely from GATEWAY_USER_TOKENS presence (serrrfirat + Copilot).
- Multi-user heartbeat capped at 8 concurrent tasks to avoid flooding
the LLM provider (serrrfirat + Copilot).
- Fixed inject_model_override doc comment accuracy (Copilot).
- Added comment explaining multi-tenant notification routing priority
(Copilot).
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* feat: user-scoped webhook endpoint for multi-tenant isolation
Adds POST /api/webhooks/u/{user_id}/{path} — a user-scoped webhook
endpoint that filters the routine lookup by user_id, preventing
cross-user webhook triggering when paths collide.
The existing /api/webhooks/{path} endpoint remains unchanged for
backward compatibility in single-user deployments.
Changes:
- get_webhook_routine_by_path gains user_id: Option<&str> param
- Both postgres and libsql implementations add AND user_id = ? filter
when user_id is provided
- New webhook_trigger_user_scoped_handler extracts (user_id, path)
from URL and passes to shared fire_webhook_inner logic
- Route registered on public router (webhooks are called by external
services that can't send bearer tokens)
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* feat: add TenantCtx for compile-time tenant isolation
Implements zmanian's architectural proposal from #1614 review:
two-tier scoped database access (TenantScope/AdminScope) so handler
code cannot accidentally bypass tenant scoping.
TenantScope (default): wraps user_id + Arc<dyn Database>, auto-binds
user_id on every operation. ID-based lookups return None for cross-
tenant resources. No escape hatch — forgetting to scope is a compile
error.
AdminScope (explicit opt-in): cross-tenant access for system-level
components (heartbeat, routine engine, self-repair, scheduler, worker).
TenantCtx bundles TenantScope + workspace + cost guard + per-user
rate limiting. Constructed once per request in handle_message, threaded
through all command handlers and ChatDelegate.
Key changes:
- New src/tenant.rs (~920 lines): TenantScope, AdminScope, TenantCtx,
TenantRateState, TenantRateRegistry
- All command handlers: user_id: &str → ctx: &TenantCtx
- ChatDelegate: cost check/record/settings via self.tenant
- System components: store field changed to AdminScope
- Config: TENANT_MAX_LLM_CONCURRENT, TENANT_MAX_JOBS_CONCURRENT env vars
- Fixes bug: /status <job_id> cross-tenant leak (now auto-filtered)
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>
* Fix REPL single-message hang and cap CI test duration
* Fix Clippy nested-if lint in REPL startup
* Fix single-message approval flow
* Handle empty single-message REPL exits
* Wait for one-shot event routines before exit
* Fix MCP lifecycle trace user scope
* Fix REPL single-message hang and cap CI test duration
* Fix Clippy nested-if lint in REPL startup
* Fix single-message approval flow
* Handle empty single-message REPL exits
* Wait for one-shot event routines before exit
* feat(agent): thread per-tool reasoning from LLM through to REPL, HTTP, SSE, and DB
Add end-to-end agent reasoning summaries so users can see *why* the
agent chose specific tools, not just what it did.
- Add `reasoning: Option<String>` to `ToolCall` (all providers)
- Populate from LLM response content in `Reasoning::respond_with_tools`
and `select_tools`, with per-tool override when providers supply it
- Extend `Turn` with `narrative` and `TurnToolCall` with `rationale` +
`tool_call_id` for identity-based result matching
- Persist reasoning in DB via existing tool_calls JSON (no migration)
- Add `StatusUpdate::ReasoningUpdate` and `SseEvent::ReasoningUpdate` +
`SseEvent::JobReasoning` for real-time streaming
- Emit reasoning events in both chat dispatcher and worker job path
- Add `/reasoning [N|all]` command for inspecting turn reasoning
- Surface `narrative` and `rationale` in HTTP `/api/chat/history`
Based on the design from #361 and #456, reconstructed cleanly with
Option<String> to minimize blast radius (vs mandatory String that broke
compilation in #456).
Closes#456
Co-Authored-By: panosAthDBX <[email protected]>
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* fix: address PR review feedback from Gemini and Copilot
- Fix `_ => Ok(None)` in agent_loop.rs to avoid accidental shutdown
- Fix fallback in record_tool_result_for/record_tool_error_for to use
first pending call instead of last_mut (parallel execution safety)
- Include per-tool decisions in WASM channel reasoning messages
- Apply truncate_at_tool_tags + clean_response to shared_reasoning in
select_tools (parity with respond_with_tools)
- Persist turn-level narrative to DB in tool_calls JSON wrapper
- Parse both old (array) and new (object) tool_calls formats in
build_turns_from_db_messages for backward compatibility
- Populate reasoning from action.reasoning in execute_plan ToolCalls
[skip-regression-check]
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* fix: address second round of review comments + merge fixes
- Add reasoning: None to new github_copilot.rs ToolCall sites (from staging merge)
- Run cargo fmt on 4 files with formatting diffs
- Truncate narrative to 1000 chars before DB persistence
- Clone turn data and drop session lock in /reasoning command
- Extract ToolDecisionDto::from_json_array shared helper (deduplicate
worker/job.rs and orchestrator/api.rs)
- Add unit tests for wrapped tool_calls JSON format with narrative
[skip-regression-check]
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* fix: address third round of review comments (Copilot + serrrfirat)
- Reword ToolCall.reasoning docstring to reflect provider-supplied or
fallback contract
- Sanitize narrative through SafetyLayer before storage/emission
- Clean per-tool reasoning via truncate_at_tool_tags + clean_response
in select_tools (parity with shared reasoning)
- Convert 4 approval-path recording sites in thread_ops.rs to
identity-based record_tool_result_for/record_tool_error_for
- Preserve tool_call_id and reasoning through restore_from_messages
- Fix has_result/has_error to reject JSON null values
- Truncate tool_call_id to 128 chars before DB persistence
- Add 4 unit tests for record_tool_result_for/error_for edge cases
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* fix: address zmanian review — sanitize JobDelegate reasoning + warn on dropped results
- Sanitize narrative and per-tool rationale through SafetyLayer in
JobDelegate reasoning events (parity with ChatDelegate)
- Add tracing::warn when record_tool_result_for/error_for drops a
result because no matching or pending tool call exists
- Add 3 unit tests for reasoning normalization (thinking tags,
tool tags, empty-after-cleaning)
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* fix: address 4 remaining unreplied review comments
- Clean per-tool reasoning in respond_with_tools via truncate_at_tool_tags
+ clean_response (parity with select_tools)
- Handle wrapped JSON format in rebuild_chat_messages_from_db so cold
hydration works after persist_tool_calls format change
- Update persist_tool_calls doc comment to describe new JSON shape
- Sanitize per-tool rationale through SafetyLayer in ChatDelegate before
emission and storage (parity with JobDelegate)
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* fix: address zmanian review round 2
- Add tracing::debug on fallback-to-pending path in record_tool_result_for
and record_tool_error_for (item 1)
- Add comment explaining why /reasoning is special-cased in agent_loop.rs
(item 4)
- Items 2 (narrative persistence), 3 (rationale sanitization), and 5
(catch-all fix) were already addressed in prior commits
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
---------
Co-authored-by: panosAthDBX <[email protected]>
Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>
