* fix: add FullJobWatcher to track full_job lifecycle for concurrency (#1318)
full_job routines previously bypassed max_concurrent and global concurrency
limits because execute_full_job() returned RunStatus::Ok immediately after
dispatch. This meant running_count was decremented and the routine_run row
was finalized before the actual job completed.
Introduce FullJobWatcher struct that polls store.get_job() every 5s until
the linked job reaches a non-active state, then maps the final JobState to
RunStatus. execute_full_job now creates and awaits the watcher, keeping both
the DB-level running row and the in-memory running_count elevated for the
full job duration.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* test: full_job concurrency regression tests (issue #1318)
Add two integration tests verifying full_job routine concurrency:
1. full_job_max_concurrent_blocks_second_fire_while_first_active:
Inserts a Running routine_run (simulating an in-flight full_job) and
verifies fire_manual returns MaxConcurrent error for max_concurrent=1.
2. global_concurrency_counts_live_full_job_runs:
Elevates running_count to simulate a live full_job holding the global
slot, verifies check_cron_triggers skips due routines, then releases
the slot and verifies the routine fires.
Also makes running_count_for_test() unconditionally public so integration
tests (separate crate) can access it.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* style: fmt and clippy fixes for full_job concurrency tests
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* fix: address PR review feedback on FullJobWatcher
- Add #[doc(hidden)] to running_count_for_test() to hide from public API
- Derive MAX_POLLS from POLL_INTERVAL to keep constants coupled
- Check job state before first sleep to finalize promptly for fast jobs
- Update execute_full_job doc comment to reflect blocking behavior
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>
One flaky test (test_builtin_echo_tool timeout) was stopping the entire
e2e coverage suite via -x, preventing 118+ remaining tests from running
and generating coverage data.
Tests are independent (each gets a fresh browser context via the
function-scoped page fixture), so removing -x is safe.
Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>
* fix: consume matched event routine messages
* style: run rustfmt for event routine fix
* fix: preserve preprocessing for routine-triggered messages
* fix: match routines against rewritten input
* refactor: narrow check_event_triggers API and simplify routine_engine_slot
Address Copilot review feedback:
- Change check_event_triggers to accept (user_id, channel, content) instead
of &IncomingMessage, eliminating the need to clone the full message
(including attachments) when hooks rewrite content.
- Remove routine_trigger_message and the Cow<IncomingMessage> indirection;
the event-trigger check now inlines the is_internal + UserInput guard and
passes the post-hook content string directly.
- Make routine_engine_slot non-optional since Agent::new() always
initializes it. Removes the redundant Option wrapper and simplifies
accessor/setter methods.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>
* fix: add debug_assert invariant guards to critical code paths (closes#1215)
Add three debug_assert! calls to catch impossible-in-correct-code states
early in debug builds without affecting release performance:
- execute_tool_with_safety: assert tool_name is non-empty at entry
- JobContext::transition_to: assert state machine transition is valid
- CircuitBreakerProvider::record_success: assert circuit is not Open
(check_allowed() must gate all calls before record_success())
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
* test: add regression test for empty tool name invariant guard
Covers the debug_assert!(!tool_name.is_empty()) added in execute_tool_with_safety.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
---------
Co-authored-by: Claude Sonnet 4.6 <[email protected]>
* feat: upgrade MiniMax default model to M2.7
- Add MiniMax-M2.7 and MiniMax-M2.7-highspeed to model list
- Set MiniMax-M2.7 as default model
- Keep all previous models as alternatives
- Update related tests
* fix: use canonical model name in test per review
Use MiniMax-M2.7-highspeed (canonical casing) in the reasoning
models test for consistency with the documentation and provider
configuration.
[skip-regression-check]
These tests guard against catastrophic regex backtracking (seconds/minutes),
not 12ms differences. CI runners with coverage instrumentation (cargo-llvm-cov)
consistently exceed the 100ms threshold due to overhead, causing flaky failures.
500ms still catches real regressions while tolerating CI variability.
[skip-regression-check]
Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>
* fix: Rate limiter returns retry after None instead of a duration
linter fix
* review fixes
* fix: rate limiter returns None for retry_after duration
Add regression test to src/llm/retry.rs that verifies RateLimited errors
always have a fallback duration (never None) due to the 60-second fallback
applied in all rate limit error creation sites (nearai_chat.rs,
anthropic_oauth.rs, embeddings.rs).
The production code fix adds `.or(Some(Duration::from_secs(60)))` to ensure
the error message never displays "retry after None" to the user.
[skip-regression-check]
Co-Authored-By: Claude Haiku 4.5 <[email protected]>
---------
Co-authored-by: Claude Haiku 4.5 <[email protected]>
- Remove duplicate build_nearai_model_fetch_config() definition from setup/wizard.rs
(function already exists in llm/models.rs and is imported)
- Add missing cheap_model and smart_routing_cascade fields to LlmConfig
initializer in build_nearai_model_fetch_config() (llm/models.rs)
- Pass request_timeout_secs to create_registry_provider() call
(llm/mod.rs:432)
All clippy checks pass with zero warnings (--no-default-features --features libsql).
Co-Authored-By: Claude Haiku 4.5 <[email protected]>
Resolved merge conflicts in 5 files:
1. src/agent/job_monitor.rs - Used is_internal flag approach (HEAD) for safe internal message marking. Removed metadata-based approach which could be spoofed by external channels.
2. src/agent/agent_loop.rs - Used is_internal check (HEAD) for routing internal messages, consistent with security model where is_internal field cannot be spoofed.
3. src/agent/dispatcher.rs - Included notify_metadata in job context (main), needed for job routing through JobMonitorRoute.
4. src/setup/wizard.rs - Added build_nearai_model_fetch_config() function (main) for model selection during setup.
5. src/tools/builtin/job.rs - Used both comments from HEAD (clarifying notify_channel and notify_user logic) while removing metadata field from JobMonitorRoute (consistent with job_monitor.rs).
All conflicts resolved with security-first approach: use is_internal boolean field for internal message marking (cannot be spoofed), while passing routing metadata through context.
Co-Authored-By: Claude Haiku 4.5 <[email protected]>
* feat: add LLM_CHEAP_MODEL for generic smart routing across all backends
Add generic cheap model support that works with any LLM backend, not just
NearAI. New env vars: LLM_CHEAP_MODEL (cheap model for any backend) and
SMART_ROUTING_CASCADE (top-level cascade flag).
Resolution order: LLM_CHEAP_MODEL > NEARAI_CHEAP_MODEL (backward compat).
Registry-based providers (OpenAI, Anthropic, Groq, etc.) clone their
RegistryProviderConfig with the cheap model swapped in. Bedrock returns
an explicit error (not yet supported). All error paths use ok_or_else
with proper LlmError variants -- no unwrap/expect in production code.
* refactor: address Gemini review — remove unnecessary async, extract cheap_model_name()
- Remove async from create_cheap_provider_for_backend() and
create_cheap_llm_provider() — neither contains .await calls
- Extract duplicated cheap model resolution logic into
LlmConfig::cheap_model_name() helper method (DRY)
- Revert tests from tokio::test async back to sync #[test]
- Add test_cheap_model_name_resolution() unit test for the helper
---------
Co-authored-by: SMKRV <[email protected]>
Route messages and replies to the correct Telegram forum topic via
message_thread_id. Key behaviors:
- Parse message_thread_id, is_topic_message, is_forum from incoming updates
- Thread agent sessions by "chat_id:topic_id" for forum groups only
(non-forum reply threads are excluded via is_forum guard)
- Pass message_thread_id through all send methods (text, photo, document)
- Normalize thread_id=1 (General topic) to None for sendMessage/sendPhoto/
sendDocument since Telegram rejects it, but preserve it for sendChatAction
where Telegram requires it for typing indicators
- Hoist bot_username workspace read to avoid duplicate WASM host call per
group message
Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>
* feat(orchestrator): read ORCHESTRATOR_PORT env var for configurable API port
The orchestrator internal API port was hardcoded to 50051 in two places
(ContainerJobConfig and OrchestratorApi::start call), making it impossible
to run multiple IronClaw instances on the same host — the second instance
fails with "Address already in use".
NETWORK_SECURITY.md already documents ORCHESTRATOR_PORT as configurable,
and ContainerJobConfig.orchestrator_port is propagated to worker containers
via IRONCLAW_ORCHESTRATOR_URL, but the env var was never actually read.
Extract resolve_orchestrator_port() that reads ORCHESTRATOR_PORT and falls
back to 50051. Includes tests for valid, invalid, and out-of-range values.
* test: add ENV_LOCK mutex for env-var test serialization
Address Gemini review: add std::sync::Mutex to serialize env var access
across test threads. Keep unsafe blocks — required in Rust edition 2024
where std::env::set_var/remove_var are unsafe functions.
---------
Co-authored-by: SMKRV <[email protected]>
* feat(transcription): add Chat Completions API provider for audio transcription
The existing transcription pipeline only supports the OpenAI Whisper API
(/v1/audio/transcriptions with multipart upload). Providers like OpenRouter
expose audio transcription through the Chat Completions API instead, using
base64-encoded audio in the `input_audio` content type.
Add `ChatCompletionsTranscriptionProvider` that sends audio as base64 in
a chat completion request and extracts the transcript from the response.
Compatible with OpenRouter, OpenAI GPT-4o-audio, and any provider that
supports audio input via Chat Completions.
Config changes:
- TRANSCRIPTION_PROVIDER=chat_completions selects the new provider
- TRANSCRIPTION_API_KEY overrides provider-specific keys
- LLM_API_KEY used as fallback for chat_completions provider
- Default model per provider (whisper-1 for openai, gemini-2.0-flash for
chat_completions)
* style: address review feedback — formatting, idiomatic patterns
- Fix rustfmt formatting for provider constructor chain
- Use or_else for resolve_api_key priority chain (Gemini review)
- Use trim_end_matches('/') instead of while loop (Gemini review)
---------
Co-authored-by: SMKRV <[email protected]>
* fix(jobs): make completed->completed transition idempotent to prevent race errors
Both execution_loop and the worker wrapper in execute() can race to call
mark_completed(). Previously the second call hit "Cannot transition from
completed to completed" and errored the job despite successful completion.
This narrowly allows only the Completed->Completed self-transition as
idempotent (early return with debug log, no duplicate history entry).
All other self-transitions remain rejected to preserve state machine
strictness.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
* style: fix assert! formatting in idempotent completion test
Co-Authored-By: Claude Opus 4.6 <[email protected]>
---------
Co-authored-by: Claude Opus 4.6 <[email protected]>
* fix(llm): persist refreshed Anthropic OAuth token after Keychain re-read (#1136)
The Anthropic OAuth provider stored its token as an immutable SecretString.
When a 401 triggered a Keychain re-read, the fresh token was used for a
single retry but never persisted — every subsequent request reused the
expired original token, causing repeated auth failures.
Changes:
- Wrap token in RwLock<SecretString> so it can be updated after refresh
- Persist refreshed token via update_token() on successful retry
- Add 500ms delay before Keychain re-read to give Claude Code time to
complete its async token refresh write (reduces race window)
- Add regression test verifying token updates persist across reads
Closes#1136
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
* style: fix formatting
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>
* fix(worker): prevent orphaned tool_results and fix parallel merging
Two fixes for tool result handling in the Worker:
1. Preserve reasoning text from select_tools() in the RespondResult
content field so it appears in the assistant_with_tool_calls message
pushed by execute_tool_calls. Without this, the LLM's reasoning
context was lost when using the select_tools path.
2. Merge consecutive tool_result messages into a single User message
in rig_adapter's convert_messages(). When parallel tools execute,
each produces a separate ChatMessage with role: Tool. Without
merging, these become consecutive User messages which Anthropic
rejects. Now consecutive tool results are merged into one User
message with multiple ToolResult content items.
Includes regression tests for both fixes.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
* fix(worker): use find_map for first non-empty reasoning extraction
The previous code only checked the first ToolSelection's reasoning,
missing cases where the first selection has empty reasoning but
subsequent ones do not. Switch to find_map to get the first non-empty
reasoning across all selections.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
---------
Co-authored-by: Claude Opus 4.6 <[email protected]>
* feat(heartbeat): fire_at time-of-day scheduling with IANA timezone support
- HEARTBEAT_FIRE_AT=HH:MM — fire heartbeat at a specific time of day instead
of on a rolling interval; format is 24h HH:MM (e.g. "14:00")
- HEARTBEAT_TIMEZONE=Region/City — IANA timezone name for fire_at (e.g.
"Pacific/Auckland", "America/New_York"). Defaults to UTC.
- When fire_at is set, interval_secs is ignored
- Config also readable from settings.toml [heartbeat] section
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
* feat(heartbeat): wire fire_at + timezone into HeartbeatConfig runner
Missed file from heartbeat scheduling commit. HeartbeatConfig struct in
agent/heartbeat.rs now carries fire_at: Option<NaiveTime> and timezone: Tz
so the runner can schedule against a fixed time of day.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
* fix: add chrono-tz dependency for heartbeat fire_at timezone support
The chrono-tz crate was used in the heartbeat fire_at commits but
its Cargo.toml entry was lost during rebase conflict resolution.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
* style: rustfmt fix for chained method call
Co-Authored-By: Claude Opus 4.6 <[email protected]>
* test(heartbeat): add fire_at scheduling and DST safety tests
- test_default_config_has_no_fire_at: interval-based default unchanged
- test_with_fire_at_builder: builder sets time and timezone
- test_duration_until_next_fire_is_bounded: result always 1s–24h
- test_duration_until_next_fire_dst_timezone_no_panic: US Eastern DST
- test_resolved_tz_defaults_to_utc: missing timezone falls back to UTC
- test_resolved_tz_parses_iana: IANA string resolves correctly
Co-Authored-By: Claude Opus 4.6 <[email protected]>
* fix(heartbeat): restore drift-free interval, add settings.json fallback for fire_at
- Interval path: restore tokio::time::interval (drift-free) instead of
tokio::time::sleep which drifts by loop body execution time
- fire_at config: fall back to settings.heartbeat.fire_at when
HEARTBEAT_FIRE_AT env var is not set, consistent with other settings
Addresses Gemini Code Assist review feedback.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
---------
Co-authored-by: IronClaw <[email protected]>
Co-authored-by: Claude Sonnet 4.6 <[email protected]>
