The LLM was ignoring tool results and answering from training data
because the compact output metadata didn't include what tools returned.
Tool results lived only as ActionResult messages (role: Tool) which
some providers flatten or the model ignores.
Now the code step output includes:
- stdout from Python print() statements
- [tool_name result] with the actual output (truncated to 4K per tool)
- [tool_name error] for failed tools
- [return] for the code's return value
- Total output truncated to 8K chars to prevent context bloat
This ensures the model sees web_search results, API responses, etc.
in the next iteration and can reason about them instead of hallucinating.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Engine v2 now shows live progress in the CLI (and any channel):
- "Thinking..." when a step starts
- Tool name + success/error when actions execute
- "Processing results..." when a step completes
Implementation:
- ThreadManager holds a broadcast::Sender<ThreadEvent> (capacity 256)
- ExecutionLoop.emit_event() writes to thread.events AND broadcasts
- ThreadManager.subscribe_events() returns a receiver
- Router uses tokio::select! to listen for events while waiting for
thread completion, forwarding them as StatusUpdate to the channel
This replaces the polling approach with zero-latency event streaming.
Agent.channels visibility widened to pub(crate) for bridge access.
102 tests passing, zero clippy warnings.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Models sometimes write FINAL() outside code blocks — as plain text
after an explanation. The Hyperliquid case: model outputs a long
analysis then FINAL("""...""") at the end, not inside ```repl fences.
Fixes:
- extract_final_from_text(): regex-based FINAL detection in text
responses, matching the official RLM's find_final_answer() fallback
- Handles: double-quoted, single-quoted, triple-quoted, unquoted,
nested parens
- Checked in LlmResponse::Text handler BEFORE tool intent nudge
(FINAL takes priority)
9 new tests:
- codeact_final_in_text_response: FINAL("answer") in plain text
- codeact_final_triple_quoted_in_text: FINAL("""multi\nline""") in text
- final_double_quoted, final_single_quoted, final_triple_quoted,
final_unquoted, final_with_nested_parens, final_after_long_text,
no_final_returns_none
102 tests passing (93 + 9 new), zero clippy warnings.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Comprehensive test coverage for the Monty Python execution path:
- codeact_simple_final: Python code calls FINAL('answer') → thread completes
- codeact_tool_call_then_final: code calls test_tool() → FunctionCall
suspends VM → MockEffects returns result → code resumes → FINAL()
- codeact_pure_python_computation: sum([1,2,3,4,5]) → FINAL('Sum is 15')
with no tool calls — pure Python in Monty
- codeact_multi_step: first step prints output (no FINAL), second step
sees output metadata and calls FINAL — tests iterative REPL flow
- codeact_error_recovery: first step has NameError → error flows to LLM
as stdout → second step recovers with FINAL — tests error transparency
- codeact_context_variables_available: code accesses `goal` and `context`
variables injected by the RLM context builder
- codeact_multiple_tool_calls_in_loop: for loop calls test_tool() 3 times
→ 3 FunctionCall suspensions → all results collected → FINAL
- codeact_llm_query_recursive: code calls llm_query('prompt') → VM
suspends → MockLlm provides sub-agent response → result returned as
Python string variable
93 tests passing (85 prior + 8 new), zero clippy warnings.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
The engine now operates in CodeAct/RLM mode:
System prompt (executor/prompt.rs):
- Instructs LLM to write Python in ```repl fenced blocks
- Documents available tools as callable Python functions
- Documents llm_query(), llm_query_batched(), FINAL()
- Documents context variables (context, goal, step_number, previous_results)
- Strategy guidance: examine context, break into steps, use tools, call FINAL()
Code block detection (bridge/llm_adapter.rs):
- extract_code_block() scans LLM text responses for ```repl or ```python blocks
- When detected, returns LlmResponse::Code instead of LlmResponse::Text
- The ExecutionLoop routes Code responses through Monty for execution
No structured tool definitions sent to LLM:
- Tools are described in the system prompt as Python functions
- The LLM call sends empty actions array, forcing text-mode responses
- This ensures the LLM writes code blocks (CodeAct) instead of
structured tool calls (which would bypass the REPL)
85 tests passing, zero clippy warnings.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
The engine was creating a fresh ThreadManager and InMemoryStore per
message, losing all context between turns. A follow-up question like
"what are the latest 10 issues?" had no memory of the prior "how many
issues" response.
Fixes:
- EngineState (ThreadManager, ConversationManager, InMemoryStore) now
persists across messages via OnceLock, initialized on first use
- ConversationManager builds message history from prior conversation
entries (user messages + agent responses) and passes it to new threads
- ThreadManager.spawn_thread_with_history() accepts initial_messages
that are prepended before the current user message
- System notifications (thread started/completed) are filtered out of
the history (not useful as LLM context)
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
The ExecutionLoop was sending empty messages to the LLM because the
thread was spawned with the user's input as the goal but no messages.
Fixes:
- ThreadManager.spawn_thread() now adds the goal as an initial user
message before starting the execution loop
- ExecutionLoop.run() injects a default system prompt if none exists
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Restructure phases 6-8 to clarify execution model:
- Monty is the sole Python executor for CodeAct/RLM. No WASM or Docker
Python runtimes for LLM-generated code.
- WASM sandbox is for third-party tool isolation (existing infra, Phase 8)
- Docker containers are for thread-level isolation of high-risk work (Phase 8)
- Two-phase commit moves to Phase 6 (integration) at the adapter boundary
Phase renumbering:
- Old Phase 6 (Tier 2-3) → removed as separate phase
- Old Phase 7 (integration) → Phase 6
- Old Phase 8 (cleanup) → Phase 7
- New Phase 8: WASM tools + Docker thread isolation (infra integration)
Updated progress table: Phases 1-5 marked DONE with test counts and commits.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Conversation is now a UI layer, not an execution boundary. Multiple
threads can run concurrently within one conversation; threads can
outlive their originating conversation.
New types (types/conversation.rs):
- ConversationSurface: channel + user + entries + active_threads
- ConversationEntry: sender (User/Agent/System) + content + origin_thread_id
- ConversationId, EntryId (UUID newtypes)
- EntrySender enum (User, Agent{thread_id}, System)
ConversationManager (runtime/conversation.rs):
- get_or_create_conversation(channel, user) — indexed by (channel, user)
- handle_user_message() — injects into active foreground thread or spawns new
- record_thread_outcome() — adds agent/system entries, untracks completed threads
- get_conversation(), list_conversations()
This enables the key architectural insight: a user can ask "what's the
weather?" while a deployment thread is still running. Both produce entries
in the same conversation.
85 tests passing, zero clippy warnings.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Cross-referenced our implementation against the official RLM (alexzhang13/rlm),
fast-rlm (avbiswas/fast-rlm), and Prime Intellect's verifiers implementation.
Key enhancements:
- FINAL(answer) / FINAL_VAR(name): explicit termination pattern matching
all three reference implementations. Code can signal completion at any
point, not just via return value.
- llm_query_batched(prompts): parallel recursive sub-calls via tokio::spawn,
matching fast-rlm's asyncio.gather pattern and Prime Intellect's llm_batch.
- Output truncation increased to 8000 chars (from 120), matching Prime
Intellect's 8192 default. Shows [TRUNCATED: last N chars] or [FULL OUTPUT].
- Step 0 orientation preamble: auto-injects context metadata (message count,
total chars, goal, last user message preview) before first code step,
matching fast-rlm's auto-print pattern.
- Error-to-LLM flow: Python parse errors, runtime errors, NameErrors,
OS errors, and async errors now flow back as stdout content instead of
terminating the step, enabling LLM self-correction on next iteration.
Only VM panics (catch_unwind) terminate as EngineError.
74 tests passing, zero clippy warnings.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
