optimclaw/CLAUDE.md

NEAR Agent Development Guide

Project Overview

LLM-powered autonomous agent for the NEAR AI marketplace. Features:

• Multi-channel input: Full TUI (Ratatui), HTTP webhook with secret auth (Slack/Telegram stubs)
• Parallel job execution with state machine and self-repair for stuck jobs
• Extensible tools: Built-in tools, WASM sandbox, MCP client, dynamic builder
• Persistent memory: Workspace with hybrid search (FTS + vector via RRF)
• Prompt injection defense: Sanitizer, validator, policy rules, leak detection
• Heartbeat system: Proactive periodic execution with checklist

Build & Test

# Format code
cargo fmt

# Lint (address warnings before committing)
cargo clippy --all --benches --tests --examples --all-features

# Run all tests
cargo test

# Run specific test
cargo test test_name

# Run with logging
RUST_LOG=near_agent=debug cargo run

Project Structure

src/
├── lib.rs              # Library root, module declarations
├── main.rs             # Entry point, CLI args, startup
├── config.rs           # Configuration from env vars
├── error.rs            # Error types (thiserror)
│
├── agent/              # Core agent logic
│   ├── agent_loop.rs   # Main Agent struct, message handling loop
│   ├── router.rs       # MessageIntent classification
│   ├── scheduler.rs    # Parallel job scheduling
│   ├── worker.rs       # Per-job execution with LLM reasoning
│   ├── self_repair.rs  # Stuck job detection and recovery
│   ├── heartbeat.rs    # Proactive periodic execution
│   ├── session.rs      # Session/thread/turn model with state machine
│   ├── session_manager.rs # Thread/session lifecycle management
│   ├── compaction.rs   # Context window management with turn summarization
│   ├── context_monitor.rs # Memory pressure detection
│   ├── undo.rs         # Turn-based undo/redo with checkpoints
│   ├── submission.rs   # Submission parsing (undo, redo, compact, clear, etc.)
│   └── task.rs         # Sub-task execution framework
│
├── channels/           # Multi-channel input
│   ├── channel.rs      # Channel trait, IncomingMessage, OutgoingResponse
│   ├── manager.rs      # ChannelManager merges streams
│   ├── cli/            # Full TUI with Ratatui
│   │   ├── mod.rs      # TuiChannel implementation
│   │   ├── app.rs      # Application state
│   │   ├── render.rs   # UI rendering
│   │   ├── events.rs   # Input handling
│   │   ├── overlay.rs  # Approval overlays
│   │   └── composer.rs # Message composition
│   ├── http.rs         # HTTP webhook (axum) with secret validation
│   ├── slack.rs        # Stub
│   └── telegram.rs     # Stub
│
├── safety/             # Prompt injection defense
│   ├── sanitizer.rs    # Pattern detection, content escaping
│   ├── validator.rs    # Input validation (length, encoding, patterns)
│   ├── policy.rs       # PolicyRule system with severity/actions
│   └── leak_detector.rs # Secret detection (API keys, tokens, etc.)
│
├── llm/                # LLM integration (NEAR AI only)
│   ├── provider.rs     # LlmProvider trait, message types
│   ├── nearai.rs       # NEAR AI chat-api implementation
│   ├── reasoning.rs    # Planning, tool selection, evaluation
│   └── session.rs      # Session token management with auto-renewal
│
├── tools/              # Extensible tool system
│   ├── tool.rs         # Tool trait, ToolOutput, ToolError
│   ├── registry.rs     # ToolRegistry for discovery
│   ├── sandbox.rs      # Process-based sandbox (stub, superseded by wasm/)
│   ├── builtin/        # Built-in tools
│   │   ├── echo.rs, time.rs, json.rs, http.rs
│   │   ├── file.rs     # ReadFile, WriteFile, ListDir, ApplyPatch
│   │   ├── shell.rs    # Shell command execution
│   │   ├── memory.rs   # Memory tools (search, write, read, tree)
│   │   └── marketplace.rs, ecommerce.rs, taskrabbit.rs, restaurant.rs (stubs)
│   ├── builder/        # Dynamic tool building
│   │   ├── core.rs     # BuildRequirement, SoftwareType, Language
│   │   ├── templates.rs # Project scaffolding
│   │   ├── testing.rs  # Test harness integration
│   │   └── validation.rs # WASM validation
│   ├── mcp/            # Model Context Protocol
│   │   ├── client.rs   # MCP client over HTTP
│   │   └── protocol.rs # JSON-RPC types
│   └── wasm/           # Full WASM sandbox (wasmtime)
│       ├── runtime.rs  # Module compilation and caching
│       ├── wrapper.rs  # Tool trait wrapper for WASM modules
│       ├── host.rs     # Host functions (logging, time, workspace)
│       ├── limits.rs   # Fuel metering and memory limiting
│       ├── allowlist.rs # Network endpoint allowlisting
│       ├── credential_injector.rs # Safe credential injection
│       ├── loader.rs   # WASM tool discovery from filesystem
│       ├── rate_limiter.rs # Per-tool rate limiting
│       └── storage.rs  # Linear memory persistence
│
├── workspace/          # Persistent memory system (OpenClaw-inspired)
│   ├── mod.rs          # Workspace struct, memory operations
│   ├── document.rs     # MemoryDocument, MemoryChunk, WorkspaceEntry
│   ├── chunker.rs      # Document chunking (800 tokens, 15% overlap)
│   ├── embeddings.rs   # EmbeddingProvider trait, OpenAI implementation
│   ├── search.rs       # Hybrid search with RRF algorithm
│   └── repository.rs   # PostgreSQL CRUD and search operations
│
├── context/            # Job context isolation
│   ├── state.rs        # JobState enum, JobContext, state machine
│   ├── memory.rs       # ActionRecord, ConversationMemory
│   └── manager.rs      # ContextManager for concurrent jobs
│
├── estimation/         # Cost/time/value estimation
│   ├── cost.rs         # CostEstimator
│   ├── time.rs         # TimeEstimator
│   ├── value.rs        # ValueEstimator (profit margins)
│   └── learner.rs      # Exponential moving average learning
│
├── evaluation/         # Success evaluation
│   ├── success.rs      # SuccessEvaluator trait, RuleBasedEvaluator, LlmEvaluator
│   └── metrics.rs      # MetricsCollector, QualityMetrics
│
├── secrets/            # Secrets management
│   ├── crypto.rs       # AES-256-GCM encryption
│   ├── store.rs        # Secret storage
│   └── types.rs        # Credential types
│
└── history/            # Persistence
    ├── store.rs        # PostgreSQL repositories
    └── analytics.rs    # Aggregation queries (JobStats, ToolStats)

Key Patterns

Error Handling

• Use thiserror for error types in error.rs
• Never use .unwrap() in production code (tests are fine)
• Map errors with context: .map_err(|e| SomeError::Variant { reason: e.to_string() })?

Async

• All I/O is async with tokio
• Use Arc<T> for shared state across tasks
• Use RwLock for concurrent read/write access

Traits for Extensibility

• Channel - Add new input sources
• Tool - Add new capabilities
• LlmProvider - Add new LLM backends
• SuccessEvaluator - Custom evaluation logic
• EmbeddingProvider - Add embedding backends (workspace search)

Tool Implementation

#[async_trait]
impl Tool for MyTool {
    fn name(&self) -> &str { "my_tool" }
    fn description(&self) -> &str { "Does something useful" }
    fn parameters_schema(&self) -> serde_json::Value {
        serde_json::json!({
            "type": "object",
            "properties": {
                "param": { "type": "string", "description": "A parameter" }
            },
            "required": ["param"]
        })
    }

    async fn execute(&self, params: serde_json::Value, ctx: &JobContext)
        -> Result<ToolOutput, ToolError>
    {
        let start = std::time::Instant::now();
        // ... do work ...
        Ok(ToolOutput::text("result", start.elapsed()))
    }

    fn requires_sanitization(&self) -> bool { true } // External data
}

State Transitions

Job states follow a defined state machine in context/state.rs:

Pending -> InProgress -> Completed -> Submitted -> Accepted
                     \-> Failed
                     \-> Stuck -> InProgress (recovery)
                              \-> Failed

Configuration

Environment variables (see .env.example):

DATABASE_URL=postgres://user:pass@localhost/near_agent

# NEAR AI (required)
NEARAI_SESSION_TOKEN=sess_...
NEARAI_MODEL=claude-3-5-sonnet-20241022
NEARAI_BASE_URL=https://private.near.ai

# Agent settings
AGENT_NAME=near-agent
MAX_PARALLEL_JOBS=5

# Embeddings (for semantic memory search)
OPENAI_API_KEY=sk-...                   # For OpenAI embeddings
# Or use NEAR AI embeddings:
# EMBEDDING_PROVIDER=nearai
# EMBEDDING_ENABLED=true
EMBEDDING_MODEL=text-embedding-3-small  # or text-embedding-3-large

# Heartbeat (proactive periodic execution)
HEARTBEAT_ENABLED=true
HEARTBEAT_INTERVAL_SECS=1800            # 30 minutes
HEARTBEAT_NOTIFY_CHANNEL=tui
HEARTBEAT_NOTIFY_USER=default

NEAR AI Provider

Uses the NEAR AI chat-api (https://api.near.ai/v1/responses) which provides:

• Unified access to multiple models (OpenAI, Anthropic, etc.)
• User authentication via session tokens
• Usage tracking and billing through NEAR AI

Session tokens have the format sess_xxx (37 characters). They are authenticated against the NEAR AI auth service.

Database

Single migration in migrations/V1__initial.sql. Tables:

Core:

• conversations - Multi-channel conversation tracking
• agent_jobs - Job metadata and status
• job_actions - Event-sourced tool executions
• dynamic_tools - Agent-built tools
• llm_calls - Cost tracking
• estimation_snapshots - Learning data

Workspace/Memory:

• memory_documents - Flexible path-based files (e.g., "context/vision.md", "daily/2024-01-15.md")
• memory_chunks - Chunked content with FTS (tsvector) and vector (pgvector) indexes
• heartbeat_state - Periodic execution tracking

Requires pgvector extension: CREATE EXTENSION IF NOT EXISTS vector;

Run migrations: refinery migrate -c refinery.toml

Safety Layer

All external tool output passes through SafetyLayer:

1. Sanitizer - Detects injection patterns, escapes dangerous content
2. Validator - Checks length, encoding, forbidden patterns
3. Policy - Rules with severity (Critical/High/Medium/Low) and actions (Block/Warn/Review/Sanitize)

Tool outputs are wrapped before reaching LLM:

<tool_output name="search" sanitized="true">
[escaped content]
</tool_output>

Testing

Tests are in mod tests {} blocks at the bottom of each file. Run specific module tests:

cargo test safety::sanitizer::tests
cargo test tools::registry::tests

Key test patterns:

• Unit tests for pure functions
• Async tests with #[tokio::test]
• No mocks, prefer real implementations or stubs

Current Limitations / TODOs

1. Slack/Telegram channels - Stubs only, need implementation
2. Domain-specific tools - marketplace.rs, restaurant.rs, taskrabbit.rs, ecommerce.rs return placeholder responses; need real API integrations
3. Integration tests - Need testcontainers setup for PostgreSQL
4. MCP stdio transport - Only HTTP transport implemented
5. WIT bindgen integration - Auto-extract tool description/schema from WASM modules (stubbed)
6. Capability granting after tool build - Built tools get empty capabilities; need UX for granting HTTP/secrets access
7. Tool versioning workflow - No version tracking or rollback for dynamically built tools

Completed

• ✅ Workspace integration - Memory tools registered, workspace passed to Agent and heartbeat
• ✅ WASM sandboxing - Full implementation in tools/wasm/ with fuel metering, memory limits, capabilities
• ✅ Dynamic tool building - tools/builder/ has LlmSoftwareBuilder with iterative build loop
• ✅ HTTP webhook security - Secret validation implemented, proper error handling (no panics)
• ✅ Embeddings integration - OpenAI and NEAR AI providers wired to workspace for semantic search
• ✅ Workspace system prompt - Identity files (AGENTS.md, SOUL.md, USER.md, IDENTITY.md) injected into LLM context
• ✅ Heartbeat notifications - Route through channel manager (broadcast API) instead of logging-only
• ✅ Auto-context compaction - Triggers automatically when context exceeds threshold
• ✅ Embedding backfill - Runs on startup when embeddings provider is enabled
• ✅ Clippy clean - All warnings addressed via config struct refactoring
• ✅ Tool approval enforcement - Tools with requires_approval() (shell, http, file write/patch, build_software) now gate execution, track auto-approved tools per session
• ✅ Tool definition refresh - Tool definitions refreshed each iteration so newly built tools become visible in same session
• ✅ Worker tool call handling - Uses respond_with_tools() to properly execute tool calls when select_tools() returns empty

Adding a New Tool

1. Create src/tools/builtin/my_tool.rs
2. Implement the Tool trait
3. Add mod my_tool; and pub use in src/tools/builtin/mod.rs
4. Register in ToolRegistry::register_builtin_tools() in registry.rs
5. Add tests

Adding a New Channel

1. Create src/channels/my_channel.rs
2. Implement the Channel trait
3. Add config in src/config.rs
4. Wire up in main.rs channel setup section

Debugging

# Verbose logging
RUST_LOG=near_agent=trace cargo run

# Just the agent module
RUST_LOG=near_agent::agent=debug cargo run

# With HTTP request logging
RUST_LOG=near_agent=debug,tower_http=debug cargo run

Code Style

• Use crate:: imports, not super::
• No pub use re-exports unless exposing to downstream consumers
• Prefer strong types over strings (enums, newtypes)
• Keep functions focused, extract helpers when logic is reused
• Comments for non-obvious logic only

Workspace & Memory System

Inspired by OpenClaw, the workspace provides persistent memory for agents with a flexible filesystem-like structure.

Key Principles

1. "Memory is files, not RAM" - If you want to remember something, write it explicitly
2. Flexible structure - Create any directory/file hierarchy you need
3. Self-documenting - Use README.md files to describe directory structure
4. Hybrid search - Combines FTS (keyword) + vector (semantic) via Reciprocal Rank Fusion

Filesystem Structure

workspace/
├── README.md              <- Root runbook/index
├── MEMORY.md              <- Long-term curated memory
├── HEARTBEAT.md           <- Periodic checklist
├── IDENTITY.md            <- Agent name, nature, vibe
├── SOUL.md                <- Core values
├── AGENTS.md              <- Behavior instructions
├── USER.md                <- User context
├── context/               <- Identity-related docs
│   ├── vision.md
│   └── priorities.md
├── daily/                 <- Daily logs
│   ├── 2024-01-15.md
│   └── 2024-01-16.md
├── projects/              <- Arbitrary structure
│   └── alpha/
│       ├── README.md
│       └── notes.md
└── ...

Using the Workspace

use crate::workspace::{Workspace, OpenAiEmbeddings, paths};

// Create workspace for a user
let workspace = Workspace::new("user_123", pool)
    .with_embeddings(Arc::new(OpenAiEmbeddings::new(api_key)));

// Read/write any path
let doc = workspace.read("projects/alpha/notes.md").await?;
workspace.write("context/priorities.md", "# Priorities\n\n1. Feature X").await?;
workspace.append("daily/2024-01-15.md", "Completed task X").await?;

// Convenience methods for well-known files
workspace.append_memory("User prefers dark mode").await?;
workspace.append_daily_log("Session note").await?;

// List directory contents
let entries = workspace.list("projects/").await?;

// Search (hybrid FTS + vector)
let results = workspace.search("dark mode preference", 5).await?;

// Get system prompt from identity files
let prompt = workspace.system_prompt().await?;

Memory Tools

Four tools for LLM use:

• memory_search - Hybrid search, MUST be called before answering questions about prior work
• memory_write - Write to any path (memory, daily_log, or custom paths)
• memory_read - Read any file by path
• memory_tree - View workspace structure as a tree (depth parameter, default 1)

Hybrid Search (RRF)

Combines full-text search (PostgreSQL ts_rank_cd) and vector similarity (pgvector cosine) using Reciprocal Rank Fusion:

score(d) = Σ 1/(k + rank(d)) for each method where d appears

Default k=60. Results from both methods are combined, with documents appearing in both getting boosted scores.

Heartbeat System

Proactive periodic execution (default: 30 minutes):

1. Reads HEARTBEAT.md checklist
2. Runs agent turn with checklist prompt
3. If findings, notifies via channel
4. If nothing, agent replies "HEARTBEAT_OK" (no notification)

use crate::agent::{HeartbeatConfig, spawn_heartbeat};

let config = HeartbeatConfig::default()
    .with_interval(Duration::from_secs(60 * 30))
    .with_notify("user_123", "telegram");

spawn_heartbeat(config, workspace, llm, response_tx);

Chunking Strategy

Documents are chunked for search indexing:

• Default: 800 words per chunk (roughly 800 tokens for English)
• 15% overlap between chunks for context preservation
• Minimum chunk size: 50 words (tiny trailing chunks merge with previous)