mirror of
https://github.com/outbackdingo/optimclaw.git
synced 2026-08-29 00:49:31 +00:00
Merge origin/main into feat/gemini-cli-oauth and resolve conflicts
This commit is contained in:
Artem
@@ -12,7 +12,12 @@ the most common configurations.
|
||||
| Anthropic | `anthropic` | `ANTHROPIC_API_KEY` | Claude models |
|
||||
| OpenAI | `openai` | `OPENAI_API_KEY` | GPT models |
|
||||
| Google Gemini | `gemini_oauth` | OAuth (browser) | Gemini models; function calling |
|
||||
| io.net | `ionet` | `IONET_API_KEY` | Intelligence API |
|
||||
| Mistral | `mistral` | `MISTRAL_API_KEY` | Mistral models |
|
||||
| Yandex AI Studio | `yandex` | `YANDEX_API_KEY` | YandexGPT models |
|
||||
| Cloudflare Workers AI | `cloudflare` | `CLOUDFLARE_API_KEY` | Access to Workers AI |
|
||||
| Ollama | `ollama` | No | Local inference |
|
||||
| AWS Bedrock | `bedrock` | AWS credentials | Native Converse API |
|
||||
| OpenRouter | `openai_compatible` | `LLM_API_KEY` | 300+ models |
|
||||
| Together AI | `openai_compatible` | `LLM_API_KEY` | Fast inference |
|
||||
| Fireworks AI | `openai_compatible` | `LLM_API_KEY` | Fast inference |
|
||||
@@ -110,6 +115,55 @@ Pull a model first: `ollama pull llama3.2`
|
||||
|
||||
---
|
||||
|
||||
## AWS Bedrock (requires `--features bedrock`)
|
||||
|
||||
Uses the native AWS Converse API via `aws-sdk-bedrockruntime`. Supports standard AWS
|
||||
authentication methods: IAM credentials, SSO profiles, and instance roles.
|
||||
|
||||
> **Build prerequisite:** The `aws-lc-sys` crate (transitive dependency via AWS SDK)
|
||||
> requires **CMake** to compile. Install it before building with `--features bedrock`:
|
||||
> - macOS: `brew install cmake`
|
||||
> - Ubuntu/Debian: `sudo apt install cmake`
|
||||
> - Fedora: `sudo dnf install cmake`
|
||||
|
||||
### With AWS credentials (IAM, SSO, instance roles)
|
||||
|
||||
```env
|
||||
LLM_BACKEND=bedrock
|
||||
BEDROCK_MODEL=anthropic.claude-opus-4-6-v1
|
||||
BEDROCK_REGION=us-east-1
|
||||
BEDROCK_CROSS_REGION=us
|
||||
# AWS_PROFILE=my-sso-profile # optional, for named profiles
|
||||
```
|
||||
|
||||
The AWS SDK credential chain automatically resolves credentials from environment
|
||||
variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`), shared credentials file
|
||||
(`~/.aws/credentials`), SSO profiles, and EC2/ECS instance roles.
|
||||
|
||||
### Cross-region inference
|
||||
|
||||
Set `BEDROCK_CROSS_REGION` to route requests across AWS regions for capacity:
|
||||
|
||||
| Prefix | Routing |
|
||||
|---|---|
|
||||
| `us` | US regions (us-east-1, us-east-2, us-west-2) |
|
||||
| `eu` | European regions |
|
||||
| `apac` | Asia-Pacific regions |
|
||||
| `global` | All commercial AWS regions |
|
||||
| _(unset)_ | Single-region only |
|
||||
|
||||
### Popular Bedrock model IDs
|
||||
|
||||
| Model | ID |
|
||||
|---|---|
|
||||
| Claude Opus 4.6 | `anthropic.claude-opus-4-6-v1` |
|
||||
| Claude Sonnet 4.5 | `anthropic.claude-sonnet-4-5-20250929-v1:0` |
|
||||
| Claude Haiku 4.5 | `anthropic.claude-haiku-4-5-20251001-v1:0` |
|
||||
| Amazon Nova Pro | `amazon.nova-pro-v1:0` |
|
||||
| Llama 4 Maverick | `meta.llama4-maverick-17b-instruct-v1:0` |
|
||||
|
||||
---
|
||||
|
||||
## OpenAI-Compatible Endpoints
|
||||
|
||||
All providers below use `LLM_BACKEND=openai_compatible`. Set `LLM_BASE_URL` to the
|
||||
|
||||
@@ -0,0 +1,195 @@
|
||||
# Smart Model Routing for IronClaw
|
||||
|
||||
**Status:** Implemented
|
||||
**Author:** Microwave
|
||||
**Date:** 2026-02-19
|
||||
|
||||
## What
|
||||
|
||||
Automatic model selection based on request complexity. The router analyzes each user message and selects an appropriate model tier (flash/standard/pro/frontier), then maps that tier to a configured model.
|
||||
|
||||
## Why
|
||||
|
||||
1. **Cost optimization** — Simple requests ("hi", "what time is it") don't need expensive models
|
||||
2. **User experience** — Simple requests return faster with lightweight models
|
||||
3. **NEAR AI native** — Default backend uses NEAR AI inference where costs vary by model
|
||||
4. **Zero-config value** — Users benefit immediately without configuration
|
||||
5. **Not just power users** — Everyone gets smart defaults, power users can override
|
||||
|
||||
## How
|
||||
|
||||
### Architecture
|
||||
|
||||
```
|
||||
User Message
|
||||
│
|
||||
▼
|
||||
┌──────────────────┐
|
||||
│ Pattern Overrides │ ← Fast-path for obvious cases (greetings, security audits)
|
||||
└────────┬─────────┘
|
||||
│ no match
|
||||
▼
|
||||
┌──────────────────┐
|
||||
│ Complexity Scorer │ ← 13-dimension analysis
|
||||
└────────┬─────────┘
|
||||
│ score 0-100
|
||||
▼
|
||||
┌──────────────────┐
|
||||
│ Tier Mapping │ ← 0-15: flash, 16-40: standard, 41-65: pro, 66+: frontier
|
||||
└────────┬─────────┘
|
||||
│ tier
|
||||
▼
|
||||
┌──────────────────┐
|
||||
│ Model Selection │ ← Currently: cheap provider (Flash/Standard/Pro) vs primary (Frontier)
|
||||
└────────┬─────────┘ Target: per-tier model mapping via config
|
||||
│
|
||||
▼
|
||||
LLM Provider
|
||||
```
|
||||
|
||||
### Complexity Scorer (13 Dimensions)
|
||||
|
||||
Each dimension produces a 0-100 score. Weighted sum determines total.
|
||||
|
||||
| Dimension | Weight | Signals |
|
||||
|-----------|--------|---------|
|
||||
| Reasoning Words | 14% | "why", "explain", "compare", "trade-offs" |
|
||||
| Token Estimate | 12% | Prompt length |
|
||||
| Code Indicators | 10% | Backticks, syntax, "implement", "PR" |
|
||||
| Multi-Step | 10% | "first", "then", "after", "steps" |
|
||||
| Domain Specific | 10% | Technical terms (configurable) |
|
||||
| Creativity | 7% | "write", "summarize", "tweet", "blog" |
|
||||
| Question Complexity | 7% | Multiple questions, open-ended starters |
|
||||
| Precision | 6% | Numbers, "exactly", "calculate" |
|
||||
| Ambiguity | 5% | Vague references |
|
||||
| Context Dependency | 5% | "previous", "you said" |
|
||||
| Sentence Complexity | 5% | Commas, conjunctions, clause depth |
|
||||
| Tool Likelihood | 5% | "read", "deploy", "install" |
|
||||
| Safety Sensitivity | 4% | "password", "auth", "vulnerability" |
|
||||
|
||||
**Multi-dimensional boost:** +30% when 3+ dimensions score above threshold.
|
||||
|
||||
### Tier Boundaries
|
||||
|
||||
| Score | Tier | Typical Use Case |
|
||||
|-------|------|------------------|
|
||||
| 0-15 | flash | Greetings, acknowledgments, quick lookups |
|
||||
| 16-40 | standard | Writing, comparisons, defined tasks |
|
||||
| 41-65 | pro | Multi-step analysis, code review |
|
||||
| 66+ | frontier | Critical decisions, security audits |
|
||||
|
||||
### Pattern Overrides
|
||||
|
||||
Fast-path rules that bypass scoring for obvious cases:
|
||||
|
||||
```yaml
|
||||
# Force flash tier
|
||||
- "^(hi|hello|hey|thanks|ok|sure|yes|no)$"
|
||||
- "^what.*(time|date|day)"
|
||||
|
||||
# Force frontier tier
|
||||
- "security.*(audit|review|scan)"
|
||||
- "vulnerabilit(y|ies).*(review|scan|check|audit)"
|
||||
|
||||
# Force pro tier
|
||||
- "deploy.*(mainnet|production)"
|
||||
```
|
||||
|
||||
### Configuration
|
||||
|
||||
> **Note:** The current implementation supports smart routing via
|
||||
> `NEARAI_CHEAP_MODEL` and `SMART_ROUTING_CASCADE` env vars, plus
|
||||
> `domain_keywords` on `SmartRoutingConfig`. The full `llm.routing` YAML
|
||||
> schema below is the target design — not all knobs are wired yet.
|
||||
|
||||
**Default (zero-config):**
|
||||
```yaml
|
||||
llm:
|
||||
routing:
|
||||
enabled: true # default
|
||||
```
|
||||
|
||||
**Power user overrides (target schema):**
|
||||
```yaml
|
||||
llm:
|
||||
routing:
|
||||
enabled: true
|
||||
tiers:
|
||||
flash: "claude-3-5-haiku-latest"
|
||||
standard: "claude-sonnet-4-5-latest"
|
||||
pro: "claude-sonnet-4-5-latest"
|
||||
frontier: "claude-opus-4-5-latest"
|
||||
thinking:
|
||||
pro: "low"
|
||||
frontier: "medium"
|
||||
overrides:
|
||||
- pattern: "my-custom-pattern"
|
||||
tier: "pro"
|
||||
domain_keywords: # Custom keywords for your domain
|
||||
- "mycompany"
|
||||
- "myproduct"
|
||||
- "internal-tool"
|
||||
```
|
||||
|
||||
If `domain_keywords` is not set, uses `DEFAULT_DOMAIN_KEYWORDS` which covers common web3/infra terms.
|
||||
|
||||
**Disable routing (pin model):**
|
||||
```yaml
|
||||
llm:
|
||||
routing:
|
||||
enabled: false
|
||||
model: "claude-opus-4-5"
|
||||
```
|
||||
|
||||
**Bring your own keys:**
|
||||
```yaml
|
||||
llm:
|
||||
backend: anthropic
|
||||
api_key: "sk-..."
|
||||
routing:
|
||||
enabled: true # still works with external providers
|
||||
```
|
||||
|
||||
### Integration Points
|
||||
|
||||
1. **RoutingProvider** — New wrapper implementing `LlmProvider` trait (like `FailoverProvider`)
|
||||
2. **Scorer** — Pure function, no I/O, fast (~1ms)
|
||||
3. **Config schema** — Extend `LlmConfig` with `routing` section
|
||||
4. **Telemetry** — Log routing decisions for observability
|
||||
|
||||
### Model Agnosticism
|
||||
|
||||
**Critical:** No hardcoded model names in the router logic itself.
|
||||
|
||||
- Tier→model mappings come from config
|
||||
- Default mappings use `-latest` patterns where supported
|
||||
- NEAR AI backend handles actual model resolution
|
||||
- Router only knows about tiers
|
||||
|
||||
### Layers of Control
|
||||
|
||||
| Layer | User Type | Config |
|
||||
|-------|-----------|--------|
|
||||
| 1. Zero-config | Everyone | `routing.enabled: true` (default) |
|
||||
| 2. Tier tuning | Power users | Custom `routing.tiers` mapping |
|
||||
| 3. Pattern overrides | Power users | Custom `routing.overrides` |
|
||||
| 4. Model pinning | Power users | `routing.enabled: false` + `model: X` |
|
||||
| 5. Own API keys | Power users | `backend: anthropic` + `api_key` |
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
1. [x] Port scorer to Rust (`src/llm/smart_routing.rs`)
|
||||
2. [x] Implement router wrapper (`src/llm/smart_routing.rs`)
|
||||
3. [x] Extend config schema (`src/config.rs`)
|
||||
4. [x] Wire into provider creation (`src/llm/mod.rs`)
|
||||
5. [x] Add telemetry/logging
|
||||
6. [x] Tests with real conversation samples
|
||||
7. [x] Codex + Gemini security review
|
||||
8. [x] Documentation updated (this spec)
|
||||
|
||||
## Expected Outcomes
|
||||
|
||||
- **50-70% cost reduction** for typical usage patterns
|
||||
- **Faster responses** for simple requests
|
||||
- **Zero config required** for default benefits
|
||||
- **Full control** for power users who want it
|
||||
Reference in New Issue
Block a user