mirror of
https://github.com/outbackdingo/optimclaw.git
synced 2026-08-25 14:53:34 +00:00
d73e35cfb0
* feat: add AWS Bedrock LLM provider via native Converse API * fix: use JSON parsing for tool result error detection instead of brittle substring matching * refactor: extract duplicated inference config builder into helper function * fix: address review feedback — safe casts, input validation, and tests - Safe u32→i32 cast for max_tokens using try_from with clamp - Remove brittle string-based error detection fallback for tool results - Validate BEDROCK_CROSS_REGION against allowed values (us/eu/apac/global) - Validate message list is non-empty before Converse API call - Log when using default us-east-1 region - Update llm_backend doc comment to list all backends - Add tests for build_inference_config and empty message handling * fix: persist AWS_PROFILE for Bedrock named profile auth The wizard collected the profile name but only printed a hint to set it manually. Now it saves to settings and writes AWS_PROFILE to the bootstrap .env, consistent with how BEDROCK_REGION and other Bedrock settings are persisted. * feat: gate AWS Bedrock behind optional `bedrock` feature flag The AWS SDK dependencies (aws-config, aws-sdk-bedrockruntime, aws-smithy-types) require cmake and a C compiler to build aws-lc-sys. Gate them behind an opt-in `bedrock` feature flag so default builds are unaffected. Build with: cargo build --features bedrock All config, settings, and wizard code stays unconditional (no AWS deps) so users can configure Bedrock even without the feature compiled — they get a clear error at startup directing them to rebuild. * fix: address review feedback and adapt Bedrock provider to registry architecture (takeover #345) - Resolve merge conflicts with main's registry-based provider system - Add missing cache_creation_input_tokens/cache_read_input_tokens fields - Add missing content_parts field in test ChatMessage - Fix string literal type mismatches in wizard env_vars (.to_string()) - Remove non-functional bearer token auth (AWS_BEARER_TOKEN_BEDROCK) from wizard and documentation per reviewer feedback from @zmanian and @serrrfirat - Remove stale BEDROCK_ACCESS_KEY proxy entry from provider table - Update Bedrock provider to use is_bedrock string check (LlmBackend enum removed) - Add bedrock_profile fallback from settings in config resolution [skip-regression-check] Co-Authored-By: cgorski <[email protected]> Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: use main's Cargo.lock as base to preserve dependency versions Regenerating Cargo.lock from scratch caused transitive dependency version drift that broke the html_to_markdown fixture test in CI. Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: bedrock config bugs — spurious warning, alias normalization, profile fallback - Move is_bedrock check before unknown-backend warning to prevent spurious "unknown backend" log for bedrock users - Normalize backend aliases ("aws", "aws_bedrock") to "bedrock" so the provider factory matches correctly - Add settings.bedrock_profile fallback for AWS_PROFILE, consistent with region and cross_region resolution [skip-regression-check] Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: address Copilot review feedback — bearer token cleanup, stop_sequences, model dedup - Remove stale bearer token refs from setup README and CHANGELOG - Remove dead bedrock_api_key secret injection mapping - Pass stop_sequences through to Bedrock InferenceConfiguration - Remove "API key" from wizard menu description (bearer token removed) - Skip duplicate LLM_MODEL write for bedrock backend in wizard - Fix cargo fmt formatting [skip-regression-check] Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: address review feedback — async new(), remove LiteLLM entry, wizard fixes - Remove dead LiteLLM-based bedrock entry from providers.json (native Converse API intercepts before registry lookup) - Make BedrockProvider::new() async to avoid block_in_place panic in current_thread runtimes; propagate async to create_llm_provider, build_provider_chain, and init_llm - Document CMake build prerequisite in docs/LLM_PROVIDERS.md - Clear bedrock_profile when user selects "default credentials" in wizard - Fix selected_model clearing to match established pattern (conditional on provider switch, not unconditional) - Add regression tests for bedrock model preservation and profile clearing Addresses review feedback from @zmanian on PR #713. Streaming support tracked in #741. [skip-regression-check] Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: address remaining review comments — CLAUDE.md backends, wizard UX - Add `bedrock` to CLAUDE.md inline backend list (#10) - Skip full setup re-run when keeping existing Bedrock config (#11) - Clear stale bedrock_profile on empty named-profile input (#12) - Add regression test for empty profile clearing Co-Authored-By: Claude Opus 4.6 <[email protected]> --------- Co-authored-by: Chris Gorski <[email protected]> Co-authored-by: cgorski <[email protected]> Co-authored-by: Claude Opus 4.6 <[email protected]>
228 lines
6.3 KiB
Markdown
228 lines
6.3 KiB
Markdown
# LLM Provider Configuration
|
|
|
|
IronClaw defaults to NEAR AI for model access, but supports any OpenAI-compatible
|
|
endpoint as well as Anthropic and Ollama directly. This guide covers the most common
|
|
configurations.
|
|
|
|
## Provider Overview
|
|
|
|
| Provider | Backend value | Requires API key | Notes |
|
|
|---|---|---|---|
|
|
| NEAR AI | `nearai` | OAuth (browser) | Default; multi-model |
|
|
| Anthropic | `anthropic` | `ANTHROPIC_API_KEY` | Claude models |
|
|
| OpenAI | `openai` | `OPENAI_API_KEY` | GPT models |
|
|
| Google Gemini | `gemini` | `GEMINI_API_KEY` | Gemini models |
|
|
| io.net | `ionet` | `IONET_API_KEY` | Intelligence API |
|
|
| Mistral | `mistral` | `MISTRAL_API_KEY` | Mistral models |
|
|
| Yandex AI Studio | `yandex` | `YANDEX_API_KEY` | YandexGPT models |
|
|
| Cloudflare Workers AI | `cloudflare` | `CLOUDFLARE_API_KEY` | Access to Workers AI |
|
|
| Ollama | `ollama` | No | Local inference |
|
|
| AWS Bedrock | `bedrock` | AWS credentials | Native Converse API |
|
|
| OpenRouter | `openai_compatible` | `LLM_API_KEY` | 300+ models |
|
|
| Together AI | `openai_compatible` | `LLM_API_KEY` | Fast inference |
|
|
| Fireworks AI | `openai_compatible` | `LLM_API_KEY` | Fast inference |
|
|
| vLLM / LiteLLM | `openai_compatible` | Optional | Self-hosted |
|
|
| LM Studio | `openai_compatible` | No | Local GUI |
|
|
|
|
---
|
|
|
|
## NEAR AI (default)
|
|
|
|
No additional configuration required. On first run, `ironclaw onboard` opens a browser
|
|
for OAuth authentication. Credentials are saved to `~/.ironclaw/session.json`.
|
|
|
|
```env
|
|
NEARAI_MODEL=claude-3-5-sonnet-20241022
|
|
NEARAI_BASE_URL=https://private.near.ai
|
|
```
|
|
|
|
---
|
|
|
|
## Anthropic (Claude)
|
|
|
|
```env
|
|
LLM_BACKEND=anthropic
|
|
ANTHROPIC_API_KEY=sk-ant-...
|
|
```
|
|
|
|
Popular models: `claude-sonnet-4-20250514`, `claude-3-5-sonnet-20241022`, `claude-3-5-haiku-20241022`
|
|
|
|
---
|
|
|
|
## OpenAI (GPT)
|
|
|
|
```env
|
|
LLM_BACKEND=openai
|
|
OPENAI_API_KEY=sk-...
|
|
```
|
|
|
|
Popular models: `gpt-4o`, `gpt-4o-mini`, `o3-mini`
|
|
|
|
---
|
|
|
|
## Ollama (local)
|
|
|
|
Install Ollama from [ollama.com](https://ollama.com), pull a model, then:
|
|
|
|
```env
|
|
LLM_BACKEND=ollama
|
|
OLLAMA_MODEL=llama3.2
|
|
# OLLAMA_BASE_URL=http://localhost:11434 # default
|
|
```
|
|
|
|
Pull a model first: `ollama pull llama3.2`
|
|
|
|
---
|
|
|
|
## AWS Bedrock (requires `--features bedrock`)
|
|
|
|
Uses the native AWS Converse API via `aws-sdk-bedrockruntime`. Supports standard AWS
|
|
authentication methods: IAM credentials, SSO profiles, and instance roles.
|
|
|
|
> **Build prerequisite:** The `aws-lc-sys` crate (transitive dependency via AWS SDK)
|
|
> requires **CMake** to compile. Install it before building with `--features bedrock`:
|
|
> - macOS: `brew install cmake`
|
|
> - Ubuntu/Debian: `sudo apt install cmake`
|
|
> - Fedora: `sudo dnf install cmake`
|
|
|
|
### With AWS credentials (IAM, SSO, instance roles)
|
|
|
|
```env
|
|
LLM_BACKEND=bedrock
|
|
BEDROCK_MODEL=anthropic.claude-opus-4-6-v1
|
|
BEDROCK_REGION=us-east-1
|
|
BEDROCK_CROSS_REGION=us
|
|
# AWS_PROFILE=my-sso-profile # optional, for named profiles
|
|
```
|
|
|
|
The AWS SDK credential chain automatically resolves credentials from environment
|
|
variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`), shared credentials file
|
|
(`~/.aws/credentials`), SSO profiles, and EC2/ECS instance roles.
|
|
|
|
### Cross-region inference
|
|
|
|
Set `BEDROCK_CROSS_REGION` to route requests across AWS regions for capacity:
|
|
|
|
| Prefix | Routing |
|
|
|---|---|
|
|
| `us` | US regions (us-east-1, us-east-2, us-west-2) |
|
|
| `eu` | European regions |
|
|
| `apac` | Asia-Pacific regions |
|
|
| `global` | All commercial AWS regions |
|
|
| _(unset)_ | Single-region only |
|
|
|
|
### Popular Bedrock model IDs
|
|
|
|
| Model | ID |
|
|
|---|---|
|
|
| Claude Opus 4.6 | `anthropic.claude-opus-4-6-v1` |
|
|
| Claude Sonnet 4.5 | `anthropic.claude-sonnet-4-5-20250929-v1:0` |
|
|
| Claude Haiku 4.5 | `anthropic.claude-haiku-4-5-20251001-v1:0` |
|
|
| Amazon Nova Pro | `amazon.nova-pro-v1:0` |
|
|
| Llama 4 Maverick | `meta.llama4-maverick-17b-instruct-v1:0` |
|
|
|
|
---
|
|
|
|
## OpenAI-Compatible Endpoints
|
|
|
|
All providers below use `LLM_BACKEND=openai_compatible`. Set `LLM_BASE_URL` to the
|
|
provider's OpenAI-compatible endpoint and `LLM_API_KEY` to your API key.
|
|
|
|
### OpenRouter
|
|
|
|
[OpenRouter](https://openrouter.ai) routes to 300+ models from a single API key.
|
|
|
|
```env
|
|
LLM_BACKEND=openai_compatible
|
|
LLM_BASE_URL=https://openrouter.ai/api/v1
|
|
LLM_API_KEY=sk-or-...
|
|
LLM_MODEL=anthropic/claude-sonnet-4
|
|
```
|
|
|
|
Popular OpenRouter model IDs:
|
|
|
|
| Model | ID |
|
|
|---|---|
|
|
| Claude Sonnet 4 | `anthropic/claude-sonnet-4` |
|
|
| GPT-4o | `openai/gpt-4o` |
|
|
| Llama 4 Maverick | `meta-llama/llama-4-maverick` |
|
|
| Gemini 2.0 Flash | `google/gemini-2.0-flash-001` |
|
|
| Mistral Small | `mistralai/mistral-small-3.1-24b-instruct` |
|
|
|
|
Browse all models at [openrouter.ai/models](https://openrouter.ai/models).
|
|
|
|
### Together AI
|
|
|
|
[Together AI](https://www.together.ai) provides fast inference for open-source models.
|
|
|
|
```env
|
|
LLM_BACKEND=openai_compatible
|
|
LLM_BASE_URL=https://api.together.xyz/v1
|
|
LLM_API_KEY=...
|
|
LLM_MODEL=meta-llama/Llama-3.3-70B-Instruct-Turbo
|
|
```
|
|
|
|
Popular Together AI model IDs:
|
|
|
|
| Model | ID |
|
|
|---|---|
|
|
| Llama 3.3 70B | `meta-llama/Llama-3.3-70B-Instruct-Turbo` |
|
|
| DeepSeek R1 | `deepseek-ai/DeepSeek-R1` |
|
|
| Qwen 2.5 72B | `Qwen/Qwen2.5-72B-Instruct-Turbo` |
|
|
|
|
### Fireworks AI
|
|
|
|
[Fireworks AI](https://fireworks.ai) offers fast inference with compound AI system support.
|
|
|
|
```env
|
|
LLM_BACKEND=openai_compatible
|
|
LLM_BASE_URL=https://api.fireworks.ai/inference/v1
|
|
LLM_API_KEY=fw_...
|
|
LLM_MODEL=accounts/fireworks/models/llama4-maverick-instruct-basic
|
|
```
|
|
|
|
### vLLM / LiteLLM (self-hosted)
|
|
|
|
For self-hosted inference servers:
|
|
|
|
```env
|
|
LLM_BACKEND=openai_compatible
|
|
LLM_BASE_URL=http://localhost:8000/v1
|
|
LLM_API_KEY=token-abc123 # set to any string if auth is not configured
|
|
LLM_MODEL=meta-llama/Llama-3.1-8B-Instruct
|
|
```
|
|
|
|
LiteLLM proxy (forwards to any backend, including Bedrock, Vertex, Azure):
|
|
|
|
```env
|
|
LLM_BACKEND=openai_compatible
|
|
LLM_BASE_URL=http://localhost:4000/v1
|
|
LLM_API_KEY=sk-...
|
|
LLM_MODEL=gpt-4o # as configured in litellm config.yaml
|
|
```
|
|
|
|
### LM Studio (local GUI)
|
|
|
|
Start LM Studio's local server, then:
|
|
|
|
```env
|
|
LLM_BACKEND=openai_compatible
|
|
LLM_BASE_URL=http://localhost:1234/v1
|
|
LLM_MODEL=llama-3.2-3b-instruct-q4_K_M
|
|
# LLM_API_KEY is not required for LM Studio
|
|
```
|
|
|
|
---
|
|
|
|
## Using the Setup Wizard
|
|
|
|
Instead of editing `.env` manually, run the onboarding wizard:
|
|
|
|
```bash
|
|
ironclaw onboard
|
|
```
|
|
|
|
Select **"OpenAI-compatible"** for OpenRouter, Together AI, Fireworks, vLLM, LiteLLM,
|
|
or LM Studio. You will be prompted for the base URL and (optionally) an API key.
|
|
The model name is configured in the following step.
|