diff --git a/.env.example b/.env.example index 1510b08f..dfd7e55d 100644 --- a/.env.example +++ b/.env.example @@ -33,13 +33,27 @@ NEARAI_AUTH_URL=https://private.near.ai # LLM_BASE_URL=http://localhost:1234/v1 # LLM_API_KEY=sk-... # optional for local servers -# === OpenRouter (via OpenAI-compatible) === -# LLM_MODEL=anthropic/claude-sonnet-4 +# === OpenRouter (300+ models via OpenAI-compatible) === +# LLM_MODEL=anthropic/claude-sonnet-4 # see openrouter.ai/models for IDs # LLM_BACKEND=openai_compatible # LLM_BASE_URL=https://openrouter.ai/api/v1 # LLM_API_KEY=sk-or-... # LLM_EXTRA_HEADERS=HTTP-Referer:https://myapp.com,X-Title:MyApp +# === Together AI (via OpenAI-compatible) === +# LLM_MODEL=meta-llama/Llama-3.3-70B-Instruct-Turbo +# LLM_BACKEND=openai_compatible +# LLM_BASE_URL=https://api.together.xyz/v1 +# LLM_API_KEY=... + +# === Fireworks AI (via OpenAI-compatible) === +# LLM_MODEL=accounts/fireworks/models/llama4-maverick-instruct-basic +# LLM_BACKEND=openai_compatible +# LLM_BASE_URL=https://api.fireworks.ai/inference/v1 +# LLM_API_KEY=fw_... + +# For full provider setup guide see docs/LLM_PROVIDERS.md + # Channel Configuration # CLI is always enabled diff --git a/README.md b/README.md index dd5f3ba5..a75e897c 100644 --- a/README.md +++ b/README.md @@ -143,6 +143,23 @@ and secrets encryption (using your system keychain). Settings are persisted in t connected database; bootstrap variables (e.g. `DATABASE_URL`, `LLM_BACKEND`) are written to `~/.ironclaw/.env` so they are available before the database connects. +### Alternative LLM Providers + +IronClaw defaults to NEAR AI but works with any OpenAI-compatible endpoint. +Popular options include **OpenRouter** (300+ models), **Together AI**, **Fireworks AI**, +**Ollama** (local), and self-hosted servers like **vLLM** or **LiteLLM**. + +Select *"OpenAI-compatible"* in the wizard, or set environment variables directly: + +```env +LLM_BACKEND=openai_compatible +LLM_BASE_URL=https://openrouter.ai/api/v1 +LLM_API_KEY=sk-or-... +LLM_MODEL=anthropic/claude-sonnet-4 +``` + +See [docs/LLM_PROVIDERS.md](docs/LLM_PROVIDERS.md) for a full provider guide. + ## Security IronClaw implements defense in depth to protect your data and prevent misuse. diff --git a/docs/LLM_PROVIDERS.md b/docs/LLM_PROVIDERS.md new file mode 100644 index 00000000..b6d6cf12 --- /dev/null +++ b/docs/LLM_PROVIDERS.md @@ -0,0 +1,172 @@ +# LLM Provider Configuration + +IronClaw defaults to NEAR AI for model access, but supports any OpenAI-compatible +endpoint as well as Anthropic and Ollama directly. This guide covers the most common +configurations. + +## Provider Overview + +| Provider | Backend value | Requires API key | Notes | +|---|---|---|---| +| NEAR AI | `nearai` | OAuth (browser) | Default; multi-model | +| Anthropic | `anthropic` | `ANTHROPIC_API_KEY` | Claude models | +| OpenAI | `openai` | `OPENAI_API_KEY` | GPT models | +| Ollama | `ollama` | No | Local inference | +| OpenRouter | `openai_compatible` | `LLM_API_KEY` | 300+ models | +| Together AI | `openai_compatible` | `LLM_API_KEY` | Fast inference | +| Fireworks AI | `openai_compatible` | `LLM_API_KEY` | Fast inference | +| vLLM / LiteLLM | `openai_compatible` | Optional | Self-hosted | +| LM Studio | `openai_compatible` | No | Local GUI | + +--- + +## NEAR AI (default) + +No additional configuration required. On first run, `ironclaw onboard` opens a browser +for OAuth authentication. Credentials are saved to `~/.ironclaw/session.json`. + +```env +NEARAI_MODEL=claude-3-5-sonnet-20241022 +NEARAI_BASE_URL=https://private.near.ai +``` + +--- + +## Anthropic (Claude) + +```env +LLM_BACKEND=anthropic +ANTHROPIC_API_KEY=sk-ant-... +``` + +Popular models: `claude-sonnet-4-20250514`, `claude-3-5-sonnet-20241022`, `claude-3-5-haiku-20241022` + +--- + +## OpenAI (GPT) + +```env +LLM_BACKEND=openai +OPENAI_API_KEY=sk-... +``` + +Popular models: `gpt-4o`, `gpt-4o-mini`, `o3-mini` + +--- + +## Ollama (local) + +Install Ollama from [ollama.com](https://ollama.com), pull a model, then: + +```env +LLM_BACKEND=ollama +OLLAMA_MODEL=llama3.2 +# OLLAMA_BASE_URL=http://localhost:11434 # default +``` + +Pull a model first: `ollama pull llama3.2` + +--- + +## OpenAI-Compatible Endpoints + +All providers below use `LLM_BACKEND=openai_compatible`. Set `LLM_BASE_URL` to the +provider's OpenAI-compatible endpoint and `LLM_API_KEY` to your API key. + +### OpenRouter + +[OpenRouter](https://openrouter.ai) routes to 300+ models from a single API key. + +```env +LLM_BACKEND=openai_compatible +LLM_BASE_URL=https://openrouter.ai/api/v1 +LLM_API_KEY=sk-or-... +LLM_MODEL=anthropic/claude-sonnet-4 +``` + +Popular OpenRouter model IDs: + +| Model | ID | +|---|---| +| Claude Sonnet 4 | `anthropic/claude-sonnet-4` | +| GPT-4o | `openai/gpt-4o` | +| Llama 4 Maverick | `meta-llama/llama-4-maverick` | +| Gemini 2.0 Flash | `google/gemini-2.0-flash-001` | +| Mistral Small | `mistralai/mistral-small-3.1-24b-instruct` | + +Browse all models at [openrouter.ai/models](https://openrouter.ai/models). + +### Together AI + +[Together AI](https://www.together.ai) provides fast inference for open-source models. + +```env +LLM_BACKEND=openai_compatible +LLM_BASE_URL=https://api.together.xyz/v1 +LLM_API_KEY=... +LLM_MODEL=meta-llama/Llama-3.3-70B-Instruct-Turbo +``` + +Popular Together AI model IDs: + +| Model | ID | +|---|---| +| Llama 3.3 70B | `meta-llama/Llama-3.3-70B-Instruct-Turbo` | +| DeepSeek R1 | `deepseek-ai/DeepSeek-R1` | +| Qwen 2.5 72B | `Qwen/Qwen2.5-72B-Instruct-Turbo` | + +### Fireworks AI + +[Fireworks AI](https://fireworks.ai) offers fast inference with compound AI system support. + +```env +LLM_BACKEND=openai_compatible +LLM_BASE_URL=https://api.fireworks.ai/inference/v1 +LLM_API_KEY=fw_... +LLM_MODEL=accounts/fireworks/models/llama4-maverick-instruct-basic +``` + +### vLLM / LiteLLM (self-hosted) + +For self-hosted inference servers: + +```env +LLM_BACKEND=openai_compatible +LLM_BASE_URL=http://localhost:8000/v1 +LLM_API_KEY=token-abc123 # set to any string if auth is not configured +LLM_MODEL=meta-llama/Llama-3.1-8B-Instruct +``` + +LiteLLM proxy (forwards to any backend, including Bedrock, Vertex, Azure): + +```env +LLM_BACKEND=openai_compatible +LLM_BASE_URL=http://localhost:4000/v1 +LLM_API_KEY=sk-... +LLM_MODEL=gpt-4o # as configured in litellm config.yaml +``` + +### LM Studio (local GUI) + +Start LM Studio's local server, then: + +```env +LLM_BACKEND=openai_compatible +LLM_BASE_URL=http://localhost:1234/v1 +LLM_MODEL=llama-3.2-3b-instruct-q4_K_M +# LLM_API_KEY is not required for LM Studio +``` + +--- + +## Using the Setup Wizard + +Instead of editing `.env` manually, run the onboarding wizard: + +```bash +ironclaw onboard +``` + +Select **"OpenAI-compatible"** for OpenRouter, Together AI, Fireworks, vLLM, LiteLLM, +or LM Studio. You will be prompted for the base URL and (optionally) an API key. +The model name is configured in the following step.