gitops/optimclaw
1
0
Fork 0
mirror of https://github.com/outbackdingo/optimclaw.git synced 2026-08-25 14:53:34 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
cb01800f73d220e93852789ec0315b26b24b0edd
optimclaw/docs/LLM_PROVIDERS.md
T
ArtemGitHub[email protected] <[email protected]>Claude Opus 4.6
 8638895879 feat(gemini_oauth): full Gemini CLI OAuth integration with Cloud Code API (#1356) 
* feat: integrate Gemini CLI OAuth with Cloud Code API

- Add gemini_oauth.rs: full OAuth flow with PKCE, token refresh,
  and Cloud Code project discovery (loadCodeAssist + onboardUser)
- Route preview/gemini-3 models through cloudcode-pa.googleapis.com
  with proper project ID injection in request payload
- Trigger OAuth login during onboarding wizard (not first chat message)
- Support manual redirect URL paste as fallback (tokio::select race)
- Parse 429 rate-limit errors with retry_after from Google response
- Add static model list: gemini-1.5/2.0/2.5/3.0/3.1 variants
- Add GeminiOauthConfig with default credentials path (~/.gemini/)

* feat(gemini): implement function calling, generationConfig, and update models

- Implement function calling support (functionDeclarations, functionResponse)
- Add functionCall SSE parsing and empty stream retry support
- Add generationConfig (temperature, maxOutputTokens)
- Add thinkingConfig for Gemini 3 and thinking models
- Add toolConfig (functionCallingConfig.mode)
- Fix .expect() panics with .ok_or_else()
- Restrict oauth credentials file permissions to 0600
- Update docs and FEATURE_PARITY.md
- Update wizard to current Gemini 3.1 and 2.5 models

* fix: address code review issues in gemini-cli OAuth integration

- Add cache_read_input_tokens/cache_creation_input_tokens fields (value 0)
- Implement manual Debug for OAuthCredential to redact tokens
- Fix hardcoded /tmp: use GeminiOauthConfig::default_credentials_path()
- Replace emoji output with plain text markers
- Propagate Client::builder() errors instead of silent fallback
- Use tokio::fs for all file I/O in CredentialManager (was std::fs)
- Use if let Some(ref pid) to avoid consuming credential.project_id
- Extract uses_cloud_code_api() helper; route by major version (gemini-2+)
- Concatenate multiple system messages into systemInstruction
- Include functionCall parts in assistant message conversion
- Add 401 retry loop with allow_retry flag for auth failures
- Remove biased from tokio::select! in OAuth callback handler
- Remove hardcoded context_length 1M; vary by model family
- Change GOOG_API_CLIENT from Node.js spoof to gl-rust/1.0.0
- Implement list_models() with static model list
- Move create_gemini_oauth_provider() before test module (clippy)
- Fix 9 additional clippy warnings (collapsible_if, map_or, needless_borrow)
- Run cargo fmt

* Add dedicated regression tests for Gemini OAuth fixes

* style: fix formatting in Gemini OAuth regression tests

* feat(gemini-oauth): implement code review v3 refinements

- Add force_refresh() for 401 retry (bypass timestamp check)
- Standardize Gemini model list across docs, wizard, and provider
- Restore gemini-3 check for thinkingConfig
- Redact sensitive tokens in GoogleTokenRefreshResponse Debug output
- Use dynamic version for GOOG_API_CLIENT
- Improve model_metadata() context length heuristics
- Use strip_prefix("data:") for safer SSE parsing
- Skip re-auth in wizard if keeping existing provider

* feat(gemini_oauth): full Cloud Code API integration with project discovery

- Register gemini_oauth as a dedicated backend in config/llm.rs (skip
  registry fallback, preserve backend name, suppress unknown-backend warning)
- Fix app.rs credential guard to exclude backends with dedicated configs
  (gemini_oauth, bedrock) from the provider.is_none() check
- Auto-discover Cloud Code project_id via loadCodeAssist when credentials
  lack it (e.g. created by the original Gemini CLI)
- Persist discovered project_id to credentials file for subsequent runs
- Add safety settings (BLOCK_NONE), gated behind GEMINI_SAFETY_BLOCK_NONE env
- Add thinkingConfig: budget-based for Gemini 2.5, level-based for Gemini 3.x
  (without includeThoughts to avoid empty responses from reasoning.rs stripping)
- Add thought signature injection for Gemini 3.x preview APIs
- Add history curation to filter invalid model outputs before re-sending
- Add extended generationConfig env vars (topP, topK, seed, penalties,
  responseMimeType, responseJsonSchema, cachedContent)
- Add custom headers support via GEMINI_CLI_CUSTOM_HEADERS
- Add API key auth mode (GEMINI_API_KEY + GEMINI_API_KEY_AUTH_MECHANISM)
- Add SSE metadata extraction (modelVersion, credits, promptFeedback,
  groundingMetadata, citationMetadata, cachedContentTokenCount)
- Add countTokens API support
- Add new models to wizard (gemini-3.1-pro-preview-customtools,
  gemini-3-pro-preview, gemini-3.1-flash-lite-preview)
- Update docs/LLM_PROVIDERS.md with new models and routing rules
- Rewrite regression tests with comprehensive coverage (23 unit tests pass)

* fix: CI violations — add safety comment on expect, fix fmt

- Add '// safety: hardcoded literal' to regex .expect() to satisfy
  the no-panic-in-prod CI check
- Fix cargo fmt whitespace in collapsible if-let chain

* fix: address PR review feedback from gemini-code-assist

- Fix parse_custom_headers to preserve commas in values by splitting
  only on commas followed by a header-name:colon pattern (manual scan
  instead of simple split(','))
- Use matches! macro for backend exclusion check in app.rs
- Merge SSE metadata extraction into single pass (was iterating twice)
- Replace fragile substring-based context_length with explicit match
  on known Gemini model IDs via gemini_context_length()
- Add missing models to regression test (8 models, not 5)

* fix: address Copilot PR review feedback

- Fix empty text part for assistant messages with tool calls
  (curate_contents could drop entire model turn)
- Propagate cache_read/creation_input_tokens in complete_with_tools
- Log warning on save_credential failure instead of silently ignoring
- Fix doc comment to mention underscore in header name pattern
- Handle gemini-oauth (hyphen variant) in setup wizard display
- Fix docs: thinkingConfig uses thinkingBudget/thinkingLevel, not
  includeThoughts

* fix: add missing allow_always field after staging merge

* fix(gemini_oauth): align header parser doc with implementation [skip-regression-check]

Update parse_custom_headers doc comments to include underscore in the
header-name character class, matching the actual implementation.
Also fix formatting from merge.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* fix(gemini_oauth): curate_contents per-part filtering and dead code removal

Fix curate_contents to filter invalid parts individually instead of
dropping entire model turn sequences. Previously a single empty text
part would discard all consecutive model turns including valid
functionCall parts, breaking the tool-call flow.

Also remove unused MID_STREAM_* constants.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* style(gemini_oauth): rustfmt formatting [skip-regression-check]

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* fix(llm): support smart routing cheap model for gemini_oauth backend

Add explicit gemini_oauth handling in create_cheap_provider_for_backend()
to create a GeminiOauthProvider with the cheap model swapped in. Without
this, setting LLM_CHEAP_MODEL with gemini_oauth backend would fail with
a confusing "no registry provider config available" error.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* docs: add Gemini OAuth env vars to .env.example [skip-regression-check]

Document GEMINI_MODEL, GEMINI_CREDENTIALS_PATH, GEMINI_API_KEY, and
all extended generation config env vars in the example config file.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

---------

Co-authored-by: [email protected] <[email protected]>
Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>
2026-03-21 22:41:44 -07:00

322 lines
9.9 KiB
Markdown
Raw Blame History

# LLM Provider Configuration
IronClaw defaults to NEAR AI for model access, but supports any OpenAI-compatible
endpoint as well as Anthropic, Ollama, and Google Gemini directly. This guide covers
the most common configurations.
## Provider Overview
| Provider | Backend value | Requires API key | Notes |
|---|---|---|---|
| NEAR AI | `nearai` | OAuth (browser) | Default; multi-model |
| Anthropic | `anthropic` | `ANTHROPIC_API_KEY` | Claude models |
| OpenAI | `openai` | `OPENAI_API_KEY` | GPT models |
| Google Gemini | `gemini_oauth` | OAuth (browser) | Gemini models; function calling |
| io.net | `ionet` | `IONET_API_KEY` | Intelligence API |
| Mistral | `mistral` | `MISTRAL_API_KEY` | Mistral models |
| Yandex AI Studio | `yandex` | `YANDEX_API_KEY` | YandexGPT models |
| MiniMax | `minimax` | `MINIMAX_API_KEY` | MiniMax-M2.7 models |
| Cloudflare Workers AI | `cloudflare` | `CLOUDFLARE_API_KEY` | Access to Workers AI |
| GitHub Copilot | `github_copilot` | `GITHUB_COPILOT_TOKEN` | Multi-models |
| Ollama | `ollama` | No | Local inference |
| AWS Bedrock | `bedrock` | AWS credentials | Native Converse API |
| OpenRouter | `openai_compatible` | `LLM_API_KEY` | 300+ models |
| Together AI | `openai_compatible` | `LLM_API_KEY` | Fast inference |
| Fireworks AI | `openai_compatible` | `LLM_API_KEY` | Fast inference |
| vLLM / LiteLLM | `openai_compatible` | Optional | Self-hosted |
| LM Studio | `openai_compatible` | No | Local GUI |
---
## NEAR AI (default)
No additional configuration required. On first run, `ironclaw onboard` opens a browser
for OAuth authentication. Credentials are saved to `~/.ironclaw/session.json`.
```env
NEARAI_MODEL=claude-3-5-sonnet-20241022
NEARAI_BASE_URL=https://private.near.ai
```
---
## Anthropic (Claude)
```env
LLM_BACKEND=anthropic
ANTHROPIC_API_KEY=sk-ant-...
```
Popular models: `claude-sonnet-4-20250514`, `claude-3-5-sonnet-20241022`, `claude-3-5-haiku-20241022`
---
## OpenAI (GPT)
```env
LLM_BACKEND=openai
OPENAI_API_KEY=sk-...
```
Popular models: `gpt-4o`, `gpt-4o-mini`, `o3-mini`
---
## Google Gemini (OAuth)
Uses Google OAuth with PKCE (S256) for authentication — no API key required.
On first run, a browser opens for Google account login. Credentials (including
refresh token) are saved to `~/.gemini/oauth_creds.json` with `0600` permissions.
```env
LLM_BACKEND=gemini_oauth
GEMINI_MODEL=gemini-2.5-flash
```
### Supported features
| Feature | Status | Notes |
|---|---|---|
| Function calling | ✅ | `functionDeclarations` / `functionCall` / `functionResponse` |
| `generationConfig` | ✅ | `temperature`, `maxOutputTokens` passed from request |
| `thinkingConfig` | ✅ | `thinkingBudget`/`thinkingLevel` for thinking-capable models (does NOT set `includeThoughts`) |
| `toolConfig` | ✅ | `functionCallingConfig.mode`: `AUTO`/`ANY`/`NONE` |
| SSE streaming | ✅ | Cloud Code API with `streamGenerateContent?alt=sse` |
| Token refresh | ✅ | Automatic via refresh token |
### Popular models
| Model | ID | Notes |
|---|---|---|
| Gemini 3.1 Pro | `gemini-3.1-pro-preview` | Latest, strongest reasoning |
| Gemini 3.1 Pro Custom Tools | `gemini-3.1-pro-preview-customtools` | Enhanced tool use |
| Gemini 3 Pro | `gemini-3-pro-preview` | Preview |
| Gemini 3 Flash | `gemini-3-flash-preview` | Fast preview with thinking |
| Gemini 3.1 Flash Lite | `gemini-3.1-flash-lite-preview` | Preview, lightweight |
| Gemini 2.5 Pro | `gemini-2.5-pro` | Stable, strong reasoning |
| Gemini 2.5 Flash | `gemini-2.5-flash` | Fast, good quality |
| Gemini 2.5 Flash Lite | `gemini-2.5-flash-lite` | Fastest, lightweight |
### Cloud Code API vs standard API
Models containing `-preview` (with hyphen) or `gemini-3` in the name, as well
as any `gemini-` model with major version >= 2, route through the Cloud Code
API (`cloudcode-pa.googleapis.com`) which supports SSE streaming
and project-scoped access. Other models use the standard Generative Language
API (`generativelanguage.googleapis.com`).
---
## GitHub Copilot
GitHub Copilot exposes chat endpoint at
`https://api.githubcopilot.com`. IronClaw uses that endpoint directly through the
built-in `github_copilot` provider.
```env
LLM_BACKEND=github_copilot
GITHUB_COPILOT_TOKEN=gho_...
GITHUB_COPILOT_MODEL=gpt-4o
# Optional advanced headers if your setup needs them:
# GITHUB_COPILOT_EXTRA_HEADERS=Copilot-Integration-Id:vscode-chat
```
`ironclaw onboard` can acquire this token for you using GitHub device login. If you
already signed into Copilot through VS Code or a JetBrains IDE, you can also reuse
the `oauth_token` stored in `~/.config/github-copilot/apps.json`. If you prefer,
`LLM_BACKEND=github-copilot` also works as an alias.
Popular models vary by subscription, but `gpt-4o` is a safe default. IronClaw keeps
model entry manual for this provider because GitHub Copilot model listing may require
extra integration headers on some clients. IronClaw automatically injects the standard
VS Code identity headers (`User-Agent`, `Editor-Version`, `Editor-Plugin-Version`,
`Copilot-Integration-Id`) and lets you override them with
`GITHUB_COPILOT_EXTRA_HEADERS`.
---
## Ollama (local)
Install Ollama from [ollama.com](https://ollama.com), pull a model, then:
```env
LLM_BACKEND=ollama
OLLAMA_MODEL=llama3.2
# OLLAMA_BASE_URL=http://localhost:11434 # default
```
Pull a model first: `ollama pull llama3.2`
---
## MiniMax
[MiniMax](https://platform.minimax.io) provides high-performance language models with 204,800 token context windows.
```env
LLM_BACKEND=minimax
MINIMAX_API_KEY=...
```
Available models: `MiniMax-M2.7` (default), `MiniMax-M2.7-highspeed`, `MiniMax-M2.5`, `MiniMax-M2.5-highspeed`
To use the China mainland endpoint, set:
```env
MINIMAX_BASE_URL=https://api.minimaxi.com/v1
```
---
## AWS Bedrock (requires `--features bedrock`)
Uses the native AWS Converse API via `aws-sdk-bedrockruntime`. Supports standard AWS
authentication methods: IAM credentials, SSO profiles, and instance roles.
> **Build prerequisite:** The `aws-lc-sys` crate (transitive dependency via AWS SDK)
> requires **CMake** to compile. Install it before building with `--features bedrock`:
> - macOS: `brew install cmake`
> - Ubuntu/Debian: `sudo apt install cmake`
> - Fedora: `sudo dnf install cmake`
### With AWS credentials (IAM, SSO, instance roles)
```env
LLM_BACKEND=bedrock
BEDROCK_MODEL=anthropic.claude-opus-4-6-v1
BEDROCK_REGION=us-east-1
BEDROCK_CROSS_REGION=us
# AWS_PROFILE=my-sso-profile # optional, for named profiles
```
The AWS SDK credential chain automatically resolves credentials from environment
variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`), shared credentials file
(`~/.aws/credentials`), SSO profiles, and EC2/ECS instance roles.
### Cross-region inference
Set `BEDROCK_CROSS_REGION` to route requests across AWS regions for capacity:
| Prefix | Routing |
|---|---|
| `us` | US regions (us-east-1, us-east-2, us-west-2) |
| `eu` | European regions |
| `apac` | Asia-Pacific regions |
| `global` | All commercial AWS regions |
| _(unset)_ | Single-region only |
### Popular Bedrock model IDs
| Model | ID |
|---|---|
| Claude Opus 4.6 | `anthropic.claude-opus-4-6-v1` |
| Claude Sonnet 4.5 | `anthropic.claude-sonnet-4-5-20250929-v1:0` |
| Claude Haiku 4.5 | `anthropic.claude-haiku-4-5-20251001-v1:0` |
| Amazon Nova Pro | `amazon.nova-pro-v1:0` |
| Llama 4 Maverick | `meta.llama4-maverick-17b-instruct-v1:0` |
---
## OpenAI-Compatible Endpoints
All providers below use `LLM_BACKEND=openai_compatible`. Set `LLM_BASE_URL` to the
provider's OpenAI-compatible endpoint and `LLM_API_KEY` to your API key.
### OpenRouter
[OpenRouter](https://openrouter.ai) routes to 300+ models from a single API key.
```env
LLM_BACKEND=openai_compatible
LLM_BASE_URL=https://openrouter.ai/api/v1
LLM_API_KEY=sk-or-...
LLM_MODEL=anthropic/claude-sonnet-4
```
Popular OpenRouter model IDs:
| Model | ID |
|---|---|
| Claude Sonnet 4 | `anthropic/claude-sonnet-4` |
| GPT-4o | `openai/gpt-4o` |
| Llama 4 Maverick | `meta-llama/llama-4-maverick` |
| Gemini 2.0 Flash | `google/gemini-2.0-flash-001` |
| Mistral Small | `mistralai/mistral-small-3.1-24b-instruct` |
Browse all models at [openrouter.ai/models](https://openrouter.ai/models).
### Together AI
[Together AI](https://www.together.ai) provides fast inference for open-source models.
```env
LLM_BACKEND=openai_compatible
LLM_BASE_URL=https://api.together.xyz/v1
LLM_API_KEY=...
LLM_MODEL=meta-llama/Llama-3.3-70B-Instruct-Turbo
```
Popular Together AI model IDs:
| Model | ID |
|---|---|
| Llama 3.3 70B | `meta-llama/Llama-3.3-70B-Instruct-Turbo` |
| DeepSeek R1 | `deepseek-ai/DeepSeek-R1` |
| Qwen 2.5 72B | `Qwen/Qwen2.5-72B-Instruct-Turbo` |
### Fireworks AI
[Fireworks AI](https://fireworks.ai) offers fast inference with compound AI system support.
```env
LLM_BACKEND=openai_compatible
LLM_BASE_URL=https://api.fireworks.ai/inference/v1
LLM_API_KEY=fw_...
LLM_MODEL=accounts/fireworks/models/llama4-maverick-instruct-basic
```
### vLLM / LiteLLM (self-hosted)
For self-hosted inference servers:
```env
LLM_BACKEND=openai_compatible
LLM_BASE_URL=http://localhost:8000/v1
LLM_API_KEY=token-abc123 # set to any string if auth is not configured
LLM_MODEL=meta-llama/Llama-3.1-8B-Instruct
```
LiteLLM proxy (forwards to any backend, including Bedrock, Vertex, Azure):
```env
LLM_BACKEND=openai_compatible
LLM_BASE_URL=http://localhost:4000/v1
LLM_API_KEY=sk-...
LLM_MODEL=gpt-4o # as configured in litellm config.yaml
```
### LM Studio (local GUI)
Start LM Studio's local server, then:
```env
LLM_BACKEND=openai_compatible
LLM_BASE_URL=http://localhost:1234/v1
LLM_MODEL=llama-3.2-3b-instruct-q4_K_M
# LLM_API_KEY is not required for LM Studio
```
---
## Using the Setup Wizard
Instead of editing `.env` manually, run the onboarding wizard:
```bash
ironclaw onboard
```
Select **"OpenAI-compatible"** for OpenRouter, Together AI, Fireworks, vLLM, LiteLLM,
or LM Studio. You will be prompted for the base URL and (optionally) an API key.
The model name is configured in the following step.
Reference in New Issue View Git Blame Copy Permalink