Files
optimclaw/src/llm/codex_auth.rs
T
ZeroTrustandGitHub 1b59eb6b39 feat: Reuse Codex CLI OAuth tokens for ChatGPT backend LLM calls (#693)
* feat: add Codex auth.json token reuse for LLM authentication

When LLM_USE_CODEX_AUTH=true, IronClaw reads the Codex CLI's auth.json
(default ~/.codex/auth.json) and extracts the API key or OAuth access
token. This lets IronClaw piggyback on a Codex login without
implementing its own OAuth flow.

New env vars:
  - LLM_USE_CODEX_AUTH: enable Codex auth fallback (default: false)
  - CODEX_AUTH_PATH: override path to auth.json

* fix: handle ChatGPT auth mode correctly

Switch base_url to chatgpt.com/backend-api/codex when auth.json
contains ChatGPT OAuth tokens. The access_token is a JWT that only
works against the private ChatGPT backend, not the public OpenAI API.

Refactored codex_auth.rs to return CodexCredentials (token +
is_chatgpt_mode) instead of just a string key.

* fix: Codex auth takes highest priority over secrets store

When LLM_USE_CODEX_AUTH=true, Codex credentials are now loaded before
checking env vars or the secrets store overlay. Previously the secrets
store key (injected during onboarding) would shadow the Codex token.

* feat: Responses API provider for ChatGPT backend

- New CodexChatGptProvider speaks the Responses API protocol
- Auto-detects model from /models endpoint (gpt-4o -> gpt-5.2-codex)
- Adds store=false (required by ChatGPT backend)
- Error handling with timeout for HTTP 400 responses
- Message format translation: Chat Completions -> Responses API
- SSE response parsing for text, tool calls, and usage stats
- 7 unit tests for message conversion and SSE parsing

* fix: SSE parser uses item_id instead of call_id for tool call deltas

The Responses API sends function_call_arguments.delta events with
item_id (e.g. fc_...) not call_id (e.g. call_...). The parser now
keys pending tool calls by item_id from output_item.added and
tracks call_id separately for result matching.

* fix: strip empty string values from tool call arguments

gpt-5.2-codex fills optional tool parameters with empty strings
(e.g. timestamp: ""), which IronClaw's tool validation rejects.
Strip them before passing to tool execution.

* fix: prevent apiKey mode fallback to ChatGPT token

When auth_mode is explicitly 'apiKey' but the key is missing/empty,
do not fall through to check for a ChatGPT access_token. This prevents
returning credentials with is_chatgpt_mode: true and routing to the
wrong LLM provider.

* refactor: reuse single reqwest::Client across model discovery and LLM calls

Create Client once in with_auto_model, pass &Client to
fetch_default_model, and move it into the provider struct.
Eliminates the redundant Client::new() that wasted a connection pool.

* fix: bump client_version to 1.0.0 to unlock gpt-5.3-codex and gpt-5.4

The /models endpoint gates newer models behind client_version.
Version 0.1.0 only returns up to gpt-5.2-codex, while 1.0.0+
also returns gpt-5.3-codex and gpt-5.4.

* feat: user-configured LLM_MODEL takes priority over auto-detection

Fetch the full model list from /models endpoint. If LLM_MODEL is set,
validate it against the supported list and warn with available models
if not found. If LLM_MODEL is not set, auto-detect the highest-priority
model. Also bumps client_version to 1.0.0 to unlock gpt-5.3/5.4.

* fix: add 10s timeout to model discovery HTTP request

Prevents startup from blocking indefinitely if chatgpt.com
is slow or unreachable. Uses reqwest per-request timeout.

* docs: add private API warning for ChatGPT backend endpoint

The chatgpt.com/backend-api/codex endpoint is private and
undocumented. Add warning in module docs and a runtime log
on first use to inform users of potential ToS implications.

* feat: implement OAuth 401 token refresh for Codex ChatGPT provider

On HTTP 401, if a refresh_token is available, the provider now
automatically refreshes the access token via auth.openai.com/oauth/token
(same protocol as Codex CLI) and retries the request once. Refreshed
tokens are persisted back to auth.json.

Changes:
- codex_auth: read refresh_token, add refresh_access_token() and
  persist_refreshed_tokens()
- codex_chatgpt: RwLock for api_key, 401 detection + retry in
  send_request, send_http_request helper
- config/llm: thread refresh_token/auth_path through RegistryProviderConfig
- llm/mod: pass refresh params to with_auto_model

* refactor: lazy model detection via OnceCell, remove block_in_place

Model is no longer resolved during provider construction. Instead,
resolve_model() uses tokio::sync::OnceCell to lazily fetch from
/models on the first LLM call. This eliminates the block_in_place
+ block_on workaround in create_codex_chatgpt_from_registry.

- with_auto_model (async) -> with_lazy_model (sync constructor)
- resolve_model() added with OnceCell-based lazy init
- build_request_body takes model as parameter
- model_name() returns resolved or configured_model as fallback

* feat: support multimodal content (images) in Codex ChatGPT provider

message_to_input_items now checks content_parts for user messages.
ContentPart::Text maps to input_text and ContentPart::ImageUrl maps
to input_image, matching the Responses API format used by Codex CLI.
Falls back to plain text when content_parts is empty.

Also updates client_version to 0.111.0 for /models endpoint.

Adds test: test_message_conversion_user_with_image

* refactor: move codex_auth module from src/ to src/llm/

codex_auth is only used by the LLM layer (codex_chatgpt provider
and config/llm). Moving it under src/llm/ reflects its actual scope.

- Remove pub mod codex_auth from lib.rs
- Add pub mod codex_auth to llm/mod.rs
- Update imports: super::codex_auth, crate::llm::codex_auth

* Fix codex provider style issues

* Use SecretString throughout codex auth refresh flow

* Use SecretString for codex access tokens

* Reuse provider client for codex token refresh

* Stream Codex SSE responses incrementally

* Fix Windows clippy and SQLite test linkage

* Trigger checks after regression skip label

* Tighten codex auth module handling
2026-03-16 07:43:45 +00:00

378 lines
12 KiB
Rust

//! Read Codex CLI credentials for LLM authentication.
//!
//! When `LLM_USE_CODEX_AUTH=true`, IronClaw reads the Codex CLI's
//! `auth.json` file (default: `~/.codex/auth.json`) and extracts
//! credentials. This lets IronClaw piggyback on a Codex login without
//! implementing its own OAuth flow.
//!
//! Codex supports two auth modes:
//! - **API key** (`auth_mode: "apiKey"`) → uses `OPENAI_API_KEY` field
//! against `api.openai.com/v1`.
//! - **ChatGPT** (`auth_mode: "chatgpt"`) → uses `tokens.access_token`
//! (OAuth JWT) against `chatgpt.com/backend-api/codex`.
//!
//! When in ChatGPT mode, the provider supports automatic token refresh
//! on 401 responses using the `refresh_token` from `auth.json`.
use std::path::{Path, PathBuf};
use secrecy::{ExposeSecret, SecretString};
use serde::{Deserialize, Serialize};
/// ChatGPT backend API endpoint used by Codex in ChatGPT auth mode.
const CHATGPT_BACKEND_URL: &str = "https://chatgpt.com/backend-api/codex";
/// Standard OpenAI API endpoint used by Codex in API key mode.
const OPENAI_API_URL: &str = "https://api.openai.com/v1";
/// OAuth token refresh endpoint (same as Codex CLI).
const REFRESH_TOKEN_URL: &str = "https://auth.openai.com/oauth/token";
/// OAuth client ID used for token refresh (same as Codex CLI).
const CLIENT_ID: &str = "app_EMoamEEZ73f0CkXaXp7hrann";
/// Credentials extracted from Codex's `auth.json`.
#[derive(Debug, Clone)]
pub struct CodexCredentials {
/// The bearer token (API key or ChatGPT access_token).
pub token: SecretString,
/// Whether this is a ChatGPT OAuth token (vs. an OpenAI API key).
pub is_chatgpt_mode: bool,
/// OAuth refresh token (only present in ChatGPT mode).
pub refresh_token: Option<SecretString>,
/// Path to the auth.json file (for persisting refreshed tokens).
pub auth_path: Option<PathBuf>,
}
impl CodexCredentials {
/// Returns the correct base URL for the auth mode.
///
/// - ChatGPT mode → `https://chatgpt.com/backend-api/codex`
/// - API key mode → `https://api.openai.com/v1`
pub fn base_url(&self) -> &'static str {
if self.is_chatgpt_mode {
CHATGPT_BACKEND_URL
} else {
OPENAI_API_URL
}
}
}
/// Partial representation of Codex's `$CODEX_HOME/auth.json`.
#[derive(Debug, Deserialize)]
struct CodexAuthJson {
auth_mode: Option<String>,
#[serde(rename = "OPENAI_API_KEY")]
openai_api_key: Option<String>,
tokens: Option<CodexTokens>,
}
#[derive(Debug, Deserialize)]
struct CodexTokens {
access_token: SecretString,
refresh_token: Option<SecretString>,
}
/// Request body for OAuth token refresh.
#[derive(Serialize)]
struct RefreshRequest<'a> {
client_id: &'a str,
grant_type: &'a str,
refresh_token: &'a str,
}
/// Response from the OAuth token refresh endpoint.
#[derive(Debug, Deserialize)]
struct RefreshResponse {
access_token: SecretString,
refresh_token: Option<SecretString>,
}
/// Default path used by Codex CLI: `~/.codex/auth.json`.
pub fn default_codex_auth_path() -> PathBuf {
let home_dir = dirs::home_dir().unwrap_or_else(|| {
tracing::warn!(
"Could not determine home directory; falling back to current working directory for Codex auth.json path"
);
PathBuf::from(".")
});
home_dir.join(".codex").join("auth.json")
}
/// Load credentials from a Codex `auth.json` file.
///
/// Returns `None` if the file is missing, unreadable, or contains
/// no usable credentials.
pub fn load_codex_credentials(path: &Path) -> Option<CodexCredentials> {
let content = match std::fs::read_to_string(path) {
Ok(c) => c,
Err(e) => {
tracing::debug!("Could not read Codex auth file {}: {}", path.display(), e);
return None;
}
};
let auth: CodexAuthJson = match serde_json::from_str(&content) {
Ok(a) => a,
Err(e) => {
tracing::warn!("Failed to parse Codex auth file {}: {}", path.display(), e);
return None;
}
};
let is_chatgpt = auth
.auth_mode
.as_deref()
.map(|m| m == "chatgpt" || m == "chatgptAuthTokens")
.unwrap_or(false);
// API key mode: use OPENAI_API_KEY field.
if !is_chatgpt {
if let Some(key) = auth.openai_api_key.filter(|k| !k.is_empty()) {
tracing::info!("Loaded API key from Codex auth.json (API key mode)");
return Some(CodexCredentials {
token: SecretString::from(key),
is_chatgpt_mode: false,
refresh_token: None,
auth_path: None,
});
}
// If auth_mode was explicitly `apiKey`, do not fall back to checking for a token.
if auth.auth_mode.is_some() {
return None;
}
}
// ChatGPT mode: use access_token as bearer token.
if let Some(tokens) = auth.tokens
&& !tokens.access_token.expose_secret().is_empty()
{
tracing::info!(
"Loaded access token from Codex auth.json (ChatGPT mode, base_url={})",
CHATGPT_BACKEND_URL
);
return Some(CodexCredentials {
token: tokens.access_token,
is_chatgpt_mode: true,
refresh_token: tokens.refresh_token,
auth_path: Some(path.to_path_buf()),
});
}
tracing::debug!(
"Codex auth.json at {} contains no usable credentials",
path.display()
);
None
}
/// Attempt to refresh an expired access token using the refresh token.
///
/// On success, returns the new `access_token` and persists the refreshed
/// tokens back to `auth.json`. This follows the same OAuth protocol as
/// Codex CLI (`POST https://auth.openai.com/oauth/token`).
///
/// Returns `None` if the refresh token is missing, the request fails,
/// or the response is malformed.
pub async fn refresh_access_token(
client: &reqwest::Client,
refresh_token: &SecretString,
auth_path: Option<&Path>,
) -> Option<SecretString> {
let req = RefreshRequest {
client_id: CLIENT_ID,
grant_type: "refresh_token",
refresh_token: refresh_token.expose_secret(),
};
tracing::info!("Attempting to refresh Codex OAuth access token");
let resp = match client
.post(REFRESH_TOKEN_URL)
.header("Content-Type", "application/json")
.json(&req)
.timeout(std::time::Duration::from_secs(10))
.send()
.await
{
Ok(r) => r,
Err(e) => {
tracing::warn!("Token refresh request failed: {e}");
return None;
}
};
if !resp.status().is_success() {
let status = resp.status();
let body = resp.text().await.unwrap_or_default();
tracing::warn!("Token refresh failed: HTTP {status}: {body}");
if status.as_u16() == 401 {
tracing::warn!(
"Refresh token may be expired or revoked. \
Please re-authenticate with: codex --login"
);
}
return None;
}
let refresh_resp: RefreshResponse = match resp.json().await {
Ok(r) => r,
Err(e) => {
tracing::warn!("Failed to parse token refresh response: {e}");
return None;
}
};
let new_access_token = refresh_resp.access_token.clone();
// Persist refreshed tokens back to auth.json
if let Some(path) = auth_path {
if let Err(e) = persist_refreshed_tokens(
path,
refresh_resp.access_token.expose_secret(),
refresh_resp
.refresh_token
.as_ref()
.map(ExposeSecret::expose_secret),
) {
tracing::warn!(
"Failed to persist refreshed tokens to {}: {e}",
path.display()
);
} else {
tracing::info!("Refreshed tokens persisted to {}", path.display());
}
}
Some(new_access_token)
}
/// Update `auth.json` with refreshed tokens, preserving other fields.
fn persist_refreshed_tokens(
path: &Path,
new_access_token: &str,
new_refresh_token: Option<&str>,
) -> Result<(), Box<dyn std::error::Error>> {
let content = std::fs::read_to_string(path)?;
let mut json: serde_json::Value = serde_json::from_str(&content)?;
if let Some(tokens) = json.get_mut("tokens") {
tokens["access_token"] = serde_json::Value::String(new_access_token.to_string());
if let Some(rt) = new_refresh_token {
tokens["refresh_token"] = serde_json::Value::String(rt.to_string());
}
}
let updated = serde_json::to_string_pretty(&json)?;
let tmp_path = path.with_extension("json.tmp");
std::fs::write(&tmp_path, updated)?;
if let Err(e) = std::fs::rename(&tmp_path, path) {
let _ = std::fs::remove_file(&tmp_path);
return Err(Box::new(e));
}
set_auth_file_permissions(path)?;
Ok(())
}
#[cfg(unix)]
fn set_auth_file_permissions(path: &Path) -> Result<(), Box<dyn std::error::Error>> {
use std::os::unix::fs::PermissionsExt;
std::fs::set_permissions(path, std::fs::Permissions::from_mode(0o600))?;
Ok(())
}
#[cfg(not(unix))]
fn set_auth_file_permissions(_path: &Path) -> Result<(), Box<dyn std::error::Error>> {
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
use std::io::Write;
use tempfile::NamedTempFile;
#[test]
fn loads_api_key_mode() {
let mut f = NamedTempFile::new().unwrap();
writeln!(
f,
r#"{{"auth_mode":"apiKey","OPENAI_API_KEY":"sk-test-123"}}"#
)
.unwrap();
let creds = load_codex_credentials(f.path()).expect("should load");
assert_eq!(creds.token.expose_secret(), "sk-test-123");
assert!(!creds.is_chatgpt_mode);
assert_eq!(creds.base_url(), OPENAI_API_URL);
}
#[test]
fn loads_chatgpt_mode() {
let mut f = NamedTempFile::new().unwrap();
writeln!(
f,
r#"{{"auth_mode":"chatgpt","tokens":{{"id_token":{{}},"access_token":"eyJ-test","refresh_token":"rt-x"}}}}"#
)
.unwrap();
let creds = load_codex_credentials(f.path()).expect("should load");
assert_eq!(creds.token.expose_secret(), "eyJ-test");
assert!(creds.is_chatgpt_mode);
assert_eq!(
creds
.refresh_token
.as_ref()
.expect("refresh token should be present")
.expose_secret(),
"rt-x"
);
assert_eq!(creds.base_url(), CHATGPT_BACKEND_URL);
}
#[test]
fn api_key_mode_ignores_tokens() {
let mut f = NamedTempFile::new().unwrap();
writeln!(
f,
r#"{{"auth_mode":"apiKey","OPENAI_API_KEY":"sk-priority","tokens":{{"id_token":{{}},"access_token":"eyJ-fallback","refresh_token":"rt-x"}}}}"#
)
.unwrap();
let creds = load_codex_credentials(f.path()).expect("should load");
assert_eq!(creds.token.expose_secret(), "sk-priority");
assert!(!creds.is_chatgpt_mode);
}
#[test]
fn returns_none_for_missing_file() {
assert!(load_codex_credentials(Path::new("/tmp/nonexistent_codex_auth.json")).is_none());
}
#[test]
fn returns_none_for_empty_json() {
let mut f = NamedTempFile::new().unwrap();
writeln!(f, "{{}}").unwrap();
assert!(load_codex_credentials(f.path()).is_none());
}
#[test]
fn returns_none_for_empty_key() {
let mut f = NamedTempFile::new().unwrap();
writeln!(f, r#"{{"auth_mode":"apiKey","OPENAI_API_KEY":""}}"#).unwrap();
assert!(load_codex_credentials(f.path()).is_none());
}
#[test]
fn api_key_mode_missing_key_does_not_fallback_to_chatgpt() {
// Bug: if auth_mode is "apiKey" but key is missing, the old code would
// fall through to check for a ChatGPT token, returning is_chatgpt_mode: true.
let mut f = NamedTempFile::new().unwrap();
writeln!(
f,
r#"{{"auth_mode":"apiKey","OPENAI_API_KEY":"","tokens":{{"id_token":{{}},"access_token":"eyJ-bad","refresh_token":"rt-x"}}}}"#
)
.unwrap();
assert!(load_codex_credentials(f.path()).is_none());
}
}