mirror of
https://github.com/outbackdingo/optimclaw.git
synced 2026-08-26 23:50:17 +00:00
docs: add comprehensive subdirectory CLAUDE.md files and update root (#589)
* docs: add comprehensive subdirectory CLAUDE.md files and update root The repo has grown significantly. This adds module-level CLAUDE.md files for the five most complex subsystems, and updates the root CLAUDE.md to reflect the actual current state of the codebase. New files: - src/agent/CLAUDE.md — full module map (19 files), session/thread/turn model, agentic loop flow, compaction strategies with correct thresholds, scheduler invariants, self-repair details, complete submission command reference table - src/channels/web/CLAUDE.md — complete API route table (50+ endpoints), SSE event type reference, auth/rate limiting gotchas, connection limits, CORS headers, step-by-step endpoint addition guide - src/db/CLAUDE.md — dual-backend build commands, sub-trait structure (7 sub-traits, ~67 methods), SQL dialect differences, boolean/timestamp gotchas, complete schema table, in-memory test helper, shared handle pattern - src/llm/CLAUDE.md — corrected LlmProvider trait signatures, provider chain decorator order, NEAR AI dual-auth and session renewal details, circuit breaker thresholds, previously undocumented smart_routing.rs and recording.rs - tests/e2e/CLAUDE.md — conftest fixtures and async scoping, environment injected into the binary, mock_llm canned responses, writing guide with correct asyncio usage, gotchas section Root CLAUDE.md updates: - Added E2E test setup and integration test commands - Documented ~15 undocumented modules: cli/, registry/, hooks/, tunnel/, observability/, webhook_server.rs, cost_guard.rs, job_monitor.rs, etc. - Corrected libSQL backend path (libsql/ directory, 8 sub-modules) - Updated Database trait method count (~67, split across 7 sub-traits) - Fixed stale references: config.rs → config/channels.rs, main.rs → app.rs - Added Hook, Observer, Tunnel traits to extensibility section - Added tunnel and observability env vars to Configuration section - Removed resolved TODO (webhook trigger is now shipped) - Added Module Specifications entries for all 5 new CLAUDE.md files Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * docs: address PR review comments and reduce CLAUDE.md size - Fix 7-sub-trait count (was 6) and ~78 async methods (was ~60/~67) in both CLAUDE.md and src/db/CLAUDE.md - Add missing types.rs to secrets/ file tree (CLAUDE.md) - Add missing tls.rs to src/db/CLAUDE.md Files table - Fix method counts: ConversationStore 12, JobStore 13, RoutineStore 15 - Add Windows venv activation note to E2E setup commands - Collapse agent/, web/, llm/, db/ file trees to one-liners (detail lives in their respective CLAUDE.md files) - Replace verbose Database and LLM Providers sections with summaries linking to src/db/CLAUDE.md and src/llm/CLAUDE.md - Root CLAUDE.md: 43,868 → 35,270 chars (fixes >40k perf warning) [skip-regression-check] Co-Authored-By: Claude Sonnet 4.6 <[email protected]> --------- Co-authored-by: Claude Sonnet 4.6 <[email protected]>
This commit is contained in:
Henry Park
co-authored by
Claude Sonnet 4.6
Claude Sonnet 4.6
parent
45ec691f4c
commit
633b234e44
@@ -0,0 +1,174 @@
|
||||
# IronClaw E2E Tests
|
||||
|
||||
Python/Playwright test suite that runs against a live ironclaw instance. Added in PR #553 ("Trajectory benchmarks and e2e trace test rig").
|
||||
|
||||
## Setup
|
||||
|
||||
```bash
|
||||
cd tests/e2e
|
||||
|
||||
# Create virtualenv (one-time)
|
||||
python -m venv .venv
|
||||
source .venv/bin/activate # or .venv\Scripts\activate on Windows
|
||||
|
||||
# Install dependencies
|
||||
pip install -e .
|
||||
|
||||
# Install browser binaries (one-time)
|
||||
playwright install chromium
|
||||
```
|
||||
|
||||
Dependencies: `pytest`, `pytest-asyncio`, `pytest-playwright`, `pytest-timeout`, `playwright`, `aiohttp`, `httpx`. Optional: `anthropic` (vision extras). Requires Python >= 3.11.
|
||||
|
||||
## Running Tests
|
||||
|
||||
```bash
|
||||
# Activate venv first
|
||||
source .venv/bin/activate
|
||||
|
||||
# Run all scenarios (conftest.py builds the binary and starts all servers automatically)
|
||||
pytest scenarios/
|
||||
|
||||
# Run a specific scenario
|
||||
pytest scenarios/test_chat.py
|
||||
pytest scenarios/test_sse_reconnect.py
|
||||
|
||||
# Run with verbose output
|
||||
pytest scenarios/ -v
|
||||
|
||||
# Run with a specific timeout (default is 120s per test, set in pyproject.toml)
|
||||
pytest scenarios/ --timeout=60
|
||||
|
||||
# Run with a headed browser (useful for debugging)
|
||||
HEADED=1 pytest scenarios/
|
||||
```
|
||||
|
||||
## Test Scenarios
|
||||
|
||||
| File | What it tests |
|
||||
|------|--------------|
|
||||
| `test_connection.py` | Gateway reachability, tab navigation, auth rejection (no token shows auth screen) |
|
||||
| `test_chat.py` | Send message via browser UI, verify streamed response from mock LLM; also tests empty-message suppression |
|
||||
| `test_html_injection.py` | XSS vectors injected directly via `page.evaluate("addMessage('assistant', ...)")` are sanitized by `renderMarkdown`; user messages are shown as escaped plain text |
|
||||
| `test_skills.py` | Skills tab UI visibility, ClawHub search (skipped if registry unreachable), install + remove lifecycle |
|
||||
| `test_sse_reconnect.py` | SSE reconnects after programmatic `eventSource.close()` + `connectSSE()`; history is reloaded after reconnect |
|
||||
| `test_tool_approval.py` | Approval card appears, buttons disable on approve/deny, parameters toggle; all triggered via `page.evaluate("showApproval(...)")` — no real tool call needed |
|
||||
|
||||
## `helpers.py`
|
||||
|
||||
Shared constants and utilities imported by every test file and `conftest.py`.
|
||||
|
||||
- **`SEL`** — dict of CSS/ID selectors for all DOM elements (chat input, message bubbles, approval card, tab buttons, skill search, etc.). Update this dict when frontend HTML changes; tests import selectors from here rather than hardcoding them.
|
||||
- **`TABS`** — ordered list of tab names: `["chat", "memory", "jobs", "routines", "extensions", "skills"]`.
|
||||
- **`AUTH_TOKEN`** — hardcoded to `"e2e-test-token"`. Used by `conftest.py` when starting the server (`GATEWAY_AUTH_TOKEN`) and by the `page` fixture when navigating (`/?token=e2e-test-token`).
|
||||
- **`wait_for_ready(url, timeout, interval)`** — polls a URL until HTTP 200 or timeout; used to wait for the gateway and mock LLM to become available.
|
||||
- **`wait_for_port_line(process, pattern, timeout)`** — reads a subprocess's stdout line-by-line until a regex match; used to extract the dynamically assigned mock LLM port from `MOCK_LLM_PORT=XXXX`.
|
||||
|
||||
## `conftest.py` and Fixtures
|
||||
|
||||
All fixtures are defined in `tests/e2e/conftest.py`. Running `pytest scenarios/` from the `tests/e2e/` directory picks up this conftest automatically (it is one level above `scenarios/`).
|
||||
|
||||
### Session-scoped fixtures (run once per `pytest` invocation)
|
||||
|
||||
| Fixture | What it does |
|
||||
|---------|-------------|
|
||||
| `ironclaw_binary` | Checks `target/debug/ironclaw`; if absent, runs `cargo build --no-default-features --features libsql` (timeout 600s). |
|
||||
| `mock_llm_server` | Starts `mock_llm.py --port 0`, reads the assigned port from stdout, waits for `/v1/models` to return 200. Yields the base URL. |
|
||||
| `ironclaw_server` | Starts the ironclaw binary with a minimal env (see below), waits for `/api/health` (timeout 60s). Yields the base URL. On teardown sends **SIGINT** (not SIGTERM) so the tokio ctrl_c handler triggers a graceful shutdown and LLVM coverage data is flushed. |
|
||||
| `browser` | Launches a single Chromium instance (headless by default; set `HEADED=1` for headed). Shared across all tests. |
|
||||
|
||||
### Function-scoped fixtures
|
||||
|
||||
| Fixture | What it does |
|
||||
|---------|-------------|
|
||||
| `page` | Creates a fresh browser **context** (viewport 1280×720) and **page** per test, navigates to `/?token=e2e-test-token`, and waits for `#auth-screen` to become hidden before yielding. Closes the context after each test. |
|
||||
|
||||
The function-scoped `page` fixture means **each test gets a clean browser context** (cookies, storage, etc.) but reuses the same ironclaw server and browser process. Tests that need the server URL directly (e.g., `test_auth_rejection`) accept `ironclaw_server` as an additional parameter.
|
||||
|
||||
### Environment passed to ironclaw in tests
|
||||
|
||||
The `ironclaw_server` fixture injects a minimal, deterministic environment:
|
||||
|
||||
```
|
||||
GATEWAY_ENABLED=true, GATEWAY_HOST=127.0.0.1, GATEWAY_PORT=<dynamic>
|
||||
GATEWAY_AUTH_TOKEN=e2e-test-token, GATEWAY_USER_ID=e2e-tester
|
||||
CLI_ENABLED=false
|
||||
LLM_BACKEND=openai_compatible, LLM_BASE_URL=<mock_llm_url>, LLM_MODEL=mock-model
|
||||
DATABASE_BACKEND=libsql, LIBSQL_PATH=<tmpdir>/e2e.db
|
||||
SANDBOX_ENABLED=false, ROUTINES_ENABLED=false, HEARTBEAT_ENABLED=false
|
||||
EMBEDDING_ENABLED=false, SKILLS_ENABLED=true
|
||||
ONBOARD_COMPLETED=true # prevents setup wizard
|
||||
```
|
||||
|
||||
The binary is also started with `--no-onboard`. Coverage env vars (`CARGO_LLVM_COV*`, `LLVM_*`, `CARGO_ENCODED_RUSTFLAGS`, `CARGO_INCREMENTAL`) are forwarded from the outer environment when present.
|
||||
|
||||
## Mock LLM (`mock_llm.py`)
|
||||
|
||||
An `aiohttp`-based OpenAI-compatible server used by tests that need deterministic LLM responses without hitting a real provider.
|
||||
|
||||
```bash
|
||||
# Start manually (port auto-selected, printed as MOCK_LLM_PORT=XXXX)
|
||||
python mock_llm.py --port 0
|
||||
```
|
||||
|
||||
It serves `POST /v1/chat/completions` (streaming + non-streaming) and `GET /v1/models`. Responses are pattern-matched from `CANNED_RESPONSES` against the last user message. Unmatched messages return `"I understand your request."`. The model name reported is always `"mock-model"`.
|
||||
|
||||
To add a new canned response:
|
||||
```python
|
||||
# In mock_llm.py
|
||||
CANNED_RESPONSES = [
|
||||
(re.compile(r"your pattern", re.IGNORECASE), "Your response"),
|
||||
...
|
||||
]
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
`conftest.py` handles all server startup automatically — you do not need to start ironclaw manually before running `pytest`. The conftest builds the binary (libsql feature), starts the mock LLM, and starts ironclaw with a fresh temp database on every `pytest` invocation.
|
||||
|
||||
If you need to test against a manually started ironclaw, you can skip conftest by running pytest with `--co` (collect-only) to understand what would run, or by calling the httpx/REST helpers directly without the `page` fixture.
|
||||
|
||||
## Writing New Scenarios
|
||||
|
||||
1. Create `scenarios/test_my_feature.py`.
|
||||
2. All async functions are automatically recognized as tests — `asyncio_mode = "auto"` is set globally in `pyproject.toml`. Do **not** add `@pytest.mark.asyncio`; it is redundant and raises a warning.
|
||||
3. Use the `page` fixture for browser tests (function-scoped, fresh context each test). Use `ironclaw_server` directly for pure HTTP tests.
|
||||
4. Import selectors from `helpers.SEL` and `helpers.AUTH_TOKEN` — do not hardcode selectors or tokens inline.
|
||||
5. Use `httpx.AsyncClient` for REST calls; `aiohttp` for SSE streaming.
|
||||
6. Keep new fixtures session-scoped where possible; server startup is expensive. Function-scoped fixtures (like `page`) are fine for browser state that must be clean per test.
|
||||
|
||||
```python
|
||||
import httpx
|
||||
from helpers import AUTH_TOKEN
|
||||
|
||||
async def test_my_endpoint(ironclaw_server):
|
||||
headers = {"Authorization": f"Bearer {AUTH_TOKEN}"}
|
||||
async with httpx.AsyncClient() as client:
|
||||
r = await client.get(f"{ironclaw_server}/api/health", headers=headers)
|
||||
assert r.status_code == 200
|
||||
```
|
||||
|
||||
For browser tests:
|
||||
```python
|
||||
from helpers import SEL
|
||||
|
||||
async def test_my_ui_feature(page):
|
||||
# page is already navigated and authenticated
|
||||
chat_input = page.locator(SEL["chat_input"])
|
||||
await chat_input.wait_for(state="visible", timeout=5000)
|
||||
# ... interact with the page ...
|
||||
```
|
||||
|
||||
### Gotchas
|
||||
|
||||
- **`asyncio_default_fixture_loop_scope = "session"`** — all async fixtures share one event loop. Do not use `asyncio.run()` inside fixtures; use `await` directly.
|
||||
- **The `page` fixture navigates with `/?token=e2e-test-token` and waits for `#auth-screen` to be hidden.** Tests receive a page that is already past the auth screen and has SSE connected.
|
||||
- **`test_skills.py` makes real network calls to ClawHub.** Tests skip (not fail) if the registry is unreachable via `pytest.skip()`.
|
||||
- **`test_html_injection.py` and `test_tool_approval.py` inject state via `page.evaluate(...)`.** They test the browser-side rendering pipeline and do not depend on the LLM or backend tool execution.
|
||||
- **Browser is Chromium only.** `conftest.py` uses `p.chromium.launch()`; there is no Firefox or WebKit variant.
|
||||
- **Default timeout is 120 seconds** (pyproject.toml). Individual `wait_for` calls inside tests use shorter timeouts (5–20s) for faster failure messages.
|
||||
- **The libsql database is a temp directory** created fresh per `pytest` invocation; tests do not share state across runs.
|
||||
|
||||
## CI Integration
|
||||
|
||||
E2E tests run in CI with `cargo-llvm-cov` for coverage collection. The CI workflow (`fix(ci): persist all cargo-llvm-cov env vars for E2E coverage` — PR #559) sets `LLVM_PROFILE_FILE` and related vars before spawning the ironclaw binary so coverage from E2E runs is captured.
|
||||
Reference in New Issue
Block a user