feat: chat onboarding and routine advisor (#927)

* feat: port NPA psychographic profiling system into IronClaw

Port the complete psychographic profiling system from NPA into IronClaw,
including enriched profile schema, conversational onboarding, profile
evolution, and three-tier prompt augmentation.

Personal onboarding moved from wizard Step 9 to first assistant
interaction per maintainer feedback — the First Contact system prompt
block now instructs the LLM to conduct a natural onboarding conversation
that builds the psychographic profile via memory_write.

Changes:
- Enrich profile.rs with 5 new structs, 9-dimension analysis framework,
  custom deserializers for backward compatibility, and rendering methods
- Add conversational onboarding engine with one-step-removed questioning
  technique, personality framework, and confidence-scored profile generation
- Add profile evolution with confidence gating, analysis metadata tracking,
  and weekly update routine
- Replace thin interaction style injection with three-tier system gated on
  confidence > 0.6 and profile recency
- Replace wizard Step 9 with First Contact system prompt block that drives
  conversational onboarding during the user's first interaction
- Add autonomy progression to SOUL.md seed and personality framework to
  AGENTS.md seed

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* feat: replace chat-based onboarding with bootstrap greeting and workspace seeds

Remove the interactive onboarding_chat.rs engine in favor of a simpler
bootstrap flow: fresh workspaces get a proactive LLM greeting that
naturally profiles the user. Identity files are now seeded from
src/workspace/seeds/ instead of being hardcoded. Also removes the
identity-file write protection (seeds are now managed), adds routine
advisor integration, and includes an e2e trace for bootstrap greeting.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* feat(safety): sanitize identity file writes via Sanitizer to prevent prompt injection

Identity files (SOUL.md, AGENTS.md, USER.md, IDENTITY.md) are injected into
every system prompt. Rather than hard-blocking writes (which broke onboarding),
scan content through the existing Sanitizer and reject writes with High/Critical
severity injection patterns. Medium/Low warnings are logged but allowed.

Also clarifies AGENTS.md identity file roles (USER.md = user info, IDENTITY.md =
agent identity) and adds IDENTITY.md setup as an explicit bootstrap step.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* docs: update profile_onboarding_completed comment to reflect current wiring

The field is now actively used by the agent loop to suppress BOOTSTRAP.md
injection — remove the stale "not yet wired" TODO.

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix(setup): use env_or_override for NEARAI_API_KEY in model fetch config

When the user authenticates via NEAR AI Cloud API key (option 4),
api_key_login() stores the key via set_runtime_env(). But
build_nearai_model_fetch_config() was using std::env::var() which
doesn't check the runtime overlay — so model listing fell back to
session-token auth and re-triggered the interactive NEAR AI
authentication menu.

Switch to env_or_override() which checks both real env vars and the
runtime overlay.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix(agent): correct channel/user_id in bootstrap greeting persist call

persist_assistant_response was called with channel="default",
user_id="system" but the assistant thread was created via
get_or_create_assistant_conversation("default", "gateway") which owns
the conversation as user_id="default", channel="gateway". The mismatch
caused ensure_writable_conversation to reject the write with:

  WARN Rejected write for unavailable thread id user=system channel=default

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix(web): remove all inline event handlers for CSP compliance

The Content-Security-Policy header (added in f48fe95) blocks inline JS
via script-src 'self'. All onclick/onchange attributes in index.html
are replaced with getElementById().addEventListener() calls. Dynamic
inline handlers in app.js (jobs, routines, memory breadcrumb, code
blocks, TEE report) are replaced with data-action attributes and a
single delegated click handler on document.

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix(agent): align bootstrap message user/channel and update fixture schema field

- Bootstrap IncomingMessage now uses ("default", "gateway") consistently
  with persist and session registration calls
- Update bootstrap_greeting.json fixture: schema_version → version to
  match current PROFILE_JSON_SCHEMA

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* style: cargo fmt

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix(safety): address PR review — expand injection scanning and harden profile sync

- BOOTSTRAP.md: fix target "profile" → "context/profile.json" so the
  write hits the correct path and triggers profile sync
- IDENTITY_FILES: add context/assistant-directives.md to the scanned
  set since it is also injected into the system prompt
- sync_profile_documents(): scan derived USER.md and assistant-directives
  content through Sanitizer before writing, rejecting High/Critical
  injection patterns
- profile_evolution_prompt(): wrap recent_messages_summary in <user_data>
  delimiters with untrusted-data instruction to mitigate indirect
  prompt injection
- routine-advisor skill: update cron examples from 6-field to standard
  5-field format for consistency with routine_create tool docs

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* style: cargo fmt

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* fix(setup): detect env-provided LLM keys during quick-mode onboarding

Quick-mode wizard now checks LLM_BACKEND, NEARAI_API_KEY,
ANTHROPIC_API_KEY, and OPENAI_API_KEY env vars to pre-populate
the provider setting, so users aren't re-prompted for credentials
they already supplied. Also teaches setup_nearai() to recognize
NEARAI_API_KEY from env (previously only checked session tokens).

Includes web UI cleanup (remove duplicate event listeners) and
e2e test response count adjustment.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* fix(test): update routine_create_list to expect 7-field normalized cron

The cron normalizer now always expands to 7-field format, so the
stored schedule is "0 0 9 * * * *" not "0 0 9 * * *".

[skip-regression-check]

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* feat(setup): skip LLM provider prompts when NEARAI_API_KEY is present

In quick mode, if NEARAI_API_KEY is set in the environment and the
backend was auto-detected as nearai, skip the interactive inference
provider and model selection steps. The API key is persisted to the
secrets store and a default model is set automatically.

Also simplify the static fallback model list for nearai to a single
default entry.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* fix: unify default model, static bootstrap greeting, and web UI cleanup

- Add DEFAULT_MODEL const and default_models() fallback list in
  llm/nearai_chat.rs; use from config, wizard, and .env.example so the
  default model is defined in one place
- Restore multi-model fallback list in setup wizard (was reduced to 1)
- Move BOOTSTRAP_GREETING to module-level const (out of run() body)
- Replace LLM-based bootstrap with static greeting (persist to DB before
  channels start, then broadcast — eliminates startup LLM call and race)
- Fix double env::var read for NEARAI_API_KEY in quick setup path
- Move thread sidebar buttons into threads-section-header (web UI)
- Remove orphaned .thread-sidebar-header CSS and fix double blank line
- Update bootstrap e2e test for static greeting (no LLM trace needed)

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* fix(safety): move prompt injection scanning into Workspace write/append

Addresses PR #927 review comments (#1, #3) — identity file write
protection and unsanitized profile fields in system prompt.

Instead of scanning at the tool layer (memory.rs) or the sync layer
(sync_profile_documents), injection scanning now lives in
Workspace::write() and Workspace::append() for all files that are
injected into the system prompt. This ensures every code path that
writes to these files is protected, including future ones.

- Add SYSTEM_PROMPT_FILES const and reject_if_injected() in workspace
- Add WorkspaceError::InjectionRejected variant
- Add map_write_err() in memory.rs to convert InjectionRejected to
  ToolError::NotAuthorized
- Remove redundant IDENTITY_FILES/Sanitizer from memory.rs
- Remove redundant sanitizer calls from sync_profile_documents()
- Move sanitization tests to workspace::tests
- Existing integration test (test_memory_write_rejects_injection)
  continues to pass through the new path

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* style: cargo fmt

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* fix: address Copilot review — merge marker order, orphan thread, stale fixture

- merge_profile_section: search for END marker after BEGIN position to
  avoid matching a stray END earlier in the file
- Bootstrap phase 2: use get_or_create_session + Thread::with_id instead
  of resolve_thread(None) to avoid creating an orphan thread
- setup_nearai: use env_or_override for NEARAI_API_KEY consistency with
  runtime overlay
- Delete orphaned bootstrap_greeting.json fixture (no test references it)
- Add test_merge_end_marker_must_follow_begin regression test

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* style: cargo fmt

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* style: fmt agent_loop.rs (CI stable rustfmt)

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* fix: lazy-init sanitizer, check profile non-empty before skipping bootstrap

Address Copilot review:
- Use LazyLock<Sanitizer> to avoid rebuilding Aho-Corasick + regexes
  on every workspace write
- has_profile check now requires non-empty content, not just file
  existence, to prevent empty profile.json from suppressing onboarding
- Add seed_tests integration tests (libsql-backed) verifying:
  - Empty profile.json does not suppress BOOTSTRAP.md seeding
  - Non-empty profile.json correctly suppresses bootstrap for upgrades

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* style: cargo fmt

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* fix: duplicate language handler, empty LLM_BACKEND, test_rig style

Address Copilot review on PR #927:
- Remove duplicate language-option click listeners (delegated
  data-action handler already covers them)
- Guard LLM_BACKEND env prefill against empty string to prevent
  suppressing API-key-based auto-detection
- Use destructured local `keep_bootstrap` instead of `self.keep_bootstrap`
  in test_rig for consistency after destructure

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* fix: update stale BOOTSTRAP.md write-protection comment [skip-regression-check]

BOOTSTRAP.md is now in SYSTEM_PROMPT_FILES and gets injection scanning
on write. The old comment incorrectly stated it was not write-protected.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* fix: replace debug_assert panics with graceful error returns [skip-regression-check]

debug_assert! in execute_tool_with_safety and JobContext::transition_to
panicked in test builds before the graceful error path could run.
Existing tests (test_cancel_job_completed, test_execute_empty_tool_name_returns_not_found)
already cover these paths — they were the ones failing.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* fix: address Copilot review — schema label, env var check, path normalization, profile validation

1. Label ANALYSIS_FRAMEWORK and PROFILE_JSON_SCHEMA sections separately
   in bootstrap prompt so the LLM knows which blob is the target structure.

2. Wizard quick-mode backend auto-detection now rejects empty env vars
   (std::env::var().is_ok_and(|v| !v.is_empty())) to avoid selecting the
   wrong backend when e.g. NEARAI_API_KEY="" is set.

3. Normalize the target path before comparing with paths::PROFILE in
   memory_write so non-canonical variants like "context//profile.json"
   still trigger profile sync.

4. seed_if_empty now requires valid JSON parse of context/profile.json
   before treating it as a populated profile. Corrupted content no longer
   permanently suppresses bootstrap seeding.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* style: cargo fmt

* fix: address Copilot review — append scan, profile validation, env_or_override

1. Workspace::append() now scans the combined content (existing + new)
   for prompt injection, not just the appended chunk. Prevents split-
   injection evasion across multiple appends.

2. seed_if_empty() now deserializes into PsychographicProfile instead of
   serde_json::Value for profile validation. Stray/legacy JSON that
   doesn't match the expected schema no longer suppresses bootstrap.

3. Wizard quick-mode backend auto-detection now uses env_or_override()
   to honor runtime overlays and injected secrets. LLM_BACKEND value
   is trimmed before storage.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* test: add bootstrap_onboarding_clears_bootstrap E2E trace test

Exercises the full onboarding flow end-to-end:
1. Bootstrap greeting fires automatically on fresh workspace
2. User converses for 3 turns (name, tools, work style)
3. Agent writes psychographic profile to context/profile.json
4. Profile sync generates USER.md and assistant-directives.md
5. Agent writes IDENTITY.md (chosen persona)
6. Agent clears BOOTSTRAP.md via memory_write(target: "bootstrap")

Verifies:
- BOOTSTRAP.md is non-empty before onboarding, empty after
- bootstrap_completed flag is set
- Profile contains expected user data (name, profession, interests)
- USER.md contains profile-derived content (name, tone, profession)
- Assistant-directives.md references user and communication style
- IDENTITY.md contains agent's chosen persona name
- All memory_write calls succeed

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* fix: address Copilot review — slash collapse, env_or_override, cron trim [skip-regression-check]

1. memory.rs path normalization now uses the same char-by-char loop as
   Workspace::normalize_path() to fully collapse consecutive slashes
   (e.g. "context///profile.json" → "context/profile.json").

2. Quick-mode NEARAI_API_KEY check (line 239) now uses env_or_override()
   consistently with the backend auto-detection block above it.

3. normalize_cron_expression() trims input before field counting so the
   passthrough branch (7+ fields) also strips whitespace.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

---------

Co-authored-by: Jay Zalowitz <[email protected]>
Co-authored-by: Claude Opus 4.6 <[email protected]>

src/agent/agent_loop.rs

@@ -31,6 +31,13 @@ use crate::skills::SkillRegistry;
 use crate::tools::ToolRegistry;
 use crate::workspace::Workspace;
 
+/// Static greeting persisted to DB and broadcast on first launch.
+///
+/// Sent before the LLM is involved so the user sees something immediately.
+/// The conversational onboarding (profile building, channel setup) happens
+/// organically in the subsequent turns driven by BOOTSTRAP.md.
+const BOOTSTRAP_GREETING: &str = include_str!("../workspace/seeds/GREETING.md");
+
 /// Collapse a tool output string into a single-line preview for display.
 pub(crate) fn truncate_for_preview(output: &str, max_chars: usize) -> String {
     let collapsed: String = output
@@ -340,6 +347,32 @@ impl Agent {
 
     /// Run the agent main loop.
     pub async fn run(self) -> Result<(), Error> {
+        // Proactive bootstrap: persist the static greeting to DB *before*
+        // starting channels so the first web client sees it via history.
+        let bootstrap_thread_id = if self
+            .workspace()
+            .is_some_and(|ws| ws.take_bootstrap_pending())
+        {
+            tracing::debug!(
+                "Fresh workspace detected — persisting static bootstrap greeting to DB"
+            );
+            if let Some(store) = self.store() {
+                let thread_id = store
+                    .get_or_create_assistant_conversation("default", "gateway")
+                    .await
+                    .ok();
+                if let Some(id) = thread_id {
+                    self.persist_assistant_response(id, "gateway", "default", BOOTSTRAP_GREETING)
+                        .await;
+                }
+                thread_id
+            } else {
+                None
+            }
+        } else {
+            None
+        };
+
         // Start channels
         let mut message_stream = self.channels.start_all().await?;
 
@@ -671,6 +704,30 @@ impl Agent {
             None
         };
 
+        // Bootstrap phase 2: register the thread in session manager and
+        // broadcast the greeting via SSE for any clients already connected.
+        // The greeting was already persisted to DB before start_all(), so
+        // clients that connect after this point will see it via history.
+        if let Some(id) = bootstrap_thread_id {
+            // Use get_or_create_session (not resolve_thread) to avoid creating
+            // an orphan thread. Then insert the DB-sourced thread directly.
+            let session = self.session_manager.get_or_create_session("default").await;
+            {
+                use crate::agent::session::Thread;
+                let mut sess = session.lock().await;
+                let thread = Thread::with_id(id, sess.id);
+                sess.active_thread = Some(id);
+                sess.threads.entry(id).or_insert(thread);
+            }
+            self.session_manager
+                .register_thread("default", "gateway", id, session)
+                .await;
+
+            let mut out = OutgoingResponse::text(BOOTSTRAP_GREETING.to_string());
+            out.thread_id = Some(id.to_string());
+            let _ = self.channels.broadcast("gateway", "default", out).await;
+        }
+
         // Main message loop
         tracing::debug!("Agent {} ready and listening", self.config.name);
 
@@ -864,9 +921,6 @@ impl Agent {
     }
 
     async fn handle_message(&self, message: &IncomingMessage) -> Result<Option<String>, Error> {
-        // Log at info level only for tracking without exposing PII (user_id can be a phone number)
-        tracing::info!(message_id = %message.id, "Processing message");
-
         // Log sensitive details at debug level for troubleshooting
         tracing::debug!(
             message_id = %message.id,
@@ -946,10 +1000,6 @@ impl Agent {
         }
 
         // Resolve session and thread
-        tracing::debug!(
-            message_id = %message.id,
-            "Resolving session and thread"
-        );
         let (session, thread_id) = self
             .session_manager
             .resolve_thread(

src/agent/routine.rs

@@ -688,16 +688,36 @@ pub fn content_hash(content: &str) -> u64 {
     hasher.finish()
 }
 
+/// Normalize a cron expression to the 7-field format expected by the `cron` crate.
+///
+/// The `cron` crate requires: `sec min hour day-of-month month day-of-week year`.
+/// Standard cron uses 5 fields: `min hour day-of-month month day-of-week`.
+/// This function auto-expands:
+/// - 5-field → prepend `0` (seconds) and append `*` (year)
+/// - 6-field → append `*` (year)
+/// - 7-field → pass through unchanged
+pub fn normalize_cron_expression(schedule: &str) -> String {
+    let trimmed = schedule.trim();
+    let fields: Vec<&str> = trimmed.split_whitespace().collect();
+    match fields.len() {
+        5 => format!("0 {} *", trimmed),
+        6 => format!("{} *", trimmed),
+        _ => trimmed.to_string(),
+    }
+}
+
 /// Parse a cron expression and compute the next fire time from now.
 ///
+/// Accepts standard 5-field, 6-field, or 7-field cron expressions (auto-normalized).
 /// When `timezone` is provided and valid, the schedule is evaluated in that
 /// timezone and the result is converted back to UTC. Otherwise UTC is used.
 pub fn next_cron_fire(
     schedule: &str,
     timezone: Option<&str>,
 ) -> Result<Option<DateTime<Utc>>, RoutineError> {
+    let normalized = normalize_cron_expression(schedule);
     let cron_schedule =
-        cron::Schedule::from_str(schedule).map_err(|e| RoutineError::InvalidCron {
+        cron::Schedule::from_str(&normalized).map_err(|e| RoutineError::InvalidCron {
             reason: e.to_string(),
         })?;
     if let Some(tz) = timezone.and_then(crate::timezone::parse_timezone) {
@@ -878,6 +898,7 @@ mod tests {
     use crate::agent::routine::{
         FullJobPermissionMode, MAX_TOOL_ROUNDS_LIMIT, RoutineAction, RoutineGuardrails, RunStatus,
         Trigger, content_hash, describe_cron, effective_full_job_tool_permissions, next_cron_fire,
+        normalize_cron_expression,
     };
 
     #[test]
@@ -1157,6 +1178,55 @@ mod tests {
         assert_eq!(Trigger::Manual.type_tag(), "manual");
     }
 
+    #[test]
+    fn test_normalize_cron_5_field() {
+        // Standard cron: min hour dom month dow
+        assert_eq!(normalize_cron_expression("0 9 * * 1"), "0 0 9 * * 1 *");
+        assert_eq!(
+            normalize_cron_expression("0 9 * * MON-FRI"),
+            "0 0 9 * * MON-FRI *"
+        );
+    }
+
+    #[test]
+    fn test_normalize_cron_6_field() {
+        // 6-field: sec min hour dom month dow
+        assert_eq!(
+            normalize_cron_expression("0 0 9 * * MON-FRI"),
+            "0 0 9 * * MON-FRI *"
+        );
+    }
+
+    #[test]
+    fn test_normalize_cron_7_field_passthrough() {
+        // Already 7-field: no change
+        assert_eq!(
+            normalize_cron_expression("0 0 9 * * MON-FRI *"),
+            "0 0 9 * * MON-FRI *"
+        );
+    }
+
+    #[test]
+    fn test_next_cron_fire_5_field_accepted() {
+        // Standard 5-field cron should now work through normalization
+        let result = next_cron_fire("0 9 * * 1", None);
+        assert!(
+            result.is_ok(),
+            "5-field cron should be accepted: {result:?}"
+        );
+        assert!(result.unwrap().is_some());
+    }
+
+    #[test]
+    fn test_next_cron_fire_5_field_with_timezone() {
+        let result = next_cron_fire("0 9 * * MON-FRI", Some("America/New_York"));
+        assert!(
+            result.is_ok(),
+            "5-field cron with timezone should be accepted: {result:?}"
+        );
+        assert!(result.unwrap().is_some());
+    }
+
     #[test]
     fn test_action_lightweight_backward_compat_no_use_tools() {
         // Simulate old DB record without use_tools field