mirror of
https://github.com/outbackdingo/optimclaw.git
synced 2026-08-29 08:59:31 +00:00
* feat(workspace): layered memory with sensitivity-based privacy redirect Introduce MemoryLayer type for named memory layers with sensitivity levels and write permissions. Layers map to synthetic user_id values in workspace tables, enabling shared/private memory isolation. - Add MemoryLayer, LayerSensitivity types with default_for_user() - Add layer-aware write methods (write_to_layer, append_to_layer) - Add PatternPrivacyClassifier to guard shared layer writes - Add optional 'layer' parameter to memory_write tool and HTTP API - Add 'redirected' and 'actual_layer' fields to write response - Add MEMORY_LAYERS env var (JSON) for layer configuration - Workspace user_id now derived from GATEWAY_USER_ID (was hardcoded "default") - 10 integration tests for layered memory operations Addresses prerequisite for Issue #59 (multi-tenancy). Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: add explicit default to memory_write layer schema Add "default": "private" to the layer parameter's JSON schema so LLM tool consumers can see the default without reading code. Co-Authored-By: Claude Opus 4.6 <[email protected]> * refactor: extract resolve_layer_target to deduplicate layer writes Consolidate shared layer-lookup, writable check, and privacy classification logic from write_to_layer and append_to_layer into a single resolve_layer_target helper. Flagged on #349 review — the duplication originates in this PR. Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: address review feedback on layered memory PR - Fix email regex pipe bug in TLD character class (privacy.rs) - Add append support to web memory_write handler via `append` field - Validate MemoryLayer name/scope: reject empty, check duplicates - Remove hardcoded 'private' default from tool schema; omit layer fields from output when no layer specified - Document scope isolation risk for multi-tenant (Issue #59) Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: address adversarial review findings - CRITICAL: fix identity file protection bypass via trailing slash (normalize target path before protection checks) - HIGH: check private layer is writable before privacy redirect - HIGH: map LayerNotFound/ReadOnly to proper 4xx HTTP status codes - HIGH: honor `append` field in non-layer HTTP write path - MEDIUM: remove redundant DB fetch in append_to_layer (narrower TOCTOU window) - MEDIUM: remove dead memory_write_handler from handlers/memory.rs Co-Authored-By: Claude Opus 4.6 <[email protected]> * feat: opt-in privacy classifier, force override, confidence scoring Address review feedback from @zmanian: - Privacy classifier is now opt-in via with_privacy_classifier() instead of always-on. Default hardcoded patterns (doctor, therapy, email, phone) had unacceptable false positive rates in household contexts. LLM chooses the correct layer via system prompt; regex can't improve on that. - Add ConfigurablePrivacyClassifier for operator-supplied patterns. - PatternPrivacyClassifier defaults narrowed to hard PII only (SSN, credit card, credentials). - Add force param to write_to_layer/append_to_layer to skip classifier. - PrivacyClassifier trait returns SensitivityResult { is_sensitive, confidence } instead of bool, ready for probabilistic classifiers. Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: remove redundant heartbeat match arm in memory_write The heartbeat arm was identical to the catch-all — resolved_path already points to paths::HEARTBEAT when target is "heartbeat". Addresses review feedback from gemini-code-assist on #1112. Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: return Result from PatternPrivacyClassifier::new() Replace .expect() with proper error propagation per project no-panics policy. Remove Default impl (unused in production). Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> * refactor: move memory_layers from GatewayConfig to WorkspaceConfig Resolve merge conflicts between HEAD (transcription, search, env helpers) and the workspace config branch. GatewayConfig no longer owns memory_layers; WorkspaceConfig::resolve() handles parsing, validation (name length >64, character set, empty scope, duplicates), and fallback defaults. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> * test: strengthen privacy classifier and layer isolation coverage Add 8 privacy classifier edge case tests (format variants, keywords, longer documents, empty/partial inputs) and 5 layer write isolation integration tests (cross-scope invisibility, overwrite, empty path, sensitive-to-private no-redirect). Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> * fix: tautological test assertion and add WorkspaceConfig validation tests Replace always-true `is_ok() || is_err()` in write_empty_path_to_layer with actual behavior assertion (write succeeds with normalized empty path). Add 8 unit tests for WorkspaceConfig::resolve() covering valid JSON parsing, invalid JSON, empty/long/invalid-char layer names, empty scopes, duplicates, and default fallback behavior. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> * style: cargo fmt after staging merge Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> --------- Co-authored-by: Claude Opus 4.6 <[email protected]> Co-authored-by: [email protected] <[email protected]>
277 lines
8.1 KiB
Rust
277 lines
8.1 KiB
Rust
use regex::Regex;
|
|
|
|
/// Result of privacy classification, including confidence level.
|
|
///
|
|
/// Confidence enables downstream callers to apply thresholds (e.g., only
|
|
/// redirect above 0.8) and supports future upgrade to LLM-based classifiers
|
|
/// that produce probabilistic scores.
|
|
#[derive(Debug, Clone)]
|
|
pub struct SensitivityResult {
|
|
pub is_sensitive: bool,
|
|
pub confidence: f32,
|
|
}
|
|
|
|
/// Classifies content as potentially sensitive for privacy purposes.
|
|
///
|
|
/// Used to guard writes to shared memory layers -- if content is flagged
|
|
/// as sensitive, it can be redirected to the private layer instead.
|
|
pub trait PrivacyClassifier: Send + Sync {
|
|
/// Classify content and return sensitivity with confidence score.
|
|
fn classify(&self, content: &str) -> SensitivityResult;
|
|
}
|
|
|
|
/// Pattern-based privacy classifier using regex matching.
|
|
///
|
|
/// Default patterns target hard PII (SSN, credit card numbers) where silent
|
|
/// redirect is clearly correct. Ambiguous terms (health vocabulary, contact
|
|
/// info) are intentionally excluded — they cause false positives in household
|
|
/// contexts and silently redirect content the user intended to share.
|
|
///
|
|
/// Operators who need broader coverage should use `ConfigurablePrivacyClassifier`
|
|
/// with domain-specific patterns.
|
|
pub struct PatternPrivacyClassifier {
|
|
patterns: Vec<Regex>,
|
|
}
|
|
|
|
impl PatternPrivacyClassifier {
|
|
pub fn new() -> Result<Self, regex::Error> {
|
|
let pattern_strs = [
|
|
// SSN — always PII
|
|
r"\b\d{3}-\d{2}-\d{4}\b",
|
|
// Credit card (basic) — always PII
|
|
r"\b\d{4}[\s-]?\d{4}[\s-]?\d{4}[\s-]?\d{4}\b",
|
|
// Credentials and auth tokens — high-confidence PII
|
|
r"(?i)\b(password|passwd|api[_-]?key|auth[_-]?token|secret[_-]?key)\b",
|
|
];
|
|
let patterns = pattern_strs
|
|
.iter()
|
|
.map(|p| Regex::new(p))
|
|
.collect::<Result<Vec<_>, _>>()?;
|
|
Ok(Self { patterns })
|
|
}
|
|
}
|
|
|
|
impl PrivacyClassifier for PatternPrivacyClassifier {
|
|
fn classify(&self, content: &str) -> SensitivityResult {
|
|
let is_sensitive = self.patterns.iter().any(|p| p.is_match(content));
|
|
SensitivityResult {
|
|
is_sensitive,
|
|
// Regex is binary — matched or not. Always full confidence.
|
|
confidence: if is_sensitive { 1.0 } else { 0.0 },
|
|
}
|
|
}
|
|
}
|
|
|
|
/// User-configurable privacy classifier.
|
|
///
|
|
/// Accepts custom regex patterns at construction time, allowing operators
|
|
/// to tune sensitivity for their use case (e.g., drop health terms that
|
|
/// cause false positives, add domain-specific patterns).
|
|
///
|
|
/// ```
|
|
/// use ironclaw::workspace::privacy::ConfigurablePrivacyClassifier;
|
|
/// use ironclaw::workspace::privacy::PrivacyClassifier;
|
|
///
|
|
/// let classifier = ConfigurablePrivacyClassifier::new(vec![
|
|
/// r"\b\d{3}-\d{2}-\d{4}\b".into(), // SSN only
|
|
/// ]).unwrap();
|
|
/// assert!(classifier.classify("SSN: 123-45-6789").is_sensitive);
|
|
/// assert!(!classifier.classify("saw the doctor today").is_sensitive);
|
|
/// ```
|
|
pub struct ConfigurablePrivacyClassifier {
|
|
patterns: Vec<Regex>,
|
|
}
|
|
|
|
impl ConfigurablePrivacyClassifier {
|
|
/// Create a classifier from user-supplied regex strings.
|
|
///
|
|
/// Returns an error if any pattern fails to compile.
|
|
pub fn new(pattern_strs: Vec<String>) -> Result<Self, regex::Error> {
|
|
let patterns = pattern_strs
|
|
.iter()
|
|
.map(|p| Regex::new(p))
|
|
.collect::<Result<Vec<_>, _>>()?;
|
|
Ok(Self { patterns })
|
|
}
|
|
}
|
|
|
|
impl PrivacyClassifier for ConfigurablePrivacyClassifier {
|
|
fn classify(&self, content: &str) -> SensitivityResult {
|
|
let is_sensitive = self.patterns.iter().any(|p| p.is_match(content));
|
|
SensitivityResult {
|
|
is_sensitive,
|
|
confidence: if is_sensitive { 1.0 } else { 0.0 },
|
|
}
|
|
}
|
|
}
|
|
|
|
#[cfg(test)]
|
|
mod tests {
|
|
use super::*;
|
|
|
|
fn classifier() -> PatternPrivacyClassifier {
|
|
PatternPrivacyClassifier::new().unwrap()
|
|
}
|
|
|
|
// Hard PII — must always trigger
|
|
#[test]
|
|
fn detects_ssn() {
|
|
let result = classifier().classify("My SSN is 123-45-6789");
|
|
assert!(result.is_sensitive);
|
|
assert_eq!(result.confidence, 1.0);
|
|
}
|
|
|
|
#[test]
|
|
fn detects_credit_card() {
|
|
let result = classifier().classify("Card: 4111 1111 1111 1111");
|
|
assert!(result.is_sensitive);
|
|
assert_eq!(result.confidence, 1.0);
|
|
}
|
|
|
|
#[test]
|
|
fn detects_password() {
|
|
assert!(classifier().classify("my password is hunter2").is_sensitive);
|
|
}
|
|
|
|
#[test]
|
|
fn detects_api_key() {
|
|
assert!(
|
|
classifier()
|
|
.classify("set the api_key to sk-1234")
|
|
.is_sensitive
|
|
);
|
|
}
|
|
|
|
// Household content — must NOT trigger (previous false positives)
|
|
#[test]
|
|
fn allows_normal_household_content() {
|
|
let result = classifier().classify("We need to buy groceries for dinner Saturday");
|
|
assert!(!result.is_sensitive);
|
|
assert_eq!(result.confidence, 0.0);
|
|
}
|
|
|
|
#[test]
|
|
fn allows_doctor_mention() {
|
|
assert!(
|
|
!classifier()
|
|
.classify("the doctor's office called about Saturday")
|
|
.is_sensitive
|
|
);
|
|
}
|
|
|
|
#[test]
|
|
fn allows_email_address() {
|
|
assert!(
|
|
!classifier()
|
|
.classify("email [email protected] about the leak")
|
|
.is_sensitive
|
|
);
|
|
}
|
|
|
|
#[test]
|
|
fn allows_phone_number() {
|
|
assert!(
|
|
!classifier()
|
|
.classify("call the restaurant at 555-123-4567")
|
|
.is_sensitive
|
|
);
|
|
}
|
|
|
|
#[test]
|
|
fn allows_medical_terms_in_context() {
|
|
assert!(
|
|
!classifier()
|
|
.classify("Started new medication for anxiety")
|
|
.is_sensitive
|
|
);
|
|
}
|
|
|
|
#[test]
|
|
fn configurable_with_custom_patterns() {
|
|
let c = ConfigurablePrivacyClassifier::new(vec![
|
|
r"\b\d{3}-\d{2}-\d{4}\b".into(), // SSN only
|
|
])
|
|
.unwrap();
|
|
assert!(c.classify("SSN: 123-45-6789").is_sensitive);
|
|
// Health terms no longer trigger with SSN-only config
|
|
assert!(!c.classify("saw the doctor today").is_sensitive);
|
|
}
|
|
|
|
#[test]
|
|
fn configurable_rejects_bad_regex() {
|
|
let result = ConfigurablePrivacyClassifier::new(vec!["[invalid".into()]);
|
|
assert!(result.is_err());
|
|
}
|
|
|
|
#[test]
|
|
fn configurable_empty_patterns_allows_everything() {
|
|
let c = ConfigurablePrivacyClassifier::new(vec![]).unwrap();
|
|
assert!(!c.classify("My SSN is 123-45-6789").is_sensitive);
|
|
}
|
|
|
|
// Format variants
|
|
#[test]
|
|
fn detects_credit_card_no_separators() {
|
|
assert!(
|
|
classifier()
|
|
.classify("card 4111111111111111 on file")
|
|
.is_sensitive
|
|
);
|
|
}
|
|
|
|
#[test]
|
|
fn detects_credit_card_with_dashes() {
|
|
assert!(
|
|
classifier()
|
|
.classify("Card: 4111-1111-1111-1111")
|
|
.is_sensitive
|
|
);
|
|
}
|
|
|
|
#[test]
|
|
fn detects_ssn_bare() {
|
|
assert!(classifier().classify("123-45-6789").is_sensitive);
|
|
}
|
|
|
|
#[test]
|
|
fn detects_auth_token_keyword() {
|
|
assert!(
|
|
classifier()
|
|
.classify("set auth_token to abc123")
|
|
.is_sensitive
|
|
);
|
|
}
|
|
|
|
#[test]
|
|
fn detects_secret_key_keyword() {
|
|
assert!(
|
|
classifier()
|
|
.classify("the secret_key is sk-prod-xyz")
|
|
.is_sensitive
|
|
);
|
|
}
|
|
|
|
#[test]
|
|
fn detects_pii_in_longer_document() {
|
|
let content = "Meeting notes from Thursday.\n\
|
|
Discussed budget and timeline.\n\
|
|
SSN is 999-88-7777 for the insurance form.\n\
|
|
Action items: follow up with vendor.";
|
|
assert!(classifier().classify(content).is_sensitive);
|
|
}
|
|
|
|
#[test]
|
|
fn empty_string_is_not_sensitive() {
|
|
assert!(!classifier().classify("").is_sensitive);
|
|
}
|
|
|
|
#[test]
|
|
fn partial_ssn_not_sensitive() {
|
|
assert!(
|
|
!classifier()
|
|
.classify("code 123-45 in the system")
|
|
.is_sensitive
|
|
);
|
|
}
|
|
}
|