mirror of
https://github.com/outbackdingo/optimclaw.git
synced 2026-08-31 08:39:24 +00:00
feat(workspace): layered memory with sensitivity-based privacy redirect (#1112)
* feat(workspace): layered memory with sensitivity-based privacy redirect Introduce MemoryLayer type for named memory layers with sensitivity levels and write permissions. Layers map to synthetic user_id values in workspace tables, enabling shared/private memory isolation. - Add MemoryLayer, LayerSensitivity types with default_for_user() - Add layer-aware write methods (write_to_layer, append_to_layer) - Add PatternPrivacyClassifier to guard shared layer writes - Add optional 'layer' parameter to memory_write tool and HTTP API - Add 'redirected' and 'actual_layer' fields to write response - Add MEMORY_LAYERS env var (JSON) for layer configuration - Workspace user_id now derived from GATEWAY_USER_ID (was hardcoded "default") - 10 integration tests for layered memory operations Addresses prerequisite for Issue #59 (multi-tenancy). Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: add explicit default to memory_write layer schema Add "default": "private" to the layer parameter's JSON schema so LLM tool consumers can see the default without reading code. Co-Authored-By: Claude Opus 4.6 <[email protected]> * refactor: extract resolve_layer_target to deduplicate layer writes Consolidate shared layer-lookup, writable check, and privacy classification logic from write_to_layer and append_to_layer into a single resolve_layer_target helper. Flagged on #349 review — the duplication originates in this PR. Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: address review feedback on layered memory PR - Fix email regex pipe bug in TLD character class (privacy.rs) - Add append support to web memory_write handler via `append` field - Validate MemoryLayer name/scope: reject empty, check duplicates - Remove hardcoded 'private' default from tool schema; omit layer fields from output when no layer specified - Document scope isolation risk for multi-tenant (Issue #59) Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: address adversarial review findings - CRITICAL: fix identity file protection bypass via trailing slash (normalize target path before protection checks) - HIGH: check private layer is writable before privacy redirect - HIGH: map LayerNotFound/ReadOnly to proper 4xx HTTP status codes - HIGH: honor `append` field in non-layer HTTP write path - MEDIUM: remove redundant DB fetch in append_to_layer (narrower TOCTOU window) - MEDIUM: remove dead memory_write_handler from handlers/memory.rs Co-Authored-By: Claude Opus 4.6 <[email protected]> * feat: opt-in privacy classifier, force override, confidence scoring Address review feedback from @zmanian: - Privacy classifier is now opt-in via with_privacy_classifier() instead of always-on. Default hardcoded patterns (doctor, therapy, email, phone) had unacceptable false positive rates in household contexts. LLM chooses the correct layer via system prompt; regex can't improve on that. - Add ConfigurablePrivacyClassifier for operator-supplied patterns. - PatternPrivacyClassifier defaults narrowed to hard PII only (SSN, credit card, credentials). - Add force param to write_to_layer/append_to_layer to skip classifier. - PrivacyClassifier trait returns SensitivityResult { is_sensitive, confidence } instead of bool, ready for probabilistic classifiers. Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: remove redundant heartbeat match arm in memory_write The heartbeat arm was identical to the catch-all — resolved_path already points to paths::HEARTBEAT when target is "heartbeat". Addresses review feedback from gemini-code-assist on #1112. Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: return Result from PatternPrivacyClassifier::new() Replace .expect() with proper error propagation per project no-panics policy. Remove Default impl (unused in production). Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> * refactor: move memory_layers from GatewayConfig to WorkspaceConfig Resolve merge conflicts between HEAD (transcription, search, env helpers) and the workspace config branch. GatewayConfig no longer owns memory_layers; WorkspaceConfig::resolve() handles parsing, validation (name length >64, character set, empty scope, duplicates), and fallback defaults. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> * test: strengthen privacy classifier and layer isolation coverage Add 8 privacy classifier edge case tests (format variants, keywords, longer documents, empty/partial inputs) and 5 layer write isolation integration tests (cross-scope invisibility, overwrite, empty path, sensitive-to-private no-redirect). Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> * fix: tautological test assertion and add WorkspaceConfig validation tests Replace always-true `is_ok() || is_err()` in write_empty_path_to_layer with actual behavior assertion (write succeeds with normalized empty path). Add 8 unit tests for WorkspaceConfig::resolve() covering valid JSON parsing, invalid JSON, empty/long/invalid-char layer names, empty scopes, duplicates, and default fallback behavior. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> * style: cargo fmt after staging merge Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> --------- Co-authored-by: Claude Opus 4.6 <[email protected]> Co-authored-by: [email protected] <[email protected]>
This commit is contained in:
standardtoaster
+178
-3
@@ -45,6 +45,8 @@ mod document;
|
||||
mod embedding_cache;
|
||||
mod embeddings;
|
||||
pub mod hygiene;
|
||||
pub mod layer;
|
||||
pub mod privacy;
|
||||
#[cfg(feature = "postgres")]
|
||||
mod repository;
|
||||
mod search;
|
||||
@@ -61,6 +63,17 @@ pub use search::{
|
||||
FusionStrategy, RankedResult, SearchConfig, SearchResult, fuse_results, reciprocal_rank_fusion,
|
||||
};
|
||||
|
||||
/// Result of a layer-aware write operation.
|
||||
///
|
||||
/// Contains the written document plus metadata about whether the write
|
||||
/// was redirected to a different layer (e.g., sensitive content redirected
|
||||
/// from shared to private).
|
||||
pub struct WriteResult {
|
||||
pub document: MemoryDocument,
|
||||
pub redirected: bool,
|
||||
pub actual_layer: String,
|
||||
}
|
||||
|
||||
use std::sync::Arc;
|
||||
|
||||
use chrono::{NaiveDate, Utc};
|
||||
@@ -344,20 +357,29 @@ pub struct Workspace {
|
||||
bootstrap_completed: std::sync::atomic::AtomicBool,
|
||||
/// Default search configuration applied to all queries.
|
||||
search_defaults: SearchConfig,
|
||||
/// Memory layers this workspace has access to.
|
||||
memory_layers: Vec<crate::workspace::layer::MemoryLayer>,
|
||||
/// Optional privacy classifier for shared layer writes.
|
||||
/// When None, writes go exactly where requested — no silent redirect.
|
||||
privacy_classifier: Option<Arc<dyn crate::workspace::privacy::PrivacyClassifier>>,
|
||||
}
|
||||
|
||||
impl Workspace {
|
||||
/// Create a new workspace backed by a PostgreSQL connection pool.
|
||||
#[cfg(feature = "postgres")]
|
||||
pub fn new(user_id: impl Into<String>, pool: Pool) -> Self {
|
||||
let user_id_str = user_id.into();
|
||||
let memory_layers = crate::workspace::layer::MemoryLayer::default_for_user(&user_id_str);
|
||||
Self {
|
||||
user_id: user_id.into(),
|
||||
user_id: user_id_str,
|
||||
agent_id: None,
|
||||
storage: WorkspaceStorage::Repo(Repository::new(pool)),
|
||||
embeddings: None,
|
||||
bootstrap_pending: std::sync::atomic::AtomicBool::new(false),
|
||||
bootstrap_completed: std::sync::atomic::AtomicBool::new(false),
|
||||
search_defaults: SearchConfig::default(),
|
||||
memory_layers,
|
||||
privacy_classifier: None,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -365,14 +387,18 @@ impl Workspace {
|
||||
///
|
||||
/// Use this for libSQL or any other backend that implements the Database trait.
|
||||
pub fn new_with_db(user_id: impl Into<String>, db: Arc<dyn crate::db::Database>) -> Self {
|
||||
let user_id_str = user_id.into();
|
||||
let memory_layers = crate::workspace::layer::MemoryLayer::default_for_user(&user_id_str);
|
||||
Self {
|
||||
user_id: user_id.into(),
|
||||
user_id: user_id_str,
|
||||
agent_id: None,
|
||||
storage: WorkspaceStorage::Db(db),
|
||||
embeddings: None,
|
||||
bootstrap_pending: std::sync::atomic::AtomicBool::new(false),
|
||||
bootstrap_completed: std::sync::atomic::AtomicBool::new(false),
|
||||
search_defaults: SearchConfig::default(),
|
||||
memory_layers,
|
||||
privacy_classifier: None,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -444,6 +470,32 @@ impl Workspace {
|
||||
self
|
||||
}
|
||||
|
||||
/// Configure memory layers for this workspace.
|
||||
///
|
||||
/// Also updates read_user_ids to include all layer scopes.
|
||||
pub fn with_memory_layers(mut self, layers: Vec<crate::workspace::layer::MemoryLayer>) -> Self {
|
||||
self.memory_layers = layers;
|
||||
self
|
||||
}
|
||||
|
||||
/// Set a privacy classifier for shared layer writes.
|
||||
///
|
||||
/// When set, writes to shared layers are checked against the classifier
|
||||
/// and redirected to the private layer if sensitive content is detected.
|
||||
/// When unset (the default), writes go exactly where requested.
|
||||
pub fn with_privacy_classifier(
|
||||
mut self,
|
||||
classifier: Arc<dyn crate::workspace::privacy::PrivacyClassifier>,
|
||||
) -> Self {
|
||||
self.privacy_classifier = Some(classifier);
|
||||
self
|
||||
}
|
||||
|
||||
/// Get the configured memory layers.
|
||||
pub fn memory_layers(&self) -> &[crate::workspace::layer::MemoryLayer] {
|
||||
&self.memory_layers
|
||||
}
|
||||
|
||||
/// Get the user ID.
|
||||
pub fn user_id(&self) -> &str {
|
||||
&self.user_id
|
||||
@@ -501,7 +553,9 @@ impl Workspace {
|
||||
/// Append content to a file.
|
||||
///
|
||||
/// Creates the file if it doesn't exist.
|
||||
/// Adds a newline separator between existing and new content.
|
||||
/// Uses a single `\n` separator (suitable for log-style entries).
|
||||
/// For semantic separation (e.g., memory entries), use `append_memory()`
|
||||
/// which uses `\n\n`.
|
||||
pub async fn append(&self, path: &str, content: &str) -> Result<(), WorkspaceError> {
|
||||
let path = normalize_path(path);
|
||||
let doc = self
|
||||
@@ -526,6 +580,127 @@ impl Workspace {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Resolve the target scope for a layer write, optionally applying privacy guards.
|
||||
///
|
||||
/// Validates that the layer exists and is writable. When a privacy classifier
|
||||
/// is configured on the workspace AND `force` is false, checks shared-layer
|
||||
/// writes for sensitive content and redirects to the private layer.
|
||||
///
|
||||
/// By default no classifier is set — writes go exactly where requested.
|
||||
/// This is intentional: the LLM chooses the correct layer via system prompt
|
||||
/// guidance, and a regex classifier can't improve on that decision without
|
||||
/// unacceptable false positive rates in household contexts (e.g., "doctor",
|
||||
/// "therapy", phone numbers). Operators who want a safety net can configure
|
||||
/// one via `with_privacy_classifier()`.
|
||||
///
|
||||
/// # Multi-tenant safety (Issue #59)
|
||||
///
|
||||
/// Layer scopes are currently used directly as `user_id` for DB operations.
|
||||
/// In a multi-tenant deployment, an operator could configure a scope that
|
||||
/// collides with another user's ID, granting write access to their data.
|
||||
/// Future work should namespace or validate scopes to prevent this.
|
||||
///
|
||||
/// Returns `(scope, actual_layer_name, redirected)`.
|
||||
fn resolve_layer_target(
|
||||
&self,
|
||||
layer_name: &str,
|
||||
content: &str,
|
||||
force: bool,
|
||||
) -> Result<(String, String, bool), WorkspaceError> {
|
||||
use crate::workspace::layer::{LayerSensitivity, MemoryLayer};
|
||||
|
||||
let layer = MemoryLayer::find(&self.memory_layers, layer_name).ok_or_else(|| {
|
||||
WorkspaceError::LayerNotFound {
|
||||
name: layer_name.to_string(),
|
||||
}
|
||||
})?;
|
||||
|
||||
if !layer.writable {
|
||||
return Err(WorkspaceError::LayerReadOnly {
|
||||
name: layer_name.to_string(),
|
||||
});
|
||||
}
|
||||
|
||||
if !force
|
||||
&& layer.sensitivity == LayerSensitivity::Shared
|
||||
&& let Some(ref classifier) = self.privacy_classifier
|
||||
&& classifier.classify(content).is_sensitive
|
||||
{
|
||||
tracing::warn!(
|
||||
layer = layer_name,
|
||||
"Redirected sensitive content to private layer"
|
||||
);
|
||||
let private = MemoryLayer::private_layer(&self.memory_layers)
|
||||
.ok_or(WorkspaceError::PrivacyRedirectFailed)?;
|
||||
if !private.writable {
|
||||
return Err(WorkspaceError::PrivacyRedirectFailed);
|
||||
}
|
||||
return Ok((private.scope.clone(), private.name.clone(), true));
|
||||
}
|
||||
|
||||
Ok((layer.scope.clone(), layer_name.to_string(), false))
|
||||
}
|
||||
|
||||
/// Write to a specific memory layer.
|
||||
///
|
||||
/// Checks that the layer exists and is writable. Uses the layer's scope
|
||||
/// as the user_id for the database write. For shared layers, sensitive
|
||||
/// content is automatically redirected to the private layer unless
|
||||
/// `force` is set.
|
||||
pub async fn write_to_layer(
|
||||
&self,
|
||||
layer_name: &str,
|
||||
path: &str,
|
||||
content: &str,
|
||||
force: bool,
|
||||
) -> Result<WriteResult, WorkspaceError> {
|
||||
let (scope, actual_layer, redirected) =
|
||||
self.resolve_layer_target(layer_name, content, force)?;
|
||||
let path = normalize_path(path);
|
||||
let doc = self
|
||||
.storage
|
||||
.get_or_create_document_by_path(&scope, self.agent_id, &path)
|
||||
.await?;
|
||||
self.storage.update_document(doc.id, content).await?;
|
||||
self.reindex_document(doc.id).await?;
|
||||
let document = self.storage.get_document_by_id(doc.id).await?;
|
||||
Ok(WriteResult {
|
||||
document,
|
||||
redirected,
|
||||
actual_layer,
|
||||
})
|
||||
}
|
||||
|
||||
/// Write to a layer, with append semantics.
|
||||
pub async fn append_to_layer(
|
||||
&self,
|
||||
layer_name: &str,
|
||||
path: &str,
|
||||
content: &str,
|
||||
force: bool,
|
||||
) -> Result<WriteResult, WorkspaceError> {
|
||||
let (scope, actual_layer, redirected) =
|
||||
self.resolve_layer_target(layer_name, content, force)?;
|
||||
let path = normalize_path(path);
|
||||
let doc = self
|
||||
.storage
|
||||
.get_or_create_document_by_path(&scope, self.agent_id, &path)
|
||||
.await?;
|
||||
let new_content = if doc.content.is_empty() {
|
||||
content.to_string()
|
||||
} else {
|
||||
format!("{}\n\n{}", doc.content, content)
|
||||
};
|
||||
self.storage.update_document(doc.id, &new_content).await?;
|
||||
self.reindex_document(doc.id).await?;
|
||||
let document = self.storage.get_document_by_id(doc.id).await?;
|
||||
Ok(WriteResult {
|
||||
document,
|
||||
redirected,
|
||||
actual_layer,
|
||||
})
|
||||
}
|
||||
|
||||
/// Check if a file exists.
|
||||
pub async fn exists(&self, path: &str) -> Result<bool, WorkspaceError> {
|
||||
let path = normalize_path(path);
|
||||
|
||||
Reference in New Issue
Block a user