mirror of
https://github.com/outbackdingo/optimclaw.git
synced 2026-08-30 01:19:34 +00:00
feat: 10 infrastructure improvements from zeroclaw (#126)
* refactor: break up agent_loop.rs into four focused modules Split the monolithic 2835-line agent_loop.rs into: - agent_loop.rs (722L): Agent struct, event loop, message dispatch - dispatcher.rs (635L): Agentic tool loop, tool execution, auth detection - commands.rs (484L): System commands, job handlers, heartbeat, summarize - thread_ops.rs (1059L): Thread lifecycle, approval, undo/redo, persistence Each module gets its own impl Agent block. Agent fields changed to pub(super) so sibling modules in the agent package can access them. All 16 existing tests pass in their new locations. Inspired by ZeroClaw's agent module split (agent.rs, loop_.rs, dispatcher.rs, prompt.rs, memory_loader.rs). Co-Authored-By: Claude Opus 4.6 <[email protected]> * feat: add cost caps and guardrails for autonomous agent spending Daily budget (MAX_COST_PER_DAY_CENTS) and hourly action rate (MAX_ACTIONS_PER_HOUR) limits prevent runaway agents from burning through API credits, especially in daemon/heartbeat modes. - CostGuard with pre-flight check and post-call recording - Sliding window for hourly rate, midnight-UTC daily reset - 80% threshold warning, atomic fast-path for exceeded budget - Wired into dispatcher loop (check before LLM call, record after) Co-Authored-By: Claude Opus 4.6 <[email protected]> * feat: add circuit breaker on LLM providers Wraps LlmProvider with a Closed/Open/HalfOpen state machine that trips after consecutive transient failures, preventing request storms against a degraded backend. Automatically probes for recovery. - CircuitBreakerProvider implements LlmProvider (drop-in wrapper) - Transient error classification (server, rate-limit, network, auth infra) - Client errors (wrong model, context overflow) don't trip the breaker - Configurable via CIRCUIT_BREAKER_THRESHOLD and CIRCUIT_BREAKER_RECOVERY_SECS - Composes with existing FailoverProvider (circuit breaker wraps failover) - 12 tests covering full state machine and error classification Co-Authored-By: Claude Opus 4.6 <[email protected]> * feat: add tunnel abstraction for remote access Trait-based tunnel system with lifecycle management (start/stop/health) for exposing the agent to the internet through external tunnel binaries. Five providers: - Cloudflare Tunnel (cloudflared, Zero Trust token auth) - Tailscale (serve for tailnet, funnel for public) - ngrok (with optional custom domain) - Custom (arbitrary command with {host}/{port} placeholders) - None (local-only, no external exposure) Config via TUNNEL_PROVIDER + provider-specific env vars. Extends existing TunnelConfig with optional managed provider alongside the static TUNNEL_URL path. Factory, shared process management, and 37 tests covering all providers and edge cases. Co-Authored-By: Claude Opus 4.6 <[email protected]> * feat: add OS service management (launchd/systemd) Adds `ironclaw service {install,start,stop,status,uninstall}` for running the agent as a background daemon. macOS uses launchd plists under ~/Library/LaunchAgents, Linux uses systemd user units. Co-Authored-By: Claude Opus 4.6 <[email protected]> * feat: add observability trait system with noop, log, and multi backends Introduces an Observer trait for recording agent lifecycle events and metrics, with pluggable backends. The noop backend compiles to zero overhead, log backend uses tracing, and multi fans out to multiple observers. Configured via OBSERVABILITY_BACKEND env var. Co-Authored-By: Claude Opus 4.6 <[email protected]> * feat: add in-memory LLM response cache with TTL and LRU eviction CachedProvider wraps any LlmProvider and caches complete() responses keyed by SHA-256(model + messages). Tool-calling requests are never cached since they trigger side effects. Configurable via RESPONSE_CACHE_ENABLED, RESPONSE_CACHE_TTL_SECS, and RESPONSE_CACHE_MAX_ENTRIES env vars. Co-Authored-By: Claude Opus 4.6 <[email protected]> * feat: add memory hygiene with cadence-gated daily log cleanup Adds workspace::hygiene module that automatically deletes daily log documents older than a configurable retention period (default 30 days). Runs on a 12-hour cadence tracked via a local state file to avoid redundant passes. Best-effort design: failures are logged, never fatal. Co-Authored-By: Claude Opus 4.6 <[email protected]> * feat: add doctor diagnostics command for active health probing Probes external dependencies (Docker, cloudflared, ngrok, tailscale), validates NEAR AI session, checks database connectivity, and verifies workspace directory. Complements the passive `status` command. Co-Authored-By: Claude Opus 4.6 <[email protected]> * feat: add structured TOML config file support Adds ~/.ironclaw/config.toml as a configuration layer between env vars and database settings. Priority: env var > TOML file > DB > defaults. - `ironclaw config init` generates a commented config.toml from current settings - `ironclaw --config path/to/config.toml` loads a custom config file - Settings.merge_from() only overlays non-default values from the TOML file - `ironclaw config path` now shows TOML file status Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: address codex review findings - apply_toml_overlay now returns Result and errors on explicit missing or invalid config paths (was log-only, violating the documented contract that explicit paths are fatal) - custom tunnel url_pattern is now used to filter extracted URLs, not just as a gate for scanning stdout - systemd ExecStart path is now quoted to handle spaces in paths Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: address PR review feedback - Cache key now includes max_tokens, temperature, and stop_sequences so different request parameters produce distinct keys - to_cents() uses .trunc() + parse::<u64> instead of f64 intermediary, avoiding precision loss for large values - Tailscale public URL no longer includes local port (serve/funnel expose on standard HTTPS port 443) Co-Authored-By: Claude Opus 4.6 <[email protected]> * feat: wire up tunnel lifecycle and fix audit findings Connect the tunnel module to the rest of the application so that setting TUNNEL_PROVIDER actually starts a managed tunnel at boot and stops it on shutdown. Previously create_tunnel() was never called outside tests. Changes: - Expand TunnelSettings with provider credential fields (settings.rs) - TunnelConfig::resolve() falls back to DB settings when env vars unset - Start tunnel at boot, stop on shutdown, show URL in boot screen - Setup wizard collects provider-specific credentials (ngrok, cloudflare, tailscale, custom, static URL) - Fix public_url() returning None under lock contention (SharedUrl) - Fix local_host parameter ignored by cloudflare/ngrok/tailscale - Fix tailscale silent fallback to "localhost" on bad JSON - Fix ngrok globally mutating config via add-authtoken (use env var) - Add 10s timeout to tailscale status --json Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: address PR review comments - Document split_whitespace limitation in CustomTunnel doc comment - Remove unnecessary quotes from systemd ExecStart directive Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: address PR review feedback (round 3) - doctor: missing libSQL DB on fresh install is Pass, not Fail - service: quote ExecStart path for systemd space handling Co-Authored-By: Claude Opus 4.6 <[email protected]> * fix: correct cost guard doc comment (LLM calls, not LLM/tool) Co-Authored-By: Claude Opus 4.6 <[email protected]> --------- Co-authored-by: Claude Opus 4.6 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 4.6
parent
436dda0f2f
commit
a158eee1b0
@@ -0,0 +1,140 @@
|
||||
//! Cloudflare Tunnel via the `cloudflared` binary.
|
||||
|
||||
use anyhow::{Result, bail};
|
||||
use tokio::io::AsyncBufReadExt;
|
||||
use tokio::process::Command;
|
||||
|
||||
use crate::tunnel::{
|
||||
SharedProcess, SharedUrl, Tunnel, TunnelProcess, kill_shared, new_shared_process,
|
||||
new_shared_url,
|
||||
};
|
||||
|
||||
/// Wraps `cloudflared` with token-based auth from the Zero Trust dashboard.
|
||||
pub struct CloudflareTunnel {
|
||||
token: String,
|
||||
proc: SharedProcess,
|
||||
url: SharedUrl,
|
||||
}
|
||||
|
||||
impl CloudflareTunnel {
|
||||
pub fn new(token: String) -> Self {
|
||||
Self {
|
||||
token,
|
||||
proc: new_shared_process(),
|
||||
url: new_shared_url(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[async_trait::async_trait]
|
||||
impl Tunnel for CloudflareTunnel {
|
||||
fn name(&self) -> &str {
|
||||
"cloudflare"
|
||||
}
|
||||
|
||||
async fn start(&self, local_host: &str, local_port: u16) -> Result<String> {
|
||||
let origin = format!("http://{local_host}:{local_port}");
|
||||
let mut child = Command::new("cloudflared")
|
||||
.args([
|
||||
"tunnel",
|
||||
"--no-autoupdate",
|
||||
"run",
|
||||
"--token",
|
||||
&self.token,
|
||||
"--url",
|
||||
&origin,
|
||||
])
|
||||
.stdout(std::process::Stdio::piped())
|
||||
.stderr(std::process::Stdio::piped())
|
||||
.kill_on_drop(true)
|
||||
.spawn()?;
|
||||
|
||||
// cloudflared prints the public URL on stderr
|
||||
let stderr = child
|
||||
.stderr
|
||||
.take()
|
||||
.ok_or_else(|| anyhow::anyhow!("Failed to capture cloudflared stderr"))?;
|
||||
|
||||
let mut reader = tokio::io::BufReader::new(stderr).lines();
|
||||
let mut public_url = String::new();
|
||||
|
||||
let deadline = tokio::time::Instant::now() + tokio::time::Duration::from_secs(30);
|
||||
while tokio::time::Instant::now() < deadline {
|
||||
let line =
|
||||
tokio::time::timeout(tokio::time::Duration::from_secs(5), reader.next_line()).await;
|
||||
|
||||
match line {
|
||||
Ok(Ok(Some(l))) => {
|
||||
tracing::debug!("cloudflared: {l}");
|
||||
if let Some(idx) = l.find("https://") {
|
||||
let url_part = &l[idx..];
|
||||
let end = url_part
|
||||
.find(|c: char| c.is_whitespace())
|
||||
.unwrap_or(url_part.len());
|
||||
public_url = url_part[..end].to_string();
|
||||
break;
|
||||
}
|
||||
}
|
||||
Ok(Ok(None)) => break,
|
||||
Ok(Err(e)) => bail!("Error reading cloudflared output: {e}"),
|
||||
Err(_) => {} // line timeout, keep waiting
|
||||
}
|
||||
}
|
||||
|
||||
if public_url.is_empty() {
|
||||
child.kill().await.ok();
|
||||
bail!("cloudflared did not produce a public URL within 30s. Is the token valid?");
|
||||
}
|
||||
|
||||
if let Ok(mut guard) = self.url.write() {
|
||||
*guard = Some(public_url.clone());
|
||||
}
|
||||
|
||||
let mut guard = self.proc.lock().await;
|
||||
*guard = Some(TunnelProcess { child });
|
||||
|
||||
Ok(public_url)
|
||||
}
|
||||
|
||||
async fn stop(&self) -> Result<()> {
|
||||
if let Ok(mut guard) = self.url.write() {
|
||||
*guard = None;
|
||||
}
|
||||
kill_shared(&self.proc).await
|
||||
}
|
||||
|
||||
async fn health_check(&self) -> bool {
|
||||
let guard = self.proc.lock().await;
|
||||
guard.as_ref().is_some_and(|tp| tp.child.id().is_some())
|
||||
}
|
||||
|
||||
fn public_url(&self) -> Option<String> {
|
||||
self.url.read().ok().and_then(|guard| guard.clone())
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn constructor_stores_token() {
|
||||
let tunnel = CloudflareTunnel::new("cf-token".into());
|
||||
assert_eq!(tunnel.token, "cf-token");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn public_url_none_before_start() {
|
||||
assert!(CloudflareTunnel::new("tok".into()).public_url().is_none());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn stop_without_start_is_ok() {
|
||||
assert!(CloudflareTunnel::new("tok".into()).stop().await.is_ok());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn health_false_before_start() {
|
||||
assert!(!CloudflareTunnel::new("tok".into()).health_check().await);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,245 @@
|
||||
//! Custom tunnel via an arbitrary shell command.
|
||||
|
||||
use anyhow::{Result, bail};
|
||||
use tokio::io::AsyncBufReadExt;
|
||||
use tokio::process::Command;
|
||||
|
||||
use crate::tunnel::{
|
||||
SharedProcess, SharedUrl, Tunnel, TunnelProcess, kill_shared, new_shared_process,
|
||||
new_shared_url,
|
||||
};
|
||||
|
||||
/// Bring-your-own tunnel binary.
|
||||
///
|
||||
/// `start_command` supports `{port}` and `{host}` placeholders.
|
||||
/// If `url_pattern` is set, stdout is scanned for a URL matching that
|
||||
/// substring. If `health_url` is set, health checks poll that endpoint.
|
||||
///
|
||||
/// **Note:** The command is split on whitespace, so quoted arguments like
|
||||
/// `--arg "hello world"` won't work. Each token must be a single word.
|
||||
///
|
||||
/// Examples:
|
||||
/// - `bore local {port} --to bore.pub`
|
||||
/// - `ssh -R 80:localhost:{port} serveo.net`
|
||||
pub struct CustomTunnel {
|
||||
start_command: String,
|
||||
health_url: Option<String>,
|
||||
url_pattern: Option<String>,
|
||||
proc: SharedProcess,
|
||||
url: SharedUrl,
|
||||
}
|
||||
|
||||
impl CustomTunnel {
|
||||
pub fn new(
|
||||
start_command: String,
|
||||
health_url: Option<String>,
|
||||
url_pattern: Option<String>,
|
||||
) -> Self {
|
||||
Self {
|
||||
start_command,
|
||||
health_url,
|
||||
url_pattern,
|
||||
proc: new_shared_process(),
|
||||
url: new_shared_url(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[async_trait::async_trait]
|
||||
impl Tunnel for CustomTunnel {
|
||||
fn name(&self) -> &str {
|
||||
"custom"
|
||||
}
|
||||
|
||||
async fn start(&self, local_host: &str, local_port: u16) -> Result<String> {
|
||||
let cmd = self
|
||||
.start_command
|
||||
.replace("{port}", &local_port.to_string())
|
||||
.replace("{host}", local_host);
|
||||
|
||||
let parts: Vec<&str> = cmd.split_whitespace().collect();
|
||||
if parts.is_empty() {
|
||||
bail!("Custom tunnel start_command is empty");
|
||||
}
|
||||
|
||||
let mut child = Command::new(parts[0])
|
||||
.args(&parts[1..])
|
||||
.stdout(std::process::Stdio::piped())
|
||||
.stderr(std::process::Stdio::piped())
|
||||
.kill_on_drop(true)
|
||||
.spawn()?;
|
||||
|
||||
let mut public_url = format!("http://{local_host}:{local_port}");
|
||||
|
||||
if self.url_pattern.is_some()
|
||||
&& let Some(stdout) = child.stdout.take()
|
||||
{
|
||||
let mut reader = tokio::io::BufReader::new(stdout).lines();
|
||||
let deadline = tokio::time::Instant::now() + tokio::time::Duration::from_secs(15);
|
||||
|
||||
while tokio::time::Instant::now() < deadline {
|
||||
let line =
|
||||
tokio::time::timeout(tokio::time::Duration::from_secs(3), reader.next_line())
|
||||
.await;
|
||||
|
||||
match line {
|
||||
Ok(Ok(Some(l))) => {
|
||||
tracing::debug!("custom-tunnel: {l}");
|
||||
if let Some(url) = extract_url(&l) {
|
||||
let matches_pattern = self
|
||||
.url_pattern
|
||||
.as_ref()
|
||||
.is_none_or(|pat| url.contains(pat.as_str()));
|
||||
if matches_pattern {
|
||||
public_url = url;
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(Ok(None) | Err(_)) => break,
|
||||
Err(_) => {}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if let Ok(mut guard) = self.url.write() {
|
||||
*guard = Some(public_url.clone());
|
||||
}
|
||||
|
||||
let mut guard = self.proc.lock().await;
|
||||
*guard = Some(TunnelProcess { child });
|
||||
|
||||
Ok(public_url)
|
||||
}
|
||||
|
||||
async fn stop(&self) -> Result<()> {
|
||||
if let Ok(mut guard) = self.url.write() {
|
||||
*guard = None;
|
||||
}
|
||||
kill_shared(&self.proc).await
|
||||
}
|
||||
|
||||
async fn health_check(&self) -> bool {
|
||||
if let Some(ref url) = self.health_url {
|
||||
return reqwest::Client::new()
|
||||
.get(url)
|
||||
.timeout(std::time::Duration::from_secs(5))
|
||||
.send()
|
||||
.await
|
||||
.is_ok();
|
||||
}
|
||||
|
||||
let guard = self.proc.lock().await;
|
||||
guard.as_ref().is_some_and(|tp| tp.child.id().is_some())
|
||||
}
|
||||
|
||||
fn public_url(&self) -> Option<String> {
|
||||
self.url.read().ok().and_then(|guard| guard.clone())
|
||||
}
|
||||
}
|
||||
|
||||
/// Extract the first `https://` or `http://` URL from a line of text.
|
||||
fn extract_url(line: &str) -> Option<String> {
|
||||
let idx = line.find("https://").or_else(|| line.find("http://"))?;
|
||||
let url_part = &line[idx..];
|
||||
let end = url_part
|
||||
.find(|c: char| c.is_whitespace())
|
||||
.unwrap_or(url_part.len());
|
||||
Some(url_part[..end].to_string())
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[tokio::test]
|
||||
async fn empty_command_returns_error() {
|
||||
let tunnel = CustomTunnel::new(" ".into(), None, None);
|
||||
let result = tunnel.start("127.0.0.1", 8080).await;
|
||||
assert!(result.is_err());
|
||||
assert!(
|
||||
result
|
||||
.unwrap_err()
|
||||
.to_string()
|
||||
.contains("start_command is empty")
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn start_without_pattern_returns_local() {
|
||||
let tunnel = CustomTunnel::new("sleep 1".into(), None, None);
|
||||
let url = tunnel.start("127.0.0.1", 4455).await.unwrap();
|
||||
assert_eq!(url, "http://127.0.0.1:4455");
|
||||
tunnel.stop().await.unwrap();
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn start_with_pattern_extracts_url() {
|
||||
let tunnel = CustomTunnel::new(
|
||||
"echo https://public.example".into(),
|
||||
None,
|
||||
Some("public.example".into()),
|
||||
);
|
||||
let url = tunnel.start("localhost", 9999).await.unwrap();
|
||||
assert_eq!(url, "https://public.example");
|
||||
tunnel.stop().await.unwrap();
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn pattern_filters_non_matching_urls() {
|
||||
// The command outputs two lines: first a non-matching URL, then a matching one.
|
||||
// The pattern filter should skip the first and grab the second.
|
||||
// No shell quoting needed; Command passes args directly to the binary.
|
||||
let tunnel = CustomTunnel::new(
|
||||
r"printf http://internal:1234\nhttps://real.tunnel.io/abc\n".into(),
|
||||
None,
|
||||
Some("tunnel.io".into()),
|
||||
);
|
||||
let url = tunnel.start("localhost", 9999).await.unwrap();
|
||||
assert_eq!(url, "https://real.tunnel.io/abc");
|
||||
tunnel.stop().await.unwrap();
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn replaces_host_and_port_placeholders() {
|
||||
let tunnel = CustomTunnel::new(
|
||||
"echo http://{host}:{port}".into(),
|
||||
None,
|
||||
Some("http://".into()),
|
||||
);
|
||||
let url = tunnel.start("10.1.2.3", 4321).await.unwrap();
|
||||
assert_eq!(url, "http://10.1.2.3:4321");
|
||||
tunnel.stop().await.unwrap();
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn health_with_unreachable_url_is_false() {
|
||||
let tunnel = CustomTunnel::new(
|
||||
"sleep 1".into(),
|
||||
Some("http://127.0.0.1:9/healthz".into()),
|
||||
None,
|
||||
);
|
||||
assert!(!tunnel.health_check().await);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn extract_url_finds_https() {
|
||||
assert_eq!(
|
||||
extract_url("tunnel ready at https://foo.bar.com/path more text"),
|
||||
Some("https://foo.bar.com/path".to_string())
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn extract_url_finds_http() {
|
||||
assert_eq!(
|
||||
extract_url("url=http://localhost:8080"),
|
||||
Some("http://localhost:8080".to_string())
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn extract_url_none_when_absent() {
|
||||
assert_eq!(extract_url("no url here"), None);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,329 @@
|
||||
//! Tunnel abstraction for exposing the agent to the internet.
|
||||
//!
|
||||
//! Wraps external tunnel binaries (cloudflared, ngrok, tailscale, etc.) behind
|
||||
//! a common trait. The gateway starts a tunnel after binding its local port
|
||||
//! and stops it on shutdown.
|
||||
//!
|
||||
//! Supported providers:
|
||||
//! - **cloudflare** - Zero Trust tunnels via `cloudflared`
|
||||
//! - **tailscale** - `tailscale serve` (tailnet) or `tailscale funnel` (public)
|
||||
//! - **ngrok** - instant public URLs via `ngrok`
|
||||
//! - **custom** - any command with `{host}`/`{port}` placeholders
|
||||
//! - **none** - local-only, no external exposure
|
||||
|
||||
mod cloudflare;
|
||||
mod custom;
|
||||
mod ngrok;
|
||||
mod none;
|
||||
mod tailscale;
|
||||
|
||||
pub use cloudflare::CloudflareTunnel;
|
||||
pub use custom::CustomTunnel;
|
||||
pub use ngrok::NgrokTunnel;
|
||||
pub use none::NoneTunnel;
|
||||
pub use tailscale::TailscaleTunnel;
|
||||
|
||||
use std::sync::Arc;
|
||||
|
||||
use anyhow::{Result, bail};
|
||||
use tokio::sync::Mutex;
|
||||
|
||||
/// Lock-free URL storage. Uses `std::sync::RwLock` so `public_url()` (sync)
|
||||
/// never returns a spurious `None` due to async lock contention.
|
||||
pub(crate) type SharedUrl = Arc<std::sync::RwLock<Option<String>>>;
|
||||
|
||||
pub(crate) fn new_shared_url() -> SharedUrl {
|
||||
Arc::new(std::sync::RwLock::new(None))
|
||||
}
|
||||
|
||||
// ── Tunnel trait ─────────────────────────────────────────────────
|
||||
|
||||
/// Provider-agnostic tunnel with lifecycle management.
|
||||
///
|
||||
/// Implementations wrap an external tunnel binary. The gateway calls
|
||||
/// `start()` after binding its local port and `stop()` on shutdown.
|
||||
#[async_trait::async_trait]
|
||||
pub trait Tunnel: Send + Sync {
|
||||
/// Human-readable provider name (e.g. "cloudflare", "tailscale").
|
||||
fn name(&self) -> &str;
|
||||
|
||||
/// Start the tunnel exposing `local_host:local_port` externally.
|
||||
/// Returns the public URL on success.
|
||||
async fn start(&self, local_host: &str, local_port: u16) -> Result<String>;
|
||||
|
||||
/// Stop the tunnel process gracefully.
|
||||
async fn stop(&self) -> Result<()>;
|
||||
|
||||
/// Check if the tunnel process is still alive.
|
||||
async fn health_check(&self) -> bool;
|
||||
|
||||
/// Return the public URL if the tunnel is running, `None` otherwise.
|
||||
fn public_url(&self) -> Option<String>;
|
||||
}
|
||||
|
||||
// ── Shared child-process handle ──────────────────────────────────
|
||||
|
||||
/// Wraps a spawned tunnel child process.
|
||||
pub(crate) struct TunnelProcess {
|
||||
pub child: tokio::process::Child,
|
||||
}
|
||||
|
||||
pub(crate) type SharedProcess = Arc<Mutex<Option<TunnelProcess>>>;
|
||||
|
||||
pub(crate) fn new_shared_process() -> SharedProcess {
|
||||
Arc::new(Mutex::new(None))
|
||||
}
|
||||
|
||||
/// Kill a shared tunnel process if running.
|
||||
pub(crate) async fn kill_shared(proc: &SharedProcess) -> Result<()> {
|
||||
let mut guard = proc.lock().await;
|
||||
if let Some(ref mut tp) = *guard {
|
||||
tp.child.kill().await.ok();
|
||||
tp.child.wait().await.ok();
|
||||
}
|
||||
*guard = None;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
// ── Configuration types ──────────────────────────────────────────
|
||||
|
||||
/// Provider-specific config for Cloudflare tunnels.
|
||||
#[derive(Debug, Clone, Default)]
|
||||
pub struct CloudflareTunnelConfig {
|
||||
/// Token from the Cloudflare Zero Trust dashboard.
|
||||
pub token: String,
|
||||
}
|
||||
|
||||
/// Provider-specific config for Tailscale tunnels.
|
||||
#[derive(Debug, Clone, Default)]
|
||||
pub struct TailscaleTunnelConfig {
|
||||
/// Use `tailscale funnel` (public) instead of `tailscale serve` (tailnet).
|
||||
pub funnel: bool,
|
||||
/// Override the hostname (default: auto-detect from `tailscale status`).
|
||||
pub hostname: Option<String>,
|
||||
}
|
||||
|
||||
/// Provider-specific config for ngrok tunnels.
|
||||
#[derive(Debug, Clone, Default)]
|
||||
pub struct NgrokTunnelConfig {
|
||||
/// ngrok auth token (required).
|
||||
pub auth_token: String,
|
||||
/// Custom domain (requires ngrok paid plan).
|
||||
pub domain: Option<String>,
|
||||
}
|
||||
|
||||
/// Provider-specific config for custom tunnel commands.
|
||||
#[derive(Debug, Clone, Default)]
|
||||
pub struct CustomTunnelConfig {
|
||||
/// Shell command with `{port}` and `{host}` placeholders.
|
||||
pub start_command: String,
|
||||
/// HTTP endpoint to poll for health checks.
|
||||
pub health_url: Option<String>,
|
||||
/// Substring to match in stdout for URL extraction.
|
||||
pub url_pattern: Option<String>,
|
||||
}
|
||||
|
||||
/// Full tunnel configuration.
|
||||
#[derive(Debug, Clone, Default)]
|
||||
pub struct TunnelProviderConfig {
|
||||
/// Provider name: "none", "cloudflare", "tailscale", "ngrok", "custom".
|
||||
pub provider: String,
|
||||
pub cloudflare: Option<CloudflareTunnelConfig>,
|
||||
pub tailscale: Option<TailscaleTunnelConfig>,
|
||||
pub ngrok: Option<NgrokTunnelConfig>,
|
||||
pub custom: Option<CustomTunnelConfig>,
|
||||
}
|
||||
|
||||
// ── Factory ──────────────────────────────────────────────────────
|
||||
|
||||
/// Create a tunnel from config. Returns `None` for provider "none" or empty.
|
||||
pub fn create_tunnel(config: &TunnelProviderConfig) -> Result<Option<Box<dyn Tunnel>>> {
|
||||
match config.provider.as_str() {
|
||||
"none" | "" => Ok(None),
|
||||
|
||||
"cloudflare" => {
|
||||
let cf = config.cloudflare.as_ref().ok_or_else(|| {
|
||||
anyhow::anyhow!("TUNNEL_PROVIDER=cloudflare but no TUNNEL_CF_TOKEN configured")
|
||||
})?;
|
||||
Ok(Some(Box::new(CloudflareTunnel::new(cf.token.clone()))))
|
||||
}
|
||||
|
||||
"tailscale" => {
|
||||
let ts = config.tailscale.as_ref().cloned().unwrap_or_default();
|
||||
Ok(Some(Box::new(TailscaleTunnel::new(ts.funnel, ts.hostname))))
|
||||
}
|
||||
|
||||
"ngrok" => {
|
||||
let ng = config.ngrok.as_ref().ok_or_else(|| {
|
||||
anyhow::anyhow!("TUNNEL_PROVIDER=ngrok but no TUNNEL_NGROK_TOKEN configured")
|
||||
})?;
|
||||
Ok(Some(Box::new(NgrokTunnel::new(
|
||||
ng.auth_token.clone(),
|
||||
ng.domain.clone(),
|
||||
))))
|
||||
}
|
||||
|
||||
"custom" => {
|
||||
let cu = config.custom.as_ref().ok_or_else(|| {
|
||||
anyhow::anyhow!("TUNNEL_PROVIDER=custom but no TUNNEL_CUSTOM_COMMAND configured")
|
||||
})?;
|
||||
Ok(Some(Box::new(CustomTunnel::new(
|
||||
cu.start_command.clone(),
|
||||
cu.health_url.clone(),
|
||||
cu.url_pattern.clone(),
|
||||
))))
|
||||
}
|
||||
|
||||
other => bail!(
|
||||
"Unknown tunnel provider: \"{other}\". Valid: none, cloudflare, tailscale, ngrok, custom"
|
||||
),
|
||||
}
|
||||
}
|
||||
|
||||
// ── Tests ────────────────────────────────────────────────────────
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use tokio::process::Command;
|
||||
|
||||
fn assert_tunnel_err(cfg: &TunnelProviderConfig, needle: &str) {
|
||||
match create_tunnel(cfg) {
|
||||
Err(e) => assert!(
|
||||
e.to_string().contains(needle),
|
||||
"Expected error containing \"{needle}\", got: {e}"
|
||||
),
|
||||
Ok(_) => panic!("Expected error containing \"{needle}\", but got Ok"),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn factory_none_returns_none() {
|
||||
let cfg = TunnelProviderConfig::default();
|
||||
assert!(create_tunnel(&cfg).unwrap().is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn factory_empty_returns_none() {
|
||||
let cfg = TunnelProviderConfig {
|
||||
provider: String::new(),
|
||||
..Default::default()
|
||||
};
|
||||
assert!(create_tunnel(&cfg).unwrap().is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn factory_unknown_provider_errors() {
|
||||
let cfg = TunnelProviderConfig {
|
||||
provider: "wireguard".into(),
|
||||
..Default::default()
|
||||
};
|
||||
assert_tunnel_err(&cfg, "Unknown tunnel provider");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn factory_cloudflare_missing_config_errors() {
|
||||
let cfg = TunnelProviderConfig {
|
||||
provider: "cloudflare".into(),
|
||||
..Default::default()
|
||||
};
|
||||
assert_tunnel_err(&cfg, "TUNNEL_CF_TOKEN");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn factory_cloudflare_with_config_ok() {
|
||||
let cfg = TunnelProviderConfig {
|
||||
provider: "cloudflare".into(),
|
||||
cloudflare: Some(CloudflareTunnelConfig {
|
||||
token: "test-token".into(),
|
||||
}),
|
||||
..Default::default()
|
||||
};
|
||||
let t = create_tunnel(&cfg).unwrap().unwrap();
|
||||
assert_eq!(t.name(), "cloudflare");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn factory_tailscale_defaults_ok() {
|
||||
let cfg = TunnelProviderConfig {
|
||||
provider: "tailscale".into(),
|
||||
..Default::default()
|
||||
};
|
||||
let t = create_tunnel(&cfg).unwrap().unwrap();
|
||||
assert_eq!(t.name(), "tailscale");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn factory_ngrok_missing_config_errors() {
|
||||
let cfg = TunnelProviderConfig {
|
||||
provider: "ngrok".into(),
|
||||
..Default::default()
|
||||
};
|
||||
assert_tunnel_err(&cfg, "TUNNEL_NGROK_TOKEN");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn factory_ngrok_with_config_ok() {
|
||||
let cfg = TunnelProviderConfig {
|
||||
provider: "ngrok".into(),
|
||||
ngrok: Some(NgrokTunnelConfig {
|
||||
auth_token: "tok".into(),
|
||||
domain: None,
|
||||
}),
|
||||
..Default::default()
|
||||
};
|
||||
let t = create_tunnel(&cfg).unwrap().unwrap();
|
||||
assert_eq!(t.name(), "ngrok");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn factory_custom_missing_config_errors() {
|
||||
let cfg = TunnelProviderConfig {
|
||||
provider: "custom".into(),
|
||||
..Default::default()
|
||||
};
|
||||
assert_tunnel_err(&cfg, "TUNNEL_CUSTOM_COMMAND");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn factory_custom_with_config_ok() {
|
||||
let cfg = TunnelProviderConfig {
|
||||
provider: "custom".into(),
|
||||
custom: Some(CustomTunnelConfig {
|
||||
start_command: "echo tunnel".into(),
|
||||
health_url: None,
|
||||
url_pattern: None,
|
||||
}),
|
||||
..Default::default()
|
||||
};
|
||||
let t = create_tunnel(&cfg).unwrap().unwrap();
|
||||
assert_eq!(t.name(), "custom");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn kill_shared_no_process_is_ok() {
|
||||
let proc = new_shared_process();
|
||||
assert!(kill_shared(&proc).await.is_ok());
|
||||
assert!(proc.lock().await.is_none());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn kill_shared_terminates_child() {
|
||||
let proc = new_shared_process();
|
||||
|
||||
let child = Command::new("sleep")
|
||||
.arg("30")
|
||||
.stdout(std::process::Stdio::null())
|
||||
.stderr(std::process::Stdio::null())
|
||||
.spawn()
|
||||
.expect("sleep should spawn");
|
||||
|
||||
{
|
||||
let mut guard = proc.lock().await;
|
||||
*guard = Some(TunnelProcess { child });
|
||||
}
|
||||
|
||||
kill_shared(&proc).await.unwrap();
|
||||
assert!(proc.lock().await.is_none());
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,142 @@
|
||||
//! ngrok tunnel via the `ngrok` binary.
|
||||
|
||||
use anyhow::{Result, bail};
|
||||
use tokio::io::AsyncBufReadExt;
|
||||
use tokio::process::Command;
|
||||
|
||||
use crate::tunnel::{
|
||||
SharedProcess, SharedUrl, Tunnel, TunnelProcess, kill_shared, new_shared_process,
|
||||
new_shared_url,
|
||||
};
|
||||
|
||||
/// Wraps `ngrok` with optional custom domain support (paid plan).
|
||||
pub struct NgrokTunnel {
|
||||
auth_token: String,
|
||||
domain: Option<String>,
|
||||
proc: SharedProcess,
|
||||
url: SharedUrl,
|
||||
}
|
||||
|
||||
impl NgrokTunnel {
|
||||
pub fn new(auth_token: String, domain: Option<String>) -> Self {
|
||||
Self {
|
||||
auth_token,
|
||||
domain,
|
||||
proc: new_shared_process(),
|
||||
url: new_shared_url(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[async_trait::async_trait]
|
||||
impl Tunnel for NgrokTunnel {
|
||||
fn name(&self) -> &str {
|
||||
"ngrok"
|
||||
}
|
||||
|
||||
async fn start(&self, local_host: &str, local_port: u16) -> Result<String> {
|
||||
let mut args = vec!["http".to_string(), format!("{local_host}:{local_port}")];
|
||||
if let Some(ref domain) = self.domain {
|
||||
args.push("--domain".into());
|
||||
args.push(domain.clone());
|
||||
}
|
||||
args.extend(["--log", "stdout", "--log-format", "logfmt"].map(String::from));
|
||||
|
||||
let mut child = Command::new("ngrok")
|
||||
.args(&args)
|
||||
.env("NGROK_AUTHTOKEN", &self.auth_token)
|
||||
.stdout(std::process::Stdio::piped())
|
||||
.stderr(std::process::Stdio::piped())
|
||||
.kill_on_drop(true)
|
||||
.spawn()?;
|
||||
|
||||
let stdout = child
|
||||
.stdout
|
||||
.take()
|
||||
.ok_or_else(|| anyhow::anyhow!("Failed to capture ngrok stdout"))?;
|
||||
|
||||
let mut reader = tokio::io::BufReader::new(stdout).lines();
|
||||
let mut public_url = String::new();
|
||||
|
||||
let deadline = tokio::time::Instant::now() + tokio::time::Duration::from_secs(15);
|
||||
while tokio::time::Instant::now() < deadline {
|
||||
let line =
|
||||
tokio::time::timeout(tokio::time::Duration::from_secs(3), reader.next_line()).await;
|
||||
|
||||
match line {
|
||||
Ok(Ok(Some(l))) => {
|
||||
tracing::debug!("ngrok: {l}");
|
||||
// ngrok logfmt: url=https://xxxx.ngrok-free.app
|
||||
if let Some(idx) = l.find("url=https://") {
|
||||
let url_start = idx + 4; // skip "url="
|
||||
let url_part = &l[url_start..];
|
||||
let end = url_part
|
||||
.find(|c: char| c.is_whitespace())
|
||||
.unwrap_or(url_part.len());
|
||||
public_url = url_part[..end].to_string();
|
||||
break;
|
||||
}
|
||||
}
|
||||
Ok(Ok(None)) => break,
|
||||
Ok(Err(e)) => bail!("Error reading ngrok output: {e}"),
|
||||
Err(_) => {}
|
||||
}
|
||||
}
|
||||
|
||||
if public_url.is_empty() {
|
||||
child.kill().await.ok();
|
||||
bail!("ngrok did not produce a public URL within 15s. Is the auth token valid?");
|
||||
}
|
||||
|
||||
if let Ok(mut guard) = self.url.write() {
|
||||
*guard = Some(public_url.clone());
|
||||
}
|
||||
|
||||
let mut guard = self.proc.lock().await;
|
||||
*guard = Some(TunnelProcess { child });
|
||||
|
||||
Ok(public_url)
|
||||
}
|
||||
|
||||
async fn stop(&self) -> Result<()> {
|
||||
if let Ok(mut guard) = self.url.write() {
|
||||
*guard = None;
|
||||
}
|
||||
kill_shared(&self.proc).await
|
||||
}
|
||||
|
||||
async fn health_check(&self) -> bool {
|
||||
let guard = self.proc.lock().await;
|
||||
guard.as_ref().is_some_and(|tp| tp.child.id().is_some())
|
||||
}
|
||||
|
||||
fn public_url(&self) -> Option<String> {
|
||||
self.url.read().ok().and_then(|guard| guard.clone())
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn constructor_stores_domain() {
|
||||
let tunnel = NgrokTunnel::new("tok".into(), Some("my.ngrok.app".into()));
|
||||
assert_eq!(tunnel.domain.as_deref(), Some("my.ngrok.app"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn public_url_none_before_start() {
|
||||
assert!(NgrokTunnel::new("tok".into(), None).public_url().is_none());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn stop_without_start_is_ok() {
|
||||
assert!(NgrokTunnel::new("tok".into(), None).stop().await.is_ok());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn health_false_before_start() {
|
||||
assert!(!NgrokTunnel::new("tok".into(), None).health_check().await);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,62 @@
|
||||
//! No-op tunnel for local-only access.
|
||||
|
||||
use anyhow::Result;
|
||||
|
||||
use crate::tunnel::Tunnel;
|
||||
|
||||
/// No-op tunnel, no external exposure. `public_url()` always returns `None`.
|
||||
pub struct NoneTunnel;
|
||||
|
||||
#[async_trait::async_trait]
|
||||
impl Tunnel for NoneTunnel {
|
||||
fn name(&self) -> &str {
|
||||
"none"
|
||||
}
|
||||
|
||||
async fn start(&self, local_host: &str, local_port: u16) -> Result<String> {
|
||||
Ok(format!("http://{local_host}:{local_port}"))
|
||||
}
|
||||
|
||||
async fn stop(&self) -> Result<()> {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn health_check(&self) -> bool {
|
||||
true
|
||||
}
|
||||
|
||||
fn public_url(&self) -> Option<String> {
|
||||
None
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn name_is_none() {
|
||||
assert_eq!(NoneTunnel.name(), "none");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn start_returns_local_url() {
|
||||
let url = NoneTunnel.start("127.0.0.1", 7788).await.unwrap();
|
||||
assert_eq!(url, "http://127.0.0.1:7788");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn stop_is_noop() {
|
||||
assert!(NoneTunnel.stop().await.is_ok());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn health_is_always_true() {
|
||||
assert!(NoneTunnel.health_check().await);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn public_url_is_always_none() {
|
||||
assert!(NoneTunnel.public_url().is_none());
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,140 @@
|
||||
//! Tailscale tunnel via `tailscale serve` or `tailscale funnel`.
|
||||
|
||||
use anyhow::{Result, bail};
|
||||
use tokio::process::Command;
|
||||
|
||||
use crate::tunnel::{
|
||||
SharedProcess, SharedUrl, Tunnel, TunnelProcess, kill_shared, new_shared_process,
|
||||
new_shared_url,
|
||||
};
|
||||
|
||||
/// Uses `tailscale serve` (tailnet-only) or `tailscale funnel` (public).
|
||||
///
|
||||
/// Requires Tailscale installed and authenticated (`tailscale up`).
|
||||
pub struct TailscaleTunnel {
|
||||
funnel: bool,
|
||||
hostname: Option<String>,
|
||||
proc: SharedProcess,
|
||||
url: SharedUrl,
|
||||
}
|
||||
|
||||
impl TailscaleTunnel {
|
||||
pub fn new(funnel: bool, hostname: Option<String>) -> Self {
|
||||
Self {
|
||||
funnel,
|
||||
hostname,
|
||||
proc: new_shared_process(),
|
||||
url: new_shared_url(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[async_trait::async_trait]
|
||||
impl Tunnel for TailscaleTunnel {
|
||||
fn name(&self) -> &str {
|
||||
"tailscale"
|
||||
}
|
||||
|
||||
async fn start(&self, local_host: &str, local_port: u16) -> Result<String> {
|
||||
let subcommand = if self.funnel { "funnel" } else { "serve" };
|
||||
|
||||
let hostname = if let Some(ref h) = self.hostname {
|
||||
h.clone()
|
||||
} else {
|
||||
let output = tokio::time::timeout(
|
||||
tokio::time::Duration::from_secs(10),
|
||||
Command::new("tailscale")
|
||||
.args(["status", "--json"])
|
||||
.output(),
|
||||
)
|
||||
.await
|
||||
.map_err(|_| anyhow::anyhow!("tailscale status --json timed out after 10s"))??;
|
||||
|
||||
if !output.status.success() {
|
||||
bail!(
|
||||
"tailscale status failed: {}",
|
||||
String::from_utf8_lossy(&output.stderr)
|
||||
);
|
||||
}
|
||||
|
||||
let status: serde_json::Value = serde_json::from_slice(&output.stdout)
|
||||
.map_err(|e| anyhow::anyhow!("Failed to parse tailscale status JSON: {e}"))?;
|
||||
status["Self"]["DNSName"]
|
||||
.as_str()
|
||||
.ok_or_else(|| anyhow::anyhow!("tailscale status missing Self.DNSName field"))?
|
||||
.trim_end_matches('.')
|
||||
.to_string()
|
||||
};
|
||||
|
||||
let target = format!("http://{local_host}:{local_port}");
|
||||
let child = Command::new("tailscale")
|
||||
.args([subcommand, &target])
|
||||
.stdout(std::process::Stdio::piped())
|
||||
.stderr(std::process::Stdio::piped())
|
||||
.kill_on_drop(true)
|
||||
.spawn()?;
|
||||
|
||||
let public_url = format!("https://{hostname}");
|
||||
|
||||
if let Ok(mut guard) = self.url.write() {
|
||||
*guard = Some(public_url.clone());
|
||||
}
|
||||
|
||||
let mut guard = self.proc.lock().await;
|
||||
*guard = Some(TunnelProcess { child });
|
||||
|
||||
Ok(public_url)
|
||||
}
|
||||
|
||||
async fn stop(&self) -> Result<()> {
|
||||
let subcommand = if self.funnel { "funnel" } else { "serve" };
|
||||
if let Err(e) = Command::new("tailscale")
|
||||
.args([subcommand, "reset"])
|
||||
.output()
|
||||
.await
|
||||
{
|
||||
tracing::warn!("tailscale {subcommand} reset failed: {e}");
|
||||
}
|
||||
|
||||
if let Ok(mut guard) = self.url.write() {
|
||||
*guard = None;
|
||||
}
|
||||
kill_shared(&self.proc).await
|
||||
}
|
||||
|
||||
async fn health_check(&self) -> bool {
|
||||
let guard = self.proc.lock().await;
|
||||
guard.as_ref().is_some_and(|tp| tp.child.id().is_some())
|
||||
}
|
||||
|
||||
fn public_url(&self) -> Option<String> {
|
||||
self.url.read().ok().and_then(|guard| guard.clone())
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn constructor_stores_hostname_and_mode() {
|
||||
let tunnel = TailscaleTunnel::new(true, Some("myhost.ts.net".into()));
|
||||
assert!(tunnel.funnel);
|
||||
assert_eq!(tunnel.hostname.as_deref(), Some("myhost.ts.net"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn public_url_none_before_start() {
|
||||
assert!(TailscaleTunnel::new(false, None).public_url().is_none());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn health_false_before_start() {
|
||||
assert!(!TailscaleTunnel::new(false, None).health_check().await);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn stop_without_start_is_ok() {
|
||||
assert!(TailscaleTunnel::new(false, None).stop().await.is_ok());
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user