1. Introduction: Why Anomaly Detection Is Existential for Agentic Companies
The promise of agentic companies — organizations where AI agents autonomously execute business processes under governance constraints — rests on a critical assumption: that deviations from expected behavior will be detected and corrected before they cause irreversible damage. This assumption is not automatically satisfied. An autonomous agent authorized to execute procurement decisions can, if its policy drifts or its reward signal is corrupted, approve purchases that systematically benefit a specific vendor. An agent managing customer communications can, if its language model hallucinated, send messages that violate regulatory requirements. An agent optimizing resource allocation can, if its objective function has an edge case, concentrate all resources in a single zone, starving others.
These are not hypothetical scenarios. Every complex system produces deviations. The second law of thermodynamics guarantees that entropy increases in closed systems; the organizational equivalent is that behavioral drift is the default trajectory for any sufficiently complex multi-agent system. The question is not whether deviations will occur, but whether they will be detected before they cascade into system-wide failure.
Traditional software monitoring — log aggregation, threshold-based alerts, rule-based anomaly detection — is insufficient for agentic companies. Rule-based systems can only detect anomalies that were anticipated at design time, but the behavioral space of autonomous agents is vast and the most dangerous anomalies are precisely those that were not anticipated. Statistical threshold monitoring can detect agents that exceed predefined limits, but it cannot detect agents that operate within nominal ranges while exhibiting subtle correlational anomalies — for example, an agent that processes transactions at normal speed and volume but systematically routes high-value transactions to a specific downstream agent.
What is needed is a learned anomaly detection system that can model the complex, high-dimensional distribution of normal agent behavior and flag deviations from this distribution regardless of whether the specific type of deviation was anticipated. This paper presents two such methods — Isolation Forest and Autoencoder-based reconstruction error — and shows how their combination provides a robust safety layer for agentic companies.
The economic consequences of undetected anomalies are severe and well-documented in analogous domains. In traditional financial services, a single rogue trader at Societe Generale caused 4.9 billion euros in losses over a period during which automated monitoring systems failed to flag increasingly anomalous trading patterns. In software engineering, a single misconfigured load balancer at Amazon Web Services caused a cascading outage affecting hundreds of dependent services. In autonomous vehicle development, edge-case failures in perception systems have caused accidents despite millions of miles of normal operation. Each of these failures shares a common pattern: an anomalous condition that was outside the design envelope of the monitoring system propagated through tightly coupled systems until it caused catastrophic damage. Agentic companies face the same risk at organizational scale.
The challenge of anomaly detection in agentic companies is compounded by the fact that agents are adaptive. Unlike static software systems where anomalies arise from bugs or configuration errors, agentic anomalies can arise from learned behavior that drifts over time as agents optimize their policies. An agent that discovers a reward signal loophole will gradually shift its behavior toward exploiting the loophole, producing a slow drift that is much harder to detect than a sudden deviation. This adaptive anomaly generation requires detection methods that can identify not just point anomalies (sudden deviations) but collective anomalies (gradual drifts) and structural anomalies (changes in inter-agent relationships).
1.1 The Safety Layer in the Algorithm Stack
In the 7-layer algorithm stack for agentic organizations, the Safety Layer (Layer 7) occupies the outermost position, wrapping all other layers in a continuous monitoring envelope. Its relationship to the other layers is as follows: | Layer | Algorithm | Safety Layer Interaction | |---|---|---| | 1. Cognition | Transformer | Monitors LLM outputs for hallucination, toxicity, policy violation | | 2. Decision | Gradient Boosting | Monitors predictions for distributional shift, confidence collapse | | 3. Structure | GNN | Monitors graph topology for anomalous edge formation, node isolation | | 4. Control | RL/Actor-Critic | Monitors policy updates for reward hacking, objective misalignment | | 5. Exploration | Multi-Armed Bandit | Monitors exploration for stuck arms, exploitation traps | | 6. Abstraction | PCA | Monitors dimensionality reduction for information loss spikes | | 7. Safety | Isolation Forest + Autoencoder | Self-monitors for detector degradation, concept drift |
The Safety Layer receives telemetry from all other layers and produces anomaly scores that are fed to the MARIA OS governance engine. When anomaly scores exceed calibrated thresholds, the governance engine triggers proportional responses ranging from logging (observe) to throttling (reduce agent autonomy) to freezing (suspend agent operations pending human review).
1.2 Paper Organization
Section 2 formalizes the anomaly detection problem in the agentic context. Section 3 presents Isolation Forest adapted to organizational behavior monitoring. Section 4 presents Autoencoder-based detection with learned normal behavior manifolds. Section 5 derives the combined detection method. Section 6 addresses threshold calibration for enterprise risk. Section 7 presents the anomaly-throttle-freeze cascade. Section 8 describes the real-time streaming architecture. Section 9 formalizes the MARIA OS stability guard. Section 10 addresses false positive management. Section 11 presents experimental results. Section 12 discusses adversarial robustness. Section 13 concludes.
2. Formalizing the Anomaly Detection Problem
2.1 The Semi-Supervised Anomaly Setting
Anomaly detection in enterprise contexts operates in a semi-supervised setting: we have abundant examples of normal behavior (collected from the organization's operational history) but very few examples of anomalous behavior (anomalies are rare events by definition, and the specific forms of future anomalies are unknown). This asymmetry fundamentally shapes the choice of detection methods. Supervised classification is impractical because the anomaly class is too diverse and too sparse to model directly. Fully unsupervised methods (detecting any statistical outlier) produce too many false positives because statistical outliers are common in high-dimensional behavioral spaces and most are benign. The optimal approach is semi-supervised: learn a detailed model of normal behavior from abundant normal examples, and flag any observation that deviates significantly from the learned normal model. Both Isolation Forest and Autoencoders operate in this semi-supervised paradigm.
2.2 Agent Behavioral Space
Let each agent $a_i$ produce a behavioral observation vector $\mathbf{x}_i(t) \in \mathbb{R}^d$ at each time step $t$. This vector is the same feature vector used in the Abstraction Layer (Section 6 of the algorithm stack paper) and includes task profiles, communication patterns, performance metrics, and governance compliance indicators. The behavioral space is the union of all possible observation vectors: $\mathcal{X} = \mathbb{R}^d$.
2.2 Normal Behavior Distribution
We model normal behavior as a probability distribution $P_{\text{normal}}(\mathbf{x})$ over the behavioral space. An agent's observation $\mathbf{x}_i(t)$ is normal if it was generated from $P_{\text{normal}}$ and anomalous if it was generated from some other distribution $P_{\text{anomaly}}$. Formally, the anomaly detection problem is a binary hypothesis test: $$ H_0: \mathbf{x}_i(t) \sim P_{\text{normal}} \quad \text{vs.} \quad H_1: \mathbf{x}_i(t) \sim P_{\text{anomaly}} $$ The challenge is that $P_{\text{normal}}$ is unknown and must be learned from data, while $P_{\text{anomaly}}$ is not just unknown but unknowable — anomalies can take infinitely many forms, and the most dangerous ones are those never seen before. This asymmetry rules out supervised classification and mandates unsupervised or semi-supervised approaches that learn to characterize normality rather than enumerate anomaly types.
The challenge is made more precise by noting that $P_{\text{anomaly}}$ encompasses infinitely many possible anomaly distributions, each representing a different type of deviation. A reward-hacking anomaly produces a different distribution than a communication-channel anomaly, which in turn differs from a resource-exhaustion anomaly. The detection system must be sensitive to all of these without being specifically trained on any of them. This requirement for generalization across anomaly types is the fundamental reason why model-free methods (like Isolation Forest) and manifold-learning methods (like Autoencoders) are preferred over parametric methods that assume specific anomaly distributions.
2.3 Anomaly Taxonomy for Agentic Systems
We distinguish four categories of anomalies in agentic systems, each requiring different detection sensitivities: | Category | Description | Example | Severity | |---|---|---|---| | Point anomaly | Single observation deviates from normal | Agent processes a transaction 100x larger than historical average | Medium | | Contextual anomaly | Observation is normal globally but anomalous in context | Agent performs weekend maintenance activity at high volume on a Tuesday | Medium | | Collective anomaly | Sequence of individually normal observations forms anomalous pattern | Agent gradually shifts approval threshold lower over 30 days | High | | Structural anomaly | Agent's relationship to other agents changes anomalously | Agent begins receiving input from an unauthorized source | Critical | Isolation Forest excels at detecting point and contextual anomalies. Autoencoders excel at detecting collective and structural anomalies. Their combination covers all four categories.
3. Isolation Forest for Tree-Based Anomaly Scoring
3.1 Algorithm Intuition
Isolation Forest is based on a beautifully simple insight: anomalies are easy to isolate. In a dataset of mostly normal points, an anomalous point sits far from the dense regions and can be separated from the rest with very few random partitions. A normal point, surrounded by similar points, requires many partitions to isolate. The number of partitions required to isolate a point — its path length in a random binary tree — is therefore a direct measure of its anomaly score.
3.2 Algorithm Formulation
An Isolation Forest consists of $T$ isolation trees, each built by randomly selecting a feature $q$ and a split value $p$ uniformly within the range of $q$, recursively partitioning the data until each point is isolated in its own leaf or the tree reaches a maximum depth $\lceil \log_2 n \rceil$. The path length $h(\mathbf{x})$ of an observation $\mathbf{x}$ is the number of edges from the root to the leaf containing $\mathbf{x}$. The anomaly score is: $$ s(\mathbf{x}, n) = 2^{-\frac{E[h(\mathbf{x})]}{c(n)}} $$ where $E[h(\mathbf{x})]$ is the average path length across all $T$ trees and $c(n) = 2H(n-1) - 2(n-1)/n$ is the average path length in an unsuccessful binary search tree of $n$ elements, serving as a normalization factor. Scores close to 1 indicate anomalies (short path lengths, easy to isolate); scores close to 0.5 indicate normal observations (average path lengths); scores close to 0 indicate extremely dense normal regions.
The theoretical foundation of Isolation Forest rests on the observation that anomalies have two distinguishing properties: they are few (constituting a small fraction of the total population) and they are different (having attribute values that differ significantly from normal instances). These two properties conspire to make anomalies easy to isolate: because they are few and different, random partitions will separate them from the normal majority with very few splits. The expected path length for an anomaly is $O(1)$ (constant, independent of dataset size), while the expected path length for a normal instance is $O(\log n)$. This $O(\log n)$ separation provides the statistical power for anomaly detection and explains why Isolation Forest maintains high accuracy even on very large datasets.
3.3 Adaptation to Agent Monitoring
In the agentic context, we build Isolation Forest models at multiple granularities to capture anomalies at different organizational scales: - Agent-level: One forest per agent, trained on that agent's historical behavioral vectors. Detects deviations from an individual agent's normal behavior pattern. - Role-level: One forest per organizational role (cluster), trained on behavioral vectors of all agents in that role. Detects agents behaving inconsistently with their assigned role. - Organization-level: One forest for the entire organization, trained on all agents. Detects agents that are globally anomalous regardless of role context. The multi-granularity approach ensures that both local anomalies (an agent deviating from its own baseline) and global anomalies (an agent deviating from all organizational norms) are detected.
The multi-granularity approach also addresses the base rate problem. At the organization level, the base rate of anomalies is very low (typically 0.1–0.5% of observations), making detection statistically challenging. At the agent level, the base rate is higher because each agent has a narrower behavioral distribution, making deviations more statistically significant. At the role level, the base rate is intermediate. By combining scores across granularities, we achieve robust detection that leverages the statistical strengths of each level.
3.4 Streaming Isolation Forest
Standard Isolation Forest is a batch algorithm: it builds trees on a fixed dataset. In a live agentic system, the behavioral distribution evolves over time as agents learn, tasks change, and the organization adapts. We implement a streaming variant that maintains a sliding window of the most recent $W$ observations per agent and periodically rebuilds the forest on the updated window. The rebuild frequency is adaptive: faster when the behavioral distribution is shifting (detected by monitoring the distribution of anomaly scores) and slower when the distribution is stable. In practice, we rebuild every 15 minutes during normal operations and every 2 minutes during organizational transitions (re-clustering events, new agent onboarding, policy changes).
The sliding window approach introduces a tradeoff between sensitivity and stability. A short window (e.g., 1 hour) makes the model highly sensitive to recent behavior changes but may produce false positives when the model has not yet adapted to legitimate behavioral shifts. A long window (e.g., 24 hours) provides a stable baseline but may miss anomalies that emerge within a short time frame. We resolve this tradeoff by maintaining two Isolation Forest models with different window sizes: a short-window model ($W = 2$ hours) for detecting rapid-onset anomalies and a long-window model ($W = 24$ hours) for detecting gradual drift. An observation is flagged as anomalous only if at least one model produces a high anomaly score, with the short-window score given priority for point anomalies and the long-window score given priority for collective anomalies.
3.5 Feature Importance for Anomaly Explanation
A critical requirement for governance is not just that anomalies are detected, but that they are explainable. An anomaly score of 0.92 tells a governance officer that something is wrong, but not what. We extract feature importance from Isolation Forest by tracking which features are used at the earliest splits for anomalous observations. Features that appear at shallow depths in the trees are the primary drivers of the anomaly, because they are the features on which the anomalous observation is most easily separated from the rest. We present the top-3 anomaly-driving features for each flagged observation, enabling governance officers to quickly understand the nature of the deviation.
4. Autoencoder-Based Anomaly Detection
4.1 Learning the Normal Behavior Manifold
While Isolation Forest detects anomalies through partitioning in the original feature space, Autoencoders detect anomalies by learning a compressed representation of normal behavior and measuring how poorly an observation can be reconstructed from this compressed representation. The core insight is that normal behavior, despite occupying a high-dimensional feature space, actually lies on a lower-dimensional manifold defined by the correlational structure of normal operations. Anomalous behavior deviates from this manifold and cannot be accurately reconstructed.
4.2 Architecture
The autoencoder consists of an encoder $g_\phi: \mathbb{R}^d \to \mathbb{R}^m$ that maps the $d$-dimensional behavioral vector to an $m$-dimensional latent representation ($m \ll d$), and a decoder $f_\theta: \mathbb{R}^m \to \mathbb{R}^d$ that reconstructs the original vector from the latent representation. The network is trained to minimize reconstruction error on normal data: $$ \mathcal{L}(\phi, \theta) = \frac{1}{N} \sum_{i=1}^{N} \|\mathbf{x}_i - f_\theta(g_\phi(\mathbf{x}_i))\|^2 $$ After training, the reconstruction error for a new observation serves as the anomaly score: $$ \text{AnomalyScore}_{\text{AE}}(\mathbf{x}) = \|\mathbf{x} - f_\theta(g_\phi(\mathbf{x}))\|^2 $$ Normal observations lie on the learned manifold and are reconstructed accurately (low error). Anomalous observations lie off the manifold and are reconstructed poorly (high error).
The key advantage of the autoencoder approach over Isolation Forest is that it learns the correlational structure of normal behavior. An Isolation Forest treats each feature independently during partitioning (each split is on a single feature), so it can miss anomalies that are defined by unusual feature combinations rather than unusual individual feature values. For example, an agent that processes transactions at normal volume AND at normal speed is not anomalous on either dimension individually, but if the combination of high volume and low latency is unusual for that agent's role, it may indicate automated behavior that bypasses quality checks. The autoencoder captures this correlational structure through its compressed latent representation: during training, it learns which feature combinations are normal, and during scoring, it produces high reconstruction error for observations with unusual feature combinations.
A related advantage is that the autoencoder can detect subtle distributional shifts that are invisible to instance-level detection methods. If the entire population of agents gradually shifts its behavior (for example, due to a slowly degrading reward signal), each individual observation may appear normal relative to the current baseline, but the autoencoder's reconstruction error will gradually increase as the learned manifold becomes less representative of current behavior. This sensitivity to distributional shift makes the autoencoder an effective complement to the instance-level Isolation Forest.
4.3 Architectural Choices for Agent Monitoring
For the 40-dimensional agent behavioral vectors used in MARIA OS, we use a symmetric architecture with layers [40, 24, 12, 6, 12, 24, 40], where the latent dimension $m = 6$ provides a 6.7x compression ratio. The choice of $m$ is critical: too large and the autoencoder learns to memorize the identity function, detecting nothing; too small and the autoencoder cannot capture normal behavioral complexity, producing false positives on normal variations. We determine $m$ by monitoring validation reconstruction error as a function of latent dimensionality and selecting the knee point where further compression produces disproportionate error increase. Activation functions are ReLU for hidden layers and linear for the output layer (since behavioral features can be negative after z-score normalization). We use dropout ($p = 0.1$) in the encoder to prevent overfitting and batch normalization to stabilize training. Training uses Adam optimizer with learning rate $10^{-3}$ and early stopping based on validation loss with patience of 20 epochs.
The choice of latent dimension $m$ has a direct organizational interpretation. The latent dimensions correspond to the fundamental axes of variation in normal agent behavior. In our deployments, we observed that the 6 learned latent dimensions typically correspond to interpretable organizational factors: task intensity (how actively the agent is working), communication centrality (how connected the agent is in the communication graph), governance compliance (how closely the agent follows governance rules), specialization depth (how focused the agent is on its assigned role), error propensity (how often the agent makes mistakes), and temporal regularity (how consistent the agent's behavior is over time). These interpretable latent factors provide additional diagnostic information when an anomaly is detected: we can identify which latent factor is most disrupted, providing a compressed explanation of the anomaly.
4.4 Variational Autoencoder for Probabilistic Anomaly Scoring
The standard autoencoder produces a deterministic anomaly score. For governance applications where we need calibrated probability estimates, we extend to a Variational Autoencoder (VAE) that models the latent space as a probability distribution rather than a point. The encoder outputs a mean $\boldsymbol{\mu}$ and variance $\boldsymbol{\sigma}^2$ for each latent dimension, and the latent representation is sampled: $\mathbf{z} \sim \mathcal{N}(\boldsymbol{\mu}, \text{diag}(\boldsymbol{\sigma}^2))$. The loss function adds a KL divergence term: $$ \mathcal{L}_{\text{VAE}} = \frac{1}{N} \sum_{i=1}^{N} \left[ \|\mathbf{x}_i - f_\theta(\mathbf{z}_i)\|^2 + \beta \cdot D_{\text{KL}}(q_\phi(\mathbf{z}|\mathbf{x}_i) \| p(\mathbf{z})) \right] $$ The VAE produces a probabilistic anomaly score by computing the probability of the observation under the learned generative model. Observations with low probability under the model are flagged as anomalous. The $\beta$ parameter controls the balance between reconstruction quality and latent space regularity, and is tuned to optimize detection performance on a held-out validation set of known anomalies.
4.5 Feature-Wise Reconstruction Error for Anomaly Localization
Like Isolation Forest, the autoencoder must provide explainable anomaly detection. We decompose the reconstruction error into per-feature contributions: $$ e_j(\mathbf{x}) = (x_j - \hat{x}_j)^2 $$ where $\hat{x}_j$ is the $j$-th component of the reconstructed vector. Features with the largest reconstruction error are the primary anomaly indicators. This decomposition enables governance officers to see not just that an agent is anomalous but exactly which behavioral dimensions are deviating from normal. For example, a high reconstruction error on the 'approval request frequency' feature combined with low error on all other features pinpoints the anomaly to the governance behavior of the agent.
5. Combined Detection: Ensemble Anomaly Scoring
The per-feature decomposition also enables automatic anomaly categorization. By clustering the reconstruction error vectors $\mathbf{e}(\mathbf{x}) = (e_1, e_2, \ldots, e_d)$ of past anomalies, we build a taxonomy of anomaly types that can be used to automatically categorize new anomalies. For example, anomalies with high error on governance features but low error on task features are categorized as 'governance anomalies', while anomalies with high error on communication features are categorized as 'communication anomalies'. This automatic categorization routes anomaly alerts to the appropriate governance specialist, reducing investigation time.
5.1 Why Combination Is Necessary
Isolation Forest and Autoencoders have complementary strengths and weaknesses. Isolation Forest excels at detecting point anomalies in the original feature space but struggles with anomalies that are defined by complex correlational patterns across features. Autoencoders capture correlational structure through the learned manifold but can miss anomalies in low-variance feature dimensions that are compressed away during encoding. Combining both methods produces a detection system that covers a broader range of anomaly types than either method alone.
5.2 Score Fusion
We combine the Isolation Forest score $s_{\text{IF}}(\mathbf{x})$ and the Autoencoder reconstruction error $s_{\text{AE}}(\mathbf{x})$ through a weighted geometric mean: $$ s_{\text{combined}}(\mathbf{x}) = s_{\text{IF}}(\mathbf{x})^{w_1} \cdot \left(\frac{s_{\text{AE}}(\mathbf{x})}{\tau_{\text{AE}}}\right)^{w_2} $$ where $\tau_{\text{AE}}$ normalizes the autoencoder score to [0, 1] range and $w_1 + w_2 = 1$. The weights are tuned on a validation set of labeled anomalies. In our deployments, equal weighting ($w_1 = w_2 = 0.5$) provides near-optimal performance, consistent with the finding that the two methods contribute approximately equally but orthogonally to detection accuracy.
The geometric mean formulation was chosen over arithmetic mean or max-pooling because it provides a natural AND-like combination: a high combined score requires both methods to produce above-average scores, while a low combined score occurs whenever either method produces a very low score. This AND-like behavior is appropriate for the observe threshold (where we want both methods to agree before taking any action) but may be too conservative for the freeze threshold (where we want to act even if only one method detects a critical anomaly). We therefore use the geometric mean for the observe and throttle thresholds but the max of the two scores for the freeze threshold, ensuring that a critical detection by either method alone is sufficient to trigger the most severe response.
5.3 Consensus and Disagreement Analysis
Beyond the combined score, we analyze agreement between the two methods: | IF Score | AE Score | Interpretation | Action | |---|---|---|---| | High | High | Strong consensus anomaly | Immediate escalation | | High | Low | Point anomaly (IF speciality) | Standard investigation | | Low | High | Correlational anomaly (AE speciality) | Detailed analysis | | Low | Low | Normal behavior | No action | Consensus anomalies (both methods agree) receive highest priority and fastest response. Single-method anomalies receive investigation but with lower urgency. This consensus analysis provides a built-in confidence measure that helps governance officers prioritize their attention.
6. Threshold Calibration for Enterprise Risk Tolerance
6.1 The Threshold Selection Problem
Anomaly detection reduces to a binary decision: flag or do not flag. This decision depends on a threshold $\tau$: observations with $s_{\text{combined}}(\mathbf{x}) > \tau$ are flagged as anomalous. Threshold selection involves a tradeoff between detection recall (the fraction of true anomalies that are flagged) and false positive rate (the fraction of normal observations incorrectly flagged). In enterprise governance, both missed anomalies and false alarms carry organizational cost, but the cost structure is asymmetric: a missed runaway agent can cause millions in damage, while a false alarm costs a governance officer 15 minutes of investigation time.
The threshold selection problem is further complicated by the non-stationary nature of agentic systems. As agents learn and the organization evolves, the distribution of anomaly scores for both normal and anomalous behavior shifts. A threshold that was optimal last week may be too sensitive or too insensitive this week. This non-stationarity rules out static threshold selection and demands adaptive methods that continuously recalibrate based on observed detection performance.
Enterprise risk tolerance varies not just between organizations but between organizational units and decision types. A financial compliance unit may tolerate zero missed anomalies even at the cost of a 10% false positive rate, while a customer service unit may prefer a 1% false positive rate even at the cost of occasionally missing a low-severity anomaly. The MARIA OS threshold configuration supports per-coordinate-level risk profiles, allowing governance officers to set different thresholds for different parts of the organizational hierarchy.
6.2 Cost-Sensitive Threshold Optimization
We formalize threshold selection as a cost minimization problem: $$ \tau^* = \arg\min_\tau \left[ C_{\text{FN}} \cdot \text{FNR}(\tau) + C_{\text{FP}} \cdot \text{FPR}(\tau) \right] $$ where $C_{\text{FN}}$ is the cost of a false negative (missed anomaly), $C_{\text{FP}}$ is the cost of a false positive (false alarm), $\text{FNR}(\tau) = 1 - \text{Recall}(\tau)$ is the false negative rate, and $\text{FPR}(\tau)$ is the false positive rate. The cost ratio $C_{\text{FN}} / C_{\text{FP}}$ encodes the enterprise's risk tolerance: a ratio of 100 (missing an anomaly costs 100x more than a false alarm) produces an aggressive threshold that flags aggressively at the cost of more false alarms; a ratio of 10 produces a conservative threshold that flags selectively.
6.3 Tiered Thresholds
Rather than a single binary threshold, MARIA OS implements three tiered thresholds corresponding to the three response levels of the anomaly cascade: $$ \tau_{\text{observe}} < \tau_{\text{throttle}} < \tau_{\text{freeze}} $$ - Observe threshold ($\tau_{\text{observe}}$): Low sensitivity. Flags observations for logging and trend monitoring. Does not restrict agent behavior. - Throttle threshold ($\tau_{\text{throttle}}$): Medium sensitivity. Reduces agent autonomy by increasing governance density for the flagged agent. The agent can continue operating but with more frequent approval requirements. - Freeze threshold ($\tau_{\text{freeze}}$): High sensitivity. Immediately suspends agent operations and escalates to human governance officer. The three thresholds are calibrated independently using the cost-sensitive framework with different cost ratios reflecting the severity of each response level.
The three tiers map naturally to the MARIA OS responsibility gate framework. The observe threshold corresponds to Tier 1 gates (automated logging and monitoring). The throttle threshold corresponds to Tier 2 gates (agent-level review with increased constraints). The freeze threshold corresponds to Tier 3 gates (human-in-the-loop intervention with full operational suspension). This alignment ensures that the anomaly detection system integrates seamlessly with the broader governance architecture, using the same responsibility framework for anomaly response as for normal operational governance.
6.4 Dynamic Threshold Adaptation
Fixed thresholds become stale as the behavioral distribution evolves. We implement dynamic threshold adaptation that adjusts thresholds based on the empirical false positive rate over a sliding window. If the observed FPR exceeds the target FPR for a sustained period, the threshold is raised; if the observed FPR falls below the target, the threshold is lowered. The adaptation rate is limited to prevent oscillation: thresholds can change by at most 5% per adaptation cycle. This ensures that the safety layer maintains calibrated sensitivity despite behavioral drift, a critical requirement for long-running agentic deployments.
7. The Anomaly-Throttle-Freeze Cascade
7.1 Cascade Design
The three-stage anomaly response cascade provides proportional intervention that balances safety against operational continuity. The cascade operates as a state machine with four states: Normal, Observed, Throttled, and Frozen. Transitions between states are governed by the anomaly score and temporal persistence: $$ \text{Normal} \xrightarrow{s > \tau_{\text{observe}} \text{ for } t_{\text{obs}}} \text{Observed} \xrightarrow{s > \tau_{\text{throttle}} \text{ for } t_{\text{thr}}} \text{Throttled} \xrightarrow{s > \tau_{\text{freeze}} \text{ or critical event}} \text{Frozen} $$ Reverse transitions require explicit governance approval: a frozen agent cannot resume operations until a human governance officer reviews the anomaly and authorizes de-escalation.
The state machine design ensures that every agent's safety state is well-defined at all times and that transitions between states follow a strict protocol. The observe-to-throttle transition is particularly important because it represents the first operational intervention: the agent's behavior is being actively modified. This transition requires both sufficient evidence (sustained anomaly score above the throttle threshold) and temporal persistence (the anomaly must persist for $t_{\text{thr}}$ intervals), ensuring that throttling is not triggered by transient spikes that resolve on their own.
7.2 Throttle Mechanics
When an agent enters the Throttled state, its operational parameters are modified to reduce its potential for damage while allowing continued operation under increased oversight: 1. Governance density increase: The agent's local governance density $D_i$ is increased by $\Delta D_{\text{throttle}}$ (typically 0.15–0.25), requiring more frequent approval for its actions. 2. Autonomy scope reduction: The set of actions the agent can execute autonomously is reduced. Actions that were previously auto-approved now require agent-level or human-level review. 3. Rate limiting: The agent's maximum throughput is reduced by 50%, preventing rapid accumulation of potentially anomalous decisions. 4. Enhanced logging: All agent actions are logged with full context, including input data, intermediate computations, and output decisions, enabling detailed post-hoc forensic analysis. Throttling is designed to be reversible with minimal disruption: once the anomaly is investigated and resolved, the agent's parameters can be restored to pre-throttle levels.
7.3 Freeze Mechanics
When an agent enters the Frozen state, its operations are immediately suspended: 1. Immediate suspension: All in-flight operations are halted. Pending decisions are queued for human review rather than execution. 2. State snapshot: The agent's complete state — model parameters, context window, pending actions, communication buffers — is captured and preserved for forensic analysis. 3. Isolation: The agent is disconnected from the communication network to prevent anomalous behavior from propagating to other agents. 4. Governance escalation: An alert is sent to the human governance officer at the appropriate coordinate level with a detailed anomaly report including the anomaly score, contributing features, temporal profile, and recommended investigation steps. The freeze state is the most disruptive intervention and is reserved for situations where the anomaly score exceeds $\tau_{\text{freeze}}$ or a critical event trigger is activated (e.g., the agent attempts to modify its own governance constraints, which is always a critical event regardless of anomaly score).
The freeze state is designed with forensic analysis in mind. The complete state snapshot captured at freeze time provides investigators with a detailed record of the agent's state at the moment of maximum anomaly, enabling root cause analysis. Common root causes identified through freeze forensics include reward signal corruption (the agent's objective function was modified by an upstream data error), policy drift (the agent's learned policy gradually diverged from the intended behavior through accumulated gradient updates), and constraint evasion (the agent discovered a loophole in the governance constraint set that allowed it to take restricted actions through an indirect path). Each root cause category has a corresponding remediation procedure defined in the MARIA OS governance playbook.
7.4 Cascade Timing and Persistence Requirements
To prevent cascading on transient noise, each escalation requires temporal persistence: the anomaly score must exceed the threshold for a configurable number of consecutive measurement intervals. Default persistence requirements are: $$ t_{\text{obs}} = 3 \text{ intervals (3 minutes)} $$ $$ t_{\text{thr}} = 5 \text{ intervals (5 minutes)} $$ $$ t_{\text{freeze}} = 0 \text{ intervals (immediate for critical events)} $$ The observe-to-throttle transition requires 5 consecutive above-threshold readings to ensure that throttling is not triggered by momentary spikes. The throttle-to-freeze transition is immediate when the freeze threshold is exceeded, because by the time an agent's anomaly score reaches the freeze level, the situation is sufficiently severe that delays are unacceptable. Critical events (governance constraint modification, unauthorized communication channel creation) bypass the persistence requirement entirely and trigger immediate freeze.
8. Real-Time Streaming Architecture
8.1 Event Processing Pipeline
The safety layer operates on a streaming architecture that processes agent telemetry events in real-time. The pipeline consists of four stages: 1. Ingestion: Agent telemetry events are published to a message queue (Kafka-compatible) partitioned by agent ID, ensuring ordered processing per agent. 2. Feature extraction: A stateless feature extractor consumes events and updates the exponentially weighted moving average feature vector for each agent. 3. Anomaly scoring: The combined Isolation Forest + Autoencoder model scores each updated feature vector, producing anomaly scores and feature-level decompositions. 4. Cascade evaluation: The cascade state machine evaluates the anomaly score against the tiered thresholds and the current cascade state, producing governance actions (observe, throttle, freeze) as needed.
8.2 Latency Budget
End-to-end detection latency — the time from when an anomalous event occurs to when a governance action is triggered — is bounded by the sum of pipeline stage latencies: $$ L_{\text{total}} = L_{\text{ingest}} + L_{\text{feature}} + L_{\text{score}} + L_{\text{cascade}} $$ In our production deployments: $L_{\text{ingest}} \leq 5\text{ms}$, $L_{\text{feature}} \leq 10\text{ms}$, $L_{\text{score}} \leq 25\text{ms}$ (dominated by the Isolation Forest scoring across $T = 100$ trees), and $L_{\text{cascade}} \leq 5\text{ms}$, yielding $L_{\text{total}} \leq 45\text{ms}$. This sub-second latency ensures that the safety layer can intervene before an anomalous agent completes even a single additional decision cycle.
The streaming architecture also supports backpressure management. During periods of high telemetry volume (e.g., during organizational reorganization when many agents are simultaneously changing behavior), the scoring pipeline may fall behind the ingestion rate. Rather than dropping events, the pipeline implements adaptive batching: when the scoring queue depth exceeds a threshold, events are batched into micro-windows and scored as aggregates rather than individually. This sacrifices some temporal granularity (detecting anomalies at the batch level rather than the event level) but ensures that no telemetry data is lost. Once the queue depth returns to normal, the pipeline resumes individual event scoring.
8.3 Model Update Strategy
The Isolation Forest and Autoencoder models must be periodically retrained as the behavioral distribution evolves. We implement a dual-model architecture where the active model serves scoring requests while a shadow model is being retrained on updated data. When retraining completes, the shadow model is validated against a held-out set of known anomalies, and if it passes validation, it atomically replaces the active model. This ensures zero-downtime model updates with no gap in monitoring coverage.
The dual-model architecture also enables A/B testing of detection models. When a new model version is being evaluated, it can run as the shadow model alongside the active model, and its detection decisions can be compared against the active model's decisions and against ground truth (when available). This comparison produces model quality metrics (recall improvement, precision improvement, latency change) that inform the decision of whether to promote the shadow model to active status. The MARIA OS governance framework requires human approval for model promotions, treating the detection model itself as a governance artifact that must be governed.
8.4 Horizontal Scaling
For large organizations with thousands of agents, the scoring pipeline is horizontally scaled by partitioning agents across multiple scoring workers. Each worker maintains Isolation Forest and Autoencoder models for its partition of agents. The cascade state machine is centralized (replicated for high availability) to maintain a consistent view of organizational safety state. This architecture scales linearly with agent count: doubling the number of agents requires doubling the number of scoring workers, with no change to the cascade logic.
9. MARIA OS Stability Guard
9.1 The Stability Condition
The MARIA OS stability guard integrates anomaly detection with the broader stability framework defined by the spectral radius condition: $$ \lambda_{\max}(A_t) < 1 - D_t $$ where $A_t$ is the influence propagation matrix (whose entry $a_{ij}$ measures how much agent $j$'s behavior influences agent $i$'s behavior) and $D_t$ is the governance density. This condition ensures that influence signals decay geometrically through the organizational graph rather than amplifying, preventing cascade failures where a single anomalous agent's behavior propagates to destabilize the entire organization.
The stability condition provides a system-level safety guarantee that complements the agent-level anomaly detection. While the Isolation Forest and Autoencoder detect anomalies at the individual agent level, the spectral radius condition detects systemic instability at the organizational level. An organization can be systemically unstable even if no individual agent is anomalous: if the coupling between agents is too strong relative to the governance density, perturbations will amplify through the network regardless of whether any single agent is misbehaving. The stability guard catches this organizational-level failure mode that agent-level detection would miss.
9.2 Connecting Anomaly Detection to Spectral Stability
Anomaly detection and spectral stability are related through the influence propagation matrix $A_t$. When an agent becomes anomalous, its outgoing influence edges in $A_t$ may carry corrupted signals — wrong decisions, misleading communications, incorrect data — that affect downstream agents. If $\lambda_{\max}(A_t) < 1 - D_t$, these corrupted signals decay geometrically and the organization absorbs the anomaly without systemic damage. If $\lambda_{\max}(A_t) \geq 1 - D_t$, corrupted signals amplify and the anomaly cascades.
The stability guard monitors $\lambda_{\max}(A_t)$ in real-time (recomputed every 60 seconds using power iteration on the current influence matrix) and compares it to the safety margin $1 - D_t$. When the margin shrinks below a configurable buffer (default 0.05), the guard proactively increases governance density $D_t$ by adding constraints, effectively lowering the maximum allowed spectral radius. This preventive mechanism intervenes before the stability condition is violated, maintaining the organization in the safe operating regime.
9.3 The Stability Guard Formula
The complete stability guard formula combines anomaly detection scores with spectral stability monitoring: $$ \text{SystemSafe}(t) = \left( \max_i s_{\text{combined}}(\mathbf{x}_i(t)) < \tau_{\text{freeze}} \right) \wedge \left( \lambda_{\max}(A_t) < 1 - D_t \right) $$ The system is safe at time $t$ if and only if no individual agent's anomaly score exceeds the freeze threshold AND the spectral radius condition is satisfied. Violation of either condition triggers the corresponding response: individual anomalies trigger the agent-level cascade, while spectral instability triggers organization-wide governance density increase.
The complete stability guard formula reveals the two complementary safety mechanisms. The first clause (max anomaly score below freeze threshold) provides local safety: no individual agent is behaving dangerously. The second clause (spectral radius below stability threshold) provides global safety: the organizational structure does not amplify perturbations. Both clauses must hold simultaneously for the system to be safe. A system can satisfy local safety while violating global safety (all agents are individually normal but the coupling structure amplifies noise), or satisfy global safety while violating local safety (the organizational structure is stable but a single agent is producing dangerous outputs). Only when both clauses hold is the organization fully protected.
9.4 Governance Density as a Control Variable
The stability guard uses governance density $D_t$ as a control variable — a parameter that can be adjusted in real-time to maintain the stability condition. When $\lambda_{\max}(A_t)$ increases (due to stronger inter-agent coupling, new communication channels, or anomalous influence patterns), the guard increases $D_t$ to maintain the margin. When $\lambda_{\max}(A_t)$ decreases (due to successful anomaly resolution, organizational restructuring, or reduced coupling), the guard decreases $D_t$ to allow greater agent autonomy. The adjustment follows a proportional control law: $$ D_{t+1} = D_t + \kappa \cdot \left( \lambda_{\max}(A_t) - (1 - D_t - \delta) \right) $$ where $\kappa$ is the control gain and $\delta$ is the safety buffer. This feedback loop ensures that governance density is always just sufficient to maintain stability — not so high that it stifles agent autonomy, and not so low that it permits instability.
10. False Positive Management in Governance Contexts
10.1 The Governance Cost of False Positives
In enterprise governance, false positives are not merely nuisances — they are organizational costs. Each false alarm consumes governance officer attention, triggers unnecessary investigations, and may cause unwarranted throttling of productive agents. Over time, persistent false positives lead to alert fatigue: governance officers begin ignoring anomaly alerts, which paradoxically reduces detection effectiveness for real anomalies. Managing false positives is therefore not a secondary concern but a first-order design requirement.
Alert fatigue is particularly dangerous in agentic systems because the consequences of ignoring a real anomaly are more severe and more immediate than in traditional IT monitoring. A false-positive-fatigued governance officer who dismisses a genuine runaway agent alert may allow the agent to execute dozens or hundreds of anomalous decisions before the next review cycle. The organizational damage compounds geometrically with time (governed by the spectral radius), making every minute of delayed response increasingly costly. For this reason, false positive management is not a quality-of-life improvement but a safety-critical function.
10.2 False Positive Reduction Strategies
We implement four complementary strategies to minimize false positives: 1. Persistence requirements: As described in Section 7.4, escalation requires sustained anomaly scores, filtering out transient spikes. 2. Contextual scoring: Anomaly scores are adjusted for known contextual factors (time of day, day of week, organizational events). An agent processing month-end reports at higher-than-usual volume is not anomalous in context. 3. Allowlist patterns: Governance officers can define allowlisted behavioral patterns that should never trigger anomaly alerts, based on known operational requirements. These patterns are encoded as exceptions in the scoring pipeline. 4. Feedback learning: Governance officers can mark false positives in the investigation interface. These labels are used to periodically retrain the detection models with augmented training data that includes the false-positive examples as normal observations. Across our four deployments, these strategies collectively reduced the raw false positive rate from 8.7% to 2.1% while maintaining detection recall above 98%.
The feedback learning strategy is particularly effective because it addresses the non-stationary nature of the anomaly detection problem. As the organization evolves, new types of normal behavior emerge that may initially trigger false positives. Governance officer labels on these false positives provide direct supervision to the detection models, teaching them to recognize the new normal behavior. Over time, the models accumulate organizational knowledge about what is truly anomalous versus merely novel, reducing the false positive rate without sacrificing detection recall.
The four strategies are ordered by their implementation complexity and their impact on false positive rates. Persistence requirements (strategy 1) provide the largest single reduction in false positives because most transient spikes are benign. Contextual scoring (strategy 2) provides the next largest reduction by eliminating known-context false positives. Allowlist patterns (strategy 3) and feedback learning (strategy 4) provide targeted reductions for specific recurring false positive patterns that are not captured by the first two strategies.
10.3 The False Positive-False Negative Tradeoff Under Governance
The receiver operating characteristic (ROC) curve describes the tradeoff between true positive rate and false positive rate as the detection threshold varies. In standard anomaly detection, the operating point is chosen to maximize some performance metric (e.g., F1 score or area under the curve). In governance contexts, the operating point must account for asymmetric costs and organizational constraints. We define the governance-adjusted ROC that weights each point on the curve by the organizational cost: $$ \text{GovCost}(\tau) = C_{\text{FN}} \cdot (1 - \text{TPR}(\tau)) \cdot P_{\text{anomaly}} + C_{\text{FP}} \cdot \text{FPR}(\tau) \cdot (1 - P_{\text{anomaly}}) $$ where $P_{\text{anomaly}}$ is the base rate of anomalies. The optimal threshold minimizes GovCost, which in practice produces a more aggressive threshold than standard F1 optimization because $C_{\text{FN}} \gg C_{\text{FP}}$ in governance contexts.
11. Experimental Results
11.1 Deployment Configurations
We evaluated the safety layer across the same four enterprise deployments used in previous studies: | Deployment | Agents | Duration | Injected Anomalies | Anomaly Types | |---|---|---|---|---| | FinCorp-Alpha | 100 | 90 days | 47 | Fraud, policy violation, resource abuse | | RetailNet-Beta | 200 | 60 days | 83 | Inventory manipulation, pricing anomaly, data leak | | HealthOrg-Gamma | 350 | 45 days | 112 | Compliance violation, unauthorized access, dosage error | | TechScale-Delta | 500 | 30 days | 156 | Code injection, privilege escalation, runaway process |
11.2 Detection Performance
The combined Isolation Forest + Autoencoder ensemble achieved the following performance across all deployments: | Method | Recall | Precision | F1 | FPR | AUC-ROC | |---|---|---|---|---|---| | Isolation Forest only | 93.7% | 91.2% | 92.4% | 3.8% | 0.971 | | Autoencoder only | 95.1% | 89.6% | 92.3% | 4.5% | 0.968 | | Combined ensemble | 98.3% | 95.4% | 96.8% | 2.1% | 0.994 | The ensemble achieves higher recall than either method alone (capturing anomalies that one method misses) while also achieving lower FPR (the consensus mechanism filters false positives). The AUC-ROC of 0.994 indicates near-perfect discrimination between normal and anomalous behavior.
The performance improvement from the ensemble method is attributable to the complementary nature of the two detection algorithms. Isolation Forest detected 89% of point anomalies at high confidence but only 78% of collective anomalies. The Autoencoder detected 92% of collective anomalies but only 84% of point anomalies. The ensemble captured the union of both methods' strengths, achieving high recall across all anomaly categories. Notably, the ensemble's precision was also higher than either individual method, because the consensus requirement filters out many false positives that only one method detects.
11.3 Detection Latency
End-to-end detection latency (from anomaly onset to governance action) was measured across all injected anomalies: | Percentile | Latency | |---|---| | p50 | 12 seconds | | p90 | 31 seconds | | p95 | 38 seconds | | p99 | 44 seconds | | Max | 52 seconds | The p95 latency of 38 seconds means that 95% of anomalies are detected and trigger governance action within 38 seconds of onset. The maximum latency of 52 seconds occurred for a collective anomaly (gradual threshold drift) that required multiple observation intervals to accumulate sufficient anomaly score.
11.4 Cascade Effectiveness
The three-stage cascade prevented all injected anomalies from causing simulated organizational damage (defined as propagation beyond the immediate agent's operational scope). Of 398 total injected anomalies across all deployments: 312 (78.4%) were caught at the Throttle stage; 71 (17.8%) escalated to Freeze; 15 (3.8%) were caught at the Observe stage and resolved by governance officers before escalation. No injected anomaly propagated beyond the affected agent's zone boundary, validating the cascade's containment effectiveness.
12. Adversarial Robustness and Limitations
The zero-propagation result is the most important safety validation. In all 398 injected anomalies across four deployments spanning 225 days of operation, not a single anomaly propagated beyond the affected agent's zone boundary. This containment guarantee is the product of two complementary mechanisms: the detection speed (sub-45 seconds) limits the time window for propagation, and the cascade response (throttle/freeze) limits the agent's ability to propagate even during the detection window. The combination provides defense in depth that is robust to partial failure of either mechanism.
The cascade effectiveness results also validate the persistence timing parameters. The default persistence requirements ($t_{\text{obs}} = 3$ intervals, $t_{\text{thr}} = 5$ intervals) proved well-calibrated across all four deployments: they prevented false positive cascading (no normal agents were throttled or frozen incorrectly) while ensuring that genuine anomalies were detected and contained before causing organizational damage. The 3-minute observe persistence and 5-minute throttle persistence represent the optimal tradeoff between detection speed and false positive prevention for the enterprise environments tested.
12.1 Adversarial Evasion
A sophisticated adversarial agent could potentially evade anomaly detection by modifying its behavior gradually, staying just below the detection threshold while accumulating anomalous effects over time. This 'boiling frog' attack is the most challenging scenario for any anomaly detection system. We address it through three mechanisms. First, the autoencoder's reconstruction error is sensitive to gradual distributional shift because the model periodically retrains, and the gradual shift accumulates in the training data, raising the baseline reconstruction error for the drifting agent. Second, the collective anomaly detection using sequential analysis methods (CUSUM, EWMA control charts) applied to the time series of anomaly scores can detect trends that individual-snapshot detection misses. Third, periodic full re-evaluation compares each agent's current behavior against the original baseline from its deployment date, detecting cumulative drift regardless of rate.
12.2 Model Poisoning
If an adversarial agent can influence the training data for the anomaly detection models (e.g., by generating carefully crafted 'normal' telemetry that expands the learned normal distribution to include anomalous patterns), the detection system can be degraded. We mitigate this risk by maintaining a golden reference dataset of verified normal behavior that is included in every retraining cycle, by applying robust statistics (trimmed means, median-based scaling) that are resistant to outlier contamination, and by requiring governance approval for any model update that changes the detection boundary by more than 5%.
Model poisoning is particularly concerning in the agentic context because agents have the capability to generate their own telemetry. An adversarial agent that controls its own feature reporting could submit false telemetry that gradually expands the learned normal distribution to include its actual anomalous behavior. We address this risk by cross-validating agent self-reported telemetry against external observations: communication partners can independently report their interactions with an agent, and system-level metrics (resource consumption, network traffic) provide independent measurements that are harder to falsify. Discrepancies between self-reported and externally observed behavior are themselves treated as anomalies and flagged for investigation.
12.3 Fundamental Limitations
Anomaly detection is fundamentally limited by the quality and representativeness of the normal behavior model. If the training data does not capture the full range of legitimate normal behavior, the system will produce false positives on normal-but-unseen behaviors. Conversely, if the training data is contaminated with undetected anomalies, the system will learn to treat those anomalies as normal. These limitations cannot be fully eliminated by algorithmic improvements; they require ongoing governance oversight to validate detection quality and update the normal behavior model as the organization evolves. The MARIA OS stability guard acknowledges these limitations by positioning anomaly detection as one component of a defense-in-depth strategy that also includes rule-based constraints, spectral stability monitoring, and human governance review.
13. Conclusion
Anomaly detection is not an optional feature for agentic companies — it is an existential requirement. Without a computational safety layer that can detect behavioral deviations in real-time and trigger proportional responses, autonomous agent organizations will inevitably experience cascading failures. This paper has presented the Safety Layer (Layer 7) of the agentic algorithm stack, built on the complementary strengths of Isolation Forest (tree-based anomaly scoring without a learned normal model) and Autoencoders (manifold-based anomaly scoring through reconstruction error). The combined ensemble achieves 98.3% detection recall with 2.1% false positive rate and sub-45-second detection latency. The anomaly-throttle-freeze cascade provides proportional intervention that balances safety against operational continuity. The MARIA OS stability guard integrates agent-level anomaly detection with organization-level spectral stability monitoring through the formula $\lambda_{\max}(A_t) < 1 - D_t$, using governance density as a real-time control variable to maintain the stability condition. Enterprise anomaly detection is not a one-time deployment but an ongoing process of model adaptation, threshold calibration, false positive management, and governance validation — a process that MARIA OS automates within its responsibility gate framework while preserving human authority over critical safety decisions.