Actor-Critic Reinforcement Learning for Gated Autonomy: PPO-Based Policy Optimization Under Responsibility Constraints

1. Introduction: The Control Layer Problem

Enterprise AI systems face a fundamental tension: automation demands speed and consistency, while governance demands caution and accountability. Rule-based systems resolve this tension by hard-coding every decision boundary, but they cannot adapt to novel situations. Unconstrained reinforcement learning resolves it by learning optimal policies from experience, but it cannot guarantee that learned behaviors respect governance requirements. The Control Layer of the agentic company architecture requires a middle path — reinforcement learning that is powerful enough to handle complex sequential decision-making yet constrained enough to respect responsibility boundaries at every step.

The specific challenge is medium-risk task automation. Low-risk tasks (sending status updates, formatting reports, scheduling meetings) can be fully automated with deterministic rules. High-risk tasks (approving large financial transactions, modifying production systems, hiring decisions) require mandatory human approval. But the vast middle ground — procurement decisions under $50K, customer escalation routing, inventory rebalancing, code deployment to staging environments — represents 60-75% of enterprise operational volume. These tasks are too numerous for full human review and too consequential for blind automation.

Actor-critic reinforcement learning, specifically Proximal Policy Optimization (PPO), provides the algorithmic foundation for this middle ground. The actor network learns a stochastic policy that maps states to action distributions. The critic network learns a value function that estimates expected future reward. Together, they enable policy gradient optimization with variance reduction. But standard PPO makes no accommodation for dynamic action space constraints, multi-stakeholder reward structures, or human-in-the-loop environment dynamics.

This paper extends PPO to the gated autonomy setting. We formalize the Gate-Constrained MDP, derive a modified policy gradient theorem, prove stability guarantees under PPO clipping, and demonstrate the framework in MARIA OS enterprise deployments.

1.1 Why Actor-Critic for Enterprise Control

The choice of actor-critic methods over alternative RL approaches is driven by three enterprise requirements. First, continuous action spaces: enterprise decisions often involve continuous parameters (budget allocations, priority scores, resource quantities) that are poorly handled by value-based methods like Q-learning, which require discretization. Actor-critic methods naturally handle continuous actions through parameterized policy networks. Second, sample efficiency: enterprise environments are expensive to simulate and impossible to reset — a procurement decision, once made, cannot be undone for training purposes. Actor-critic methods, especially PPO, achieve higher sample efficiency than pure policy gradient methods by reusing trajectories through importance sampling. Third, stability: enterprise policy updates must be conservative. A sudden shift in how an AI agent handles customer escalations could cascade through the organization. PPO's clipped objective provides a formal mechanism for bounding policy changes between updates.

1.2 Paper Organization

Section 2 formalizes the Gate-Constrained MDP. Section 3 develops the actor-critic architecture for gated autonomy. Section 4 derives the gate-constrained policy gradient theorem. Section 5 presents PPO adaptation with governance trust regions. Section 6 introduces multi-stakeholder reward shaping. Section 7 models human-in-the-loop approval as environment dynamics. Section 8 addresses the gate engine as constraint enforcer. Section 9 presents the MARIA OS integration. Section 10 provides experimental validation. Section 11 discusses convergence properties. Section 12 concludes.

2. The Gate-Constrained Markov Decision Process

Standard reinforcement learning operates on a Markov Decision Process defined by the tuple (S, A, T, R, gamma), where S is the state space, A is the action space, T: S x A x S -> [0,1] is the transition function, R: S x A -> R is the reward function, and gamma in (0,1) is the discount factor. In the standard formulation, the action space A is fixed — every action is available in every state. This assumption is fundamentally incompatible with governed enterprise environments where the available actions depend on the agent's authorization level, the task's risk classification, and the current state of responsibility gates.

2.1 Formal Definition

We define a Gate-Constrained MDP (GC-MDP) as the tuple (S, A, G, T, R, gamma, C) where the additional elements are: - G = {g_1, g_2, ..., g_K} is a finite set of responsibility gates - C: S x G -> 2^A is the constraint function that maps each state-gate pair to the subset of admissible actions The effective action space at state s is the intersection of all gate constraints: $$ A_{\text{eff}}(s) = \bigcap_{k=1}^{K} C(s, g_k) $$ This intersection ensures that an action is admissible only if it passes ALL active gates simultaneously. A single gate veto is sufficient to exclude an action.

The constraint function C encodes three types of restrictions that appear in enterprise governance: | Constraint Type | Formal Expression | Enterprise Example | |---|---|---| | Hard exclusion | C(s, g_k) = A \ {a_blocked} | Financial transactions above $100K excluded from auto-approve | | Conditional inclusion | C(s, g_k) = {a : phi_k(s, a) >= tau_k} | Actions permitted only if evidence quality exceeds threshold | | Rate limiting | C(s, g_k) = {a : count(a, H_t) < n_max} | Maximum 5 auto-approvals per hour for procurement | where phi_k is a gate-specific scoring function, tau_k is a gate threshold, H_t is the action history up to time t, and n_max is the rate limit.

2.2 State Representation for Enterprise Environments

The state in a GC-MDP must encode not only the task-level information but also the governance context. We define the state as a composite vector: $$ s_t = [s_t^{\text{task}}, s_t^{\text{trust}}, s_t^{\text{gate}}, s_t^{\text{history}}] $$ where s_t^task contains task-specific features (e.g., procurement amount, vendor risk score, urgency level), s_t^trust is the agent's current trust vector across all gate dimensions, s_t^gate is the binary vector of gate activation states, and s_t^history summarizes relevant action history. In MARIA OS, the trust vector is maintained by the gate engine and updated after every action based on outcome feedback.

2.3 Transition Dynamics with Gate Interactions

The transition function in a GC-MDP has a critical property: gate decisions affect transitions. When an agent requests an action that requires gate approval, the environment transitions to an intermediate waiting state s_wait where the outcome depends on the gate's (possibly human) decision. We model this as: $$ T(s' | s, a) = \begin{cases} T_{\text{direct}}(s' | s, a) & \text{if } a \in A_{\text{auto}}(s) \\ \sum_{d \in \{\text{approve, reject}\}} P(d | s, a) \cdot T_{\text{gated}}(s' | s, a, d) & \text{if } a \in A_{\text{gated}}(s) \end{cases} $$ where A_auto(s) is the set of auto-executable actions and A_gated(s) is the set requiring approval. The probability P(d | s, a) captures the gate's (or human reviewer's) decision distribution, which the agent must learn to predict.

3. Actor-Critic Architecture for Gated Autonomy

The actor-critic framework decomposes the learning problem into two cooperating networks: the actor (policy network) that selects actions, and the critic (value network) that evaluates states. In the gated autonomy setting, both networks must be aware of gate constraints.

3.1 The Actor Network: Gate-Aware Policy

The actor network pi_theta parameterizes a stochastic policy over the full action space A, but its outputs are masked by the gate constraint function before action selection. For discrete action spaces, we use a masked softmax: $$ \pi_\theta(a | s) = \frac{\exp(f_\theta(s, a)) \cdot \mathbb{1}[a \in A_{\text{eff}}(s)]}{\sum_{a' \in A} \exp(f_\theta(s, a')) \cdot \mathbb{1}[a' \in A_{\text{eff}}(s)]} $$ where f_theta(s, a) is the actor network's raw logit for action a in state s, and the indicator function zeroes out inadmissible actions. For continuous action spaces, the actor outputs parameters of a truncated Gaussian distribution whose support is restricted to A_eff(s).

The key architectural insight is that the actor learns over the full action space but is constrained at inference time. This means the network can learn representations that span the complete action space — understanding why certain actions are blocked and how to select optimally within the admissible subset. If we trained only on the constrained space, the actor would lack the representational capacity to adapt when gate constraints change (as they do when an agent's trust level evolves).

3.2 The Critic Network: Governance-Aware Value Estimation

The critic network V_phi(s) estimates the expected return from state s under the current policy. In a GC-MDP, the critic must account for the fact that value depends on gate constraints: $$ V^\pi(s) = \mathbb{E}_{\pi} \left[ \sum_{t=0}^{\infty} \gamma^t r_{t+1} \mid s_0 = s, \forall t: a_t \in A_{\text{eff}}(s_t) \right] $$ The gate constraints affect value through two channels: they restrict which actions the agent can take (reducing value when high-reward actions are blocked) and they introduce approval delays (discounting future rewards through waiting time). We train the critic with a modified temporal difference target: $$ y_t = r_t + \gamma \cdot (1 - d_t) \cdot V_\phi(s_{t+1}) + \gamma \cdot d_t \cdot \delta_{\text{wait}} \cdot V_\phi(s_{t+1}) $$ where d_t is an indicator for whether the action required gate approval, and delta_wait in (0,1) is a discount factor for approval delay.

3.3 Advantage Estimation with Gate Corrections

The advantage function A^pi(s, a) = Q^pi(s, a) - V^pi(s) measures how much better action a is compared to the average action under policy pi. In gated autonomy, we use Generalized Advantage Estimation (GAE) with a gate correction term: $$ \hat{A}_t^{\text{GAE}(\gamma, \lambda)} = \sum_{l=0}^{\infty} (\gamma \lambda)^l \delta_{t+l} + \alpha_g \cdot \text{GateBonus}(s_t, a_t) $$ where delta_t = r_t + gamma V_phi(s_{t+1}) - V_phi(s_t) is the TD residual, lambda is the GAE parameter, and GateBonus(s_t, a_t) provides a small positive reward for actions that proactively respect gate boundaries without requiring enforcement. The coefficient alpha_g controls the strength of gate-compliance incentivization.

4. The Gate-Constrained Policy Gradient Theorem

The standard policy gradient theorem states that the gradient of the expected return J(theta) with respect to policy parameters theta is: $$ \nabla_\theta J(\theta) = \mathbb{E}_{\pi_\theta} \left[ \nabla_\theta \log \pi_\theta(a|s) \cdot A^{\pi_\theta}(s, a) \right] $$ This result assumes a fixed action space. When gate constraints dynamically restrict the action space, the gradient requires modification.

4.1 Theorem Statement

Theorem 1 (Gate-Constrained Policy Gradient). Let (S, A, G, T, R, gamma, C) be a GC-MDP and let pi_theta be a gate-masked policy. The gradient of the expected return decomposes as: $$ \nabla_\theta J(\theta) = \underbrace{\mathbb{E}_{s \sim d^\pi, a \sim \pi_\theta(\cdot|s)} \left[ \nabla_\theta \log \pi_\theta(a|s) \cdot A^{\pi}(s,a) \right]}_{\text{Interior gradient}} + \underbrace{\mathbb{E}_{s \sim d^\pi} \left[ \sum_{a \in \partial A_{\text{eff}}(s)} \nabla_\theta \pi_\theta(a|s) \cdot \Delta Q(s, a) \right]}_{\text{Boundary correction}} $$ where d^pi is the discounted state visitation distribution, the partial symbol denotes the boundary of the effective action space (actions that are marginally admissible), and Delta Q(s, a) is the value difference between executing action a and its nearest admissible alternative.

4.2 Proof Sketch

The proof proceeds by partitioning the action space into interior actions (strictly within A_eff), boundary actions (on the edge of admissibility), and exterior actions (blocked by gates). For interior actions, the standard policy gradient applies directly because the gate masking function is locally constant. For boundary actions, the gate constraint introduces a discontinuity in the policy — small changes in theta can cause an action to cross the admissibility threshold, producing a discrete jump in the masked policy probability. The boundary correction term captures the gradient contribution from these threshold crossings. Formally, let M(s, a) = 1[a in A_eff(s)] be the gate mask. The masked policy is pi_theta^M(a|s) = pi_theta(a|s) M(s,a) / Z(s) where Z(s) = sum_{a'} pi_theta(a'|s) M(s,a') is the normalization constant. Taking the gradient of J(theta) = E_s[sum_a pi_theta^M(a|s) Q^pi(s,a)] and applying the product rule to the masked policy yields the two-term decomposition. The boundary correction vanishes when gate constraints are either very loose (A_eff = A, no constraints) or very tight (A_eff is a singleton, no choice), and is maximally significant when the gate boundary passes through regions of high policy probability.

4.3 Practical Implications

The boundary correction term has a crucial practical implication: it encourages the policy to learn the gate boundaries. When the actor assigns high probability to an action near the admissibility threshold, the boundary correction provides gradient signal proportional to the value difference between executing that action and its constrained alternative. Over training, this drives the policy to either strongly prefer the action (if it is valuable enough to warrant gate approval requests) or shift probability mass away from the boundary (if the constrained alternative is nearly as good). In enterprise terms, the agent learns when it is worth requesting human approval versus accepting a slightly suboptimal but auto-executable action.

5. PPO with Governance Trust Regions

Proximal Policy Optimization constrains policy updates to prevent destructive large steps. The standard PPO clipped objective is: $$ L^{\text{CLIP}}(\theta) = \mathbb{E}_t \left[ \min\left( r_t(\theta) \hat{A}_t, \; \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon) \hat{A}_t \right) \right] $$ where r_t(theta) = pi_theta(a_t | s_t) / pi_{theta_old}(a_t | s_t) is the probability ratio and epsilon is the clipping parameter. In governance-bounded settings, we need tighter control.

5.1 Adaptive Clipping for Risk-Tiered Actions

Not all actions carry the same governance risk. A procurement approval for $500 has different risk characteristics than one for $45,000. We introduce risk-adaptive clipping where the clipping parameter depends on the action's risk tier: $$ \epsilon(a_t) = \epsilon_{\text{base}} \cdot \exp(-\beta \cdot \text{risk}(s_t, a_t)) $$ where risk(s_t, a_t) in [0, 1] is the gate engine's risk assessment and beta > 0 controls sensitivity. High-risk actions get tighter clipping (smaller epsilon), preventing the policy from making large probability shifts on consequential decisions. Low-risk actions get standard clipping, allowing faster adaptation.

5.2 Trust Region Interpretation

The clipped objective implicitly defines a trust region in policy space. We make this explicit by deriving the effective trust region radius under gate-adaptive clipping. Proposition 1. Under risk-adaptive clipping with parameter epsilon(a), the effective KL divergence bound between consecutive policies satisfies: $$ D_{\text{KL}}(\pi_{\theta_{\text{old}}} \| \pi_\theta) \leq \frac{\epsilon_{\text{base}}^2}{2} \cdot \mathbb{E}_{a \sim \pi_{\theta_{\text{old}}}} \left[ \exp(-2\beta \cdot \text{risk}(s, a)) \right] $$ This bound shows that the trust region contracts as the average action risk increases. In a governance context, this means policy updates become more conservative precisely when the agent is operating in higher-risk territory — a desirable property that emerges naturally from the risk-adaptive clipping design.

5.3 Multi-Epoch Training with Gate Consistency Checks

Standard PPO performs multiple epochs of gradient updates on the same batch of trajectories, improving sample efficiency. In the gated autonomy setting, gate constraints can change between collection and training (e.g., an agent's trust level is updated based on the latest batch outcomes). We add a gate consistency check: before each training epoch, we verify that the actions in the replay buffer are still admissible under current gate constraints. Actions that have become inadmissible are either removed from the batch or assigned a penalty reward to drive the policy away from them: $$ r_t^{\text{corrected}} = \begin{cases} r_t & \text{if } a_t \in A_{\text{eff}}^{\text{current}}(s_t) \\ r_t - \lambda_{\text{penalty}} & \text{if } a_t \notin A_{\text{eff}}^{\text{current}}(s_t) \end{cases} $$ This ensures the policy learns from the most recent gate configuration, even when training on historical trajectories.

6. Multi-Stakeholder Reward Shaping

Enterprise reinforcement learning serves multiple stakeholders with potentially conflicting objectives. The operations team wants maximum task completion rate. The compliance team wants zero governance violations. The human oversight team wants minimal unnecessary escalations. We formalize this as a multi-objective reward function.

6.1 Composite Reward Function

The reward at each timestep is a weighted combination of stakeholder-specific components: $$ r_t = w_{\text{task}} \cdot r_t^{\text{task}} + w_{\text{comply}} \cdot r_t^{\text{comply}} + w_{\text{oversight}} \cdot r_t^{\text{oversight}} + w_{\text{evidence}} \cdot r_t^{\text{evidence}} $$ where: - r_t^task is the task completion reward (positive for successful outcomes, negative for failures) - r_t^comply is the compliance reward (positive for gate-compliant actions, strongly negative for violations) - r_t^oversight is the oversight efficiency reward (negative for unnecessary human escalations, positive for correct self-handling) - r_t^evidence is the evidence quality reward (positive for well-documented decisions, negative for undocumented actions) The weights w_* are set by organizational policy and can vary across departments, risk tiers, and operational contexts.

6.2 Potential-Based Reward Shaping

To accelerate learning without altering the optimal policy, we apply potential-based reward shaping following Ng et al. (1999). We define a potential function Phi(s) that encodes domain knowledge about state desirability: $$ \Phi(s) = \alpha_1 \cdot \text{TrustLevel}(s) + \alpha_2 \cdot \text{EvidenceCompleteness}(s) + \alpha_3 \cdot \text{QueueHealth}(s) $$ The shaped reward is F(s, s') = gamma * Phi(s') - Phi(s). This provides immediate feedback for actions that improve the agent's governance state (building trust, gathering evidence, reducing queue backlogs) without changing the optimal policy under the original reward — a property guaranteed by the potential-based shaping theorem.

6.3 Constraint Satisfaction via Lagrangian Relaxation

Hard governance constraints (e.g., 'never auto-approve transactions above $100K') are enforced by the gate constraint function. Soft constraints (e.g., 'keep human escalation rate below 15%') are better handled through constrained optimization. We formulate the constrained RL problem: $$ \max_\theta J(\theta) \quad \text{subject to} \quad \mathbb{E}_{\pi_\theta}[c_i(s, a)] \leq d_i \quad \forall i \in \{1, ..., m\} $$ where c_i are constraint cost functions and d_i are constraint thresholds. We solve this via Lagrangian relaxation, introducing dual variables mu_i >= 0: $$ L(\theta, \mu) = J(\theta) - \sum_{i=1}^{m} \mu_i \left( \mathbb{E}_{\pi_\theta}[c_i(s, a)] - d_i \right) $$ The primal variables theta are updated via PPO on the Lagrangian objective, and the dual variables mu are updated via gradient ascent. This produces a policy that maximizes task performance while satisfying governance constraints in expectation.

7. Human-in-the-Loop RL: Approval as Environment Dynamics

A distinctive feature of gated autonomy is that human approval decisions are part of the environment, not external interruptions. When an agent submits an action for gate approval, the human reviewer's decision (approve, reject, modify) is a stochastic environment response that the agent must learn to predict and optimize around.

7.1 The Approval MDP Augmentation

We augment the GC-MDP with approval states. When the agent selects an action a in A_gated(s), the environment transitions to an intermediate state s_pending where the agent waits for a human decision. The human decision d is drawn from a distribution P_H(d | s, a, h) where h represents the reviewer's observable characteristics (workload, expertise, historical approval patterns). The transition then proceeds based on d: $$ s_{t+1} = \begin{cases} T_{\text{approve}}(s_t, a_t) & \text{if } d = \text{approve} \\ T_{\text{reject}}(s_t, a_t) & \text{if } d = \text{reject} \\ T_{\text{modify}}(s_t, a_t, a_t') & \text{if } d = \text{modify with } a_t' \end{cases} $$

7.2 Learning the Human Approval Model

The agent maintains an internal model P_hat_H(d | s, a) of the human approval distribution. This model is trained on historical approval data and updated online as new decisions are observed. The model enables the agent to compute expected values for gated actions: $$ Q^\pi(s, a_{\text{gated}}) = P_{\hat{H}}(\text{approve} | s, a) \cdot Q^\pi_{\text{approve}}(s, a) + P_{\hat{H}}(\text{reject} | s, a) \cdot Q^\pi_{\text{reject}}(s, a) - c_{\text{wait}} $$ where c_wait is the opportunity cost of waiting for approval. The agent thus learns to balance the potential reward of a gated action against the probability of rejection and the cost of waiting. This naturally leads to behavior where the agent requests approval only when the expected value of the gated action significantly exceeds the best auto-executable alternative.

7.3 Reviewer Workload-Aware Scheduling

The agent can observe (or estimate) the reviewer's current workload and adjust its gate-request behavior accordingly. We model the reviewer's response time as a function of workload: $$ \tau_{\text{response}}(w) = \tau_0 \cdot (1 + \kappa \cdot w^2) $$ where w is the reviewer's current queue depth, tau_0 is the base response time, and kappa captures the non-linear degradation of response time under load. The agent incorporates this into its action-value computation by adjusting the discount for gated actions: $$ Q^\pi_{\text{gated}}(s, a) = \gamma^{\tau_{\text{response}}(w)} \cdot \mathbb{E}_d[Q^\pi(s', a)] $$ This creates a natural load-balancing effect: when reviewers are overloaded, gated actions are discounted more heavily, and the agent prefers auto-executable alternatives. When reviewers are idle, the discount is minimal, and the agent is more willing to request approval for potentially higher-reward actions.

8. The Gate Engine as Constraint Enforcer

In MARIA OS, the gate engine is the infrastructure component that implements the constraint function C(s, g_k). It sits between the actor's proposed action and the environment, enforcing governance constraints in real time.

8.1 Gate Engine Architecture

The gate engine operates as a middleware layer with the following pipeline: 1. Risk Assessment: Compute risk(s, a) using the multi-dimensional risk scoring model 2. Gate Selection: Determine which gates g_k are active for the current state-action pair 3. Constraint Evaluation: For each active gate, evaluate C(s, g_k) to determine if the action is admissible 4. Action Routing: Route the action to auto-execute, agent-review, or human-approval based on the gate evaluation 5. Evidence Collection: Record the gate evaluation, action, and outcome for audit and training purposes The gate engine maintains a gate state vector that tracks each gate's configuration, thresholds, and recent evaluation history. This state is included in the RL agent's observation, enabling the policy to learn gate-aware behavior.

8.2 Dynamic Gate Threshold Adaptation

Gate thresholds are not static. They adapt based on organizational performance metrics using an exponential moving average of outcome quality: $$ \tau_k^{(t+1)} = (1 - \eta) \cdot \tau_k^{(t)} + \eta \cdot \bar{q}_k^{(t)} $$ where tau_k^(t) is gate k's threshold at time t, eta is the adaptation rate, and q_bar_k^(t) is the average quality score of decisions that passed gate k in the recent window. If quality degrades (more errors in auto-approved decisions), thresholds tighten, requiring human approval for more actions. If quality is consistently high, thresholds relax, granting the agent more autonomy. This creates a feedback loop between the RL agent's competence and its operational freedom — a formal instantiation of the 'graduated autonomy' principle.

8.3 Gate Interaction Effects

When multiple gates are active simultaneously, their constraints interact. The effective action space is the intersection of all gate constraints, which can be significantly smaller than any individual gate's constraint set. We define the gate interaction coefficient: $$ \iota_G(s) = 1 - \frac{|A_{\text{eff}}(s)|}{|A|} = 1 - \frac{|\bigcap_k C(s, g_k)|}{|A|} $$ when iota_G(s) is close to 1, the gates are highly restrictive (the agent has very few admissible actions). When close to 0, the gates are permissive. We monitor this coefficient across states to detect over-constraining configurations where gate interactions reduce the agent's action space to the point of operational paralysis.

9. MARIA OS Integration Architecture

The gated autonomy RL framework integrates with MARIA OS through three interface layers: the coordinate system for agent addressing, the decision pipeline for state management, and the evidence system for audit trails.

9.1 MARIA Coordinate Mapping

Each RL agent in the system is assigned a MARIA coordinate (G.U.P.Z.A) that determines its governance context. The coordinate maps to specific gate configurations: $$ \text{GateConfig}(G_i.U_j.P_k.Z_l.A_m) = \bigcup_{\text{level} \in \{G, U, P, Z\}} \text{Gates}(\text{level}) $$ Galaxy-level gates enforce tenant-wide policies (data residency, compliance frameworks). Universe-level gates enforce business unit policies (budget authority, approval chains). Planet-level gates enforce domain policies (risk thresholds, evidence requirements). Zone-level gates enforce operational policies (rate limits, workload caps). The agent inherits constraints from all ancestor levels, creating a hierarchical constraint structure that the RL policy must navigate.

9.2 Decision Pipeline Integration

The RL agent's actions map to MARIA OS decision pipeline states. When the actor selects an action, it is translated into a decision proposal that enters the pipeline at the 'proposed' state. Gate evaluations correspond to pipeline transitions: | RL Action Type | Pipeline Transition | Gate Requirement | |---|---|---| | Auto-execute | proposed -> validated -> approved -> executed | None (within trust boundary) | | Agent-review | proposed -> validated -> approval_required -> approved | Peer agent review | | Human-approval | proposed -> validated -> approval_required -> [human decision] | Human reviewer approval | Every transition creates an immutable audit record in the decision_transitions table, ensuring complete traceability of the RL agent's decision-making process.

9.3 Evidence Bundle Generation

For each gated action, the RL agent generates an evidence bundle that justifies the action to the reviewer. The bundle includes the state representation, the policy's action probabilities, the critic's value estimate, the advantage score, and a natural-language explanation generated by the Cognition Layer's transformer. This evidence bundle is stored in the evidence system and linked to the decision record: $$ E(s, a) = \{s_{\text{features}}, \pi_\theta(\cdot|s), V_\phi(s), \hat{A}(s,a), \text{NL}_{\text{explain}}(s, a)\} $$ The evidence completeness score r_t^evidence in the reward function incentivizes the agent to generate high-quality evidence bundles, creating a self-reinforcing loop where better evidence leads to higher approval rates, which leads to higher reward, which leads to even better evidence.

10. Experimental Validation

We evaluate the gated autonomy PPO framework across four enterprise deployment scenarios within MARIA OS, comparing against three baselines: unconstrained PPO, rule-based automation, and random policy with gate constraints.

10.1 Experimental Setup

The four deployment scenarios are: 1. Procurement Automation (Sales Universe G1.U1): 12 agents handling purchase orders from $100 to $100K, with 3-tier gate structure (auto < $5K, agent-review $5K-$25K, human-approval > $25K) 2. Customer Escalation Routing (FAQ Universe G1.U3): 8 agents routing customer complaints across severity levels, with risk-based gates keyed to customer lifetime value 3. Code Deployment Pipeline (Auto-Dev Universe G1.U4): 6 agents managing staging and production deployments, with gates based on test coverage, change magnitude, and blast radius 4. Audit Evidence Collection (Audit Universe G1.U2): 10 agents gathering and verifying audit evidence, with gates based on evidence completeness and materiality thresholds Each scenario runs for 52,000 episodes (approximately 30 simulated operational days) with PPO hyperparameters: learning rate 3e-4, gamma = 0.99, lambda = 0.95, epsilon_base = 0.2, beta = 2.0, batch size 2048, 10 epochs per update.

10.2 Results Summary

10.3 Convergence Analysis

The training curves show three distinct phases. In the exploration phase (episodes 0-8,000), the agent frequently triggers gate rejections as it explores the action space, resulting in low reward and high escalation rate. In the boundary learning phase (episodes 8,000-25,000), the agent learns the gate boundaries and begins routing actions appropriately, with a sharp decrease in unnecessary escalations. In the optimization phase (episodes 25,000-52,000), the agent fine-tunes its policy within the learned gate structure, gradually improving task completion while maintaining compliance. The policy stabilizes around episode 40,000 with KL divergence between updates falling below 0.008.

10.4 Ablation Studies

We conduct ablation studies on three key components: 1. Risk-adaptive clipping: Removing risk-adaptive clipping (using fixed epsilon = 0.2 for all actions) increases policy instability by 3.2x on high-risk actions, with 4 gate compliance violations in 52K episodes vs 0 with adaptive clipping. 2. Multi-stakeholder reward: Using task-only reward (w_task = 1, all others 0) achieves 96.1% completion but only 91.2% gate compliance, as the agent learns to 'game' gates rather than respect them. 3. Human approval model: Removing the learned approval model reduces gated action quality by 18%, as the agent cannot predict which actions reviewers will approve and resorts to conservative over-escalation.

11. Convergence Properties Under Gate Constraints

A natural concern is whether gate constraints affect the convergence guarantees of PPO. We address this by establishing convergence bounds for the gated setting.

11.1 Convergence Theorem

Theorem 2 (Gated PPO Convergence). Let pi_theta be a gate-masked PPO policy trained on a GC-MDP with risk-adaptive clipping. Under standard assumptions (bounded rewards, ergodic MDP, Lipschitz-continuous policy parameterization), the policy converges to a local optimum of the constrained objective at rate: $$ J(\pi^_{\text{gated}}) - J(\pi_\theta^{(T)}) \leq \frac{C_1}{\sqrt{T}} + C_2 \cdot \iota_G^{\max} $$ where T is the number of updates, C_1 depends on the learning rate schedule, and C_2 iota_G^max is the optimality gap due to gate constraints (iota_G^max is the maximum gate interaction coefficient across states). The first term is the standard PPO convergence rate. The second term is the price of governance — the irreducible gap between the constrained optimum and the unconstrained optimum. This gap is zero when gates impose no constraints and increases as gates become more restrictive.

11.2 Regret Bounds

We define governance regret as the cumulative difference between the constrained optimal policy and the learned policy: $$ \text{Regret}(T) = \sum_{t=1}^{T} \left[ V^{\pi^_{\text{gated}}}(s_t) - V^{\pi_\theta^{(t)}}(s_t) \right] $$ Proposition 2. Under gated PPO with risk-adaptive clipping, the governance regret satisfies Regret(T) = O(sqrt(T |A| * log(|G|))), where |G| is the number of active gates. The logarithmic dependence on |G| shows that adding more gates increases regret only logarithmically, not linearly — a favorable property for enterprise environments with many concurrent governance requirements.

12. Conclusion

This paper has established Actor-Critic Reinforcement Learning — specifically PPO with gate-constrained policy gradients — as the foundational algorithm for the Control Layer of agentic enterprises. The Gate-Constrained MDP formalism captures the essential structure of governed environments: dynamic action spaces determined by responsibility gates, multi-stakeholder reward functions, and human approval as environment dynamics. The gate-constrained policy gradient theorem provides the mathematical foundation for learning optimal policies under governance constraints, decomposing the gradient into interior and boundary terms that allow the agent to learn gate boundaries. PPO's clipped objective, extended with risk-adaptive clipping, provides formal stability guarantees appropriate for enterprise deployment where policy instability has real organizational consequences.

The experimental results demonstrate that gated autonomy PPO achieves near-unconstrained performance (94.2% relative task completion) while maintaining 99.7% gate compliance and reducing human oversight burden by 61%. These results validate the core thesis: governance and performance are not fundamentally opposed. With the right algorithmic framework, more governance enables more effective automation by giving the system clear boundaries within which to optimize.

The framework's integration with MARIA OS — through coordinate-based gate configuration, decision pipeline state mapping, and evidence bundle generation — demonstrates that theoretical RL constructs can be operationalized in production enterprise systems. The key architectural insight is that responsibility gates should be part of the RL environment, not external constraints bolted onto a trained policy. When the agent learns with gates from the beginning, it develops governance-native behavior that is both more compliant and more efficient than post-hoc constraint enforcement.

Future work will explore multi-agent gated PPO where multiple agents share gate constraints and must coordinate their gate-request behavior, hierarchical gate learning where agents propose gate threshold modifications based on their accumulated competence, and transfer learning across MARIA OS universes where gate structures share common patterns.