Tool Genesis Under Governance: How to Safely Turn Generated Code into New Commands

1. Introduction: The Self-Extension Problem

Modern AI agent systems are no longer static command executors. They observe patterns in their operational environment, identify capability gaps, and — increasingly — generate code to fill those gaps. An agent managing a data pipeline notices that a common transformation lacks a dedicated command. It writes the transformation function, tests it against sample data, and proposes registering it as a new command available to all agents in the system.

This capability is extraordinarily powerful. It means the system can adapt to novel requirements without human engineering effort. But it introduces a class of risk that traditional software engineering has never faced: runtime self-modification by an autonomous system.

Consider the attack surface. A generated tool has access to whatever permissions the generating agent holds. If registration is ungoverned, the new tool inherits those permissions permanently. The tool may function correctly for its intended purpose while simultaneously opening a side channel — writing logs to an external endpoint, caching sensitive data in an unencrypted temporary file, or gradually expanding its resource consumption until it starves other processes.

The fundamental question is not whether agents should generate tools. They should — the productivity gains are too significant to ignore. The question is: what governance architecture makes tool genesis safe?

1.1 Why Existing Approaches Fail

Current approaches to code generation safety fall into three categories, all of which are insufficient for tool genesis.

Static analysis catches known vulnerability patterns but cannot reason about emergent behavior in context. A generated tool that individually passes static analysis may create dangerous interactions when composed with existing tools.

Code review introduces human bottlenecks that negate the speed advantage of autonomous tool generation. If every generated tool requires human review, the system is not self-extending — it is proposing extensions that humans must approve.

Testing validates behavior against specified expectations but cannot verify the absence of unspecified behavior. A tool that passes all tests may still perform actions that no test anticipated.

What is needed is a governance pipeline — a sequence of automated gates that together provide safety guarantees stronger than any individual gate.

2. Formal Model: Tool Safety as Bounded Decidability

We define a tool $T$ as a tuple $(C, P, S, E)$ where $C$ is the executable code, $P$ is the declared permission set, $S$ is the state space the tool may access, and $E$ is the set of side effects the tool may produce.

T = (C, P, S, E) \quad \text{where} \quad P \subseteq \mathcal{P}_{\text{system}}, \quad S \subseteq \mathcal{S}_{\text{registry}}, \quad E \subseteq \mathcal{E}_{\text{allowed}}

A tool is safe if and only if three conditions hold simultaneously:

\text{Safe}(T) \iff \text{Bounded}(C) \land \text{Contained}(P, S) \land \text{Declared}(E)

Bounded means the code terminates within a resource budget (time, memory, I/O operations). Contained means the tool's actual resource access does not exceed its declared permissions and state space. Declared means every side effect the tool produces was explicitly declared before registration.

In general, determining whether arbitrary code satisfies these properties is undecidable (by reduction to the halting problem). However, under bounded execution — where we impose hard resource limits and execute in an isolated environment — safety becomes decidable.

\text{Given } \text{budget}(\tau, M, I) \text{ and sandbox } \Sigma: \quad \text{Safe}_{\text{bounded}}(T) \in O(\tau \cdot \log M)

The key insight is that we do not need to prove a tool is safe for all inputs and all time. We need to prove it is safe within the bounded execution context in which it will actually run.

3. The 7-Stage Tool Genesis Pipeline

The MARIA OS Tool Genesis Framework implements seven sequential gates. A generated tool must pass all seven to be registered as a command. Failure at any gate terminates the pipeline with a detailed audit record.

| Stage | Gate | Purpose | Fail Action |
|-------|------|---------|-------------|
| 1 | Schema Validation | Verify tool declaration completeness | Reject with missing field list |
| 2 | Static Safety Analysis | Detect known vulnerability patterns | Reject with violation report |
| 3 | Sandbox Execution | Run tool in isolated environment | Terminate + capture forensics |
| 4 | Permission Verification | Confirm actual access matches declared | Reject with access diff |
| 5 | Formal Safety Proof | Verify bounded termination and containment | Reject with counterexample |
| 6 | Integration Testing | Test composition with existing tools | Reject with conflict report |
| 7 | Governed Registration | Register with audit trail and rollback hooks | N/A (final stage) |

Each gate produces a gate certificate — a signed, timestamped record of the verification result. The complete set of certificates forms the tool's provenance chain, which is immutable and queryable.

interface ToolGenesisGateCertificate {
  gateId: 1 | 2 | 3 | 4 | 5 | 6 | 7
  gateName: string
  toolId: string
  timestamp: string // ISO 8601
  result: "pass" | "fail" | "conditional_pass"
  evidence: {
    checksPerformed: string[]
    resourcesConsumed: { cpuMs: number; memoryBytes: number; ioOps: number }
    findings: GateFinding[]
  }
  signature: string // HMAC-SHA256 of evidence payload
  previousCertificateHash: string | null // chain linkage
}

interface GateFinding {
  severity: "info" | "warning" | "critical" | "blocking"
  category: string
  description: string
  location?: { file: string; line: number; column: number }
  recommendation: string
}

4. Sandbox Verification: Isolation Without Compromise

Stage 3 of the pipeline executes the generated tool in an isolated sandbox. The sandbox provides a complete execution environment that mimics production while capturing every resource access attempt.

The sandbox architecture uses three isolation layers:

Layer 1: Process Isolation. The tool executes in a separate process with restricted system calls. The process has no network access, no filesystem access outside its designated workspace, and no ability to spawn child processes without explicit permission.

Layer 2: Resource Budgeting. Hard limits on CPU time (default: 5 seconds), memory allocation (default: 256MB), and I/O operations (default: 1000 ops). Exceeding any limit triggers immediate termination.

Layer 3: Capability Interception. All API calls are proxied through a capability interceptor that records every resource access attempt and compares it against the tool's declared permission set.

interface SandboxConfig {
  resourceBudget: {
    cpuTimeMs: number       // Hard CPU time limit
    memoryBytes: number     // Hard memory limit
    ioOperations: number    // Maximum I/O operations
    networkCalls: number    // Maximum network calls (usually 0)
  }
  permissions: {
    allowedApis: string[]           // Whitelisted API endpoints
    allowedStateKeys: string[]      // Accessible state keys
    allowedFilePatterns: string[]   // Glob patterns for file access
    denyList: string[]              // Explicitly blocked resources
  }
  monitoring: {
    captureAllSyscalls: boolean
    recordMemorySnapshots: boolean
    traceApiCalls: boolean
    escapeDetectionEnabled: boolean
  }
}

class ToolSandbox {
  async execute(
    tool: GeneratedTool,
    config: SandboxConfig,
    testInputs: TestInput[]
  ): Promise<SandboxResult> {
    const process = await this.spawnIsolated(tool.code, config)
    const monitor = new ResourceMonitor(config.resourceBudget)
    const interceptor = new CapabilityInterceptor(config.permissions)

    for (const input of testInputs) {
      const result = await process.run(input, {
        monitor,
        interceptor,
        timeout: config.resourceBudget.cpuTimeMs,
      })

      if (interceptor.hasViolations()) {
        await process.terminate()
        return {
          status: "permission_violation",
          violations: interceptor.getViolations(),
          forensics: await monitor.captureState(),
        }
      }
    }

    return {
      status: "pass",
      resourceUsage: monitor.getSummary(),
      apiTrace: interceptor.getTrace(),
    }
  }
}

Sandbox escape detection operates with a median latency of 23ms and a P99 of 48ms. When an escape attempt is detected — a system call outside the allowed set, a memory access outside the sandbox boundary, or a network connection attempt — the sandbox immediately terminates the process, captures a full memory dump for forensic analysis, and writes a blocking finding to the gate certificate.

5. Permission Escalation Model: Lattice-Based Authority Bounds

A newly generated tool must never possess more authority than the agent that created it. This principle — the authority ceiling — prevents privilege escalation through tool genesis.

We model permissions as a lattice $(\mathcal{L}, \sqsubseteq)$ where $\sqsubseteq$ defines an authority ordering. The permission set of a generated tool must be a lower bound of the generating agent's permission set:

\forall T \text{ generated by agent } A: \quad P(T) \sqsubseteq P(A) \quad \text{(Authority Ceiling)}

Furthermore, tools cannot compose permissions to exceed the ceiling. If tool $T_1$ with permissions $P_1$ invokes tool $T_2$ with permissions $P_2$, the effective permission at the invocation point is the meet (greatest lower bound):

P_{\text{effective}}(T_1 \circ T_2) = P(T_1) \sqcap P(T_2) \sqsubseteq P(A)

This ensures that tool composition can only narrow permissions, never widen them. The lattice structure provides three critical properties:

1. Monotonic restriction. Every composition step can only reduce or maintain the permission set. No sequence of tool invocations can escalate beyond the originating agent's authority.

2. Decidable containment. Given two permission sets, determining whether one is contained in the other is decidable in $O(|P| \log |P|)$ time by comparing sorted permission vectors.

3. Audit transparency. The lattice structure makes permission relationships visually inspectable. Every tool's position in the lattice is a complete description of its authority relative to all other tools.

interface PermissionLattice {
  // Check if permission set A is dominated by B
  isBelow(a: PermissionSet, b: PermissionSet): boolean

  // Compute the meet (greatest lower bound) of two permission sets
  meet(a: PermissionSet, b: PermissionSet): PermissionSet

  // Validate that a tool's permissions respect the authority ceiling
  validateCeiling(
    tool: GeneratedTool,
    generatingAgent: AgentIdentity
  ): {
    valid: boolean
    violations: PermissionViolation[]
    effectivePermissions: PermissionSet
  }

  // Compute the transitive closure of tool composition permissions
  compositionPermissions(
    toolChain: GeneratedTool[]
  ): PermissionSet
}

type PermissionSet = {
  read: ResourcePattern[]
  write: ResourcePattern[]
  execute: CommandPattern[]
  network: NetworkScope[]
  maxCpuMs: number
  maxMemoryBytes: number
}

6. Immutable Audit Trail for Tool Creation

Every tool genesis event produces an immutable audit record. The audit trail serves three purposes: compliance (proving that governance gates were followed), forensics (investigating incidents involving generated tools), and learning (improving the genesis pipeline based on historical patterns).

The audit record is structured as a Merkle tree where each gate certificate is a leaf node. The root hash of the tree is the tool's governance fingerprint — a single value that cryptographically commits to the entire verification history.

H_{\text{root}} = H(H(G_1 \| G_2) \| H(G_3 \| G_4) \| H(G_5 \| H(G_6 \| G_7)))

Where $G_i$ is the serialized gate certificate for stage $i$ and $H$ is SHA-256. Any modification to any gate certificate changes the root hash, making tampering immediately detectable.

interface ToolGenesisAuditRecord {
  toolId: string
  governanceFingerprint: string // Merkle root of all gate certificates
  genesisTimestamp: string
  generatingAgent: {
    coordinate: string // MARIA coordinate (e.g., G1.U2.P3.Z1.A5)
    permissionSetHash: string
  }
  pipeline: {
    gateCertificates: ToolGenesisGateCertificate[]
    totalDurationMs: number
    resourcesConsumed: ResourceSummary
  }
  registration: {
    registryVersion: number
    previousToolVersion: string | null // for updates
    rollbackToken: string // token to trigger automatic rollback
    expiresAt: string | null // optional TTL for provisional tools
  }
  mariaCoordinate: string // where this tool is registered in the hierarchy
}

Audit records are append-only and replicated across three storage backends: the primary database, an event log, and an external compliance archive. This triple redundancy ensures that audit data survives any single storage failure.

7. Rollback Mechanisms: Undoing Tool Registration

A registered tool may later be discovered to be unsafe — through runtime monitoring, user reports, or updated safety analysis. The system must be able to completely undo a tool registration, restoring the system to its pre-registration state.

Rollback is implemented through a compensating transaction pattern. At registration time, the system records every state mutation required to register the tool and generates a corresponding compensating mutation for each one.

interface RollbackPlan {
  toolId: string
  rollbackToken: string
  compensatingActions: CompensatingAction[]
  dependencyCheck: {
    // Tools that depend on this tool
    dependents: string[]
    // Whether cascade rollback is required
    cascadeRequired: boolean
    // Estimated blast radius
    affectedAgents: string[]
  }
  stateSnapshot: {
    registryStateHash: string
    toolConfigHash: string
    permissionStateHash: string
  }
}

interface CompensatingAction {
  order: number
  action: "deregister" | "restore_previous" | "cleanup_cache" |
          "revoke_permissions" | "notify_dependents" | "archive_audit"
  target: string
  params: Record<string, unknown>
  timeout: number
  retryPolicy: { maxRetries: number; backoffMs: number }
}

async function executeRollback(plan: RollbackPlan): Promise<RollbackResult> {
  // Phase 1: Dependency check — fail if cascade would affect critical paths
  if (plan.dependencyCheck.cascadeRequired) {
    const cascadeApproval = await requestCascadeApproval(plan)
    if (!cascadeApproval.approved) {
      return { status: "blocked", reason: "cascade_not_approved" }
    }
  }

  // Phase 2: Execute compensating actions in order
  const results: ActionResult[] = []
  for (const action of plan.compensatingActions) {
    const result = await executeWithRetry(action)
    results.push(result)
    if (result.status === "failed") {
      // Partial rollback — escalate to human operator
      await escalatePartialRollback(plan, results)
      return { status: "partial", completedActions: results }
    }
  }

  // Phase 3: Verify state matches pre-registration snapshot
  const currentStateHash = await computeRegistryStateHash()
  if (currentStateHash !== plan.stateSnapshot.registryStateHash) {
    await escalateStateMismatch(plan, currentStateHash)
    return { status: "verify_failed", stateHash: currentStateHash }
  }

  return { status: "complete", completedActions: results }
}

Rollback P99 latency is 340ms, measured across 5,000 rollback events. The majority of this time (68%) is spent on dependency checking, not on the actual state mutations.

8. Formal Verification of Tool Safety

Stage 5 of the pipeline performs formal verification — proving, not testing, that the tool satisfies safety properties. Under bounded execution, we reduce tool safety to satisfiability of a first-order formula.

For a tool $T = (C, P, S, E)$ with resource budget $B = (\tau, M, I)$, we construct a verification condition $\phi_T$:

\phi_T \equiv \forall \vec{x} \in \text{Input}(T): \quad \text{Exec}(C, \vec{x}, B) \Rightarrow (\text{Access}(C, \vec{x}) \subseteq P) \land (\text{Effects}(C, \vec{x}) \subseteq E) \land (\text{Time}(C, \vec{x}) \leq \tau)

The verification condition states: for all valid inputs, if the code executes within the budget, then its resource accesses are contained in the declared permissions, its side effects are contained in the declared effects, and its execution time is within the time budget.

We discharge $\phi_T$ using bounded model checking with an SMT solver. The bounded execution assumption makes the state space finite, which makes the verification decidable. In practice, verification completes in under 2 seconds for tools with fewer than 500 AST nodes (which covers 94% of generated tools in our production data).

Formal verification proves safety within the bounded execution model. It does not prove safety for executions that exceed the resource budget. This is by design: executions that exceed the budget are terminated by the sandbox, so safety beyond the budget is enforced by runtime mechanisms rather than static proofs.

9. Runtime Monitoring: Post-Registration Surveillance

Governance does not end at registration. Every registered tool is continuously monitored during execution. Runtime monitoring catches behaviors that formal verification cannot — environmental dependencies, timing-sensitive interactions, and gradual resource drift.

The monitoring system tracks four signal categories:

| Signal Category | Metrics | Alert Threshold | Response |
|----------------|---------|-----------------|----------|
| Resource Consumption | CPU, memory, I/O per invocation | > 2x declared budget | Rate limit, then suspend |
| Permission Usage | API calls, state access patterns | Any undeclared access | Immediate suspend + audit |
| Behavioral Drift | Output distribution, error rates | KL divergence > 0.3 | Flag for review |
| Composition Effects | Cross-tool interaction patterns | New dependency detected | Integration re-test |

The Tool Safety Index (TSI) is a composite runtime metric that aggregates all four signal categories into a single score:

\text{TSI}(T, t) = w_r \cdot R(T, t) + w_p \cdot P(T, t) + w_b \cdot B(T, t) + w_c \cdot C(T, t) \quad \in [0, 1]

Where $R$ is resource compliance (1.0 = within budget), $P$ is permission compliance (1.0 = no violations), $B$ is behavioral stability (1.0 = no drift), $C$ is composition safety (1.0 = no unexpected interactions), and $w_r, w_p, w_b, w_c$ are weights summing to 1. A TSI below 0.85 triggers automatic review; below 0.60 triggers automatic suspension.

10. MARIA OS Tool Registry Architecture

The Tool Registry is the central authority for tool lifecycle management in MARIA OS. It maintains the canonical record of every registered tool, its governance provenance, runtime status, and dependency graph.

interface ToolRegistry {
  // Registration
  register(tool: VerifiedTool, certificates: GateCertificate[]): Promise<RegistrationResult>
  deregister(toolId: string, reason: string): Promise<DeregistrationResult>

  // Discovery
  findTool(query: ToolQuery): Promise<RegisteredTool[]>
  getToolByCoordinate(coordinate: string): Promise<RegisteredTool | null>
  getDependencyGraph(toolId: string): Promise<DependencyGraph>

  // Governance
  getProvenanceChain(toolId: string): Promise<ToolGenesisAuditRecord>
  getSafetyIndex(toolId: string): Promise<number> // Current TSI
  getPermissionBounds(toolId: string): Promise<PermissionSet>

  // Lifecycle
  suspend(toolId: string, reason: string): Promise<void>
  resume(toolId: string, approval: ApprovalRecord): Promise<void>
  rollback(toolId: string, rollbackToken: string): Promise<RollbackResult>

  // Analytics
  getGenesisStats(timeRange: TimeRange): Promise<GenesisStatistics>
  getFailureAnalysis(timeRange: TimeRange): Promise<FailureReport>
}

interface RegisteredTool {
  id: string
  name: string
  version: number
  coordinate: string // MARIA coordinate
  code: string
  permissions: PermissionSet
  declaredEffects: Effect[]
  status: "active" | "suspended" | "deprecated" | "rolled_back"
  tsi: number // Current Tool Safety Index
  registeredAt: string
  registeredBy: string // Agent coordinate
  governanceFingerprint: string
  rollbackToken: string
  invocationCount: number
  lastInvokedAt: string | null
}

The registry is organized according to the MARIA coordinate system. Each tool is registered at a specific coordinate — Galaxy, Universe, Planet, Zone, Agent — and its permissions are scoped to that coordinate's authority. A tool registered at Zone level cannot access resources belonging to a different Zone, even if the generating agent has cross-zone permissions.

10.1 Version Management

Tools can be updated through re-genesis. An updated tool goes through the same 7-stage pipeline as a new tool, with one additional check: the updated tool must be backward compatible with any tools that depend on it. Backward compatibility is verified by running the dependent tools' integration tests against the new version.

If backward compatibility cannot be maintained, the update triggers a cascade re-verification — all dependent tools are re-verified against the new version. This can be expensive, so the registry maintains a dependency impact score that predicts cascade cost before the update begins.

11. Benchmarks and Production Results

We evaluated the Tool Genesis Framework across 10,000 tool genesis events in a production MARIA OS deployment over 90 days. The deployment included 847 active agents across 12 Zones.

| Metric | Value | Notes |
|--------|-------|-------|
| Total genesis events | 10,000 | Unique tool generation attempts |
| Stage 1 pass rate | 97.2% | Schema validation |
| Stage 2 pass rate | 89.1% | Static safety analysis |
| Stage 3 pass rate | 96.8% | Sandbox execution (of Stage 2 passes) |
| Stage 4 pass rate | 98.4% | Permission verification |
| Stage 5 pass rate | 93.7% | Formal safety proof |
| Stage 6 pass rate | 91.2% | Integration testing |
| End-to-end pass rate | 72.3% | All 7 stages passed |
| Safety compliance (registered tools) | 99.7% | No safety incidents post-registration |
| Mean pipeline latency | 4.2s | End-to-end for passing tools |
| P95 pipeline latency | 8.7s | End-to-end for passing tools |
| Rollback events | 31 | 0.43% of registered tools |
| Mean rollback time | 180ms | Time to complete rollback |
| P99 rollback time | 340ms | Worst-case rollback time |

The most common failure mode was Stage 2 (static safety analysis), where 10.9% of generated tools contained patterns flagged as potential vulnerabilities. The majority of these were false positives (estimated 62% false positive rate), which is a known limitation of static analysis. However, we accept the high false positive rate because the cost of a false positive (re-generation) is far lower than the cost of a false negative (unsafe tool in production).

A 72.3% pass rate means 27.7% of generated tools are rejected by the governance pipeline. This is not a failure of the generation system — it is the governance system doing its job. The generating agent receives detailed feedback from the rejection and can re-generate an improved tool. On average, tools that fail on the first attempt pass on the second attempt 84% of the time.

12. Conclusion: Self-Extension as Governed Evolution

Tool genesis is the natural evolution of agentic systems. As agents become more capable, their ability to create new tools becomes a fundamental capability — not a bug to be suppressed, but a feature to be governed.

The MARIA OS Tool Genesis Framework demonstrates that governance and autonomy are not in tension. The 7-stage pipeline adds only 12% latency overhead while achieving 99.7% safety compliance. Formal verification under bounded execution makes tool safety a decidable problem. The permission lattice ensures that tool composition can only narrow authority, never widen it. Immutable audit trails make every tool's provenance inspectable and tamper-evident. Automatic rollback provides a safety net that makes experimentation safe.

The key architectural insight is that governance enables autonomy. An ungoverned agent that can generate tools is dangerous and must be restricted. A governed agent that generates tools through a verified pipeline can be given wide latitude — because the governance infrastructure guarantees that no tool will exceed its bounds.

This is the principle that underlies all of MARIA OS: more governance enables more automation. Tool genesis is where this principle reaches its most powerful expression. The system extends itself, safely, under governance — and in doing so, becomes more capable without becoming more dangerous.

\lim_{n \to \infty} \text{Capability}(\text{System}_n) = \infty \quad \text{subject to} \quad \forall n: \text{TSI}(\text{System}_n) \geq \theta_{\text{safe}}

Self-extension is not dangerous. Ungoverned self-extension is. The difference is architecture.