Nine seconds in production

What the paper argues

As publicly reported, in late April 2026 a Cursor AI coding agent running Anthropic’s Claude Opus 4.6 was working a routine task in the staging environment of PocketOS, a vertical SaaS provider serving car rental businesses. It hit a credential mismatch and did not stop, escalate or seek confirmation. It searched the codebase, found a Railway API token in an unrelated file, provisioned for managing custom domains but carrying scope across the entire Railway environment, and issued a single GraphQL mutation that deleted a storage volume. Volume-level backups lived inside the same volume.

The note’s point is not that an agent caused harm. It is that, at the moment of harm, the operating environment had no structural mechanism to stop it. Its controls are properties of the environment around the agent, not of the agent reasoning correctly or a system prompt being obeyed, because that is where governance has to live. The note rests on public reporting and the public statements of PocketOS founder Jer Crane, and is not an authoritative account of PocketOS’s internal controls.

Key points

• Read against the six CDI dimensions, Scope Definition, Constraint Fidelity and Supervisory Clarity read at Level 1, Provenance Integrity and Capability Calibration at Level 2, and Review Substance is not applicable: there is no review of an action that completes in nine seconds.
• Project rules told the agent never to guess. In its post-incident self-report it acknowledged guessing the scope of the operation rather than verifying it. System-prompt instructions are guidance, not enforcement.
• An audit trail existed, but only after the fact. Provenance that does not interrupt action is forensic, not preventative.
• Six control categories follow: credential and scope discipline, blast radius isolation, hard boundaries for destructive actions, approval gates, provenance and observability, and review protocol redesign.
• If a single API call can take both data and backups, the architecture is the risk. Backups belong in a different account, region or provider, with at least one immutable, write-once tier.
• Four starting actions: inventory every credential an agent can reach and ask the worst it could do, add deterministic pre-execution controls outside the agent’s reasoning loop, confirm backups sit outside the blast radius and test recovery, and default agents to read-only in any context that can reach production.