Sandbox Timeout Failures as a Code Execution Architecture Diagnosis

Separating command timeouts from sandbox failures keeps infrastructure faults out of scores.

· 8 min read
Cover illustration for “Sandbox Timeout Failures as a Code Execution Architecture Diagnosis”
Sandboxed Code Execution · November 14, 2026 · 8 min read · 1,746 words

Persistent AI agent infrastructure versus serverless functions is a question of execution lifetime, state, and failure attribution. Long-running agents need task budgets that survive pauses, backstops that reset on resume, and errors that distinguish stalled infrastructure from runaway code. This article explains how a two-tier timeout taxonomy turns sandbox failures into a diagnosis of the code execution layer.

Why a sandbox timeout is an ambiguous signal by design

The ambiguity starts in the error type’s design. At the API boundary, exec(timeout=...) can raise the same SandboxTimeoutError when the user’s process runs too long and when the sandbox host stops responding. The exception says that execution did not return as expected, but leaves the cause unknown. No scorer can recover that missing distinction from the label alone.

The two causes call for different remedies. A runaway process can warrant a task failure or a retry with stricter limits, while an unresponsive sandbox warrants infrastructure investigation and a retry without penalizing the model. Applying the first remedy to the second event quietly charges the infrastructure’s failure to the agent. The task receives a plausible-looking result, so the developer gets no immediate warning that the result measures the wrong thing.

How the ambiguity corrupts eval scores, RL reward signals, and production reliability metrics

An eval scorer turns this missing distinction into a measurement error when it maps every SandboxTimeoutError to INCORRECT or 0.0. As the sandbox degrades, the model’s apparent failure rate rises even if its capability has not changed. The benchmark now partly measures infrastructure health. That makes a score harder to interpret because the recorded outcome does not say whether the agent failed the task or lost access to the machinery needed to finish it.

A reward penalty for a stalled sandbox feeds into policy updates, pushing the agent away from behavior that happened to coincide with infrastructure degradation. The behavior need not have caused the failure. The pipeline has converted an execution-layer problem into a training signal, so an attribution error can change what the model learns.

Production dashboards have the same blind spot when all timeouts fall into one error class. A spike after a bad model release looks like one caused by sandbox host saturation, though each calls for a different search for the cause. Incident response starts from an ambiguous count. The inspect-evals-lint #95 remediation rule draws the necessary boundary for scoring. A plain exec timeout should raise rather than score, unless the time limit belongs to the task definition, in which case the scorer reads the in-sandbox exit code.

Why wall-clock timeout misconfiguration is the most common compounding factor

Misconfigured wall-clock timers compound the failure to distinguish error types by creating timeout signals that should never have represented task failure. A sandbox stack can have a per-cell execution limit, an adapter backstop, a provider-side idle-read timeout, and a recovery watchdog. Each watches a different part of execution. If they return indistinguishable errors, the operator must first discover which clock fired before investigating why it fired.

Paperclip’s PR #9232 addresses a concrete version of this problem. Sandbox-backed runs could be killed with a bare Timed out after Ns message that named neither the timer nor its configuration knob. The message offered no useful starting point. The fix raises the sandbox backstop to 4 hours, aligned with the recovery watchdog’s critical threshold, routes acpx_local through the shared timeout resolver, and makes the error identify both the timer and its setting.

AgentScope Java’s fix exposes a related gap between client and server deadlines. A client timeout could drop the HTTP stream while leaving the remote process running and accumulating compute cost. Stopping the wait was not stopping the work. Together, these cases show why timeout configuration is not just a choice of duration. A timer also needs a defined owner, a defined target, and an error that explains which part of the stack acted.

The two-tier error taxonomy that separates agent fault from infrastructure fault

The architectural response is to separate command failure from sandbox failure before either reaches a scorer. Tier 1, SandboxCommandTimeoutError, identifies a user process that exceeded its per-task wall-clock budget while the sandbox remained healthy. When the time limit is part of the task, the scorer can record failure or apply task-specific logic using the in-sandbox exit code.

Tier 2, the unsubclassed SandboxTimeoutError base class, identifies a sandbox that stopped answering. The agent’s code may or may not have finished, so there is no sound task outcome to score. The distinction is about available evidence. This error should raise and route to --retry-on-error, keeping infrastructure uncertainty out of the model’s result.

AgentScope starts to make that distinction in how it handles the protocol, below the exception API. Its Connect-Timeout-Ms header triggers a server-side kill, and a returned deadline_exceeded frame maps to ExecTimeoutException. Missing completion evidence takes a separate path. A stream ending without an exit code produces MissingExitCodeException, which maps to ExecTimeoutException when elapsed time is at least the timeout. That elapsed-time fallback is not equivalent to an in-sandbox result, however. Host polling can observe a program only after its deadline even when it finished just before it, so the in-sandbox exit code remains the reliable discriminator for command-side failure.

What proper timeout architecture looks like across the sandbox stack

To implement the taxonomy, builders should give each timer a distinct scope and handling path. The first layer is the per-task execution budget inside the sandbox, set through exec(timeout=N). The runtime enforces that budget and reports a typed command timeout. This layer gives the scorer evidence of whether the task reached its deadline. An outer timer firing does not, on its own, show that the agent ran past the time allowed for the task.

The second layer is the server-side kill deadline at the transport boundary. A deadline header such as Connect-Timeout-Ms lets the server stop the remote process and return a typed deadline_exceeded error. The kill belongs where the process is running. Without it, a client can abandon the stream without ending the execution, as the AgentScope fix demonstrates.

The third layer is an idle-read or inactivity monitor that detects a sandbox no longer producing output without exiting. Its timeout should raise, never score. The fourth layer is a wall-clock backstop for genuinely hung runs, set high enough to let the inactivity monitor act first and allow recovery logic to proceed. Every timer should log its name and configuration knob when a run starts, so the run log has what is needed to diagnose a later failure without reading source code.

Why each sandbox needs its own isolated execution environment, not a shared one

The taxonomy depends on a boundary around the workload. In a shared container or shared-kernel environment, one tenant’s runaway process can use up CPU that another tenant needs to finish within its wall-clock budget. Tenant B then records a command timeout caused by tenant A’s workload. The exception may correctly identify which deadline expired, yet still misattribute responsibility. Separate error types cannot repair a shared execution environment that lets one task distort another task’s timing.

Micro-VM isolation, including Firecracker, gives each workload its own kernel behind a hardware virtualization boundary. That boundary helps contain kernel failures and workload resource pressure instead of making them faults in a shared guest environment. Isolation makes the sandbox a meaningful diagnostic unit. It also limits what a server-side kill can reach, making it less likely to stop another tenant’s work.

The Balkanization of Execution-Security Research for AI Coding Agents documents four disclosed, patched CVEs affecting production agent harnesses, including a runc container-escape vulnerability. Those disclosures show that execution boundaries have been breached in real deployments, though they are not a blanket verdict on every container design. The cost objection has a separate answer. Sleep-and-resume architectures let an isolated VM stop accumulating idle vCPU compute cost and wake through snapshot restore in well under a second, typically tens of milliseconds. Storage, and in some configurations memory, can still incur charges.

How snapshot and resume mechanics interact with timeout state across long-running agent tasks

Long-lived agents introduce another source of false attribution when a timer acts on resumed execution without accounting for the checkpoint. Firecracker resumes a snapshot by memory-mapping the snapshot file, restoring CPU state, and continuing from the point where execution stopped. A checkpoint is not a fresh task. An in-progress execution timer needs to keep the task budget that was left when execution stopped, with no new allowance when the VM resumes.

An adapter’s wall-clock backstop has a different responsibility. It does not automatically reset on resume, so one with little time remaining can fire shortly after the agent returns, regardless of how long the resumed session has run. The clocks need different resume rules. Per-task budgets should be tracked inside the sandbox so they survive snapshot-restore cycles accurately, while host-side backstops should reset on each resume event instead of inheriting accumulated time from the prior session.

Firecracker closes guest vsock connections on resume, which breaks a long-lived control channel and may keep an agent from receiving a timeout notification. That is evidence of an infrastructure fault. Google Cloud Agent Platform’s troubleshooting guidance says that code which relies on earlier state must use the same sandbox_name across calls and check that the sandbox has not expired. Reconnecting to an expired sandbox belongs in the infrastructure-fault path.

Applying the two-tier taxonomy to eval pipelines, RL rollouts, and production observability

Eval pipelines should catch SandboxCommandTimeoutError and use the task-failure logic when the task sets the execution limit. Other sandbox timeouts should propagate and route to --retry-on-error, leaving the score unchanged. Where a sandbox does not yet implement the command-timeout subclass, an in-sandbox exit-code helper provides the fallback by reporting whether the internal timeout fired. It avoids using host observation time as a substitute for the process’s result.

RL rollout pipelines should exclude infrastructure-side timeouts from the reward signal and log them as infrastructure events. Only genuine command overruns should contribute timeout penalties to policy updates. Agent-fault timeouts belong in task-failure dashboards and point toward model debugging, while infrastructure-fault timeouts belong in sandbox-health dashboards and point toward infrastructure investigation.

The platform should expose these categories as structured log events as well as exception classes. Each event should have the sandbox ID, task ID, timer that fired, and in-sandbox exit code if one is available. Error messages should also name the timer and configuration knob, rather than leave operators with Timed out after Ns. That gives both people and tooling a usable record.

Sources

  1. The Balkanization of Execution-Security Research for AI Coding Agents: Isolation, Access Control, and Time-of-Check-to-Time-of-Use Vulnerabilities

More in Sandboxed Code Execution