The immediate development
The failure mode is moving from wrong answers to unverifiable action
A preprint posted on August 11, Agent Safety Should Be a Runtime Contract, makes a useful distinction that is becoming harder to ignore as AI agents move into real workflows. Training can influence what a model tends to do. It cannot, by itself, prove that a file was actually changed, a test really passed, a citation supports a claim, a message was sent to the intended recipient or a database mutation stayed inside policy.
The authors argue that agent safety needs a second layer outside the model: a runtime contract enforced by the harness that gives the agent tools, observes its trajectory and decides whether actions or completion claims are admissible. Their proposal has a preventive side—sandboxes, permission gates, action filters and trajectory monitors—and an evidential side that requires checkable proof such as test logs, file diffs or grounded citations before success is accepted.
The paper is a preprint, not a peer-reviewed result, and its headline empirical numbers come from audits the authors designed and coded themselves. In their review of 52 documented agent and LLM safety incidents, they classify 40 as fully preventable by a functional harness, 11 as mitigable and one as primarily an internal-goal or alignment problem. A separate false-completion audit contains 31 non-contested core cases plus one disputed illustrative case. Those counts are evidence for the authors' taxonomy, not a population-level estimate of how often runtime controls would prevent failures in production.
What makes the paper interesting is not the exact percentage. It is that other recent work is converging on the same architectural move from different directions: actions are treated as proposals, state changes are admitted through deterministic gates, execution histories are preserved, and final success is separated from the model's own assertion that it succeeded.
The unit of trust is becoming the trajectory, not the final sentence
For ordinary text generation, evaluation often begins with the output. Is the answer correct? Is it safe? Does it cite reliable evidence? An agent adds a second object that matters just as much: the trajectory, meaning the sequence of observations, tool calls, state changes, errors, retries and approvals that produced the final result.
That distinction is easy to see in a coding task. Suppose an agent reports, “I fixed the bug and all tests pass.” The statement can sound perfectly plausible even if the agent changed the wrong file, ran only a subset of tests, ignored a failing integration test or never executed the test suite at all. A language-model judge can inspect the sentence, but the stronger evidence lives elsewhere: the repository diff, command log, test output and final working tree.
The same logic applies to non-software work. A research agent can claim that three sources agree when one was never opened. A procurement agent can say it stayed under a spending limit even though one tool call exceeded it. A scheduling agent can report that invitations were sent when one API request failed. Once a task depends on external state, fluent completion language is not the same thing as completion.
Anthropic's January guidance on evaluating agents makes a related point from the measurement side. Agents operate over many turns, use tools and adapt to intermediate results, so useful evaluations need to inspect more than a final answer. Anthropic recommends graders that can examine trajectories, tool outcomes and environment state, while also warning that realistic agent evaluations are harder to make stable than single-turn tests.
This is also why AIUpdateWatch's recent browser-agent security analysis focused on permissions and cross-site action rather than prompt wording alone. Once a model can act, reliability depends on the architecture around the model as well as the model itself.
Prevention and proof solve different parts of the problem
The preventive half of a runtime contract is familiar from computer security. A tool can refuse to delete files outside a workspace. A payment agent can be blocked from moving more than a defined amount. A browser agent can be denied access to unrelated logged-in sites. Credentials can be short-lived and scoped. Human approval can be required before a consequential action is committed.
These controls answer: what is the agent allowed to do?
The evidential half answers a different question: what must be demonstrated before the system accepts that the task is done? A coding task may require a clean test run. A report may require each consequential claim to map to an opened source. A data migration may require row counts and checksums. A financial reconciliation may require independently computed totals. A document workflow may require that the final artifact exists at the expected location and matches the approved version.
This division matters because an agent can stay inside its permissions and still fail the task. Nothing unsafe has to happen. The system may simply accept a false completion claim. Conversely, a task can be completed correctly while the execution path violated a policy that should have blocked it. Safety and correctness overlap, but they are not identical.
NIST's 2026 work on software and AI agent identity is pushing the same systems boundary from another angle. Its National Cybersecurity Center of Excellence has asked how agents should be identified, authorized and audited when they act across enterprise resources. Identity establishes who or what is exercising authority. Runtime evidence establishes what that authority actually produced.
Completion should be a proposal that the runtime can reject
One of the most useful ideas in the recent literature is to demote the model's declaration of success from a fact to a proposal.
Verify-Gated Completion as Admission Control in a Governed Multi-Agent Runtime studies a bounded system where agents can propose completion but a read-only verifier decides whether the claim is admitted. Weak or ambiguous evidence fails closed. The authors report 1,791 successful verification events out of 1,800 known-outcome invoked events in one released slice, but they explicitly caution that this is not a task-success or production-reliability rate. Most rows came from one high-volume reporting cluster, and only 17 events were production-classified.
That caveat is more valuable than a bigger percentage would have been. It shows the right evaluation question: not “did the verifier usually say yes?” but “did the verifier reliably distinguish valid from invalid completion across representative tasks without becoming a new source of error or gridlock?”
A July paper called Mnemosyne takes the proposal idea into stateful workflows. Its authors treat generated actions as untrusted proposals until a deterministic admission layer checks them against an executable constraint set. The runtime maintains an append-only transition log and can reject or repair proposed state changes. In their released experiments, the gate rejected targeted violations in nine falsification tests and, in a set of live-proposer repair trials, admitted 24 of 40 repair proposals while rejecting 16. The authors report no invalid commits in that bounded test setup.
The broader architectural point is independent of those specific numbers. A stochastic model can remain creative and flexible while a smaller deterministic layer decides whether a consequential action is allowed to become reality.
Proof of execution turns agent activity into something auditable
Another recent preprint, Proof of Execution: Runtime Verification for Governed AI Agent Actions, asks what an organization would need to prove after an agent has acted. Its framework binds three things together: a contract describing what is allowed, an event stream describing what happened, and a replay context intended to make the execution reconstructable.
The proposal separates planning from enforcement, effect and recordkeeping. That separation is important. If the same model that decides an action can also decide whether the action complied with policy and can rewrite the log that records it, the audit trail is weak by construction. A governed runtime instead tries to make the action path independently checkable.
The paper reports a TypeScript prototype with low single-digit millisecond overhead on a minimal flow and roughly 4.4% overhead on concurrent batch workloads. Those figures are prototype results under the paper's own assumptions, not evidence that every enterprise agent can gain cryptographically meaningful auditability for a 4% cost. Still, they challenge one common objection: that stronger runtime records must necessarily be too expensive to use outside the highest-risk domains.
This resembles a familiar pattern from databases and distributed systems. We do not trust a database transaction because the application says “done.” We rely on commit rules, logs, consistency mechanisms and recoverable state. Agentic systems are beginning to rediscover the same principle at the language-to-action boundary.
This matters well beyond software engineering
The obvious examples come from coding agents because code gives verifiers unusually crisp evidence: tests, diffs, compilers and reproducible commands. The more consequential question is whether equivalent evidence can be defined in professional workflows where success is less binary.
Finance. An agent preparing a reconciliation should not be accepted because its narrative says the books balance. The runtime can require source totals, exception lists, ledger references and independently recomputed checks before the result is submitted.
Law and compliance. An agent can draft a memo, but consequential assertions may need linked primary authorities, jurisdiction checks and a record of which documents were actually retrieved. Verification will not decide whether the legal reasoning is ultimately correct, but it can prevent a system from claiming source support it never established.
Medicine. A clinical support agent should not be able to convert a plausible summary into an untraceable action. Evidence can include the patient data consulted, medication constraints checked, decision-support rules invoked and explicit human approval for actions that remain clinician-controlled. The runtime cannot turn uncertain medicine into deterministic truth; it can make the path to a recommendation inspectable.
Operations. Agents that schedule staff, update inventories or move work through queues can be checked against system state. Did the shift actually change? Did every downstream dependency receive the update? Did a failed API call leave the task half-complete? These are often more tractable questions than whether the model's explanation sounds competent.
The common pattern is that evidence requirements should come from the domain's real definition of completion. A generic “LLM verifier” is not enough. The strongest verifier is often the authoritative system of record, a deterministic calculation, a test harness or a human decision at a deliberately chosen boundary.
Verification can fail too
Runtime verification is not a magic layer that makes agents safe. It moves some of the trust from the model into the harness, and that creates new engineering questions.
A verifier can be incomplete. If a coding gate checks unit tests but ignores security tests, an agent can pass while still creating a serious vulnerability. A verifier can be stale if the policy it enforces no longer matches the business process. A human approval step can become meaningless if people learn to click through hundreds of low-value confirmations. An LLM-based verifier can hallucinate, share the same blind spot as the agent or be manipulated by the same untrusted context.
There is also a liveness trade-off. A fail-closed system is safer when evidence is missing, but an excessively strict gate can block legitimate work and make the agent economically useless. The design target is therefore not maximum rejection. It is accurate admission: block what should not happen, require proof where proof is meaningful and let routine valid work proceed without unnecessary friction.
Privacy introduces another complication. Detailed trajectories can contain sensitive prompts, retrieved documents, tool arguments and internal identifiers. A company that improves auditability by logging everything may create a new data-governance problem. Provenance needs retention rules, access controls and minimization of its own.
These limitations are why the August 11 paper's strongest claims should be treated as a research agenda rather than a finished safety theorem. The architecture is plausible. The evidence that it generalizes across domains is still developing.
The next benchmark problem is the gate, not only the model
Agent leaderboards usually ask whether a model completed a task. Runtime assurance needs an additional benchmark layer: whether the surrounding system correctly admitted, rejected or escalated each consequential step.
That requires harder test sets. A useful benchmark should include valid tasks that look risky, invalid actions that look superficially normal, partial failures, stale state, ambiguous completion evidence and multi-step cases where no single tool call is dangerous but the sequence is. It should measure false approvals and false blocks separately. It should also record how much latency, compute and human attention the assurance layer adds.
Anthropic's AuditBench work illustrates a related difficulty in a different setting: a tool can surface useful evidence in isolation and still fail to help an investigator agent use that evidence correctly. Evidence availability and evidence interpretation are separate capabilities. A runtime gate that generates logs no one can reliably interpret has not solved the assurance problem.
This is where the research could become genuinely important. If standardized trajectory schemas and verifiable completion rules emerge, model comparisons may start to include a new dimension: not only how capable an agent is, but how safely and auditably it can operate inside a governed system. That would be a much more useful metric for many organizations than another small gain on a static reasoning benchmark.
AIUpdateWatch's benchmark coverage already separates task-specific evidence from broad claims of model superiority. Agent assurance needs the same discipline: a runtime gate that performs well on coding tasks should not automatically be treated as validated for clinical, legal or financial workflows.
What would make runtime assurance convincing?
The next evidence should come from systems that are harder to control than a research prototype.
First, independent teams need to reproduce the core patterns across different agent frameworks and models. Second, evaluations need realistic adverse cases in which the agent is competent enough to make plausible mistakes rather than obviously broken ones. Third, runtime controls need longitudinal production evidence: how many harmful or false-completion events were prevented, how often legitimate work was blocked, how much human review was required and what failure modes escaped the gate.
Standards work will matter as well. NIST's agent-identity initiative is already treating authorization and auditing as infrastructure concerns. Similar convergence around trace formats, evidence objects and action semantics would make it easier to compare systems instead of accepting each vendor's private definition of “safe agent.”
The larger shift is straightforward. Training still matters. Better models will make fewer mistakes, follow policies more reliably and need less correction. But once AI moves from generating suggestions to changing external state, the model cannot remain the sole witness to its own success.
The most trustworthy agent may not be the one that confidently says it finished. It may be the one whose surrounding system can show, step by step, what was authorized, what actually happened, what evidence supports completion and where uncertainty still remains.
For related coverage, see AIUpdateWatch's agent-security explainer, browser-agent security analysis and evidence-quality methodology.
Primary and supporting evidence
Sources
- Ng, Han, Zhang and Wang — Agent Safety Should Be a Runtime Contract, arXiv v1, August 11, 2026.
- Rhodes and Kang — Proof of Execution: Runtime Verification for Governed AI Agent Actions, arXiv, 2026.
- Chang, Geng and Chang — Mnemosyne: Agentic Transaction Processing for Validating and Repairing AI-generated Workflows, arXiv v2, July 2026.
- Nguyen and Tran — Verify-Gated Completion as Admission Control in a Governed Multi-Agent Runtime, arXiv, May 2026.
- Anthropic — Demystifying evals for AI agents, January 9, 2026.
- Anthropic — AuditBench, 2026.
- NIST NCCoE — Software and AI Agent Identity and Authorization, 2026.
Evidence note: The runtime-contract, proof-of-execution, Mnemosyne and verify-gated-completion papers are preprints. Their experiments support bounded architectural claims, not universal production reliability. The August 11 incident taxonomy is author-coded retrospective evidence. This article treats the convergence among these projects as an emerging systems-design direction, not as proof that one runtime architecture has already become a standard.