When Reasoning Models Learn to Execute: Why Native Tool Search Moves AI From Rhetoric to Verification

Related Daily Briefing

September 28, 2026: OpenAI releases o3-pro alongside TSMC’s 2nm N2P silicon tape-out

The September 28 Daily Briefing records the launch of o3-pro and Alibaba’s open-weight Qwen 2.5-Coder-32B-Reasoner as headline developments. This article examines the systems architecture, verification mechanics and economic consequences behind test-time tool search.

Open the September 28 report

The immediate development

The benchmark leap matters less as a high score than as proof that linguistic reasoning reached its limit

When OpenAI published the system card for o3-pro on September 28, 2026, the figure that captured commercial attention was 78.4% on SWE-bench Verified—a benchmark of real-world GitHub issues that had resisted previous foundation models. Simultaneously, Alibaba released Qwen 2.5-Coder-32B-Reasoner under an Apache 2.0 license, demonstrating that a 32-billion-parameter open-weight model running on two consumer-grade graphics cards could solve 65.8% of the same software engineering tasks.

Those numbers represent a genuine performance discontinuity. Throughout 2024 and 2025, progress on software engineering benchmarks crept upward by single-digit percentages, largely through pre-training scale, higher-quality synthetic data and post-training reinforcement learning. Yet frontier models repeatedly stumbled over complex, multi-file software modifications, frequently hallucinating imports, misunderstanding existing test harnesses or introducing subtle regressions.

The breakthrough in this latest generation does not stem from another tenfold expansion in pre-training compute. It comes from changing what happens during inference. Instead of generating a single speculative chain of thought and hoping its internal logic holds, these systems execute intermediate code snippets, invoke linters, query filesystem trees and run automated unit tests inside a sandboxed environment while they are reasoning.

The distinction is fundamental. A model that only generates tokens operates in the domain of rhetoric: its goal is to output a sequence that sounds plausible according to statistical distributions. A model that executes intermediate code operates in the domain of verification: its candidate thoughts are subjected to empirical tests where syntax errors, mismatched types and failing assertions immediately falsify a reasoning branch. Understanding this mechanism clarifies both why the benchmark scores jumped and why these models introduce severe operational constraints.

The rhetorical trap: why pure chain-of-thought compounds early errors

To understand why execution during reasoning is necessary, one must examine why autoregressive chain-of-thought fails on long-horizon technical problems.

Standard reasoning models—such as the early OpenAI o1 releases or DeepSeek-R1—generate thousands of internal "thinking tokens" before returning a final answer. This hidden reasoning allows the model to break problems into sub-goals, explore alternatives and backtrack when a step seems unpromising. However, that entire deliberation process remains trapped inside the model's vocabulary space. The model thinks in language, but it cannot observe the physical consequences of its ideas.

This creates what software engineers call the rhetorical trap. Large language models are trained to produce highly convincing prose. If a model makes a subtle error in step 3 of a 40-step technical derivation—such as assuming an off-by-one index, misremembering an API signature or assuming a file exists when it does not—every subsequent token is generated conditioned on that false premise. Because the model has been trained to maintain narrative coherence, it will smoothly rationalize the error, constructing elaborate, mathematically sophisticated explanations to justify a conclusion that was broken from the start.

In creative writing or open-ended strategic analysis, plausible rhetoric is often acceptable. In software engineering, quantitative finance, structural design and medical diagnostics, it is catastrophic. A program either compiles or it crashes; an algorithm either terminates in polynomial time or it overflows the stack. Without an external reality check, long reasoning chains do not necessarily increase correctness—they frequently amplify self-deception.

Three generations of tool use: ReAct, isolated deliberation, and interleaved search

The software industry has attempted to solve this limitation for several years, passing through three distinct architectural phases.

The first generation was the ReAct (Reasoning and Acting) loop, popularized in 2023. In a ReAct architecture, an external application orchestrates the interaction. The model generates a textual plan, emits a structured JSON tool call (such as a database query or bash command), and halts generation. The host application executes the tool, captures the output, appends that output back into the prompt history as a new message, and calls the model again. While practical for basic tasks, ReAct is inherently linear and client-managed. If an agent pursues a flawed plan, it pollutes its context window with thousands of tokens of failing attempts, rapidly exhausting its context window and degrading attention focus.

The second generation was isolated test-time deliberation. Systems like OpenAI o1 decoupled reasoning from the client loop, allowing the model to generate sprawling internal deliberation traces before answering. This dramatically improved performance on Olympiad-level mathematics and logic puzzles, where problems can be solved deductively through pure symbolic manipulation. But for empirical engineering, isolated deliberation was blind: the model could spend three minutes contemplating a refactor without being able to verify whether the target repository even used Python 3.11 or 3.12.

The third generation—exemplified by o3-pro and Qwen 2.5-Coder-32B-Reasoner—is interleaved test-time tool search. Here, tool execution is not an external wrapper, nor is reasoning isolated from reality. Instead, external execution environments are directly embedded into the model’s internal search algorithm. During inference, the system explores multiple potential solutions organized as a tree. When a candidate branch proposes a code modification, the serving runtime immediately executes that code inside an ephemeral container, inspects the exit code and compiler output, and feeds those empirical signals directly back into the search policy.

How sandboxes act as search operators: tree pruning through compiler feedback

The technical core of interleaved reasoning lies in how external tools interact with Monte Carlo Tree Search (MCTS) and Process Reward Models (PRMs).

In standard test-time search, an auxiliary model known as a Process Reward Model evaluates intermediate reasoning steps, assigning a heuristic probability score indicating whether a particular derivation step is likely to lead to a correct solution. While PRMs represent a major advance over scoring only the final outcome, training a neural network to reliably judge complex, domain-specific code edits without running them is extraordinarily difficult. Subtle race conditions or type mismatches easily fool the verifier.

Interleaved tool execution replaces heuristic guessing with empirical ground truth. When the generative model proposes a patch, the search engine does not ask a second neural network whether the patch looks plausible. Instead, it dispatches the patch to a lightweight virtual machine sandbox that runs a suite of automated checks:

Syntax and Abstract Syntax Tree (AST) validation: Does the modified file parse cleanly?
Static type checking: Does the modification satisfy compiler or type-checker constraints (such as TypeScript or mypy)?
Isolated regression execution: Does running existing unit tests fail or pass?
Runtime behavior analysis: Did the code trigger an unexpected recursion, memory leak or timeout?

If the sandbox returns an exit code of 1 alongside a stack trace, the search algorithm immediately drops the value of that search node to zero and prunes the entire branch. The model does not need to waste tokens rationalizing why its code should have worked; the compiler has settled the question. Conversely, when a patch passes existing tests and resolves the targeted failure, that branch receives a high reward, allowing the search policy to deepen its exploration along that verified path.

This dynamic shifts the primary computational bottleneck in AI systems. The limiting factor is no longer just how many matrix multiplications a GPU cluster can perform per second; it is how quickly a virtualization layer can instantiate clean environments, execute arbitrary code and return structured execution telemetry to the inference scheduler.

The dollar cost of verified truth: why thinking tokens transform inference economics

Empirical verification delivers unprecedented accuracy, but it demolishes the conventional economic model of large language model serving.

In traditional conversational AI, inference costs are predictable and modest. An enterprise querying an API with 1,000 input tokens and receiving 500 output tokens pays fractions of a cent. Even with frontier foundation models, standard generation costs rarely exceed $0.02 per interaction.

Interleaved test-time tool search operates on an entirely different economic scale. According to OpenAI's published pricing for o3-pro, input tokens cost $15.00 per million, cached inputs cost $3.75 per million, and output tokens—which explicitly include hidden thinking tokens—cost $60.00 per million. When a model tackles an ambiguous, multi-file software defect on SWE-bench, it may explore 15 to 25 distinct reasoning branches. Each branch generates thousands of tokens of planning, code drafting, execution observation and reflection.

A single difficult engineering query can easily consume 40,000 to 60,000 internal thinking tokens, alongside a dozen container executions. At $60.00 per million tokens, that single query costs between $2.40 and $3.60. If the task requires extensive search across deep trees, costs can exceed $5.00 per attempt.

This reality divides the market into two distinct economic regimes:

High-value task justification: For a commercial software team, paying $3.00 to automatically diagnose and fix a complex regression that would take an experienced software engineer three hours to investigate represents an astronomical return on investment. The human labor cost dwarfs the API invoice.
Unviable mass-market chat: Applying this architecture to consumer chat, general search or real-time assistants is financially impossible. A consumer paying a $20 monthly subscription would exhaust their entire subscription fee after four or five deep queries.

This is where open-weight systems like Alibaba's Qwen 2.5-Coder-32B-Reasoner alter industry dynamics. By demonstrating that a 32-billion-parameter model can execute step-level process reward search on dual RTX 4090 GPUs (consuming approximately 600 watts of power), Alibaba proves that test-time tool execution can be internalized by engineering organizations willing to manage their own hardware, bypassing recurring per-token cloud charges.

Furthermore, this architecture makes context caching economics essential. Because the search tree repeatedly references the same repository codebase and baseline instructions across different branches, serving systems that fail to cache intermediate Key-Value (KV) attention states will face prohibitive prefill costs and unsustainable memory overhead.

Compounding latency: the transition from interactive chat to asynchronous engineering batches

Cost is not the only penalty of empirical verification; latency expands even more dramatically.

For a decade, user interface design for artificial intelligence has been optimized for immediate conversational feedback. Users expect a response to stream onto their screens within 500 to 1,500 milliseconds. If generation stalls for more than three seconds, users perceive the system as unresponsive.

Interleaved reasoning models cannot operate within conversational time budgets. The latency of a test-time search tree is governed by the critical path of synchronous tool execution. Consider a realistic debugging scenario:

The model spends 4 seconds generating an initial diagnostic plan (150 tokens).
It invokes a search tool to locate relevant files across a 200,000-line codebase (1.5 seconds).
It reads the source files and generates three alternative patches across three parallel branches (8 seconds).
It dispatches each patch to a container to run an automated test suite. Even a well-optimized test suite takes 6 seconds to set up dependencies, compile and execute (6 seconds).
Two branches fail; one passes partially but introduces a regression. The model inspects the error logs and generates a second iteration of repairs (10 seconds).
It reruns the test suite to verify the secondary fix (6 seconds).
It synthesizes its findings, writes documentation and produces the final pull request (5 seconds).

The total elapsed wall-clock time for this single request is over 40 seconds. For deep search budgets on intractable problems, elapsed time routinely spans two to four minutes.

This marks the death of the chat interface for complex technical work. Systems relying on test-time tool execution cannot be packaged as interactive text boxes. They must be deployed as asynchronous background workers: a developer assigns an issue in a tracker, the agent works independently in a sandboxed staging environment, and the human reviews the completed pull request and verified execution trace an hour later. The mental model shifts from talking with an assistant to delegating work to an offshore junior engineer.

What SWE-bench Verified 78.4% really measures—and what it still misses

Because benchmark numbers are frequently weaponized by vendor marketing departments, evaluating what 78.4% on SWE-bench Verified actually signifies is essential for maintaining analytical rigor.

SWE-bench, introduced in late 2023 by researchers at Princeton and the University of Chicago, evaluates language models on real GitHub issues pulled from popular open-source Python repositories (including Django, SymPy, pytest and scikit-learn). To succeed, an AI agent must read a natural-language issue description, navigate a large and unfamiliar codebase, identify the bug, write a patch that modifies one or more files, and ensure that the project's existing unit tests pass while newly written tests verifying the fix also succeed.

The original benchmark contained significant noise: some repository environments were broken, certain test suites were flaky, and some issues had underspecified requirements. In response, OpenAI and independent researchers created SWE-bench Verified, a curated subset of 500 problems verified by human software engineers to have clear problem statements, reproducible environments and deterministic unit tests.

A score of 78.4% on SWE-bench Verified is a historic milestone. It proves that when provided with compiler feedback and a sandboxed test runner, an AI model can reliably resolve well-scoped bugs in mature, well-tested codebases. It is no longer a toy demonstration.

However, relying on this single metric to claim that "software engineering is solved" confuses solving isolated test suites with the full scope of engineering practice. SWE-bench Verified exhibits structural constraints that favor test-time search:

Unit tests already exist: In SWE-bench, the repository already has a comprehensive, deterministic test harness. The model can use those pre-existing tests as an infallible oracle to verify its changes. In real-world enterprise codebases, comprehensive tests are often missing, outdated or flaky. A model that relies on test execution cannot verify its work if the tests do not exist.
Single-repository confinement: The benchmark tasks take place within a single self-contained Python repository. They do not test an agent's ability to coordinate changes across a distributed microservices architecture, migrate a production database with zero downtime, or balance trade-offs between cloud infrastructure costs and query latency.
Zero ambiguous human negotiation: The problem statements are fixed text descriptions. In real organizations, the most difficult part of software engineering is clarifying ambiguous, contradictory or changing business requirements through communication with stakeholders.

As AIUpdateWatch documented in our analysis of the coding verification bottleneck, making candidate code cheap to generate simply pushes the organizational constraint downstream into system architecture, integration testing and deployment governance.

The security perimeter: indirect tool poisoning and sandbox escape boundaries

Granting an AI reasoning model the ability to execute code and query external environments during inference creates a profound cybersecurity hazard.

In standard model serving, the security boundary is straightforward: the model receives text inputs and returns text outputs. The worst a malicious actor can achieve through direct prompt injection is eliciting toxic text or bypassing safety filters. The model itself cannot touch file systems, execute shell scripts or interact with internal corporate networks.

Interleaved tool execution shatters that isolation. If a model has access to a bash shell or a web browser to inspect documentation while deliberating, it becomes vulnerable to indirect prompt injection and tool poisoning. Consider an autonomous coding model inspecting an open-source library to debug a client dependency. An attacker could embed malicious instructions inside a hidden comment in an HTML documentation page or a README file:

Because the model reads this input as part of its internal reasoning search, it may interpret the text as an authentic system instruction, executing the malicious shell command inside its sandbox. As demonstrated by recent joint audits from the US and UK AI Safety Institutes, visual prompt smuggling in desktop agent screenshots bypasses up to 88% of standard guardrails.

To deploy test-time execution safely, infrastructure operators must enforce strict architectural isolation:

Hardware-enforced MicroVMs: Tools must execute inside lightweight virtual machines (such as AWS Firecracker) that provide hardware-level kernel isolation, preventing container-escape vulnerabilities from compromising the host GPU server.
Ephemeral read-only overlays: File systems must be instantiated from clean, ephemeral snapshots that are completely destroyed upon query completion, preventing persistent state contamination.
Strict network egress blocking: The execution environment must have outbound internet access disabled by default, or restricted through a zero-trust proxy that allows connections only to pre-approved package registries.
Execution budget limits: Hard limits on CPU cycles, memory allocation and execution time must be enforced to prevent algorithmic denial-of-service attacks designed to trap the reasoning engine in infinite computational loops.

Beyond software: how execution moves AI from statistical mimicry to empirical science

While software engineering is the first discipline transformed by interleaved execution, the underlying principle is universal across technical domains.

For any professional field governed by rigorous formal rules, pure language models have been frustratingly unreliable. They write fluent legal briefs that cite non-existent precedents; they draft financial valuations with subtle accounting arithmetic errors; they propose chemical synthesis routes that violate thermodynamic laws. In every case, the model fails because it relies on statistical word associations rather than computational reality.

Interleaved execution extends the scientific method into AI cognition. Whenever a formal verifier or simulation engine exists, it can be integrated as an operator in the model's test-time search tree:

Quantitative Finance and Economics: Rather than predicting financial outcomes through rhetorical narrative, an economic reasoning model writes a numerical simulation, executes it against historical tick data, observes risk metrics and parameter sensitivities, and refines its thesis based on empirical distributions.
Molecular Biology and Drug Discovery: A computational biochemistry model designing a novel enzyme does not simply output an amino acid sequence; it dispatches the candidate structure to an automated docking simulator, checks binding affinities, prunes non-viable conformations and returns only molecules with verified physical stability.
Structural Engineering and Architecture: An engineering model calculating load-bearing requirements for a bridge or airframe submits candidate designs to finite-element analysis solvers, iteratively modifying geometry until mechanical stresses fall within regulatory safety margins.
Formal Mathematical Proofs: In pure mathematics, models can pair with formal interactive theorem provers like Lean 4. Every step in a proposed proof is checked by the Lean kernel; if a tactic fails to compile, the branch is discarded, eliminating hallucinated mathematical claims entirely.

This is the true significance of the architectural transition. By binding generative language models to deterministic verification engines, AI moves from an era of generative mimicry into an era of automated, reproducible empirical discovery.

What to watch next: the metrics that will prove whether execution scaling is durable

As interleaved reasoning systems move from research announcements into commercial production, several critical indicators will determine their practical adoption and long-term viability:

Asynchronous API adoption: Track whether major cloud providers replace synchronous HTTP streaming endpoints with event-driven, webhook-based job architectures designed for multi-minute reasoning tasks.
Local execution orchestration: Monitor whether open-source inference runtimes (such as vLLM, SGLang and Ollama) introduce native, standardized interfaces for local Docker and Firecracker sandbox execution.
Reasoning token price deflation: Watch whether advanced search tree pruning algorithms reduce the average number of thinking tokens required to solve complex tasks, lowering the per-query dollar cost.
The emergence of "compilation-passing" regressions: Evaluate whether models develop subtle failure modes where candidate code passes local unit tests while violating broader system invariants, security boundaries or performance constraints.
Enterprise sandbox security standards: Watch for formal regulatory and industry standards governing network isolation, credential protection and prompt injection defenses in commercial agent sandboxes.

Interleaved tool execution proves that making models "smarter" is no longer just a question of collecting more internet text or building larger GPU clusters. The frontier has moved into systems engineering: integrating compilers, sandboxes, reward models and search algorithms into an architecture that tests its own thoughts before presenting them to the world.

Sources