ARC-AGI-3 Is Testing Something Static AI Benchmarks Miss: Learning While Acting

A 30.16% score sounds ordinary until you know what is being scored

ARC Prize’s verified results page lists Claude Opus 5 at 30.16% on ARC-AGI-3 at High reasoning effort. The same page says Opus 5 completed five additional public-demo environments that no previously tested model had beaten. ARC Prize also notes that the ARC-AGI-3 run was evaluated only at High effort because the testing window was short.

For comparison, ARC Prize reports GPT-5.6 Sol at 7.78% on ARC-AGI-3 in its Max configuration. The gap is large enough to deserve attention, but it should not be translated into a statement such as “Opus 5 is four times as intelligent.” The scores belong to one benchmark with a particular interaction protocol, scoring rule, reasoning setting and agent implementation.

That caution matters because ARC-AGI-3 is deliberately unlike a conventional exam. A score does not represent the percentage of ordinary questions answered correctly. It mixes whether the agent finishes levels with how efficiently it acts relative to first-time human players.

From answering to discovering

ARC-AGI-3 makes the system discover the problem before it can solve it

The ARC-AGI series was created to focus on adaptation to novel tasks rather than recall of familiar knowledge. ARC-AGI-3 pushes that idea into an interactive setting.

The official competition describes four capabilities: exploration, modeling, goal-setting, and planning and execution. Those categories are tightly connected. The agent cannot plan effectively until it has some idea of how the environment behaves. It cannot build that model without taking actions and observing consequences. It may not even know which future state counts as success until it has explored enough to infer the goal.

For a professional outside AI research, the easiest analogy is entering unfamiliar software with the documentation removed. You can see the interface, press controls and observe what happens, but nobody tells you the workflow. A competent system must experiment without destroying its chances of success, form a working theory of the interface, revise that theory when evidence contradicts it and eventually perform the task.

That is closer to many real uses of AI agents than answering a static multiple-choice question. A procurement agent may have to discover which vendor portal contains the relevant record. A scientific agent may need to inspect a dataset before deciding which analysis is sensible. A coding agent may discover halfway through a change that the repository behaves differently from what its initial plan assumed.

The difficult part in all of those examples is not simply generating the next sentence. It is maintaining a useful model of an environment while acting inside it.

ARC-AGI-3 penalizes aimless trial and error, not just failure

The benchmark uses a measure called Relative Human Action Efficiency, or RHAE. The basic idea is straightforward: completing a level matters, but the number of environment-changing actions required to complete it matters too.

ARC Prize establishes a human baseline from people encountering each game for the first time. For a completed level, the score compares the human baseline action count with the AI agent’s action count, and squares that ratio. If the human baseline is 10 actions and the agent also takes 10, the level score is 1.0. If the agent takes 20 actions, the score falls to 0.25. Taking ten times as many actions as the human baseline produces a score of only 0.01.

This changes the incentive. An agent cannot obtain a strong result merely by clicking, probing or trying arbitrary moves until something works. Exploration is necessary, but wasteful exploration is expensive.

There is an important technical nuance. Internal reasoning steps, tool calls and retries that do not change the environment are not counted as ARC-AGI-3 actions. That means the benchmark measures external action efficiency, not the complete computational cost of reaching each action.

An agent could therefore spend much more inference compute thinking before it acts and still appear efficient under RHAE. That is not a flaw so much as a reminder that benchmark efficiency and economic efficiency are different quantities. A system that needs ten times more inference compute to save three environment actions may look better on RHAE while costing more to operate.

This is why AIUpdateWatch treats reasoning effort as an experimental variable rather than an invisible implementation detail.

A benchmark of systems, not just weights

The model is no longer the whole system being evaluated

Interactive benchmarks make a long-running problem in AI evaluation impossible to ignore: the same underlying model can behave very differently depending on the machinery wrapped around it.

An agent can maintain notes, construct a programmatic world model, replay earlier observations, use a verifier, separate planning from execution, or delegate specific work to another component. These choices affect how the system explores and how quickly it recovers from a wrong hypothesis.

Recent ARC-AGI-3 research illustrates the effect. A July paper on a system called Tycho tested several orchestration policies that build programmatic world models while interacting with the public ARC-AGI-3 games. Under the selected policy, the authors report that both GPT-5.6 Sol and Claude Opus 5 reached 100 RHAE on the public games and completed all 183 levels. The Opus 5 run used fewer scored actions than the aggregate human baselines.

That does not contradict ARC Prize’s lower verified model results. It is measuring a different system configuration. The Tycho work adds an agent architecture designed specifically around active abstraction, model building and verification. The ARC Prize result is useful precisely because it provides a standardized verified configuration for comparison.

The practical consequence is significant. The industry is entering a period in which asking “Which model is best?” becomes less informative than asking “Which model, with which reasoning budget, tools, memory, verifier and orchestration policy, performs best on this workflow?”

That is also why a model name alone is increasingly insufficient for reproducible comparison.

A system can saturate the public games without solving the benchmark

The Tycho result is technically interesting, but the authors themselves give the most important warning: performance on public environments should not be confused with performance on held-out evaluation environments.

Public benchmark tasks serve several legitimate purposes. Researchers need something they can inspect, replay and debug. Public tasks make it possible to identify failure modes, compare agent designs and improve engineering. The problem begins when repeated exposure turns the public set into a development target.

Once researchers know the games, they can build better abstractions, prompts, planners and verifiers around their observed structure. Even without directly hard-coding game-specific solutions, the engineering process can gradually adapt to the public distribution.

This is a broader benchmark-governance problem, not something unique to ARC-AGI-3. Any widely studied evaluation eventually becomes part of the environment in which models and systems are developed. Scores remain useful, but the meaning changes.

ARC-AGI-3 tries to preserve the distinction through private and semi-private evaluation. For readers assessing headline numbers, the rule is simple: public-set saturation is evidence of what an engineered system can do on known evaluation structure; held-out performance is stronger evidence of generalization.

The site's broader AI benchmark registry follows the same principle by keeping benchmark scope and methodology visible instead of collapsing different evaluations into one universal intelligence score.

Reading the current leader correctly

What Claude Opus 5’s verified result actually establishes

The strongest part of the Opus 5 result is not the marketing comparison. ARC Prize administers and publishes the verified score, which gives readers an evaluation source independent of Anthropic’s own launch material.

The result also contains useful behavioral evidence. ARC Prize reports that Opus 5 completed five public-demo environments that previously tested models had not beaten. A replayable interactive environment provides more diagnostic information than a single pass/fail answer because researchers can inspect where the agent explored, committed to a hypothesis, corrected itself or became stuck.

At the same time, three limits should remain attached to the number.

1. It is one benchmark. ARC-AGI-3 focuses on novel interactive reasoning in abstract environments. It does not measure every form of professional competence.
2. Reasoning effort matters. Opus 5’s ARC-AGI-3 result was reported at High effort, while model families can expose multiple inference budgets. A score without its effort setting is incomplete.
3. Agent design matters. Research systems can achieve very different public-set results by changing the wrapper around the model, even when the base model stays the same.

Those caveats do not make the result unimportant. They make it more useful. A benchmark becomes informative when we understand what caused the score rather than treating the number as a single ranking of intelligence.

For businesses and professionals, the lesson is bigger than ARC-AGI

Most organizations will never run ARC-AGI-3. They should still pay attention to what this evaluation style is revealing.

AI systems are increasingly purchased for processes rather than isolated answers. A legal team may want an agent to gather records, compare clauses and prepare a structured review. A hospital may want software that navigates scheduling, documentation and coding systems. A finance group may want an agent to reconcile information across several tools. An operations team may want software that notices an exception, finds the relevant record and proposes the next action.

In these settings, raw model knowledge is only one component of performance. The system also needs to understand state, decide what information is missing, use tools correctly, retain relevant context, recover from failed actions and know when a human must take over.

That changes procurement and evaluation. A strong model benchmark should be treated as a reason to test a system, not as a substitute for testing it.

This is one reason interactive evaluation is likely to become more important. Static tests are still valuable for capabilities such as mathematics, coding and document understanding. They simply cannot tell us enough about systems expected to operate over time.

What would count as stronger evidence

The next important ARC-AGI-3 results will be about transfer, not another public-set record

The public leaderboard will continue to move, but four signals would tell us more about genuine progress.

• Held-out performance. Systems that preserve strong results on semi-private or private environments provide better evidence that the strategy generalizes beyond development examples.
• Standardized agent configurations. Clear reporting of the harness, memory, tools, reasoning budget and retry policy will make comparisons more reproducible.
• Compute-aware efficiency. RHAE measures environment actions. A complementary view of inference cost, latency and tool usage would show whether better action efficiency is economically efficient too.
• Failure-mode analysis. Replays can reveal whether systems fail because they misread the environment, forget evidence, choose the wrong goal, plan poorly or execute badly. That is more actionable than a leaderboard position alone.

The larger shift is already visible. As AI becomes more agentic, evaluation has to move closer to the conditions under which agents actually fail: incomplete information, uncertain goals, costly actions, changing state and the need to recover from mistakes.

ARC-AGI-3 is not a final test of intelligence. It is a useful example of why the next generation of benchmarks will increasingly evaluate behavior over time, not just answers at the end.

Sources

Primary research and benchmark documentation

• ARC Prize — ARC-AGI-3 Competition and capability definitions
• ARC Prize — ARC-AGI-3 scoring methodology and Relative Human Action Efficiency
• ARC Prize — Verified Claude Opus 5 ARC-AGI results
• ARC Prize — Verified GPT-5.6 series ARC-AGI results
• ARC Prize Foundation — ARC-AGI-3: A New Challenge for Frontier Agentic Intelligence
• Lehmann, Aioanei and Vahdati — Tycho: Active Abstraction with Programmatic World Models for ARC-AGI-3
• Rodionov — Do Coding Agents Need Executable World Models, Simplification, and Verification to Solve ARC-AGI-3?

Benchmark results describe performance under specific evaluation settings. They should not be read as universal rankings of intelligence or as direct estimates of real-world professional performance.