ARC-AGI-3 is designed to test how AI agents learn and adapt inside unfamiliar environments. Image: ARC Prize.
OpenAI’s GPT-6 Astra has produced one of the most eye-catching benchmark results of the year: 99.95% on ARC-AGI-3.
That headline is real, but it needs an important footnote. The score came from ARC Prize’s Provider Adapter harness, a provider-specific evaluation setup that preserves Astra’s opaque reasoning state between requests and uses context compaction for long tasks. Under ARC Prize’s neutral Standard harness, the same model’s best observed score was 62.71%.
Those numbers do not mean that one result is fake. They answer different questions about the same model. They also show why benchmark headlines should include the test conditions, not just the biggest percentage.
What is ARC-AGI-3?
ARC-AGI-3 is an interactive reasoning benchmark from the ARC Prize Foundation. It places an AI agent inside novel, game-like environments without giving it a natural-language rulebook. The agent has to explore the environment, work out what matters, discover the goal, and plan a sequence of actions.
The benchmark is different from a conventional question-and-answer test. A model cannot simply retrieve a known fact or produce one final answer. It must learn from interaction over time. It has to update its beliefs when an action produces an unexpected result, remember useful discoveries, and execute a long-horizon plan with limited feedback.
ARC Prize says the environments are solvable by humans and that a 100% score represents beating every game with human-level efficiency. The benchmark measures skill acquisition, planning, adaptation and action efficiency, rather than only whether an agent eventually reaches a goal. ARC Prize’s ARC-AGI-3 overview explains the design in more detail.
ARC-AGI-3 is an interactive test rather than a static collection of puzzles. Image: ARC Prize on X.
Do you need a harness to run ARC-AGI-3?
Yes. In the practical sense, ARC-AGI-3 must be run through an evaluation harness. The harness connects the model to the game environment, supplies observations, accepts actions, carries the interaction forward, and records the result.
There is no published “raw model with no harness” score for GPT-6 Astra. A no-harness number is not one of the official ARC-AGI-3 conditions. The official reporting currently separates two conditions:
- Standard harness: a minimal, provider-neutral interface. The model can carry forward notes that it chooses to preserve in the visible conversation.
- Provider Adapter harness: a provider-designed context-management layer. For Astra, this preserves opaque reasoning state between requests and compacts long conversations so earlier work can remain useful.
So, when people say “Astra scored 62.7% without the special harness,” the technically accurate version is: Astra scored 62.71% with the Standard harness, without the Provider Adapter. That is not the same as running the model with no harness at all.
The two Astra scores
ARC Prize tested Astra on the ARC-AGI-3 Semi-Private set and published the following best results:
| Evaluation condition | Reasoning setting | Best observed score |
|---|---|---|
| Standard harness | Max | 62.71% |
| Provider Adapter harness | High | 99.95% |
The full scorecard also shows that the Provider Adapter result stayed close to 99% across several reasoning settings, while Standard harness results varied more widely. ARC Prize says the Provider Adapter runs were faster in aggregate and used fewer total tokens across the evaluated game-and-reasoning pairs.
The important point is that the model weights did not suddenly change between the two numbers. The surrounding system changed how the model’s continuing state was handled. In a long interactive task, that can be a major capability multiplier.
OpenAI says Astra helped Playco prototype and test games inside professional development tools. Image: OpenAI.
What does a harness actually improve?
A harness does not automatically make a weak model intelligent. It can, however, determine whether the model is able to use the information it has already discovered.
Imagine an agent playing a game for hundreds of turns. It may learn that a particular object changes direction after a switch is pressed, or that an apparent obstacle is actually a transport mechanism. If that discovery is lost when the next request is sent, the agent may repeat the same exploration. If the system preserves the discovery in a useful form, the agent can build on it.
ARC Prize describes Astra’s Provider Adapter as preserving opaque reasoning state and using compaction. In simpler terms, the model can carry more of its internal work forward while the conversation grows. The Standard harness still lets the model keep visible notes, but it does not provide the same provider-specific state-management path.
This is why ARC Prize reports both results instead of pretending they measure exactly the same thing. The Standard harness is better for an apples-to-apples comparison across providers. The Provider Adapter asks a different practical question: how well does a model perform when it is used with the context-management system its own provider designed for it?
What Astra appears capable of on ARC-AGI-3
The 62.71% Standard harness result is already substantial. It shows that Astra can solve many unfamiliar interactive environments through a common interface. The 99.95% Provider Adapter result indicates that Astra can use persistent state especially effectively when the evaluation setup lets it preserve more of its ongoing reasoning.
ARC Prize’s analysis of Astra’s replays highlights several behaviors:
- It built compact symbolic descriptions of unfamiliar game mechanics.
- It tracked objects, positions, rules and unfinished plans.
- It invented shorthand notation to make complex game states easier to manage.
- It created ordered plans and updated them as new evidence arrived.
- It used fewer actions than the tested human median on 96% of ARC-AGI-3 levels in the Provider Adapter evaluation.
These are meaningful agentic behaviors. They go beyond answering isolated questions. They suggest that Astra is better at turning interaction into a working model of an environment and then using that model to plan.
ARC-AGI-3 is still a bounded benchmark. Its worlds are closed-ended, deterministic and designed around a particular interface. A strong result shows that the model can perform the tested kind of exploration and planning; it does not prove that the model can autonomously handle every open-ended real-world situation.
Gemini and the Continual Harness comparison
The harness story is not unique to OpenAI. Researchers have also tested Gemini 3.1 Pro Preview with a third-party system called Continual Harness.
Continual Harness is designed to let an agent improve its working process during a run. It can update memories, skills, prompts and subagent definitions using evidence from previous trajectories. The project evaluated it on the 25-game public ARC-AGI-3 set using Gemini 3.1 Pro Preview and reported a 20.54% score.
That is an interesting result, but it should not be presented as a direct comparison with Astra’s 99.95%. The Continual Harness experiment used the public set, while Astra’s headline result came from ARC Prize’s Semi-Private evaluation. The systems were also different: Astra used OpenAI’s Provider Adapter, while Continual Harness used an independent self-improving agent architecture.
The fair conclusion is narrower: the harness can be a significant part of an ARC-AGI-3 system, and Gemini’s performance also changes when it is placed inside a more sophisticated continual-learning loop. It is not evidence that Gemini itself scored 20.54% under ARC Prize’s official Standard harness.
ARC Prize’s verified results provide useful context. The current leaderboard lists Claude Opus 5 at 30.16%, GPT-5.6 at 7.78%, Grok 4.6 at 2.11% and Grok 4.5 at 0.32% on ARC-AGI-3. Community systems and custom harnesses can report different numbers on public tasks, but those results should be labeled separately from official Semi-Private scores.
Is the Astra result marketing, or real capability?
It is both a real capability result and a marketing headline that needs context.
The result is real in the sense that ARC Prize published the test conditions, scorecard and replay-based evaluation. Astra did not merely receive an unexplained claim from a product launch slide. Under the Provider Adapter condition, it performed extremely well on the stated benchmark.
But “GPT Astra reaches 99% on ARC-AGI-3” leaves out the part that changes how readers should interpret the number. The score is for a complete evaluated system: model, reasoning setting, environment, harness and scoring procedure. It is not a pure measurement of the model’s weights in isolation.
The Standard harness result matters because it tells us how Astra performed under the common provider-neutral interface. The Provider Adapter result matters because real users and developers may want to know what Astra can do when its native context-management features are available. Both are useful, but they should never be collapsed into one undifferentiated claim.
Does this mean Astra is AGI?
No—not from ARC-AGI-3 alone.
ARC Prize itself says that saturating ARC-AGI-3 would not be proof of AGI. The benchmark measures a valuable slice of intelligence: learning unfamiliar rules, building temporary world models, planning over many steps and adapting from experience. It does not measure the full range of open-ended human ability, including ordinary physical interaction, social understanding, long-term autonomy in changing real-world institutions and the ability to set genuinely open-ended goals.
Astra’s result is better described as a major advance in agentic reasoning and interactive problem-solving. The 62.71% Standard result shows strong capability under a shared interface. The 99.95% Provider Adapter result shows how much a capable model can gain when its native reasoning state is preserved effectively.
That is a big deal. It is also exactly why benchmark literacy matters. The future of AI evaluation may not be about model scores alone. It may be about the performance of the whole system wrapped around the model—and whether that wrapper is common to everyone, provider-specific, or a continually improving agent of its own.
What people on X are saying
Public X reactions have split into two broad camps. Some users celebrated Astra’s near-perfect ARC-AGI-3 result and highlighted reports of faster computer-use workflows, coding and professional tasks. Others focused on the harness distinction and argued that a provider-specific evaluation should not be compared casually with a neutral benchmark score.
That disagreement is useful. It shows that the key question is no longer simply whether the number is 99% or 62%. The key question is: what system produced the number, what information could it preserve, and was the same setup used for the comparison?
Early-access demos and X posts can reveal how a model feels in practice, but they are anecdotes rather than controlled evaluations. They are best used as leads for further testing, not as substitutes for a scorecard.
The bottom line
GPT-6 Astra’s ARC-AGI-3 result is genuinely impressive, but the honest headline is more precise:
GPT-6 Astra reached 99.95% on ARC-AGI-3 with ARC Prize’s Provider Adapter harness, while scoring 62.71% under the Standard harness. No official raw no-harness score has been published.
The result shows real progress in interactive reasoning, memory management and long-horizon planning. It also shows that the harness is part of the measured system. Astra may be one of the strongest general-purpose agents publicly evaluated so far, but ARC-AGI-3 does not establish that it has achieved AGI.
