Back to Writing
Model & Benchmark AnalysisFebruary 18, 20267 min read

AI Benchmarks I: Gemini 3 Scored 85% on ARC-AGI-2. What That Actually Proves.

Gemini 3 Deep Think set a record on ARC-AGI-2. Is it genuine reasoning progress, or smarter search amplified by compute? What ARC-AGI measures, what it does not, and why benchmark wins don't equal field-readiness.

Co-authored with Milind Chandramohan

BenchmarksARC-AGIReasoningInference Scaling

Gemini 3.0 Deep Think achieved nearly 85% on ARC-AGI-2 — a new record. Is this genuine progress in reasoning, or just smarter search amplified by more compute?

Three questions arise immediately: Is this AGI progress? Is it benchmark gaming? Is it useful for production?

What ARC-AGI is

Traditionally, IQ was assessed with tests like Raven's Progressive Matrices. Those failed for LLMs, which relied on statistical shortcuts, dataset contamination, or verbalization hacks instead of true abstraction. ARC-AGI — the Abstract Reasoning Corpus for Artificial General Intelligence — was created in 2019 by François Chollet to minimize data contamination, focus on abstract reasoning, and resist brute-force memorization.

Diagram
A sample ARC-AGI-2 task — grid transformations that require abstraction and generalization rather than recall.
  • ARC-AGI-1 was solvable with sufficient scaling and chain-of-thought.
  • ARC-AGI-2 requires deeper abstraction and generalization.
  • ARC-AGI-3 pushes further toward unseen transformations.

The feat of Gemini 3.0 Deep Think

In February 2026, Gemini 3.0 Deep Think scored 96% on ARC-AGI-1 and 85% on ARC-AGI-2. How to interpret this:

  • ARC-AGI-1 is effectively saturated.
  • ARC-AGI-2 is now the differentiator.
  • Inference scaling, along with program synthesis, appears decisive.
  • Humans still dominate ARC-AGI-2 (the average human scores around 60%).

The breakthrough came because Gemini 3.0 Deep Think was able to:

  • reason deeply via exploration, verification, and iterative refinement — Type 2 reasoning in the Kahneman framework
  • think for several minutes — inference scaling, another leap forward
  • programmatically synthesize solutions by writing code to evaluate them

Why this benchmark suddenly matters

The model solved problems requiring deep abstraction and generalization, narrowing the gap on abstract reasoning benchmarks. But benchmarks measure constrained abstraction, not general intelligence.

What it does not mean

A few things remain unclear from these tests:

  • Does this generalize outside grid puzzles?
  • Is this reasoning or heuristic search?
  • How much inference compute was thrown at it? Is the cost per task scalable?

Importantly, ARC-AGI does not test field-readiness. From the lens of deployment and agentic AI, it does not measure long-term planning, agency, world models, tool reliability, memory stability, or real-world robustness.

The next race may not be about bigger models. It may be about clear, disciplined reasoning under resource constraints.

The benchmark series ahead

As LLMs achieve breakthroughs, we plan to add context on the benchmarks that matter:

  • MMLU, GPQA, and the illusion of knowledge — understanding vs. recall.
  • SWE-Bench and real engineering — does passing mean you can ship production software?
  • Agent benchmarks (or the lack thereof) — measuring multi-step, stateful, tool-driven agents.
  • Toward domain-specific benchmarks — are generic benchmarks useful in real industries?

In our internal evaluations, we found that real-world task drift occurs much earlier than benchmark drift. More on that in future articles.