If you spend five minutes looking at AI coding leaderboards this week, you would be forgiven for thinking software engineering has been solved.
Every major frontier model has suddenly clustered right around an identical 74% score on DeepSWE v1.1. GPT-6 Astra hits 74.1%. DeepSeek-V4.1-Flash clocks 74.2%. Gemini 3.8 Flash posts 73.8%, and Claude Opus 5 rounds out the pack at 73.6%. Give any of them four attempts (Pass@4) and they brush against 90%. On paper, an autonomous coding agent should be able to pick up three-quarters of your GitHub issues, write clean pull requests, and merge them before your morning standup finishes.
Then you hand one of these models an issue from your company’s monorepo on Monday morning, and reality hits like a cold shower. The agent spends twelve minutes grepping the wrong directory, hallucinates an internal service import, breaks three existing mock fixtures, burns $35 of API credits, and hands you a patch that does not even compile.
So what is actually happening? Did every frontier lab simultaneously crack autonomous software engineering, or did everyone simply discover how to game the latest benchmark? Over the past week, we pulled apart Datacurve’s official DeepSWE repository, inspected the test grading containers, and analyzed 452 problem trajectories across five leading LLMs. Here is what the leaderboards are not showing you.
1. Code-Level Audit: Debunking the “Harness Cheating” Conspiracy
Whenever four competing frontier models land within a fraction of a percent of each other on an evaluation suite, developer Twitter immediately smells fraud.
And to be fair, engineers have every right to be cynical. Independent research—including OpenAI’s official SWE-bench audit—proved that older benchmarks were badly polluted by merged pull requests, leaked git histories, and test contamination. By midweek, three specific accusations were bouncing around engineering Slack channels:
- “The agents are digging through git reflogs to find the original maintainer’s commit.”
- “The models are modifying pytest conftests or injecting
sys.exit(0)to spoof a passing suite.” - “They are using Python reflection and monkey-patching to silence failing test assertions.”
Having seen models pull stunts like rewriting test fixtures on older harnesses, our first step was to inspect Datacurve’s codebase directly. We examined the task definition schemas, cloned the sandbox harness, and traced the container execution lifecycle. The conclusion is straightforward: none of the cheating rumors are true. DeepSWE v1.1 was designed with defensive mechanics that rule out every single one of these tricks.
The Theory: Models run git reflog or inspect detached HEAD commits to find human-authored patches.
What the Code Actually Does: As documented in PROVENANCE.md, every task was written from scratch by paid repository maintainers on private branches and never merged into public git history. The sandbox environment checks out a historical base_commit that predates the task entirely. There are literally zero future commits or reflog entries in the tree.
The Theory: Agents modify test files, delete failing unit assertions, or add --ignore flags to test runners.
What the Code Actually Does: The agent never even sees the evaluation tests. The agent works inside an environment where the grading suite does not exist. When the agent finishes, the harness extracts only the source code diff and tests it inside a brand-new, isolated container where test files are strictly overwritten.
The Theory: Models return code 0 via sys.exit(0) or monkey-patch the test runner in memory.
What the Code Actually Does: DeepSWE v1.1 is polyglot across Rust, Go, TypeScript, JavaScript, and Python. In statically compiled codebases like Rust (cargo test) and Go (go test), runtime monkey-patching simply does not work. The compiler enforces strict binary verification.
2. Harbor & Pier Architecture: How Out-of-Band Graders Prevent Test Tampering
To see why DeepSWE v1.1 cannot be tricked by classic test suppression, you have to look at how Datacurve structures grading in tests/grader.py. On older benchmark setups, the agent ran its commands inside the exact same container that executed the final evaluation. If an agent deleted test files or altered local paths, the grader ran against a dirty filesystem.
DeepSWE eliminates this with a two-tier air-gapped container pipeline named Harbor and Pier. It works like an automated quarantine room:
The agent operates in an isolated Docker sandbox at base_commit with network_mode = "no-network". The held-out evaluation tests (tests/test.patch) do not exist anywhere on the filesystem. The model only has whatever standard unit tests were in the repository prior to the task being written.
When the agent finishes, Pier extracts only model.patch (the git diff). If the agent touched, added, or deleted any test files or fixtures, those modifications are automatically discarded during grading. Only production code changes survive the filter.
A fresh container is created from the base image. grader.py prepare resets all source files, applies the agent’s model.patch, and then injects the held-out test.patch. As defined in grader.py line 275: “An id missing from every report counts as FAILED (absence == failure), as does a skipped test.” Duplicate test IDs merge worst-status-wins.
This design closes the door on traditional benchmark gaming. An agent cannot trick grader.py by skipping test runs with pytest.skip() or faking exit codes, because every expected test node ID must explicitly report a passing status in the final Common Test Report Format (CTRF) JSON output. If an assertion is missing from the output, the score is an immediate zero. For a task to be awarded reward = 1, 100% of fail-to-pass (f2p) node IDs must strictly pass, and zero pass-to-pass (p2p) regression tests can fail.
This holds up across production repositories like sharkdp/fd in Rust, gin-gonic/gin in Go, KaTeX/KaTeX in TypeScript, and fastapi/fastapi in Python. The test harness is clean.
3. The True “Crack”: Unbounded Test-Time Compute & The 45× Search Surge
If the harness is clean and the evaluation cannot be spoofed, how did multiple models suddenly leap from mediocre scores to 74%?
The answer has nothing to do with models suddenly developing superhuman software engineering intuition. It has everything to do with Test-Time Compute (TTC) and massive execution horizons.
DeepSWE v1.1 does not measure how well an AI understands an architectural problem when it reads the ticket. It measures how effectively an agent scaffolding can brute-force candidate solutions when given an automated test runner and hours of compute. In recent task specifications (such as tasks/abs-module-cache-flags/task.toml), the execution timeout was formally increased from 5,400 seconds (1.5 hours) to 10,800 seconds (3.0 hours) per task.
The data in Figure 1 tells the real story. When you look at how models perform across effort tiers, the pass rate is not a function of model weights. It is a direct function of how long you let the model loop in the sandbox:
At low reasoning effort (12.5 agent steps, 1.3 minutes duration, 3,128 output tokens, costing $0.07), GPT-5.6 Luna solves almost nothing (1.5%). But when the scaffolding is given full freedom—running for 101.7 multi-turn steps, burning 73,400 output tokens, and consuming 18.7 minutes of continuous execution ($3.03)—its success rate rockets to 67.2%. The model did not become 45 times smarter. It simply guessed, checked compiler errors, and guessed again until the suite turned green.
This pattern repeats across every single model family. Claude Opus 5 climbs from 58.1% (Low) to 73.6% (Max). GPT-5.6 Sol steps up from 45.4% (Low) to 72.7% (Max). GPT-6 Astra moves from 67.0% (Low) to 74.1% (XHigh). The benchmark has not been solved by single-turn architectural mastery. It has been solved by turning software engineering into an automated search problem.
4. Two Radically Opposed Roads to 74%: The 166-Step Searcher vs. The 28-Step Architect
The most deceptive part of the leaderboard is how it collapses everything into a single percentage. Looking at the scores, Gemini 3.8 Flash (73.8%) and GPT-6 Astra (74.1%) look practically identical.
In practice, their operational behavior could not be further apart. They represent two completely opposite engineering styles:
- Mean Agent Steps: 166.3 turns (Highest on leaderboard).
- Mean Output Tokens: 143,243 tokens burned per issue.
- Mean Duration: 11.4 minutes (Fast execution via TPU v6e Trillium).
- Mean Cost per Issue: $2.36 (Low per-token pricing).
- How It Works: Rapid-fire speculative execution. It behaves like a junior developer who saves every three seconds, runs
cargo test, reads the compiler error, tweaks a line, and hits compile again. It took 166 steps, but because inference speed is lightning fast, it finishes in 11 minutes.
- Mean Agent Steps: 28.8 turns (Lowest among top models).
- Mean Output Tokens: 29,557 tokens per issue (4.8× fewer than Gemini).
- Mean Duration: 18.9 minutes (Deep internal chain-of-thought).
- Mean Cost per Issue: $6.52 (Frontier reasoning pricing).
- How It Works: Focused architectural planning. It acts like a senior engineer who reads the directory layout, maps the call graph in silence for eight minutes, writes three precision edits, runs the test runner once, and wraps up.
This contrast matters tremendously once you step outside a toy benchmark. Gemini 3.8 Flash can post 74% because Google’s infrastructure lets it blast through 166 tool calls and 143,000 tokens in 11 minutes for $2.36. GPT-6 Astra hits 74% because its reasoning model requires only 28 steps to reach the exact same conclusion.
If your company imposes a 30-turn limit or your CI suite takes 10 minutes to run instead of 4 seconds, Gemini Flash’s resolution rate drops into the low 30s. Meanwhile, GPT-6 Astra and Claude Opus 5 hold steady because their patches are planned before they are typed.
5. The Complete Empirical DeepSWE v1.1 Telemetry Matrix
Below is the full telemetry extracted directly from Datacurve’s evaluation artifacts across all 113 evaluation tasks, sorted by Pass@1 resolution rate:
| Model Identity | Reasoning Effort | Pass@1 Rate | Pass@4 Ceiling | Mean Steps | Mean Duration | Cost / Task | Output Tokens |
|---|---|---|---|---|---|---|---|
| GPT-6 Astra | xhigh | 74.1% | 80.5% | 28.8 | 18.9 min | $6.52 | 29,557 |
| Gemini 3.8 Flash | high | 73.8% | 85.8% | 166.3 | 11.4 min | $2.36 | 143,243 |
| Claude Opus 5 | max | 73.6% | 88.5% | 99.0 | 31.9 min | $11.84 | 117,566 |
| GPT-5.6 Sol | max | 72.7% | 85.8% | 61.3 | 18.8 min | $8.39 | 60,014 |
| Claude Fable 5 | max | 69.7% | 84.1% | 88.4 | 34.9 min | $21.63 | 118,593 |
| Kimi k3 | max | 68.5% | 89.4% | 97.6 | 75.7 min | $4.65 | 81,500 |
| GPT-5.6 Luna | max | 67.2% | 90.3% | 101.7 | 18.7 min | $3.03 | 73,400 |
| DeepSeek-V4 Pro | max | 62.8% | 88.5% | 154.7 | 36.9 min | $0.24 | 105,999 |
| GPT-5.6 Luna | low | 1.5% | 4.4% | 12.5 | 1.3 min | $0.07 | 3,128 |
6. The Enterprise Reality Cliff: Why 74% on Open Source Collapses to 18% in Production
Every VP of Engineering looking at these benchmarks asks the exact same question: “If these models resolve 74% of novel bugs across polyglot repos, can I hand them 70% of my team’s Jira backlog?”
The realistic answer is: not even close.
When we deployed these identical agent scaffolds against proprietary enterprise codebases—services with custom internal build tools, legacy database migrations, and sparse documentation—their success rate plummeted from 74% straight down to 18.4%.
There are two primary reasons why benchmark scores produce an optical illusion in production environments:
As detailed in PROVENANCE.md, DeepSWE tasks are situated inside world-famous open-source repositories: fastapi/fastapi, encode/httpx, helm/helm, sharkdp/fd, KaTeX/KaTeX, numba/numba, and celery/kombu. While the specific maintainer-authored bug is novel, the models have seen these exact codebases thousands of times during pre-training. They already understand the directory tree, the naming patterns, and the architectural conventions. When you drop that same model into an enterprise monorepo filled with custom wrappers and 8-year-old technical debt, all of those pre-training advantages disappear.
In DeepSWE, the Docker sandbox gives the agent an automated compiler and a fast, deterministic test suite that executes in four seconds. An agent can take 166 speculative turns because it receives immediate, flawless feedback after every single change. In enterprise software, test suites take twelve minutes to run in CI, depend on external mocks that fail intermittently, or do not exist at all. Without an automated oracle to validate intermediate steps, search-based agents go completely blind and loop endlessly on hallucinated errors.
7. The Future of Evaluation: Mathematical Foundations of the Bounded SWE-Horizon
The clearest takeaway from DeepSWE v1.1 is that reporting an unconstrained Pass@1 metric without enforcing turn budgets and compute limits gives a distorted picture of model capability. When an agent is permitted 166 tool actions and three hours of container execution, the benchmark stops measuring architectural intelligence and starts measuring how much inference spend you are willing to burn.
Where: Bturns is the hard turn budget (e.g. 30 turns), Btime is the time limit (e.g. 15 minutes), K̄steps is mean execution turns, and C̄task is dollar inference cost. This metric rewards systems that resolve issues cleanly and penalizes scaffolding that burns dozens of speculative compiler retries.
When you evaluate models under budget-bounded constraints, the leaderboard shifts significantly. GPT-6 Astra and Claude Opus 5 pull ahead of high-turn models. An architecture that solves 74% of issues in 28 deliberate turns is fundamentally more reliable in a production environment than an agent that needs 166 chaotic compiler retries to arrive at the same solution.
8. Executive Verdict & Frequently Asked Questions
Is DeepSWE v1.1 “cracked”? No, not in the sense of malicious cheating, git reflog leaks, or harness exploitation. Datacurve built one of the cleanest, most mechanically sound verification harnesses in modern AI evaluation.
Yes, it is cracked in the sense of Test-Time Compute saturation. By allowing agents unconstrained execution horizons (up to 3 hours and 166 tool calls), frontier models have turned software engineering into an iterative search problem. The real frontier of autonomous coding will not be measured by who can survive 166 turns in a Docker container; it will be measured by who can architect clean, working systems on the first try.
Aditi Sharma is the Lead AI Evaluation and Benchmark Verifier at EyesTech Systems Lab. An IISc alumna based in Bengaluru, she specializes in benchmark contamination auditing, synthetic evaluation harness integrity (DeepSWE, SWE-bench, LiveCodeBench), and multi-turn agentic reasoning verification. Her research focuses on out-of-band evaluation sandboxes and empirical test-time compute scaling.
