Pixel Canary’s 90% Next.js Agent Evals score did not translate into a working result in our Cline Desktop test. A Three.js scene and a Python reporting tool both ended with socket errors before producing code. The attempts overlapped: about 38 minutes passed from the first submission to the final error, while their individual recorded durations add up to about 46 minutes. These are delivery failures, not a measured score for the model’s coding quality.
What the 90% benchmark actually says
The Next.js Agent Evals leaderboard, last run September 25, 2026, lists Pixel Canary at 90% success, tied with GPT 6 Astra (high) and above Kimi K3’s 84%. Vercel’s launch note describes the baseline as 28 successful tasks out of 31, or 90.3%; with bundled Next.js documentation in AGENTS.md, it reports 30 out of 31, or 96.8%.
That score is pass@4. A task counts as successful when any of four attempts passes. The leaderboard says infrastructure failures are discarded and rerun. Its Pixel Canary row uses OpenCode, whereas our tests used Cline Desktop and gave each task one attempt. The benchmark does not report the chance that a single Cline run will finish without a connection error.
The same leaderboard records an average duration of 1,015.80 seconds for Pixel Canary versus 251.89 seconds for GPT 6 Astra (high) on its eval set. That is about four times as long in that harness. It is a reason to examine wall-clock cost, but it is not a tokens-per-second measurement or a diagnosis of the errors we saw.
The setup and the boundary of our test
We used Cline Desktop v0.0.37 on macOS, Cline Usage-Billing, Pixel Canary (free), and the Extra thinking setting. The fixtures were small and synthetic. Each prompt requested a concrete implementation, a build or tests, and an honest final account of what was verified. We inspected the saved Cline transcripts and the source files after each run. We did not run another model on these fixtures, so this is a Cline field test, not a controlled model ranking.
Two earlier setup attempts under the macOS Documents directory failed with Operation not permitted while Cline tried to use the shell. We cancelled them and moved clean copies of the fixtures to /tmp/pixel-canary-20260926/. Those permission errors are excluded from the 0/2 result because the workspaces were not accessible. The two attempts below are the ones in which Cline could read the projects.
The planned Next.js repair task was not run. The benchmark’s Next.js result is a public source we analyzed, not a claim that our own Next.js task failed.
Three.js: project inspection, then a 20-minute thinking gap
The first prompt asked for a Vite and TypeScript solar-system demo built with Three.js. A complete result would have shown a moving Sun, Earth, and Moon; orbital paths and lighting; pause, reset, speed, and focus controls; object picking; keyboard access and reduced-motion handling; resize behavior; cleanup of listeners and GPU resources; and a successful build. The project was an intentionally plain starter with Three.js already installed.
We submitted the clean run at 12:21:45 IST. At 12:24:30, Cline’s saved conversation shows its first batch of shell commands: it listed the project, read package and TypeScript configuration, checked index.html, and inspected installed packages and Node/npm versions. At 12:27:08, it read src/main.ts and src/style.css and checked the available Three.js controls and type package. Those are sensible reconnaissance steps, but neither batch wrote an implementation.
After the second batch, the Cline UI records “Thought for 1201s”—just over 20 minutes without another visible tool action. The saved conversation ends at 12:47:09 with the socket-connection error shown below. During the live observation the session had appeared completed earlier, around 12:38, while its last visible item was still “Thinking”; the saved transcript’s later error timestamp is why we use 25 minutes 24 seconds as the recorded submission-to-error duration.
Open the screenshot at full size to inspect the thinking duration and the error text.
The starter src/main.ts remained 156 bytes and src/style.css remained 120 bytes, both with their original modification times. There was no new scene, no final response, and no build by Cline. We therefore cannot grade motion, controls, mobile behavior, accessibility, or rendering quality: no application was delivered to test.
Python: inspection commands, then an 800-second thinking gap
The second task was less visual and more deterministic. The prompt asked Cline to finish a report.py CLI that reads a UTF-8 CSV with date,category,amount, groups exact decimal money totals by month and category, writes sorted JSON, rejects malformed nonblank rows with line-numbered errors, avoids partial stdout on failure, adds focused unit tests, and runs them. The valid fixture included 0.10 and 0.20 in the same category, a negative refund, and another month. The invalid fixture included the impossible date 2026-02-30. These cases would have exposed float rounding and date-validation mistakes had code been produced.
We submitted this run at 12:39:37 IST, while the Three.js session’s eventual error had not yet appeared in its saved record. At 12:42:00, Cline listed the Python workspace and tried git status; the fixture was not a Git repository, so that command exited 128. At 12:44:24, it read report.py, valid.csv, and invalid.csv, checked Python and pytest, and tried cat -A to inspect the CSV. The macOS version of cat rejected -A. Cline recovered by running od -c on the fixtures at 12:46:39.
The git and cat -A errors were minor inspection missteps. Neither explains the final failure; Cline continued after them. The last visible phase was “Thought for 800s”, followed by the same socket-connection message at 12:59:59. The recorded submission-to-error duration was 20 minutes 22 seconds.
Open this screenshot at full size to see the 800-second pause and the selected Extra thinking setting.
report.py remained the original 354-byte stub with NotImplementedError, and no unit-test file appeared. Cline never ran the requested test suite. We cannot say whether its implementation would have handled exact decimal sums, invalid dates, or clean error output, because it supplied no implementation.
The shared failure, and what we cannot infer from it
Both clean sessions ended with Cline reporting: “The run failed: The socket connection was closed unexpectedly.” The repeated symptom matters. In this setup it prevented a usable artifact twice, even though the agent could read the workspaces and perform inspection commands. A developer waiting for a finished task would have received no code to review.
The message identifies a closed connection, not the component that closed it. Plausible points include Cline’s client, the provider gateway, network transport, or model-serving infrastructure. We have no trace that isolates one of them. The screenshots also show Extra thinking was selected; we did not change it or run a lower-thinking comparison. The two prompts differ in complexity, but both reached the same delivery error after long periods without tool progress.
The timing needs one distinction: Three.js ran from 12:21:45 to 12:47:09 and Python from 12:39:37 to 12:59:59. Those periods overlap. Adding their recorded durations gives about 46 session-minutes; the real clock advanced about 38 minutes from the first submission to the last error. Calling that “37 minutes combined” would blur the overlap and understate the Three.js record.
Verdict for this Cline setup
For our two first attempts, the end-to-end completion rate was 0/2. We saw reads and diagnostic shell commands, then long thinking phases and transport errors. We saw no delivered code, no build, no tests, and no final answer. That is enough to call this a poor practical experience in Cline on September 26, 2026. It is not enough to declare Pixel Canary intrinsically bad at Three.js or Python, or to overturn a pass@4 Next.js benchmark run in a different agent.
A useful follow-up comparison would keep the same fixtures and time limit, record first-attempt completion separately from retries, and log the client, provider, elapsed time, code changes, build or test result, and transport errors. Until that exists, the defensible takeaway is narrower and more useful: the impressive benchmark score did not protect these two Cline sessions from failing before any coding result could be evaluated.
Sources and evidence
- Next.js Agent Evals leaderboard, agent names, duration, and pass@4 rules
- Vercel’s Pixel Canary launch note
- Public Next.js eval runner and fixtures
- Our Cline Desktop screenshots, saved message timestamps, and post-run source-file inspection from September 26, 2026. All times above are India Standard Time.
