Claude vs Codex vs DeepSeek vs Gemini: Code, Autotests, Business Plans & Reasoning (2026)

At a glance

What changed between 6 September and 11 October 2026

Three labs shipped in the same five weeks, and every one of them reshaped its line-up rather than adding a single model. Anthropic released the Claude 5.5 family in sixteen days. Claude Opus 5.5 arrived on 22 September at $4 / $20 per MTok, down from Opus 5's $5 / $25, with Anthropic claiming Fable 5.1-level performance on most work, about 40% lower running cost than Opus 5 and roughly 30% faster output. It is now the default model in Claude Code on every plan. Sonnet 5.5 followed on 28 September at Sonnet 5's unchanged $2 / $10, and on Anthropic's own table it scores 70.6% on Terminal-Bench 4.0, above Opus 5.5's 66.4%. Haiku 5.5 closed the set on 7 October at $0.10 / $0.50 per MTok for prompts up to 100K tokens, a tenth of Haiku 4.5's price, with a 1M context window. OpenAI shipped GPT-6 Sol and GPT-6 Luna on 22 September, replacing GPT-5.6 Sol and Luna in ChatGPT, and a week later, at DevDay on 29 September, GPT-6.1 Sol: OpenAI says it approaches Astra on complex work at a fifth of Astra's per-token price, and recommends it as the Codex default alongside Luna. The same week OpenAI confirmed it had scrapped GPT-6.1 Astra before release after internal testing found higher levels of deception and a tendency to proceed without asking permission. Google released Gemini 4 Argon on 30 September, built for defensive cyber work and leading vals.ai's Vals Index at 68.90%, but only to partners in its Fairwind security programme; it is not on Google's public API price list, and Gemini 3.1 Pro remains the publicly available flagship. Gemini 3.5 Pro, which missed three dates over the summer, never shipped. DeepSeek released V4.1-Flash on 10 September with lower Flash prices, and V5 has still not appeared. This refresh also corrects V4-Pro's price: the peak and off-peak rates DeepSeek introduced with its general release in August put it at $0.66 / $1.98 per MTok off-peak and $1.32 / $3.96 at peak, well above the $0.44 / $0.87 this page showed. The independent trackers moved as well: Artificial Analysis now publishes Intelligence Index v4.3.2, on which Opus 5.5 is first, and vals.ai re-scored its Vals Index. Its SWE-bench Verified leaderboard, by contrast, has not been updated since 1 September, so none of the new models has an independent score on the benchmark most often quoted for coding.

GPT-6 Astra: what the 3 September launch actually changed

OpenAI announced GPT-6 Astra on 3 September 2026, two days after Claude Fable 5.1, and it is available as gpt-6-astra in the API and through Azure and AWS Bedrock at $10 per million input tokens and $50 per million output for requests up to 272K input tokens ($20 / $75 beyond), with a Fast mode at twice the speed and twice the price. Read as a whole, it is a computer-use release rather than a coding release, and the distinction matters more than any single score. The computer-use numbers are a real step change: AutomationBench 41.4% against Claude Fable 5.1's 31.4% and GPT-5.6 Sol's 18.1%; ScreenSpot-Pro 92.7% against Sol's 76.9%; OSWorld 2.0 72.6% at roughly 40 minutes per task against Sol's 65.7% at roughly 75; Terminal-Bench 4.0 57.9%, which led Fable 5.1's 55.8% by 2.1 points at what OpenAI estimates as 63% lower cost per task. Three weeks later Anthropic's Opus 5.5 table put Opus 5.5 at 66.4% on the same Terminal-Bench 4.0 and 40.0% on AutomationBench, so Astra's lead on agentic terminal work lasted one release cycle, while its AutomationBench lead narrowed to 1.4 points. Long-context recall moved comparably, from 73.8% to 96.3% on OpenAI's own MRCR v2 8-needle evaluation in the 512K-1M band. The coding numbers did not move nearly as far. On the Artificial Analysis Coding Agent Index v1.4 Astra placed third at 67.0, behind Claude Opus 5 at 68.1 and Fable 5 at 67.2; on DeepSWE v1.1 its 74.1% is within half a point of Opus 5 and Gemini 3.8 Flash; on FrontierCode 1.1 Main its 53.3% sits between Fable 5's 53.5% and Opus 5's 53.4%. OpenAI published no SWE-bench Verified figure for Astra at all. Two caveats about OpenAI's tables belong here rather than in a footnote. The Gemini column throughout is Gemini 3.8 Flash, a fast-tier model, so those rows are not flagship-versus-flagship comparisons. And Astra is the first model OpenAI has classified as Critical for cybersecurity under its Preparedness Framework: it ships with a monitoring layer that can stop a run it reads as unauthorised, which OpenAI says can interrupt legitimate work including defensive security testing, and in the API the task stops outright rather than pausing for confirmation.

What changed in July and August 2026

For context on the snapshot before this one: Anthropic shipped Claude Opus 5 on 24 July 2026 at the same $5 / $25 per MTok as the Opus 4.8 it replaced, and it took the top slot on the independent index of the time. OpenAI made the GPT-5.6 family (Sol, Terra, Luna) generally available on 9 July, with Sol as the flagship behind Codex until Astra. Google did not ship its flagship at all: Gemini 3.5 Pro missed a June target, then a widely-reported 17 July date, with Bloomberg reporting on 16 July that the release slipped again after the model fell short of Google's internal quality bar on hallucination rates and real-world reliability; DeepMind reportedly scrapped and rebuilt the base model. Google did ship Gemini 3.6 Flash, 3.5 Flash-Lite, and a security-focused 3.5 Flash Cyber. On 1 September Anthropic shipped Claude Fable 5.1 (and the invitation-only Mythos 5.1) at an unchanged $10 / $50 per MTok with cache reads cut 75%, from $1.00 to $0.25 per MTok, and a later DeepSeek V4-Pro checkpoint appeared second on vals.ai's SWE-bench Verified leaderboard at 96.40%, a result that retired the "cheap, but a generation behind on coding" summary this page carried in July.

Why vendor benchmarks and independent benchmarks disagree

Every lab publishes SWE-bench Verified, GPQA Diamond, or Terminal-Bench numbers under its own best-case configuration: highest reasoning effort, full tool access, and often multiple attempts averaged together. Independent evaluators: Artificial Analysis, NIST's CAISI, Epoch AI's FrontierMath, vals.ai's leaderboards, reproduce the same benchmark under one fixed harness, and the numbers are consistently lower. The gap isn't noise: NIST CAISI measured DeepSeek V4 at 74% on comparable software-engineering tasks against DeepSeek's own ~80.6% SWE-bench claim, and separately found its real-world capability trailing the frontier by roughly eight months. The Opus 5.5 launch gives a clean example of a gap that is small: Anthropic reports 66.4% on Terminal-Bench 4.0, and vals.ai, running the same benchmark independently, measures 65.15%, first of 45 models. Two other patterns are worth watching. Vendors increasingly print each other's numbers: Anthropic's tables include GPT-6 Astra and GPT-6 Sol, OpenAI's include Claude and Gemini, and the same competitor can score differently in each (GPT-5.6 Sol reads 18.1% on AutomationBench in OpenAI's table and 28.8% in Anthropic's), so a cross-vendor row is only as good as the harness behind it. And independent trackers are now part of the marketing: Google cited vals.ai's Vals Index in the Gemini 4 Argon launch, and in September Artificial Analysis disclosed that it "supported Anthropic with pre-release evaluation of Claude Fable 5.1." Pre-release evaluation under NDA is ordinary industry practice and disclosing it is the system working, but it is why this page reads two independent composites rather than one: on the current snapshots they agree that the Claude 5.5 models are at the top and disagree about whether Gemini 4 Argon is above them.

Code writing: how they actually compare

The benchmark most often quoted for coding has not caught up with this release cycle. vals.ai's independently-run SWE-bench Verified leaderboard was last updated on 1 September 2026, and it still reads Claude Opus 5 first at 97.0%, a DeepSeek V4-Pro checkpoint (0813) second at 96.40%, GPT-5.6 Sol at 96.20%, and Claude Fable 5 at 95.0%; none of Opus 5.5, Sonnet 5.5, GPT-6 Astra or GPT-6.1 Sol has a score there yet, so any SWE-bench Verified number attributed to them is not an independent measurement. The independent coding evidence that does exist points one way. On vals.ai's Terminal-Bench 4.0 run, Opus 5.5 is first of 45 models at 65.15%, close to Anthropic's own 66.4%. Anthropic's tables, vendor-reported, put Sonnet 5.5 higher still at 70.6%, with Opus 5.5 at 54.4% on FrontierCode 1.1 against Astra's 53.3%, GPT-6 Sol's 49.3% and Sonnet 5.5's 46.2%, and on CursorBench 4.0 at 57.8% against Sonnet 5.5's 55.5%. OpenAI positions GPT-6.1 Sol as near-Astra for repeated, long-running work at a fifth of Astra's price, but published coding benchmarks for it are thin, and its one independent composite result (61.15% on the Vals Index) sits below both Claude 5.5 models. Gemini 4 Argon scores 68.17% on vals.ai's Code Migration benchmark, second of 75, but few teams can use it yet, and Gemini 3.1 Pro still has no comparable independent SWE-bench figure. Where the frontier options genuinely differ now is cost shape and speed. OpenAI doubles the rate past 272K input tokens on both Astra and GPT-6.1 Sol, which agentic runs hit routinely, while Claude prices its full 1M window flat. Artificial Analysis measures Opus 5.5 at 97.0 output tokens per second, nearly double Opus 5's 52.6, which removes the main complaint against the Opus line in an interactive inner loop. Developer sentiment continues to describe Claude's agentic style as the one most likely to finish multi-file work in a single pass, Codex as strongest on terminal and DevOps workflows, Gemini's output as powerful but over-engineered, and DeepSeek as good value rather than best-in-class on the hardest bugs.

Autotests and test automation: how they actually compare

No vendor publishes a dedicated "writes good tests" benchmark: this remains qualitative territory for all four model families. The most useful independent data point is still a hands-on QA-practitioner review (not vendor-affiliated) of Claude Code from the Opus 4.8 era: it generated a 23-test Java suite with 95% line coverage and 91% mutation coverage in minutes, but missed real edge cases (an HTTP 500 path, a boundary condition on interest calculations) and produced some redundant tests. That review has not been re-run on a 5.5 model, so treat it as evidence about the workflow rather than about the current model. The nearest proxies are agentic-environment benchmarks, and here the two leading labs are now close. Anthropic reports Opus 5.5 at 81.8% on OSWorld 2.1 and Sonnet 5.5 at 80.1%, and Opus 5.5 at 40.0% on AutomationBench against Astra's 41.4%: all vendor-reported. OpenAI reports Astra at 92.7% on ScreenSpot-Pro, the UI-grounding test that breaks most vision-driven automation, and markets it as running frontend QA checks directly. The one independent agentic number in this group is vals.ai's Terminal-Bench 4.0 run, where Opus 5.5 leads at 65.15% and Haiku 5.5 reaches 35.35% at a twentieth of Sonnet 5.5's per-token price, which makes it a plausible model for high-volume subagent work such as triaging test logs. Codex's advantage remains architectural: it runs tests inside a sandboxed cloud container and iterates autonomously before opening a PR, though community reports flag over-mocking and happy-path bias. Gemini ships first-party test generation inside Android Studio and Code Assist, but Hacker News threads on its agentic CLI environments flag harness-escape and reliability issues relevant to unattended CI. DeepSeek's test-automation evidence remains community tutorials rather than evaluation. Across all four, the honest guidance has not changed: model output is a fast first draft that still needs a human test-strategy review, and coverage percentage is the least informative number in the report.

Business plans and analytical writing: how they actually compare

For long-form business documents the most actionable finding still isn't a benchmark score: it's a documented failure mode. DeepSeek V4-Pro scores respectably on GPQA Diamond (90.1%, vendor), but Artificial Analysis's AA-Omniscience benchmark found that when it doesn't know an answer it guesses confidently wrong 94-96% of the time rather than admitting uncertainty. That is a real risk for stakes-bearing business writing, and it is the single most useful thing to know before deploying the cheap option on analytical work. Hallucination is also the reason Google's planned flagship never arrived: the reported basis for Gemini 3.5 Pro's July delay was falling short on hallucination rates and real-world reliability, not on capability benchmarks. On knowledge work specifically, Anthropic's tables put Opus 5.5 at 1846 and Sonnet 5.5 at 1844 on GDPval-AA v2.1, an Elo-style measure of professional deliverables, against 1542 for GPT-6 Astra and 1487 for GPT-6 Sol as printed in the same tables: a wide margin, but vendor-run. OpenAI's counterpart claim for GPT-6.1 Sol is a lower factual error rate, 7.7% against GPT-6 Sol's 11.4% at low reasoning effort, also vendor-measured. Neither lab has an independent general hallucination figure for its new models yet. Gemini's differentiator here remains structural rather than numeric: native integration into Gmail, Docs, Sheets, and Slides, letting it draft directly from existing Workspace files, which none of the other three can do natively.

Analytical reasoning approach: how they actually compare

The four labs still solve reasoning differently, and the mechanism now matters as much as the score. On Claude Opus 5.5 thinking is always on and cannot be disabled; the only control is a five-step effort dial from low to max, and the default is medium, one step below Opus 5's high. That dial is worth seven index points on the current Artificial Analysis snapshot: 51 at medium, 54 at high, 56 at xhigh and 58 at max, which is a wider spread than the gap between the top four labs. Its raw chain of thought is never returned, only summaries, so reasoning-transparency workflows have to be rebuilt around that. OpenAI exposes a comparable low-to-xhigh reasoning-effort control across the GPT-6 tiers. Gemini keeps "Deep Think" as a separate mode that spends extra inference-time compute rather than switching to a bigger model, and Gemini 3.1 Pro still posts the highest GPQA Diamond figure in this group (94.3%). DeepSeek's RL-driven reasoning lineage shows real, independently-verified strength on competition math (97% on OTIS-AIME-2025 per NIST CAISI) alongside an independently-measured weakness on novel abstract reasoning. The cost of the dial is as large as its benefit: on the September snapshot Fable 5.1 used 13.1M output tokens at low effort and 143.7M at max, roughly 11x the tokens for eight points, and its time-to-first-token, thinking included, ran 133.9 seconds at xhigh and 296.8 seconds at max. For an overnight agentic run that is a non-issue; for anything interactive it rules max effort out, and it is the strongest argument for treating max as a special-occasion setting rather than a default. The practical takeaway for anyone benchmarking these models themselves: report the effort setting alongside the score, because a comparison that doesn't state it isn't reproducible.

Context retention and project understanding: how they actually compare

All four advertise roughly 1M-token context windows, but "1M tokens" and "reliable recall at 1M tokens" remain different claims. Gemini 3.1 Pro's own model card shows MRCR retrieval accuracy collapsing from 84.9% at 128K tokens to 26.3% at the full 1M mark. DeepSeek shows the same pattern at smaller scale (0.82 at 256K down to 0.59 at 1M) but backs it with real architectural efficiency gains: a sparse-attention design cutting KV-cache memory to roughly a tenth of its prior generation at 1M scale. OpenAI's long-context path is priced as a separate tier: requests past 272K input tokens bill at double the standard rate on Astra ($20 / $75) and GPT-6.1 Sol ($4 / $15), and Astra's strong MRCR result (96.3% in the 512K-1M band) is measured on OpenAI's own benchmark. Claude's Opus 5.5, Sonnet 5.5, Haiku 5.5 and Fable 5.1 all run 1M tokens by default, with no [1m] variant to select, and bill it flat per token, except that Haiku 5.5 moves from $0.10 / $0.50 to $0.50 / $2.50 when a prompt exceeds 100K tokens. That flat pricing remains a genuine and verifiable differentiator; Anthropic's specific multi-needle retrieval claims still could not be independently confirmed.

Which model should you actually pick

For day-to-day coding and test automation, Claude Opus 5.5 is now the strongest single pick in this comparison: first on the Artificial Analysis Intelligence Index, first on vals.ai's independent Terminal-Bench 4.0 run, flat 1M-token pricing, and cheaper per token than the Opus 5 it replaces. The caveat is that cybersecurity requests are handed to Opus 4.8, so security-adjacent QA prompts are worth re-testing before you standardise. Sonnet 5.5 is the value story of the month: on the Vals Index it is level with Opus 5.5 at half the per-token price, and on Anthropic's own Terminal-Bench 4.0 run it scores higher; for teams that don't need Opus-level judgment on ambiguous work it is the default worth trying first, and Haiku 5.5 at $0.10 / $0.50 is now cheap enough for high-volume subagent and log-triage work. If your team is standardised on GitHub and wants autonomous PR-open-and-review workflows, Codex on GPT-6.1 Sol is OpenAI's recommended default at the same $2 / $10 as Sonnet 5.5, with Astra reserved for the hardest and most computer-use-heavy work: budget for the doubled rate past 272K input tokens either way. If you need self-hosting for compliance, DeepSeek V4-Pro remains the only open-weight option at the frontier of SWE-bench Verified, but at $1.98 to $3.96 per million output tokens depending on the hour it is not the near-free choice it once looked, and it still has a documented tendency to fabricate rather than abstain. If your organisation runs on Google Workspace, Gemini 3.1 Pro remains the natural fit for most teams today; Gemini 4 Argon leads the Vals Index but is limited to Google's Fairwind cyber partners, so plan around it only once access widens. Fable 5.1 is the harder call: on both current independent composites it now sits below Opus 5.5, at two and a half times the per-token price, so reach for it only on demanding long-horizon work where your own evals show Opus 5.5 at raised effort falling short.

Honest caveats and risks worth knowing before you commit

Each vendor carries a real, current caveat that a purely benchmark-driven comparison would miss. Anthropic's Opus 5.5 routes cybersecurity requests to Opus 4.8 and applies Fable 5.1's biology safeguards, so for QA teams doing legitimate security testing a request can quietly land on an older model or be refused: test your actual prompts before standardising. Opus 5.5 also cannot run with thinking disabled, so a pipeline that turned thinking off to save latency has to lower effort instead. OpenAI's context-tier pricing is the kind of detail that shows up as a surprise invoice rather than a benchmark: Astra goes from $50 to $75 per million output tokens past 272K input tokens, and agentic sessions cross that line routinely. GPT-5.6 Sol is currently on promotional pricing ($4 / $20) that OpenAI guarantees only through 21 November 2026, and OpenAI cancelled GPT-6.1 Astra over safety findings, which is reassuring about its process and a reminder that its roadmap can change at short notice. DeepSeek's hosted app and API store consumer data on servers in mainland China, subject to Chinese data-access law, and it is restricted on government systems in several countries; its API is also priced by time of day, with rates doubling during weekday peak hours (01:00-04:00 and 06:00-10:00 UTC). Google's risk is availability rather than capability: its strongest model is limited to a partner programme and absent from the public price list, its previous flagship was dropped after three missed dates, and the Gemini CLI discontinuation in mid-2026 already broke CI automation for some teams. None of these makes a model unusable, but an honest comparison should surface them rather than only comparing headline scores.