DeepSeek V4.1 Flash 1017: What '9 of 11' Actually Means

DeepSeek released V4.1-Flash-1017 on October 17. The model card carries an 11-row benchmark table, and in that table the new build beats GPT-5.6 Sol on nine of the eleven rows. The claim is true. You can check it line by line in about two minutes.

The table is DeepSeek's own, and it does not include GPT-6 Astra. Independent testing against Astra gives a more varied picture: scores are close on two agentic evaluations, while gaps on terminal coding and science range from twenty-five to fifty points.

This is what the numbers support, thirteen days after the table was published, and what changed in the eight days that followed.

What DeepSeek actually published

The table compares the 1017 build against the September V4.1-Flash, Kimi K3, GLM-5.3, GPT-5.6 Sol and Claude Opus 5. Two footnotes sit under it, and both matter:

• Terminal-Bench 4.0 was run with the Claude Code harness; DeepSWE v1.1 with Mini-SWE-V2.
• The daggered HLE figures are the text-only subset.

Everything else is unstated: who ran the rival models, on what dates, with what sampling, and whether the Sol and Opus columns were re-run or copied from other people's reports. The card does not say. A benchmark table without that information is a claim about a model's behaviour on that table, not a measurement of the model in the abstract.

TECHi's write-up was unusually careful about this: "Every score here is DeepSeek's, produced on test harnesses it chose, and none has been independently reproduced." That sentence is the correct frame for the whole table. For context on the previous build, see our V4-Flash-0731 teardown; for why the harness matters as much as the weights, the DeepSeek Harness breakdown covers the tooling side.

The 9-of-11 tally, checked

Here is the DeepSeek-versus-Sol subset with the arithmetic done:

Nine wins, two losses. Against Kimi K3 it is ten of eleven; against Claude Opus 5 it loses seven of eleven, with the widest gaps on HLE (13.7 points) and Terminal-Bench 4.0 (9.5).

Most of the eleven rows are coding, agentic or vision-with-tools suites. There is no chat-preference evaluation, so the tally reflects the selected tasks rather than every kind of model use.

Three of the nine wins are under four points. Without uncertainty estimates or repeated runs, the table alone cannot establish how reliable those margins are.

Where the nine wins come from. Eight sit on suites built outside DeepSeek: Terminal-Bench 4.0 by the Laude Institute, Stanford researchers and the open-source community; DeepSWE v1.1 by Datacurve (2607.07946); ProgramBench (2605.03546); NL2Repo-Bench (2512.12730); Zapier's AutomationBench (2604.18934); Agents' Last Exam by UC Berkeley's RDI (2606.05405); Chartography by Surge AI (2608.10677); and BabyVision (2601.06521). External benchmark authorship does not independently verify DeepSeek's results.

The ninth is the exception. Apex Bench v1.1 has no public paper, repository or leaderboard that I could find; the only published numbers are DeepSeek's own, and benchmark trackers list it as a vendor-reported reference rather than an independently run suite. It is also the largest win in the table, at 17.8 points over Sol. That is not evidence of anything wrong — it may simply be an internal evaluation DeepSeek chose to publish — but it is the one row a reader cannot check.

Eight days later, the ruler changed

On October 22, Artificial Analysis had DeepSeek V4.1 Flash 1017 at 45 on its Intelligence Index, ranked second among 121 comparable models in its large open-weights class (archived page). That version of the index (v4.3.2) weighted ten evaluations and included SciCode, GDPval-AA and AA-LCR.

On October 30, the same model reads 32, and the index is v5.0 with nine evaluations: AA-Briefcase v1.1, AutomationBench-AA, Terminal-Bench 4.0, FrontierCode 2.0, AA-Omniscience v1.1, GDP.pdf, HLE-Diamond, CritPt v2 and Terminal-Bench-Science 0.1. SciCode, GDPval-AA and AA-LCR are gone; FrontierCode, HLE-Diamond and TB-Science are new.

The change in score does not by itself show that the model got worse: the instrument changed. The 22 October figure is preserved in the archive snapshot linked above; the v5.0 score is on the live page. The two figures should be cited with their index versions.

This is the same failure mode as the vendor table, one level up: a number is only meaningful with its instrument attached.

What the independent numbers say now

Artificial Analysis runs its own harness rather than accepting vendor figures. Here is DeepSeek V4.1 Flash 1017 against GPT-6 Astra, both in their reasoning (max) configuration, as published on 30 October:

One row needs a warning before you line it up against the vendor table. AA's AutomationBench-AA is its independent run of Zapier's AutomationBench v1.0.6, using a private held-out split and its own scoring method. DeepSeek reports 60.5 and AA reports 67%, but without matching the task set and scoring, the figures are not directly comparable. AA's methodology describes its implementation.

On AA-Briefcase, an agentic knowledge-work evaluation, DeepSeek scores 1570 Elo against Astra's 1569. On AutomationBench-AA, the scores are 67% and 68%. Those close results sit alongside a twenty-five-point composite gap and wider gaps on several other evaluations.

The composite combines knowledge work, coding, science and general knowledge, each weighted at 25%; AA publishes the evaluation weights. Readers should consider the individual evaluations that best resemble their work as well as the composite.

The price side is where the claim survives

DeepSeek's pricing did not change with the update, and the off-peak tier is the part most comparisons ignore (official pricing):

Off-peak rates are half of peak: $0.15 input, $0.60 output, $0.003 cached. Peak is 01:00–04:00 and 06:00–10:00 UTC, Monday to Friday, excluding Chinese public holidays — which means a European or US working day is mostly off-peak, and a scheduler that batches overnight work pays half price for the same weights.

That is about a 10× difference in cost per Index task and a 42× difference in blended token price. The blend assumes a 7:2:1 ratio of cache-hit input, uncached input and output tokens; actual costs depend on usage. The Index cost is across its full suite, not AA-Briefcase alone. The weights are MIT-licensed, with 552B backbone parameters and 16B active per generated token. If you are choosing between the Flash and Pro tiers rather than against OpenAI, our V4 pricing and selection guide covers that decision; the GLM-5.2 breakdown offers another open-weights comparison.

The results describe a low-cost open-weights model with close scores on two agentic evaluations and substantial gaps on several other tests.

Reading this table, and the next one

Three habits cover most of it.

Find the harness footnote before the score. DeepSeek reports 42.3 on Terminal-Bench 4.0 under Claude Code. AA reports 34% using mini-swe-agent. The harness is one possible explanation for the gap; other evaluation settings may also differ.

Check the comparator. DeepSeek's table has no Astra row; comparisons with Astra require a different source.

Treat small gaps cautiously. Four of DeepSeek's nine reported wins over Sol are under five points. The published table does not provide enough information to determine whether those margins would persist across repeated runs.

Bottom line

DeepSeek's table reports nine wins over Sol. Independent results against Astra show close scores on AA-Briefcase and AutomationBench-AA, alongside substantial gaps on the composite and several individual evaluations.

For workloads resembling AA-Briefcase or AutomationBench-AA, the close scores and lower token prices make Flash worth evaluating. For workloads resembling Terminal-Bench-Science or GDP.pdf, the measured gaps warrant closer attention. Check the harness, comparator and index version before drawing conclusions.