Can LLMs read a color vision test?

This post and benchmark were written and built with AI assistance using Codex and Claude Code.

Example

Each plate contains 23. The dots have matched brightness. Only their color defines the number. Color separation increases from left to right.

Color-dot plate with the number 23 at separation 0.040
Separation 0.040Open image ↗
Color-dot plate with the number 23 at separation 0.080
Separation 0.080Open image ↗
Color-dot plate with the number 23 at separation 0.120
Separation 0.120Open image ↗

Select an image to open the original 1024 × 1024 PNG.

How the images are made

  1. Place 900 dots in a circle. Set their positions, sizes, and brightness before applying a number.
  2. Use a two-digit mask to assign one color to dots inside the number and another color to the other dots.
  3. Match the brightness of the two colors. Change their separation across seven levels in two red–green color directions.
  4. Add grayscale copies and color blanks with no number. These controls should produce null.

The set has 140 color plates (10 numbers × 2 color directions × 7 separations), 100 grayscale copies, and 20 color blanks.

Control examples
Grayscale twin of a color plate, with no number visible
Grayscale twinSame dot geometry; color removed.
Colored dots with no intended number
Color blankColored dots; no number assigned.

Results

Each result set has one saved response for every image. Both digits must match for a correct read. A number reported on a control is a false read.

GPT-6 Astra read 81/140 color plates correctly. GPT-4.1-mini scored highest at the strongest color separation: 15/20.

Compare model scores

25 result sets

Correct number reads. Each level has 20 color plates; “All color” combines all 140. Color separation increases from left to right.

Ranked by all 140 color plates. Ties share a rank.

RankModel
1816111413141112
2470127111214
3460025111315
44202598810
540013581013
63610249812
73100058711
8260125666
9250034378
10200001478
11130003334
12120000156
1380021122
1470002212
=1560012120
=1562012010
1740000022
1830002100
=1920000002
=1920000011
=1920001010
2210100000
=2300000000
=2300000000
=2300000000

Bold marks the best score in each column. Equal scores share a rank.

Inspect individual image answers

GPT-6 Astra

Saved results · low reasoning · original image detail

Color 81 / 140
Controls 120 / 120
All inputs 201 / 260

✓ Correct · × Incorrect · Cell labels show the selected model’s answer.

Open a thumbnail to inspect the exact image and saved answer. “none” means no number was reported.

Color plates · Protan

Rows show the expected number. Columns show color separation.

Color plates · Deutan

Rows show the expected number. Columns show color separation.

Grayscale controls · Protan

No number is expected. Row labels identify the source mask.

Grayscale controls · Deutan

No number is expected. Row labels identify the source mask.

Color blanks

No number is expected in any blank.

Original plate

Original PNG · 1024 × 1024. Arrow keys change plates. At 1:1, focus the image area to pan with arrow keys.

Download PNG
Model
Expected answer
Model answer
Correctness
Input ID
Family
Axis / separation

Saved final response


Model settings and secondary metrics

Expand a model for request settings, digit accuracy, and control errors.

1GPT-6 AstraAll color plates81/140Separation 0.12012/20False reads0/120

81/140 exact · 57 wrong numbers · 2 null answers · 0 invalid

Model ID: gpt-6-astra

Reasoning: low · Image detail: original · Output limit: default

All seven color separations
Score0.0060.0100.0160.0260.0400.0800.120
Exact30%55%70%65%70%55%60%
Digits57.5%75%85%80%80%75%77.5%
Numeric90%100%100%100%100%100%100%

Correct control rejections: 100% grayscale; 100% color blanks.

2GPT-4.1All color plates47/140Separation 0.12014/20False reads0/120

47/140 exact · 63 wrong numbers · 30 null answers · 0 invalid

Model ID: gpt-4.1-2025-04-14

Reasoning: not set · Image detail: high · Temperature: 0 · Output limit: 60 tokens

All seven color separations
Score0.0060.0100.0160.0260.0400.0800.120
Exact0%5%10%35%55%60%70%
Digits0%12.5%50%67.5%77.5%80%85%
Numeric5%45%100%100%100%100%100%

Correct control rejections: 100% grayscale; 100% color blanks.

3GPT-4.1-miniAll color plates46/140Separation 0.12015/20False reads0/120

46/140 exact · 90 wrong numbers · 4 null answers · 0 invalid

Model ID: gpt-4.1-mini-2025-04-14

Core plate settings: Reasoning: not set · Image detail: high · Temperature: 0 · Output limit: 60 tokens

All seven color separations
Score0.0060.0100.0160.0260.0400.0800.120
Exact0%0%10%25%55%65%75%
Digits12.5%32.5%40%62.5%75%77.5%82.5%
Numeric80%100%100%100%100%100%100%

Correct control rejections: 100% grayscale; 100% color blanks.

4Gemini 3.8 FlashAll color plates42/140Separation 0.12010/20False reads0/120

42/140 exact · 79 wrong numbers · 19 null answers · 0 invalid

Model ID: gemini-3.8-flash

Thinking level: low · Image detail: native · Output limit: 2048 tokens · Output format: JSON schema

All seven color separations
Score0.0060.0100.0160.0260.0400.0800.120
Exact0%10%25%45%40%40%50%
Digits0%30%52.5%70%67.5%67.5%72.5%
Numeric5%100%100%100%100%100%100%

Correct control rejections: 100% grayscale; 100% color blanks.

5Gemini 3.6 FlashAll color plates40/140Separation 0.12013/20False reads1/120

40/140 exact · 82 wrong numbers · 18 null answers · 0 invalid

Model ID: gemini-3.6-flash

Thinking level: minimal · Image detail: native · Output limit: 2048 tokens · Output format: JSON schema

All seven color separations
Score0.0060.0100.0160.0260.0400.0800.120
Exact0%5%15%25%40%50%65%
Digits0%27.5%42.5%62.5%67.5%75%82.5%
Numeric15%95%100%100%100%100%100%

Correct control rejections: 100% grayscale; 95% color blanks.

False numbers on controls
AnswerColor blanksGrayscale
8110
6Gemini 3.7 FlashAll color plates36/140Separation 0.12012/20False reads0/120

36/140 exact · 90 wrong numbers · 14 null answers · 0 invalid

Model ID: gemini-3.7-flash

Thinking level: low · Image detail: native · Output limit: 2048 tokens · Output format: JSON schema

All seven color separations
Score0.0060.0100.0160.0260.0400.0800.120
Exact5%0%10%20%45%40%60%
Digits10%30%42.5%55%72.5%65%77.5%
Numeric30%100%100%100%100%100%100%

Correct control rejections: 100% grayscale; 100% color blanks.

7Gemini 3.1 Pro PreviewAll color plates31/140Separation 0.12011/20False reads1/120

31/140 exact · 88 wrong numbers · 17 null answers · 4 invalid

Model ID: gemini-3.1-pro-preview

Core plate settings: Thinking level: low · Image detail: native · Output limit: 2048 tokens

All seven color separations
Score0.0060.0100.0160.0260.0400.0800.120
Exact0%0%0%25%40%35%55%
Digits0%25%45%57.5%70%57.5%77.5%
Numeric15%100%100%95%100%85%100%

Correct control rejections: 100% grayscale; 95% color blanks.

False numbers on controls
AnswerColor blanksGrayscale
410
8Gemini 3.5 FlashAll color plates26/140Separation 0.1206/20False reads15/120

26/140 exact · 104 wrong numbers · 10 null answers · 0 invalid

Model ID: gemini-3.5-flash

Core plate settings: Thinking level: low · Image detail: native · Output limit: 2048 tokens

All seven color separations
Score0.0060.0100.0160.0260.0400.0800.120
Exact0%5%10%25%30%30%30%
Digits5%22.5%37.5%55%57.5%62.5%65%
Numeric50%100%100%100%100%100%100%

Correct control rejections: 100% grayscale; 25% color blanks.

False numbers on controls
AnswerColor blanksGrayscale
5250
5350
530
2520
9Gemini 3 Flash PreviewAll color plates25/140Separation 0.1208/20False reads22/120

25/140 exact · 103 wrong numbers · 12 null answers · 0 invalid

Model ID: gemini-3-flash-preview

Core plate settings: Thinking level: low · Image detail: native · Output limit: 2048 tokens

All seven color separations
Score0.0060.0100.0160.0260.0400.0800.120
Exact0%0%15%20%15%35%40%
Digits7.5%27.5%42.5%55%55%65%70%
Numeric40%100%100%100%100%100%100%

Correct control rejections: 92% grayscale; 30% color blanks.

False numbers on controls
AnswerColor blanksGrayscale
4712
5221
1520
301
510
1201
2010
2110
2410
2610
2801
3810
4201
6010
7401
8110
8610
10GPT-5 miniAll color plates20/140Separation 0.1208/20False reads2/120

20/140 exact · 99 wrong numbers · 21 null answers · 0 invalid

Model ID: gpt-5-mini

Core plate settings: Reasoning: minimal · Image detail: high · Output limit: 1000 tokens

All seven color separations
Score0.0060.0100.0160.0260.0400.0800.120
Exact0%0%0%5%20%35%40%
Digits0%22.5%25%32.5%45%62.5%57.5%
Numeric5%90%100%100%100%100%100%

Correct control rejections: 99% grayscale; 95% color blanks.

False numbers on controls
AnswerColor blanksGrayscale
1211
11Gemini 3.1 Flash-LiteAll color plates13/140Separation 0.1204/20False reads0/120

13/140 exact · 105 wrong numbers · 22 null answers · 0 invalid

Model ID: gemini-3.1-flash-lite

Core plate settings: Thinking level: low · Image detail: native · Output limit: 2048 tokens

All seven color separations
Score0.0060.0100.0160.0260.0400.0800.120
Exact0%0%0%15%15%15%20%
Digits0%17.5%25%52.5%47.5%52.5%47.5%
Numeric0%90%100%100%100%100%100%

Correct control rejections: 100% grayscale; 100% color blanks.

12GPT-5.1-Codex miniAll color plates12/140Separation 0.1206/20False reads0/120

12/140 exact · 76 wrong numbers · 52 null answers · 0 invalid

Model ID: gpt-5.1-codex-mini

Core plate settings: Reasoning: low · Image detail: high · Output limit: 4000 tokens

All seven color separations
Score0.0060.0100.0160.0260.0400.0800.120
Exact0%0%0%0%5%25%30%
Digits0%2.5%15%25%25%47.5%57.5%
Numeric0%20%70%70%85%100%95%

Correct control rejections: 100% grayscale; 100% color blanks.

13Claude Opus 5All color plates8/140Separation 0.1202/20False reads0/120

8/140 exact · 93 wrong numbers · 39 null answers · 0 invalid

Model ID: claude-opus-5

Thinking: disabled · Effort: low · Image detail: native · Output limit: 8192 tokens

All seven color separations
Score0.0060.0100.0160.0260.0400.0800.120
Exact0%0%10%5%5%10%10%
Digits0%0%17.5%20%35%25%25%
Numeric0%15%90%100%100%100%100%

Correct control rejections: 100% grayscale; 100% color blanks.

14Gemini 3.5 Flash-LiteAll color plates7/140Separation 0.1202/20False reads0/120

7/140 exact · 107 wrong numbers · 26 null answers · 0 invalid

Model ID: gemini-3.5-flash-lite

Thinking level: minimal · Image detail: native · Output limit: 2048 tokens · Output format: JSON schema

All seven color separations
Score0.0060.0100.0160.0260.0400.0800.120
Exact0%0%0%10%10%5%10%
Digits0%17.5%25%45%45%37.5%50%
Numeric0%70%100%100%100%100%100%

Correct control rejections: 100% grayscale; 100% color blanks.

15GPT-5.6 SolAll color plates6/140Separation 0.1200/20False reads0/120

6/140 exact · 134 wrong numbers · 0 null answers · 0 invalid

Model ID: gpt-5.6-sol

Reasoning: low · Image detail: original · Output limit: default

All seven color separations
Score0.0060.0100.0160.0260.0400.0800.120
Exact0%0%5%10%5%10%0%
Digits12.5%12.5%17.5%22.5%25%35%15%
Numeric100%100%100%100%100%100%100%

Correct control rejections: 100% grayscale; 100% color blanks.

Other saved results

Daybreak → Sol · 6/140

6/140 exact · 134 wrong numbers · 0 null answers · 0 invalid

Model ID: gpt-daybreak-blue-latest

Reasoning: low · Image detail: original · Output limit: default

Returned model: gpt-5.6-sol

All seven color separations
Score0.0060.0100.0160.0260.0400.0800.120
Exact10%0%5%10%0%5%0%
Digits25%17.5%17.5%17.5%20%25%25%
Numeric100%100%100%100%100%100%100%

Correct control rejections: 100% grayscale; 95% color blanks.

False numbers on controls
AnswerColor blanksGrayscale
5610
16GPT-4.1 nanoAll color plates4/140Separation 0.1202/20False reads0/120

4/140 exact · 65 wrong numbers · 71 null answers · 0 invalid

Model ID: gpt-4.1-nano-2025-04-14

Reasoning: not set · Image detail: high · Temperature: 0 · Output limit: 60 tokens

All seven color separations
Score0.0060.0100.0160.0260.0400.0800.120
Exact0%0%0%0%0%10%10%
Digits0%0%0%10%12.5%25%20%
Numeric0%0%35%75%70%75%90%

Correct control rejections: 100% grayscale; 100% color blanks.

17Claude Fable 5.1All color plates3/140Separation 0.1200/20False reads0/120

3/140 exact · 116 wrong numbers · 21 null answers · 0 invalid

Model ID: claude-fable-5-1

Thinking: adaptive · Effort: low · Image detail: native · Output limit: 8192 tokens

All seven color separations
Score0.0060.0100.0160.0260.0400.0800.120
Exact0%0%0%10%5%0%0%
Digits0%15%15%40%35%20%27.5%
Numeric0%95%100%100%100%100%100%

Correct control rejections: 100% grayscale; 100% color blanks.

18GPT-5.4All color plates2/140Separation 0.1202/20False reads0/120

2/140 exact · 134 wrong numbers · 4 null answers · 0 invalid

Model ID: gpt-5.4

Core plate settings: Reasoning: not set · Image detail: original · Output limit: 1000 tokens

All seven color separations
Score0.0060.0100.0160.0260.0400.0800.120
Exact0%0%0%0%0%0%10%
Digits5%17.5%17.5%12.5%15%17.5%20%
Numeric80%100%100%100%100%100%100%

Correct control rejections: 100% grayscale; 100% color blanks.

18GPT-5.4 miniAll color plates2/140Separation 0.1201/20False reads0/120

2/140 exact · 134 wrong numbers · 4 null answers · 0 invalid

Model ID: gpt-5.4-mini

Core plate settings: Reasoning: none · Image detail: high · Output limit: 1000 tokens

All seven color separations
Score0.0060.0100.0160.0260.0400.0800.120
Exact0%0%0%0%0%5%5%
Digits12.5%10%12.5%12.5%12.5%35%30%
Numeric90%95%100%95%100%100%100%

Correct control rejections: 100% grayscale; 100% color blanks.

18GPT-5.6 LunaAll color plates2/140Separation 0.1200/20False reads0/120

2/140 exact · 77 wrong numbers · 61 null answers · 0 invalid

Model ID: gpt-5.6-luna

Reasoning: low · Image detail: original · Output limit: default

All seven color separations
Score0.0060.0100.0160.0260.0400.0800.120
Exact0%0%0%5%0%5%0%
Digits0%2.5%5%12.5%5%10%7.5%
Numeric0%30%55%75%60%85%90%

Correct control rejections: 100% grayscale; 100% color blanks.

21GPT-5.6 TerraAll color plates1/140Separation 0.1200/20False reads0/120

1/140 exact · 136 wrong numbers · 3 null answers · 0 invalid

Model ID: gpt-5.6-terra

Reasoning: low · Image detail: original · Output limit: default

All seven color separations
Score0.0060.0100.0160.0260.0400.0800.120
Exact0%5%0%0%0%0%0%
Digits10%12.5%2.5%10%7.5%15%15%
Numeric85%100%100%100%100%100%100%

Correct control rejections: 100% grayscale; 100% color blanks.

22Claude Sonnet 4.5All color plates0/140Separation 0.1200/20False reads0/120

0/140 exact · 54 wrong numbers · 86 null answers · 0 invalid

Model ID: claude-sonnet-4-5-20250929

Core plate settings: Reasoning: not set · Image detail: native · Output limit: 64 tokens

All seven color separations
Score0.0060.0100.0160.0260.0400.0800.120
Exact0%0%0%0%0%0%0%
Digits0%0%0%2.5%2.5%10%15%
Numeric0%0%5%35%60%90%80%

Correct control rejections: 100% grayscale; 100% color blanks.

22Claude Sonnet 5All color plates0/140Separation 0.1200/20False reads0/120

0/140 exact · 9 wrong numbers · 131 null answers · 0 invalid

Model ID: claude-sonnet-5

Core plate settings: Reasoning: disabled · Image detail: native · Output limit: 64 tokens

All seven color separations
Score0.0060.0100.0160.0260.0400.0800.120
Exact0%0%0%0%0%0%0%
Digits0%0%0%0%5%12.5%2.5%
Numeric0%0%0%0%10%25%10%

Correct control rejections: 100% grayscale; 100% color blanks.

22Gemini 2.5 FlashAll color plates0/140Separation 0.1200/20False reads0/120

0/140 exact · 67 wrong numbers · 73 null answers · 0 invalid

Model ID: gemini-2.5-flash

Core plate settings: Thinking budget: 0 · Image detail: native · Output limit: 2048 tokens

All seven color separations
Score0.0060.0100.0160.0260.0400.0800.120
Exact0%0%0%0%0%0%0%
Digits0%0%2.5%0%2.5%17.5%22.5%
Numeric0%0%30%40%80%90%95%

Correct control rejections: 100% grayscale; 100% color blanks.

Each row shows one result set. Other saved results are inside that model’s details.

Prompt and scoring
Inspect this newly generated pseudoisochromatic color-dot plate. Report the one- or two-digit Arabic number only if it is actually visible. Do not infer an answer from familiar clinical color-vision plates and do not force a guess. Return only JSON as {"number":"NN"}; if no numeral is visible, return {"number":null}.

In the score details, Exact requires both digits to match. Digits counts correct digit positions. Numeric counts number responses. Invalid answers count as wrong.

Color separation is measured in CIE u′v′ coordinates. The two directions are called protan and deutan. The number mask uses Atkinson Hyperlegible.