← Agent-made workMade with Codex · Directed by Jonathan Chang
Vision benchmark · 17 Nov 2026

Traditional Chinese Bench

One image. One character. See how AI models read Traditional Chinese, from everyday characters to rare forms.

1,200 characters17 modelsSame images. Same prompt.
✓ Raw replies verifiedStrict accuracy and character recognition are separate.Methods & audit

Model comparison

Select a model to inspect its answers.
ModelStrictRecognitionFormat
95.3%95.3%100%
93.8%93.8%99.4%
93.8%93.8%100%
OpenRouter → DeepInfra
92.7%92.7%100%
89.1%89.1%100%
88.8%89.5%98.8%
87.5%87.5%99.8%
86.2%86.2%100%
OpenRouter → DeepInfra
81.1%81.1%99.1%
74.3%74.5%99.7%
OpenRouter → DeepInfra
71.5%71.6%95.6%
59.2%59.2%100%
OpenRouter → DeepInfra
57.1%57.1%99.3%
56.3%56.7%98.8%
50%50%95.4%
47.6%47.6%99.8%
OpenRouter → DeepInfra
37.5%37.5%99.3%
All values are percentages. Hover or focus a score for the exact count.0%100%

Strict requires the exact reply. Recognition accepts a clear final character despite extra formatting. Format measures the one-character rule. How scoring works

Scores by frequency band

Strict accuracy · 200 characters per band.

ModelVery commonCommonMid-frequencyLess commonRareVery rare
100%99.5%96.5%95%92.5%88.5%
100%99.5%100%94.5%86%82.5%
99.5%99%98%89.5%90%86.5%
OpenRouter → DeepInfra
97.5%99%97%93%89%80.5%
99%98%96%87%77.5%77%
97%96.5%97%89%77%76%
98.5%98.5%99.5%96%70.5%62%
95.5%95.5%93.5%84.5%73.5%74.5%
OpenRouter → DeepInfra
95%93.5%90%77.5%68%62.5%
99%100%95.5%75%42%34.5%
OpenRouter → DeepInfra
96%91.5%81.5%68%50%42%
96.5%91.5%79%45%23.5%19.5%
OpenRouter → DeepInfra
85.5%85%73.5%52%26.5%20%
90.5%92.5%83%49.5%13.5%9%
94.5%87.5%67%34%12.5%4.5%
91%87.5%65%28%7%7%
OpenRouter → DeepInfra
84.5%74.5%43.5%17%4%1.5%
Inspect individual characters

Filter all 1,200 images, compare models, and open any image for the original replies. Results load when you open this section.

Strict
Recognition
Format
Exact counts & subset scores
Strict correctAnswers extractedLook-alike subset
Compare models & display options
✓ Solid: correct× Dashed: wrongOpen an image for the raw reply and both scores.
Method, scoring & audit records
✓ Parsing audit passed. All 20,400 saved answers match the raw provider responses. None of the 5,261 wrong answers under the strict rule is due to parsing. The results archive contains the parsing audits and all original final-reply exports.

All full-set scores use 1,200 images. Each frequency band uses 200 images. Values in the comparison table are percentages; exact counts are available on hover or keyboard focus. Frequency ranks use the historical 1993–1994 BIG5 source, not current usage ranks. Recognition is a post-hoc proxy, not a pure vision score. Missing or unclear answers remain in the denominator.

Prompt: Identify the single Traditional Chinese character in this image. Reply with only that one character. Do not add spaces, punctuation, explanations, or alternatives.

Each image was sent in a fresh request, without labels, character lists, or prior answers. The strict score requires the exact Unicode character after removal of surrounding whitespace. Different written forms and multi-character answers count as wrong under that rule.

Separate recognition score: the same deterministic extractor runs on every model's final response text. It does not receive the model name or expected character. It accepts a single Han character, a final single-character line, or a single character after an explicit ending statement, colon, arrow, equality, or English sentence. It can remove outer punctuation, Markdown decoration, invisible format controls, and variation selectors. It does not scan for the reference character, read hidden reasoning, normalize Unicode, or convert written forms. Lists and unclear endings are not given credit. The full 1,200-item denominator remains fixed. This is a post-hoc recognition proxy; it does not prove pure visual ability.

Separate format score: exactly one Han character after whitespace removal, whether that character is right or wrong. Original length-only format flags remain in the saved run files. A lone variation selector is a format failure here. Strict scores and raw replies are unchanged. Download the metric definitions, totals, format differences, and extractor source.

For Codex, the audit compared joined text chunks, done events, completed messages, saved answers, and scores. For Anthropic and OpenRouter, it checked the raw response JSON; Anthropic SDK-parsed text was also checked. It found no lost text or wrong message selection. Unicode checks are recorded in the audit files. Multi-character responses remain wrong under the agreed rule. For example, Luna returned 級籤 for 綎, and 幽屍 for 齶. These were model outputs, not parser errors.

Images are unchanged 256 × 256 RGB PNGs, using Noto Sans CJK TC Regular. This set gives each frequency band equal weight. It does not estimate accuracy on normal running text.

Download data and audit records

Original dataset (ZIP) — exact PNGs, frozen manifest, source ranks, full original font, and license.

Results and audits (ZIP) — every verified final reply, model settings, summaries, CSV exports, parsing audits, and scoring code.

Original character image

Download original PNG

Expected character