New benchmark tests AI translation of Teochew Chinese
Summary and headline written by AI from the source article. How we work
Researchers created TeochewBench, a new resource for testing how well artificial intelligence translates Teochew Chinese into Mandarin Chinese and English.
The benchmark contains 300 Teochew Hanzi expressions, reviewed by native speakers to ensure accuracy. Teochew is a dialect of Chinese spoken by a large community, particularly in parts of China and Southeast Asia, but lacks extensive digital resources for language technology.
The dataset groups expressions into five types: basic vocabulary, everyday sentences, uniquely Teochew phrases, expressions sensitive to politeness and context, and culturally specific idioms. The creators evaluated 11 widely used AI language models on the benchmark, generating over 6,600 translations. They also ran a control test where the AI simply copied the original Teochew text to measure how much shared Chinese characters influence translation scores.
Results showed that Qwen3.5-27B performed best overall, achieving a score of 60.63 using a metric called chrF-style. However, the models struggled with highly specific Teochew expressions, averaging a score of 27.52 compared to 69.25 for simpler phrases. This suggests that capturing the nuances of Teochew culture and language remains a challenge for current AI. The researchers hope TeochewBench will spur development of better translation tools for under-represented languages.
