Riddles reveal cultural gaps in AI reasoning

Researchers developed CulturalRiddles, a benchmark of over 1,700 original riddles sourced from 62 cultures across 45 countries and 51 languages.

The benchmark aims to test how well large language models (LLMs) understand cultural context, moving beyond simple factual recall. Riddles were created through a community effort to ensure authenticity and require not just language skills, but also reasoning about social customs and specific cultural associations. The team evaluated 28 different LLM setups using both automated scoring and human feedback.

Results showed even the most advanced models struggled, achieving less than 22% accuracy on riddles from some cultural groups. This suggests current AI systems have limited ability to apply knowledge within diverse cultural frameworks. The CulturalRiddles benchmark and its associated data are publicly available, and researchers encourage further contributions to broaden its scope. This work offers a new way to assess and improve AI’s cultural competency as language models become more globally integrated.