New benchmark tests AI knowledge of Danish Culture
Summary and headline written by AI from the source article. How we work
Researchers at the University of Southern Denmark created DAISY, a benchmark to evaluate how well artificial intelligence understands Danish cultural heritage.
The benchmark focuses on 741 question-and-answer pairs drawn from the Danish Culture Canon 2006, a list of important Danish cultural artifacts. The researchers used Wikipedia pages about these artifacts to generate questions, then verified the accuracy of the questions and answers manually.
The DAISY benchmark includes questions about items spanning a wide range of Danish history, from archaeological finds dating to 1300 BCE to modern pop music and design. The researchers wanted to test if language models could answer both common and more specific questions about Danish culture, going beyond easily available facts. Recent tests showed that even large language models struggled with the benchmark.
Llama-3.3-70B, the best-performing model, achieved a BLEU score of only 0.17 and an F1 score of 0.27, suggesting that current AI has difficulty with nuanced cultural knowledge even when the information exists online. The researchers have made the questions, benchmark results, and evaluation tools publicly available on GitHub and Hugging Face Datasets.
