Language models infer Korean words despite limited recall
Summary and headline written by AI from the source article. How we work
Researchers developed a new benchmark, KoAGE, to test how well large language models understand and create new Korean words.
The framework, called NeoTrace, automatically gathers and dates neologisms, newly coined words, from various sources. KoAGE assesses models on both recognizing existing words and generating new ones, comparing their creations to actual Korean neologisms used online. Analysis of 21 models revealed a gap between a model’s ability to recall definitions and its capacity to understand words in context.
While models only correctly defined 13 to 30 percent of words appearing before a specific cutoff year, they demonstrated an ability to grasp meaning from a single example of usage. However, the models struggled to analyze how established neologisms were formed. When tasked with creating new words, single models often mimicked human word-building patterns, while multiple models working together tended to favor abbreviations, influenced by how the task was presented.
KoAGE allows researchers to distinguish between what language models already “know”, what they learn from exposure to language, and what they can independently “coin”. This work offers a new way to evaluate language models beyond simple memorization, especially for languages with rapidly evolving vocabularies.
