Study finds AI tokenization gap across languages
A study by the Coalition for Language Equity in AI Research found that some languages require substantially more AI tokens to process than others. Among the 20 most popular internet languages, the costliest used 1.5 to 5 times as many tokens as the cheapest; the gap in one model grew 17 times when the study expanded to 200 languages. Rest of World also reported that script and representation in tokenizer training data affect usage, with English and dominant-language tokenizers often favoring languages they encounter more frequently.
Rest of World reports that unequal tokenization can raise costs and latency while reducing effective context capacity for users of less-resourced languages and scripts.
People who use less widely spoken languages could end up paying more to use artificial intelligence than those who speak English or other dominant languages.
A new study published last month by the Coalition for Language Equity in AI Research found AI models require far more tokens for some languages than others. Tokens are the units of text or data that the models process. The researchers have called this a hidden “tokenizer tax,” which can make AI more expensive in languages with smaller online presence. A tokenizer is a tool that breaks down raw text, images, or audio into tokens.
0:00/4:31

Tokenization example. A graphic representation of the tokenization of the sentence “People should benefit from AI, regardless of the language they speak.”in English and Malayalam using The OpenAI tokenizer API (https://platform.openai.com/tokenizer) . The English sentence gets split into recognizable words, while the Malayalam sentence gets multiple fragments per word without respect for multi-byte characters.CLEAR