Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

Study finds AI tokenization gap across languages

AI summary

A study by the Coalition for Language Equity in AI Research found that some languages require substantially more AI tokens to process than others. Among the 20 most popular internet languages, the costliest used 1.5 to 5 times as many tokens as the cheapest; the gap in one model grew 17 times when the study expanded to 200 languages. Rest of World also reported that script and representation in tokenizer training data affect usage, with English and dominant-language tokenizers often favoring languages they encounter more frequently.

In-depth insight

Rest of World reports that unequal tokenization can raise costs and latency while reducing effective context capacity for users of less-resourced languages and scripts.

People who use less widely spoken languages could end up paying more to use artificial intelligence than those who speak English or other dominant languages.

A new study published last month by the Coalition for Language Equity in AI Research found AI models require far more tokens for some languages than others. Tokens are the units of text or data that the models process. The researchers have called this a hidden “tokenizer tax,” which can make AI more expensive in languages with smaller online presence. A tokenizer is a tool that breaks down raw text, images, or audio into tokens.

0:00/4:31

Image 1: Text displaying a message in English and Malayalam about the benefits of AI for all people, regardless of their language

Tokenization example. A graphic representation of the tokenization of the sentence “People should benefit from AI, regardless of the language they speak.”in English and Malayalam using The OpenAI tokenizer API (https://platform.openai.com/tokenizer) . The English sentence gets split into recognizable words, while the Malayalam sentence gets multiple fragments per word without respect for multi-byte characters.CLEAR

#Coalition for Language Equity in AI Research#OpenAI#RESPIN

Original · 3 min readRead the original
Group chat

No comments yet. Start the conversation.