研究发现AI分词存在语言成本差
语言公平联盟(Coalition for Language Equity in AI Research)的研究发现,不同语言处理所需的AI token数量存在显著差异。在互联网最常用的20种语言中,成本最高的语言比最低者多用1.5至5倍token;当研究扩展至200种语言时,某个模型的token成本差距扩大至17倍。Rest of World还报道称,文字系统及分词器训练数据中的语言覆盖度会影响使用量,现有分词器往往更有利于英语等高资源语言。
Rest of World报道称,分词不平等可能增加低资源语言用户的使用成本和延迟,同时压缩可用的上下文容量。
People who use less widely spoken languages could end up paying more to use artificial intelligence than those who speak English or other dominant languages.
A new study published last month by the Coalition for Language Equity in AI Research found AI models require far more tokens for some languages than others. Tokens are the units of text or data that the models process. The researchers have called this a hidden “tokenizer tax,” which can make AI more expensive in languages with smaller online presence. A tokenizer is a tool that breaks down raw text, images, or audio into tokens.
0:00/4:31

Tokenization example. A graphic representation of the tokenization of the sentence “People should benefit from AI, regardless of the language they speak.”in English and Malayalam using The OpenAI tokenizer API (https://platform.openai.com/tokenizer) . The English sentence gets split into recognizable words, while the Malayalam sentence gets multiple fragments per word without respect for multi-byte characters.CLEAR