Reddit用户解析GPT-4o对“strawberry”的分词
AI 总结
一名Reddit用户发布了一段六分钟的动画教程,展示GPT-4o如何对“strawberry”进行分词,其中包括真实的token ID和简化的字节对编码过程。创作者称,“ strawberry”对应token ID 101830,而“strawberry”“ strawberry”和“Strawberry”会产生不同的token序列。创作者还表示,同一句印地语使用GPT-4分词器时需要21个token,使用GPT-4o时需要5个,并称所有展示的切分均由tiktoken 0.14.0生成。
I made this explainer for my small channel and would like feedback from people learning this, especially if anything is wrong.
It follows one word the whole way:
- "How many r's are in strawberry?" is 8 tokens in o200k_base (the GPT-4o / GPT-5 tokenizer). " strawberry" is a single token, id 101830, so the three r's are hidden inside one id.
- Why pieces of words: letters make sequences about 4x longer, whole words need a huge vocabulary and can't handle new words.
- A tiny byte pair encoding run on a made-up text, merge by merge, until strawberry becomes st | raw | berry.
- The space belongs to the word: "strawberry", " strawberry" and "Strawberry" are three different token sequences.
- Why reasoning models count letters better: in OpenAI's 2025 paper "Why Language Models Hallucinate", DeepSeek-R1 spells DEEPSEEK out letter by letter before counting, while DeepSeek-V3 answered 2 or 3 (the answer is 1).
- Bonus: the same Hindi sentence took 21 tokens with GPT-4's tokenizer and 5 with GPT-4o's.
Every split comes from running tiktoken 0.14.0. The narration is an AI voice reading my own script.
Video: https://www.youtube.com/watch?v=FdWcUJAhfpo&pp=0gcJCSsMAYcqIYzv
submitted by /u/Rajucg [link] [comments]