创建

登录 ReadmeX

或

了解我们如何处理你的信息: 隐私政策

ReadmeX

陪你读懂 AI 动态

今天,想先了解什么?

选一份动态总结,或直接聊聊你的问题。

社区和关注动态总结只属于你自己。

资讯

Reddit用户解析GPT-4o对“strawberry”的分词

AI 总结

一名Reddit用户发布了一段六分钟的动画教程,展示GPT-4o如何对“strawberry”进行分词,其中包括真实的token ID和简化的字节对编码过程。创作者称,“ strawberry”对应token ID 101830,而“strawberry”“ strawberry”和“Strawberry”会产生不同的token序列。创作者还表示,同一句印地语使用GPT-4分词器时需要21个token,使用GPT-4o时需要5个,并称所有展示的切分均由tiktoken 0.14.0生成。

I made this explainer for my small channel and would like feedback from people learning this, especially if anything is wrong.

It follows one word the whole way:

  • "How many r's are in strawberry?" is 8 tokens in o200k_base (the GPT-4o / GPT-5 tokenizer). " strawberry" is a single token, id 101830, so the three r's are hidden inside one id.
  • Why pieces of words: letters make sequences about 4x longer, whole words need a huge vocabulary and can't handle new words.
  • A tiny byte pair encoding run on a made-up text, merge by merge, until strawberry becomes st | raw | berry.
  • The space belongs to the word: "strawberry", " strawberry" and "Strawberry" are three different token sequences.
  • Why reasoning models count letters better: in OpenAI's 2025 paper "Why Language Models Hallucinate", DeepSeek-R1 spells DEEPSEEK out letter by letter before counting, while DeepSeek-V3 answered 2 or 3 (the answer is 1).
  • Bonus: the same Hindi sentence took 21 tokens with GPT-4's tokenizer and 5 with GPT-4o's.

Every split comes from running tiktoken 0.14.0. The narration is an AI voice reading my own script.

Video: https://www.youtube.com/watch?v=FdWcUJAhfpo&pp=0gcJCSsMAYcqIYzv

submitted by /u/Rajucg [link] [comments]

#GPT-4o#o200k_base#tiktoken#OpenAI#DeepSeek-R1#DeepSeek-V3

原文约 1 分钟读完阅读原文
群聊

还没有评论,来说说你的看法。