o200k_base
AI overview
Sign in and the AI will write an overview from our coverage.
Headlines · 1
- Reddit Creator Explains GPT-4o Tokenization of ‘Strawberry’
A Reddit user published a six-minute animated explainer showing how GPT-4o tokenizes “strawberry,” including real token IDs and a small byte-pair encoding exercise. The creator says “ strawberry” is token ID 101830, while “strawberry,” “ strawberry” and “Strawberry” produce different token sequences. The creator also reports that one Hindi sentence takes 21 tokens with GPT-4’s tokenizer and five with GPT-4o’s, and says all displayed splits were generated using tiktoken 0.14.0.
r/MachineLearning · 🔥 0
Experience and discussion from the community
Share my o200k_base experienceAsk about o200k_base
Nobody has shared their experience with o200k_base yet.