Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

o200k_base

AI overview

Sign in and the AI will write an overview from our coverage.

Headlines · 1

  1. Reddit Creator Explains GPT-4o Tokenization of ‘Strawberry’

    A Reddit user published a six-minute animated explainer showing how GPT-4o tokenizes “strawberry,” including real token IDs and a small byte-pair encoding exercise. The creator says “ strawberry” is token ID 101830, while “strawberry,” “ strawberry” and “Strawberry” produce different token sequences. The creator also reports that one Hindi sentence takes 21 tokens with GPT-4’s tokenizer and five with GPT-4o’s, and says all displayed splits were generated using tiktoken 0.14.0.

    r/MachineLearning · 🔥 0

Experience and discussion from the community

Share my o200k_base experienceAsk about o200k_base

Nobody has shared their experience with o200k_base yet.