ttok 1.0 switches its default to the GPT-5/GPT-6 tokenizer
Simon Willison released ttok 1.0, which changes the command-line tool's default tokenizer from GPT-4 to the GPT-5/GPT-6 family; the prior release was 0.4. OpenAI has not confirmed that GPT-6 shares the GPT-5 family tokenizer, but a commit by William Liu documents an experiment in which seven GPT models (5.5, 5.6 Sol/Terra/Luna, and 6 Astra/Sol/Luna) all reported 44,794 tokens and matched on every one of 31 fixtures. On that corpus, the author says, GPT-6 introduces no input-count change.
For tools that rely on token counting, the default tokenizer choice directly affects cost estimates and context budgeting.
I released ttok 0.4, ran uv tool upgrade ttok, piped a file into the new version... and realized that it was defaulting to the GPT-4 tokenizer when it should very clearly default to GPT-5/GPT-6 instead!
I figured switching the default was a reasonable excuse to finally ship a 1.0.
OpenAI haven't actually confirmed that GPT-6 uses the same tokenizer as the GPT-5 family yet - there's an angry issue about it - but I found this commit by William Liu which reports on an experiment he ran confirming that the tokenizers are likely the same:
All seven GPT models (5.5, 5.6 Sol/Terra/Luna, 6 Astra/Sol/Luna) report 44,794 tokens and match each other on every one of the 31 fixtures. GPT-6 introduces no input-count change on this corpus.