ttok 1.0 默认改用 GPT-5/GPT-6 分词器
Simon Willison 发布命令行工具 ttok 1.0，把默认分词器从 GPT-4 换成 GPT-5/GPT-6 系列，此前版本为 0.4。OpenAI 尚未确认 GPT-6 与 GPT-5 系列使用同一分词器，但 William Liu 的一项提交记录了实验：7 个 GPT 模型（5.5、5.6 Sol/Terra/Luna、6 Astra/Sol/Luna）在全部 31 个测试样本上都报告 44,794 个 token，彼此完全一致。据此，作者称在该语料上 GPT-6 没有带来输入 token 数量的变化。
对依赖 token 计数的开发工具来说，默认分词器的选择会直接影响成本估算和上下文预算。
I released ttok 0.4, ran uv tool upgrade ttok, piped a file into the new version... and realized that it was defaulting to the GPT-4 tokenizer when it should very clearly default to GPT-5/GPT-6 instead!
I figured switching the default was a reasonable excuse to finally ship a 1.0.
OpenAI haven't actually confirmed that GPT-6 uses the same tokenizer as the GPT-5 family yet - there's an angry issue about it - but I found this commit by William Liu which reports on an experiment he ran confirming that the tokenizers are likely the same:
All seven GPT models (5.5, 5.6 Sol/Terra/Luna, 6 Astra/Sol/Luna) report 44,794 tokens and match each other on every one of the 31 fixtures. GPT-6 introduces no input-count change on this corpus.