Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

How Tokens, Transformers and Autoregressive Generation Produce Text

AI summary

This tutorial uses a DeepSeek conversation to explain how text is tokenized, converted into embeddings, processed by Transformer attention and MLPs, and scored into candidate tokens. It also distinguishes parallel processing of existing input from step-by-step generation of new output, while noting that KV Cache can reuse earlier computations. The reported token counts and IDs were reproduced with the DeepSeek-V4-Pro-Base tokenizer, while illustrative logits and probabilities are not measured model outputs.

In-depth insight

The article gives beginners a concrete technical foundation for understanding LLM output, including tokenizer-dependent token counts, causal masking, attention, sampling and the role of KV Cache.

你在 DeepSeek 里问:“为什么大海看起来是蓝色的?”

页面上,回复会一段一段出现,而不是等整句话都写完后一起显示。假设模型给出这样一句回答:

海水会吸收较多红光,蓝光更容易被散射回来,所以大海常呈蓝色。

这次问答会引出三个问题:模型为什么每次只生成一个 Token;输入文字怎样变成模型能计算的数值;模型生成每个 Token 时,怎样利用前面已经出现的内容。

一、模型不是一次确定整句话,而是逐个生成 Token

这句回答是构造示例;以下切分是用 DeepSeek-V4-Pro-Base 官方 Tokenizer 对这串文字实际运行的结果:

#DeepSeek#DeepSeek-V4-Pro-Base

Original · 7 min readRead the original
Group chat

No comments yet. Start the conversation.