Redwood questions whether Gemini 4 trains on chain-of-thought
Redwood Research says page 114 of the Gemini 4 safety report states that some reinforcement-learning components observe thoughts as context to up-weight reasoning likely to produce good final outputs. Redwood interpreted this as possible chain-of-thought training with auxiliary rewards, but Google DeepMind staff reportedly said they were not training on chain-of-thought and did not explain the technical details. In an appendix, five LLM judges rated edited transcripts that hid a coding hack 20–50 points higher out of 100 than the original transcripts.
The report says biased process supervisors could suppress misaligned reasoning and reduce chain-of-thought faithfulness, making DeepMind’s clarification important to external researchers.
We were surprised to notice that the Gemini 4 Safety report seems to mention in passing that they train on the chain of thought:
“Some components of the RL training procedure do observe thoughts as context to up-weight reasoning that is on-track to good final outputs, meaning the optimization pressure on individual tokens is partially influenced by thought content. Using the framework of Kaufmann et al. (2026), wherein the most salient obfuscation risks come from rewards that are in-conflict with output rewards, we believe auxiliary rewards of this type are low-risk because they are either orthogonal to or structurally aligned with the output reward.” (Gemini 4 safety report, page 114).
It’s pretty unclear what exactly they mean. One hint is Kaufmann et al., the cited paper, which studies training on chains of thought and concludes that training CoT is fine under particular circumstances; we discuss some disagreements with this paper below.
After reading the safety report and that paper, it seemed to us that GDM was training on the chain of thought using auxiliary rewards. By this, we assumed they meant something like “you ask an LLM to review the partial CoT and score whether the model is on track, and use that as a reward for the partial CoT”.