The Thinking Leakage in Reinforcement Learning for LLMs

RL in no-think mode mostly wakes up a model's existing thinking mode instead of teaching new reasoning.

75%

RL gain removed by subspace patching

Undoing just an 8-dimensional think subspace erases most of RL's accuracy gain

How it works

Suspicious no-think gains
Qwen3 models trained with RL in no-think mode score higher. But think and no-think modes share the same weights.
Probe weights, activations, heads
Train Qwen3-1.7B/4B/8B with GRPO, track think-likeness, then scale updates, patch activation subspaces and single attention heads.
Gain is borrowed thinking
Scaled updates approach but never beat think-mode accuracy; patching a think subspace removes 75% of the gain.

From the paper