Context Language Models

Let a language model rewrite its own working memory like a file, instead of following fixed rules.

11.4%

Higher accuracy on BrowseComp-Plus

Beats the best context strategy while also using less compute.

How it works

Context overflows
Long agent tasks outgrow a 32K context. Fixed rules like summarizing at a length limit lose or hallucinate facts.
Context as a file
The live context is mirrored to a file. The model edits it with Bash, and edits sync before its next turn.
Better and cheaper
Without training, it beats summarization and tool-based baselines on research, coding and day-long repo optimization, often with fewer FLOPs.

From the paper