The Pensieve Paradigm: Stateful Language Models Mastering Their Own Context

ABSTRACT

A fundamental limitation of current LLMs is their stateless, autoregressive nature. They are passive predictors designed to perform sequence completion within an externally-provided context, unable to actively manage their own reasoning process. We introduce Stateful Language Models (StateLMs), a new class of foundation models endowed with a learned capability to self-engineer their context. StateLM maintains an efficient 'sawtooth' context length by using tools like deleteContext, readChunk, and updateNote. This allows the model to establish a dynamic reasoning loop: reading a segment, recording key info into persistent notes, and removing raw text to maintain a compact, noise-free state regardless of total context length.