CHURN

Cumulative History Under Repeated (state) mutatioNs

A non-saturating benchmark for in-context state tracking without chain-of-thought

Erik Wiffin
Code Email

Abstract

Cumulative History Under Repeated (state) mutatioNs (CHURN) is a process for determining how many state mutations an LLM can track without using reasoning or external tools like a scratchpad. The motivation is twofold: (1) establish a benchmark for how many things an LLM can keep track of at once, and (2) establish a benchmark that will resist saturation.

CHURN works by describing an initial world state of I items with A attributes each, providing k operations each modifying one aspect of the world state, and then measuring the model's accuracy at describing the final state. The headline number is k½: the operation count at which a model's state-recall accuracy first falls below 50%.

Motivation

A story is a state machine wearing prose

A common use-case for LLMs is creative writing. A common failure mode in this use-case is the narrative "forgetting" events and continuing in ways that are physically impossible. Two characters are talking, one turns away, and the next sentence has them noticing each other's facial expression. This phenomenon is obvious in small models, but it's difficult to measure. In larger models, it's difficult to induce without generating and reading large amounts of prose.

The following is an example of this failure mode. We begin with a short story about a cat named Whiskers.

Whiskers and the Ball of Yarn

The afternoon sun came through the bay window in one wide golden slab, and Whiskers had spent the better part of an hour maintaining it. He was a round, opinionated tuxedo cat — white socks, white chin, a smudge of black on his nose like a badly wiped inkstain — and he took sunbeam duty seriously.

The living room belonged to him, though Eleanor paid the mortgage. She sat in the striped armchair with her feet on the ottoman and a cup of peppermint tea balanced on the armrest, pretending to read her library book but mostly watching the cat. On the shelf beside her, Captain Bubbles the goldfish patrolled his bowl, and the grandfather clock in the corner ticked in its patient, disapproving way. It had stopped chiming years ago and now merely cleared its throat at the hour, which everyone agreed was more polite.

At half past three, Whiskers abandoned the sunbeam for the wicker basket by Eleanor's chair and, after a moment of theatrical indifference, extracted her ball of red yarn. He batted it across the rug. He pounced. He wrestled it into surrender, rolled onto his back, and bicycle-kicked it with his hind feet while a long red tail of yarn unspooled behind him, looping around the leg of the coffee table like a tiny crime scene.

Eleanor turned a page. "You're doing very important work," she told him.

Whiskers agreed. He wound up for one tremendous swat — and the ball shot across the rug, skittered past the ottoman, and vanished under the couch with a soft, final thump.

Silence. The clock ticked. Captain Bubbles paused mid-lap.

Whiskers flattened himself against the floorboards and reached one paw into the dark beneath the couch, all the way to the shoulder. His claws found dust, a long-lost bottle cap, the corner of the rug — everything in the world except yarn.

This story introduces several characters (Whiskers, Eleanor, Captain Bubbles), objects (the ball of yarn, the wicker basket, the couch), and a bit of state (the ball of yarn is stuck under the couch). We then use two different models to continue the story.

Gemma E2B (2.3B parameters)

A low, guttural sound rumbled in Whiskers’ chest, a sound somewhere between a purr and a threat. He nudged the dark space again, his whiskers twitching with renewed purpose. The silence of the room, usually a comforting blanket, now felt charged, expectant. He pushed his head further into the gloom, his nose working with the intense focus of a tiny, furry detective. He nudged a stray sock, a forgotten pen, a stray piece of lint—all the mundane treasures of the under-couch kingdom. Nothing.

Frustration, a sharp, prickly sensation, began to prickle at the edges of his concentration. He let out a soft, indignant huff, a sound that was entirely too human for a creature of his refined sensibilities. He shifted, his muscles coiling, and then, with a sudden, decisive movement, he hooked a claw into the edge of the basket. He pulled.

The wicker basket, usually a sanctuary, offered a slight resistance, a stubborn refusal to yield. Whiskers braced his front paws against the woven edge, his back arching in a display of magnificent, feline leverage. He gave a mighty heave, a series of determined pushes and pulls that shook the very frame of the chair. With a final, decisive shove, the basket tipped.

It didn't fall completely, but it tilted enough. And there, nestled in the shadow where the sun no longer reached, was the red yarn. It had tumbled out, a glorious, tangled mess, catching the faint light in a chaotic halo. Whiskers blinked, his ears swiveling, his entire body vibrating with triumph. He didn't hesitate this time. He launched himself forward, a blur of tuxedo fur and pure, unadulterated joy, and with a swift, practiced flick of his paw, he batted the ball of yarn right out of the basket and into his waiting mouth. He began to chew, a satisfied, rumbling sound vibrating deep in his chest, the hunt finally over.

Gemma 31B (30.7B parameters)

He let out a sharp, indignant meow, a sound that clearly communicated the injustice of the situation to everyone in the room. He retreated from the couch and sat back on his haunches, staring at the dark gap with narrowed eyes, as if he could intimidate the yarn into returning through sheer force of will.

Eleanor sighed, setting her tea down. "Now, what have we done?" she asked, though she made no move to get up. She knew the rules of the game; the hunt was the point, and intervening would be a breach of feline etiquette.

Whiskers decided that a frontal assault had failed and opted for a tactical flank. He scrambled to the other side of the sofa, his paws skidding on the polished wood, and attempted to fish the ball out from the opposite end. He could see a glimmer of red—a single, teasing strand of yarn poking out from the shadows—but as he lunged, he managed only to wedge his own head firmly between the couch frame and the wall.

For a moment, he remained there, frozen in a state of profound embarrassment, his hind legs paddling the air. With a sudden, frantic wiggle and a series of muffled huffs, he popped free, sending himself tumbling backward into a heap of tuxedo fur. He lay there for a second, blinking up at the ceiling, before his gaze locked onto the red strand once more. With one final, desperate surge of energy, he dove headfirst into the abyss, his entire body disappearing beneath the upholstery. There was a frantic scuffle, a muffled thud, and then a triumphant, guttural purr.

Whiskers emerged from under the couch backward, his eyes wide and wild, clutching the red ball firmly between his front paws.

As expected, the smaller model does noticeably worse at tracking the state of the ball of yarn. The ball of yarn being stuck under the couch is the critical tension at the end of the introduction! Our smaller model tracks that Whiskers has lost access to it, but teleports the yarn back inside the wicker basket. Since this is a short story, the larger model is able to track the position of the yarn; however, if we were to fully use Gemma 31B's 256K token context window for prose continuation, we would not expect it to accurately track the state of every object or character in the narrative.

CHURN is not an exact measure of this phenomenon, but measures similar capabilities and anecdotally matches the observed capabilities of various models generating prose.

Benchmark saturation

A parallel motivation for the CHURN benchmark is benchmark saturation. We are increasingly seeing previously useful benchmarks get saturated (Phan et al., 2025). Once a model is able to score 100% on a benchmark, further improvements to the model can no longer be tracked. By designing a benchmark that can scale to arbitrarily large k, we should be able to prevent CHURN from saturating as long as context windows continue to grow. As we will see in the results later, a 31B parameter model can barely track 100 state operations (around 2000 input tokens) — unless we start seeing frontier models with unusually short context windows, this is unlikely to be a problem.

Prior research

Entity Tracking in Language Models (Kim & Schuster, 2023) establishes that LLMs can track entity state, and that fine-tuning can improve model performance at this goal, but does not establish a process that can scale with recent advances in frontier models.

Needle In A Haystack — Pressure Testing LLMs (Kamradt, 2023) establishes a measure for an LLM's ability to retrieve information from arbitrarily long context but doesn't measure an LLM's ability to make use of and reason about the entire context.

Show Your Work: Scratchpads for Intermediate Computation with Language Models (Nye et al., 2021) establishes chain-of-thought as a method for reasoning about complicated state. This is specifically a method that is not available for one-shot narrative generation, so we've designed this benchmark to prevent the models from using it.

The benchmark

World

A world is a small set of items, each holding one value per attribute (item → attribute → value). Items come from a fixed noun vocabulary; attributes (color, location, owner, …) each have a closed vocabulary of exactly 10 values, so the chance rate is a uniform 1/10 per cell. The initial state is rendered as prose ("The apple's color is red. Its location is vault. …"), and a seeded RNG makes every world and operation sequence exactly reproducible.

Operations

Each operation is one natural-language statement that mutates the world. Three types, drawn with equal probability:

TypeStatementSemantics
relative "The apple's color is now the same as the car's." Copy another item's current value.
swap "The apple and the car swap their color." Exchange two items' current values.
conditional "If the apple's color is red, it becomes blue." Fires only if the condition matches the current value; half are no-ops the model must recognize.

All three depend on the current state, so the final state cannot be computed without tracking every operation. There is deliberately no unconditional "set" operation: direct writes truncate dependency chains.

These operations have two attractive properties:

  1. They can be randomly generated. Given a set of items and attributes, arbitrary numbers of each operation can be generated as needed. Duplicates will occur, but since each operation is non-idempotent, this is not an issue.
  2. Their impact on state can be deterministically tracked. As we generate operations, we can track a ground truth against which to compare the output of the LLM.

Scoring and floors

The LLM is provided with the initial world state and the sequence of operations, and then prompted for a JSON representation of the resulting world matching a defined schema. Any model we'd be interested in benchmarking should be sufficiently capable of following a JSON schema that invalid representations are treated as accuracy failures.

Models are run with hidden reasoning turned off, no provided tools, and the JSON response prevents them from doing any inline reasoning to calculate state.

Scoring is determined by how many item/attribute pair values the model is able to accurately return in the JSON response. Model accuracy is compared against two model-free floors, since even random chance should result in a score greater than zero.

  • Stale baseline — the score of answering with the initial world state at every k. At low k most cells are untouched, so this floor starts near 100% and decays slowly; a model curve is evidence of state tracking only where it sits above this line.
  • Chance pedestal — the 1/10 rate of guessing within the closed vocabulary, the asymptote both floors share at high k.

The CHURN score

The summary number is k½: the operation count at which a model's pooled accuracy first falls below 50%, linearly interpolated between sampled steps. Scores are reported against the world size as CHURN-IxA for a world of I items × A attributes — the results below are CHURN-5x5; a harder configuration with more items would be CHURN-10x5, and so on.

Results

Four small open models (the Gemma 4 family, run locally, reasoning off, temperature 0) on the standardized 5-items × 5-attributes world. To be clear about what this table is not: it is not a leaderboard, and the ordering is not the finding — of course the 31B model beats the E2B, and models allowed chain-of-thought have been able to solve tasks like this for a while. The finding is that the method located a clean failure point for every model, and the same procedure would locate one for a frontier model, just at larger k.

ModelCHURN-5x5 (k½)
gemma-4-31b86.9
gemma-4-12b63.0
gemma-4-26b-a4b60.7
gemma-4-e2b27.8
stale-baseline floor (parroting the initial state)18.5
Accuracy vs. accumulated operations for gemma-4-e2b
Accuracy vs. accumulated operations for gemma-4-12b
Accuracy vs. accumulated operations for gemma-4-26b-a4b
Accuracy vs. accumulated operations for gemma-4-31b

State-recall accuracy vs. accumulated operations, one panel per model. Colored curve: pooled accuracy with 95% Wilson band. Dashed vertical: k½. Gray dashed: stale baseline (answering with the initial state). Gray dotted: chance pedestal (1/10). Runs early-abort after sustained collapse, so weaker models' curves end sooner.

References