StateBench-style Harness
State-tracking chains, bounded replay controls and exact memory tools with reproducible curves.
- Role
- Solo, AI-assisted prototype session
- Stack

Problem
Long assignment chains test whether a model can preserve changing state. A useful harness must separate reasoning failure from context truncation and tool-assisted execution.
What I built
An AI-assisted generator produces immutable, overwrite, self-referential and aliased-reference/swap chains from 8 to 64,000 assignments, with explicit semantics and an exact interpreter. Real Qwen3 0.6B and 1.7B Q4 CPU models are evaluated at 8, 32 and 128 assignments.
The live experiment compares direct answers, model-written JSON scratchpads passed between chunks, and model-delegated execution with persisted state. Model-written values are never repaired from the answer key. Exact token/context guards prevent silent input truncation.
Results
- Both models’ direct accuracy falls from 75% at 8 assignments to 25% at 32 and 0% at 128. Each point contains only four cases, one per difficulty level; all 72 model/method records are audited, with 171 actual calls and zero transport or context errors.
- Model-written state does not improve aggregate accuracy: 2/12 correct for 0.6B and 3/12 for 1.7B. The separate execution-tool method reaches 12/12 for 1.7B, recovering 100 percentage points at 128 assignments. The 0.6B model gets only 1/12 because it frequently fails the tool protocol.
- Tool delegation offloads tracking to a deterministic interpreter, then requires the model to read its output; it is not improved intrinsic memory. Twelve tests pass, including all four generators at 64K. The frontier 13% figure, official dataset, 64K live inference and recurrent hybrid were not reproduced.