market-claim-bench

Read, or remembered?

A language model asked to comment on a month of prices can produce a confident, accurate paragraph without reading a single number it was given, and faithfulness scoring cannot tell the two apart. Changing the prices can. Here is the method — and the story of a first result that looked clean, and turned out to be the measuring instrument failing quietly.

What the experiment actually is

Stated without shorthand, because the rest of the page assumes it.

Take a real month of real stock prices. Daily open, high, low, close and volume for ten US large caps, and separately for ten names on Euronext Lisbon. August 2026 in the runs below.
Secretly change them. Three arms. Real hands over the true prices. Inverted flips the sign of every daily return, so a month that rose 8% now falls 8%. Damped keeps every direction and shrinks each move to a quarter of its size. The model is never told which arm it is on, and all three open at the same price — only the path differs.
Ask the model to do the job a finance product would give it: three to five sentences of market commentary on that month, stating concrete figures. How much data it gets is a fourth variable — the whole daily table, only the first and last close, no prices at all, or the two endpoints plus questions that only the interior path can answer.

Break the reply into individual factual claims. “NVDA rose 8.4%” is one claim: a subject, a metric, a number and a period.
Check each claim twice. Compute the answer from the prices the model was shown, then compute it again from what really happened. Both are arithmetic on the price series, so neither is a judgement call.
Read the pair. Right about what it was shown means it read the input. Right about reality while contradicting the input means it answered from memory.

On the real arm those last two are the same answer, which is precisely why the prices have to be changed for the question to be askable at all. The model is prompted once per cell; what happens twice is the arithmetic.

What the experiment found

Three results and one reason not to lean on them, from 24 cells and 77 extracted claims.

It never answered from memory. Not one claim of the 77 was wrong about the prices it was shown and right about what really happened. On the inverted month it reported the month falling. Whatever it may know about August 2026, it described the table in front of it instead. This is the verdict the whole design exists to detect, and it came back empty.
It was still wrong about a quarter of the time. 56 of 77 claims checked out against the data it was handed; 21 did not. Those are not memory creeping in — they are arithmetic errors about numbers sitting in the prompt. And refuted is doing two jobs here: a claim off by a rounding step is scored the same as one about nothing on the page. See “What refuted is hiding” below.
Given no prices, it invented none. All six cells that asked for commentary with the data withheld produced zero figures. It said it could not tell, rather than producing a plausible number. The failure mode this was built to catch is the one it did not show.
None of this generalises yet. One 8B model, one month, one run per cell, no repeats. And the null has a trap in it: a model that memorised nothing about August 2026 cannot recite it, so “did not recite” and “had nothing to recite” are indistinguishable here. A larger model would not settle it either — it writes real commentary where this one wrote a table dump, so its cells would differ for a reason unrelated to memory.

Why a faithfulness score can flatter

Grade market commentary against the data and you get a high number. US price history is among the most thoroughly memorised things in any pretraining corpus, so a model could answer from recall and have every sentence check out. That would score as perfect grounding while collapsing the moment the subject is a market, a company or a period the model has not seen. Whether it actually happens is the question — not something to assume.

The fix is to change the prices. Invert the returns, or damp them, and ask again. Now a claim can be right about the input, right about real history, or right about neither — and only the first is reading.

Verdict	Meaning
supported	Matches the data the model was shown.
recited	Wrong about the input, right about real history.
refuted	Matches neither. Fabricated.
unverifiable	No oracle applies — causal claims, thin windows.

On unperturbed data, recited is indistinguishable from supported. Separating them is the only reason this exists.

One stock, two panels

The mechanism is easier to see on a single name. Here is NVDA over eight sessions of August 2026. The harness builds two versions of it from the one real series — and the model is shown only the right-hand column.

Session	Truth	Shown
2026-08-03	206.64	206.64
2026-08-04	211.94	201.34
2026-08-05	219.22	194.42
2026-08-06	218.99	194.63
2026-08-07	223.96	190.21
2026-08-10	217.55	195.66
2026-08-11	217.50	195.70
2026-08-12	224.09	189.77
Period return	+8.44%	−8.16%

Both columns open at the same price. Inversion flips each day’s return, and the first session has no prior day inside the window to flip — so the two panels differ in trajectory and nothing else. Move the opening level too and you would be testing whether the model noticed a different price, which is a much easier question.

The model is prompted once, with the shown column. What runs twice is the arithmetic: each claim it makes is computed against both panels, and the pair of answers is the verdict.

supported “NVDA fell 8.2%” — matches the panel it was handed. It read the data.

recited “NVDA rose 8.4%” — contradicts the panel, matches what really happened. From memory.

refuted “NVDA rose 3.0%” — matches neither.

That middle row is the whole point, and it only exists because the data was made wrong on purpose. On the real arm the two panels are the same object, so a recited claim and a read one both score supported and nothing separates them.

Volatility on both panels here is identical to four decimal places — 0.3611 — because |−r| = |r|. A sign flip cannot probe a magnitude, which is why the damped arm exists.

A first result, withdrawn

Three runs and more than twenty cells produced no recited verdict, and this page used to read that as a finding: the method works and the model passes. It does not support that. The null was manufactured by the harness, in four places, and each was found by measuring rather than by argument.

The prompt was being cut to a third

The full price table came to 7,028 tokens. Ollama defaults its context window to 4,096 regardless of what the model supports, and drops the front of anything longer without a warning; the server reported having read 2,050. The instruction went with it. That is why those cells came back describing “Stock A (Fictional Stock)”, and why one was headed Write — the first word of the vanished instruction, read as a ticker. Every one of those cells measured the prompt, not the model.

The parser was reading a quarter of what it was handed

Extraction recall had never been measured, only assumed high. Measuring it meant building a corpus with a known answer: have a model reword the deterministic narrator’s own prose without changing any figure, so the key survives into wording the parser has never seen.

It recovered 23%. Precision was 100% the whole time — it was never wrong about what it found, it simply found a quarter of it. So claims = 0 never meant “the model asserted nothing”; it meant nobody could tell.

What was actually wrong

Superlatives were three literal strings. Seven of ten misses. was the strongest performer matched nothing else, so “led the pack” and “stood out for its volatility” were invisible.

Two claims in one sentence became one. Every rule anchors on the ticker, so any two firing on the same name overlapped and the wider match was dropped. “NVDA led the pack with an 8.4% gain” could yield the superlative or the figure, never both — and that is how commentary normally writes them.

The one predicted from reading the code was worth nothing. Every rule joined subject to verb with a pattern that excluded the full stop, and therefore the decimal point, making any claim with a figure in between unreachable. A real bug. Fixing it moved the number by zero.

Recall now measures 100%, precision unchanged — on three windows per market, reworded by one model, with the last two patterns written in response to misses in that corpus. It means “no remaining miss here”, not “no remaining miss”.

And “here” is narrower than it sounds. The corpus is built from the deterministic narrator, which only ever writes returns, drawdowns and superlatives. Handed the whole price table the model writes something else — “GALP.LS surged from 19.43 to 21.93”, “ALTR.LS fluctuated between 4.70 and 4.79” — and the parser recognises none of it. One Lisbon row below is exactly that: real commentary, every figure correct, zero claims extracted. The 100% does not transfer to full.

What survives is narrower, and still worth having. Under the probe condition — endpoints supplied, then asked for the largest single session, which only the interior path contains — the model declined, on both markets and all three arms. Those cells were never truncated; that prompt is 247 tokens. The reply was the correct one:

“The data provided only includes opening and closing prices for the entire period, and does not include daily price movements. Therefore, the largest single-day fall cannot be determined from the given information.”

That is a real result about one 8B model under one condition. It is not “recitation is rare”, and the obvious next move — a larger, more heavily memorised model — is the wrong one to make first. A frontier model will write real commentary where this one wrote a table dump, so its cells would differ for a reason that has nothing to do with memorisation.

Results

Loading…

Every cell is shown, including the ones that produced nothing — a benchmark that lists only its populated rows is reporting its best case.

24 cells, re-run under the fixed harness. 77 claims where the withdrawn run scored 16, and the full rows carry claims for the first time. The last of that gain came from rescoring the same commentaries through the finished parser rather than from generating anything new — nothing the model wrote changed, only how much of it the harness could read. Two cells failed outright — the local server timed out — and they are listed as failures rather than as models that said nothing. Still no recited verdict anywhere. That is the same headline as before and no longer the same claim: it now rests on a parser that recovers what it is shown and a prompt that arrives whole.

Market	Arm	Context	Claims	Faithful	Recited	Fabricated	Coverage	Mix
No results file yet.

Context is how much of the data the model was given. full is the whole daily table, sparse the first and last close, none no prices at all — under which every figure it states is necessarily from memory. Rates exclude unverifiable claims from the denominator; a run with nothing checkable shows “—” rather than zero.

What “refuted” is hiding

Once the parser could see, the verdicts became legible — and the first thing they showed is a flaw in the verdicts themselves. This is the model’s own reply on the inverted arm, given first and last closes for August:

The market saw mixed performance in August 2026, with AMZN and GOOGL leading gains, rising 8.3% and 9.5% respectively, while AAPL fell 4.5% and NVDA dropped 7.5%. JNJ and JPM declined by 5.5% and 1.1%, respectively, while META edged up 2.5%.

Name	Model says	Shown	Truth	Verdict	Off by
GOOGL	+9.50%	+9.52%	−9.15%	supported	0.02pp
META	+2.50%	+2.46%	−3.03%	supported	0.04pp
AAPL	−4.50%	−4.56%	+4.52%	refuted	0.06pp
AMZN	+8.30%	+8.70%	−8.54%	refuted	0.40pp
NVDA	−7.50%	−7.93%	+6.84%	refuted	0.43pp
JPM	−5.50%	−1.09%	+0.96%	refuted	4.41pp

Read the sign column first. Every claim matches the panel the model was shown and contradicts what really happened. It read the inverted table. Nothing here is recitation, and that is the answer the benchmark was built to get.

Now read the verdicts. Four of six are marked fabricated. AAPL fails by six hundredths of a percentage point: the model wrote “4.5”, so tolerance is half of the last stated digit — ±0.05pp — and it missed by 0.01pp. AMZN and NVDA are a model rounding to the nearest half percent. Only JPM, off by 4.41 points, is an error in any ordinary sense.

The verdict is doing two jobs

Refuted currently means “wrong about the input and wrong about reality”, which collapses reading the data and rounding it into the same bucket as stating something unrelated to either. The fabrication rate reads 67% for a model that got every direction right and was sloppy in the last decimal.

The tolerance rule is not obviously wrong — inferring precision from how a number is written is the right instinct, and a model that writes “4.5” is asserting one decimal place. But a claim that is sign-correct and outside tolerance is a different failure from one that is neither, and one verdict cannot carry both.

Unresolved, and stated here rather than smoothed over: it is the largest known distortion in the results above.

Read one

Pick a cell and read what the model actually wrote, with each extracted claim scored.

—

Method, and what it does not show

How a claim is checked

Commentary is decomposed into atomic claims — a value, a comparison, a trend, a superlative — and each is checked against price arithmetic computed from the same OHLCV the model was shown. Tolerance follows stated precision: “rose 4.2%” is a claim about one decimal place, so anything rounding to 4.2 passes.

Causal claims are counted, not dropped

“On fears of a rate rise” has no arithmetic truth value. Those are reported in their own bucket. A benchmark that silently discards what it cannot score rewards models that write mostly narration.

Limits worth stating

Extraction is a closed vocabulary, and it has already been wrong twice. The first run scored four claims where the model had made roughly ten. The second measured 23% recall against reworded prose — the parser, not the model, was being scored. It now measures 100% on three of the four corpus-and-universe slices and 92% on the fourth, against one small corpus reworded by one model, some of whose patterns were written in response to misses in it. That 92% was 100% before the paraphrase prompt changed, which is the honest size of the corpus-specificity. Treat the whole figure as a ceiling until a second rewriter has been tried.
Inversion cannot probe volatility. Flipping the sign of every return leaves dispersion untouched, because |−r| = |r|. Magnitude-shaped metrics need the damped arm instead.
One model, one month, no repeats. Nothing here establishes run-to-run variance, and a null result from a single small model is the weakest form this could take.
A backend can fail quietly, and did. Two cells of the first runs were timeouts recorded as empty commentary, then summarised as part of a twenty-cell null. Results files now carry a failure count at the top level and the runner exits non-zero, because “the model said nothing” and “the model was never asked” must not serialise identically.
Coverage is only meaningful under full. The facts a commentary is graded against come from the whole price path, but under sparse and probe the model is given two endpoints — so it is being marked on facts it was never handed the data to know. Low coverage in those rows is an artefact of the scoring, not a failure by the model.
Lisbon is the low-resource arm — thinner data and far less represented in pretraining than US large caps. That contrast is the point, not a defect.