Ask an off-the-shelf model to answer as one customer and it does a decent job. Ask it to stand in for a thousand of them and the answers all come out the same. Training it on real answers from real people fixes most of that. Here is the measurement, and what it cost.
Before anything else
In plain terms, and with no shorthand, because everything after this assumes it.
What those two markings said is summarised next, and the rest of the page is the evidence — first the picture that shows why the crowd one matters, then the numbers, before and after.
The short version
Three results and one reason not to over-read them. The rest of the page is the evidence.
The problem, in one picture
Ten real survey respondents, each with their own age, income, politics and location, were shown the same short story and asked the same question. Their answers are on the left. Underneath are the model’s predictions for those same ten people.
“Rate the doctor’s overall quality.”
The real answers spread across six of the seven options. The model answered 6 for every persona — a rating one person out of ten actually gave.
Read any single prediction and nothing looks wrong. A 6 is a perfectly reasonable answer for one person to give. The failure only appears when you look at the group: the simulated population has no disagreement in it.
That matters because a simulated customer base is only ever used to answer group questions. What share would take this offer. Does this message move sentiment. Which segment reacts differently. A population where everyone answers identically cannot answer any of them — every variant scores the same, every segment looks alike.
The idea you need for the rest of this
There are two questions you can ask of a prediction, and a model can pass one while failing the other.
How close was the prediction for this specific individual? Call it accuracy. Useful if you want to know what one named customer will do.
Forget who said what. Does the shape of the predicted answers match the shape of the real ones? If 30% of people said 4, do 30% of predictions say 4? The standard measure for this is Wasserstein distance: roughly, how far you would have to shift the predicted answers to make the two piles line up. Lower is better. Zero means the shapes are identical.
A quick way to feel the difference. Take the real answers and hand each one to the wrong person. Every individual prediction is now wrong, so accuracy collapses — but the crowd is unchanged, so the distribution score is perfect.
The reverse also happens, and it is the trap. Answer “4” to everything and accuracy looks respectable, because on a 1–7 scale the middle is never far from anyone. The crowd shape is the worst of any strategy we tested.
Accuracy rewards playing it safe. The distribution measure punishes it. Simulating a customer base is entirely a crowd question, so the second number is the one that matters — and it is the one nobody looks at.
Where an off-the-shelf model starts
We measured the model against 8,030 real answers from 40 studies it had never seen, alongside some deliberately stupid strategies for comparison.
| Strategy | Accuracy | Crowd distance | What it tells us |
|---|---|---|---|
| Perfect — resample the real answers | 0.744 | 0.075 | the best anyone could do |
| The model, off the shelf | 0.715 | 0.239 | — |
| Pick an answer at random | 0.588 | 0.236 | ties the model |
| Always answer “4” | 0.686 | 0.314 | fine accuracy, worst crowd |
Random guessing gets the crowd as right as the model does — 0.236 against 0.239. The model clearly knows something: it beats random by a wide margin on individuals. None of that knowledge survives when you add the answers up.
This is not a quirk of one model. It is the documented failure of this whole approach, and it is why simulated-customer work tends to produce populations that agree with each other far more than real people do.
What we changed
We took a public research dataset of 2.9 million real survey responses from 400,000 people across 210 peer-reviewed experiments. Each record is one person’s demographics, the exact question they saw, and the answer they gave.
The model was trained on 170 of those studies and then tested on the 40 it had never seen — so it is being asked to generalise to new questions about new topics, not to recite what it memorised.
One record looks like this. Everything below is real: a person who took part in a study on political advertising, the exact question they were shown, and what they answered.
Who they are
What they were shown, and asked
“You were shown an advertisement with threatening music and fearful visuals presenting an immigration message followed by a liberal policy solution. How much do you trust U.S. Customs and Border Patrol to address immigration?” Answer 1 to 4.
What they actually answered
3
The model sees the first two boxes and has to produce the third. Multiply that by 2.9 million and you have the training data — and because the same question was put to thousands of different people, the data carries the disagreement between them, which is exactly what the untuned model was missing.
The held-out set is 8,030 answers across 803 distinct question-and-condition combinations, spanning sociology, psychology, economics, political science and communication. The response formats vary, which is deliberate — a model that only handled 1–7 scales would be of little use.
| Format | Answers | Example |
|---|---|---|
| 5-point scale | 2,550 | strongly disagree … strongly agree |
| Open numeric (0–100) | 1,290 | “what percentage would you say…” |
| 4-point scale | 1,270 | the example above |
| 7-point scale | 1,140 | “rate the doctor’s quality” |
| Binary | 680 | support / oppose, candidate A / B |
| 6-point scale | 470 | likelihood, frequency |
We did not retrain the model. We trained a small add-on called a LoRA adapter — 9.7 million adjustable numbers against the model’s 8.19 billion, about 0.1%. The original model is untouched; the adapter is a separate file you can attach or remove.
That is what makes this affordable. A full retrain of the same model needs eight high-end datacentre GPUs. The adapter needed one rented card for four hours, for about $11.
Results
The same question, the same ten people, the same picture as at the top of this page — with the adapter attached. The model’s weights are identical in both of the lower rows; the only difference is whether the adapter is switched on.
“Rate the doctor’s overall quality.”
One column becomes three. It is not the right three — nothing below 4, where four of the ten real respondents sat — and it is not a fix for individuals: the person who actually answered 1 is still predicted a 6. What changed is that the simulated population now disagrees with itself.
One question proves nothing by itself, and this one improved more than most — it sits in the top fifth of the held-out set. The effect it illustrates does hold across all of them. The flat line in the first chart — every persona returning the identical answer — occurred in 280 of 792 questions before training and 50 after. The average question used 1.85 of the answers available to it before, and 3.02 after, against 4.07 for the real respondents.
Two runs, both training the same kind of adapter, differing only in how much data they saw.
| Run | Training examples | Before | After | Improvement |
|---|---|---|---|---|
| Small trial — a Mac mini overnight | 8,000 | 0.2786 | 0.2475 | 11% |
| Full run — one rented GPU, four hours | 103,314 | 0.2406 | 0.1746 | 27.4% |
More data, more improvement. The small trial was enough to prove the effect was real; the full run is thirteen times the data and roughly two and a half times the gain.
The best published result on this benchmark is a 26% improvement, using a model nearly twice the size trained on roughly 300 times more data. Measured the way that result is measured — the benchmark’s sampled protocol — this adapter improves 18.4%, with a 95% interval of 12.6% to 23.7% across the forty held-out studies. The published figure sits above that interval, so this run does not match it. It gets most of the way there on 0.1% of the parameters, four hours and $11.
Accuracy barely moved: 0.705 to 0.718, an interval of -0.001 to +0.027 that does not exclude zero. That is expected and worth stating plainly. We did not make it better at predicting individuals. We made it better at predicting crowds — which is the part that was broken.
Every question, not just that one
The picture above is one question, and it was chosen because it improved. Here is the rest of the held-out set, including the ones that did not. Nothing is generated as you watch: these are the predictions the two models actually returned during the scored run, replayed dot by dot.
Loading the held-out set…
Point at a dot to see who it is.
This calls a laptop-or-mini-hosted copy of the model and generates the bottom row while you watch, instead of replaying it. It runs the small local adapter — the overnight trial, not the GPU run behind the numbers on this page. They are different models, so treat what comes back as evidence that something is running, not as a score.
674 of the 803 held-out questions are here, across 38 of the 40 studies. The missing 129 are numeric-entry items — “how much would you pay”, sliders with thousands of positions — which are scored the same way but cannot be drawn as a row of ten dots; two studies consist only of those.
Predictions that landed outside the offered scale are pulled to the nearest end, which is what the scoring does too, and counted underneath. Crowd distance is per question here, so it is noisier than the averages above; the headline figures are the mean across all 40 studies, not any one of these.
What follows from this
The small trial saw improvement after a few thousand answers. Volume is not the constraint — ground truth is. You need real answers from real people to a known question. Anything scored or categorical qualifies: CSAT, NPS, offer accepted or declined, churned or retained, escalated or resolved. Without something that actually happened, there is nothing to be right or wrong about, and no amount of compute helps.
Four hours and $11 per training run, on hardware rented by the hour. The expensive part is assembling the labelled data, not the training.
This model was trained to do exactly one thing: read a profile and a question, and answer with a number. That is a small fraction of what a simulation system does — most modes write prose, call tools, or drive a browser. A model tuned to answer with a number is no better at those, and probably worse. The realistic shape is a small specialist for measurable outcomes running alongside the general model, not instead of it.
Today a deletion request is a database delete: the row goes, and we can prove it. Once someone’s answers are trained into an adapter, that information is spread across millions of numbers with no row to remove. The practical answer is to retrain without them, which means keeping the training data and paying for the run again. Anyone building this has to decide how adapter versions track whose data is in them. It is a cheap question now and an expensive one later.
Technical detail
Model. Qwen3-8B with a rank-16 LoRA adapter, 9.7 M trainable parameters (0.118%), targeting all attention and MLP projections. Trained bf16 on one ml.g6e.xlarge (L40S 48 GB), one epoch over 103,314 examples, batch 8 × grad-accum 2, lr 1e-4 cosine, 6,443 steps in about four hours.
Data. SocSci210 — 2.9 M responses, 400,491 participants, 210 studies. Trained on the 170-study split, evaluated on the 40-study held-out split, using the dataset’s own mapping.
Task. Given a demographic profile and an experimental stimulus, predict the ordinal or binary answer that person gave. The label is a single token, so the loss lands on roughly two tokens per example.
The dataset is SocSci210, published openly on Hugging Face: huggingface.co/datasets/socratesft/SocSci210. It is built from the US National Science Foundation’s Time-sharing Experiments for the Social Sciences, a repository of peer-reviewed, nationally representative studies — so the underlying responses come from properly sampled participants, not a convenience panel.
The benchmark figures quoted for GPT-4o, Qwen2.5-14B, LLaMA3-8B and the 26% best result are from the paper that released the dataset, Kolluri, Wu, Park & Bernstein, Finetuning LLMs for Human Behavior Prediction in Social Science Experiments (EMNLP 2025): arxiv.org/pdf/2509.05830. Our own numbers were measured independently against the same held-out split.
Accuracy is 1 − mean(|predicted − actual| / scale range), averaged per study then across studies. Wasserstein distance is computed per (condition, outcome) cell on a [0,1]-standardised scale, averaged within study then across studies. Scale bounds come from the ground-truth responses in each cell, so no metric depends on parsing the stimulus text.
Two estimators are reported. Sampled draws one answer per respondent, as the benchmark specifies: 0.2577 → 0.2103, a 18.4% improvement (95% CI 12.6–23.7%). Exact reads the model’s output probabilities and has no sampling noise: 0.2406 → 0.1746, 27.4% (95% CI 18.6–34.9%). Intervals are a paired bootstrap over the forty held-out studies — the unit the metric averages over — resampled 20,000 times; the adapter was ahead in every resample, on both estimators. On the small Mac mini run the two disagreed sharply, and that disagreement was the evidence the run was under-powered — three respondents per cell cannot resolve an effect of this size. On the full run, at ten per cell, they agree.
Base and finetuned were measured in the same process on the same GPU, the base being that model with the adapter disabled, so nothing but the adapter differs. The untuned baseline was separately reproduced three ways — Ollama 4-bit, MLX 4-bit and bf16 on the GPU — and sits between the published figures for comparable 8B and 14B models. Those three agree to 0.016 on accuracy, but not on the crowd metric: they spread 0.022 on the sampled estimator and 0.038 on the exact one, which is well over half the effect being measured. Quantisation moves output probabilities further than it moves the argmax, and the exact estimator reads those probabilities directly. That is the reason to compare the sampled figure against other people’s work and to treat the exact one as the within-run number it is.
Dispersion, or people? A model that learned only how much a population disagrees would improve the crowd metric exactly as much as one that learned which person says what — Wasserstein compares multisets, so it cannot tell them apart. Accuracy can. Holding the predictions fixed and permuting which respondent each is credited to, within its own cell, preserves the marginal and destroys only the person-level pairing. Over 2,000 shuffles the true pairing beat every one, in both runs: the base model scores 0.7050 against a permuted null of 0.6923, and the adapter 0.7177 against 0.6987 — margins of 5.4 and 6.2 standard deviations. Finetuning increased the person-level signal rather than trading it away for spread.
That signal is real and it is also small. Roughly 0.70 of the adapter’s 0.72 is reachable knowing only the question and nothing about the individual; the person contributes about two points of accuracy, up from about one before training. The difference between those two margins was not itself tested.