small-forecaster

Just scale it up?

A small learned weather forecaster, built on GraphCast's recipe, was scaled four ways and graded against the European weather centre's own forecast. Scaling closed most of the gap for tomorrow and the day after, and little of it for day five.

What the experiment actually is

Stated without shorthand, because the rest of the page assumes it.

Pick the yardsticks. The weakest honest forecast is persistence: “in six hours it will be what it is now”. The strongest available is ECMWF's operational forecast. Every score below is the share of the distance from the first to the second that a model covered, averaged over a year: 0% is no better than persistence, 100% is as good as ECMWF.
Teach a small network one step. Show it the weather map over the North Atlantic and Europe now and six hours ago, and ask for the map six hours from now. Eight years of past maps, 2010–2017, are the lessons.
Chain the steps. Feed each answer back in as the new “now”, twenty times, to reach five days.

Grade it on a year it never saw. Every day of 2020, against ERA5 — the reconstructed record of what the weather was — and against what 268 airports actually measured.
Change one thing and do it again. More years, a wider map, layers higher up in the atmosphere, a bigger network.
Train every version twice from different random starts, so the difference between two identical runs is known before any change is called real.

One moment, five answers

Barcelos, forecast made at midnight on 1 July 2020, for 13:00 that day.

Forecaster	Said
Persistence — “it's 17.7 now, so 17.7”	17.7 °C
The 30-year average for that day and hour	23.9 °C
ECMWF	22.2 °C
The largest model here	21.1 °C
What happened	22.0 °C

What the experiment found

Scaling closed most of the gap for the first two days and little of it at five. The best model covers 88% of the distance from persistence to ECMWF for temperature one day ahead, 39% at three days and 6% at five. One test year, 2020, over one region; graded against ERA5 over the whole map.
Each lever moved a different thing. A wider view fixed the first two days, upper-air layers fixed pressure and wind, and only a bigger network moved temperature at three and four days. More years of data helped least. Each slope rests on two or three settings, two runs each.
Over land a bigger network kept paying; over the sea it stopped. Graded at the airports, the share closed at five days went from 4% to 10% to 23% as the network grew 4× and then 9×. Over open sea every model is worse than persistence at five days, and the largest size added nothing. The bigger networks also trained longer; size and computing time are not separated.
ECMWF still wins at six hours. Graded against ERA5, the small model looked better than ECMWF at six hours from the start. Against airport thermometers it is not, at any size. That early win was how it was graded, not the weather — see below.

The ladder

Share of the persistence-to-ECMWF gap closed, 2020, mean of two runs per row (the first row had one). “ERA5” grades the whole map, 25–60°N, 45°W–15°E; “airports” grades 268 lowland airports in it. Two identical runs differ by about 1 point up to one day ahead and by 3–8 points at three to five days, so smaller changes are not claimed.

Step	What changed	Temp 1 day	Temp 3 days	Temp 3 days, airports	Pressure 1 day	Wind 1 day
1	2 years, map from 45°W	61	18	26	49	53
2	8 years	63	14	26	53	57
3	Map widened to 95°W	71	21	31	60	66
4	+ upper air, 1.5 and 5.5 km up	66	12	29	68	78
5	Upper air at ¼ weight	74	21	31	68	78
6	Network 4× (7.6 M parameters)	84	34	38	71	81
7	Network 9× (17 M parameters)	88	39	49	72	83

More years helped pressure and wind by 2–5 points up to three days, and temperature not at all past one day. Both steps trained for the same number of steps.
A wider view was the largest single gain at one and two days. The original map began at 45°W, and weather arrives from the west: the model was asked to forecast what flowed in from outside its view. The worst error sat at the western edge at 12–24 hours, and widening the map removed it. It did not change days three to five.
Upper air helped pressure, wind and rain, and made temperature worse: the same network now predicted fifteen fields instead of five, so surface temperature carried a third of the training weight. Giving the ten upper-air fields a quarter of the weight gave temperature back and kept the other gains.
A bigger network was the only change that moved temperature at three and four days. It also took longer to train: 2.5, 5 and 12 hours per run on one Mac mini.

The six-hour win that was not

Graded against ERA5, the small model beat ECMWF six hours ahead from the first version on — at step 3, 0.70 °C of temperature error against ECMWF's 0.83. The model learned from ERA5, so it was being graded by the source it imitates — and ERA5 is smoother than any real forecast. ECMWF's maps carry about twice the fine detail of ERA5's, and a score that squares every error penalises detail that lands slightly in the wrong place.

Against airport thermometers, ECMWF wins at six hours on all three fields, at every model size. The last column is ERA5 itself scored as a forecast: one number per 28 km grid square cannot match one airport, so that is roughly the floor, and ECMWF at six hours sits on it.

The ERA5 scores on this page are kept as measured. They are fair for comparing the models with each other; they are not evidence that any model beats ECMWF.

Six hours ahead	ECMWF	Smallest	Largest	ERA5
Temperature, °C	1.79	1.92	1.89	1.74
Pressure, hPa	0.71	1.01	0.91	0.72
Wind speed, m/s	1.74	1.83	1.80	1.82

Typical error against 268 airports, 2020. “Smallest” is step 3, the first on the wide map.

Land and sea

Graded against ERA5 over the whole map, the second size step paid less than the first at three to five days. Graded at the airports, it paid as much. Removing each airport's fixed offset from its grid square did not reconcile the two. Grading against ERA5 only at the airports' own squares did: the disagreement was about place, not about the grader.

The map is about two thirds sea. Sea temperature moves slowly, persistence is hard to beat there, and the distance from persistence to ECMWF is small, so the share closed swings widely. Over sea the largest model is still worse than persistence at five days.

ERA5 reads 0.35 °C warmer than the lowland airports on average, and the largest model drifts 0.5 °C cold by day five, so part of its airport score is the two cancelling. With each forecaster's average error removed first, the share closed at three days is still 35, 41.5 and 48.5 across the three sizes.

Temperature, share closed	1.9 M	7.6 M	17 M
3 days, ERA5 at airport squares	33	41	50
3 days, airports	31	38	49
3 days, ERA5 over all land	39	50	55
3 days, ERA5 over all sea	3	17	21
5 days, ERA5 at airport squares	2	10	21
5 days, ERA5 over all sea	−45	−20	−21

Also tried: training on its own mistakes

Trained one step at a time, the model only ever starts from a true map; a five-day forecast starts nineteen times from its own guesses. Fine-tuning on four chained steps raised temperature at three to five days by 8–10 points. An equal amount of ordinary extra training gave 1–4 of those, leaving 4–9 for the chaining — borderline against the noise. For wind and rain the extra “gain” was blur: the forecast maps lost 35–55% of their fine detail, which lowers the error score without being a better forecast. Not carried forward.

Method, and what it does not show

Method

WeatherBench 2's public copy of ERA5 at 0.25°, every six hours. Surface: 2 m temperature, 10 m wind, sea-level pressure, six-hour rain. Upper air: geopotential, temperature, wind and humidity at 500 and 850 hPa. Training 2010–2017, validation 2018, test 2020. ECMWF HRES 00 UTC runs and airport (METAR) reports from the same source.

A three-level U-Net, base width 48, 96 or 144 channels: 1.9 M, 7.6 M and 17 M parameters. Input: the state six hours ago and now, land–sea mask, terrain height, position, time of day and of year. Output: the six-hour change. Latitude-weighted squared error, AdamW, ten epochs, batch 16, seeds 0 and 1, on an M4 Mac mini. The scorer reproduces WeatherBench 2's published ECMWF score for Europe to four decimal places.

What it does not show

One year, one region. Nothing here says how the slopes behave elsewhere or in other years.
Slopes, not projections. Each lever was measured at two or three settings. Where the lines would meet ECMWF is not estimated.
Airports measure points. Lowland airports only; wind as speed, since reports carry no direction vector; pressure from the altimeter setting. No airport check for rain.
Rain is graded against ERA5 only, itself a model's estimate, so it is not one of the headline fields.
Not GraphCast. The same recipe — two states in, the change out, rolled forward — on a plain image network and two levels instead of 37. GraphCast trained for about four weeks on 32 TPU v4 chips.
Pressure drift. Several runs drift by about 1 hPa over five days, in either direction. Not explained.