Methods lab
Ideas for improving on the plain polling average are tested here before any of them is allowed near a forecast. Each test is written down, including its pass mark, before the results are run. Shown here are the tests that did not make the average less accurate overall; none has yet passed its mark.
E001: Does correcting for pollster house effects beat a plain average?
Rejected: no demonstrated improvement. Across 1,420 races, the adjustment changed the average miss by -0.011 points (95% range -0.057 to 0.027), and helped in 5 of 10 elections. The pass mark, set in advance, was a range entirely below zero and at least 6 of 10 elections.
| Election | Races | Plain average miss | Adjusted miss | Adjustment helped |
|---|---|---|---|---|
| 2006 | 149 | 4.74 | 4.75 | no |
| 2008 | 160 | 4.33 | 4.33 | no |
| 2010 | 190 | 5.62 | 5.65 | no |
| 2012 | 144 | 4.97 | 5.00 | no |
| 2014 | 125 | 5.16 | 5.25 | no |
| 2016 | 126 | 6.13 | 6.12 | yes |
| 2018 | 161 | 4.58 | 4.52 | yes |
| 2020 | 146 | 6.32 | 6.14 | yes |
| 2022 | 97 | 4.45 | 4.45 | yes |
| 2024 | 122 | 4.46 | 4.43 | yes |
Preregistration: docs/research/experiments/E001-house-effects.md
E002: Does subtracting each pollster's own miss against results help?
Rejected: no demonstrated improvement. Across 1,420 races, the adjustment changed the average miss by -0.005 points (95% range -0.210 to 0.172), and helped in 4 of 10 elections. The pass mark, set in advance, was a range entirely below zero and at least 6 of 10 elections.
| Election | Races | Plain average miss | Adjusted miss | Adjustment helped |
|---|---|---|---|---|
| 2006 | 149 | 4.74 | 4.79 | no |
| 2008 | 160 | 4.33 | 4.30 | yes |
| 2010 | 190 | 5.62 | 5.70 | no |
| 2012 | 144 | 4.97 | 4.98 | no |
| 2014 | 125 | 5.16 | 5.56 | no |
| 2016 | 126 | 6.13 | 6.09 | yes |
| 2018 | 161 | 4.58 | 4.92 | no |
| 2020 | 146 | 6.32 | 5.64 | yes |
| 2022 | 97 | 4.45 | 4.70 | no |
| 2024 | 122 | 4.46 | 4.04 | yes |
Preregistration: docs/research/experiments/E002-pollster-error-correction.md
E003: Does it help if midterms learn only from midterms, and presidential years only from presidential years?
Rejected: no demonstrated improvement. Across 1,420 races, the adjustment changed the average miss by -0.027 points (95% range -0.184 to 0.101), and helped in 4 of 10 elections. The pass mark, set in advance, was a range entirely below zero and at least 6 of 10 elections.
| Election | Races | Plain average miss | Adjusted miss | Adjustment helped |
|---|---|---|---|---|
| 2006 | 149 | 4.74 | 4.94 | no |
| 2008 | 160 | 4.33 | 4.33 | yes |
| 2010 | 190 | 5.62 | 5.67 | no |
| 2012 | 144 | 4.97 | 4.86 | yes |
| 2014 | 125 | 5.16 | 5.36 | no |
| 2016 | 126 | 6.13 | 6.36 | no |
| 2018 | 161 | 4.58 | 4.59 | no |
| 2020 | 146 | 6.32 | 5.77 | yes |
| 2022 | 97 | 4.45 | 4.57 | no |
| 2024 | 122 | 4.46 | 4.08 | yes |
Preregistration: docs/research/experiments/E003-same-type-pollster-correction.md
How much does hindsight flatter a backtest?
The same 2024 polling average, replayed at several dates before the election, three ways. Compared on the races all three could estimate, the leaky replays look better than the honest one: that gap is what a careless backtest would wrongly claim. B1: 21-day window, one question per poll, latest poll per pollster, mean.
| Before the election | Shared races | strict miss | listed miss | hindsight miss |
|---|---|---|---|---|
| 90 days before | 2 | 3.81 | 1.69 | 1.91 |
| 30 days before | 62 | 4.45 | 4.07 | 3.88 |
| 14 days before | 51 | 4.56 | 4.13 | 4.35 |
| 7 days before | 17 | 2.09 | 2.25 | 2.31 |
| 1 days before | 0 | — | — | — |
- Strict: only what archived snapshots prove was public
- Listed: by the aggregator's listing time
- Hindsight: by fieldwork date (the leaky way)
- Archived snapshots end: PRES 2024-11-05, HOUSE 2024-10-11, SEN 2024-10-11, GOV 2024-10-11. Strict replays after those dates see no newer polls, so they estimate fewer races.