Iowahawk breaks out the calculator on poll reliability:
So if the sample size is 400, the margin of error is 1/20 = 5%; if the sample size is 625 the margin of error is 1/25 = 4%; if the sample size is 1000, it’s about 3%.
Works pretty well if you’re interested in hypothetical colored balls in hypothetical giant urns, or survival rates of plants in a controlled experiment, or defects in a batch of factory products. It may even work well if you’re interested in blind cola taste tests. But what if the thing you are studying doesn’t quite fit the balls & urns template?
- What if 40% of the balls have personally chosen to live in an urn that you legally can’t stick your hand into?
- What if 50% of the balls who live in the legal urn explicitly refuse to let you select them?
- What if the balls inside the urn are constantly interacting and talking and arguing with each other, and can decide to change their color on a whim?
- What if you have to rely on the balls to report their own color, and some unknown number are probably lying to you?
- What if you’ve been hired to count balls by a company who has endorsed blue as their favorite color?
- What if you have outsourced the urn-ball counting to part-time temp balls, most of whom happen to be blue?
- What if the balls inside the urn are listening to you counting out there, and it affects whether they want to be counted, and/or which color they want to be?
If one or more of the above statements are true, then the formula for margin of error simplifies to
Margin of Error = Who the hell knows?
I think that the disparity among the polls is pretty good evidence of this. A lot of it, particularly the weighting is guess work, educated or otherwise. There’s only one poll that matters (though with all of the chicanery going on, even that one is going to be in doubt, particularly if it’s close on Tuesday). What a mess.