Demystifying Sampling Error

Consider the following hypothetical newspaper excerpt:

An October poll of 800 registered voters found that, if the election were held that day, Candidate-X would beat Candidate-Y, 55% to 40%.  While Candidate-X held a 15 point lead, his numbers have “slipped” from a September poll that showed Candidate-X at 58% and Candidate-Y at 39%.

Although data are reported this way all the time, something in the above paragraph is incorrect.  What is it?  Simply put, there is no statistical basis for claiming that Candidate-X’s lead has “slipped” at all.

Why?  Because the margin of error for a poll of eight hundred people is plus or minus 4 percentage points, and Candidate-X’s September-to-October difference fell within that 4-point margin.  So, statistically speaking, there was no change between September and October.

From another angle… if you asked a question from this poll one hundred times, ninety-five of those times the percentage of people giving a particular answer would fall within 4 points of the percentage that gave that same answer in this poll.  Statisticians refer to this as a confidence interval (the “ninety five out of a hundred” referring to the 95% confidence limit).

And here’s another very important way to think about it:  For every 20 times you repeat a poll, one of those times you will get an answer that is completely wrong (because that poll was the one in twenty where the results fell outside the margin of error).

The bottom line:  Never place all of your faith in a single survey.  Only by looking at numerous surveys will you have the most accurate picture.