Over the last two months, there has been a clear gap between live interview and automated (IVR) pollsters: Obama seems to have a big lead in live polling, but the robots find a closer race. A majority of surveys in key battleground states have been conducted by automated polling firms. While live interviews dominate national polling (every major media poll is conducted with live interviews), only a few live interview firms conduct polls in the battleground states, since they're expensive. On average, the live interview polls that reached out to cell phone voters found Obama performing substantially better than their automated counterparts in Virginia, Ohio, and Florida—three states where multiple live and automated firms have conducted public polls since mid-July.
But the issue extends outside of these three critical East Coast states. Throughout the summer, cheap, mainly automated Michigan polls showed a close race or a Romney lead, while live interview polls showed Obama cruising in Pennsylvania. Nationally, Rasmussen Reports consistently shows Romney performing a net-3 points better (or more) than other likely voter surveys—all of which are conducted with live interviews. And the only survey showing Obama with a narrow lead among registered voters following the DNC was IBD/CSM/TIPP—an automated poll. The weakness of this argument is the relatively small number of live interview state polls, but this week, NYT/CBS/Quinnipiac, NBC/WSJ/Marist, Marquette University, and perhaps CNN will conduct additional live interview polls in battleground states.
If the election was conducted over a live telephone interview, Obama would probably win and clearly. But IVR pollsters have a strong record over the last decade, so their results can't be dismissed out of hand. The problem is that cell phones have seriously complicated the argument on behalf of IVR polling over the last few years. Automated pollsters are unable to contact voters who only posses a cell phone, and those voters are disproportionately young and non-white. While the number of cell phone-only voters was too small to appreciably influence the accuracy of the polls in 2008, Pew Research found that the absence of cell phone voters started to produce a more readily observable bias toward Republicans in landline-only polls in 2010. Cell phone-only voters weren’t just younger and non-white, they were more Democratic than other demographically similar voters, so weighting wasn't a sufficient remedy. When polls generally underestimated the strength of incumbent Democratic Senate candidates in 2010, cell phones were raised as a possible explanation.
The science of cell phone polling is imprecise, as pollsters disagree about whether cell phones should represent 18 or 25 or 30 percent of the sample. But at this point, the debate over whether polls need to reach out to cell phone voters is over; the question is how. One of the most prolific automated pollsters of the last decade, SurveyUSA, has begun to conduct live interviews with cell phone respondents, even thought it makes them vulnerable to low-cost competitors. But some automated firms have adopted indirect, cheaper methods of unproven effectiveness, like Rasmussen's internet panels.
Several automated polls have an outstanding record, but their accomplishments were earned prior to the rise of cell-phone only voters, and it is telling that firms like SurveyUSA have calculated that it was necessary to reach out to cell phone voters, even though it raised the cost of their services. Reporting on the importance of cell phone voters is also consistent with the studies published by Pew Research.
Ultimately, polls should be judged by their methodology, not just their record. A good poll starts with a representative sample, so the possibility that landline-only pollsters can't sample a quarter of the electorate could undermine a prerequisite to their ability to sustain success. At this stage, it's unclear whether the alternative, low-cost remedies devised by Rasmussen or other pollsters can satisfactorily compensate for the absence of telephone interviews with cell phone voters, and the emerging gap between pollsters conducting live interviews with cell phone voters and those that do not suggests that the alternative measures might not quite be up to the task. The gap between the automated and live pollsters could fade in the coming weeks, but if it doesn't, you would rather be ahead in the polls capable of surveying the entire electorate.