Is there a problem with the New York Times and the Kaiser Family Foundation’s poll of four U.S. Senate elections? After the poll results were published Monday morning, the methodology was called into question, particularly in response to a poll of the Arkansas Senate race between incumbent Democrat Mark Pryor and Republican challenger Tom Cotton. Most previous polls showed the race fairly close, with a small lead switching back and forth between the two candidates, depending on the poll. The Times poll, however, showed a 10-point advantage for Pryor, his largest yet.
How could this be? It could be an outlier, the result of some statistical noise that isn’t reflective of the reality. After all, the Pryor campaign's own internal poll, released last week, shows the Democrat only up by three points. Or it could be a leading indicator that Pryor is winning the race, at least for now. But there were some curious elements in the poll’s sample that suggest the results may not be accurate. For instance, of those registered voters polled, 27 percent said they voted for Mitt Romney in the 2012 election, while 26 percent said they voted for Barack Obama and 32 percent said they didn’t vote at all. That’s a one-point difference between Romney voters and Obama voters. But in Arkansas in 2012, Romney beat Obama by 24 points. There were some similarly disparate findings between the 2012 reality and the sample of 2012 voting preference in the three other states the Times polled: Kentucky, Louisiana, and North Carolina.
If the poll did not accurately capture how Arkansas voted in 2012, how can we be sure it’s accurately capturing how Arkansas will vote in 2014?
Facing this criticism, the Times’s Nate Cohn responded twice. In his first response, Cohn offered some caveats the poll’s findings. Southern Democrats, he cautioned, still had problems ahead in 2014 despite their good showing in the poll. He admitted, too, that “Pryor probably does not have a 10-point lead,” but added, “it does seem clear that Mr. Pryor has retaken the lead in a race that many analysts believed was tilting toward Mr. Cotton, who has not led in a survey of any kind since February.”
It’s fair analysis, as far as it goes, but it doesn’t address the questions about the Times poll’s sample. Is it clear that Pryor has “retaken the lead” from a poll that has a sample that voted nothing like the actual electorate did in 2012 in Arkansas? Cohn addressed this more directly in a second response, in which he said the critique about the 2012 voting preference sample was “understandable, but misguided.”
Here’s Cohn:
At first glance, the self-reported vote for 2012 in the poll data does seem off. In Arkansas, for instance, Mitt Romney had only a one-point lead, 27 to 26 percent, with 32 percent of respondents saying that they didn’t vote. In 2012, Mr. Romney won Arkansas by a huge margin, 60 percent to 36 percent. It’s not surprising, then, that readers are suggesting that the poll favors Democrats, especially considering the poll’s strong showing for Mark Pryor, the Democratic candidate. But there’s a well-known bias toward the victor in post-election surveys. Respondents who voted for the loser often say that they don’t remember whom they supported, or say they supported someone else. The signs of that bias are obvious upon closer examination. The poll accurately captures Mr. Obama’s support, but tends to underestimate Mr. Romney’s performance.
Cohn argues that the disparity in the poll’s sample comes from this victor’s bias; that is to say, the sample includes fewer people saying they voted for Mitt Romney in 2012 than actually did. If that were the case, that one-point difference between Romney and Obama voters is much larger and being obscured by this bias. Furthermore, there are other indications the poll accurately reflects reality. Obama’s approval ratings in Arkansas (32 percent approve, 59 percent disapprove) aren’t far off from what other polls have found. For Cohn, and the Times, this is enough reason to have confidence in the results.
But is it? The victor’s bias Cohn mentions is a real polling phenomenon, though it usually affects polling by a few percentage points, and it manifests itself when the victor in question is either very popular or very unpopular. One extreme example came in early 2002, right after 9/11 and when George W. Bush was at 82 percent support. Voters told pollsters they supported Bush over Al Gore in the 2000 election by 12 points.
If the victor’s bias were to explain the disparity in the Times poll, though, there’d be more reason to think Romney’s margin of support would be greater than 24 points, considering how unpopular Obama is in Arkansas right now.
The Times could clear this up by releasing the cross-tabs and internal polling data for the rest of us to discover why this disparity exists in the results. Pryor may have the advantage the Times poll says he has. But this apparent error in the sample doesn’t inspire confidence in its results.