Pollsters can expect to take their share of blame when their campaigns lose, and this year has been no exception. Not long after Barack Obama and Democrats had a strong night on Nov. 6, Republicans began to complain publicly that the polls conducted by their campaigns and by affiliated groups implied considerably more optimistic outcomes for them than actually occurred.
Perhaps these Republicans shouldn’t have been so surprised. When public polls conducted by independent organizations clash with the internal polls released by campaigns, the public polls usually prove more reliable.
Take, for example, the gubernatorial recall election in Wisconsin earlier this year. Independent polls had the Republican incumbent, Scott Walker, favored to retain his office by about six percentage points. A series of polls conducted for Democratic groups showed a roughly tied race instead.
Mr. Walker in fact won by seven points: the independent polls called the outcome almost exactly, while the internal polls were far from the mark.
Cases like these are fairly typical. My database of campaign polls released to the public in United States House races found that they were about six points more favorable to their candidate than independent surveys on average — and that they were typically less accurate in the end.
The traditional explanation for this phenomenon is that the subset of campaign polls that are released to the public is subject to a type of selection bias. Campaigns conduct polls all the time, but only occasionally disclose these results to the public and will be much more inclined to do so when the numbers are favorable for their candidates (especially in comparison to independent polls). In essence, the internal polls that filter their way into the public domain may be the outliers.
This is certainly an important part of the story, but my view is that it lets the campaigns off a little too easily.
Pollsters must make a lot of choices and assumptions about turnout, wording of questions, whether to include third-party candidates in the polls, how hard to push “leaners” toward their preferred candidates, among other factors. These can be difficult choices, even if one is operating in good faith in an effort to be as accurate as possible.
But when campaigns release internal polls to the public, their goal is usually not to provide the most accurate information. Instead, they are most likely trying to create a favorable news narrative — and they may fiddle with these assumptions until they get the desired result.
Some reporters make the mistake of assuming that information is valuable simply because it is private or proprietary. But the information that makes it to the reporter’s ears, or into his in-box, may be something that the campaign wants him to hear or see.
The release of internal campaign polls is not that different from any other form of spin in this respect — but that is precisely the point. Internal numbers that a campaign releases to the public should be thought of less as scientific surveys and more as talking points.
But sometimes, internal polls make their way to the public through leaks that are not authorized by the campaigns. Or reporters and analysts may see the “real” numbers, or reasonably explicit characterizations of them, on background, on the condition that they not report them. (I saw some of the Obama campaign’s internal polling in 2008, along with that of some other Republican and Democratic campaigns in a few instances over the years.)
Perhaps these internal polls are more accurate?
My experience has been that these polls can also exaggerate the standing of their candidate, if perhaps not by quite as large a margin as those that are authorized for an on-the-record release. An interesting example of this comes from Noam Scheiber of The New Republic, who says he received data on Mr. Romney’s internal polling in six states from an aide to Mr. Romney’s campaign. In addition, Mr. Romney’s chief pollster, Neil Newhouse, disclosed the campaign’s polling to Mr. Scheiber in a seventh state, Ohio.
On average, the polls had Mr. Obama ahead by just one point between the seven states. They had Mr. Obama trailing in Colorado and New Hampshire and tied with Mr. Romney in Iowa.
In fact, Mr. Obama won all seven states, and by an average margin of 5.7 percentage points based on the ballots counted so far. (Several of the states have yet to certify their results.) Therefore, the polls were biased in Mr. Romney’s direction by nearly five percentage points, on average.
It should be mentioned that most of the independent polls this year were also slightly biased (in a statistical sense) toward Mr. Romney. In the same seven states, the final Real Clear Politics averages overrated Mr. Romney’s standing by 2.5 percentage points. The final FiveThirtyEight forecasts were less biased statistically, in part because our forecast model was designed to respond aggressively to movement in the polls in the closing days of the campaign, which favored Mr. Obama after Hurricane Sandy. Still, our forecasts in these seven states had a one-point bias toward Mr. Romney, on average, compared against the actual results.
Further, the Romney campaign’s polls did not have him winning the Electoral College, as they had Mr. Romney behind in Ohio, Minnesota, Pennsylvania and Wisconsin, according to Mr. Scheiber. Assuming that the campaign also had Mr. Romney trailing in Nevada and Michigan, but leading in Virginia, Florida and North Carolina, then Mr. Romney would have been leading in states with 261 electoral votes, but trailing Mr. Obama in states with 271. Mr. Romney’s internal polls implied better chances for him than the public polls did — but if Mr. Scheiber’s reporting is right, they still had him as the underdog.
Nor is it clear which strategic choices Mr. Romney might have made differently had his campaign’s polls been more accurate. It isn’t as though, if Mr. Romney felt he had a 10 percent chance of winning instead of a 50 percent chance, he was simply going to stop campaigning.
Nonetheless, the seeming inaccuracy of Mr. Romney’s internal polls ought to present a warning to future campaigns. The problems with internal polls may run deeper than the tendency for campaigns to report them to the public in a selective or manipulative way. The campaigns may also be fooling themselves.
Our self-perceptions are very often more optimistic than the reality; 80 percent of people think they are above-average drivers, for example.
These problems can be worse when we join together to form businesses or organizations. Honest self-assessment is a challenge for any business, and it is one reason that management consultants are sometimes engaged at considerable expense to provide a supposedly more objective and unbiased take on the state of the organization’s operations. (Much of Mr. Romney’s success in business, of course, came precisely because he was able to identify companies whose organizational cultures prevented them from functioning efficiently.)
A pollster working within a campaign may face a variety of perverse incentives that compete with his ability to produce the most accurate possible results to his candidate. He may worry about harming the morale of the candidate or the campaign if he delivers bad news. Or he may be worried that the campaign will no longer be interested in his services if the candidate feels the race is hopeless.
Groupthink and confirmation bias are also risks in any organization, particularly under the stress of the end stages of a political campaign.
So should the campaigns simply give up on conducting their own polls – and instead base their decisions on the forecasts issued by Real Clear Politics or FiveThirtyEight?
If internal polls were solely used to diagnose the state of the horse race, then there might be an argument for this — at least in presidential elections, where the public polling is normally quite reliable.
But the campaigns also use internal polling for message testing and a variety of other purposes. And they may desire a more granular take on the race than the public polls provide — for example, in an effort to measure the effectiveness of an advertisement in a particular media market, or among a particular demographic subgroup. Pollsters like Mr. Newhouse are an important part of campaigns, and most of them do their jobs well.
Campaigns should foster organizational cultures in which their pollsters are enabled to provide the most value.
Campaigns might consider how pollsters are compensated; they could tie some of the pollster’s compensation to the accuracy of its final polls, for instance.
Some campaigns have had success with hiring more than one pollster and having them work relatively autonomously from one another. This can serve as a check against groupthink — and may increase the likelihood the different assumptions that the pollsters might introduce will be thought over and debated.
But most important, campaigns would be wise not to have their pollsters serve as public spokesmen or spin doctors for the campaign. Campaigns have other personnel who specialize in those tasks.
The role of the pollster should be just the opposite of this, in fact: to provide a reality check such that the campaign does not begin to believe its own spin.
An earlier version of this post referred to a series of statements made under oath by the Democratic pollster Harrison Hickman, who testified in the federal case against John Edwards. The post incorrectly characterized Mr. Hickman as describing the release of internal polls to the news media as a form of “propaganda.” Instead, Mr. Hickman's statements in court referred to public polls conducted by independent organizations. During the trial, Mr. Hickman said that he was more likely to circulate public polls that showed more favorable results for his candidate, Mr. Edwards, even if they contradicted his own views on the race. The statements did not pertain to internal polling conducted by Mr. Hickman, and Mr. Hickman said in court that he had told Mr. Edwards that he had little chance to win the Democratic nomination in 2008. "The prosecutors were in possession of all of the polling we conducted for the Edwards campaign as well as all other documents we created for them and all of my email to the campaign or sent on behalf of the campaign," Mr. Hickman said in an email to FiveThirtyEight. "The prosecutors most certainly would have raised questions about the accuracy of our polling or our characterization of it if there were legitimate questions that could have been raised. In any case, there were no internal polls involved."