The media seem to be united in their contempt for the polls in the U.S. presidential election. Here’s Freddy Gray in The Spectator:
“Can we just agree never to pay any attention to polls from now on?…The big pollsters all assured us that they had learned the lesson of 2016*…Some projections even gave Biden a 97% chance of winning. Everyone who isn’t delusional knew that was absurd.”
That got me thinking: What does “wrong” mean anyway?
Some pollsters might say, “Well, Freddy, we said Biden’s victory was 97% certain and look what’s happened! “Journalists, on the other hand, would say pollsters got it wrong because his margin of victory is less than their figures suggested. That “blue wave” just didn’t materialise.
So, I wonder: Are polls fundamentally flawed or is it the interpretation of them that’s out of kilter?
Here are my thoughts:
Why believe what people tell you?
If I asked you, “Do you like oranges?”, you could give me a pretty accurate answer. If on the other hand , if I ask, “ Are you going to eat an orange next Thursday?”, the chances of an accurate answer diminish sharply. You may, of course, eat oranges every day, so it’s highly probable that you will eat one that day but, If you’re, like me, an irregular consumer, it’s only possible that you will.
So why, when we ask people who are they going to vote for in an upcoming election, do we believe what they say? My company does commercial market research, not political polling, and, if we test a new baked beans concept, do we believe it when 50% say they will “definitely buy” it? Well, we don’t disbelieve it, but we don’t report to our client that they can expect almost half the population to rush out and buy their product. The point is, (a) figures like this are known to include overclaim and (b) people’s willingness to buy will be affected by all sorts of extraneous factors – price, availability, product performance, etc, etc. That’s why survey data forms only part of predictive models of future behaviour.
The essential problem with polling is that it assumes that people know what they’re going to do and, moreover, can tell you. It’s strange that pollsters are almost the only part of the market research industry to cling wholeheartedly to that belief, although it’s encouraging that one polling company have fairly recently developed a multiple regression model to predict voting intention from previous behavioural activity.
Polls are estimates not audits
When journalists say (as they often do) that the polls “got it wrong”, what they usually mean is that that Biden (or whoever) polled 52%, or 54%, whereas he actually got 50%. In other words, they treat the data as if it’s an audit (with 99.999% accuracy) rather than an estimate with a significant margin of error attached. There’s over 300 million people in the US and, miraculously, it is possible to construct a poll based on a few thousand people that tells you with reasonable accuracy what percentage of people think X or Y. With reasonable accuracy – but even the largest polls are subject to margins of error of +/-2 percentage points. So, 52% for Biden could be 54% or 50%. And 48% for Trump could be 46% or 50% . In other words, they could be neck and neck, or 8 points apart.
A 48/52 split produces a range of possibilities each of which have a probability. A 46/54 split produces the same range of possible outcomes but with very different probabilities. Hence it was always possible (despite Biden’s consistent poll lead) that Trump could win but perhaps not very probable, and so it turned out. In the event, it looks like Biden won by c.3 percentage points and four million votes.
When you’re comparing the voting intentions for two rival contenders, all you can ever be confident of is that A’s figure is higher than B’s. The bigger the gap, the more confident you can be thar A will win. But that’s not the same thing as saying, “A will win by a landslide”. Unless the gap is more than 5 percentage points, we can usually only be c.95% confident that the difference in real. In other words, there’s usually around a 1 in 20 (or 5%) chance that A and B are not different.
Now, 5% is a low probability, but it doesn’t mean it won’t happen. Leicester City won the English Premier League after being given a 1 in 5000 chance by the bookmakers. Low probability events do occur. So, when I hear a journalist wail that that the polls gave a failed candidate, say, a 90% chance of victory, I’m tempted to ask: Would you take a ride in a plane that had a 10% chance of crashing?
Polls are only as good as the people they speak to
If you’ve followed the argument so far, you’re probably thinking, “ Yeah, yeah, but Freddie is right, the US polls forecast a big win for Biden, and it’s much quite closer than that”. Fair point, and I think the lesson of this election is that there may be a systematic polling bias, either in favour of the Democrats or against Trump. It looks to me that Trump’s vote is consistently underestimated and that could be due to sample deficiencies. Perhaps a lot of Trump’s voters just don’t show up on online panels, or maybe – as US pollster, Frank Luntz suggests – they deliberately boycott polls. It could be that polls are seen as part of the “fake news establishment” which Trump and his fans deride.
Then there’s also the ‘social proof’ argument- the one that says some Trump voters ‘lie’ about their preference, because supporting him is not socially acceptable. I’ve always been sceptical about ‘shy’ punters, seeing it as a hangover from the days of telephone and face to face polling. Why, after all, would people lie online, when there’s no human interlocutor to deceive? I think the answer is that people don’t lie, but they can deceive themselves. For example, support in England for the government’s Covid-19 regulations appears to be running at over 60%, outscoring the naysayers by over 2:1. Yet, it doesn’t feel that way; most people seem to accept the impositions grudgingly, rather than ‘supporting’ them. Conforming is probably the socially acceptable route – particularly when it’s mainly the ‘awkward squad’ who oppose the restrictions.
Pollsters tell the media what want to hear
Pollsters love to get on the media and the media love to get stories from pollsters. There’s at best a symbiotic relationship and at worst a kind of Faustian pact. My view is that pollsters take too much credit when they get it right but too little responsibility when they get it wrong. This is not new. Throughout the 80s Robert Worcester (founder of MORI) crowed that the polls in UK were consistently right, up until 1992, when they came a cropper at that year’s General Election. His response? Not our fault guv, the census was out of date.
The problem is that polling is by far the most visible part of the market research industry and frankly I am getting a bit fed up with them periodically pouring a bucket of ordure over the rest of the industry at election times. I know people who hold market research in near contempt and that, in large part, is due to the use and misuse of opinion polling. So, for once, I (almost) agree with Donald Trump: I do blame the polls, or – more specifically – the pollsters.
*Actually in 2016, most polls forecast a marginal victory for Hilary, which was correct in terms of the popular vote, but she did not win the Electoral College.