Tel: +44 (0)20 3667 8700
Ask a Silly Question: Why the Polls Got it Wrong

Yes, yet another blog about why the opinion polls failed to predict accurately the 2015 UK general election. But before you switch off, let me assure you that I'm not going to talk about 'shy Tories' (is anyone shy in the internet age?), differential turnout figures, or 'late swings' (another convenient myth in my view). Instead I want to focus on an issue that has been a hot topic in the commercial MR world for at least a decade now: Are we asking the right questions?

Mark Earls (author of Herd and most recently, Copy Copy Copy) once challenged the market research industry to 'stop asking silly questions of unreliable witnesses... or at least stop listening to the answers'. Ouch! I thought this harsh because some of us in MR twigged some time ago that people do not always answer the question we think we're asking them. See a recent blog by Mark Earls by clicking here»

People don't usually 'lie' in surveys (why should they?), but often they don't know their own minds, and sometimes they're really answering a different question to the one we're asking. Thus some may interpret a purchase intent question as a kind of 'brand liking' scale- I'll say I'll buy it because I like it, but I don't really know if I will. Often we think we're measuring behaviour when what we're really measuring is attitude, or a vague disposition.

What evidence is there that people 'don't know their own minds'? Lots: if one takes even a cursory glance at the vast amount of literature published in the fields of neuroscience, cognitive science and behavioural economics over the last few decades, it's clear that much behaviour is driven or framed by heuristics, emotion, or social forces (Kahneman's System 1, in other words). The fact is their behaviour is extremely difficult to predict from attitudinal data - and that's what, essentially polling data is.

So how could we do it better? First of all if we accept that much of our thinking actually lies below the level of awareness and is driven by non-rational (heuristics) and emotional factors, we might reconsider the value of asking voting intention and try something less direct and rational.

A few years ago my company developed a radical new approach called Metaphorix® - using metaphor based animations - which uses no words or numbers and deliberately avoids direct questions. We are a brand research consultancy, not a pollster, and have only used it once in a political context, but to spectacular effect. ITN asked us to conduct a snap poll on the London mayoral election when most (traditional) opinion polls put pretender, Boris Johnson, and the incumbent, Ken Livingstone, neck and neck, and it was widely believed that Ken would retain the crown because of his greater experience and gravitas. We discovered considerable emotional negativity towards Ken, particularly on our key Metaphorix® predictor - brand proximity - which put Ken at -14 and Boris on +11. Indeed, Boris outscored Ken on nearly every emotional measure -including warmth and empathy - and went on to win because as Damasio says, "when emotion comes into conflict with thinking, emotion wins".

Just before the election we ran an experiment using Metaphorix® and an entirely new, implicit approach to personality identification. Cameron's Metaphorix® proximity score was unimpressive at -14, but Miliband's much worse at -25 : not as great a margin as Ken trailed Boris, but big enough to suggest a problem for him, and an opportunity for Cameron. Moreover our implicit data indicated reluctance to identify with Miliband. Using a new implicit technique that measures distance between a brand (or in this case candidate) and self-perception we found that, despite being perceived as much posher, Cameron was much closer to the voters for being caring, sincere and even down to earth. Crucially Cameron almost matched voters' self-perception for honesty (-6) with Miliband at -14.

This narrative is consistent with that put forward by some commentators: that Cameron won not because he is loved but because the alternative was worse. Is this 'emotional differential' sufficient explain Cameron's victory? Maybe, maybe not, but it feels much closer to the mark than all the ineffectual wailing about 'shy Tories' and differential turnouts.

Politicians, marketers and researchers still cling to the belief that people know what they will do and, that when they tell you, you should believe what they say. People (be they voters and consumers) are rarely irrational, but they are frequently non-rational in their behaviours, allowing themselves to be guided by emotion, gut-feel and the feeling of what's right. When they are, market research is often lousy at measuring it. What we pick up can be a post-rationalised set of 'reasons why', rather than a real explanation of their behaviour. It's not only pollsters who 'don't know' because, a lot of the time, we don't know either!

Click here to go back to main blog page»

David Penn's Blog