No thanks, I'll keep struggling.

Blog & News


How can it go so wrong when research says it's right?

Category: Blog & News

Two companies. Two well intended product changes. Two expensive and embarrassing examples of how failing to understand real-world behaviour can backfire.


I’m Glad I’m not Milo

Glad wrap and Milo have been in the news this year for the wrong reasons.

GLAD introduced a new method of cutting their cling wrap, moving the cutter from an exposed blade on the box to a lid-mounted.

Nestle changed the recipe of their malted chocolate drink Milo in New Zealand, removing vanilla and changing the mix of vitamins.

In both cases, people flipped out and called for boycotts, leaving the companies scrambling to placate their consumers and shore-up their market share.

As a past product manager I felt their pain.

Making changes to a product is no easy thing, and would have followed extensive business justification and research.

In fact when interviewed a spokesperson for GLAD had this to say;

Before making these changes, GLAD completed rigorous and extensive in-home research in Australia. The results were overwhelmingly positive favouring the changes including the movement of the cutter bar to the lid, with the safety aspect of this front of mind. In fact, more than 60% of those Australians, who participated in the research, preferred the improved product overall.”


So why is there such a chasm between what businesses think their consumers want and what they actually do?

It seems businesses are doing one of three things;

  • getting the wrong kind of research,
  • ignoring what the research is really telling them or
  • cherry picking research to support a decision they want to make

Getting the wrong kind of research

If you are relying on what consumers tell you they want, don’t.

It doesn’t work for predicting the outcome of elections and it doesn’t work for product development. Examples of this ‘say vs do’ gap include;

  • 60% of US voters claiming they’ll turn out to vote but only 40% do
  • 50% of Victorians say they eat healthily but only 7% eat their veggies
  • Americans claim they love Guinness but actually buy Bud Light

It doesn’t work because asking people in focus groups, surveys or interviews to tell you why they did or would do something engages our System 2 rationalising, slow thinking, cold state brain (known often as the “Rider”) which will give you a well intentioned justification of past behaviour or well intentioned rationalisation of future behaviour.

While these answers look great in a research report and can be used as handy stats to quote in PR, the problem is System 2 is unlikely to be the one making the decision to use or buy your product.

System 1, our fast-thinking, impulsive, short-cutting, habitual, hot-state brain is the one who makes most of the decisions everyday (also known as the “Elephant”).

Unfortunately you can’t easily ask System 1 what is likely to happen in the future because merely asking the question prompts System 2 to take the wheel.

Behavioural Economics helps you resolve this in two ways, by providing a;

  1. framework of predictable factors that influence behaviour so you can anticipate what people are likely to do in a given set of circumstances, and
  2. system of creating experiments in observable rather than self-reported behaviour so you don’t have to ask, you can watch

Ignoring what the research is telling them

When you are a product manager your role is to make the product better, and as the expert you spend a lot of time and energy thinking about it. In fact, more time than your consumer ever would.

And that’s a problem because you develop ideas about what changes you should make, commission research about those changes and, thanks to Confirmation Bias, look for results that support rather than disaffirm your views.

Compounding this, the effort you have expended heightens your sense of ownership, making walking away mighty hard due to Sunk Cost - your desire to hold out for a return on the investment you’ve made.

How do you stop yourself from ignoring what research is telling you? Structure the research to disaffirm your hypothesis, looking for reasons not to do what you want to do, and make sure the decision not to proceed with a change is as valid as proceeding so you won’t feel the pain of wasted effort.


Cherry picking research

Where ignoring research can in part happen subconsciously thanks to Confirmation Bias, cherry picking is more deliberate. This is about trawling the results to find anything that justifies a decision already taken, and sadly for consumers, researchers and business culture, all too common. I’ve done it in the past and I’m not proud.

How to prevent cherry picking? Too big a topic to tackle here because it gets to the politics of an organisation and its culture - the Social Norms at play and people’s fear of failure. As a leader though you can at the very least interrogate the research and ask to see the full story, reminding yourself that in today’s world of social media the claims you go to market with are subject to relentless scrutiny.


Who you need to understand best

To avoid the situation Milo and GLAD have found themselves in, you need to know two people best.

  1. Know thy consumer
  2. Know thyself

To “know thy consumer” make sure your research includes a behavioural analysis, drawing on observed as well as (or instead of) self-reported behaviour.

And to “know thyself”, create safe guards to compensate for you and your team’s humanness - your biases and decision-making heuristics - because you will otherwise be setting yourself up for failure.

For more;
Why you should ignore what customers say they want

Say vs Do Gap examples;

This article also appeared in Smartcompany.