Download as PDF, PPTX
Uploaded byEmma Howell
Bias in Research
The document discusses inherent biases in research, exemplified by Coca-Cola's New Coke debacle in the 1980s, where biased design and oversight led to flawed decision-making despite extensive testing. It emphasizes the importance of recognizing and mitigating biases in research sessions, from planning to execution and analysis. Suggestions for reducing bias include challenging assumptions, using diverse participant recruitment, and avoiding leading questions.
Bias in Research
- 1. Bias in Research EmmaHowell @oohmawolfie
- 2. Every bit ofresearch I have ever done is biased
- 3. Every bit ofresearch I have ever done is biased you
- 4. Standard Operating Procedures: Anorder of actions/activities/ sentences that are kept constant across participants.
- 5. Don’t throw sandwiches at baboons. Itmakes science bad.
- 6. I’ve moved from primatesand flying sandwiches to limited budget and leading questions
- 7. In 1983, Coke’smarket share plummeted In the mid 1980s, the youthful branding of Pepsi saw Coke’s market share decline from 60% to under 24%.
- 8. They developed anew, sweeter recipe Coke executives developed a new, sweeter formula. They carried out 200,000 blind taste tests. More than half preferred the new flavour.
- 9. But, consumers hatedNew Coke There were months of protests and over 400,000 complaints. Within 3 months, New Coke was pulled and the original recipe back.
- 10. What went wrong? Theyhad carried out a quarter of a million taste tests and spent $4 million on development. They were left with $30million of New Coke.
- 11. Their research wasbiased.. Which lead to a huge mistake in their interpretation of the findings. They: • Only tested in one area • Used closed questions in surveys • Failed to consider brand perception • Only tested taste.
- 12. Coke, 1980s researchIntegrity You may design the wrong stuff Wrong stuff can cost money
- 13. Coke, 1980s research Let’sthink about how your research sessions are biased
- 14. Think about thelast research session you were involved in… Think of all of the ways that it was biased They were asked leading questions Room had a two way mirror Participant was distracted and nervous 5 mins
- 15. Coke, 1980s research Wecan have a massive impact on the way our participants behave
- 19. Coke, 1980s research Butit’s not just the research session we need to think about…
- 20. Project born Researchplanning Research Analysis Delivering research
- 21. Coke, 1980s research Let’sthink about how your research sessions are biased projects
- 22. Think about yourprojects roughly in these chunks Before During After Kick off Designing the research Planning the research Set up Research session Analysis Deliverables 15 mins
- 23. Add in anybias you can think of Before During After Kick off Designing the research Planning the research Set up Research session Analysis Deliverables There was already a solution Picked the wrong methodology Too many people on discussion guide Perceptions of participants screened Participant got lost Not enough time Forced to tell a certain story 15 mins
- 24. Before During After Kickoff Designing the research Planning the research Set up Research session Analysis Deliverables There was already a solution Picked the wrong methodology Screener Perceptions of participants screened Participant got lost Not enough time Forced to tell a certain story Tick box exercise Think we already know the users Recruiter bias Media that day Researchers energy Analysis from memory Who’s in the presentation Stakeholders not interested in research Guessing what areas to explore Client too focussed on specific objectives Outfit (heels on a farm) Dolly Parton research hours Remembering the ‘good’ participants Passed around out of context
- 25. Coke, 1980s research Whatcan we do about research bias?
- 26. Pick two postit notes Write down 1 or 2 ways that you could minimise that bias There’s already a designed solution List all assumptions at the beginning of the project Directly challenge assumptions in discussion guide Educate stakeholders 5 mins
- 27. Before the session •List all your assumptions early on. Have them on the wall • Use diverse recruitment methods • Be mindful of building a picture about participants from the screener • Counterbalance activities • Consider effect of time of day/year • Educate your stakeholders
- 28. During the session •Think about what the non verbal cues you’re giving off • Avoid leading questions • Don’t take notes in the room • Directly challenge your assumptions/hypotheses • Avoid emotive language
- 29. After the session •Analysis as a team sport • Try reframing your results • Pay equal weight to participants • Are the patterns you are seeing really there?
- 30. Coke, 1980s research Wedo the best we can to simulate a scenario that is as close to what users would actually do. However, no amount of realism in the tasks, data, software or environment can change the fact that the whole thing is contrived. This doesn’t mean it’s not worth doing. Jeff Sauro: ‘9 biases in usability testing’ “ “
- 31.