Explore and Evaluate - A different intro to research and data
•
4 likes•3,006 views
Based on the Medium article: https://medium.com/@ben.dressler/two-keys-to-data-and-product-research-cfc4e0d75ae4 A beginners guide to data and research that is different. Instead of using any jargon like "qual" and "quant" this presentation gives a basic view of how to think about data. Ben Dressler is a user researcher at Spotify in Stockholm, Sweden.
Recommended
More Related Content
Similar to Explore and Evaluate - A different intro to research and data
Similar to Explore and Evaluate - A different intro to research and data (20)
Recently uploaded
Recently uploaded (20)
Explore and Evaluate - A different intro to research and data
- 1. Explore and Evaluate Aguide to research and data forthe uninitiated Picture: Niels Heidenreich
- 2. What do you know about data?
- 3. Qual Quant
- 4. Qual Quant Subjective Objective Longitudinal InterdependentFidelity Significant p-valueconfidence interval n alpha normally distribute correlated ordinal behavioural attitudinal self-report
- 5. All clear?
- 6. Think of research as a jigsaw puzzle. Picture: Curt Smith
- 7. First you explore the lay of the land. What are the main colours? Where are the corners and edges? What are good distinct starting points?
- 8. Then you start evaluating individual pieces Is this the same colour as that?Are those part ofthe same object? Does that piece fit where I think it fits?
- 9. Explore / Evaluate The one distinction to get right! Picture: Pablo Ruiz Múzquiz
- 10. Exploration research is like a fishing net Picture: Christian Haugen
- 11. Explore You can go fishing with simple tools or complex machinery You won’t know exactly what you will get in advance Your catch will give you ideas about what is under the surface Throwing the net out repeatedlywill have diminishing returns Missing a fish will have no serious consequences
- 12. Checklist What is it you are exploring? Describe the population and the part oftheir life that you’re interested in Are you exploring all sources available to you? From talking to people on the street to digging through usage logs Is your scope wide enough? Do not restrict the flow of incoming data. Let yourself be surprised.
- 13. Evaluative research is like a sniper rifle Picture: Ian Norman
- 14. Evaluate A sniper rifle is a sophisticated and specific device You will either hit or miss - there is little ambiguity. You need a good idea where and what you’re aiming at Your required accuracy is a function of the target size Zooming in on the target reduces your breadth ofvision Missing a shot can have serious consequences
- 15. Checklist What are your hypotheses? Be specific - you want to be able to evaluate with a clear yes or no. What accuracy do you need? How small is the effect you want to evaluate? Can you get enough sample for that? When is it time to zoom out? Be aware that yourview is very limited. Go back to exploration frequently.
- 17. Explore Kick off Time frame depends on Resources depend on Deliverables Evaluate As early as possible Once you have formed hypotheses Size of area to explore, existing knowledge, method used Whether hypothesis is time based (e.g. retention) Accessibility of the population and what methods you need to employ Whether a framework for evaluating hypotheses exists (e.g. AB testing) Condensation of the signals into themes, potentially hypotheses Yes/no evaluation of the hypothesis in question
- 18. Explore User Testing Surveys A/B Tests Evaluate • Small sample (3-15) • Focus on describing the entirety of observations • Recommend to fix/evaluate issues • Add caveats around reported opinions • Match sample with desired precision • Standardise task description and order • Count success/fail • Report completion rates with confidence intervals • Small-mid sample (<50 per segment) • Focus on open questions • Keep closed questions general • Report relative frequencies • Report confidence intervals • Large sample (150+ per segment) • Focus on closed questions • Mix general and specific questions • Report relevant means with added confidence intervals • Very large sample • Compare high level metrics • Drill into funnels to discover differences • Look at click maps • Report all significant differences • Very large sample • Focus on metrics as formulated in hypotheses • Report significant differences on hypothesis metrics
- 20. 1 Continuous delivery Explore Evaluate Hypothesis A/B test Short, concurrent exploration sprints. Exploration through usertesting andA/B test result analysis to generate hypotheses. Regular A/B testing (every 2-3 weeks) to validate hypotheses and produce more data to explore.
- 21. 2 Big bang innovation Explore Evaluate Long, initial exploration phase. Moving through prototype stage while evaluating and refining functionality and experience. Build & release as a final evaluation. Gathering data Build and test Form hypotheses
- 22. Initial exploration of available data, identify themes to test. High level evaluation of hypothesis themes. Break down themes into test waves. Iterate through themes in parallel. Next waves informed by test result and concurrent exploration. 3 Iterative redesign Explore Evaluate Explore Evaluate Direction Start evaluating themes Define themes to take forward Iterate on MVPtests Release and iterate
- 23. What do you know about data?
- 24. What do you know about data?Explore Evaluate
- 26. Data visualisation: Niels Heidenreich https://www.flickr.com/photos/schoschie/ LHC: Pablo Ruiz Múzquiz https://www.flickr.com/photos/angelaypablo/ Fishing net: Christian Haugen https://www.flickr.com/photos/christianhaugen/3290641932/ Rifle: Ian Norman https://www.flickr.com/photos/inorman/ Roundabout:Tauno Tõhk https://www.flickr.com/photos/toehk/ Puzzle: Curt Smith https://www.flickr.com/photos/curtsm/5295530152/ Picture credits