How Agile Can Be Better Than Good Enough_FINAL.pptx
•Download as PPTX, PDF•
0 likes•47 views
A case study of what Agile Online Qualitative research is. Showing how asychronous online qual is more agile and effective than focus groups.
![2
Slide /
Agile Summit 2020
[SPONSOR
LOGO]
[PresentationTitle]
How Agile Qualitative Research Can Be Better
Than ‘Good Enough’
Decisions today happen at speed
Agile qual keeps insight central
Reduces risk; maximises success
The idea of ‘agile’ can make
research more effective](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
Recommended
Recommended
More Related Content
Similar to How Agile Can Be Better Than Good Enough_FINAL.pptx
Similar to How Agile Can Be Better Than Good Enough_FINAL.pptx (20)
More from Feeling Mutual ltd.
More from Feeling Mutual ltd. (8)
Recently uploaded
Recently uploaded (20)
How Agile Can Be Better Than Good Enough_FINAL.pptx
- 1. Agile Summit 2020 How AgileQualitative Research Can Be BetterThan ‘Good Enough’ How Agile Qualitative Research Can Be Better Than ‘Good Enough’ Tom Woodnutt, Feeling Mutual
- 2. 2 Slide / Agile Summit 2020 [SPONSOR LOGO] [PresentationTitle] How Agile Qualitative Research Can Be Better Than ‘Good Enough’ Decisions today happen at speed Agile qual keeps insight central Reduces risk; maximises success The idea of ‘agile’ can make research more effective
- 3. 3 Slide / Agile Summit 2020 [SPONSOR LOGO] [PresentationTitle] How Agile Qualitative Research Can Be Better Than ‘Good Enough’ Where is the line? What do we borrow? What do we reject? What should ‘agile’ mean?
- 4. 4 Slide / Agile Summit 2020 [SPONSOR LOGO] [PresentationTitle] How Agile Qualitative Research Can Be Better Than ‘Good Enough’ ‘Agile’ has taken over the ‘Waterfall’ Built on on-going feedback Soft-launching ‘good enough’ designs The ‘agile’ philosophy is fundamentally ‘insight’ led
- 5. 5 Slide / Agile Summit 2020 [SPONSOR LOGO] [PresentationTitle] How Agile Qualitative Research Can Be Better Than ‘Good Enough’ Which ‘agile’ principles apply to research ? a) ‘Test and learn’ continuously b) Collaborating closely c) ‘Good enough’ is good enough
- 6. 6 Slide / Agile Summit 2020 [SPONSOR LOGO] [PresentationTitle] How Agile Qualitative Research Can Be Better Than ‘Good Enough’ a) ‘Good enough’ is good enough b) ‘Test and learn’ continuously c) Collaborating closely Too often, ‘Agile research’ just means ‘quick and dirty’ BUT Risk of relaxing standards too far BUT Researchers need independence BUT Too much testing can kill ideas
- 7. 10 Slide / Agile Summit 2020 [SPONSOR LOGO] [PresentationTitle] How Agile Qualitative Research Can Be Better Than ‘Good Enough’ Three weeks turnaround Little prior knowledge Challenging recruitment 2 markets The brief: Compress discovery and testing into one phase Top 10 World’s Most Innovative Company” (2019)
- 8. 11 Slide / Agile Summit 2020 [SPONSOR LOGO] [PresentationTitle] How Agile Qualitative Research Can Be Better Than ‘Good Enough’ How best to use the 3 weeks? a) Balancing testing and learning b) Agency collaboration at speed c) Maximising validity
- 9. 12 Slide / Agile Summit 2020 [SPONSOR LOGO] [PresentationTitle] How Agile Qualitative Research Can Be Better Than ‘Good Enough’ Typical agile qual prioritises ‘testing’ over ‘learning’ WEEK 1 WEEK 2 WEEK 3 RECRUITMENT CONCEPT TEST 1 UK ITERATE CONCEPTS TEST 2 US ITERATE Squeezed recruitment to free more time to ‘test & learn’ Breaks between two dips of real-time qual discussion, across two weeks (online or face to face) BUT, what if a concept change after phase 1, loses something important? Or something unimportant is overplayed based on just 2 groups?
- 10. 13 Slide / Agile Summit 2020 [SPONSOR LOGO] [PresentationTitle] How Agile Qualitative Research Can Be Better Than ‘Good Enough’ ‘Incubated Iteration’ using asynchronous online qual WEEK 1 WEEK 2 WEEK 3 RECRUITMENT I T E R A T E DISCOVERY US & UK CONCEPT TEST US & UK ANALYSIS SUMMARY REPORT Ensure quality recruitment Daily moderation Daily insight summaries ‘Incubated iteration’ Revealing concepts on penultimate day
- 11. 14 Slide / Agile Summit 2020 [SPONSOR LOGO] [PresentationTitle] How Agile Qualitative Research Can Be Better Than ‘Good Enough’ a) Investing in discovery to test better concepts Day 1-3 for discovery Exploring cultural context Getting into their real-worlds Gathering multi-media feedback Using tools to elicit emotion Day 4-5 for concept testing Day 6 for final questions
- 12. 15 Slide / Agile Summit 2020 [SPONSOR LOGO] [PresentationTitle] How Agile Qualitative Research Can Be Better Than ‘Good Enough’ Asynchronous online qual gets at least 12 times more feedback WEEK 1 WEEK 2 WEEK 3 RECRUITMENT F I E L D DISCOVERY CONCEPT TEST ANALYSIS SUMMARY ITERATION WEEK 1 WEEK 2 WEEK 3 RECRUITMENT CONCEPT TEST 1 FIRST ITERATION CONCEPTS TEST 2 SECOND ITERATION 2 phased iteration real-time qual ‘Incubated iteration’ Asynchronous qual In real-time group discussions, only one person can feedback at a time So after 4 x 2hr groups (of 8) Total time spent just on discovery = just 4hrs Total time spent on testing = just 4hrs In Asynchronous Online Qual, everyone gives feedback in parallel, at the same time So after 6 days with 32 people Total time spent on discovery = 48 hrs Total time spent on concept = 48hrs X2 Realtime Groups X2 Realtime Groups
- 13. 16 Slide / Agile Summit 2020 [SPONSOR LOGO] [PresentationTitle] How Agile Qualitative Research Can Be Better Than ‘Good Enough’ b) Agency collaboration at speed via google drive “Incubated Iteration” Close collaboration Shared, daily summary Ongoing updates to concepts Compatible with planning model
- 14. 17 Slide / Agile Summit 2020 [SPONSOR LOGO] [PresentationTitle] How Agile Qualitative Research Can Be Better Than ‘Good Enough’ c) Maximising validity through asynchronous online qual Pure private response Priming (with reality) via discovery No over-exposure to concepts
- 15. 18 Slide / Agile Summit 2020 [SPONSOR LOGO] [PresentationTitle] How Agile Qualitative Research Can Be Better Than ‘Good Enough’ ‘incubated optimisation’ 12 x more discovery with asynchronous online qual (ALBEIT with one round of concept feedback) = Stronger inputs to the concepts = More vivid outputs from mobile video = More valid feedback with less group effects = Tighter collaboration with agency = Better concepts at the end
- 16. 19 Slide / Agile Summit 2020 [SPONSOR LOGO] [PresentationTitle] How Agile Qualitative Research Can Be Better Than ‘Good Enough’ Diverse apps (from education to productivity, media and creativity) What is differentiating benefit? What is the core need? Competing with Apple iTunes brief: agile brand proposition development
- 17. 20 Slide / Agile Summit 2020 [SPONSOR LOGO] [PresentationTitle] How Agile Qualitative Research Can Be Better Than ‘Good Enough’ Droga5’s Planning Framework Brand purpose COMMS PLATFORM 1 - Culture Emergent cultural trends 4. Company brand credibility to deliver 5. Context Role of channels and moments 2 - Consumer Opportunity to influence consumer 3 - Category Competitive point of difference
- 18. 21 Slide / Agile Summit 2020 [SPONSOR LOGO] [PresentationTitle] How Agile Qualitative Research Can Be Better Than ‘Good Enough’ We structured the study around Droga5’s planning framework DAY 1 CULTURE / CONSUMER Day 2 - CATEGORY / CONTEXT Day 3 - COMPANY Day 4 - CONCEPTS Day 5 - AREAS OF INTEREST Apple Mac Relationship Use of Mac Apps Mac App store experience Customer journey Reactions to Setapp Customer journey Reactions to concepts And their ideas Areas of interest Video capture Days 4-5 - Two days for their reactions to concepts and our probes Day 5 – Covered areas of interest after the concepts, to avoid priming them. Days1-3 - Started with discovery into: Culture, Consumer, Context and Company
- 19. 22 Slide / Agile Summit 2020 [SPONSOR LOGO] [PresentationTitle] How Agile Qualitative Research Can Be Better Than ‘Good Enough’ Question: “Share any quotations you have around you to inspire you…” They work so hard to keep on doing the things they love Day 1 - Culture
- 20. 23 Slide / Agile Summit 2020 [SPONSOR LOGO] [PresentationTitle] How Agile Qualitative Research Can Be Better Than ‘Good Enough’ Question: “Record your deskspace and computer desktop. What does it tell us about you?” It’s a battle to stay focused Day 1 - Consumer “I live by the principle ‘Clear Desk - Clear Mind’ as you can see. It’s hard to work with too many distractions”
- 21. 24 Slide / Agile Summit 2020 [SPONSOR LOGO] [PresentationTitle] How Agile Qualitative Research Can Be Better Than ‘Good Enough’ Question: “Draw a picture that represents how you feel about your Mac apps?” They crave enhanced productivity Day 1 - Consumer “My trusty dog Mac helps me do more. (But sometimes he starts playing with his spinning rainbow ball unannounced)"
- 22. 25 Slide / Agile Summit 2020 [SPONSOR LOGO] [PresentationTitle] How Agile Qualitative Research Can Be Better Than ‘Good Enough’ Question: “Go to the Mac App store / Setapp and record a video, commentating on how you find it?” Setapp curates order from chaos (when you need it most) Day 2 – Category, Day 3 – Company / Context VS
- 23. 26 Slide / Agile Summit 2020 [SPONSOR LOGO] [PresentationTitle] How Agile Qualitative Research Can Be Better Than ‘Good Enough’ Millennials need help to stay productive, so they can do the work that they love Setapp simplifies a bewildering choice in Mac Apps
- 24. 27 Slide / Agile Summit 2020 [SPONSOR LOGO] [PresentationTitle] How Agile Qualitative Research Can Be Better Than ‘Good Enough’ Question: “How do you feel about these ideas that the brand could stand for?” The emotional experience of enhanced productivity resonates Day 4/5 – Concepts CONCEPT 1: ‘For the disrupters’ CONCEPT 2: ‘When you’re on fire’ CONCEPT 3: ‘Have more head- space’
- 25. 28 Slide / Agile Summit 2020 [SPONSOR LOGO] [PresentationTitle] How Agile Qualitative Research Can Be Better Than ‘Good Enough’ ‘Find your flow’
- 26. 29 Slide / Agile Summit 2020 [SPONSOR LOGO] [PresentationTitle] How Agile Qualitative Research Can Be Better Than ‘Good Enough’ More ‘discovery’; less testing Asynchronous online & mobile for: - more pure concept feedback (private responses) - more in-depth exploration (12 x more discovery) - more in-situ insights (mobile ‘in the moment’) - more vivid reporting (video and image feedback) Better concepts ‘Incubated Iteration’ using asynchronous online and mobile
Editor's Notes
- Hello everyone. My name’s Tom Woodnutt. And today I’m going to be talking about how Agile Qual can be better than ‘good enough’. I’m going to share my approach to an agile brand proposition development brief that I received from one of the world’s most innovative brands (according to Fast Company). I’ll share my interpretation of how ‘agile’ should (and should not) be applied to concept development.
- As we know, the concept of ‘agile’ originally came from software development. Some of the tenets that make up Agile software development can make qualitative research more useful today. (CLICK) Client and agency workflows and decision making are more time-pressured than ever. CLICK Qual Research must shape its delivery to stay in time with more dynamic decision making CLICK It is vital to keep people’s needs at the heart of decision making, in order to reduce risk and encourage ideas that truly address genuine needs.
- We need to be clear on which aspects of ‘agile software development’ we embrace, and which ones we ignore. In defining what agile qual methods looks like, we need to be careful to preserve the integrity and validity of what we do. But we also have to push ourselves to deliver insights that can be acted on more quickly and effectively.
- Research and agile software design principles should be compatible. The ethos of agile software development is fundamentally insight led. It recognises that people’s needs and behaviour must lead the design process. Before sharing my interpretation of ‘agile’ – I want to return to its conceptual inception. CLICK Software development used to take a linear ‘waterfall’ approach’ to building, which involved coding the entire system as well as possible, before testing and launching it (CLICK). But engineers started to question whether you could ever anticipate all requirements from the start. It’s impossible to accurately predict what people need and what they will do until you see how they interact with the system. CLICK So the agile approach was proposed, which involves launching different chunks of the overall design early, to a ‘good enough’ functional standard, so they can undergo a series of tests and learning, which get applied continuously.
- Which agile principles should be applied to qual research? To use the terminology of Wendy Gordon, ‘Agile’ is a fat word, that is to say, it carries a number of competing meanings. In fact, when it It was put forward by a group of software engineers back in the (2001), 12 principles were outlined in their manifesto. I believe Qualitative researchers must be very clear on which principles to apply, how and when. CLICK Three core principles are usually referenced when researchers talk about agile methods – testing and learning more than once in a project / collaborating and communicating closely across teams / relaxing traditional standards in places to make the process ‘good enough’ rather than ‘perfect’ It’s obvious how these can be useful. But I think too little attention is paid to how they can be destructive.
- Too often, ‘Agile research’ just means ‘quick and dirty’. CLICK The idea of ‘good enough being good enough’ is sometimes used a justification for relaxing standards. CLICK But bad research is worse than no research. I get worried when I read about companies relaxing standards of recruitment and sampling, offering poor incentives and minimal (or even no) reporting- all in the name of ‘agile’ research. We should be flexible but not on standards of quality. CLICK Similarly, just because you have the opportunity to run concepts past people more than once and continually ‘test and learn’ , does not mean you should. Too often the emphasis is on doing as much testing as possible for the sake of it rather than trying to learn as much as possible. CLICK CLICK Without clearly defined roles, close collaboration can mean research agencies lose autonomy and blindly follow the conclusions of the design or ad agency team.
- I’ve seen projects described as ‘agile’ that assume that it’s OK to reduce standards, in order to work with less budgets or shorter timelines. CLICK that can be in terms of recruitment, paying less incentives and charging less consultancy. This means there is less probing and lower participation. CLICK Sometimes the reporting is left to the agency and the researcher is just an enabler – which can create a problem for the end-client unless they completely trust the agency to report without any unconscious bias. CLICK Also too often there is a ‘rough and ready’ mentality that results in a first draft of concepts which are poorly thought through. It’s almost as if it doesn’t matter, because they’ll be tested more than once, so they’ll end up OK. But I think this isn’t the best way to interpret agile.
- You often hear agile workflows talk about: ‘test and learn’. But too often the emphasis is on the ‘testing’ rather than the learning’. CLICK In software dev, this makes sense: You need to build a ‘good enough’ design and let people interact with it, so you can test whether your assumptions were right and iron out bugs. with software, the TESTING is the learning In proposition dev you need to isolate the right need or insight, understand more of the cultural and emotional context You need the right learning before you have a concept worth testing. CLICK Unless we have the luxury of more time and budget AND Unless the client has already done lots of discovery work, I think one round of concept feedback after a period of Discovery, can be better than trying to squeeze in two and lose the opportunity for the discovery work.
- Thirdly, I am a huge fan of collaboration. CLICK I think that researchers’ conclusions will always be better when they have in depth, on going conversations with the agency (be that innovation, design, or advertising) as well as the client; not just before but also during fieldwork. CLICK However, if the researcher is just a facilitator – in other words, just doing moderation and perhaps basic summaries - and the final conclusions are led by the agency - I think channelling this agenda is useful if it inspires and improves the researcher’s analysis but not necessarily if it defines the conclusions. So there needs to be a balance between the agencies influence over the results and the researcher’s freedom to make their own conclusion.
- These were the kind of considerations that came to mind when I received this brief from Setapp. Setapp, from one of the world’s most innovative companies (according to Fast Company) theylike the Netflix for mac apps subscription model – they are a software company – and I wanted to borrow from their culture of innovation, by making it as agile as possible, without compromising the integrity of the research. They wanted to re-fresh the brand strategy in order to meet aggressive growth targets. CLICK But they only had three weeks from sign off of the project to sign off of the final brand proposition. CLICK And they didn’t know much about their users before. So they wanted us to combine discovery and concept testing – which might ordinarily be in two phases, into one. CLICK we also had to speak to entrepreneurs, IT and design professionals. CLICK Moreover we had to do the study in 2 markets US and UK
- In the proposal I had to work out the best way to deliver an agile project. And how to slice up the 3 weeks. CLICK I had to balance ‘testing the concepts vs learning about the audience in general’ so they could be written in the first place CLICK I had to work closely with the agency so they could take any learnings quickly and easily apply them to the concepts CLICK And I had to ensure the project was completely valid and following best practice qual standards. I’m now going to go through these three principles and show how I applied them in this case study.
- Typical agile qual focuses more on ‘testing’ than ‘learning’. CLICK The more I read about Agile Qual, the less I wanted to follow the conventions that were promoted in the public domain. If I had followed what others do, it would have looked something like this. CLICK A basic standard of recruitment for the first week, using less demanding sample criteria to keep it quick free up enough time for two phases of testing. CLICK Usually using real-time ‘live’ methods (e.g. face to face or webcam group discussions). With one market following the other, and a period of concept iteration between them. CLICK – So in this model, the concepts have to be ready by the first dip of fieldwork after the first week of recruitment. And then the concepts are changed based on the first market. CLICK but the proble with changing the concepts early on, after just two groups, is that something important might be lost after groups 1 and 2? Or something unimportant could be overplayed. In this model, the concepts are being changed based on a limited amount of input. Since the client didn’t know much about their audience, it felt like we would be over-prioritising concept development at the expense of more general discovery.
- So instead we ran the asynchronous discussion over 6 days in each market, at the same time. Rather than doing real-time discussion that were live and one after the other . CLICK We allowed for an extra half a week to get recruitment right. CLICK And we did ongoing moderation and analysis each day and shared the highlights, we updated the concepts collaboratively and only introduced them on the last couple of days. I’m calling this CLICK incubated iteration. Because the iteration took place in an incubated fashion – with just one phase of testing in the last couple of days.
- So in the context of this brief – we prioritised discovery over testing. That’s because we wanted to get in depth learning before committing to the concepts. This upfront time for discovery allowed us to look at the cultural context, to get into their real worlds via mobile feedback, to gather selfie videos and photos from their lives and using tools to explore their emotions. CLICK So by the time it was day 4 and 5, when we tested the concepts, we had already knew all about them and were more confident that we were in the right place. Even though this approach meant sacrificing having two phases of testing and iterating the concepts inbetween them, we believe that the final concepts benefited more by having that extra discovery time. So my point is that just because it was an agile brief – and just because ‘agile’ is so often a proxy for multiple waves to ‘testing and learning’- researchers still need to look at the context of the brief and decide on which agile principles to use and which to reject.
- I’m going to show how prioritising ‘testing’ concepts, using a typical ‘realtime qual approach that you see here on the left will mean you learn significantly less than you would in the asynchronous online qual approach. So this is what might be a typical agile qual approach. You spend the first week recruiting. Then CLICK you do 2 x 2hr groups in the first market. That could be a face to face group or a real time webcam group. In both scenarios let’s say you spend the first hour on discovery – so exploring things like their attitudes, feelings and behaviour. Then you spend the second hour getting feedback on the concepts. So you can see how after 4 x 2hr groups you’ve got a total of just 4 hours worth of discovery into their lives. That’s because in realtime methods only one person can speak at once. So although you pay them an incentive for 2hrs of their time. You only get 15mins (if that once you deduct the time the moderator speaks!). Now I’m not denying 4hours worth of group feedback on concepts is useful and 4hrs worth of discovery are useful. But there’s a real risk that the concepts being tested are not going to be as good as they could be. Especially in the context of an agile workflow. The fact is you sometimes get the sense that concepts are launched which are just ‘good enough’ because you’ll have at least 2 rounds of feedback and they’ll be great by the end of it. But I would argue that those concepts will end up way better if you use asynchronous online qual with just one exposure to one set of concepts CLICK – with the concepts being iterated behind the scenes over the first three days of discovery. In this ‘incubated iteration’ approach, on the right here, you get 12 times more Discovery feedback from people than in the realtime, ‘2 phased’ iteration. In this asynchronous example, the first 3 days are discovery, and all 32 people are giving you a full 30mins feedback a day in parallel – so it’s more like 32 depths. They can all feedback at once. So you get a full 90mins feedback on discovery alone – per person. So that’s 90mins input into discovery, per person which comes to 48hrs worth of discovery in total, compared to just 4hrs of discovery in total in the real-time, 2 phased iteration model. CLICK SO by the time you test the concepts for the first time, you have got 12 times more discovery learnings about their lives, the culture and context – fully integrated into the concepts. I would argue that often that extra learning is worth more than the extra testing.
- You need the right learning before testing, No point testing crap concepts Concepts are crap without the right insight
- You need the right learning before testing, No point testing crap concepts Concepts are crap without the right insight
- You need the right learning before testing, No point testing crap concepts Concepts are crap without the right insight
- It was so important that our findings could be quickly fed into the Droga5’s planning framework. So we broke the discovery phase up around the model We Each day covered one or two of the Cs in their model. CLICK So day 1-3 covered the culture (exploring what inspires them), The consumer (their relationship with Mac Apps) the category (and how they see the competition) and Setapp the company themselves (and how they react to / use the service). On day 4-5 we shared three concepts which had been developed over the previous three days. And finally on the last day we probed further on the concepts and also picked off some areas of interest that the agency instinctively thought might be relevant. We didn’t want to explore them before the concepts in case it influenced their subsequent reactions.
- work so they can do what they love. We put this cultural desire for achievement and productivity at the heart of the brand strategy
- Day 2, consumer – On day 2 we looked at their relationship with their Apple Macs We asked them to take a video of the typical place in which they use their Mac. And to share photos of their physical and digital desktops. We noticed that many of them kept immaculate workspaces and virtual desktops, which they carefully controlled in order to minimise distraction and feel in control This reinforced our belief that the core need that Setapp should focus on was to stay productive
- We also got them to draw how they feel about using Macs Apps As this drawing shows, it reveals a loyal partnership, focused on getting things done This further validated our focus on the productivity benefit from Setapp
- On Day 3, To explore the ‘Category’, We sent them to the Mac App Store which revealed a weakness in how hard it is to find what you need Day 4 ‘Company’, they went to the Setapp site and raved about the idea of a more curated, personalised experience This highlighted the importance of Setapp’s curation (helping solve problems just when you need it most)
- Although Setapp curate diverse types of mac apps (from creativity to personal finance and productivity) – It was clear that the productivity apps were the most culturally and emotionally valued ones. The weakness of the mac app store on itunes – was obviously in the complexity of choice. Which played in neatly to Setapps focus on curating the right apps for people. By Day 4 we knew had done enough discovery, to have faith in placing the focus of the concepts on Productivity. On Day 1 that was not clear. We could have had an extra round of feedback. But what was the point, if that first set of concepts were not anchored in an insight and need that resonated?
- 15 Across those 5 days we updated the 3 concepts which we then shared on the last two days. Each idea too a different perspective on the idea of enhanced productivity We allowed two days for this so they could react to our probes
- 15 Finally this led to the big strategic idea which is to ‘find your flow’ with setapp And the advertising is in development now
- 15 Finally this led to the big strategic idea which is to ‘find your flow’ with setapp And the advertising is in development now