Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Research Operations at Scale (Christian Rohrer at DesignOps Summit 2017)

8,723 views

Published on

Christian Rohrer: "Research Operations at Scale"
DesignOps Summit 2017 • November 6-7, 2017 • Queens, NY, USA
http://designopssummit.com

Published in: Design
  • Be the first to comment

Research Operations at Scale (Christian Rohrer at DesignOps Summit 2017)

  1. 1. RE SEARCH OPERATIONS AT S CALE: ENABLING OT HERS WHILE MAINTAINI NG EXCELLENCE C H R I S T I A N P. R O H R E R , P H D @christianrohrer and crohrer@yahoo.com https://www.linkedin.com/in/crohrer/
  2. 2. S O … W H AT C O M E S T O M I N D ?
  3. 3. KNOWN AS A CREDIT CARD COMPANY
  4. 4. BUT IT’S ACTUALLY A BANK…
  5. 5. G O A L : A T E C H C O M PA N Y 
 
 ( T H AT J U S T H A P P E N S T O D O B A N K I N G ) Desirability ViabilityFeasibility
  6. 6. T O P S T R AT E G Y: 
 
 H U M A N - C E N T E R E D D E S I G N Desirability ViabilityFeasibility (AND This means RESEARCH!)
  7. 7. Desirability ViabilityFeasibility BUT WHAT
 ABOUT…
 
 
 EFFICIENCY
 
 
 RISK
 
 
 REGULATIONS
  8. 8. DESIGN RESEARCH AT A BANK IS MIRED BY ”GOO” Research Goo Risk: Strategy Leaks Risk: Privacy Laws Risk: Compliance Reg: File 1099s Reg: Consumer Protection Reg: Complaints Inefficiency: 
 Recruiting & Scheduling Inefficiency: 
 Technical/Tool 
 Complexity
  9. 9. AT C A P I TA L O N E , T H E R E A R E D E D I C AT E D R O L E S F O R R E S E A R C H O P E R AT I O N S A N D C U R R I C U L A Research Operations • Research Operations Leads: Research request intake, coordinate studies, manage recruitment vendors, scheduling participants, communicating research schedules • Technicians: manage lab and field equipment, videography Research Curricula • Instructors: Develop and teach courses on research methods & procedures to non-research employees • Specialists: Provide remote research tool licenses, access and training (e.g., survey tools, online studies, diary apps, etc.)
  10. 10. YOU CAN RUN A STUDY LIKE AN EVENT THIS RESU LTS IN 3 MAJOR BENEFITS RISK & REGULATION ARE HANDLED IN ADVANCE OR BY TRAINED PERSONNEL OTHER “GOO” LIKE RECRUITING, SCHEDULING AND TECHNICAL SETUP IS LARGELY REMOVED
  11. 11. EXAMPLE: “WINE AND DINE”
  12. 12. Should you do this?
  13. 13. CONTEXT
  14. 14. RESEARCH OPERATIONS MUST ADDRESS CHALLENGES FROM THREE KEY SOURCES Recent Trends in Research The Context of your business or industry Universal Challenges inherent in research
  15. 15. What about you? IT/INTERNAL B2BB2C Major distinctions: • Customers’ needs • Users’ choices • Customer/User Access
  16. 16. Or maybe you’re a startup.
 Who are your users today? What about tomorrow’s users? Are you getting direct access to your users? Diffusion of innovation
  17. 17. © 2015 Christian Rohrer WHAT PEOPLE SAY WHAT PEOPLE DO WHY & 
 HOW TO FIX HOW MANY & 
 HOW MUCH QUALITATIVE (DIRECT) QUANTITATIVE (INDIRECT) ATTITUDINAL BEHAVIORAL You really should be doing both qualitative and quantitative research
  18. 18. RESEARCH OPERATIONS MUST ADDRESS THE KEY SOURCES OF POTENTIAL CHALLENGES Recent Trends in Research The Context of your business or industry Universal Challenges inherent in research
  19. 19. Trends in Research
  20. 20. AGILE IS HERE (DEAL WITH IT) • Why? Agile is a must to attract engineers • So UCD must fit Agile • By default, everything goes into a sprint • Everyone’s on the same team (pod) • Code early, no docs Source: http://scrumology.com/an-excerpt-from-improving-agility-agile-certifications/
  21. 21. Source: T O P R E S E A R C H C H A L L E N G E S W I T H 
 T R A D I T I O N A L A G I L E • No UX roles defined • Little up front research • Hard to fit research into a sprint • Need to feed development • Dev centricity
  22. 22. R E C O N C I L I N G P R O C E S S E S Source: Peter Source: IDEO Source: Michael James & Luke Walter Developer & Product Owner Centric Designer & User CentricNote: all of these are “human-centered” 
 (just for very different humans)
  23. 23. WORKING WITH AGILE – 2 MAIN APPROACHES • Approach 1: Use agile methodology, but in a separate design phase • Approach 2: Use an equally attractive, but more user-centered compatible processes (e.g., Lean) • IMO: Doing strict agile, in which UX is simply added to Dev Pods does NOT work for Human-Centered Design processes. You cannot design or research anything significant in the same sprint as development.
  24. 24. 24 EXAMPLE FROM INTEL SECURITY’S TRUEKEY DESIGN PROCESS (APPLYING AGILE TO EARLIER DESIGN PHASES) 2 weeks CXD sprint2-3 days before sprint starts What is relevant here: - Uses the same agile tools and nomenclature - Design happens BEFORE Development - Validate/Optimize designs/flows before code (on every sprint) - Research is scheduled in the ”design sprints” before design executes Dev
  25. 25. L E A N A P P R O A C H : B U I L D - M E A S U R E - L E A R N BUILD MEASURELEARN Products (MVP) UsageUptake
  26. 26. L E A N A P P R O A C H : B U I L D - M E A S U R E - L E A R N BUILD MEASURELEARN Products (MVP) UsageUptake Improvements Viable/Desirable Usability Interest Prototypes Ideas & Concepts
  27. 27. M O D I F I E D L E A N A P P R O A C H : B U I L D - S T U D Y- L E A R N BUILD MEASURELEARN Products (MVP) UsageUptake Improvements Viable/Desirable Usability Interest Prototypes Ideas & Concepts STUDY
  28. 28. Source: It’s our Research by Tomer Sharon Today, researchers can’t be the only ones driving research, or it would be a huge… Another Trend: Research Participation
  29. 29. DIY RESEARCH • Over the past 5-6 years, everyone wants to do research: • All design functions • Product managers • Developers • Even executives! • Removing barriers to research now essential. • Just enough training and support are needed
  30. 30. DIY RESEARCH TIPS • Get the right vendors for you • Reach the right user tyoe • Remove goo • Get tools that work • Provide training and support • Incorporate into YOUR process
  31. 31. RESEARCH OPERATIONS MUST ADDRESS THE KEY SOURCES OF POTENTIAL CHALLENGES Recent Trends in Research The Context of your business or industry Universal Challenges inherent in research
  32. 32. Universal
 Challenges
  33. 33. The 5 Purposes of Research 1. Understand people and problems 2. Inspire creativity and innovation 3. Validate concepts and solutions 4. Improve and inform designs 5. Assess and measure experience News flash!
 These require different skill sets, not always found in one person.
  34. 34. THE TALENT POOL: THERE ARE MANY SKILLS AND TALENTS • It’s hard to find people that can “do it all” • Not everybody wants to • You should allow for BOTH specialists and hybrid players on your team • This enlarges the talent pool
  35. 35. THE TALENT POOL: SMALL, GROWS SLOWLY, VICIOUSLY FOUGHT OVER “We will always be outnumbered.” -Whitney Quissenberry, panel member at UXPA2016, “UX Past, Present & Future” • Universities that produce 
 Engineers: 1000s • Universities that produce UX Professionals: about 75 Conclusion: Don’t waste your design and research talent. Make them as efficient and strategic as you can. Source: indeed.com
  36. 36. W H AT K I N D O F R E S E A R C H W O R K S B E S T F O R D E S I G N E R S ? 
 
 W H Y ?
  37. 37. WHAT KIND OF RESEARCH AND INSIGHTS WORK BEST FOR EXECUTIVES AND PRODUCT PROFESSIONALS?
 
 WHY
  38. 38. MANY THINK SIMPLISTICALLY:
 
 “BIG NUMBERS GOOD.
 SMALL NUMBERS BAD.” “Sorry, but I’ll never make a decision based on research with just 5 people.” 
 –Product Management
  39. 39. OK, LET’S JUST ACKNOWLEDGE THAT METRICS MATTER 39 • “You can’t manage what you can’t measure” • “I need a dashboard to control my business” • “How does our NPS compare?” • “Invest in data scientists and big data” • “I want to be more scientific” • “Build, measure, learn” • “The design needs to be validated” Conversion rates, downloads, NPS, customer reviews, A/B testing performance, uptime, abandon rate, analytics data, path/click flows NOT measuring the experience. They measure the product performance.
  40. 40. A User-Centered Model of User Experience Look & FeelThe visual and industrial design is clear, professional and appropriate Sound Clear and relevant language, content and information architecture. The price of entry. In today’s world of technology, data, and design, there is no excuse for something to be hard. Ease Emotional
 Outcome It doesn’t matter how easy it is if it doesn’t address the real human need in the right way. Effectiveness Sources: • Rohrer’s Simple Model of UX (2006-2016) • Forrester: Three Es of CX (2015)
 40
  41. 41. The PURE Method Pragmatic Usability Rating by Experts Source: Link goes here
  42. 42. 42 PURE: A way to measure “friction” PURE provides a cognitive load “score” based on : 1. A well-defined user type 2. A small set of “fundamental tasks” 3. The “performance” of those tasks (e.g., the Happy Path) for that user type, based on heuristics & design principles Like in golf, smaller numbers are better, and green is good.
  43. 43. In PURE, we judge the experience based on a ”rubric,” as in skating & gymnastics • A panel of raters each silently rates an experience
 flow they are all witnessing at the same time • A 1-3 rubric defines how
 much “friction” there is, based
 on known UX heuristics • PURE rates every step
 of the task (like a “move”) • Rather than
 averaging scores,
 the panel decides on 
 the score of each step Photo credit: Ezra Shaw/Getty Images
  44. 44. A DIFFERENT KIND OF METRIC – ONE FROM A PANEL OF “EXPERTS”
 CONSIDER AN ANALOGY FROM THE REALM OF MOVIES… THE TOMATOMETER PURE is kind of like this: An score based on expert analysis
  45. 45. ACTUAL PURE SCORECARD OVER TIME (VERSION OVER VERSION) 45
  46. 46. ACTUAL COMPETITIVE PURE SCORECARD 46
  47. 47. E X E C S WA N T N U M B E R S … S O G I V E T H E M N U M B E R S ! H E R E ’ S W H AT H A P P E N S W I T H P U R E • People are intrigued. Then, either: o Owners and Designers of the product want to fix ease of use problems o Owners and Designers of the product push back, refuting PURE • Either outcome is good! We are finally focusing on improving an important aspect of user experience: Ease of Use. • It’s better to have numbers from actual user studies, but if you can’t do it easily, this is acceptable. It will cause positive change in your organization.
  48. 48. K N O W L E D G E M A N A G E M E N T I S H A R D
  49. 49. R E Q U I R E M E N T S O F A G O O D K N O W L E D G E M A N A G E M E N T S Y S T E M • Easy to find what we have and know • Easy to contribute new info and docs • Must handle more than text docs (shared cognitive artifacts like journey maps) • Tags and metadata • Include social and behavioral components (who contributed, who knows more, who used what, who likes what) • A place where knowledge can grow and be used, not just be stored and accessed
  50. 50. A J O U R N E Y M A P O F K N O W L E D G E Idea Question Research Finding Finding Counter Finding Counter Finding Better Idea Another Test Conclusion Fail Hypothesis Pivot Finding THE
 Idea Winner! Usability Study Find 
 & Fix Find 
 & Fix Find 
 & Fix Find 
 & Fix Find 
 & Fix Find 
 & Fix Build &
 Release Market data Success!
  51. 51. T H E P R O B L E M S PA C E I S L A R G E • Remove risk, deal with regulation, and address other industry-specific challenges • Teach/empower non-researchers • Find a way too fit into agile • Support qualitative research • Provide quantitative data • Keep track of and better use your insights across your design process
  52. 52. T H E P U R P O S E O F C A P I TA L O N E R E S E A R C H We believe opportunity lies in the understanding of the complexity of people. DEEP EXPERTISE INTERPRETATON EMPOWERMENTPARTNERSHIP Ultimately, we help bring insights into design.
  53. 53. @christianrohrer and crohrer@yahoo.com https://www.linkedin.com/in/crohrer/ For more on PURE, contact me or use #puremethod Thank you

×