Christian Rohrer: "Research Operations at Scale" DesignOps Summit 2017 • November 6-7, 2017 • Queens, NY, USA http://designopssummit.com

1. RE SEARCH OPERATIONS AT S CALE: ENABLING
OT HERS WHILE MAINTAINI NG EXCELLENCE
C H R I S T I A N P. R O H R E R , P H D
@christianrohrer and crohrer@yahoo.com
https://www.linkedin.com/in/crohrer/

2. S O …
W H AT C O M E S T O M I N D ?

3. KNOWN AS A
CREDIT CARD
COMPANY

4. BUT IT’S ACTUALLY
A BANK…

5. G O A L : A T E C H
C O M PA N Y
( T H AT J U S T H A P P E N S
T O D O B A N K I N G )
Desirability
ViabilityFeasibility

6. T O P S T R AT E G Y:
H U M A N - C E N T E R E D
D E S I G N
Desirability
ViabilityFeasibility
(AND This means RESEARCH!)

7. Desirability
ViabilityFeasibility
BUT WHAT
ABOUT…
EFFICIENCY
RISK
REGULATIONS

8. DESIGN RESEARCH AT A BANK IS MIRED BY ”GOO”
Research Goo
Risk: Strategy Leaks
Risk: Privacy Laws
Risk: Compliance
Reg: File 1099s
Reg: Consumer Protection
Reg: Complaints
Inefficiency:
Recruiting & Scheduling
Inefficiency:
Technical/Tool
Complexity

9. AT C A P I TA L O N E , T H E R E A R E D E D I C AT E D R O L E S
F O R R E S E A R C H O P E R AT I O N S A N D C U R R I C U L A
Research Operations
• Research Operations Leads: Research request intake, coordinate studies, manage recruitment
vendors, scheduling participants, communicating research schedules
• Technicians: manage lab and field equipment, videography
Research Curricula
• Instructors: Develop and teach courses on research methods & procedures to non-research employees
• Specialists: Provide remote research tool licenses, access and training (e.g., survey tools, online
studies, diary apps, etc.)

10. YOU CAN RUN A
STUDY LIKE AN
EVENT
THIS RESU LTS IN 3 MAJOR BENEFITS
RISK & REGULATION
ARE HANDLED IN
ADVANCE OR BY
TRAINED PERSONNEL
OTHER “GOO” LIKE
RECRUITING, SCHEDULING
AND TECHNICAL SETUP IS
LARGELY REMOVED

11. EXAMPLE: “WINE AND DINE”

12. Should you do this?

13. CONTEXT

14. RESEARCH OPERATIONS MUST ADDRESS
CHALLENGES FROM THREE KEY SOURCES
Recent Trends in Research
The Context of your business or industry
Universal Challenges inherent in research

15. What
about
you? IT/INTERNAL
B2BB2C
Major distinctions:
• Customers’ needs
• Users’ choices
• Customer/User Access

16. Or maybe you’re a startup.
Who are your users today?
What about tomorrow’s users?
Are you getting direct access to your users?
Diffusion of innovation

17. © 2015 Christian Rohrer
WHAT PEOPLE SAY
WHAT PEOPLE DO
WHY &
HOW TO FIX
HOW MANY &
HOW MUCH
QUALITATIVE (DIRECT) QUANTITATIVE (INDIRECT)
ATTITUDINAL
BEHAVIORAL
You really should be doing both qualitative and
quantitative research

18. RESEARCH OPERATIONS MUST ADDRESS THE KEY
SOURCES OF POTENTIAL CHALLENGES
Recent Trends in Research
The Context of your business or industry
Universal Challenges inherent in research

19. Trends in Research

20. AGILE IS HERE (DEAL WITH IT)
• Why? Agile is a must
to attract engineers
• So UCD must fit Agile
• By default, everything
goes into a sprint
• Everyone’s on the
same team (pod)
• Code early, no docs
Source: http://scrumology.com/an-excerpt-from-improving-agility-agile-certifications/

21. Source:
T O P R E S E A R C H C H A L L E N G E S W I T H
T R A D I T I O N A L A G I L E
• No UX roles
defined
• Little up front
research
• Hard to fit research
into a sprint
• Need to feed
development
• Dev centricity

22. R E C O N C I L I N G P R O C E S S E S
Source: Peter
Source: IDEO
Source: Michael James & Luke Walter
Developer & Product Owner Centric
Designer & User CentricNote: all of these are “human-centered”
(just for very different humans)

23. WORKING WITH AGILE – 2 MAIN APPROACHES
• Approach 1: Use agile methodology, but in a separate design phase
• Approach 2: Use an equally attractive, but more user-centered
compatible processes (e.g., Lean)
• IMO: Doing strict agile, in which UX is simply added to Dev Pods does
NOT work for Human-Centered Design processes. You cannot design
or research anything significant in the same sprint as development.

24. 24
EXAMPLE FROM INTEL SECURITY’S TRUEKEY DESIGN
PROCESS (APPLYING AGILE TO EARLIER DESIGN PHASES)
2 weeks CXD sprint2-3 days before sprint starts
What is relevant here:
- Uses the same agile tools and nomenclature
- Design happens BEFORE Development
- Validate/Optimize designs/flows before code (on every sprint)
- Research is scheduled in the ”design sprints” before design executes
Dev

25. L E A N A P P R O A C H : B U I L D - M E A S U R E - L E A R N
BUILD
MEASURELEARN
Products (MVP)
UsageUptake

26. L E A N A P P R O A C H : B U I L D - M E A S U R E - L E A R N
BUILD
MEASURELEARN
Products (MVP)
UsageUptake
Improvements
Viable/Desirable
Usability
Interest
Prototypes
Ideas & Concepts

27. M O D I F I E D L E A N A P P R O A C H : B U I L D - S T U D Y- L E A R N
BUILD
MEASURELEARN
Products (MVP)
UsageUptake
Improvements
Viable/Desirable
Usability
Interest
Prototypes
Ideas & Concepts
STUDY

28. Source: It’s our Research by Tomer Sharon
Today, researchers can’t be the only ones
driving research, or it would be a huge…
Another Trend: Research Participation

29. DIY RESEARCH
• Over the past 5-6 years, everyone wants to
do research:
• All design functions
• Product managers
• Developers
• Even executives!
• Removing barriers to research now
essential.
• Just enough training and support are
needed

30. DIY RESEARCH TIPS
• Get the right vendors for you
• Reach the right user tyoe
• Remove goo
• Get tools that work
• Provide training and support
• Incorporate into YOUR process

31. RESEARCH OPERATIONS MUST ADDRESS THE KEY
SOURCES OF POTENTIAL CHALLENGES
Recent Trends in Research
The Context of your business or industry
Universal Challenges inherent in research

32. Universal
Challenges

33. The 5 Purposes of Research
1. Understand people and problems
2. Inspire creativity and innovation
3. Validate concepts and solutions
4. Improve and inform designs
5. Assess and measure experience
News flash!
These require
different skill
sets, not
always found
in one person.

34. THE TALENT POOL: THERE ARE MANY SKILLS AND
TALENTS
• It’s hard to find people that can
“do it all”
• Not everybody wants to
• You should allow for BOTH
specialists and hybrid players
on your team
• This enlarges the talent pool

35. THE TALENT POOL: SMALL, GROWS SLOWLY,
VICIOUSLY FOUGHT OVER
“We will always be outnumbered.”
-Whitney Quissenberry, panel member at
UXPA2016, “UX Past, Present & Future”
• Universities that produce
Engineers: 1000s
• Universities that produce UX
Professionals: about 75
Conclusion: Don’t waste your design and
research talent. Make them as efficient and
strategic as you can. Source: indeed.com

36. W H AT K I N D O F
R E S E A R C H W O R K S
B E S T F O R D E S I G N E R S ?
W H Y ?

37. WHAT KIND OF RESEARCH
AND INSIGHTS WORK BEST
FOR EXECUTIVES AND
PRODUCT PROFESSIONALS?
WHY

38. MANY THINK
SIMPLISTICALLY:
“BIG NUMBERS
GOOD.
SMALL NUMBERS
BAD.”
“Sorry, but I’ll never make a
decision based on research
with just 5 people.”
–Product Management

39. OK, LET’S JUST ACKNOWLEDGE THAT METRICS MATTER
39
• “You can’t manage what you can’t
measure”
• “I need a dashboard to control my
business”
• “How does our NPS compare?”
• “Invest in data scientists and big data”
• “I want to be more scientific”
• “Build, measure, learn”
• “The design needs to be validated”
Conversion rates, downloads, NPS, customer
reviews, A/B testing performance, uptime,
abandon rate, analytics data, path/click flows
NOT measuring the experience. They
measure the product performance.

40. A User-Centered Model of User Experience
Look & FeelThe visual and industrial design is clear,
professional and appropriate
Sound
Clear and relevant language, content and
information architecture.
The price of entry. In today’s
world of technology, data, and
design, there is no excuse for
something to be hard.
Ease
Emotional
Outcome
It doesn’t matter how easy it is if
it doesn’t address the real human
need in the right way.
Effectiveness
Sources:
• Rohrer’s Simple Model of UX (2006-2016)
• Forrester: Three Es of CX (2015)
40

41. The PURE Method
Pragmatic Usability Rating by Experts
Source: Link goes here

42. 42
PURE: A way to measure “friction”
PURE provides a cognitive load
“score” based on :
1. A well-defined user type
2. A small set of “fundamental tasks”
3. The “performance” of those tasks
(e.g., the Happy Path) for that user
type, based on heuristics & design
principles
Like in golf, smaller numbers are
better, and green is good.

43. In PURE, we judge the experience based on a ”rubric,” as in skating & gymnastics
• A panel of raters each silently rates an experience
flow they are all witnessing at the same time
• A 1-3 rubric defines how
much “friction” there is, based
on known UX heuristics
• PURE rates every step
of the task (like a “move”)
• Rather than
averaging scores,
the panel decides on
the score of each step
Photo credit: Ezra Shaw/Getty Images

44. A DIFFERENT KIND OF METRIC – ONE FROM A PANEL OF “EXPERTS”
CONSIDER AN ANALOGY FROM THE REALM OF MOVIES… THE TOMATOMETER
PURE is kind of like this:
An score based on expert analysis

45. ACTUAL PURE SCORECARD OVER TIME (VERSION OVER VERSION)
45

46. ACTUAL COMPETITIVE PURE SCORECARD
46

47. E X E C S WA N T N U M B E R S … S O G I V E T H E M
N U M B E R S ! H E R E ’ S W H AT H A P P E N S W I T H P U R E
• People are intrigued. Then, either:
o Owners and Designers of the product want to fix ease of use problems
o Owners and Designers of the product push back, refuting PURE
• Either outcome is good! We are finally focusing on improving an
important aspect of user experience: Ease of Use.
• It’s better to have numbers from actual user studies, but if you can’t do it
easily, this is acceptable. It will cause positive change in your organization.

48. K N O W L E D G E
M A N A G E M E N T
I S H A R D

49. R E Q U I R E M E N T S O F A
G O O D K N O W L E D G E
M A N A G E M E N T S Y S T E M
• Easy to find what we have and know
• Easy to contribute new info and docs
• Must handle more than text docs (shared
cognitive artifacts like journey maps)
• Tags and metadata
• Include social and behavioral components
(who contributed, who knows more, who
used what, who likes what)
• A place where knowledge can grow and be
used, not just be stored and accessed

50. A J O U R N E Y M A P O F K N O W L E D G E
Idea
Question
Research
Finding
Finding
Counter
Finding
Counter
Finding
Better
Idea Another
Test
Conclusion
Fail
Hypothesis
Pivot
Finding
THE
Idea
Winner!
Usability
Study
Find
& Fix
Find
& Fix
Find
& Fix
Find
& Fix
Find
& Fix
Find
& Fix
Build &
Release
Market
data
Success!

51. T H E P R O B L E M S PA C E I S L A R G E
• Remove risk, deal with regulation,
and address other industry-specific
challenges
• Teach/empower non-researchers
• Find a way too fit into agile
• Support qualitative research
• Provide quantitative data
• Keep track of and better use your
insights across your design process

52. T H E P U R P O S E O F C A P I TA L O N E R E S E A R C H
We believe opportunity lies in the
understanding of the complexity of people.
DEEP EXPERTISE INTERPRETATON EMPOWERMENTPARTNERSHIP
Ultimately, we help bring insights into design.

53. @christianrohrer and crohrer@yahoo.com
https://www.linkedin.com/in/crohrer/
For more on PURE, contact me or use #puremethod
Thank you