UXPA 2023: Managing UX Research Vendors for Success

©2022 Discover Financial Services
1
Li Wen Huang
Discover Financial Services
Managing UX Research Vendors for
Success

2
Public
Agenda
Introduction
Finding the Right Partners
Evaluating Fit
Setting Up Yourself and Vendors for
Success
Friction Points
Questions

3
Public
About Discover
We provide banking and credit products that
help people achieve their goals. In payments,
our networks connect banks, cardmembers and
merchants around the world with innovative
commerce solutions.
• Credit Cards
• Checking & Savings
• CDs, MMAs, IRAs,
• Loans: Personal, Student, Home Equity
• Discover Global Network, Diner’s Club
Our Operating Model
• Outsource design, research and dev work
‒ Internal team manages outsourced work,
set strategy & business goals

4
Public
About Me
• Been with Discover for 15+ years focused on UX research
• Proposed and started UX Research and Digital Voice of the Customer programs in
E-Business supporting credit card portfolio business
‒ Identified and onboarded UXR vendors and continue to expand our capabilities
‒ Manage ongoing relationships with vendors
‒ Started with 2 projects in 2008, peaked at managing 46 projects/year
• Moved to Consumer Insights in 2015 and began supporting the entire enterprise
‒ Focused on external customer and consumer facing experiences
‒ Managing 2 UX researchers
‒ Part of CI team of 15 who support market research, brand tracking, ad/media tracking, competitive research, behavior
insights & UXR
• Things to know about me:
‒ I’ve been called a force of nature in performance reviews
‒ Internal teams and design agencies seek me out for help; I’ve gotten projects from internal word of mouth

5
Public
Disclaimer
Everything presented here are my own personal views and experiences
and I am not speaking on behalf of Discover Financial Services.

6
Public
Why are you here?
• Who are you and what you want to know/learn?

7
Finding the Right Partners

8
Public
Why Outsourcing UXR Works
• Politics
‒ Vendors are a neutral 3rd parties and you need the research to come
from an independent source
• Special expertise
‒ Access to experts and resources focused solely on UXR, including
difficult recruits, etc.
‒ Can engage multiple vendors/agencies with different strengths and
capabilities
‒ Can train new/internal team members on UXR methods and best
practices
• Bandwidth
‒ No need to find and hire UXRs
‒ Can expand and contract with ebb and flow of work
‒ Projects that are too big or too small for internal teams

9
Public
B2B Dating Game
Before You Start, Define What You Want
• What does your company need?
‒ Full service vs. self-service
‒ Types of research or specific UXR activities (i.e.,
recruiting)
‒ What’s your process (agile, waterfall, etc.)
‒ Timelines
‒ Do you have budget?
‒ Contractual or other business requirements

10
Public
Full-Service UX Vendors
Pros
• They do everything from scoping, screening/recruiting
users, discussion guide, moderation, report writing and
highlight clips
• Can do very specific audience targeting and balancing
of recruit
‒ May be able to handle customer lists
‒ Access to multiple panels/international panels, etc.
• May be able to conduct types of research that are
more complex or beyond self-service capabilities:
ethnography, multi-method research, etc.
• You can give them access to self-service platforms to
set up run research for you, when allowed
• High-level of expertise, experience and knowlege
Cons
• Often more costly than self-service model
• May not be as fast as self-service
• May not be able to work with same researchers for
each project
• Will likely need to briefed on how your process and
industry works
• Smaller agencies may not have the bandwidth you
need
• May not be good at certain types of research vs.
others

11
Public
Self-Service UX Vendors
Pros
• DIY research platforms have their own panels and
research can be completed in hours
‒ Unmoderated usability session videos are available
for sharing after sessions are done
• Research platforms can provide access to multiple
tools, for one-stop research
• Can set up many people to run research on their own
• Can be very cost-effective depending on how the
subscription is set up
• Provide training for users and tech support
Cons
• You need to take the time to set up, run, and review
results and write your own reports for the fastest
results
• Professional services can set up research for you, but
may not be 100% dedicated to you and take longer
than self-service
‒ May only provide general insights, since they are
not familiar with your industry or experience
• Participants are more likely to lie to get into research
and may not be your real target (check YouTube)
• Garbage in = garbage out
‒ Even with templates and guides, you can do a lot of
bad research

12
Public
Specialized Vendors: Recruiters, User Panels, Tools
Pros
• Provides the service you need without all the other un-
needed frills
• Can be very cost-effective
• Some vendors may not need to be “onboarded” and
services can be purchased via credit card
Cons
• You may still need to go through vendor onboarding
and periodic compliance assessment process, even if
you only use them occasionally
• If it’s a self-service tool, you still need to have the time
and resources to set up and manage

13
Public
Types of research you need and what they can provide
For full-service vendors:
• How often or how much experience
do they have with a type of
research
• Cost and timelines for research
meet your expectations
‒ Compared to industry standard
costs

14
Public
How to Find Vendors
• Word of mouth
‒ If you have colleagues in the field who use vendors or know
people who work with UXR agencies, it’s a good way to find who
is out there and who is good
• Online directories
‒ Can identify companies that your connections may not know or
used
• Conference exhibitors
‒ Can meet to discuss capabilities with company reps
• Google

15
Evaluating Fit

16
Public
Evaluating if they are a fit
• Size and capacity
• Financial stability/viability
• Focus of their business: are they competitive research or
market research with a UXR add-on? Or are they UXR only?
• Example reports and case studies
• Demos/mock sessions
• Pilot research/proof of concepts
‒ Some vendors will do a free pilot of proof of concept
‒ Others may allow for a trial period to evaluate and allow
either party to walk away free
• Referrals and references
‒ Definitely check references

17
Public
Your Process vs. Their Process: Don’t Assume Anything
• Ask for concrete examples of what they have done
and what they can accommodate
• Confirm your internal processes can also
accommodate the vendor’s process
• What data are you allowed to share with a vendor?
Can the vendor securely handle your data?
‒ Do you have specific procedures they need to
follow? Clarify and validate with risk, compliance
and legal
• Are they familiar with any rules and regulations that
affect your industry

18
Public
Corporate Culture
• Have similar or acceptable viewpoints about
customers
• Have similar or acceptable viewpoints about
research
• Expectations on the working relationship

19
Public
Contracts should be built for Worst Case Scenarios
• Build in flexibility that works both ways
‒ Timelines and dates can shift if you’re not ready or the vendor
needs more time to adjust to scope changes
‒ You’re not locked in or obligated to pay future work that may not
move forward
‒ Ability to credit pre-paid work towards different projects
‒ Fair payment terms (50% at signing, 50% on completion). Pay
100% up front only for trusted partners
• For new partners, consider a proof-of-concept or trial period, which
gives you the ability to back out if it doesn’t work out as expected or to
move forward if it does
• All appropriate laws, regulations, etc. are outlined
• Be Fair!
‒ Don’t get a bad rap by trying to make it one-sided

20
Setting up Yourself and Vendors for
Success

21
Public
Put a process in place
• Develop a process, document it, share it
‒ What can run in parallel, what cannot
‒ Much of planning of UX research such as recruiting,
discussion guide development, etc. can be done in
parallel
‒ Do you want vendors to use your report template or
are you okay with theirs
‒ Provide any procedures that vendors may need to
follow before you begin (customer list handling,
NPI, etc.)
• Start small and grow it
‒ Refine as you go; work via e-mail vs. meetings, etc.
‒ Highlight clips vs. verbatims
• Track projects, so you can get a sense of timing

22
Public
Set expectations for internal teams and keep reinforcing them
• The role of customers and consumers in research
‒ Ensure internal teams understand consumers are not here to
solve your problem; they’re here to provide feedback (and not
all feedback is valid or appropriate)
• Role of research team/vendor
‒ Vendor should outline all tasks, timelines, etc. and what they
need from you
• Role of company teams
‒ Building prototypes
‒ Approving screeners/discussion guides, etc.
‒ Attending live research sessions
‒ Reviewing recorded sessions (watch parties)
‒ Incorporating changes

23
Public
You are the expert on your industry, brief your vendors
• Translate your insider lingo
• Explain how processes work—don’t assume they know it
• If you don’t understand the process or project, get the
background or make sure the product owners can explain
it in laymen’s terms
• Explain what is considered good vs. bad “This really
sucks!” – vacuum cleaner manufacturer = good; everyone
else, probably bad!
• Provide typical scenarios, representative data,
• If you’re testing edge cases, communicate this to your
vendor

24
Public
Get the right information from the start
• Goals and key performance indicators for the project
• Who is their audience/target
• What do they already know (analytics, past research, etc.)
• What they want to understand or learn
‒ Is this something that research can answer?
• Work in progress designs/flows
‒ What user/customer problem will this solve
• When do they need research results

25
Public
Showing Value - Reports
• Review reports before they are shared
‒ Make sure key questions are addressed
‒ Edit to ensure internal users understand the insights
and recommendations (user-friendly language)
‒ Incorporate additional insights from other research if
needed
‒ Stay in your lane—focus on research results and recs,
not dictating design, dev, etc.
‒ Ensure and adapt reports based on audience (tactical
team vs. executive leadership)
• Own the research—understand how it’s done, why it’s
done this way—ask questions, so you understand and
can explain it yourself
• Become a UX research expert & champion—take
classes, webinars, attend conferences, etc.
Reporting Best Practices
1. Order recommendations by descending
importance. Top rec should be most
important, such as “go/no go”
2. Don’t cite numbers when doing qualitative
research
3. Include screenshots for future reference
4. Anticipate and address questions you may
get, including supporting external data

26
Public
Proving Value
• Vendor scorecards
‒ Meeting expectations on deliverables, timing, cost, etc.
• Retrospective
‒ For redesigns, pre-launch metrics vs. post-launch metrics
• Summer Intern Project--Analysis of changes incorporated into designs
based on UXR research
‒ After launch, confirm what changes were made based on research
‒ Only include in-scope findings/recs
‒ Weight by business value/need & usability severity (H, M, L)
‒ Additional insights outside of usability

27
Public
Communicate, collaborate and compromise
• Support your vendor if they are right
‒ Don’t expect your vendors to fight your battles
‒ If they are presenting, brief them and arm them with information
that will support their position
‒ Voice your support: give credit where credit is due
‒ Don’t claim other people’s work as your own
‒ Don’t sugar coat things to win favor; be honest but be nice
• Be willing to say No nicely but firmly
‒ Scope creep can dilute your research value and increase timelines
‒ Understand that changes of directions will affect timelines, costs
and even the assigned vendor team—be reasonable
• If saying no is not an option, see where you can compromise and
clearly communicate it

28
Public
Don’t get pushed around
• It’s your research, so own it—provide clear guidance to
the vendor of what works and what doesn’t in your
organization
‒ You may need to say it multiple times or in different
ways or to different people
• Don’t compromise on things that truly matter: integrity,
values and your customers
‒ Are they willing to stake their reputation and business on
it? Or their personal salaries?
• Inaction may set a precedent for the future
‒ Address it up front

29
Friction Points

30
Public
You inherited a vendor and you…
Want to change things
• Evaluate what’s working and
what’s not
‒ Build a case: feedback from
teams; be specific
‒ Get leadership POV
• Outline what changes you want to
make
‒ Get leadership alignment and
support: speak in their language
of improved business value,
expense management, etc.
‒ Share changes and set up
meeting with the vendor to
discuss
‒ Give vendor time to respond
Hate them but can’t get rid
of them
• Try to address the reason you
hate them
‒ Is it personality? Process?
Quality?
‒ Can it be worked out or
can you build a case
against them?
• Can you find another vendor to
replace them?
‒ Create a business case for
another vendor or different work
model
‒ Be willing to start small and see
if you can prove better business
value
Don’t know what you’re
doing
• Educate yourself!
‒ Ask the vendor to help you
‒ Project management training
‒ UX research training/webinars
‒ Read, network, find a mentor

31
Public
Win-Win Negotiations
• Don’t look at things from your perspective only; see it from both
sides
‒ How does this change benefit the vendor as well as you?
‒ Find common ground on where you do agree
‒ Take a step back if things get heated
• Be willing to compromise and get small wins
‒ If you’re asking a vendor to change what they do, it may
take time for them to understand your perspective
‒ Own any changes you ask for
• Agree to disagree, but don’t air it publicly

32
Public
If something goes wrong, do you know what to do?
• Does your vendor know what they need to do if something
goes wrong?
‒ Share requirements and procedures up front
‒ How do they notify you of a problem
‒ Documentation of extent of issue and impact
‒ Mitigating actions by you or vendor
• Do you know what to do if something goes wrong?
‒ Reporting, forms, etc.

33
Public
Breaking Up
• Do you terminate or just stop using them?
‒ Is there a possibility you’ll want to use them in the
future?
• Understand contract terms for termination
‒ Number of days notice
‒ How termination needs to be communicated
• You can’t avoid the awkward conversation/meeting to
discuss why
‒ Keep it professional and cordial
• What do you do if you still have projects in process?
‒ Is it better to let them finish or to walk away?
‒ Can someone else take on the work without
affecting cost and timelines?

34
Public
Questions?

UXPA 2023: Managing UX Research Vendors for Success

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to UXPA 2023: Managing UX Research Vendors for Success

Similar to UXPA 2023: Managing UX Research Vendors for Success (20)

More from UXPA International

More from UXPA International (20)

Recently uploaded

Recently uploaded (20)

UXPA 2023: Managing UX Research Vendors for Success