Users do not always accurately describe what they mean or feel. There are many reasons for this, ranging from politeness to poor introspection, to lack of sufficient technical vocabulary. Fortunately, UX researchers have tools in their trade to deduce what was really meant. We call this UX Fracking, a mixed methods approach that is optimized for extracting hidden user insights. We will illustrate the dangers of inadequate, superficial research, and how this may lead to outcomes incapable of addressing the users’ core issues. We will explore ways to avoid these pitfalls by leveraging mixed research methods to test hypotheses about the users’ intent and needs. This starts with a thorough understanding of who the user is, their goals, and how they work today, to an approach that combines surveys, interviews, and comment analysis with behavioral observation, and finally, validating the newly discovered user insights with the users themselves.

1. CIO Design | UXPA 2023, Austin
UX Fracking
Using mixed methods to extract hidden insights

2. Feb 2021
UX Fracking
UXPA 2023, Austin
• UX Fracking:
o Is intentional UX research
o Uses mixed methods
o Tests hypotheses about what users are thinking, doing, or
feeling
o Goes below the surface of token research to crack open
the insights
• “Hidden insights” - because users are notoriously bad at
telling you what they think.
• Understanding what is really going on requires skills, context,
and a scientific, curious outlook.

3. UX Research
Creative Director
Jason Telner, Ph.D. Annette Tassone, Ph.D.
Office of the CIO, CIO Design, IBM
Senior User Experience
Researcher & Designer
Jon Temple, Ph.D.
Design Principal,
UX Design Creative
Director
UX Fracking
UXPA 2023, Austin

4. UX Research: We can
do better!
UX Fracking
UXPA 2023, Austin

5. UX Fracking
UXPA 2023, Austin
The design
cycle –User
research and
design
research
Define
Ideate
Protype
Test
Empathize
Discovery:
User
research
Validation:
Design
research

6. UX Fracking
UXPA 2023, Austin
Design
Research Map
Research Led
Sanders, L. (2008) On modeling an evolving map of design practice and design
research. Interactions 15(6) pp13-17. add t
• Research-led perspective
relies on the data from
qualitative and
quantitative research
methods to inform
design and has the roots
of their methods based in
traditional disciplines
such as applied
psychology,
anthropology, sociology.
• Design-led approaches
are more modern
methods that focus on
the experiences of users
of a product or service.
k.

7. User Experience
Research Map
Scripted (lab-based) use of
product
Natural use of product De-contextualized/not using product
Combination/hybrid
• User experience research
consist of the dimensions of
qualitative vs. quantitative
research approaches, as
well as behavioral vs
attitudinal data.
• Various research methods
lie along the spectrum and
those in the middle consist
of mixed methods and data
sources.
• The methods can also be
categorized by if they
examine users using a
product in their natural
environments, in a lab
based setting, not using the
product or some
combination of approaches

8. Cultural shifts
To save time, we see
general movement
away from users and
their needs and
towards the design.
Move away from
strategic to the
tactical.
User Research :: UX Design
Design Research :: UI Design
UX Fracking
UXPA 2023, Austin
D
e
s
i
g
n
U
s
e
r

9. Do research
faster!
• Commoditization of
UX Research
• No time to acquire
domain knowledge
• Top-down design
(stakeholders, PO)
• UXR adds time
• Do all research
within a
development sprint
(don’t do that!)

10. Design
research vs. UX
research –
compare and
contrast
UX Fracking
UXPA 2023, Austin
What is the
purpose of
the
research?
Design Research
User Experience Research
To inform design of products
services or user interfaces.
To better understand how
users interact and feel about a
product
What
methods
are used?
Use a variety of qualitative
and quantitative methods to
inform the look and feel of a
product or service.
Use a variety of qualitative and
quantitative methods to
understand and improve how a
user interacts and experiences a
product or service.
What are
example
research
activities?
Investigate which design a
user prefers with regards to
how it looks.
Investigate how a user
interacts with a proposed
design and what their
experience is like.

11. Hallmarks of
research
unlikely to
provide deeper
insights
• Focus on preferences – “they
liked A more than B”. (But why?)
• Testing navigation that doesn’t
support real tasks – “they found
it!”
• Looking at just the happy path
• Using narrow tasks that are not
end-to-end
• Sticking to “the script”
• Unsupervised research conducted
by untrained, non-UXR
• Starting research over from scratch
– dismiss “old” data
• Users that never complain or act
negative
• Using personas built in a co-
creation workshop w/o real
users
• Failing to include alternative
outcomes or designs
• Generalizing from a single outlier
• Consider user language to be
technically precise
• Comment analysis: Always
backing into preexisting
categories
• Focusing on extremely rare edge
cases
But may have their
place within the
context of design
research.

12. Dangers of doing superficial research
• Inaccurate insights
• Misinformed design decisions
• Missed opportunities
• Lower user satisfaction
• Higher training costs
• Higher support costs
• Wasted time and money
• High redesign costs
• Failure to innovate and differentiate
Dangers are similar to not doing any UXR; work
products that evaluate without revealing.

13. What is “UX Fracking”?
UX Fracking
UXPA 2023, Austin

14. • Inject “fracturing
fluid”
• Data-driven decisions
• Extremely lucrative
way to get hidden
natural resources
• Controversial
practice (groundwater
contamination,
wasteful, flaming
faucets)
Natural Gas
Fracking

15. UX Fracking
UXPA 2023, Austin
• Metaphor - intentional mixed-
methods research
• Data to decide if big UXR payoff
• Seek hidden UX insights
• Not controversial – except where
companies are not design or research
led
UX Fracking
1

16. Extracting Hidden UX
insights
Great UXR helps us understand what
is in the mind of the user.
Which may or may not be what they
are saying.
UX Fracking is another way to
extract insights from our users.
UX Fracking
UXPA 2023, Austin

17. UX Fracking
UXPA 2023, Austin
UX Fracking in
7 steps
1
1. Listen to what users say (with a critical ear)
2. Watch what users do (with a critical eye)
3. Think end-to-end about the user’s tasks
4. Build testable hypotheses about what was thought or meant
5. Test hypotheses by chaining mixed methods
6. Synthesize findings across methods
7. Review what you think you have learned with users

18. 1. Listen to what users say (with a
critical ear)
1
Always, always talk
to users
Interview them…
Survey them…
Interview them
again!
Words are a clue;
need a lot of
context to interrupt
what is meant.
Users do not use the
same words in the
same way as experts.
Not technically
accurate.
Users are not
designers. Do not
take their
suggestions
literally.
Image from wired.com

19. Role of implicit versus explicit
memory
Human information processing
limitations.
Users may feel or understand
something they cannot
describe.
From: SimplyPsychology

20. “Faster horses”
Henry Ford famously (never) said this:
“If I had asked people what they wanted,
they would have said faster horses.”
Illustrates many users/consumers lack
sufficient creativity and imagination.
Doesn’t mean users do not recognize
value (superiority of recognition over
recall).

21. When users design…
Remember:
• Users Don’t Always Know
What They Want
• Users Are Not Trained
Interaction Designers
• Users Design for Themselves
“Make sure that you do not take
user feedback or requests for
design changes at face value.”
(UX Matters, 2011)
Homer Simpson’s dream car. (Cup holder for beer not
included)

22. Real world example
Long ago, in a usability lab far far
away…
Usability test: Users couldn’t find the
“key” information on a busy
webpage.
Asked them to redesign the page.
I got this:
• Move everything above the fold
• Bold everything that might be
important
• Flaming letters?

23. 2. Watch what users do (with a
critical eye)
1
Methods:
Usability testing –
too late for current
release but rich
quantitative data
for the future.
Disconnect
between what they
do and say is PURE
GOLD!
A-B Tests
Diary studies – watch
users in “the wild”.
Usage metrics
provide more
context.
“Relax, and remember we are testing the
application and not you. You can begin!”

24. Behavioral data requires critical
analysis, too
During WWII, American bombers
shot down.
Naïve analysis: Double up the armor
where there are holes.
Statistician Abraham Wald pointed
out:
• Data from bombers who returned
(Survivor bias)
• "Missing data"
Hidden insight led to improving the
armor where no holes (cockpit,
engines, tail).
Look at those bullet holes! Better double up the armor
there, right? Guess again.

25. 3. Think end-to-end about the user’s
tasks
1
Projects are often
siloed. “In scope”.
But may not fully
reflect what the
user does or need.
Thinking end-to-
end uncovers
hidden pain points,
gaps in
functionality,
missed
opportunities.
For example:
How do they get
started?
How do they know
when they are done?
Methods:
• Journey maps
• Task analysis
• Friction analysis

26. 4. Build testable hypotheses about
what was thought or meant
1
Use “hypothesis” in
a similar manner to
Jeff Gothelf.
Doesn’t need to be
formal or require
significance testing.
Have both a null and
an alternative
hypothesis.
Hypotheses must
be measurable and
capable of being
rejected.

27. • Experience: I’ve seen this
elsewhere – is it true here?
• Violation of a key UX heuristic.
Experience & best practices
2
4
• Stakeholders
• Product Owner
• Development team
• Insightful user comment
Stakeholders &
extended team
• Disconnect between the user model
and the expert or business model
(gaps, workarounds).
• Disconnect between UX methods –
user fails test but reports high
satisfaction.
• Known pain points may lead to
other questions.
User disconnects &
top pain points • Root decisions in research – and
test hypotheses to decide
between alternatives.
• (You don’t need a formal A-B test
to make an A-B decision).
Design decisions
UX Fracking
UXPA 2023, Austin
1
3
Where do
hypotheses come
from?
Anywhere!

28. 5. Test hypotheses by chaining
mixed methods
1
Triangulate on a
problem with a
combination of
methods.
Discovery phase:
fishing, look for
patterns.
Deepening expertise
in a domain: Knowing
what to look for, test
more hypotheses.
Goal is to feel
reasonably confident
you have surfaced an
important insight
that will improve the
solution.
Build out a test
plan, or topological
map, that captures
your journey.

29. 6. Synthesize findings across
methods
1
Proper
methodology is
important; be
prepared to speak
to it.
Record enough for
posterity.
Focus not on
standalone work
products.
The UX insight is the
deliverable.
Focus should be on
the problem, the new
insight.
Helps you understand
the solution better,
what should be done.

30. 7. Review what you think you have
learned with users
1
Review your
findings with users,
give them a chance
to react.
Recognition is
better than recall.
What they could not
articulate, they
might recognize.
If you are wrong, let
the users surprise
you.
“I say: I noticed y'all missed the Submit button
because the scroll bar was hidden. Maybe a pattern
that works great on mobile isn't so great on the
desktop."

31. Story time: How we used mixed
methods to help us understand
the true source of friction
UX Fracking
UXPA 2023, Austin

32. UX Fracking topology
NPS
Feedback
Comments
Walkthrough
of new Device
selection
design
Hypothesis:
Data migration
is the biggest
pain point
Interviews
about end-to-
end device
migration
Friction
analysis
(Analyze +
Quantify)
Survey to
validate
discovery with
larger sample
Compare
to
alternative
design 1
Compare
to
alternative
design 2
Summarize
Longitudinal
UX metrics at
task level
Compare
Monitor
Discovery
Test Hypothesis
Journey map
of As-Is
Evaluate remediation
Validate

33. Business wanted to refresh an old design
Website in need of a refresh
UX Fracking
UXPA 2023, Austin

34. Dunnings – borrowed from collection
agencies
Business wanted
old devices
returned
promptly
UX Fracking
UXPA 2023, Austin

35. Business model
Simone needs
a new laptop
Simone returns
old device
Selects
replacement
device
Receives reminder
emails
Schedules a
return
1. Select a new device
2. Receive reminders to return old device
3. Return the old device

36. App Feedback Walkthrough results
2
1
• Employees (18)
Early research
• App-specific feedback
(comment analysis)
UX Fracking
UXPA 2023, Austin

37. Employee quotes
“I am not a technical person and I work 60-80 hours at a client site. When things
don't work out I am stuck. I have no time to do it and it takes hours longer for a
nontechnical person when things don't go correctly (like installing Notes).”
“I cannot spend a whole day setting up a computer.”
“The new machine migration was
hard.”
“If you are not an expert it's really hard. I am getting a lot of requests
from people around to help or guide them in this process. The main
issues is notes data files/location.”
"Migrating data using Crashplan is not practical at all, and with USB ports disabled, it
was very difficult to know what files/settings to migrate outside of my home folder. "
"What directions there were on the WEB were not real clear when
everything did not start automatically. If not for the help desk then it
would have taken much much longer."
"It would help if the DEVICES@IBM could have a list of applications that we need to be installed before
the device is sent out. Some of this information is available via the IBM Standard Asset Manager
Software list."
[Before returning machine] “2 months. When I’ve got an email from IBM to return the laptop.
Otherwise, I would probably have sent it back after about 4 months.”

38. NULL HYPOTHESIS:
The process is straightforward, and everyone knows what to do*. But users
are very busy and do not have time, so they wait as long as they can.
HYPOTHESIS 1:
The process is complex and time-consuming, and most do not know what
to do.
Users take their time to ensure they haven’t “missed” anything that might
later prove important.
UX Fracking begins with a question…
Question: Why are users really taking so long to return an old device?

39. Interviews
3
Understanding the end-to-end process
• Product Owners
• Subject Matter Experts
(SMEs)
• End users
Journey map
4
• High-level
• Main phases, key steps
• End to end
UX Fracking
UXPA 2023, Austin

40. Business versus the user perspective
Simone needs a
new laptop
Simone returns
old device
Selects
replacement
device
Receives reminder
emails
Schedules a
return
Experience from a BUSINESS perspective
Asks manager
about device
refresh rules
Reads policy info Selects
replacement
device
Receives
device
Follows setup
process
Figure out how to
migrate data
Learn how &
then transfer
bookmarks
Manually
migrates data
Manually
installs missing
software
Wipes data /
passwords /
packs up old
device
Receives Dunning
emails
Schedules a
return
Experience from a USER’S perspective

41. Identifying problem areas
What would you say is the biggest pain point in
this area? (n=27)
Data Migration 15 56%
Installing apps 4 15%
Network setup 2 7%
Platform adoption 1 4%
Device setup 2 7%
Dunning Email 3 11%
How confident are you that you got
everything you need? (n=23)
Confident 12 52%
Somewhat confident 5 22%
Not at all confident 5 22%
Unclear 1 4%
How long did you wait to return your
old machine? (n=24)
Returned same day 1 4%
Within 1 week 4 17%
Within 1 month 7 29%
After 1 month 4 17%
Not returned yet 8 33%
UX Fracking
UXPA 2023, Austin

42. As-is experience: The Journey to A New Device
Tyson
Employee
Scenario
Tyson just started a new project requiring a lot of number
crunching and decided to upgrade his aging PC.
Goal
Quickly select a new PC and confidently migrate all of his data and apps
over, without losing a day of work.
1. How do I get a new device? 2. Order new device 3. Migrate data and apps 4. Return old device
My machine is slow. Am
I eligible for a new
machine? Who do I ask?
Not really sure what to
search for, so I ask my
manager, who told me
to go to Devices@IBM.
Found Devices@IBM
link in search. Not sure
whether I am eligible
will request a refresh
anyway just in case.
I need a more powerful
machine now. But
difficult to compare all
of the specs, which
seem buried.
I don’t see the
‘Developer machine’ I
am told is the best. Will
ask my manager how to
request an exception.
Exception was
approved. UPS tells me I
should receive the new
device soon.
Received my new PC
today. There is a link to
setup instructions in the
box.
Confused about how
to move my data
over. Code42?
Migration tool? A
friend says to copy
everything to Box; it’s
really slow, like half a
day.
Installing apps from
the IBM app store
was a breeze!
Missing some key apps. Need to
track them down one by one. One
requires a new Software License.
I can’t get the VPN to
work. Calling the Help
Desk.
I can’t get IBM
Notes to work.
Calling the Help
Desk again!
Keep receiving
“Dunning” emails to
return my machine. But
I am not sure I have all
of my data or apps yet.
My manager says not to
worry.
It’s been 2 months and I
think I have everything.
Time to return my old
clunker (and shut off
those ”Dunnings”).
Manager Devices@IBM Setupmypc or
Help@IBM
Box, Code42, or
Migration tool
IBM App Store SLM catalog,
App Store, other
Help desk ThinkDesk, UPS,
manager, etc

43. Surveys
4
Does our small group findings generalize?
• End users
UX Fracking
UXPA 2023, Austin

44. 44
Findings
Reasons to
deter you from
refreshing
Breakdown of Categorized Responses for Device Refresh Survey

45. Data migration
I have not
completed
the migration
1%
1 - 3 hours
16%
4 - 7 hours
23%
1 - 2 days
27%
3 - 5 days
16%
1 - 3 weeks
12%
1 - 2 months
3%
More than 2
months
2%
Value Percent Count
I have not completed the migration 0.8% 3
1 - 3 hours 16.4% 61
4 - 7 hours 23.3% 87
1 - 2 days 27.1% 101
3 - 5 days 15.8% 59
1 - 3 weeks 11.5% 43
1 - 2 months 3.5% 13
More than 2 months 1.6% 6
Totals 373
Yes
41%
(147 )
No
59% (213)
26.7%
52.1%
27.4%
6.2%
0
10
20
30
40
50
60
From a colleague In person from
Helpdesk,
Deskside support,
or THINKDESK
Over the phone or
chat with
Helpdesk
Other - Write-in
Percent
Did you need help with your migration
(including all data and applications)?
32% took 3 days or more….
How long did it take?

46. Friction analysis: As-Is
5 6
Task analysis, identify
pain points
PURE scoring of As-
Is steps
• Task analysis - detailed analysis
at the step level
• Use to synthesize findings (pain
points) across methods
• Use a spreadsheet
• Optionally, journey map
• PURE (Pragmatic User
Ratings by Expert)
• Multiple scorers
Current (As Is)
UX Fracking
UXPA 2023, Austin

47. PURE Methodology
Pragmatic User Rating by Experts
From NN/g Nielsen Norman Group: https://www.nngroup.com/articles/pure-method/

48. Scoring process
1. Detailed task analysis
(discrete substeps)
2. Map pain points /
quotes against steps
3. Individual raters
assign score
4. Team score
5. Add them up
As-is PURE for Device
management: 69

49. Compare alternative designs
• Full Data migration
• Full app migration
• PURE: 42
• Reduction in friction: 39%
Design alternative 1:
As Is: 69 (PURE)
Design alternative 2:
• Mirror hard drive in cloud
• Re-installing apps still
manual
• PURE: 58
• Reduction in friction: 16%
UX Fracking
UXPA 2023, Austin

50. UX Fracking
UXPA 2023, Austin
What did we
learn? Why does this matter?
• Majority of development for
device selection, had relatively
low friction, and for emails
urging device returns.
• Should prioritize highest points
of friction over lower.
• Improved automation of
migration likely to lower overall
friction than just telling people.
Winner:
H1
Hypothesis confirmed:
Data migration and app
installation are very time
consuming, difficult to do,
and are directly
responsible to the delay in
returning old machines.
The user model around acquiring
new devices:
Sure, device selection design could
be improved.
But my biggest pain points – was
data migration, reinstalling apps.
These are unsupported.
I don't understand how to start, it
takes a long time to do, and I have
low confidence I was successful.
“
“

51. Where did we
end up? In the beginning:
• Seen but not seen: “I know this!”
• Diffusion of responsibility: "Not my app!"
With UX Fracking, drew attention to issues “hidden in plain sight”:
• Dismissed claims: “everyone knows how to do that!”
• Measured the highest points of friction
UX Fracking
UXPA 2023, Austin

52. Story time: How we determined
why “search” sucked and what
should be done about it
UX Fracking
UXPA 2023, Austin

53. Anecdotal:
“Search
sucks!”
• Search
usage
• SEO
• Alternative
engines
Survey
Feedback
Comments
Hypothesis:
Search may
suck – but
users mean
something
else
Intercept
survey: Ask
about search
experience
Discovery
Test Hypothesis
Summarize
Non-UX
Interviews
about search
experience
Analyze
patterns of
success and
failure
Compare
reported
outcome to an
idealized
outcome
Recommend
pivots
UX Fracking topology

54. NULL HYPOTHESIS:
Users are unable to find the solutions provided.
HYPOTHESIS 1:
Users are unable to find solutions that do not exist or do not meet
expectations.
UX Fracking begins with a question…
Question: What do users mean when they say “Search sucks”?

55. NPS Comment analysis (N=387)
Search-specific complaints (35%)
• “The website looks nice, but it's hard to find
what you need.”
• “Search brings up too much data that has very
low relevance to the search. I wasn't able to
find the 2018 holiday calendar for
Poughkeepsie.”
Content is missing or not relevant (18%)
• “It's difficult to find out the solution to your
specific problem”
• “handles main cases, but not rare cases”
• “Too vague, more helpful searching forums in
most cases”
• “Things I can't solve by myself are the more
obscure issues that aren't handled by a FAQ.”
• “I have installed outlook and it was not easy at
all to find related information.”
Interviews with people who identified
themselves:
If I didn’t find it for any reason, I would agree with
the statement: “search sucks.”

56. UX Fracking
UXPA 2023, Austin
Intercept survey
Reasons for failed searches
N=570
Yes: 52% No: 48%

57. Intercept survey
Comparing what was sought to what could be found
UX Fracking
UXPA 2023, Austin
86% (n=132) were
looking for content
that did not (yet)
exist.
Evaluated the ease of
locating this
information and
compared this back to
the reasons for search
failure.
“Bunch of garbage text for normies
to not feel anxious but no normal
information on how to resolve a
problem.”
“We need a better system in
place to get direct answer to
many avenues to find the correct
answer.”
Survey respondents

58. UX Fracking
UXPA 2023, Austin
What did we
learn? Why does this matter?
• Correct diagnosis
• Enhancements to the search engine
unlikely to help
• Focus needs to be related to
content: missing content, answering
specific questions, quality
Winner:
H1
Hypothesis confirmed:
Complaints about "search"
were not technically about
"search".
The user model around failed
searches:
I used search but...
I did not find anything that helped.
Missing content or content that didn't
meet expectations.
I used search; I did not find an
answer; so I consider this to be
'search' related.
“
“

59. Where did we
end up? UX Fracking surfaced issues completely hidden:
• Naïve comment analysis by keywords alone would
have missed this key insight
• Listened to what users were really trying to tell us:
“I didn’t find what I needed!”
• Disentangled search from content
UX Fracking
UXPA 2023, Austin

60. Parting words
UX Fracking
UXPA 2023, Austin

61. UX Fracking
UXPA 2023, Austin
Summary
Remember: If UX is not
answering these questions,
someone not trained in UX
will act as if they know the
answers.
And they will likely get it
wrong.
UX Fracking can be
combined with any UXR
program to get to the
bottom of key user
assumptions or decide
between design approaches.
UX Fracking is a mixed
methods approach that uses
"hypotheses" to help test
out poorly understood or
hidden user issues.

62. Thank you
UX Fracking
UXPA 2023, Austin

63. Special thanks to Adobe Firefly
UX Fracking
UXPA 2023, Austin

65. Backup
UX Fracking
UXPA 2023, Austin

66. User Experience Index (UXI)
Computation
1. Research end-to-end tasks to determine
the actual “friction” for that task.
2. Design the ideal user experience and
estimate the target friction for the task.
Compute the “Change in Friction”.
3. Create a normalized difference score
(enables cross-task comparison) by
converting it to a 100-point scale with
larger numbers being better. Assign grade.
4. To compare the same task against
different targets or goals, use the Change
in Friction (CIF) not the UXI. Multiply by
100%.
Actual Friction Target Friction
-
Actual Friction
Scores range from 0 to 1, with lower scores being better
(0 means actual experience matches ”ideal”)
UXI = 100 * 1 - Change in Friction
Change in
Friction (CIF)
=
Use the CIF score to compare the same task with different targets; use UXI to
compare different tasks over time against a single but realistic target or goal.

68. Task Ease (3Q21) Ease (3Q22)
Sign in and/or authenticate during installations 70% 75%
Install the software and tools that I need 64% 72%
Connect a new device to the IBM network 63% 69%
Find the software and tools that I need 60% 65%
Return an old device 60% 66%
Select the appropriate computer that meets my needs 56% 67%
Purchase an old laptop/desktop for my personal use from IBM 54% 53%
Setup a new device 51% 64%
Contact the help desk to solve problems encountered during setup 48% 59%
Find help information to solve problems encountered during setup 46% 58%
Refresh/upgrade my laptop/desktop 43% 57%
Transfer all existing data from my old device to a new device 42% 55%
Request an emergency replacement for my laptop/desktop 41% 55%
Find the relevant policies and procedures about acquiring devices or accessories 37% 48%
Manage registration - 66%
Long-term monitoring
1 Very difficult --- Neither easy nor difficult --- Very easy 7