UXD v. Analytics - IUE12 Ann Arbor

UXD v. Analytics
What have YOU got for me? No, what have you got for ME?

Internet User Experience 2012
July 18, 2012

Chris Farnum and Farris Khan
Twitter: #IUE12 #Measure @crfarnum @BobbleHeadGuru 1

Background – The ProQuest Platform
• Search application for
journals, news, magazines,
and many other types
• Academic and corporate
audiences – one platform, two
skins
• Majority of academic
customers now have migrated,
corporate migrations
underway
• Summary – the bootstrap days
are done, we are now
extending and refining

The UXD Role At ProQuest
• Agile methodology Core UXD
• 12 team members Design Processes

• Work on large scale
projects
– Global Prioritized Develop/Test
– Multi-year requirements

• We do projects to
translate business Planning
requirements into user
experiences Iteration release Product
• Work closely with release

platform management,
technical teams, and QA


The UXD Approach, In General…

• Mostly Qualitative - Often focused on one-on-
one sessions, with a small group of carefully
chosen users
• Focused on “Why did you do that?”
• Often used for tactical design decisions
• Often offers a deep and narrow perspective
on users
• Practitioners develop a gut feel, instinct for
what will work for their users’ preferences


A New Development For UXD

UXD was recently placed within the
same platform management team
as our digital analytics lead, Farris.

The big question…
Who is this guy and what’s this
report he’s publishing about the
platform we designed?


Overlapping Territories? Competition?

• UXD and Digital Analytics both claim lots of
expertise about users
• Quantitative v. Qualitative?
• Influence over strategic direction?


What is Analytics?

Actionability Smart
System
Strategy and
Operations are largely
driven by Analytics

Decision
Support
Decisions are made
based on the data
Predictive Model is
Intelligence established
The trend-line is long
Measure enough to determine
what is in the acceptable
A credible set of v. is in the “needs
“report card” metrics improvement” range
established
Maturity


Daily Heatmap


Weekly Heatmap


Monthly Heatmap


Trend-line = Uncooked Spaghetti


Trend-line: Good


Better

Twitter: #IUE12 #Measure @crfarnum @BobbleHeadGuru

Even Better


Comparing UXD and DA
Digital Analytics User Experience Design
Mostly Quantitative Mostly Qualitative
Page or widget level statistics, Often focused on one-on-one
can be enhanced via surveys sessions, with a small group of
and segmentation carefully chosen users
Focused on “What did most Focused on “Why did you do
users do?” that?”
Often used for performance Often used for tactical design
reporting, strategic decision decisions
making
Often offers a broad and Often offers a deep and
shallow perspective on users narrow perspective on users
Practitioners develop a keen Practitioners develop a gut
eye for trends, changes and feel, instinct for what will
patterns in user data work for their users’
preferences


Farris and Chris work in back-to-back cubes and
this leads to interesting conversations…


Conversation #1 – Interpreting Report Data

One day Farris asked,

“Why do some of our segments have
significantly higher satisfaction scores than
others?”


Conversation #1 - Why Is Satisfaction Higher
For Some Segment v. Others?

NOTE: Data has been altered


Conversation #1 – Interpreting Report Data
UXD: Personas and Segmentation


Conversation #1 – Summary

• DA interprets heartbeat reports:
– Important to understand institutional
baselines
– Analysis of KPIs by audience segment
• UXD can offer:
– Insight into personas and design priorities
– Help on whether deltas are expected or
unexpected


Conversation #2 – Narrowing Options

One day Chris said,

“There’s a customer service request to improve
the ease of narrowing results. Is there any user
data that can give me the background I need to
design some new approaches?”


Conversation #2 – Narrowing Options
UXD: Many existing options. Do we need more, less,…?


Conversation #2 - Open Ended Comments, Area
of Focus
“Narrow Results”
The following pages contain analysis from “representative comments” from November data.

The data is a subset of 272 comments from respondents with comments who were in the
bottom middle quartile in satisfaction, meaning they were slightly below the median in
satisfaction. This group is considered an area of focus because their scores are “on the fence”
and it is realistically possible to “win over” these visitors (moving them from the bottom middle
quartile to the upper middle quartile).

The 24 comments that are highlight the concept of “narrowing results” which was identified to
be the largest topic associated with the areas of focus (search) in this group.

NOTES:
1. This topic was selected based on the frequency of the term. We can conduct similar analysis
for other topics.
2. The first page contains the actual open ended “verbatim” comments. The second page
contains respondent level detail about the visitor. This is an abridged version of the larger set of
data that we can obtain.



• UXD needs background info before beginning
to design solutions:
– Does the issue impact many users or just a few?
– Observed behavior patterns, ratings and
comments can provide insight
• DA can offer both qualitative and quantitative
data:
– Satisfaction scores based on surveys
– Analysis of a segment of the data
– Review comments
– A virtual user research study


Conversation #3 – Interpreting Emerging Trends

One day Farris said,

“I was preparing my monthly report and noticed
something new. Page views per visit
dropped, but other statistics remained constant.
What does this mean?”


Conversation #3 – Monthly Heatmap


Conversation #3 – Weekly Heatmap


Conversation #3 – Interpreting Emerging Trends
UXD: Updates to site navigation, database selection



• DA spots trends in regular reports:
– Determines whether changes in KPIs are
significant
– Trends may suggest priorities for planning
site development
• UXD can offer:
– Possible explanations for changes
– Communication about upcoming site
updates
– Assistance in identifying important KPIs,
creating and refining survey questions…


Conversation #4 – Is This Design Successful?

One day Chris said,

“We show price notifications to corporate
customers for each transaction. How often do
users turn off these notifications?”


Conversation #4 – Is This Design Successful?
Price notifications in the corporate interface
– Pay per document
when not subscribed
– Design question -
should we show
notifications for every
paid transaction?
– Is option to “do not
show” sufficient for
users who don’t want
the interruption?
– Are further
refinements needed
or is the current
design working fine?


Conversation #4 - Comparative Analysis


Conversation #4 – Also…

• Path tracking – needs development effort
to embed tags
• Session Replay



• UXD would like to evaluate designs and
features:
– We want to know if a feature is serving it’s
purpose or needs refining
– If features aren’t used or are unpopular, we may
want to remove or replace them
• DA can offer deep analysis
– However, extra effort is needed
– Prioritization relative to other projects will be an
issue


Next Steps - Evolving Our Process & Working Together

• Strategic decision making - transition from gut feel 
data driven
• Deeper analysis based on a richer understanding of
user personas and motivation
• Measure clicks and paths, not just pages


Small Group Tweet Up

• Brainstorm how you would like to see
analytics and UxD fit together in your
organization
• Everyone contribute
• Pick top 3
• Post on Twitter: #IUE12 #Measure


Questions and Comments

Farris Khan
Farris.Khan@ProQuest.com
@Bobbleheadguru

Chris Farnum
Chris.Farnum@ProQuest.com
@crfarnum

Slideshare
http://www.slideshare.net/ChrisFarnum/farnum-khan-iue2012


UXD v. Analytics - IUE12 Ann Arbor

Recommended

Recommended

More Related Content

Similar to UXD v. Analytics - IUE12 Ann Arbor

Similar to UXD v. Analytics - IUE12 Ann Arbor (20)

More from Chris Farnum

More from Chris Farnum (13)

Recently uploaded

Recently uploaded (20)

UXD v. Analytics - IUE12 Ann Arbor