Diane Bartoli discusses how analytics can drive effective product development. She outlines several tools that can be used including ethnographic research, workflow mapping, and quantitative surveys to understand user needs and pain points. Case studies of Elsevier Evergreen show how these tools were used at different stages of development to gather feedback and measure results. While there is a lot of available data, asking the right questions is key. Analytics should be integrated throughout the product development process from alpha to beta testing to ensure products truly meet user needs.

1. Analytics to Drive Effective Product Development
Diane Bartoli
Vice President, Product Development
SSP Boston, June 1, 2011

2. About Me
Vice President of Product Management
Elsevier Health Sciences
Clinical Reference Group
Analytics EXPERT
NOT*
*(Just a product manager desperate for data to
inform good product development...)
2

3. With seemingly endless data, is the world not your...?
3

4. A few of the tools at your disposal...
WHICH TO USE???
4

5. The trick is not capturing the data,
it’s asking the right questions...

6. Case Study: Elsevier Evergreen
A new Online Medical Reference product
in development intended for Doctors in
the Hospital setting
6

7. Where is the need? Understanding the user
• Ethnographic research brings end-users to life
• Highlights pain points
• Understanding motivations and ambitions

8. What is needed and WHEN? Mapping workflow…
Patient care management workflows Collateral workflows
Diagnosis
1
5
2
Keeping current
•Conferences / society meetings
Create Care Plan •Reading current literature
2a
Treatment 2b
Medical treatment Surgical treatment
Procedures
4
Sharing information
•Patient specific information
•Case presentation (within dept)
After treatment care plan •Presentation to other bodies (e.g.
Treatment committees, drug committees
3 etc)
•External presentations (Grand rounds,
association presentations etc).
Patient Education & Compliance
• Collateral workflows take 24% of the
•Patient care management workflows take over 45% of the average physician's time
average physician’s time
Non-surgical Surgical
Source: Qualitative research (76 in person respondents)

9. Quantifying qualitative findings....
Key Findings
• Validation of critical information
needs for physicians: trustworthy,
current content
Daily activity
Weekly activity
• Validation of critical pain points:
Weekly/monthly
Monthly activity
lack of comprehensive access to
Annual activity
information, clinical search
Total time spent (hrs)
relevance not optimal today
• Quantification of physician
workflow: Understanding what
they do, how often they do it, how
important those activities are,
when they need information and
what information they need
• Business justification: Sizing
market opportunity and potential
growth
% of time medical reference products are requested
9
9

10. Zeroing in on the goal…
The Physician’s No current offering delivers on all three components doctors
Dilemma need in a medical information product
Comprehensive Trusted
• Regardless of • Despite generally preferring E
where they start over P, physicians find PRINT
Online
their online search, resources more trustworthy
40-53% of doctors Books (75% vs 59%) than current E
use more than one offerings
source Online
Aggregators Journals
Elsevier
Clinical
Evergreen
Reviews
Wikis
Speed to Answer
• Residents report significant
use of Google (64%),
Wikipedia (27%) and WebMD
(36%) given need for fast and
easy reference

11. Building products with analytics at the core: Alpha,
Beta....Zeta Volunteered
comments :
Capturing the voice
of the user
Understanding
Motivation: Why
are you here? Are
you satisfied?
Elsevier Evergreen
Quantifying results:
Quantitative
Survey at close of
Alpha Period
Measuring the Right things:
Composite metrics intended to get to
“workflow”
11

12. Lots of Data.... What to do with it?
System Data:
• What are they looking for? Search logs help tune
search relevance and recall
Usage Data:
• What was used? What was not used?
• Are users coming back?
• Are we inspiring loyalty and engaging our users?
Satisfaction:
• Why are they coming to our site?
• Did they really find what they were looking for?
• Touchy-feely: visual design? Cultural differences?
Quantifying results:
• Prioritization of development
• Directing go-to-market planning
• Return on investment?

13. Some lessons
• Setting appropriate expectations
• Imperfect data yields imperfect results
• Apples, oranges, and some pears too: matching mis-
matched data
• The Holy Grail: marrying customer demographics to usage
and conversion
13

14. More questions?
Diane Bartoli,
VP, Product Development
D.bartoli@elsevier.com
215-239-3602
14