The document outlines a process for driving data-driven change in legacy organizations, emphasizing the importance of involving data at the beginning of product development and asking the right questions to identify and prioritize user problems. It discusses a case study on improving document retrieval, detailing hypotheses formation and evaluation, leading to design changes that enhanced user experience. Key takeaways include the necessity of collaboration among teams and the establishment of proper metrics for tracking success.

1. How to drive data
driven change in a
legacy organization
Subhasree Chatterjee
Sr. Data Analyst
LexisNexis

2. About me
Lead Data Analyst Master’s in
Business Analytics
Long distance
runner

3. Who we are
• Innovating to support each
customer's success
• Combining information & analytics
to help customers make more
informed decisions & achieve
better outcomes
• Advancing the rule of law around
the world
What we do
• Help lawyers with comprehensive,
efficient research
• Collaborate with universities to
educate students, and
governments and courts by making
laws accessible and strengthening
legal infrastructures
• Partner with leading global
associations to advance the rule of
law

4. Typical product workflow
Search Document View
Document
Retrieval

5. Data-Driven Product Design
Identification &
Prioritization Phase
• Get involved at
the beginning
• Ask the right
questions
• Utilize different
sources of data
Hypotheses
formation phase
• Define specific
and testable
hypotheses
• Create metrics to
define success
• Set up proper
tracking
mechanisms and
dashboards
Delivery Phase
• A/B testing
• Feature
Performance
Dashboard
• User feedback
• User satisfaction
score
How to shift from “gut feel”/received wisdom to data informed design change

6. Qual+Quant Data Triangulation Process
WHAT?
Why?

7. Case Study:
Enhancement to Document
Retrieval

8. Received wisdom around document
retrieval
TOP PREDICTOR OF USER
SUCCESS
HIGHER = BETTER

9. Identify & Prioritize the Customer Problem
EVIDENCE
• CX, NPS, and UX feedback indicate that users are having trouble with
document retrieval
PRIORITIZATION
• Number of document retrieval is a top NPS driver (both positive and
negative)
• Document retrieval is a key indicator of overall task success
CUSTOMER
PROBLEM
• Users are experiencing high document retrieval failure and
cancellation rates

10. Form Hypotheses
1. Users have difficulty distinguishing between the document retrieval icons(print,email,download)
in the toolbar, causing them to accidentally choose the wrong action
2. Users are drawn to the strong affordance of the “quick” action icon
3. Pop-up blockers are causing the retrieval to fail

11. Evaluate Hypotheses
1. Users have difficulty distinguishing between the retrieval icons in
the toolbar, causing them to accidentally choose the wrong action
36% of users retrieve via more than one method on a single document
“One of my biggest problems is printing. I typically have to print a document
multiple times before I can get a full download that will print. And even when it
works, it’s slow.”
WHAT?
WHY?

12. Evaluate Hypotheses
2. Users are drawn to the strong affordance of the “quick” action icon
80% of users retrieve via the “quick” action icon
“I actually did not know you could do all of the items in this survey. I usually just
press email and okay. It is interesting to know that I could customize how the
document is emailed.” -- Associate
WHAT?
WHY?

13. Evaluate Hypotheses
3. Pop-up blockers are causing the document retrieval to fail
Cancellation/failure rate were around 10%
“How do I print a case? when I print, it does not go to my printer, nor does a printer
dialog box pop up so I can choose a printer. and I can't find anywhere to set printer
settings.” -- Associate
WHAT?
WHY?

14. Prototype and Deliver
AMBIGUOUS BUTTONS WERE
CLEANED UP
DOCUMENT RETRIEVAL WITH
SETTINGS WAS MADE THE
DEFAULT OPTION
A “PROCESSING BUDDY” WAS
INTRODUCED

15. A/B Test: Document retrieval with settings
won’t have any effect on cancellation rate

16. Design changes Launched. Panic!!

17. Right
Metrics to
look at
New toolbar
iconography
“Ambiguous”
icons removed
New
processing
window

18. Lessons
Learnt
• Importance of defining the right metrics
• More is always not better!
• Importance of proper tracking of user actions
• Importance of working together with other
teams to rule out the opposing ways behavioral
data can be interpreted

19. How to make this process work for you?
1. Get involved at the beginning. Data (both qualitative and quantitative) is very important at all stages of the product development
lifecycle, but arguably the most important at the beginning, while assessing the feature itself
2. Ask the right questions.
• Is there really a problem to solve?
• How big is the problem?
• What change are we trying to bring about and why?
• How do we drive the necessary change in behavior?
3. Eyes on the prize. How do we measure success or failure?
• Create metrics to define success. Just one composite metric or some vanity metrics won’t help. Have specific sets of metrics for
a specific chunk of a problem.
• Make sure to have proper tracking mechanisms to measure and have a visual setup to track the movement of those metrics
4. There is no such thing as no data, even when there is time pressure
• Use usage data, customer support call data, nps data, heuristic reviews, etc. to gain a better understanding of how many users
utilize the feature and what their pain points may be
5. Collaboration is everything! Having Product, UX and data science consistently talking to each other is important both to avoid wrinkles
going forward and work efficiently

20. Key Questions to ask
•What user problem are we trying to solve
•What evidence do we have on the problem
•What’s users’ reaction on the solution
•How do we know we have solved it

21. Thank You
subhasree.chatterjee@lexisnexis.com
www.linkedin.com/in/chatterjeesubhasree