For the past three years, The Wall Street Journal has used experimentation to help create their membership experience. Starting with acquisition and expanding to engagement, product, and retention testing, WSJ has built a data-driven culture that leverages these experiments to deliver a premium experience that grows their base and keeps members coming back for more. In this webinar, Olivia Simon from The Wall Street Journal’s optimization team shares how WSJ created their robust and successful testing program. You’ll learn: How to think about testing in a membership ecosystem How WSJ prioritizes tests and earns stakeholder buy-in About WSJ’s emphasis on testing throughout the customer acquisition funnel How WSJ expanded their testing funnel beyond acquisition, and the key testing moments they target for consumer engagement and retention

1. 1
Optimizing Membership at
the Wall Street Journal
Olivia Simon & Adam Szakacs
How the Wall Street Journal leverages experimentation
to create their subscription ecosystem
+

2. 2
Adam Szakacs
Manager, Customer Success
Olivia Simon
Manager, Optimization
Introductions

3. 3
Housekeeping
● We are recording this webinar
● You’ll receive the recording and slides
● We’ll answer all questions at the end

4. optimizely.com/digital-change-agents/

5. 5
Agenda
● Introduction
● About Optimizely
● The Wall Street Journal’s Journey
● Q&A

6. Revenue, share of wallet, funnel conversion, risk
mitigation, ops efficiency
What
We Do Next gen “Test and Learn” platform for
enterprise-wide digital experience optimization
Over 1B impressions daily
Replaces Digital Guesswork
With Evidence-based Optimization
Digital Experimentation
SaaS Platform
Apply the scientific method to “at-scale”
business decision making.
20X increase in Yield

7. “Our success is a function of
how many experiments we do
per year, per month, per week,
per day.”
Jeff Bezos
“Our aim is to create the best
product for our customers, and
we do that through constant
innovation and testing.”
Gillan Tans, CEO
“Our company culture
encourages experimentation
and free flow of ideas.”
Larry Page
“We use experimentation and
testing to inform as much of the
business as we possibly can” -
Gregory Peters, CPO
Today’s Digital Leaders Win By
Using Experimentation At-Scale

8. What
We Enable
1500+
+51%
+$21M
+19%
+28%
Improved subscription
acquisition
Users Scale and agility
of experiments; users
incremental
in revenue
Improved Driver
Acquisition
increase in cross-sell
performance

9. Spent with Digital Media
19hr
week
2018
Digital-Centric Customer
Spent with Digital Media
41
$396m
Mobile commerce Mobile commerce
$1 8t.
2008
Digital-Convenient Customer
hr
week
Your customer has fundamentally
changed how they engage

10. Experimentation: Testing variations of digital experiences
with real users and deploying what wins
+21%
-5%

11. Frontend UI Backend Business
Logic & Data
The anatomy of an experience
i.e. navigation,
search location
& visual treatment
Copy
Images &
Colors
Layout Search algorithms
Personalize content
based on previous
behavior
Recommendations
Make your headlines
more personal

12. Device coverage Complete
experience
coverage
Open platform
Trusted results
quickly
Consumer-grade
usability
Security &
compliance
Program
Management
Strategic & tactical
partnership
Build
Digital
Products
Market
& Sell
Service
& Retain
What you need
Marketing
Product &
Engineering
Support
The Optimizely Solution
Consumer Grade
Usability
Program
Management
Purpose- Built
Statistics Engine
& Machine
Learning
Open Platform
Data &
Integrations
Security &
Compliance
OTT
Voice/IOT
Mobile
Web / SPA
Frontend
Experimentation
& Personalization
Server-side
Experimentation
Best in class
Services &
Success Teams
SUCCESS ACCELERATIONEXPERIENCE COVERAGEDEVICE COVERAGE TRUSTED RESULTS ENTERPRISE SCALE

13. Success Acceleration
MATURITYANDCOVERAGE
LEVEL 1
Executional
Start
LEVEL 2
Foundational
Growth
LEVEL 3
Cross-functional
Advancement
LEVEL 4
Operational
Excellence
LEVEL 5
Culture of
Experimentation
Start Anywhere & Scale Up
To a Culture of
Experimentation

14. We’re Proud to Work With Great Global Enterprises
24 of the Fortune 100 have chosen Optimizely to drive their digital experience

15. Optimizing Membership
at WSJ

16. DOW JONES OPTIMIZATION
Que(test)ions
1. Where do you test?
2. How do you prioritize what you test?
3. Show me the tests?
4. Advice on starting a program or boosting
my current program?

17. DOW JONES OPTIMIZATION
How we view the membership testing lifecycle:
Acquisition:
How do we acquire new
members?
Engagement:
How do get new members to
take transactional product
actions that will make them long
term subscribers?
Product:
How do we increase the
usefulness of WSJ?
Retention:
Once a member decides to
leave (or has left), how do we
keep them (or win them back)?

18. DOW JONES OPTIMIZATION
What does the WSJ ecosystem look like?
Acquisition:
Engagement:
Homepage placements
Article roadblocks
Shop
Checkout
Onboarding
Newsletter Center
Platforms Center
Customer Center

19. DOW JONES OPTIMIZATION
What does the WSJ ecosystem look like?
Product:
Retention:
Home page
Article pages
Videos / Video Center
Online Cancel
Winback shop
Winback emails
Credit Card Decline

20. DOW JONES OPTIMIZATION
So where do we test?

21. DOW JONES OPTIMIZATION
Questions
1. Where do you test?
2. How do you prioritize what you test?
3. Show me the tests?
4. Advice on starting a program or boosting
my current program?

22. DOW JONES OPTIMIZATION
How do you prioritize?
Expected
Uplift
Traffic to the
Experience
X
Value of KPI
X
Conversion
Rate
X
=
Level of
Effort
—
Expected Money Back to The Business

23. DOW JONES OPTIMIZATION
Low Hanging Fruit 101

24. DOW JONES OPTIMIZATION
Questions
1. Where do you test?
2. How do you prioritize what you test?
3. Show me the tests?
4. Advice on starting a program or boosting
my current program?

25. DOW JONES OPTIMIZATION
Testing Principles
Direct users to the
desired action
by making smarter,
data-driven design
choices.
Visual Cues
Choosing the right
words is important.
Don’t assume your
users know obvious
information about
your product.
Language
Tailoring calls to action
for the right time and
place will increase the
likelihood users take the
desired action.
Calls To Action
On any given page or
series of pages, make
the process of moving
through the page(s) fit
your objective.
Process Flow

26. DOW JONES OPTIMIZATION
Principles in Action: Language
9.77% Lift
Variant added simple
cancelation messaging

27. DOW JONES OPTIMIZATION
49% Lift
Principles in Action: Visual Cues
CONTROL VARIANT
- Center align sale price
- Bold Images

28. DOW JONES OPTIMIZATION
13% Lift
Principles in Action: Process Flow
CONTROL
20 Steps, 1 Min
VARIANT
15 Steps, 45 seconds

29. DOW JONES OPTIMIZATION
6% Lift
Principles in Action: Calls To Action
CONTROL VARIANT

30. DOW JONES OPTIMIZATION
But what happens after you get people through the door?
First 100 Days

31. DOW JONES OPTIMIZATION
Onboarding
One page that clicks out to actions

32. DOW JONES OPTIMIZATION
Onboarding
One page that sends you
down the page to complete the actions

33. DOW JONES OPTIMIZATION
Onboarding
Immediately after you finish your purchase!

34. DOW JONES OPTIMIZATION
Onboarding
What if we mimic the experience you have when you’re checking out….

35. DOW JONES OPTIMIZATION
Onboarding
And apply our our design best practices?

36. DOW JONES OPTIMIZATION
175%
Increase
333%
Increase
Onboarding Results
Member Actions During
Onboarding
FY17 FY19
Downloading our App 3% 13%
Signing up for newsletters 4% 11%

37. DOW JONES OPTIMIZATION
Questions
1. Where do you test?
2. How do you prioritize what you test?
3. Show me the tests?
4. Advice on starting a program or boosting
my current program?

38. DOW JONES OPTIMIZATION
How to Make Everyone Love Testing in Your Org
Prove your value
by grabbing the
low hanging fruit
Use data to stay
objective and to
ruthlessly prioritize
Make friends!

39. Optimizing Membership
at WSJ