The document outlines the evolution of a consumer experience team's approach to product testing and development, emphasizing the importance of data-driven decision-making, audience engagement, and iterative experimentation. It discusses challenges faced in optimizing content and advertising strategies, as well as the integration of cross-functional teams and consumer science in refining processes. Key future steps include enhancing mobile app testing, initiating complex editorial testing, and continuing to evolve user experiences through automated analysis and dynamic testing.

2. TO INFORM,
ENGAGE AND
EMPOWER
THE WORLD

3. CONSUMER
EXPERIENCE TEAM
PRODUCT ENGINEERING
CUSTOMER CAPABILITY TEAMS

4. CONSUMER
SCIENCE TEAM
IMPROVING OUR DECISIONS
Objective
Facts
What
Confirms
Your
Beliefs
What you see

5. OUR
EXPERIMENTATIO
N JOURNEY

6. X6
EARLY DAYS
•2014 Redesign Beta
- Simple redirect test
•Headline testing POC
- Optimizely
- Visual Revenue
•Optimizing template monetization
•Homepage zone arrangements
•Ad Hoc experiments
WE DIDN’T KNOW MUCH

7. X7
AWARENESS TURNING POINT
1.30%
-29.26%
0.80%
-47.30%
2.99%
-0.97% 0.26%
1.02%
Var.1: Replace “TOP STORIES” - Desktop Var.2: Replace “News and Buzz”- Desktop Var.3: Replace “More from CNN” - Desktop Mobile Web – Replace More from CNN
-20.38%
-1%
+34%
Overall Footer:
-61.43%
- 86%
+24%
+46%

8. X8
CHALLENGES WE FACED
•News cycle
- Audience mix changes with big news events
•Content matters and changes quickly
•Optimal layout depends on content
•Advertising business model centers on attention
- Proxy metrics to habit are hard to identify
•Every single interaction represents a micro-transaction
•No paid subscription means no big payback conversions
EASIER SAID THAN DONE

9. EDITORIAL TESTING
Leverage existing toolkit
Test programming
approaches
FOCUS
Quick turnaround
Large reach
Focused analysis
TEST TYPES

10. X10
•Newsletter acquisition
- Color and copy variations
- Segmented targeting
- Size variations
- Activation triggers
- Registration campaigns
- Custom code
- Color and copy variations
ACQUISITION TESTING
CRO
Lifecycle management
FOCUS
Quick turnaround
Large reach
Automated analysis
TEST TYPES

11. REVENUE OPTIMIZATION
Revenue modules
Templates
Ad Footprint
FOCUS
Longitudinal
Small reach
Deep analysis
TEST TYPES
•Optimize template monetization
- Custom code
- Placement on template
- Full downstream analysis
- Revenue impact analysis

12. PRODUCT TESTING
FOCUS
New Product features
UX changes
TEST TYPES
Longitudinal
Small reach
Deep analysis

13. HOW WE GOT
HERE

14. X14
ORGANIZATIONAL EVOLUTION
•Creation of Audience Development
• Focus on growth and habit
• Lifecycle management
•Consumer Science approach
- Road towards COE
•Product’s adoption of cross functional
team approach
•Build up of UX Research team
•Scaling the organization
CHANGING HOW TEAMS INTERACT

15. X15
CULTURAL EVOLUTION
• Discovery driven Product Development
• Right people, right problems
• Hiring the right team
• Audience segmentation awareness
• Cold, Casual, Core
• Audience metrics
• Building habit
• Measuring satisfaction
• Retention Proxy metrics
GROWTH OF PRODUCT AND ENGINEERING

16. X16
PROCESS
• Starting with what we know
• Data / observations / feedback
• Earlier involvement of
Consumer Science team
• Relentless focus on refining the
process
EMPHASIS ON TEST & LEARN
DATA
QUAL
TESTING
RELEAS
E
Validate
and repeat
QUANT
TESTING
“Test
against
Data”
“Test
against few
volunteers”
“Test at
scale”
Fidelity & effort

17. X17
TECHNOLOGY
• Optimizely Web
• Optimizely Full Stack
• Improved event instrumentation
• List Vars on Adobe
• Custom metrics on Optimizely
• Program documentation and
measurement
• Snowflake, Databricks, Tableau & Looker

18. X18
ENGINEERING
EXPERIMENTATION WITH CONFIDENCE

19. X19
ENGINEERING
RUTHLESS FOCUS

20. X20
ENGINEERING
ROLLING “CANARY” RELEASE

21. X21
7
APP TEST: “FOR YOU” PERSONALIZED FEED
Test VariationControl Preview

22. X22
5
APP TESTING
•Important to monitor adoption
rates before launching
experiment
•Launch to larger percentage of
app audience to reach
sufficient sample size faster
•Simultaneously test on mobile
web to help inform app testing
WHAT WE’VE LEARNED

23. X23
8
NEXT STEPS
•Continue optimizing iOS mobile app test
process
•Launch Android mobile app tests
•Evaluate OTT testing requirements
•Enable server-side testing on both web
and apps

24. X24
8
LONG TERM
•Dynamic experience testing (Server-side)
•Automated and faster analysis
•Complex editorial testing
•Cross platform experiments
•Continued evolution of user experience

25. X25
•Trustworthiness is fundamental
•Move fast and methodically
•Get involved from inception
•Leverage qualitative to refine
solutions before coding
•Define success before testing
•Communicate broadly
•Focus on process
KEY
LEARNINGS