1.
Congratulations to our
Outperform Award Winners!
Most Dramatic
Business Impact
Most Customer-
Obsessed Company
Culture
Most Transformative
Innovation
Most Inspiring Social
Impact

2.
2
 Welcome to Day 2 of Opticon!
 Sessions today will be recorded and will be available after
Opticon
 Share your feedback and rate sessions on the mobile app

3.
Will Close
Director of Digital Experiments, Nike
Alex Alwan
Sr. Product Manager Optimization,
Fox Networks
From iOS to TiVo
In-App Digital experience testing

4.
Will Close
Director of Digital Experiments, Nike
Running Digital Experiments at
Nike

5.
5
At Nike: Making Sport a Daily Habit
 Iconic, inspirational shoe & sportswear brand.
 Experimentation and athlete feedback has always been critical to Nike’s
product development process (and still is today with LeBron, KD...)
Founder Bill Bowerman was the Men’s Track Coach of the University of Oregon. He experimented with the
design of footwear for athletes – building shoes to the exact specifications of his runners.

6.
6
2015: Will joined Nike, to start a “Test & Learn” culture
 Sports brand, with a culture of winning.
 One of the world’s strongest brands.
 Strong intuition (art) has led decision-making at Nike for years.
 With unprecedented success, globally.
 Applying consumer data (science) was a new concept to digital product owners.
 Analytics systems in particular were still immature.
 Custom in-house, JavaScript testing solution on Nike.com.
 4 person team was dedicated to running Nike.com tests only.
With a technology background (Microsoft, Xbox, Dell) and specifically in product management, web
development, Analytics and A/B Testing.

7.
7
Mobile Cart: Primary button (Checkout) is out of view
Move Checkout button to top & make it “sticky”

8.
8
Mobile Cart: Primary button (Checkout) is out of view
Move Checkout button to top & make it “sticky”
% to Checkout = +5.5% lift
Multi-item Conversion = +5.67% lift

9.
9
In Checkout: Using a simple consumer-value proposition
Membership value proposition was lengthy marketing-speak. In Checkout, most consumers
want to quickly Place their Order, not read.
Selection of Membership Checkbox
= +172% lift
New NIKE Members = +30.3% lift

10.
10
Mobile PDP: How to best show multiple related products?
Static 2x2 layout vs. carousel vs. 2x2 layout with pagination.
Related Usage = +14.4% lift
Revenues from Related = +97% lift

11.
11
Small, centralized team executing tests
 4 team members.
 Ran about 80-90 tests a year.
 Technically constrained, front-end only.
 Lots of low-hanging fruit.
 Examples showed big impact.
Using a home-grown Nike.com testing tool, we showed huge impacts from very small UX
changes and leadership recognized the value of A/B testing.

12.
“
”
macklemore
The greats weren’t GREAT because at
birth they could paint.
The greats were great because they
PAINT A LOT.

13.
13
2018: Nike deploys Optimizely X Full Stack to scale
To enable experimentation from any connected, digital experience – Nike adopts the Full
Stack capability across the enterprise.

14.
14
 Enable Testing by All Product Teams
 Critical to Product Planning
 Informs key roadmap priorities
 Launch + product KPIs increase
 Features & experiments
 Create 3-4 options, monitor results
 All tests run by 1 small, central team
 Behavior tagging = one of many goals
 Engagement data = nice-to-have
 Success = the product launches
 We launch = Features only
 We prototype & design = 1 version
Before Today

15.
1515
No technical limits
 Support for Client-side &
Server-side experiments
 Built for performance
Easy to integrate
 Lots of example code to
leverage.
 Developer documentation
was easy to follow.
 Integration was quick.
Increase Test volume
 16 Digital experience
squads are integrated &
running experiments.
 Concurrent Test volume
has grown 7x – from
approx. 5 to 35.
Changing norms
 Growing organic interest in
“Test & Learn” capability.
 Leadership is requiring all
teams to experiment.
 North-star goal: All new
UX changes in Nike Digital
experiences are rolled out
via Feature Tests.
Test from anywhere
Benefits to-date from federating A/B Test execution to all Nike digital experience squads, via
Optimizely X Full Stack.

16.
16
Recent Test Results (1 of 3)
Subhead

17.
17
Recent Test Results (1 of 3)
Subhead

18.
18

19.
19

20.
20

21.
21

22.
22
Visibility is key
 Creating visibility across the
entire Nike program is
helpful for leadership.
 Many different teams have
different ways to work.
Standardizing where
appropriate creates
efficiencies.
 Communicate in multiple
formats & cadences (Email,
Meetings, Slack..)
Bold ideas matter
 Test features that are going
to be released anyway.
 Good or Bad, Bold ideas
help illustrate a learning –
and get the audience’s
attention (story).
 Engage your Design & UX
teams early and often – to
take more risk.
Build a community
 Enable a broad group of
“Test & Learn” teammates.
 Collaborate frequently with
this group & make sure they
are supported.
 Use the Voting system for
Social feedback on tests.
 Give them the Stage &
Forum to Share.
Growing up – culture insights
During this maturation process, we recognized a few insights to “growing up”.

23.
23
Optimizely Teams + Slack
 Slack integration for
Optimizely Teams & Program
Management is an easy
method to make intake of
ideas easy across the
company.
 /Optimizely.idea [Idea Name]
Feature Tests vs. A/B
 There are a lot of benefits to
planning your Mobile Apps
in a modular way – using
Optimizely Features.
 All Mobile App teams are
embracing Feature Tests.
Tell a Story
 Don’t hide behind statistics
and data points.
 Testing is about validating a
hypothesis & learning.
 Share a valuable, insightful
story. Simplify.
 Use the thumb.
Growing up – culture tips & tricks
During this maturation process, we recognized a few insights to “growing up”.

24.
Alex Alwan
Sr. Product Manager, Optimization
Running Digital
Experiments at Fox

25.
Alex Alwan
Sr. Product Manager, Product Strategy
Product Management is an art
Experimentation is not a choice
Experimentation is a tactic not a goal

26.
Optimization Team
From 1 to 2 to 3 to 6 to 4 to 2 people
From “Let’s give AB testing a try” to “Why the heck we didn’t AB test that?”
From “dunno man, that looks complicated” to “Just follow these steps”

27.
Experimentation Purposes
Improvement Testing
Discovery Testing
Feature Testing
Great to achieve OKRs
”Splitting sign-up and payment screens will improve
conversion rate”
Great to understand what’s going on, then improve
“What would happen if we remove Shows Favoriting from
onboarding?”
Great to validate/measure features’ impact
“What’s the impact of live streaming on VOD
viewership?”

28.
Scaling Is Challenging
It’s hard to change people
New Skills = New Responsibilities
http://millhoppervet.com
Someone: “Why AB test? Just launch and see
what happens”
Me:
Use success stories
Convince the leaders first

29.
Product Owner vs. Experiment Owner
Responsible
Accountable
Consulted
Informed
Scaling Is Challenging

30.
Scaling Is Challenging
“But.. it’s nice and warm
inside the box”
Optimization is a tactic not a goal
Focus on producing learnings not
only results
pbs.org

31.
Scaling Is Challenging
When there’s no big goal to inspire
ipwatchdog.com
You see the light at the end of the tunnel?
Optimization should serve the
company’s OKRs

32.
Scaling Tips
Start small but ask big questions
Quantity first, quality later
Integrate Data Tracking and A/B Testing systems
Center Of Excellence is NOT a gatekeeper

33.
From Web to Mobile/tvOS
Web: Extensions Full Stack: Variables
High Flexibility Specific Functionality

34.
From Web to Mobile/tvOS
More Content Starts From Redesigned
Collection
+31%

35.
From Web to Mobile/tvOS
Add-To-Favorite Rate
+15%
Push Opt-in Rate
+60%

36.
Tips For Mobile/tvOS Experimentation
Create a useless variable for testing purposes
Focus on generic capabilities to re-use
Don’t break the app, build feature flags

37.
Feature Flags
Takes one out of three possible values:
“ON”: Display feature/design to all users
“OFF: Don’t display feature/design at all
“OPTIMIZELY”: Wait for Optimizely's decision
If Invalid, fall-back to “OFF”

38.
What’s Next?
Optimization As Part Of Product Strategy

39.
Company OKR
Product OKRProduct OKR
Kye Result Kye Result Kye Result Kye Result
Initiative Initiative Initiative Initiative Initiative
Hypotheses Hypotheses
Where do we need to go?

40.
Company OKR
Product OKRProduct OKR
Key Result Key Result Key Result Key Result
Initiative Initiative Initiative Initiative Initiative
Hypotheses Hypotheses
How do we know we’re getting there?

41.
Company OKR
Product OKRProduct OKR
Key Result Key Result Key Result Key Result
Initiative Initiative Initiative Initiative Initiative
Experiments Experiments
What will we do to get there?

42.
Company OKR
Product OKRProduct OKR
Key Result Key Result Key Result Key Result
Initiative Initiative Initiative Initiative Initiative
Experiments ExperimentsExperiments Experiments
How can we move fast?

43.
From OKRs to Initiatives,
and beyond!

44.
How to define and plan initiatives?
Find Areas of
Improvement
Design Variations
Create KPIs
Reports
Define
Hypotheses
Plan Features as
Experiments
Track Key
Results

45.
Create KPIs
Reports
2%
Visit -> Subscribe
3 Months
90% Tipping Point
10%
Trial Cancelation
Visitor to Subscriber Funnel By Day/Hour
Visitor to Subscriber Funnel By User Traits
Key Result Quality Metrics Deeper Analysis

46.
Find Areas of
Improvement
Traffic source User Traits User Behavior
Drops and Opportunities
Referral traffic from
politics news on
foxnews.com are 60%
more likely to convert
Safari Mobile users
drop by 85% on funnel’s
2nd step
Users who watch on
>1 platform are 90%
less likely to cancel
subscription

47.
Define
Hypotheses
User Behavior
Users who watch on
>1 platform are 90%
less likely to cancel
subscription
Highlight available platforms on the marketing
landing page
Educate users on other available platforms
through email/push messages
Display banner ads
Fix iOS deep links

48.
Find Areas of
Improvement
Design Variations
Create KPIs
Reports
Define
Hypotheses
Plan Features as
Experiments
Track Key
Results
Achieve Growth
Hit OKRs
Cultivate a data-driven culture
Empower product managers to own their OKRs
Improve efficiency
Transform features into experiments
Goals

49.
Q&A

50.
Thank you!

51.
 Head to the Expo Hall for a quick break and say hello to our
sponsors
 Up next...learn how to measure experiment impact with
Optimizely's Co-founder Pete Koomen!