Watch Dan Ross, Managing Director for Optimizely ANZ in our latest webinar from the Experimentation Insights Tour -- "7 Habits of Highly Effective Personalisation Organisations” Watch the webinar here: https://optimizely.wistia.com/medias/cun66mnkwt Take Optimizely's Maturity Assessment here: https://www.optimizely.com/maturity-model/ DESCRIPTION: Create a data-driven culture and affect business decisions at the broader company level. When most people think of experimentation or testing, they think of sales and marketing. However, to do real customer experience optimisation, you need to think about all the ways your customers are interacting with you. The right mix to support building your programme into a centre of excellence is critical: you need a team that helps create a data-driven culture. Watch this webinar so you can: * Think more deeply about the future of your program and the makeup of your team * Consider which hard and soft skill sets your testing organisation needs * Build a well-rounded optimisation team that is visible, sustainable, and efficient About Optimizely Optimizely is the world's leading experimentation platform, enabling businesses to deliver continuous experimentation and personalisation across websites, mobile apps and connected devices. Optimizely enables businesses to experiment deeply into their technology stack and broadly across the entire customer experience. The platform’s ease of use and speed of deployment empower organisations to create and run bold experiments that help them make data-driven decisions and grow faster. To date, marketers, developers and product managers have delivered over 700 billion experiences tailored to the needs of their customers. Optimizely’s global client base includes Atlassian, eBay, Fox, IBM, The New York Times, LendingClub, Hotwire, Microsoft and many more leading businesses. To learn more about customer experience optimisation, visit optimizely.com

1. 7 Habits of Highly
Effective Personalisation
Organisations
Dan Ross
Managing Director, Optimizely ANZ
dan.ross@optimizely.com
linkedin.com/in/danross9

2. By the end of today’s session, you will
have learned:
• 7 tactics you can employ to advance your personalisation practice
• The importance of experimentation in your program’s maturity
• Different technology and processes available to support your journey

3. Optimization Is A Journey

10. Skeptics still openly wonder if we can continue to deliver on this journalistic
mission, given the seeming mismatch between the economics of news media and the
scale of our operations. They suggest the days when a media company can fund a
big, ambitious newsroom are over.
This is why we are setting the goal of doubling our digital revenues over the next five
years, to reach more than $800 million in digital-only revenue by 2020.
Our Path Forward
October 7, 2015

11. Our overarching aspiration is to cultivate another generation of readers who can’t
imagine a day without The New York Times. Our first two million subscribers —
including our more than one million newspaper subscribers — grew up with The
New York Times spread out over their kitchen tables. The next million must be
fought for and won over with The Times on their phones.
The sustainable path to long-term revenue growth requires that we always prioritize
user experience and the needs of our customers over hitting quarterly revenue
targets. These deep reader relationships are our most valuable asset.
Our Path Forward
October 7, 2015

13. +155%

14. opticon2017
See this island through an
artist’s eyes
Explore 19th-century huts
in rural Japan
Copenhagen: the new global
hub for natural wines
Michigan: America’s new
architecture hub?
Visit Slovenia’s glowing
capital city, Ljubljana
Dine on modern camp
food at an Oregon lodge
Campsite booked? Not
anymore with online
reservations
Buckling up for a bumpy
ride: handling extreme
weather

18. Every article is
an experiment
Every offer is
an experiment
Every product launch
is an experiment

19. HABIT 1
Create a
Vision

20. HABIT 2
Experimentation Maturity

21. ARE WE READY TO STEP FORWARD?
FORRESTER’S PERSPECTIVE
*Source: Forrester’s Q3 2015 Global Online Testing Platform Customer Online Survey
Dimensions
of continuous
optimization
Online testing is applied
mostly to the “explore”
and “buy” phases of the
customer life cycle
Online testing is
applied
mostly to websites
Online testing practices are
mostly executing only A/B
tests
A minority (i.e., 30% or fewer) of
customer interactions are included in
online testing*
Opportunity for improvement

22. ARE WE READY TO STEP FORWARD?
FORRESTER’S PERSPECTIVE
*Source: Forrester’s Q3 2015 Global Online Testing Platform Customer Online Survey
Dimensions
of continuous
optimization
Online testing is applied
mostly to the “explore”
and “buy” phases of the
customer life cycle
Online testing is
applied
mostly to websites
Online testing practices are
mostly executing only A/B
tests
A minority (i.e., 30% or fewer) of
customer interactions are included in
online testing*
Opportunity for improvement
MATURE OPTIMISATION PROGRAMS
• Do more complicated tests than A/B
• Test through more than just a few pages
• Are segmenting analytics

23. CULTURE OF EXPERIMENTATION

24. 0
10
100
1000
VELOCITY
MATURITY
10000

25. VELOCITY
EXPERIMENTATION
HERO
MATURITY
0
10
100
1000
10000

26. VELOCITY
EXPERIMENTATION
PROGRAM
EXPERIMENTATION
HERO
MATURITY
0
10
100
1000
10000

27. VELOCITY
EXPERIMENTATION
HERO
EXPERIMENTATION
PROGRAM
CULTURE OF
EXPERIMENTATION
MATURITY
0
10
100
1000
10000

28. Experimentation Maturity Model

29. LEADING
INDICATORS
Experimentation Success
VELOCITY
The volume of experiments being run, the
reach of personalisation campaigns.
Throughput:
# of experiments per property per
month/week.
AGILITY
The degree that the experimentation
program acts on results.
Iteration:
The % of experiments put into production
and iterated upon.
EFFICIENCY
The efficiency that experiments get
through production cycle
Drag:
Average hours spent
redeveloping due to QA
QUALITY
The average likelihood that an
experiment will produce
business impact
Impact Rate:
% generating meaningful result
OPERATIONAL METRICS FOR EXPERIMENTATION

30. LEADING
INDICATORS
Experimentation Success
VELOCITY
The volume of experiments being ran,
the reach of personalization
campaigns.
Throughput:
# of experiments per property per
month/week.
AGILITY
The degree that the experimentation
program acts on results.
Iteration:
The % of experiments put into
production and iterated upon.
EFFICIENCY
The efficiency that experiments get
through production cycle
Drag:
Average hours spent
redeveloping due to QA
QUALITY
The average likelihood that an
experiment will produce
business impact
Impact Rate:
% generating meaningful result
OPERATIONAL METRICS FOR EXPERIMENTATION
MATURE EXPERIMENTATION PROGRAMS
• Are high throughput
• Develop efficiently (business as usual!)
• Get consistent wins

31. VELOCITY
EXPERIMENTATION
HERO
EXPERIMENTATION
PROGRAM
CULTURE OF
EXPERIMENTATION
MATURITY
0
10
100
1000
10000

32. Habit 2 Takeaway:
Experimentation
Maturity

33. HABIT 3
Assemble
Your
Dream Team

34. DISCOVERY IMPLEMENTATION PLANNING PRODUCTION REPORTING
PERSONALISATION PLAYBOOK
END-END PROCESS + MILESTONES

35. CORE PERSONALISATION TEAM
SKILLSETS & TEAM ROLE
Executive Sponsor Project Manager Technical Lead Developer Content

36. B U I L D I N G A P R O G R A M I S H A R D

37. I D E A T I O N & P R I O R I T I S A T I O N

39. C O L L A B O R A T I O N &
O V E R S I G H T

40. HYPOTHESIS
CREATIVE
DEVELOPMENT
SETUP &
QA
TESTING
ANALYSIS
SHARE

42. K N O W L E D G E & R E P O R T I N G

45. IDEATION
Democratise
ideation across
your organisation
COLLABORATION
Bring teams together
to collaborate on
experiments
KNOWLEDGE
SHARE
Document and
share learnings &
detailed analysis
PROGRAM
REPORTING
Track success with
program level
reporting

46. Integrated hub for capturing ideas and
enabling collaboration across your
organisation

47. HYPOTHESIS CREATIVE DEVELOPMENT SETUP & QA TESTING ANALYSIS SHARE

53. 3XTesting Velocityimprove collaboration and

54. Habit 3 Takeaway:
Assemble Your
Dream Team

55. HABIT 4
Enrich Your
Perspective

56. YOUR
Team
Status Quo:
Tech: current capabilities and limitations
People and Process
Audience Strategy
Look Internally
Your Systems
Your Analytics
Your Personas
Your Competitors
Your Strategy
Future States:
Potential capabilities
Audience Proposal
Use Cases
YOUR TEAM’S TASK
GATHER INTELLIGENCE: LOOK INWARD
1

57. YOUR
Team
Validation and
Alternate Perspectives:
Tech: Potential capabilities
People and Process: Alternate Approaches
Audience Strategy
Consult
External Experts
Vendors
Consultants
Agencies
Analyst Reports
Future States:
Potential capabilities
Audience Proposal
Use Cases
2
YOUR TEAM’S TASK
GATHER INTELLIGENCE: LOOK OUTWARD

58. YOUR
Team
Status Quo:
Tech: current capabilities and limitations
People and Process
Audience Strategy
Validation and
Alternate Perspectives:
Tech: Potential capabilities
People and Process: Alternate Approaches
Audience Strategy
Consult
External Experts
Vendors
Consultants
Agencies
Analyst Reports
Look Internally
Your Systems
Your Analytics
Your Personas
Your Competitors
Your Strategy
Future States:
Potential capabilities
Audience Proposal
Use Cases
YOUR
Brief
3
YOUR TEAM’S TASK
GATHER INTELLIGENCE: CONSOLIDATE

59. YOUR
Team
Status Quo
Validation and
Alternate Perspectives
Consult
External
Experts
Look
Internally
Future States
YOUR
Brief
3
YOUR TEAM’S TASK
GATHER INTELLIGENCE: CONSOLIDATE

60. Habit 4 Takeaway:
Enrich Your
Perspective

61. HABIT 5
Create Your
Audience
Strategy

62. Recency & Frequency
Cross-sells & Up-sells
Value Propositions
START BY EXAMINING YOUR BUSINESS
STRATEGY
Propensity Models
Customer Journey Model
Price Sensitivity

63. LAYER ON MORE AUDIENCES
LEFT- & RIGHT-BRAIN
PERSONAS ANALYTICS

64. WHAT TECHNICAL SIGNALS CAN WE LEVERAGE?
CONNECT CONCEPT TO TACTIC
Viewed 2 Products, Didn’t Buy
Keyword contains ‘discount’
Most frequently viewed
category
DMP + Uploaded Lists
Abandoned Checkout
Data Warehouse (Customer
ID
Geo-Targeting)
Came from Ad Campign = Gift
Technical
Signal Consideration-Stage
Wants a discount
Preference for a specific
product type
High-Propensity
Needs a push
VIP Member
Urban Location
Shopping for a Gift
Audience
Characteristic

65. PRIORITISE, PRIORITISE, PRIORITISE
PURSUE VARIETY OF AUDIENCES, MAXIMISE REACH/QUALITY
Obvious Need
Large
Need for Creativity
Granular
Visitor Cohort; New,
Returning, Active, Loyal
Large Geos; Coastal
Urban, State, Key Cities
Browsed Twice;
Product Category
Past Purchasers
Second Priority

66. Habit 5 Takeaway:
Create Your
Audience Strategy

67. HABIT 6
Unify

68. View of the
Customer
CONNECT YOUR DATA
HOUSEKEEPING BEFORE TECHNOLOGY

69. Everyone has to work together
for personalisation to work for you

70. Habit 6 Takeaway:
Unify

71. HABIT 7
Crawl Before
You Walk

72. PHASED INTEGRATION OF PERSONALISATION
CRAWL, WALK, RUN
0-12 weeks
Build
Phase
1
months 12-24
Build
Phase
3Build
Phase
2
months 3-12

73. Platform Implementation
Simple Audiences
Starter Campaigns,
Limited Integration of
Testing + Personalisation
Phase 2 Planning
REACH: 0-15%
PAGES: 1-3; only most critical ROI points
#
CAMPAIGNS: 2-5
AUDIENCES: Natively available, simple, large, simple conditions;
Metro, Single Behaviours
TACTICS: Modules (lightboxes), image swaps, little testing
0-12 weeks
Buil
d
Phase
1
PHASED INTEGRATION OF PERSONALISATION
CRAWL, WALK, RUN

74. Integration with 1st & 3rd
Party Data
More Campaigns
Integration of testing &
Personalisation workflows
More advanced use cases
Phase 3 Planning
Buil
d
Phase
2
months 3-12
PHASED INTEGRATION OF PERSONALISATION
CRAWL, WALK, RUN
REACH: 30-60%
PAGES: Multiple campaign/audiences on top ROI pages
#
CAMPAIGNS: 10-20 ongoing campaigns
AUDIENCES: Target intersecting audiences, 3rd & 1st party data
used, more and complex behaviours
TACTICS: Experiments drive campaign execution and iteration

75. Full system integration
Ongoing improvement
New audience strategy
Use cases continually iterated
Web personalisation data feeds
email and ad deployment
Buil
d
Phase
3
months 12-24
PHASED INTEGRATION OF PERSONALISATION
CRAWL, WALK, RUN
REACH: 75-100%
PAGES: Most pages, multiple elements per page
#
CAMPAIGNS: 25+ ongoing personalisation campaigns iterated on
AUDIENCES: Old audiences iterated, new granular audiences
TACTICS: Fully expressive strategy

76. Habit 7 Takeaway:
Crawl Before You Walk

77. Experimentation Maturity
Create a Vision
Assemble Your Dream Team
Enrich Your Perspective
Create Your Audience Strategy
Unify
Crawl Before You Walk

78. Experimentation Maturity Model

79. Dan Ross
Managing Director, Optimizely ANZ
dan.ross@optimizely.com
linkedin.com/in/danross9
T H A N K Y O U