Today, Atlassian runs experiments and personalization campaigns across all their products, campaigns, and commerce. But it wasn’t always like this. Atlassian started small and at their own pace, grew across the enterprise. Join Tom Tsao, the head of eCommerce at Atlassian, to learn how they evolved, from the origins of their program, to their process and team structure. -Learn how to build a program around digital experience optimization using Optimizely. -Get advanced use cases that deliver business impact. -Learn what metrics are important in measuring the growth of your program.

1. Better Decisions with
Experimentation
Tom Tsao
Atlasssian
LESSONS FROM THE GROWTH OF
ATLASSIAN’S EXPERIMENTATION TEAM
Wes McCabe
Optimizely

2. OPTIMIZELY.COM/DIGITAL-CHANGE-AGENTS

3. Housekeeping
● We are recording this webinar
● You’ll receive the recording and slides
● We’ll answer all questions at the end

4. Agenda
● Introduction
● What is Optimizely?
● Atlassian’s Experimentation Program
● Q&A

5. Better Decisions with
Experimentation
Tom Tsao
Atlassian
LESSONS FROM THE GROWTH OF
ATLASSIAN’S EXPERIMENTATION TEAM
Wes McCabe
Optimizely

6. Digital Experience Optimization:
Digital Products, Commerce & Campaigns
Up to 5X Increase in Yield:
Revenue, Share of wallet, Funnel conversion, Risk
mitigation, Ops efficiency
Partner of Choice:
Work with leading global enterprises & “digital
disruptors” including 26 of F100
OUR COMPANY
Digital Experimentation Platform:
Next gen “Test and Learn” System
Replaces Digital Guesswork:
with Evidence-based optimization
Speeds Innovation & Optimization:
Single platform for marketing & product teams
Best in class stats & machine learning
Consumer grade usability
Enterprise program management & prof. services
2
OUR SOLUTION

7. 5
D I G I TA L E X P E R I M E N TAT I O N
EMBRACE
OR PERISH
10X MORE
SCALE
THE GREATEST
ROI IN SaaS
THE NEXT
BIG THING
Our Point of View

8. “Our success is a function of
how many experiments
we do per year, per month,
per week, per day.”
“Instead of saying ‘I have an
idea,’ what if you said
‘I have a new hypothesis,
let’s go test it.’
“Our company culture
encourages
experimentation and
free flow of ideas.”
3
“One of the things I’m most
proud of, and I think what is
the key to our success, is
this testing framework
we’ve built.”
Experimentation is the
Next Great Business
Transformation
Jeff Bezos Larry Page
Mark Zuckerberg
Satya Nadella
The Surprising
Power of Online
Experiments

9. 10x more experiments
Consumer-grade usability
Open data integration
Maximum yield of business value
UX and feature-level experiments
and personalization at every digital
touch point
Enterprise-wide
management & governance
Captures, governs and shares
ideation, analyses & results
Optimizely X Unlocks the Experimentation Best Practices
of the World’s Greatest Digital Companies
World’s most trusted outcomes
BestinclassStatsEngine
FasttimetoresultsviaML
Accelerates digital innovation
Speeds dev ops & deployment
De-risks continuous feature delivery
Ensures success of new features
Unifying flagging & experiments
enables controlled testing of new
features while maintaining high
performance
Ideate
Manage
StoreGovern
Analyze
Share
Open Data
Integration
Security &
compliance
Stats
Engine
Stats
Accelerator
Consumer-
grade
usability
APIs &
Developer
Tools
Feature
Flags
Open source
SDKs
X-Channel
Full Stack
Experimentation
Personali-
zation
Recommen-
dations
Web
Experimentation
PRODUCTS
COMMERCE
CAMPAIGNS

10. B U S I N E S S V A L U E
VELOCITY/VOLUME
LEVEL 1
Executional
Start
LEVEL 2
Foundational
Growth
LEVEL 3
Cross-functional
Advancement
LEVEL 4
Operational
Excellence
LEVEL 5
Culture of
Experimentation
10
Our Products and Services Take
You on Your Experimentation Journey

11. TOM TSAO | HEAD OF ECOMMERCE | ATLASSIAN
Better Decisions with Experimentation
Lessons from the growth of Atlassian’s experimentation team

12. We believe behind every great human
achievement, there is a team. Our mission
is to unleash the potential of every team.
OUR MISSION

13. Team
Work
Organize Create SupportDiscuss
WHAT WE DO

14. Agenda
History of experimentation
Case studies
What’s next

15. History of experimentation

16. B U S I N E S S V A L U E
VELOCITY/VOLUME
LEVEL 1
Executional
Start
LEVEL 2
Foundational
Growth
LEVEL 3
Cross-functional
Advancement
LEVEL 4
Operational
Excellence
LEVEL 5
Culture of
Experimentation
10
Optimizely’s Products and Services Take
You on Your Experimentation Journey

17. B U S I N E S S V A L U E
VELOCITY/VOLUME
LEVEL 1
Executional
Start
LEVEL 2
Foundational
Growth
LEVEL 3
Cross-functional
Advancement
LEVEL 4
Operational
Excellence
LEVEL 5
Culture of
Experimentation
In Product
Experiments
Identified users
Controlled environment
Easier to prove value
Evaluation
Experiments
Remove credit
card for trial
Web
Experiments
Anonymous visitors
Multiple channels
Campaign
Experiments
Optimizing spend
throughout the funnel
ATLASSIAN’S EXPERIMENTATION JOURNEY
2014
2015
2016
2017

18. Experiments help us
make decisions
WHAT WE BELIEVE

19. Idea A vs B vs
C
Not HIPPO
Roadmap
decisions
Feature “spoofing”
Learnings
Listening to our
customers
Learnings =
Success + Failures
Biz Decisions
Remove credit card for
trials
Experiments Decisions

20. Case studies

21. PERSONALIZED HEADLINES DRIVE CONVERSIONS
Hypothesis
Headlines
personalized to
visitors’ company can
drive conversions
Result
+20% increase in
Enterprise contact
form submissions

22. INDUSTRY PERSONALIZATION DRIVES VISITORS TO RELEVANT CONTENT
BB is marketed to
“professional teams”
control

23. INDUSTRY PERSONALIZATION DRIVES VISITORS TO RELEVANT CONTENT
test
Hypothesis
Promoting BB
UNLIMITED free
repositories to EDU
audience, would
increase signups

24. INDUSTRY PERSONALIZATION DRIVES VISITORS TO RELEVANT CONTENT
test
Result
INCONCLUSIVE
In this use case,
personalization did not
move the metric
Hypothesis
Promoting BB
UNLIMITED free
repositories to EDU
audience, would
increase signups

25. An advanced case study
Product Bundling

26. PRODUCT “BUNDLE” CHOICES FOR JIRA
control
Do visitors really
know what they
want?
Confluence

27. PRODUCT “BUNDLE” CHOICES FOR JIRA
Hypothesis
We believed that the
3rd product bundle
was chosen by most
visitors, because they
got “3” products for
FREE
Confluence
test

28. PRODUCT “BUNDLE” CHOICES FOR JIRA
test
Hypothesis
“Uninformed choice”
bundling

29. +167 -46
JSW JSD
PRODUCT “BUNDLE” CHOICES FOR JIRA
-266
Confluence
Signup
CONVERSION

30. +167 -46
JSW JSD
PRODUCT “BUNDLE” CHOICES FOR JIRA
-266
Confluence
Signup
CONVERSION
First Day
ACTIVATION
Week1 (Trial)
+110 +50 -145
+66 +8 -71

31. +167 -46
JSW JSD
PRODUCT “BUNDLE” CHOICES FOR JIRA
-266
Confluence
Signup
CONVERSION
First Day
ACTIVATION
Week1 (Trial)
Week2 (after trial)
RETENTION
+110 +50 -145
+66 +8 -71
+151 +11 +4

32. PRODUCT “BUNDLE” CHOICES FOR JIRA
Result
Bundling more products
yields higher initial sign-
ups,
but lower retention
Hypothesis
“Uninformed choice”
bundling

33. Optimize for
conversion
LEARNINGS

34. Optimize for
conversion
LEARNINGS
retention

35. Optimize for
conversion
LEARNINGS
retention (the right metric for us)

36. What’s next

37. After
Integrated Buyer Experience Team with a
culture of experimentation
Before
Fragmented, one-off experimentation
“heroes”
Exp team Web team Web Exp team

38. The right
culture
depends on
having the right
platform
Optimizely

39. Why Optimizely?
Key reasons we chose Optimizely
Better Performance Robust
APIs
Auto-Traffic
Allocation
X-Domain
instrumentation
DCP (dynamic
customer profile)
1 2 3 4 5

40. B U S I N E S S V A L U E
VELOCITY/VOLUME
LEVEL 1
Executional
Start
LEVEL 2
Foundational
Growth
LEVEL 3
Cross-functional
Advancement
LEVEL 4
Operational
Excellence
LEVEL 5
Culture of
Experimentation
ATLASSIAN’S EXPERIMENTATION JOURNEY
Culture of
Experimentation
Integrated Buyer
Experience Team
Optimizely X
2018
In Product
Experiments
Identified users
Controlled environment
Easier to prove value
Evaluation
Experiments
Remove credit
card for trial
Web
Experiments
Anonymous visitors
Multiple channels
Campaign
Experiments
Optimizing spend
throughout the funnel
2014
2015
2016
2017