In this talk Mark Halvorson will explore how Atlassian, the creator of JIRA and Confluence, has used Magnolia CMS and other tools to operationalize A/B testing to continually optimize Atlassian's website.

2. MARK HALVORSON • HEAD OF INTERACTIVE • ATLASSIAN SOFTWARE • @halv0112
A/B Testing
HOW TO BREAK THINGS AND FAIL FAST…
WITHOUT BREAKING THINGS

3. About Atlassian
• Products include: JIRA, Confluence, Bitbucket, HipChat…
• Founded in 2002
• Over 35,000 Customers
• Over 700 employees globally
• No sales people
!
!
!
We h e l p g r e a t t e a m s b u i l d b e t t e r s o f t w a r e , t o g e t h e r.

5. Think big, act small, fail fast, learn rapidly.
"LEAN SOFTWARE DEVELOPMENT” BY MARY AND TOM POPPENDIECK (2003)
”“

6. Atlassian A/B Testing is cross functional
• Email
• Product Management
• Customer Platform
• Customer Advocates
• Internal Systems
!
!
!
C e n t r a l G r o w t h H a c k i n g Te a m t o s u p p o r t :
• Leads
• Product Marketing
• Support
• Bitbucket
• HipChat

9. SCHEDULE TO AVOID CONFLICTS
BUILD OUT GOOD CANDIDATES
TRIAGE BASED ON POTENTIAL IMPACT
COLLECT GREAT IDEAS
Our Process
ANALYZE RESULTS
RUN EXPERIMENTS

10. SCHEDULE TO AVOID CONFLICTS
BUILD OUT GOOD CANDIDATES
TRIAGE BASED ON POTENTIAL IMPACT
COLLECT GREAT IDEAS
Our Process
ANALYZE RESULTS
RUN EXPERIMENTS

11. Where do experiment ideas come
from?

12. Anywhere.

13. What are you are trying to
improve?

14. Blimp Dashboard

15. JIRA – Ticket

16. • Pebbles
• Change in copy or CTA presentation
• Rocks
• New page (e.g. no Dev Tools family page)
• Boulders
• Simplified order form.
!
!
!
Types of tests

17. Examples

18. Reduce Friction

19. Increase Visibility

20. Test drive campaigns

21. Simplify

22. SCHEDULE TO AVOID CONFLICTS
BUILD OUT GOOD CANDIDATES
TRIAGE BASED ON POTENTIAL IMPACT
COLLECT GREAT IDEAS
Our Process
ANALYZE RESULTS
RUN EXPERIMENTS

23. What are you are trying to
improve?

24. Blimp Dashboard

25. JIRA – Prioritized Backlog

26. JIRA – Ticket

27. JIRA – Ticket

28. Experiment Illuminati

29. SCHEDULE TO AVOID CONFLICTS
BUILD OUT GOOD CANDIDATES
TRIAGE BASED ON POTENTIAL IMPACT
COLLECT GREAT IDEAS
Our Process
ANALYZE RESULTS
RUN EXPERIMENTS

30. Optimizely – Build

31. Optimizely – Build

32. Prepare for results.

33. Confluence – Corresponding Test Page

34. Confluence – Corresponding Test Page

35. Confluence – Corresponding Test Page

36. Confluence – Corresponding Test Page

37. Confluence – Corresponding Test Page

38. Confluence – Corresponding Test Page

39. Confluence – Corresponding Test Page

40. Confluence – Corresponding Test Page

41. Confluence – Corresponding Test Page

42. Confluence – Corresponding Test Page

43. Confluence – Corresponding Test Page

44. Confluence – Corresponding Test Page

45. SCHEDULE TO AVOID CONFLICTS
BUILD OUT GOOD CANDIDATES
TRIAGE BASED ON POTENTIAL IMPACT
COLLECT GREAT IDEAS
Our Process
ANALYZE RESULTS
RUN EXPERIMENTS

46. JIRA – Prioritized Backlog

47. SCHEDULE TO AVOID CONFLICTS
BUILD OUT GOOD CANDIDATES
TRIAGE BASED ON POTENTIAL IMPACT
COLLECT GREAT IDEAS
Our Process
ANALYZE RESULTS
RUN EXPERIMENTS

48. Optimizely – Start Experiment

49. Optimizely – Project Code

50. Magnolia CMS – Include Optimizely Checkbox

51. Magnolia CMS – Include Optimizely Checkbox

52. Confluence – Corresponding Test Page

53. SCHEDULE TO AVOID CONFLICTS
BUILD OUT GOOD CANDIDATES
TRIAGE BASED ON POTENTIAL IMPACT
COLLECT GREAT IDEAS
Our Process
ANALYZE RESULTS
RUN EXPERIMENTS

54. Optimizely – Results

55. What are you are trying to
improve?

56. Blimp Dashboard

57. Optimizely – Results

58. Confluence – Corresponding Test Page

59. Confluence – Results Archive

60. • Ideas are everywhere
• Everything affecting conversion is tested
• No traffic wasted
• Be aggressive
• Regular cadence
• Never run a test that you wouldn’t want to win
Our Test Philosophy

61. • Ideas are everywhere collect as many as you can, prioritize based on
impact.
• Use ticket number everywhere so everyone is referring to the same
test.
• Involve stakeholders and developers… and get approvals.
Key Take Aways

63. Thank you!
MARK HALVORSON • HEAD OF INTERACTIVE • ATLASSIAN SOFTWARE • @halv0112