This presentation was given by Mark Halvorson at the Optimizely Experience, Silicon Valley.

1. A/B Testing
How to break things and fail fast…
without breaking things
MARK HALVORSON • HEAD OF INTERACTIVE • ATLASSIAN SOFTWARE • @halv0112

2. About Atlassian
We help great teams build better software, together.
• Products include: JIRA, Confluence, Bitbucket, HipChat…
• Founded in 2002
• Over 37,000 Customers
• Over 700 employees globally
• No sales people
!
!
!

4. “ Think Big, Act Small, Fail Fast, Learn Rapidly.
”
"LEAN SOFTWARE DEVELOPMENT” BY MARY AND TOM POPPENDIECK (2003)

5. Atlassian A/B Testing is cross functional
Central “Growth Hacking” team to support:
• Email
• Product Management
• Customer Platform
• Customer Advocates
• Internal Systems
• Leads
• Product Marketing
• Support
• Bitbucket
• HipChat

8. Our Process
COLLECT GREAT IDEAS
TRIAGE BASED ON POTENTIAL IMPACT
BUILD OUT GOOD CANDIDATES
SCHEDULE TO AVOID CONFLICTS
RUN EXPERIMENTS
ANALYZE RESULTS

9. Our Process
COLLECT GREAT IDEAS
TRIAGE BASED ON POTENTIAL IMPACT
BUILD OUT GOOD CANDIDATES
SCHEDULE TO AVOID CONFLICTS
RUN EXPERIMENTS
ANALYZE RESULTS

10. Where do
experiment ideas
come from?

11. Anywhere.

12. What are you are
trying to improve?

13. Blimp Dashboard

14. JIRA – Ticket

15. Types of tests
• Pebbles
• Change in copy or CTA presentation
• Rocks
• New page (e.g. no Dev Tools family page)
• Boulders
• Simplified order form

16. Examples

17. Reduce Friction
A B

18. Increase Visibility
A B

19. Simplify
A B

20. Our Process
COLLECT GREAT IDEAS
TRIAGE BASED ON POTENTIAL IMPACT
BUILD OUT GOOD CANDIDATES
SCHEDULE TO AVOID CONFLICTS
RUN EXPERIMENTS
ANALYZE RESULTS

21. What are you are
trying to improve?

22. Blimp Dashboard

23. JIRA – Prioritized Backlog

24. JIRA – Ticket

25. JIRA – Ticket

26. Experiment Illuminati

27. Our Process
COLLECT GREAT IDEAS
TRIAGE BASED ON POTENTIAL IMPACT
BUILD OUT GOOD CANDIDATES
SCHEDULE TO AVOID CONFLICTS
RUN EXPERIMENTS
ANALYZE RESULTS

28. Optimizely – Build

29. Optimizely – Build

30. Prepare for results.

31. Confluence – Corresponding Test Page

32. Confluence – Corresponding Test Page

33. Confluence – Corresponding Test Page

34. Confluence – Corresponding Test Page

35. Confluence – Corresponding Test Page

36. Confluence – Corresponding Test Page

37. Confluence – Corresponding Test Page

38. Confluence – Corresponding Test Page

39. Confluence – Corresponding Test Page

40. Confluence – Corresponding Test Page

41. Confluence – Corresponding Test Page

42. Confluence – Corresponding Test Page

43. Our Process
COLLECT GREAT IDEAS
TRIAGE BASED ON POTENTIAL IMPACT
BUILD OUT GOOD CANDIDATES
SCHEDULE TO AVOID CONFLICTS
RUN EXPERIMENTS
ANALYZE RESULTS

44. JIRA – Prioritized Backlog

45. Our Process
COLLECT GREAT IDEAS
TRIAGE BASED ON POTENTIAL IMPACT
BUILD OUT GOOD CANDIDATES
SCHEDULE TO AVOID CONFLICTS
RUN EXPERIMENTS
ANALYZE RESULTS

46. Optimizely – Start Experiment

47. Optimizely – Project Code

48. Magnolia CMS – Include Optimizely
Checkbox

49. Magnolia CMS – Include Optimizely
Checkbox

50. Confluence – Corresponding Test Page

51. Our Process
COLLECT GREAT IDEAS
TRIAGE BASED ON POTENTIAL IMPACT
BUILD OUT GOOD CANDIDATES
SCHEDULE TO AVOID CONFLICTS
RUN EXPERIMENTS
ANALYZE RESULTS

52. Optimizely – Results

53. What are you are
trying to improve?

54. Blimp Dashboard

55. Optimizely – Results

56. Confluence – Corresponding Test Page

57. Confluence – Results Archive

58. Our Test Philosophy

59. Our Test Philosophy
• Everything affecting conversion is tested

60. Our Test Philosophy
• Everything affecting conversion is tested
• Be aggressive

61. Our Test Philosophy
• Everything affecting conversion is tested
• Be aggressive
• Regular cadence, no traffic wasted

62. Our Test Philosophy
• Everything affecting conversion is tested
• Be aggressive
• Regular cadence, no traffic wasted
• Never run a test that you wouldn’t want to win

63. Key Take Aways

64. Key Take Aways
• Ideas are everywhere collect as many as you can, prioritize based on impact.

65. Key Take Aways
• Ideas are everywhere collect as many as you can, prioritize based on impact.
• Use ticket number everywhere so everyone is referring to the same test.

66. Key Take Aways
• Ideas are everywhere collect as many as you can, prioritize based on impact.
• Use ticket number everywhere so everyone is referring to the same test.
• Involve stakeholders and developers… and get approvals.

67. Key Take Aways
• Ideas are everywhere collect as many as you can, prioritize based on impact.
• Use ticket number everywhere so everyone is referring to the same test.
• Involve stakeholders and developers… and get approvals.

69. Thank you!
MARK HALVORSON • HEAD OF INTERACTIVE • ATLASSIAN SOFTWARE • @halv0112