The document discusses how Atlassian integrated support and development teams to enhance customer service through data-driven insights. It highlights the methodologies implemented for tracking customer pain points and prioritizing bug fixes to improve user experience. The focus on predictive analytics is emphasized as a means to forecast support needs and improve overall service delivery.

1. How Atlassian Broke Down the Walls
Between Support & Development
MATT SAXBY | SERVICE ENABLEMENT TEAM LEAD | ATLASSIAN
MATT HUNTER | SERVICE ENABLEMENT TEAM LEAD | ATLASSIAN

2. $0
$30
$60
$90
$120
07-2008
01-2009
07-2009
01-2010
07-2010
01-2011
07-2011
01-2012
07-2012
01-2013
07-2013
01-2014
07-2014
01-2015
07-2015

3. Agenda
Reporting
Insights
Prediction
The Problem

4. DEVELOPMENT CUSTOMERS
SUPPORT
PRODUCT

5. 9SUPPORT CENTERS
350+CUSTOMER SUCCESS STAFF
89,200+CUSTOMERS

6. CUSTOMERS
SUPPORT
PRODUCT
DEVELOPMENT

7. Insights
Readiness
SERVICE
ENABLEMENT
SUPPORT
DEVELOPMENT

8. 88
SERVICE ENABLEMENT EXISTS TO
Create a frictionless product experience for our
customers

9. ShirtVoteTM
INTRODUCING

11. Shirts, for everything
For Products

12. Shirts, for everything
For Products

13. Shirts, for everything
For Products For Events

14. Shirts, for everything
For Products For Events

15. Shirts, for everything
For Products For Events For Celebrations

16. Shirts, for everything
For Products For Events For Celebrations

17. Shirts, for everything
For Products For Events For Celebrations For Fun

18. SHIRT VOTE
Add a new shirt
design
View existing shirt
designs ordered by
most votes
JIRA Service Desk
2015
ShipIt 30
2015
40,000 Customers
2014
Game of Codes
2014
200 votes
180 votes
120 votes
80 votes
Need Help?

19. SHIRT VOTE
Full screen
images giving
you an up close
and personal
look at the shirt
Description with all
the detail you would
need
JIRA Service Desk
2015
Vote
Designed to promote JIRA
Service Desk and give a great
way for users (both inside and
outside Atlassian) able to pro…
Ability to vote
for the shirts
you love
200 votes

20. Agenda
Reporting
Insights
Prediction
The Problem

21. How do you tell if a product
is causing customer pain?

22. Things to
Consider:
Actionable
You can tell what has caused
customer pain to go up.
Representative
When the number goes up customer
pain goes up.
Trackable
Easy to see current status when
needed

23. In App
How to create tickets?
1 2
support.shirtvote.com

24. 60Support tickets in a month

25. 90Support tickets in a month

26. 40Support tickets in a month

27. Things to
Consider:
Actionable
You can tell what has caused
customer pain to go up
Representative
When the number goes up customer
pain goes up
Trackable
Easy to see current status when
needed

28. Tickets Over Time
50
56
62
68
74
80
January February March April May June July

29. Customer Growth Over Time
0
200
400
600
800
January February March April May June July August

30. Contact Index
Support
Tickets
=Active Users
Contact
Index

31. Contact Index
60
550
10.9%=

32. Contact Index Over Time
5
6.4
7.8
9.2
10.6
12
January February March April May June July August

33. Things to
Consider:
Actionable
You can tell what has caused
customer pain to go up
Representative
When the number goes up customer
pain goes up
Trackable
Easy to see current status when
needed

34. = 60 Support Contacts
Data Conclusion

35. Android App
I can't view shirts
When I open the app to view the list of shirts, it comes up with an error

37. = 15 Voting on Shirts
Data Conclusion
= 27 Viewing Shirts
= 12 Adding Shirts
= 6 Searching for shirts

38. Things to
Consider:
Actionable
You can tell what has caused
customer pain to go up
Representative
When the number goes up customer
pain goes up
Trackable
Easy to see current status when
needed

39. Agenda
Reporting
Insights
Prediction
The Problem

40. TURNING DATA INTO INSIGHTS
Service Enablement has built
a partnership with
development through
turning data into insights.

41. Bug Ranking Rollouts
Support Insights

42. Bug Ranking
Too much time was being wasted
deciding which bugs we should solve
first.
We needed a system to consistently
and objectively rank bugs, agreed on
across the business.

43. VIEW SHIRTS BUG VOTING ON SHIRTS BUG

44. How do we gauge interest in a Bug?
Votes and WatchersLinked Support cases

45. 10 reports20 reports
VIEW SHIRTS BUG VOTING ON SHIRTS BUG

46. Active Users License Tier
Customer Size

47. 10 customers20 customers
VIEW SHIRTS BUG VOTING ON SHIRTS BUG
1 active
users avg
3 active
users avg

48. 10 customers x
3 active users avg
20 customers x
1 active users avg
VIEW SHIRTS BUG VOTING ON SHIRTS BUG
20 active
users total
30 active
users total

49. Severity
The impact of a bug on the end user,
system or business.
How bad is it?

50. Major
Performance problems or
breaks minor functionality
Severity
Minor
Inconvenience such as a
display problem or an
inconsistency
Critical
Causes application outage,
or breaks core functionality
8 4 2

51. 30 active users impacted20 active users impacted
VIEW SHIRTS BUG VOTING ON SHIRTS BUG
Causes
outage
Minor
feature not
working

52. 30 active users impacted20 active users impacted
VIEW SHIRTS BUG VOTING ON SHIRTS BUG
Critical
Severity
Major
Severity

53. Recency
How many customers have reported
the problem in the past 90 days?

54. Customers Impacted Per MonthCustomers
0
1
2
3
4
Jan Feb Mar Apr May June July
View Bug Voting Bug

55. 30 active users impacted
Major Severity
20 active users impacted
Critical Severity
VIEW SHIRTS BUG VOTING ON SHIRTS BUG
2 reports in
last 90 days
10 reports in
last 90 days

56. Severity
How bad is the impact of the
bug?
User Impact Score
Recency
Is the bug getting better or
worse?
Impacted Users
Number of users impacted
by a bug

57. UIS Calculation
impacted users
x
severity
x
recency
=UIS

58. 30 active users impacted
Major Severity
10 reports in last 90 days
20 active users impacted
Critical Severity
2 reports in last 90 days
VIEW SHIRTS BUG VOTING ON SHIRTS BUG

59. 30 (users)
x
4 (severity)
x
10 (recency)
20 (users)
x
8 (severity)
x
2 (recency)
VIEW SHIRTS BUG VOTING ON SHIRTS BUG
= 320 = 1200

60. 30
x
4
x
10
20
x
8
x
2
VIEW SHIRTS BUG VOTING ON SHIRTS BUG
3 UIS 12 UIS

61. KEY SUMMARY UIS
BUG-4
BUG-3
Images not displaying 7
Shirts not showing in correct order 4
BUG-2 Can’t upload images 2

62. KEY SUMMARY UIS
BUG-4 Images not displaying 7
BUG-3
BUG-1
Shirts not showing in correct order 4
Viewing shirts crashes application 3
BUG-2 Can’t upload images 2

63. KEY SUMMARY UIS
BUG-4 Images not displaying 7
BUG-3
BUG-1
Shirts not showing in correct order 4
Viewing shirts crashes application 3
BUG-2 Can’t upload images 2
BUG-5 Voting on shirts reduces vote count 12

64. After
Alignment on which bugs are the most
important, bug master always grabs the
issue with the highest UIS.
Before
Frequent disagreement on the highest
bug priorities and what to work on next.

65. Bug Ranking Rollouts
Support Insights

66. SHIRT VOTE
JIRA Service Desk
2015
ShipIt 30
2015
40,000 Customers
2014
Game of Codes
2014
200 votes
180 votes
120 votes
80 votes

67. SHIRT VOTE
JIRA Service Desk
2015
ShipIt 30
2015
40,000 Customers
2014
Game of Codes
2014
200 votes
180 votes
120 votes
80 votes
Votes
Title
Date
Sort shirts in the
order that suits you
best

68. FEATURE LIFECYCLE - THE OLD
Feature is
developed
Code is reviewed
and tests are run.
Change is rolled
out to entire
customer base
Bugs related to the
feature that weren’t
caught during
review are fixed.
Dev Review Rollout Fix

69. Gradual Rollouts - The Old
0
13
25
38
50
Rollout%
0%
25%
50%
75%
100%
Week 1 Week 2 Week 3 Week 4 Week 5
% of Customers With Feature Support Tickets

70. SHIRT VOTE
JIRA Service Desk
2015
ShipIt 30
2015
40,000 Customers
2014
Game of Codes
2014
200 votes
180 votes
120 votes
80 votes
Votes
Date
Title

71. SHIRT VOTE

72. Gradual Rollouts - The Old
0
13
25
38
50
Rollout%
0%
25%
50%
75%
100%
Week 1 Week 2 Week 3 Week 4 Week 5
% of Customers With Feature Support Tickets

73. Gradual Rollouts - The Old
0
13
25
38
50
Rollout%
0%
25%
50%
75%
100%
Week 1 Week 2 Week 3 Week 4 Week 5 Week 6
% of Customers With Feature Support Tickets

74. Gradual Rollouts - The New
0
12.5
25
37.5
50
Rollout%
0%
25%
50%
75%
100%
Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7
% of Customers With Feature Support Tickets

75. Gradual Rollouts - The New
0
12.5
25
37.5
50
Rollout%
0%
25%
50%
75%
100%
Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7
% of Customers With Feature Support Tickets

76. Gradual Rollouts - The New
0
12.5
25
37.5
50
Rollout%
0%
25%
50%
75%
100%
Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7
% of Customers With Feature Support Tickets

77. BIG BANG VS GRADUAL ROLLOUT
0
2.5
5
7.5
10
Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7
Contact Rate - Gradual Contact Rate - Big Bang
Tickets Created:
Big Bang - 57
Gradual - 29

78. FEATURE LIFECYCLE
Feature is
developed
Code is reviewed
and tests are run.
Change is
gradually rolled
out.
Most bugs were
caught during
rollout, so this
feature is ready to
close off.
Dev Review Rollout Close
Tight feedback
loop to ensure
problems are
caught early
Feedback

79. Contact Rate
Create a contact rate
specifically monitoring
tickets received from
customers with the feature.
Gradual Rollout Pro Tips
Feedback Loop
Work closely with the
Development team to share
feedback asap.
Readiness
Involvement with
Development early in the
cycle ensures the support
team is ready for the change.

80. Agenda
Reporting
Insights
Prediction
The Problem

81. Hiring
Allows you to
forecast the number
of Supporters you
need to hire
Customer
Experience
Helps to tell the
customer experience
story
Development
Negotiations
Assists with
negotiating
Development
priorities
Goal Setting
Sets goals to track
the success of the
team.
Why is prediction Important?

82. Baseline
Forecast of support contact
assuming current trends
continue
Initiatives
Changes impacting product
that are expected to impact
support contact
How do you predict?

83. Baseline
Forecast of support contact
assuming current trends
continue
Initiatives
Changes impacting product
that are expected to impact
support contact
How do you predict?

84. Complex Modelling
• Prophet
• ARIMA
BASE FORECASTING
Basic Modelling
Average Yearly Contact Index
0
3
6
9
12
Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
FY16 FY17

85. BASE FORECASTING
0
3
6
9
12
Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
FY16 FY17 Average
Complex Modelling
• Prophet
• ARIMA
Basic Modelling
Average Yearly Contact Index

86. BASE FORECASTING
0
3
6
9
12
Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
Average
Complex Modelling
• Prophet
• ARIMA
Basic Modelling
Average Yearly Contact Index

87. Contact Index
Support
Tickets
=Active Users
Contact
Index

88. Support Tickets
Support
Tickets = Forecast
Contact
Index
x
Active Users

89. BASE FORECASTING
0
20
40
60
80
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
Tickets

90. Baseline
Forecast of support contact
assuming current trends
continue
Initiatives
Changes impacting product
that are expected to impact
support contact
How do you predict?

91. Contact Decrease
Bugfix, feature improvement,
support policy changes.
Contact Increase
Rollouts, infrastructure
change.
Initiative Types

92. CONTACT DECREASE
0
20
40
60
80
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
Ticket Forecast
Shirt voting bug due to be
fixed in February.

93. CONTACT DECREASE
0
20
40
60
80
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
Ticket Forecast Post Bug Fix
Shirt voting bug due to be
fixed in February.

94. Contact Decrease
Bugfix, feature improvement,
support policy changes.
Contact Increase
Rollouts, infrastructure
change.
Initiative Types

95. SHIRT VOTE
JIRA Service Desk
2015
ShipIt 30
2015
40,000 Customers
2014
Game of Codes
2014
200 votes
180 votes
120 votes
80 votes
Search for the shirts
that interest you
Search

96. The best predictor of future
behaviour is past behaviour
DR PHIL

97. Gradual Rollouts - The New
0
12.5
25
37.5
50
Rollout%
0%
25%
50%
75%
100%
Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7
% of Customers With Feature Support Tickets

98. CONTACT INCREASE
0
22.5
45
67.5
90
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
Ticket Forecast
Search feature due to be
Rolled out in May and June

99. CONTACT INCREASE
0
22.5
45
67.5
90
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
Ticket Forecast Post Rollout
Search feature due to be
Rolled out in May and June

100. INITIATIVES COMBINED
0
22.5
45
67.5
90
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
Baseline Forecast Final Forecast
Including both positive and
negative contact changes

101. Hiring
Allows you to
forecast the number
of Supporters you
need to hire
Customer
Experience
Helps to tell the
customer experience
story
Development
Negotiations
Assists with
negotiating
Development
priorities
Goal Setting
Sets goals to track
the success of the
team.
Why is prediction important?

102. Insights
Making things actionable is
fundamental to creating
change
In Summary
Prediction
You need to look to the
future to prepare your
support team and ensure
they can provide legendary
support
Reporting
Have a Trackable,
Representative and
Actionable way to gauge
customer impact.

103. Thank you!
MATT SAXBY | SERVICE ENABLEMENT TEAM LEAD | ATLASSIAN
MATT HUNTER | SERVICE ENABLEMENT TEAM LEAD | ATLASSIAN

104. What do
you think?
Ask a question to win
one of 10 Service
Desk Shirts