Greg Warden, Atlassian Facebook (and others) have raised the bar on what people expect in SaaS – performance, availability, dev-speed. Unfortunately a lot of what we learned as on-premises server developers needs to be un-learned to get to this level. In this talk we'll explore some of the learnings Atlassian has had on its journey from World Class server software to proper SaaS.

1. Damn You Facebook
GREG WARDEN • HEAD OF ENGINEERING SERVICES • ATLASSIAN • @USMILE1

2. Thanks to
Facebook, Twitter,
Netflix, Linkedin,
Etsy and others…

3. The standard for
availability,
performance and
dev speed is now
VERY HIGH

4. For us old-farts that
learned to develop
for servers, the
rules have changed

5. Damn

6. There is already
A LOT of material
out there about this
stuff

7. By people WAY
smarter than me

8. So this talk is NOT
going to be about…

9. CAP theory

10. or the
fallacies of
distributed
programming

11. or
Raft vs. Paxos

12. or CRDTs

13. It’s not about
12-factor apps

14. or
micro-services

15. or
the reactive
manifesto

16. It’s not about
circuit-breakers or
back pressure

17. or
no-sql, CQRS,
immutability,
idempotentcy, or
functional
programming

18. Sorry

19. This talk is about
thinking differently
about some of the
basics

20. it’s about un-
learning some stuff

21. practical learnings

22. Let’s start with
availability

23. Un-learning #1:
the customer admin
is sitting there on
app startup

24. You can’t just dump
some friendly error
message to a log

25. You can’t just bork
the whole system
and expect the
admin will sort it
out

26. @40000000553d9deb364aa0d4 2015-04-27 04:24:33,909 main FATAL
[atlassian.jira.startup.JiraStartupLogger]
@40000000553d9deb364aa8a4
@40000000553d9deb364aa8a4
**************************************************************************************
***********************************************
@40000000553d9deb364aac8c The following plugins are required by JIRA, but have not
been started: Gadget Spec Publisher Plugin (com.atlassian.gadgets.publisher)
@40000000553d9deb364aac8c
**************************************************************************************
***********************************************
@40000000553d9deb36589af4
@40000000553d9deb3686947c 2015-04-27 04:24:33,914 main ERROR
[atlassian.jira.upgrade.UpgradeLauncher] Skipping, JIRA is locked.
@40000000553d9deb368ada3c 2015-04-27 04:24:33,915 main INFO
[atlassian.jira.scheduler.JiraSchedulerLauncher] JIRA Scheduler not start
Now what?

27. Just text by itself,
for impact.
Graceful degradation

28. Un-learning #2:
sys properties are a good
source of user information

29. Tied to a server?

30. It would really suck
if you had to restart
the system to
change the
timezone?

31. public TimeZone getDefaultTimezone()
{
String systemDefaultTimeZoneId =
applicationProperties.getString( APKeys.JIRA_DEFAULT_TIMEZONE );
if (StringUtils.isNotEmpty(systemDefaultTimeZoneId))
{
return TimeZone.getTimeZone(systemDefaultTimeZoneId);
}
return TimeZone.getDefault();
}
The old way

32. public void acceptTenant(final String tenantId, final Map<String, String>
tenantProperties)
throws LandlordRequestException
{
final Tenant tenant = new JiraTenantImpl(tenantId);
jiraTenantAccessor.addTenant(tenant);
updateTimezoneIfProvided(tenantProperties);
tenantPluginBridge.trigger();
for (final TenantInitialDataLoader loader : tenantInitialDataLoaders)
{
loader.start(tenant);
}
eventPublisher.publish(new TenantArrivedEvent(tenant));
log.debug("Tenanted - AcceptTenantCalled");
}
Might this be better?

33. Un-learning #3:
You can take the system
down to upgrade it

34. stick with me a sec

35. Cost-effective high
availability tends to
imply multi-tenancy

36. Sharding be-damned, you
have global customers on
shared infrastructure

37. Somebody is always awake

38. So you need 0-downtime
upgrades

39. Safe 0-downtime upgrades
on shared infrastructure
tends to imply

40. You can’t upgrade your code
and schema at the same time

41. Fast five pattern

42. Fast five pattern

43. Fast five pattern

44. Fast five pattern

45. Fast five pattern

46. Fast five pattern

47. Un-learning #4
you can rely on an
in-memory cache
for global state

48. The Server

49. The Cluster

50. The Cloud
X

51. you can’t cache all
the JIRA users for
every customer on
every node

52. Un-learning #5
you have a local
filesystem

53. Decoupled remote
file service

54. And you replicate
that storage globally

55. And then:
backup / restore
import / export
search indexing
performance

56. Now lets talk about
performance

57. Un-learning #6:
Varying the resource
returned based on
cookies is a good idea

58. Ever seen a URL like
this in JIRA:
<script type="text/javascript" src="//
d2p4ir3ro0j0cb.cloudfront.net/gregdownunder.atlassian.net/s/
d7f5a59ca238699619bdd31870a15fca-CDN/en_USdre0s0/65000/8/67/_/
download/superbatch/js/batch.js?
atlassian.aui.raphael.disabled=true&amp;locale=en-US" ></
script>

59. You see this bit?
<script type="text/javascript" src="//
d2p4ir3ro0j0cb.cloudfront.net/gregdownunder.atlassian.net/s/
d7f5a59ca238699619bdd31870a15fca-CDN/en_USdre0s0/65000/8/67/_/
download/superbatch/js/batch.js?
atlassian.aui.raphael.disabled=true&amp;locale=en-US" ></
script>

60. It used to be just:
<script type="text/javascript" src="//
gregdownunder.atlassian.net/s/d7f5a59ca238699619bdd31870a15fca-
CDN/en_USdre0s0/65000/8/67/_/download/superbatch/js/batch.js?
atlassian.aui.raphael.disabled=true&amp" ></script>

61. Remember this?

62. Cookies and CDNs
don’t play nice

63. Pro tip
figure out how to
monitor the CDN
before you roll it out

64. Un-learning #7:
Local session state
is a convenient
solution for multi-
step workflows

66. two choices:
browser state
external cache

67. That’s right,
external cache

68. Pro-tip:
Write your queries
right and the DB is a
good external
cache

69. Un-learning #8:
admins are the only
ones experiencing
start-up time

70. Customer provisioning

71. Key Idea
decouple app and
customer specific data

72. Key Idea

73. Key Idea

74. Un-learning #9:
snowflakes are
pretty

75. allowing variation is
great for custom
environments but it
screws your dev
speed in SaaS

76. dev speed centers
around making lots
of little changes

77. cost of each change is
something like:
abstract cost *
# variations to consider

78. Options?
minimise the cost of
consideration

79. Options?
make sure variation is
observable
programmatically

80. Options?
capture configurations in
a query-able meta-data
service

81. Pro-tip
upgrades should assert
what the world is like
before acting

82. Immutable infrastructure

83. Un-learning #10:
logs are pretty

84. we were then told
to log in a super
structured way

85. Un-learning 10-a:
logs are pretty is an
un-learning

86. Invest in a proper
logging
infrastructure

87. Dev Speed
Learning Fast
Log Everything

88. In Conclusion

89. All that crazy
hacker-news stuff
is cool. But you
gotta start with the
basics

90. no admins
external state
fast five upgrades
start up time matters
respect the CDN
log everything

91. no admins
external state
fast five upgrades
start up time matters
respect the CDN
log everything

92. no admins
external state
fast five upgrades
start up time matters
respect the CDN
log everything

93. no admins
external state
fast five upgrades
start up time matters
respect the CDN
log everything

94. no admins
external state
fast five upgrades
start up time matters
respect the CDN
log everything

95. no admins
external state
fast five upgrades
start up time matters
respect the CDN
log everything

96. thank you!

97. Damn you Facebook - Raising the bar in SaaS
Submit your feedback:
go.atlassian.com/acfacebook