Join a couple DevOps gurus for a deep dive into AWS configuration for Bitbucket Data Center and learn how Atlassian keeps our own developers happy with speedy clone times and better availability. Specifically we'll cover: How Smart Mirrors can provide lightning fast Git access for teams all over the globe. How to use AWS auto-scaling with Bitbucket Data Center for better performance. How to provide continuous availability using an HA cluster in multiple AWS zones. How to start using Git large file support (Git LFS) to dramatically reduce clone and build times. Products covered: Bitbucket

1. How Atlassian Scales
Bitbucket Data Center on
AWS
FELIX HAEHNEL
DEVELOPER • ATLASSIAN
CHRIS SZMAJDA
SENIOR TEAM LEAD • ATLASSIAN

2. Performance at scale Auto-scaling clusters in AWS
Bitbucket
Data Center
Bitbucket
Data Center
@ Atlassian
Smart Mirrors
Disaster recovery
High availability and failover
Instant scalability

3. CLUSTERING IN AWS
SMART MIRRORS
SCALING BITBUCKET @ ATLASSIAN
The Challenge:
Scaling Bitbucket
@ Atlassian
DISASTER RECOVERY

4. Why scale @ Atlassian?

5. Scale @
Atlassian
100,000’s of builds
1,000’s of users
Terabytes of data
Repositories and forks and moar,
mostly on one central instance.
More Atlassians using Bitbucket
every day.
Continuous integration hammering
Bitbucket around the clock.

7. Sydney
Global Atlassian teams
Type your annotation here,
the dotted line can be
moved and/or re-sized.
San Fransisco
Austin
Manila
Tokyo
Amsterdam
London
Gdansk

8. • Big cool statistic
• 2,56
9
• Add-Ons in Marketplace
Repository data
GBytes
23042298231223262291228222712264224022292212217921482130201820031993198419761951194219211906189818711855184418191809179417761752174317211702168516711653

9. OMG CI load!
1,000’s of agents
100,000’s of builds
Petabytes served

10. Pull Request workflow
1,000’s of repo’s
100,000’s of PR’s
500,000+ comments

11. Downtime

12. SCALING BITBUCKET @ ATLASSIAN
SMART MIRRORS
CLUSTERING IN AWS
The Sky’s the Limit:
Clustering in AWS
DISASTER RECOVERY

13. Clustering

14. Clustering
Load balancer
Cluster nodes

15. What makes
up a Data
Center
deployment?

16. Cluster nodes
Database
File Server
(NFS)
Elasticsearch
Bitbucket Data Center deployment model
Load balancer

17. Cluster nodes
Database
File Server
(NFS)
Elasticsearch
Bitbucket Data Center deployment model
Load balancerSimple but powerful
No lock-in stacks.
Takes advantage of
your infrastructure.
Scales as you grow.

18. Data Center, in AWS?

19. Why AWS? Managed services
Elastic scale
Total Cost of
Ownership
Cheaper in the long run.
Grow from micro to massive in
minutes.
Saves time.

20. AWS Services

21. Auto Scaling
Auto Scaling Group
Elastic Load Balancer

22. Relational Database Service (RDS)
Automatic
upgrades
Read
Replicas
Multiple
Availability
Zones
Scale

23. Elasticsearch Service (ES)
Scale
Security
Code and
repository
Search

24. Disaster RecoveryPerformance Zero Downtime Backup
Snapshot in under a second.
Restore in minutes.
Low latency.
Up to 16 Terabytes.
Up to 20,000 IOPS.
Copy snapshots offsite.
Elastic Block Store (EBS)

25. Auto Scaling
Group
Amazon
RDS
NFS Server
Amazon
Elasticsearch
EBS
volume
Amazon
ELB
Bitbucket DC AWS
deployment model

26. Availability Zones

27. AWS
Availability
Zones Low latency
Typically several miles apart.
Within one Region
Equally visible across the whole
Region.
Isolated data centers
Connected by fast network links.

28. Why more
than one
AZ?

29. us-east-1a
Availability Zones
us-east-1b

30. EBSRDS
ELB
Auto Scaling Group
Elasticsearch Service
Availability Zones for Bitbucket DC
Use two or more AZ’s.
Enable cross zone
load balancing.
Enable Multi-AZ
deployment.
Deploy two or more nodes.
Enable Zone Awareness.
Snapshot often.
Consider Cloud NAS
products for AWS.

31. What about scale?

32. Elastic scale
Scaling in AWS
More nodes =
greater capacity
Load tests
Type your annotation here,
the dotted line can be
moved and/or re-sized.
AWS makes it easy to scale
up capacity.
Larger clusters can serve
bigger workloads.
Representing 10,000’s of typical
users and build agents.
Throughput(TPS)
10
20
30
40
50
60
70
Concurrent requests
0 40 80 120 160 200 240 280
1 node 2 nodes 4 nodes 8 nodes 12 nodes

33. Choose the right
instances
IOPS = Happiness ☺Scale up, then out
Scale vertically with
bigger instances first.
Then scale horizontally
with more nodes.
Bitbucket’s largest workload
is Git.
Give it enough IOPS.
Different instance types
have different balances of
CPU, memory, and I/O.
Choose wisely.
Scaling in AWS: Pro tips

34. How did we do it?

35. How we did it

36. How we did it
Loose coupling
Stateless nodes
Highly tuned caches
Keep “hot” data in fast local
storage.
Extremely light JVM memory
footprint.
Minimizes node-to-node
communication.
Don’t maintain any state
that can’t be reconstructed
from other sources.
Storage
optimizations
Minimize use of the shared
file system.

37. How we did it
Source Code Management cache
Intercepts Git requests and caches the responses on
each node (in AWS, on an Instance Store device).
Saves CPU and memory
Every cache hit saves a Git process.
Saves NFS traffic
Takes load off the shared file system.
Cluster aware
Lightweight cache consistency between nodes.
Git request
NFS
git
response
SCM
cache
another
Git request
cached
response
Git push
invalidations
lotsa Git
requests
lotsa cached
responses
CI agents generate many Git requests

38. That’s great!
But how do I deploy it all?

39. Deploying
in AWS
Auto Scaling Group
ELB
RDS instance
Elasticsearch
EC2 file server
Scaling policies
Security Group
VPC
Subnets
EBS volume
Provisioned IOPS
Instance Stores
IAM Role
Listeners
Manually create … ?

40. Deploy in a few clicks
One template =
many resources
One JSON file that describes an
entire stack of AWS resources:
ELB, ASG, EC2, EBS, RDS, ES, …
Create, update, and delete stacks
from the AWS Console or CLI.
AWS
CloudFormation

41. AWS
CloudFormation

42. Auto Scaling
Group
Amazon
RDS
NFS Server
Amazon
Elasticsearch
EBS
volume
Amazon
ELB
CloudFormation
{
"AWSTemplateFormatVersion": "2010-09-09",
"Description": "Atlassian Bitbucket Data Center",
"Metadata": {
...
},
"Parameters": {
...
},
"Resources": {
"ClusterNodeGroup": {
"Type": "AWS::AutoScaling::AutoScalingGroup",
...
},
...
}

43. Bitbucket Amazon Machine Image (AMI)
Based on
Amazon Linux
“Bitbucket Server”
Multi-role:
Any version
of Bitbucket
“Cluster node”
“File server”
Public in all AWS Regions

44. https://bitbucket.org/atlassian/
atlassian-aws-deployment

45. AWS Quick Start

46. AWS
Quick Start
Free credits!
Best practices
Single click deploy
Complete CloudFormation template
for Bitbucket Data Center and VPC.
Designed by Amazon solutions
architects for security and scale.
Amazon is offering free credits with
all Atlassian AWS Quick Starts.

48. https://atlassian.com/aws

49. CLUSTERING IN AWS
SCALING BITBUCKET @ ATLASSIAN
SMART MIRRORS
Mirror, mirror on the wall:
Smart Mirrors
DISASTER RECOVERY

50. Austin
“Prod”
(N. Virginia)
San Fransisco

51. “Prod”
(N. Virginia)
Austin
San Fransisco
Tokyo
Gdańsk
Sydney

52. San Fransisco
Tokyo
Gdańsk
“Prod”
(N. Virginia)
Sydney
Austin

53. San Fransisco
Tokyo
Gdańsk
“Prod”
(N. Virginia)
Austin
Sydney

54. Smart Mirror (AU)
FishEye/Crucible
JIRA
Confluence
Bamboo
Bitbucket Server
Minutes
0 25 50 75 100
How much faster is a mirror?

55. On average over
faster to clone!
8x

56. Setup Wizard
AWS CloudFormation
Launch Bitbucket with
application.mode=mirror.
Deploying
Mirrors in
AWS

57. That’s great!
But how does it work?

58. How it works
Primary
Bitbucket
Mirror

59. Primary
Bitbucket
Mirror
How it works

60. How it works
Primary
Bitbucket
Mirror

61. How it works
Primary
Bitbucket
Mirror

62. How it works
Primary
Bitbucket
Mirror

63. How it works
Primary
Bitbucket
Mirror

64. How it works
Primary
Bitbucket
Mirror

65. What makes
Smart Mirroring
smart?

66. Primary
Bitbucket
Mirror
How it works
Forks optimized
Saves disk space on the primary Bitbucket instance.
Mirrors fetch only once
Saves disk space and network bandwidth on the mirror.
repository
forkforksmirrored forks

67. How it works
Git LFS optimized
Large files fetched only when needed.
large files
Each large file fetched only once.
Saves disk space and bandwidth.
Primary
Bitbucket
Mirror
repository
mirrored large files

68. Primary
Bitbucket
Mirror
Mirrors can be for CI, too
CI agents
“Build” mirrors
Take load off your primary
Bitbucket instance.

69. CLUSTERING IN AWS
SMART MIRRORS
DISASTER RECOVERY
Always have a plan B:
Disaster recovery
SCALING BITBUCKET @ ATLASSIAN

70. Disasters 101
Always have a backup ALWAYS have a backup

71. Cluster nodes
Database
File Server
Elasticsearch
Backing up Bitbucket
Load balancer
Bitbucket backup client?
Only recommended for small instances.
Rsync?
Still requires some downtime.
Backup storage
Consistent snapshots

72. Backup
Zero Downtime

73. Database File Server
Zero Downtime Backup
Backup storage
Amazon RDS
All DB vendors Many FS vendors
Amazon EBS
ZFS
Atomic
snapshots
Offsite snapshots
Amazon S3

74. Downtime

75. Primary Bitbucket
(N. Virginia)
Sydney
Backups @ Atlassian
Round the clock
Backups taken
continually,
sometimes minutes
apart.
Backups copied offsite
Can be restored anywhere,
in minutes.
AWS
snapshots
Sub-second latency.

76. https://bitbucket.org/atlassianlabs/
atlassian-bitbucket-diy-backup

77. Disasters 201
Always have a standbyALWAYS have a backup

78. Disaster recovery

79. Disaster recovery
Primary instance Standby instance
Standby
database
Standby
file server
RDS read replica
All DB vendors
ZFS
Many FS vendors

80. my name is
Integrity Checker
Hello
Disaster recovery

81. https://bitbucket.org/atlassianlabs/
atlassian-bitbucket-diy-backup
Yes, it does Disaster recovery too

82. Start replicating
AWS CloudFormation
Launch Bitbucket Data Center with
DBMaster=<primary-db>.
Deploying a
standby in
AWS
setup-home-replication.sh
replicate-home.sh

83. Backups
Disaster recovery
standby
Clustering
Handles isolated
failures.
Works in seconds.
Handles failure of an
entire data center.
Failover in minutes.
Handle failures that
corrupt all your data.
Restore to any
point in time.
High availability: Pro tips

84. Smart Mirroring Disaster Recovery AWS Support and
Quick Start

85. https://atlassian.com/aws
FELIX HAEHNEL
DEVELOPER • ATLASSIAN
CHRIS SZMAJDA
SENIOR TEAM LEAD • ATLASSIAN