Serving developer binaries isn’t trivial. Such binaries are consumed by tools ,and create massive request load. Add to that support for metadata, REST API, storage quotas, stats, repo indexes on demand and global HA distribution, and you’ve got yourself a pretty complicated system to run and manage. This talk will show you how Bintray, JFrog’s social binary distribution service, works. We will speak about how the system segmentation supports massive loads across data centers with stateless vertical scaling; how Grails applications scale and how we tie up different NoSQL technologies such as CouchDB, MongoDB, ElasticSearch & Redis; how we chose between physical and virtual servers and how we manage deployments without service interruption.

1. Building a Massively Scalable
Cloud Service
from the Grounds Up

2. Yoav Landman
@yo
dman
avlan
cee tee
github
oh @ J
Frog
/yoav
.com

3. What Frog?

4. What Frog?

5. What Frog?

6. What Frog?

7. So…

8. ___________
Some Numbers
liftoff + 5 months

9. ___________
Some Numbers
Users
liftoff + 5 months
7K

10. ___________
Some Numbers
liftoff + 5 months
Users
7K
Packages
70K

11. ___________
Some Numbers
liftoff + 5 months
Users
7K
Packages
70K
Requests
1.2 B/Month

14. ___________
Requirements

15. ___________
Requirements
– Download binaries

16. ___________
Requirements
– Download binaries
– Web Front

17. ___________
Requirements
– Download binaries
– Web FRONT
– REST API

18. ___________
Requirements
– Download binaries
– Web FRONt
– REST API
– Backend services

19. We know developers

20. %new_sexy_lang% community

21. Not our fault! AWS failed again!

22. Downloads must…

23. Web application must…

24. Backend Services must…

25. Choose your battles...

26. _________________
Non-Func. Requirements

27. _________________
Non-Func. Requirements
Requirement RPS Availability

28. _________________
Non-Func. Requirements
Requirement RPS Availability
Download
10K Always

29. _________________
Non-Func. Requirements
Requirement RPS Availability
Download
10K Always
Interaction
200 Almost always

30. _________________
Non-Func. Requirements
Requirement
Download
Interaction
Services
RPS
10K
200
10
Availability
Always
Almost always
Most of the time

31. Download Server

33. No Servlets here

42. Deduplication by Checksum
File
A:
46b34
/user-­‐a/repo-­‐z/package-­‐y/file-­‐x
File
B:
a64ff7
/org-­‐c/repo-­‐m/package-­‐n/file-­‐k
/user-­‐m/repo-­‐w/package-­‐t/file-­‐f

43. Flat blobs storage
File
A:
46b34
File
B:
a64ff7

48. Mapping
/user-­‐m/repo-­‐w/package-­‐t/file-­‐f

52. Web Front

59. Web Front
Web Framework

62. ___________
Requirements

63. ___________
Requirements
– Rapid Application
Development

64. ___________
Requirements
– Rapid Application
Development
– Flexible schema

65. ___________
Requirements
– Rapid Application
Development
– Flexible schema
– Java Background

66. ___________
Requirements
– Rapid Application
Development
– Flexible schema
– Java Background
– Stateless

68. ________________
Why don’t you just use...?
Framework
Why not?

69. ________________
Why don’t you just use...?
Framework
Why not?
Angular.js Ember.js æж.js Maturity
-

70. ________________
Why don’t you just use...?
Framework
Why not?
Angular.js Ember.js æж.js Maturity
-
Wicket
State

71. ________________
Why don’t you just use...?
Framework
Why not?
Angular.js Ember.js æж.js Maturity
-
Wicket
State
JSF
Model

72. ________________
Why don’t you just use...?
Framework
Why not?
Angular.js Ember.js æж.js Maturity
-
Wicket
State
JSF
Model
Non-java
No java bg

79. Updated Grails to newer minor

81. Web Front
Data Model

83. Remember?

88. Grails means
Gorm!

89. Gorm MongoDB plugin

90. Web Front
Search

91. Search

92. 2 types of search
Full Text Search
Structured Search

93. 2 types of search
Full Text Search
Structured Search

97. ________________
Executive summary
Framework
Why not?

98. ________________
Executive summary
Framework
Lucene/
compass
Why not?
Only embedded,
resource guzzler

99. ________________
Executive summary
Framework
Lucene/
compass
solr
Why not?
Only embedded,
resource guzzler
Bad grails
integration

100. ________________
Executive summary
Framework
Lucene/
compass
solr
sphynx
Why not?
Only embedded,
resource guzzler
Bad grails
integration
No incremental index

104. vs.

105. vs.

106. You ask

107. ElasticSearch answers

109. Additional Services

110. Additional Services
Indexes, Statistics, Logs

115. Also, Redis to the resque

116. Did they just add a 4th nosql?!

119. Additional Services
Documentation

121. DevOps

122. IaaS vs. SaaS

123. Leave it to the Pros

127. SaaS for Download
Service
_________________
Component
SaaS

128. SaaS for Download
Service
_________________
Component
blob storage
SaaS
SL objectstore

129. SaaS for Download
Service
_________________
Component
blob storage
mapping
SaaS
SL objectstore
Cloudant

130. _________________
SaaS for Web and services
Component
SaaS

131. _________________
SaaS for Web and services
Component
Model
SaaS
Mongohq

132. _________________
SaaS for Web and services
Component
Model
Grails
SaaS
Mongohq
N/A

133. _________________
SaaS for Web and services
Component
Model
Grails
ElasticSearch
SaaS
Mongohq
N/A
N/A

134. _________________
SaaS for Web and services
Component
Model
Grails
ElasticSearch
Redis
SaaS
Mongohq
N/A
N/A
N/A

138. Physical vs. Virtual

139. Remember this?

140. __________
Virtualization

141. __________
Virtualization
Pros

142. __________
Virtualization
Pros
– Cheap

143. __________
Virtualization
Pros
– Cheap
– elastic

144. __________
Virtualization
Pros
– Cheap
– elastic
– Volatile

145. __________
Virtualization
Pros
– Cheap
– elastic
– Volatile
cons

146. __________
Virtualization
Pros
– Cheap
– elastic
– Volatile
cons
– Overhead

147. __________
Virtualization
Pros
– Cheap
– elastic
– Volatile
cons
– Overhead
– Tenant, not owner

150. Development Environment

151. Remember?

152. We are liberal

153. We are liberal

154. We are liberal

155. We are liberal

157. The Solution

158. The Solution

159. The Solution

160. The Solution

161. Chef What?

162. Opscode Chef

163. Opscode Chef

164. Opscode Chef

165. Opscode Chef

166. The Solution

167. The Solution

168. Vagrant Who?

169. Vagrant

170. Vagrant

171. Vagrant

172. Vagrant

173. Vagrant

174. Development

175. Development

176. Development

177. Development

178. Ops are part of the DevOps

180. 1. Vagrant boots centos
on virtualbox

181. 1. Vagrant boots centos
on virtualbox
2. Chef installs all db
and service rpms
from private YUM repo

182. 1. Vagrant boots centos
on virtualbox
2. Chef installs all db
and service rpms
from private YUM repo
3. Profit!

183. High Availability
(And Locality)

184. Cluster everything

185. Remember?

187. CDN for Download Server

189. GTD for Web Application

191. Backup
(and Vendor Lock-Out)

195. Snapshots and replicas

196. Monitoring
(Servers, State and Logs)

199. Prevent this:

201. Going to Production…

202. Remember?

204. The Solution

210. All together now

212. ______________
Conclusions time

213. ______________
Conclusions time
– Define Criticality

214. ______________
Conclusions time
– Define Criticality
– Embrace the change

215. ______________
Conclusions time
– Define Criticality
– Embrace the change
– Plan for scale, but be
realistic

216. ______________
Conclusions time
– Define Criticality
– Embrace the change
– Plan for scale, but be
realistic
– Backup everything!

217. No, thank you!