1. A Brief, Rapid History
of Scaling Instagram
(with a tiny team)
Mike Krieger
QConSF 2013
!

2. Watch the video with slide
synchronization on InfoQ.com!
http://www.infoq.com/presentations
/scaling-instagram
InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month

3. Presented at QCon San Francisco
www.qconsf.com
Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide

4. Hello!

5. Instagram

6. 30 million with 2 eng
(2010-end 2012)

7. 150 million with 6 eng
(2012-now)

8. How we scaled

9. What I would have
done differently

10. What tradeoffs you make when
scaling with that size team

11. (if you can help it, have a
bigger team)

12. perfect solutions

13. survivor bias

14. decision-making process

15. Core principles

16. Do the simplest thing first

17. Every infra moving part is another
“thread” your team has to manage

18. Test & Monitor
Everything

19. This talk
Early days
Year 1: Scaling Up
Year 2: Scaling Out
Year 3-present: Stability, Video, FB

20. Getting Started

21. 2010
2 guys on a pier

25. no one <3s it

26. Focus

28. Mike iOS, Kevin Server

29. Early Stack
Django + Apache mod_wsgi
Postgres
Redis
Gearman
Memcached
Nginx

30. If today
Django + uWSGI
Postgres
Redis
Celery
Memcached
HAproxy

31. Three months later

33. Server planning night
before launch

34. Traction!

35. Year 1: Scaling Up

36. scaling.enable()

37. Single server in LA

38. infra newcomers

39. “What’s a load average?”

40. “Can we get another
server?”

41. Doritos &
Red Bull &
Animal Crackers &
Amazon EC2

42. Underwater on recruiting

43. 2 total engineers

44. Scale "just enough" to get
back to working on app

45. Every weekend was an
accomplishment

46. “Infra is what happens when you’re busy
making other plans”
—Ops Lennon

47. Scaling up DB

48. First bottleneck: disk IO
on old Amazon EBS

49. At the time: ~400 IOPS
max

50. Simple thing first

51. Vertical partitioning

52. Django DB Routers

53. Partitions
Media
Likes
Comments
Everything else

54. PG Replication to
bootstrap nodes

55. Bought us some time

56. Almost no application logic changes
(other than some primary keys)

57. Today: SSD and provisioned
IOPS get you way further

58. Scaling up Redis

59. Purely RAM-bound

60. fork() and COW

61. Vertical partitioning by
data type

62. No easy migration story;
mostly double-writing

63. Replicating + deleting
often leaves fragmentation

64. Chaining replication =
awesome

65. Scaling Memcached

66. Consistent hashing /
ketama

67. Mind that hash function

68. Why not Redis for kv
caching?

69. Slab allocator

70. Config Management
& Deployment

71. fabric + parallel git pull
(sorry GitHub)

72. All AMI based snapshots
for new instances

73. update_ami.sh

74. update_update_ami.sh

75. Should have done Chef
earlier

76. Munin monitoring

77. df, CPU, iowait

78. Ending the year

79. Infra going from 10% time
to 70%

80. Focus on client

81. Testing & monitoring kept
concurrent fires to a minimum

82. Several ticking time
bombs

83. Year 2: Scaling Out

84. App tier

85. Stateless, but plentiful

86. HAProxy
(Dead node detection)

87. Connection limits
everywhere

88. PGBouncer
Homegrown Redis pool

89. Hard to track down kernel
panics

90. Skip rabbit hole; use instancestatus to detect and restart

91. Database Scale Out

92. Out of IO again
(Pre SSDs)

93. Biggest mis-step

94. NoSQL?

95. Call our friends

96. and strangers

97. Theory: partitioning and rebalancing
are hard to get right,
let DB take care of it

98. MongoDB (1.2 at the
time)

99. Double write, shadow
reads

100. Stressing about Primary Key

101. Placed in prod

102. Data loss, segfaults

103. Could have made it
work…

104. …but it would have been
someone’s full time job

105. (and we still only had 3
people)

106. train + rapidly
approaching cliff

107. Sharding in Postgres

108. QCon to the rescue

109. Similar approach to FB
(infra foreshadowing?)

110. Logical partitioning, done
at application level

111. Simplest thing; skipped
abstractions & proxies

112. Pre-split

113. 5000 partitions

114. note to self: pick a power
of 2 next time

115. Postgres "schemas"

116. database
schema
table
columns

117. machineA:
shard0
photos_by_user
shard1
photos_by_user
shard2
photos_by_user
shard3
photos_by_user

118. machineA:
shard0
photos_by_user
shard1
photos_by_user
shard2
photos_by_user
shard3
photos_by_user
machineA’:
shard0
photos_by_user
shard1
photos_by_user
shard2
photos_by_user
shard3
photos_by_user

119. machineA:
shard0
photos_by_user
shard1
photos_by_user
shard2
photos_by_user
shard3
photos_by_user
machineA’:
shard0
photos_by_user
shard1
photos_by_user
shard2
photos_by_user
shard3
photos_by_user

120. Still how we scale PG
today

121. 9.2 upgrade: bucardo to
move schema by schema

122. ID generation

123. Requirements
No extra moving parts
64 bits max
Time ordered
Containing partition key

124. 41 bits: time in millis (41 years of IDs)
13 bits: logical shard ID
10 bits: auto-incrementing
sequence, modulo 1024.

125. This means we can generate 1024
IDs, per shard, per table, per
millisecond

126. Lesson learned

127. A new db is a full time
commitment

128. Be thrifty with your
existing tech

129. = minimize moving parts

130. Scaling configs/host
discovery

131. ZooKeeper or DNS
server?

132. No team to maintain

133. /etc/hosts

134. ec2tag KnownAs

135. fab update_etc_hosts
(generates, deploys)

136. Limited: dead host
failover, etc

137. But zero additional infra, got
the job done, easy to debug

138. Monitoring

139. Munin: too coarse, too
hard to add new stats

140. StatsD & Graphite

141. Simple tech

142. statsd.timer
statsd.incr

143. Step change in developer
attitude towards stats

144. <5 min from wanting to
measure, to having a graph

145. 580 statsd counters
164 statsd timers

146. Ending the year

147. Launched Android

148. (doubling all of our infra, most of
which was now horizontally scalable)

149. Doubled active users in
< 6 months

150. Finally, slowly, building up
team

151. Year 3+: Stability,
Video, FB

152. Scale tools to match
team

153. Deployment &
Config Management

154. Finally 100% on Chef

155. Simple thing first: knife
and chef-solo

156. Every new hire learns
Chef

157. Code deploys

158. Many rollouts a day

159. Continuous integration

160. But push still needs a
driver

161. "Ops Lock"

162. Humans are terrible
distributed locking systems

163. Sauron

164. Redis-enforced locks

165. Rollout / major config changes
/ live deployment tracking

166. Extracting approach
Hit issue
Develop manual approach
Build tools to improve manual / hands on
approach
Replace manual with automated system

167. Monitoring

168. Munin finally broke

169. Ganglia for graphing

170. Sensu for alerting
(http://sensuapp.org)

171. StatsD/Graphite still
chugging along

172. waittime: lightweight slow
component tracking

173. s = time.time()
# do work
statsd.incr("waittime.VIEWNAME.C
OMPONENT", time.time() - s)

174. asPercent()

176. Feeds and Inboxes

177. Redis

178. In memory requirement

179. Every churned or inactive user

181. Inbox moved to
Cassandra

182. 1000:1 write/read

183. Prereq: having rbranson,
ex-DataStax

184. C* cluster is 20% of the
size of Redis one

185. Main feed (timeline) still in
Redis

186. Knobs

187. Dynamic ramp-ups and
config

188. Previously: required
deploy

189. knobs.py

190. Only ints

191. Stored in Redis

192. Refreshed every 30s

193. knobs.get(feature_name,
default)

194. Uses
Incremental feature rollouts
Dynamic page sizing (shedding load)
Feature killswitches

196. As more teams around
FB contribute

197. Decouple deploy from
feature rollout

198. Video

199. Launch a top 10 video site on day
1 with team of 6 engineers,
in less than 2 months

200. Reuse what we know

201. Avoid magic middleware

202. VXCode

203. Separate from main App
servers

204. Django-based

205. server-side transcoding

206. ZooKeeper ephemeral
nodes for detection

207. (finally worth it / doable to
deploy ZK)

208. EC2 autoscaling

209. Priority list for clients

210. Transcoding tier is
completely stateless

211. statsd waterfall

212. holding area for
debugging bad videos

213. 5 million videos in first day
40h of video / hour

214. (other than perf improvements we’ve
basically not touched it since launch)

215. FB

216. Where can we skip a few
years?

217. (at our own pace)

218. Spam fighting

219. re.compile(‘f[o0][1l][o0]w’)

220. Simplest thing did not last

221. Generic features +
machine learning

222. Hadoop + Hive + Presto

223. "I wonder how they..."

224. Two-way exchange

225. 2010 vintage infra

226. #1 impact: recruiting

227. Backend team: >10
people now

228. Wrap up

229. Core principles

230. Do the simplest thing first

231. Every infra moving part is another
“thread” your team has to manage

232. Test & Monitor
Everything

233.