![re.compile(‘f[o0][1l][o0]w’)](https://image.slidesharecdn.com/untitled-131209030902-phpapp01/85/How-a-Small-Team-Scales-Instagram-219-320.jpg)
Uploaded byC4Media
How a Small Team Scales Instagram
The document outlines a presentation by Mike Krieger on the scaling of Instagram from its early days with a minimal engineering team to handling millions of users. It discusses key strategies and principles for scaling infrastructure efficiently, including database optimization, deployment management, and team dynamics. The talk emphasizes the importance of simplicity in solutions and learning from operational challenges faced during Instagram's growth phase.
More Related Content
Similar to How a Small Team Scales Instagram
How a Small Team Scales Instagram
- 1. A Brief, RapidHistory of Scaling Instagram (with a tiny team) Mike Krieger QConSF 2013 !
- 2. Watch the videowith slide synchronization on InfoQ.com! http://www.infoq.com/presentations /scaling-instagram InfoQ.com: News & Community Site • 750,000 unique visitors/month • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • News 15-20 / week • Articles 3-4 / week • Presentations (videos) 12-15 / week • Interviews 2-3 / week • Books 1 / month
- 3. Presented at QConSan Francisco www.qconsf.com Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide
- 4.
- 5.
- 6. 30 million with2 eng (2010-end 2012)
- 7. 150 million with6 eng (2012-now)
- 8.
- 9. What I wouldhave done differently
- 10. What tradeoffs youmake when scaling with that size team
- 11. (if you canhelp it, have a bigger team)
- 12.
- 13.
- 14.
- 15.
- 16. Do the simplestthing first
- 17. Every infra movingpart is another “thread” your team has to manage
- 18.
- 19. This talk Early days Year1: Scaling Up Year 2: Scaling Out Year 3-present: Stability, Video, FB
- 20.
- 21. 2010 2 guys ona pier
- 25.
- 26.
- 28. Mike iOS, KevinServer
- 29. Early Stack Django +Apache mod_wsgi Postgres Redis Gearman Memcached Nginx
- 30. If today Django +uWSGI Postgres Redis Celery Memcached HAproxy
- 31.
- 33.
- 34.
- 35.
- 36.
- 37.
- 38.
- 39. “What’s a loadaverage?”
- 40. “Can we getanother server?”
- 41. Doritos & Red Bull& Animal Crackers & Amazon EC2
- 42.
- 43.
- 44. Scale "just enough"to get back to working on app
- 45. Every weekend wasan accomplishment
- 46. “Infra is whathappens when you’re busy making other plans” —Ops Lennon
- 47.
- 48. First bottleneck: diskIO on old Amazon EBS
- 49. At the time:~400 IOPS max
- 50.
- 51.
- 52.
- 53.
- 54.
- 55. Bought us sometime
- 56. Almost no applicationlogic changes (other than some primary keys)
- 57. Today: SSD andprovisioned IOPS get you way further
- 58.
- 59.
- 60.
- 61.
- 62. No easy migrationstory; mostly double-writing
- 63. Replicating + deleting oftenleaves fragmentation
- 64.
- 65.
- 66.
- 67. Mind that hashfunction
- 68. Why not Redisfor kv caching?
- 69.
- 70.
- 71. fabric + parallelgit pull (sorry GitHub)
- 72. All AMI basedsnapshots for new instances
- 73.
- 74.
- 75. Should have doneChef earlier
- 76.
- 77.
- 78.
- 79. Infra going from10% time to 70%
- 80.
- 81. Testing & monitoringkept concurrent fires to a minimum
- 82.
- 83.
- 84.
- 85.
- 86.
- 87.
- 88.
- 89. Hard to trackdown kernel panics
- 90. Skip rabbit hole;use instancestatus to detect and restart
- 91.
- 92. Out of IOagain (Pre SSDs)
- 93.
- 94.
- 95.
- 96.
- 97. Theory: partitioning andrebalancing are hard to get right, let DB take care of it
- 98. MongoDB (1.2 atthe time)
- 99.
- 100.
- 101.
- 102.
- 103. Could have madeit work…
- 104. …but it wouldhave been someone’s full time job
- 105. (and we stillonly had 3 people)
- 106.
- 107.
- 108. QCon to therescue
- 109. Similar approach toFB (infra foreshadowing?)
- 110. Logical partitioning, done atapplication level
- 111. Simplest thing; skipped abstractions& proxies
- 112.
- 113.
- 114. note to self:pick a power of 2 next time
- 115.
- 116.
- 117.
- 118.
- 119.
- 120. Still how wescale PG today
- 121. 9.2 upgrade: bucardoto move schema by schema
- 122.
- 123. Requirements No extra movingparts 64 bits max Time ordered Containing partition key
- 124. 41 bits: timein millis (41 years of IDs) 13 bits: logical shard ID 10 bits: auto-incrementing sequence, modulo 1024.
- 125. This means wecan generate 1024 IDs, per shard, per table, per millisecond
- 126.
- 127. A new dbis a full time commitment
- 128. Be thrifty withyour existing tech
- 129. = minimize movingparts
- 130.
- 131.
- 132. No team tomaintain
- 133.
- 134.
- 135.
- 136.
- 137. But zero additionalinfra, got the job done, easy to debug
- 138.
- 139. Munin: too coarse,too hard to add new stats
- 140.
- 141.
- 142.
- 143. Step change indeveloper attitude towards stats
- 144. <5 min fromwanting to measure, to having a graph
- 145. 580 statsd counters 164statsd timers
- 146.
- 147.
- 148. (doubling all ofour infra, most of which was now horizontally scalable)
- 149. Doubled active usersin < 6 months
- 150. Finally, slowly, buildingup team
- 151.
- 152. Scale tools tomatch team
- 153.
- 154. Finally 100% onChef
- 155. Simple thing first:knife and chef-solo
- 156. Every new hirelearns Chef
- 157.
- 158.
- 159.
- 160. But push stillneeds a driver
- 161.
- 162. Humans are terrible distributedlocking systems
- 163.
- 164.
- 165. Rollout / majorconfig changes / live deployment tracking
- 166. Extracting approach Hit issue Developmanual approach Build tools to improve manual / hands on approach Replace manual with automated system
- 167.
- 168.
- 169.
- 170.
- 171.
- 172.
- 173. s = time.time() #do work statsd.incr("waittime.VIEWNAME.C OMPONENT", time.time() - s)
- 174.
- 176.
- 177.
- 178.
- 179. Every churned orinactive user
- 181.
- 182.
- 183.
- 184. C* cluster is20% of the size of Redis one
- 185. Main feed (timeline)still in Redis
- 186.
- 187.
- 188.
- 189.
- 190.
- 191.
- 192.
- 193.
- 194. Uses Incremental feature rollouts Dynamicpage sizing (shedding load) Feature killswitches
- 196. As more teamsaround FB contribute
- 197. Decouple deploy from featurerollout
- 198.
- 199. Launch a top10 video site on day 1 with team of 6 engineers, in less than 2 months
- 200. Reuse what weknow
- 201.
- 202.
- 203. Separate from mainApp servers
- 204.
- 205.
- 206. ZooKeeper ephemeral nodes fordetection
- 207. (finally worth it/ doable to deploy ZK)
- 208.
- 209. Priority list forclients
- 210. Transcoding tier is completelystateless
- 211.
- 212. holding area for debuggingbad videos
- 213. 5 million videosin first day 40h of video / hour
- 214. (other than perfimprovements we’ve basically not touched it since launch)
- 215.
- 216. Where can weskip a few years?
- 217. (at our ownpace)
- 218.
- 219.
- 220. Simplest thing didnot last
- 221. Generic features + machinelearning
- 222. Hadoop + Hive+ Presto
- 223. "I wonder howthey..."
- 224.
- 225.
- 226.
- 227.
- 228.
- 229.
- 230. Do the simplestthing first
- 231. Every infra movingpart is another “thread” your team has to manage
- 232.
- 233.
- 234. Recruit way earlierthan you'd think
- 235.
- 236. Rocketship scaling hasbeen (somewhat) democratized
- 237. Huge thanks toIG Eng Team
- 238.
- 239. Watch the videowith slide synchronization on InfoQ.com! http://www.infoq.com/presentations/scalinginstagram