Scaling Instagram         AirBnB Tech Talk 2012                  Mike Krieger                     Instagram
me-   Co-founder, Instagram-   Previously: UX & Front-end    @ Meebo-   Stanford HCI BS/MS-   @mikeyk on everything
communicating andsharing in the real world
30+ million users in less    than 2 years
the story of how we      scaled it
a brief tangent
the beginning
Text
2 product guys
no real back-end   experience
analytics & python @        meebo
CouchDB
CrimeDesk SF
let’s get hacking
good components in   place early on
...but were hosted on a     single machine    somewhere in LA
less powerful than my    MacBook Pro
okay, we launched.   now what?
25k signups in the first         day
everything is on fire!
best & worst day of our      lives so far
load was through the        roof
first culprit?
favicon.ico
404-ing on Django,causing tons of errors
lesson #1: don’t forget     your favicon
real lesson #1: most of   your initial scaling  problems won’t be       glamorous
favicon
ulimit -n
memcached -t 4
prefork/postfork
friday rolls around
not slowing down
let’s move to EC2.
scaling = replacing allcomponents of a car  while driving it at       100mph
since...
“"canonical [architecture]of an early stage startup       in this era."  (HighScalability.com)
Nginx &Redis &Postgres &Django.
Nginx & HAProxy &Redis & Memcached &Postgres & Gearman &Django.
24h Ops
our philosophy
1 simplicity
2 optimize forminimal operational     burden
3 instrument everything
walkthrough:1 scaling the database2 choosing technology3 staying nimble4 scaling for android
1 scaling the db
early days
django ORM, postgresql
why pg? postgis.
moved db to its own    machine
but photos kept growing     and growing...
...and only 68GB of   RAM on biggest   machine in EC2
so what now?
vertical partitioning
django db routers make     it pretty easy
def db_for_read(self, model):  if app_label == photos:    return photodb
...once you untangle all    your foreign key      relationships
a few months later...
photosdb > 60GB
what now?
horizontal partitioning!
aka: sharding
“surely we’ll have hiredsomeone experiencedbefore we actually need        to shard”
you don’t get to choosewhen scaling challenges      come up
evaluated solutions
at the time, none wereup to task of being our      primary DB
did in Postgres itself
what’s painful about    sharding?
1 data retrieval
hard to know what yourprimary access patternswill be w/out any usage
in most cases, user ID
2 what happens ifone of your shards  gets too big?
in range-based schemes (like MongoDB), you split
A-H: shard0I-Z: shard1
A-D:   shard0E-H:   shard2I-P:   shard1Q-Z:   shard2
downsides (especially on    EC2): disk IO
instead, we pre-split
many many many(thousands) of logical       shards
that map to fewer  physical ones
// 8 logical shards on 2 machinesuser_id % 8 = logical shardlogical shards -> physical shard map{    0:   A,   1:   A,    ...
// 8 logical shards on 2 4 machinesuser_id % 8 = logical shardlogical shards -> physical shard map{    0:   A,   1:   A,  ...
little known but awesome    PG feature: schemas
not “columns” schema
- database:  - schema:    - table:      - columns
machineA:  shard0    photos_by_user  shard1    photos_by_user  shard2    photos_by_user  shard3    photos_by_user
machineA:            machineA’:  shard0               shard0    photos_by_user       photos_by_user  shard1               ...
machineA:            machineC:  shard0               shard0    photos_by_user       photos_by_user  shard1               s...
can do this as long asyou have more logical shards than physical        ones
lesson: take tech/toolsyou know and try first toadapt them into a simple         solution
2 which tools where?
where to cache /otherwise denormalize        data
we <3 redis
what happens when a user posts a photo?
1 user uploads photowith (optional) caption     and location
2 synchronous write tothe media database for      that user
3 queues!
3a if geotagged, async worker POSTs to Solr
3b follower delivery
can’t have every user who loads her timelinelook up all their followers  and then their photos
instead, everyone gets their own list in Redis
media ID is pushed onto a list for every personwho’s following this user
Redis is awesome forthis; rapid insert, rapid        subsets
when time to render afeed, we take small # of  IDs, go look up info in       memcached
Redis is great for...
data structures that are  relatively bounded
(don’t tie yourself to a solution where your in-memory DB is your main        data store)
caching complex objectswhere you want to more       than GET
ex: counting, sub- ranges, testing   membership
especially when TaylorSwift posts live from the        CMAs
follow graph
v1: simple DB table(source_id, target_id,        status)
who do I follow? who follows me?  do I follow X?does X follow me?
DB was busy, so westarted storing parallel   version in Redis
follow_all(300 item list)
inconsistency
extra logic
so much extra logic
exposing your support team to the idea of  cache invalidation
redesign took a page  from twitter’s book
PG can handle tens ofthousands of requests, very light memcached         caching
two takeaways
1 have a versatilecomplement to your core data storage (like Redis)
2 try not to have twotools trying to do the      same job
3 staying nimble
2010: 2 engineers
2011: 3 engineers
2012: 5 engineers
scarcity -> focus
engineer solutions that you’re not constantly returning to because       they broke
1 extensive unit-tests and functional tests
2 keep it DRY
3 loose coupling using notifications / signals
4 do most of our work inPython, drop to C when      necessary
5 frequent code reviews,  pull requests to keep   things in the ‘shared           brain’
6 extensive monitoring
munin
statsd
“how is the system right         now?”
“how does this compare  to historical trends?”
scaling for android
1 million new users in 12           hours
great tools that enable easy read scalability
redis: slaveof <host> <port>
our Redis frameworkassumes 0+ readslaves
tight iteration loops
statsd & pgfouine
know where you canshed load if needed
(e.g. shorter feeds)
if you’re tempted toreinvent the wheel...
don’t.
“our app serverssometimes kernel panic     under load”
...
“what if we write amonitoring daemon...”
wait! this is exactly what HAProxy is great at
surround yourself with awesome advisors
culture of opennessaround engineering
give back; e.g.   node2dm
focus on making what   you have better
“fast, beautiful photo       sharing”
“can we make all of ourrequests 50% the time?”
staying nimble = remind   yourself of what’s        important
your users around theworld don’t care that you  wrote your own DB
wrapping up
unprecedented times
2 backend engineerscan scale a system to  30+ million users
key word = simplicity
cleanest solution with the fewest moving parts as        possible
don’t over-optimize orexpect to know ahead of time how site will scale
don’t think “someoneelse will join & take care           of this”
will happen sooner than   you think; surround    yourself with great         advisors
when adding software tostack: only if you have to,optimizing for operational         simplicity
few, if any, unsolvablescaling challenges for a      social startup
have fun
Scaling Instagram
Scaling Instagram
Scaling Instagram
Scaling Instagram
Scaling Instagram
Scaling Instagram
Scaling Instagram
Scaling Instagram
Scaling Instagram
Scaling Instagram
Scaling Instagram
Scaling Instagram
Upcoming SlideShare
Loading in...5
×

Scaling Instagram

88,365

Published on

Instagram 扩展性实践

Published in: Technology
7 Comments
262 Likes
Statistics
Notes
No Downloads
Views
Total Views
88,365
On Slideshare
0
From Embeds
0
Number of Embeds
62
Actions
Shares
0
Downloads
1,032
Comments
7
Likes
262
Embeds 0
No embeds

No notes for slide

Scaling Instagram

  1. 1. Scaling Instagram AirBnB Tech Talk 2012 Mike Krieger Instagram
  2. 2. me- Co-founder, Instagram- Previously: UX & Front-end @ Meebo- Stanford HCI BS/MS- @mikeyk on everything
  3. 3. communicating andsharing in the real world
  4. 4. 30+ million users in less than 2 years
  5. 5. the story of how we scaled it
  6. 6. a brief tangent
  7. 7. the beginning
  8. 8. Text
  9. 9. 2 product guys
  10. 10. no real back-end experience
  11. 11. analytics & python @ meebo
  12. 12. CouchDB
  13. 13. CrimeDesk SF
  14. 14. let’s get hacking
  15. 15. good components in place early on
  16. 16. ...but were hosted on a single machine somewhere in LA
  17. 17. less powerful than my MacBook Pro
  18. 18. okay, we launched. now what?
  19. 19. 25k signups in the first day
  20. 20. everything is on fire!
  21. 21. best & worst day of our lives so far
  22. 22. load was through the roof
  23. 23. first culprit?
  24. 24. favicon.ico
  25. 25. 404-ing on Django,causing tons of errors
  26. 26. lesson #1: don’t forget your favicon
  27. 27. real lesson #1: most of your initial scaling problems won’t be glamorous
  28. 28. favicon
  29. 29. ulimit -n
  30. 30. memcached -t 4
  31. 31. prefork/postfork
  32. 32. friday rolls around
  33. 33. not slowing down
  34. 34. let’s move to EC2.
  35. 35. scaling = replacing allcomponents of a car while driving it at 100mph
  36. 36. since...
  37. 37. “"canonical [architecture]of an early stage startup in this era." (HighScalability.com)
  38. 38. Nginx &Redis &Postgres &Django.
  39. 39. Nginx & HAProxy &Redis & Memcached &Postgres & Gearman &Django.
  40. 40. 24h Ops
  41. 41. our philosophy
  42. 42. 1 simplicity
  43. 43. 2 optimize forminimal operational burden
  44. 44. 3 instrument everything
  45. 45. walkthrough:1 scaling the database2 choosing technology3 staying nimble4 scaling for android
  46. 46. 1 scaling the db
  47. 47. early days
  48. 48. django ORM, postgresql
  49. 49. why pg? postgis.
  50. 50. moved db to its own machine
  51. 51. but photos kept growing and growing...
  52. 52. ...and only 68GB of RAM on biggest machine in EC2
  53. 53. so what now?
  54. 54. vertical partitioning
  55. 55. django db routers make it pretty easy
  56. 56. def db_for_read(self, model): if app_label == photos: return photodb
  57. 57. ...once you untangle all your foreign key relationships
  58. 58. a few months later...
  59. 59. photosdb > 60GB
  60. 60. what now?
  61. 61. horizontal partitioning!
  62. 62. aka: sharding
  63. 63. “surely we’ll have hiredsomeone experiencedbefore we actually need to shard”
  64. 64. you don’t get to choosewhen scaling challenges come up
  65. 65. evaluated solutions
  66. 66. at the time, none wereup to task of being our primary DB
  67. 67. did in Postgres itself
  68. 68. what’s painful about sharding?
  69. 69. 1 data retrieval
  70. 70. hard to know what yourprimary access patternswill be w/out any usage
  71. 71. in most cases, user ID
  72. 72. 2 what happens ifone of your shards gets too big?
  73. 73. in range-based schemes (like MongoDB), you split
  74. 74. A-H: shard0I-Z: shard1
  75. 75. A-D: shard0E-H: shard2I-P: shard1Q-Z: shard2
  76. 76. downsides (especially on EC2): disk IO
  77. 77. instead, we pre-split
  78. 78. many many many(thousands) of logical shards
  79. 79. that map to fewer physical ones
  80. 80. // 8 logical shards on 2 machinesuser_id % 8 = logical shardlogical shards -> physical shard map{ 0: A, 1: A, 2: A, 3: A, 4: B, 5: B, 6: B, 7: B}
  81. 81. // 8 logical shards on 2 4 machinesuser_id % 8 = logical shardlogical shards -> physical shard map{ 0: A, 1: A, 2: C, 3: C, 4: B, 5: B, 6: D, 7: D}
  82. 82. little known but awesome PG feature: schemas
  83. 83. not “columns” schema
  84. 84. - database: - schema: - table: - columns
  85. 85. machineA: shard0 photos_by_user shard1 photos_by_user shard2 photos_by_user shard3 photos_by_user
  86. 86. machineA: machineA’: shard0 shard0 photos_by_user photos_by_user shard1 shard1 photos_by_user photos_by_user shard2 shard2 photos_by_user photos_by_user shard3 shard3 photos_by_user photos_by_user
  87. 87. machineA: machineC: shard0 shard0 photos_by_user photos_by_user shard1 shard1 photos_by_user photos_by_user shard2 shard2 photos_by_user photos_by_user shard3 shard3 photos_by_user photos_by_user
  88. 88. can do this as long asyou have more logical shards than physical ones
  89. 89. lesson: take tech/toolsyou know and try first toadapt them into a simple solution
  90. 90. 2 which tools where?
  91. 91. where to cache /otherwise denormalize data
  92. 92. we <3 redis
  93. 93. what happens when a user posts a photo?
  94. 94. 1 user uploads photowith (optional) caption and location
  95. 95. 2 synchronous write tothe media database for that user
  96. 96. 3 queues!
  97. 97. 3a if geotagged, async worker POSTs to Solr
  98. 98. 3b follower delivery
  99. 99. can’t have every user who loads her timelinelook up all their followers and then their photos
  100. 100. instead, everyone gets their own list in Redis
  101. 101. media ID is pushed onto a list for every personwho’s following this user
  102. 102. Redis is awesome forthis; rapid insert, rapid subsets
  103. 103. when time to render afeed, we take small # of IDs, go look up info in memcached
  104. 104. Redis is great for...
  105. 105. data structures that are relatively bounded
  106. 106. (don’t tie yourself to a solution where your in-memory DB is your main data store)
  107. 107. caching complex objectswhere you want to more than GET
  108. 108. ex: counting, sub- ranges, testing membership
  109. 109. especially when TaylorSwift posts live from the CMAs
  110. 110. follow graph
  111. 111. v1: simple DB table(source_id, target_id, status)
  112. 112. who do I follow? who follows me? do I follow X?does X follow me?
  113. 113. DB was busy, so westarted storing parallel version in Redis
  114. 114. follow_all(300 item list)
  115. 115. inconsistency
  116. 116. extra logic
  117. 117. so much extra logic
  118. 118. exposing your support team to the idea of cache invalidation
  119. 119. redesign took a page from twitter’s book
  120. 120. PG can handle tens ofthousands of requests, very light memcached caching
  121. 121. two takeaways
  122. 122. 1 have a versatilecomplement to your core data storage (like Redis)
  123. 123. 2 try not to have twotools trying to do the same job
  124. 124. 3 staying nimble
  125. 125. 2010: 2 engineers
  126. 126. 2011: 3 engineers
  127. 127. 2012: 5 engineers
  128. 128. scarcity -> focus
  129. 129. engineer solutions that you’re not constantly returning to because they broke
  130. 130. 1 extensive unit-tests and functional tests
  131. 131. 2 keep it DRY
  132. 132. 3 loose coupling using notifications / signals
  133. 133. 4 do most of our work inPython, drop to C when necessary
  134. 134. 5 frequent code reviews, pull requests to keep things in the ‘shared brain’
  135. 135. 6 extensive monitoring
  136. 136. munin
  137. 137. statsd
  138. 138. “how is the system right now?”
  139. 139. “how does this compare to historical trends?”
  140. 140. scaling for android
  141. 141. 1 million new users in 12 hours
  142. 142. great tools that enable easy read scalability
  143. 143. redis: slaveof <host> <port>
  144. 144. our Redis frameworkassumes 0+ readslaves
  145. 145. tight iteration loops
  146. 146. statsd & pgfouine
  147. 147. know where you canshed load if needed
  148. 148. (e.g. shorter feeds)
  149. 149. if you’re tempted toreinvent the wheel...
  150. 150. don’t.
  151. 151. “our app serverssometimes kernel panic under load”
  152. 152. ...
  153. 153. “what if we write amonitoring daemon...”
  154. 154. wait! this is exactly what HAProxy is great at
  155. 155. surround yourself with awesome advisors
  156. 156. culture of opennessaround engineering
  157. 157. give back; e.g. node2dm
  158. 158. focus on making what you have better
  159. 159. “fast, beautiful photo sharing”
  160. 160. “can we make all of ourrequests 50% the time?”
  161. 161. staying nimble = remind yourself of what’s important
  162. 162. your users around theworld don’t care that you wrote your own DB
  163. 163. wrapping up
  164. 164. unprecedented times
  165. 165. 2 backend engineerscan scale a system to 30+ million users
  166. 166. key word = simplicity
  167. 167. cleanest solution with the fewest moving parts as possible
  168. 168. don’t over-optimize orexpect to know ahead of time how site will scale
  169. 169. don’t think “someoneelse will join & take care of this”
  170. 170. will happen sooner than you think; surround yourself with great advisors
  171. 171. when adding software tostack: only if you have to,optimizing for operational simplicity
  172. 172. few, if any, unsolvablescaling challenges for a social startup
  173. 173. have fun
  1. ¿Le ha llamado la atención una diapositiva en particular?

    Recortar diapositivas es una manera útil de recopilar información importante para consultarla más tarde.

×