Scale like an ant, distribute the workload - DPC, Amsterdam, 2011

3,492 views
3,421 views

Published on

Many services / applications now a day are ill equipped with handling a sudden rush of popularity, as is often the case on the internet now a days, to a point where the services either become unavailable or unbearably slow.
By taking a chapter from the ant colonies in the wild, where their strength lies in their numbers and the fact that everyone works together towards the same goal, we can apply the same principle to our service by using systems such as
- gearman
- memcache
- daemons
- message queues
- load balancers
and many more, you can achieve greater performance, more redundancy, higher availability and have the ability to scale your services up and down as required easily.
During this talk attendees will be lead through the world of distributed systems and scalability, and shown the how, where and what, of how to take the average application and splitting it into smaller more manageable pieces

Published in: Technology, Travel
0 Comments
8 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
3,492
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
114
Comments
0
Likes
8
Embeds 0
No embeds

No notes for slide

Scale like an ant, distribute the workload - DPC, Amsterdam, 2011

  1. 1. Distribute the workload Helgi Þormar Þorbjörnsson Dutch PHP Conference, Amsterdam, 19th May 2011 Thursday, 19 May 2011
  2. 2. Who am I? Thursday, 19 May 2011
  3. 3. Helgi Thursday, 19 May 2011
  4. 4. VP of Engineering at Orchestra.io Helgi Thursday, 19 May 2011
  5. 5. VP of Engineering at Orchestra.io Developer at PEAR Helgi Thursday, 19 May 2011
  6. 6. VP of Engineering at Orchestra.io Developer at PEAR From Iceland Helgi Thursday, 19 May 2011
  7. 7. VP of Engineering at Orchestra.io Developer at PEAR From Iceland @h on Twitter Helgi Thursday, 19 May 2011
  8. 8. Why Distribute? Thursday, 19 May 2011
  9. 9. Why Distribute? Thursday, 19 May 2011
  10. 10. Why Distribute? Budget Thursday, 19 May 2011
  11. 11. Why Distribute? Budget Efficiency Thursday, 19 May 2011
  12. 12. Why Distribute? Budget Efficiency Perception Thursday, 19 May 2011
  13. 13. Budget Thursday, 19 May 2011
  14. 14. Budget Spend wisely Thursday, 19 May 2011
  15. 15. Budget Spend wisely Commodity servers Thursday, 19 May 2011
  16. 16. Budget Spend wisely Commodity servers Low overhead, high yield Thursday, 19 May 2011
  17. 17. Budget Spend wisely Commodity servers Low overhead, high yield Cloud Computing (EC2) Thursday, 19 May 2011
  18. 18. Efficiency 10 small servers > 1 big Thursday, 19 May 2011
  19. 19. Venue Security Thursday, 19 May 2011
  20. 20. 1000 people can exit quicker through 10 small doors than 1 big Thursday, 19 May 2011
  21. 21. 1000 people can exit quicker through 10 small doors than 1 big Thursday, 19 May 2011
  22. 22. 1000 people can exit quicker through 10 small doors than 1 big Thursday, 19 May 2011
  23. 23. 1000 people can exit quicker through 10 small doors than 1 big Thursday, 19 May 2011
  24. 24. Thursday, 19 May 2011
  25. 25. Thursday, 19 May 2011
  26. 26. Thursday, 19 May 2011
  27. 27. Thursday, 19 May 2011
  28. 28. Perception Thursday, 19 May 2011
  29. 29. Perception Defer intensive processes Thursday, 19 May 2011
  30. 30. Perception Defer intensive processes Give instant feedback Thursday, 19 May 2011
  31. 31. Perception Defer intensive processes Give instant feedback Users keep on browsing Thursday, 19 May 2011
  32. 32. Perception Defer intensive processes Give instant feedback Users keep on browsing Thursday, 19 May 2011
  33. 33. “It all depends on how we look at things, and not how they are in themselves.” - Carl G. Jung Thursday, 19 May 2011
  34. 34. Thursday, 19 May 2011
  35. 35. Chapter from Nature Thursday, 19 May 2011
  36. 36. Ant Colonies Thursday, 19 May 2011
  37. 37. Algorithms Thursday, 19 May 2011
  38. 38. Algorithms Scheduling Thursday, 19 May 2011
  39. 39. Algorithms Scheduling Vehicle Routing Thursday, 19 May 2011
  40. 40. Algorithms Scheduling Vehicle Routing Assignment Thursday, 19 May 2011
  41. 41. Algorithms Scheduling Vehicle Routing Assignment Sets Thursday, 19 May 2011
  42. 42. Algorithms Scheduling Vehicle Routing Assignment Sets Other Thursday, 19 May 2011
  43. 43. Algorithms Scheduling Vehicle Routing Assignment Sets Other Thursday, 19 May 2011
  44. 44. How do ants fit? Thursday, 19 May 2011
  45. 45. How do ants fit? Strength in numbers Thursday, 19 May 2011
  46. 46. How do ants fit? Strength in numbers Work together Thursday, 19 May 2011
  47. 47. How do ants fit? Strength in numbers Work together Size benefits them Thursday, 19 May 2011
  48. 48. Teamwork When faced with a problem they will solve the problem as one. Thursday, 19 May 2011
  49. 49. Thursday, 19 May 2011
  50. 50. Thursday, 19 May 2011
  51. 51. What if they were bigger? Thursday, 19 May 2011
  52. 52. Types of Ants Thursday, 19 May 2011
  53. 53. Types of Ants Military Thursday, 19 May 2011
  54. 54. Types of Ants Military Maids Thursday, 19 May 2011
  55. 55. Types of Ants Military Maids Tunnel diggers Thursday, 19 May 2011
  56. 56. Types of Ants Military Maids Tunnel diggers Food gatherers Thursday, 19 May 2011
  57. 57. How does this map to my application? Thursday, 19 May 2011
  58. 58. Thursday, 19 May 2011
  59. 59. Colony = Application Thursday, 19 May 2011
  60. 60. Colony = Application Ants = Components Thursday, 19 May 2011
  61. 61. Colony = Application Ants = Components Ants do many different types of work to keep their colony running Thursday, 19 May 2011
  62. 62. Architect for Distribution Thursday, 19 May 2011
  63. 63. Characteristics Thursday, 19 May 2011
  64. 64. Characteristics Decoupling Thursday, 19 May 2011
  65. 65. Characteristics Decoupling Elasticity Thursday, 19 May 2011
  66. 66. Characteristics Decoupling Elasticity High Availability Thursday, 19 May 2011
  67. 67. Characteristics Decoupling Elasticity High Availability Concurrency Thursday, 19 May 2011
  68. 68. Decoupling Thursday, 19 May 2011
  69. 69. Application DB API Cache FE Thursday, 19 May 2011
  70. 70. Application DB API Cache FE Thursday, 19 May 2011
  71. 71. ApplicationDB API Cache FE Thursday, 19 May 2011
  72. 72. ApplicationDB API Cache FE Cache Thursday, 19 May 2011
  73. 73. ApplicationDB API Cache FE Cache API Thursday, 19 May 2011
  74. 74. ApplicationDB API Cache FE Cache API API Thursday, 19 May 2011
  75. 75. Elasticity Thursday, 19 May 2011
  76. 76. Cloud Computing Thursday, 19 May 2011
  77. 77. Load Balancing Thursday, 19 May 2011
  78. 78. HA Proxy Nginx My Favourite Thursday, 19 May 2011
  79. 79. Monitoring Thursday, 19 May 2011
  80. 80. When do I need more servers? Thursday, 19 May 2011
  81. 81. Needs to be around from the start! Thursday, 19 May 2011
  82. 82. Keep records Thursday, 19 May 2011
  83. 83. Spot trends Thursday, 19 May 2011
  84. 84. Different types Thursday, 19 May 2011
  85. 85. Different types Hardware Performance Thursday, 19 May 2011
  86. 86. Different types Hardware Performance Software Performance Thursday, 19 May 2011
  87. 87. Different types Hardware Performance Software Performance Availability Thursday, 19 May 2011
  88. 88. Different types Hardware Performance Software Performance Availability Resourcing Thursday, 19 May 2011
  89. 89. Different types Hardware Performance Software Performance Availability Resourcing Thursday, 19 May 2011
  90. 90. Applications Thursday, 19 May 2011
  91. 91. Applications New Relic Thursday, 19 May 2011
  92. 92. Applications New Relic CloudKick Thursday, 19 May 2011
  93. 93. Applications New Relic CloudKick ScoutApp Thursday, 19 May 2011
  94. 94. Applications New Relic CloudKick ScoutApp Nagios Thursday, 19 May 2011
  95. 95. Applications New Relic CloudKick ScoutApp Nagios Cacti Thursday, 19 May 2011
  96. 96. Applications New Relic CloudKick ScoutApp Nagios Cacti Circonus Thursday, 19 May 2011
  97. 97. Automation Thursday, 19 May 2011
  98. 98. Want to sleep easy at night? Thursday, 19 May 2011
  99. 99. Want to sleep easy at night? Go out partying without worrying about getting a phone call? Thursday, 19 May 2011
  100. 100. Plug into your monitoring Thursday, 19 May 2011
  101. 101. Bringing together Monitoring and Elastic behaviour into one beautiful whole! Thursday, 19 May 2011
  102. 102. Add some intelligence to add / remove servers as needed based on current information. Thursday, 19 May 2011
  103. 103. This is why good monitoring is essential or this wouldn’t be possible Thursday, 19 May 2011
  104. 104. Just make sure it doesn’t turn into... Thursday, 19 May 2011
  105. 105. Skynet!! Thursday, 19 May 2011
  106. 106. High Availability Thursday, 19 May 2011
  107. 107. Get a highly available and resilient setup by following a few of those recommendations Thursday, 19 May 2011
  108. 108. Remember, even Google has outages Thursday, 19 May 2011
  109. 109. Benefits Thursday, 19 May 2011
  110. 110. Benefits Easy management Thursday, 19 May 2011
  111. 111. Benefits Easy management Ability to stop / start servers quickly Thursday, 19 May 2011
  112. 112. Benefits Easy management Ability to stop / start servers quickly Responsibilities are separate Thursday, 19 May 2011
  113. 113. Benefits Easy management Ability to stop / start servers quickly Responsibilities are separate Quickly move to a new cluster Thursday, 19 May 2011
  114. 114. Benefits Easy management Ability to stop / start servers quickly Responsibilities are separate Quickly move to a new cluster Reduced risk Thursday, 19 May 2011
  115. 115. What to avoid Thursday, 19 May 2011
  116. 116. Local Sessions Thursday, 19 May 2011
  117. 117. Store sessions in DB / Memcache Solution Thursday, 19 May 2011
  118. 118. Local Memory Thursday, 19 May 2011
  119. 119. Networked Memcache Solution Thursday, 19 May 2011
  120. 120. Local Files Thursday, 19 May 2011
  121. 121. Local Uploads Thursday, 19 May 2011
  122. 122. Writing to /tmp Thursday, 19 May 2011
  123. 123. Store on S3 or a networked FS Solution Thursday, 19 May 2011
  124. 124. Serve up static files from CDNs Solution Thursday, 19 May 2011
  125. 125. Servers can vanish at any given time Thursday, 19 May 2011
  126. 126. Internal APIs Thursday, 19 May 2011
  127. 127. Application S3GFS FS Internal Storage API Thursday, 19 May 2011
  128. 128. Application MySQLMongo Cache Internal DB API Thursday, 19 May 2011
  129. 129. SOA Thursday, 19 May 2011
  130. 130. Service Oriented Architecture Thursday, 19 May 2011
  131. 131. Sort of :-) Thursday, 19 May 2011
  132. 132. Eventually Consistent Thursday, 19 May 2011
  133. 133. CAP Therom Thursday, 19 May 2011
  134. 134. Consistency Availability Partition Tolerance Thursday, 19 May 2011
  135. 135. Consistency All nodes see the same data at the same time Thursday, 19 May 2011
  136. 136. Availability Node failures do not prevent survivors from continuing to operate Thursday, 19 May 2011
  137. 137. Partition Tolerance The system continues to operate despite arbitrary message loss Thursday, 19 May 2011
  138. 138. Consistency Availability Partition Tolerance Thursday, 19 May 2011
  139. 139. Queue Systems Thursday, 19 May 2011
  140. 140. Good for Thursday, 19 May 2011
  141. 141. Good for Image Processing Thursday, 19 May 2011
  142. 142. Good for Image Processing Distributed Logs Thursday, 19 May 2011
  143. 143. Good for Image Processing Distributed Logs Data Mining Thursday, 19 May 2011
  144. 144. Good for Image Processing Distributed Logs Data Mining Mass Emails Thursday, 19 May 2011
  145. 145. Good for Image Processing Distributed Logs Data Mining Mass Emails Intensive transformation Thursday, 19 May 2011
  146. 146. Good for Image Processing Distributed Logs Data Mining Mass Emails Intensive transformation Search Thursday, 19 May 2011
  147. 147. Common Tools Thursday, 19 May 2011
  148. 148. Common Tools Gearman Thursday, 19 May 2011
  149. 149. Common Tools Gearman Hadoop Thursday, 19 May 2011
  150. 150. Common Tools Gearman Hadoop Zero MQ (0MQ) Thursday, 19 May 2011
  151. 151. Common Tools Gearman Hadoop Zero MQ (0MQ) RabbitMQ Thursday, 19 May 2011
  152. 152. Common Tools Gearman Hadoop Zero MQ (0MQ) RabbitMQ And many others! Thursday, 19 May 2011
  153. 153. Gearman Thursday, 19 May 2011
  154. 154. Your Client Code Gearman Client API (C, PHP, Perl, MySQL UDF, ...) Gearman Job Server gearmand Gearman Worker API (C, PHP, Perl, Python, ...) Your Worker Code Your App Gearman Thursday, 19 May 2011
  155. 155. A Story! Thursday, 19 May 2011
  156. 156. Financial Software Thursday, 19 May 2011
  157. 157. 3000+ Clients Thursday, 19 May 2011
  158. 158. Each one has 5 external data sources Thursday, 19 May 2011
  159. 159. Each data source is a web service Thursday, 19 May 2011
  160. 160. Ran every 6 hours every day Thursday, 19 May 2011
  161. 161. Cron 1 2 3 4 5 Job 1 Gearman 1 4 3 2 5 Web Services Processing Thursday, 19 May 2011
  162. 162. But! That wasn’t enough Thursday, 19 May 2011
  163. 163. Job kicked off on login Thursday, 19 May 2011
  164. 164. Another Story! Thursday, 19 May 2011
  165. 165. CloudSplit Thursday, 19 May 2011
  166. 166. Near Real Time Cloud Analytics Thursday, 19 May 2011
  167. 167. Clients install logging agent locally Thursday, 19 May 2011
  168. 168. syslogd Thursday, 19 May 2011
  169. 169. Public API Thursday, 19 May 2011
  170. 170. Multiple Persistent Gearman Servers Thursday, 19 May 2011
  171. 171. Internal DB API Thursday, 19 May 2011
  172. 172. Agent Thursday, 19 May 2011
  173. 173. Agent syslogd Thursday, 19 May 2011
  174. 174. Agent syslogd API Thursday, 19 May 2011
  175. 175. Agent syslogd API Load Balanced Thursday, 19 May 2011
  176. 176. Agent syslogd API Gearman Gearman Load Balanced Thursday, 19 May 2011
  177. 177. Agent syslogd API Gearman Gearman Load Balanced Persistent Thursday, 19 May 2011
  178. 178. Agent syslogd API Gearman Gearman Worker Worker Worker Load Balanced Persistent Thursday, 19 May 2011
  179. 179. Agent syslogd API Gearman Gearman Worker Worker Worker Internal API Load Balanced Persistent Thursday, 19 May 2011
  180. 180. Agent syslogd API Gearman Gearman Worker Worker Worker Internal API Load Balanced Load Balanced Persistent Thursday, 19 May 2011
  181. 181. Agent syslogd API Gearman Gearman CouchDB Worker Worker Worker Internal API Load Balanced Load Balanced Persistent Thursday, 19 May 2011
  182. 182. CouchDB Setup Thursday, 19 May 2011
  183. 183. Write vs Read Thursday, 19 May 2011
  184. 184. Writes Thursday, 19 May 2011
  185. 185. Writes Multi Master setup Thursday, 19 May 2011
  186. 186. Writes Multi Master setup Replicated Thursday, 19 May 2011
  187. 187. Writes Multi Master setup Replicated Deals with writes only Thursday, 19 May 2011
  188. 188. Writes Multi Master setup Replicated Deals with writes only Thursday, 19 May 2011
  189. 189. Reads Thursday, 19 May 2011
  190. 190. Reads Multi Master setup Thursday, 19 May 2011
  191. 191. Reads Multi Master setup Replicated from write cluster Thursday, 19 May 2011
  192. 192. Reads Multi Master setup Replicated from write cluster Slaves handle website requests Thursday, 19 May 2011
  193. 193. Reads Multi Master setup Replicated from write cluster Slaves handle website requests Thursday, 19 May 2011
  194. 194. Heavy Map/Reduce usage for data Thursday, 19 May 2011
  195. 195. Supervisord Thursday, 19 May 2011
  196. 196. phpadvent.org/2009/daemonize- your-php-by-sean-coates Thursday, 19 May 2011
  197. 197. Map/Reduce Thursday, 19 May 2011
  198. 198. Map Thursday, 19 May 2011
  199. 199. Map Master gets a problem to solve Thursday, 19 May 2011
  200. 200. Map Master gets a problem to solve Breaks into multiple sub-problems Thursday, 19 May 2011
  201. 201. Map Master gets a problem to solve Breaks into multiple sub-problems Distributed to multiple workers Thursday, 19 May 2011
  202. 202. Map Master gets a problem to solve Breaks into multiple sub-problems Distributed to multiple workers A worker can take the same steps Thursday, 19 May 2011
  203. 203. Map Master gets a problem to solve Breaks into multiple sub-problems Distributed to multiple workers A worker can take the same steps Answer passed back to Master Thursday, 19 May 2011
  204. 204. Reduce Thursday, 19 May 2011
  205. 205. Reduce Takes in answers from the map workers Thursday, 19 May 2011
  206. 206. Reduce Takes in answers from the map workers Combines together to get an answer Thursday, 19 May 2011
  207. 207. Reduce Takes in answers from the map workers Combines together to get an answer There can be multiple reducers Thursday, 19 May 2011
  208. 208. process petabytes of data in few hours on commodity server farm Thursday, 19 May 2011
  209. 209. CouchDB Thursday, 19 May 2011
  210. 210. CouchDB Thursday, 19 May 2011
  211. 211. CouchDB Highly Concurrent Thursday, 19 May 2011
  212. 212. CouchDB Highly Concurrent Schema free, document based Thursday, 19 May 2011
  213. 213. CouchDB Highly Concurrent Schema free, document based RESTful API Thursday, 19 May 2011
  214. 214. CouchDB Highly Concurrent Schema free, document based RESTful API Map/Reduce Views Thursday, 19 May 2011
  215. 215. CouchDB Highly Concurrent Schema free, document based RESTful API Map/Reduce Views Easy Replication Thursday, 19 May 2011
  216. 216. Hadoop Thursday, 19 May 2011
  217. 217. Hadoop is a framework for running applications on large clusters of commodity hardware. Thursday, 19 May 2011
  218. 218. Thursday, 19 May 2011
  219. 219. The Hadoop framework transparently provides applications both reliability and data motion Thursday, 19 May 2011
  220. 220. Thursday, 19 May 2011
  221. 221. Uses Map/Reduce concept to farm out work Thursday, 19 May 2011
  222. 222. Thursday, 19 May 2011
  223. 223. Distributed FS to handled node failure automagically Thursday, 19 May 2011
  224. 224. Thursday, 19 May 2011
  225. 225. Join 2 datasets together of a significant size Thursday, 19 May 2011
  226. 226. Thursday, 19 May 2011
  227. 227. 500 GB worth of log files with a large location dataset Thursday, 19 May 2011
  228. 228. ØMQ Thursday, 19 May 2011
  229. 229. ØMQ Thursday, 19 May 2011
  230. 230. ØMQ Async Message System Thursday, 19 May 2011
  231. 231. ØMQ Async Message System Thin and lightweight Thursday, 19 May 2011
  232. 232. ØMQ Async Message System Thin and lightweight High Performance Thursday, 19 May 2011
  233. 233. ØMQ Async Message System Thin and lightweight High Performance Simple Thursday, 19 May 2011
  234. 234. ØMQ Async Message System Thin and lightweight High Performance Simple Scalable Thursday, 19 May 2011
  235. 235. Thursday, 19 May 2011
  236. 236. One socket can load balance to multiple end points Thursday, 19 May 2011
  237. 237. Thursday, 19 May 2011
  238. 238. Multiple end points can be funnelled into a single socket Thursday, 19 May 2011
  239. 239. Thursday, 19 May 2011
  240. 240. Handle deployments to multiple servers Thursday, 19 May 2011
  241. 241. Thursday, 19 May 2011
  242. 242. Scale is an example of that Thursday, 19 May 2011
  243. 243. Thursday, 19 May 2011
  244. 244. Mongrel2 is a web server that uses it in a similar way as fastcgi Thursday, 19 May 2011
  245. 245. Thursday, 19 May 2011
  246. 246. Move around text (JSON) and Binary data for real time communication Thursday, 19 May 2011
  247. 247. Thursday, 19 May 2011
  248. 248. Could have replaced syslogd and the external API in my previous example Thursday, 19 May 2011
  249. 249. Code time? :-) Thursday, 19 May 2011
  250. 250. Questions? helgi@orchestra.io Twitter: @h Joind.in: http://joind.in/3212 Thursday, 19 May 2011

×