Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

of

Distribute the workload, PHP Barcelona 2011 Slide 1 Distribute the workload, PHP Barcelona 2011 Slide 2 Distribute the workload, PHP Barcelona 2011 Slide 3 Distribute the workload, PHP Barcelona 2011 Slide 4 Distribute the workload, PHP Barcelona 2011 Slide 5 Distribute the workload, PHP Barcelona 2011 Slide 6 Distribute the workload, PHP Barcelona 2011 Slide 7 Distribute the workload, PHP Barcelona 2011 Slide 8 Distribute the workload, PHP Barcelona 2011 Slide 9 Distribute the workload, PHP Barcelona 2011 Slide 10 Distribute the workload, PHP Barcelona 2011 Slide 11 Distribute the workload, PHP Barcelona 2011 Slide 12 Distribute the workload, PHP Barcelona 2011 Slide 13 Distribute the workload, PHP Barcelona 2011 Slide 14 Distribute the workload, PHP Barcelona 2011 Slide 15 Distribute the workload, PHP Barcelona 2011 Slide 16 Distribute the workload, PHP Barcelona 2011 Slide 17 Distribute the workload, PHP Barcelona 2011 Slide 18 Distribute the workload, PHP Barcelona 2011 Slide 19 Distribute the workload, PHP Barcelona 2011 Slide 20 Distribute the workload, PHP Barcelona 2011 Slide 21 Distribute the workload, PHP Barcelona 2011 Slide 22 Distribute the workload, PHP Barcelona 2011 Slide 23 Distribute the workload, PHP Barcelona 2011 Slide 24 Distribute the workload, PHP Barcelona 2011 Slide 25 Distribute the workload, PHP Barcelona 2011 Slide 26 Distribute the workload, PHP Barcelona 2011 Slide 27 Distribute the workload, PHP Barcelona 2011 Slide 28 Distribute the workload, PHP Barcelona 2011 Slide 29 Distribute the workload, PHP Barcelona 2011 Slide 30 Distribute the workload, PHP Barcelona 2011 Slide 31 Distribute the workload, PHP Barcelona 2011 Slide 32 Distribute the workload, PHP Barcelona 2011 Slide 33 Distribute the workload, PHP Barcelona 2011 Slide 34 Distribute the workload, PHP Barcelona 2011 Slide 35 Distribute the workload, PHP Barcelona 2011 Slide 36 Distribute the workload, PHP Barcelona 2011 Slide 37 Distribute the workload, PHP Barcelona 2011 Slide 38 Distribute the workload, PHP Barcelona 2011 Slide 39 Distribute the workload, PHP Barcelona 2011 Slide 40 Distribute the workload, PHP Barcelona 2011 Slide 41 Distribute the workload, PHP Barcelona 2011 Slide 42 Distribute the workload, PHP Barcelona 2011 Slide 43 Distribute the workload, PHP Barcelona 2011 Slide 44 Distribute the workload, PHP Barcelona 2011 Slide 45 Distribute the workload, PHP Barcelona 2011 Slide 46 Distribute the workload, PHP Barcelona 2011 Slide 47 Distribute the workload, PHP Barcelona 2011 Slide 48 Distribute the workload, PHP Barcelona 2011 Slide 49 Distribute the workload, PHP Barcelona 2011 Slide 50 Distribute the workload, PHP Barcelona 2011 Slide 51 Distribute the workload, PHP Barcelona 2011 Slide 52 Distribute the workload, PHP Barcelona 2011 Slide 53 Distribute the workload, PHP Barcelona 2011 Slide 54 Distribute the workload, PHP Barcelona 2011 Slide 55 Distribute the workload, PHP Barcelona 2011 Slide 56 Distribute the workload, PHP Barcelona 2011 Slide 57 Distribute the workload, PHP Barcelona 2011 Slide 58 Distribute the workload, PHP Barcelona 2011 Slide 59 Distribute the workload, PHP Barcelona 2011 Slide 60 Distribute the workload, PHP Barcelona 2011 Slide 61 Distribute the workload, PHP Barcelona 2011 Slide 62 Distribute the workload, PHP Barcelona 2011 Slide 63 Distribute the workload, PHP Barcelona 2011 Slide 64 Distribute the workload, PHP Barcelona 2011 Slide 65 Distribute the workload, PHP Barcelona 2011 Slide 66 Distribute the workload, PHP Barcelona 2011 Slide 67 Distribute the workload, PHP Barcelona 2011 Slide 68 Distribute the workload, PHP Barcelona 2011 Slide 69 Distribute the workload, PHP Barcelona 2011 Slide 70 Distribute the workload, PHP Barcelona 2011 Slide 71 Distribute the workload, PHP Barcelona 2011 Slide 72 Distribute the workload, PHP Barcelona 2011 Slide 73 Distribute the workload, PHP Barcelona 2011 Slide 74 Distribute the workload, PHP Barcelona 2011 Slide 75 Distribute the workload, PHP Barcelona 2011 Slide 76 Distribute the workload, PHP Barcelona 2011 Slide 77 Distribute the workload, PHP Barcelona 2011 Slide 78 Distribute the workload, PHP Barcelona 2011 Slide 79 Distribute the workload, PHP Barcelona 2011 Slide 80 Distribute the workload, PHP Barcelona 2011 Slide 81 Distribute the workload, PHP Barcelona 2011 Slide 82 Distribute the workload, PHP Barcelona 2011 Slide 83 Distribute the workload, PHP Barcelona 2011 Slide 84 Distribute the workload, PHP Barcelona 2011 Slide 85 Distribute the workload, PHP Barcelona 2011 Slide 86 Distribute the workload, PHP Barcelona 2011 Slide 87 Distribute the workload, PHP Barcelona 2011 Slide 88 Distribute the workload, PHP Barcelona 2011 Slide 89 Distribute the workload, PHP Barcelona 2011 Slide 90 Distribute the workload, PHP Barcelona 2011 Slide 91 Distribute the workload, PHP Barcelona 2011 Slide 92 Distribute the workload, PHP Barcelona 2011 Slide 93 Distribute the workload, PHP Barcelona 2011 Slide 94 Distribute the workload, PHP Barcelona 2011 Slide 95 Distribute the workload, PHP Barcelona 2011 Slide 96 Distribute the workload, PHP Barcelona 2011 Slide 97 Distribute the workload, PHP Barcelona 2011 Slide 98 Distribute the workload, PHP Barcelona 2011 Slide 99 Distribute the workload, PHP Barcelona 2011 Slide 100
Upcoming SlideShare
Phar, The PHP .exe Format
Next
Download to read offline and view in fullscreen.

8 Likes

Share

Download to read offline

Distribute the workload, PHP Barcelona 2011

Download to read offline

Many services / applications now a day are ill equipped with handling a sudden rush of popularity, as is often the case on the internet now a days, to a point where the services either become unavailable or unbearably slow.
By taking a chapter from the ant colonies in the wild, where their strength lies in their numbers and the fact that everyone works together towards the same goal, we can apply the same principle to our
service by using systems such as:
- gearman
- memcache
- daemons
- message queues
- load balancers
and many more, you can achieve greater performance, more redundancy, higher availability and have the ability to scale your services up and down as required easily.
During this talk attendees will be lead through the world of distributed systems and scalability, and shown the how, where and what, of how to take the average application and splitting it into smaller more manageable pieces.

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

Distribute the workload, PHP Barcelona 2011

  1. 1. Distribute the workload Helgi Þormar Þorbjörnsson PHP Barcelona, 29th of October 2011 Saturday, 29 October 11
  2. 2. Who am I? Saturday, 29 October 11
  3. 3. Helgi Co-founded Orchestra.io Developer at PEAR From Iceland @h on Twitter Saturday, 29 October 11
  4. 4. Why Distribute? Efficiency Budget Perception Saturday, 29 October 11
  5. 5. Efficiency 10 small servers > 1 big Saturday, 29 October 11
  6. 6. Budget Spend wisely Commodity servers Cloud Computing (EC2) Saturday, 29 October 11
  7. 7. Perception Defer intensive processes Give instant feedback Users keep on browsing Saturday, 29 October 11
  8. 8. Saturday, 29 October 11
  9. 9. Ant Colonies Saturday, 29 October 11
  10. 10. Teamwork When faced with a problem they will solve the problem as one. Saturday, 29 October 11
  11. 11. Saturday, 29 October 11
  12. 12. Saturday, 29 October 11
  13. 13. Architect for Distribution Saturday, 29 October 11
  14. 14. Characteristics Decoupling Elasticity High Availability Concurrency Saturday, 29 October 11
  15. 15. Decoupling Saturday, 29 October 11
  16. 16. Application DB API Cache FE Saturday, 29 October 11
  17. 17. API Application API DB API Cache FE Cache Saturday, 29 October 11
  18. 18. Elasticity Saturday, 29 October 11
  19. 19. Cloud Computing Saturday, 29 October 11
  20. 20. Load Balancing Saturday, 29 October 11
  21. 21. My Favourite HA Proxy Nginx Saturday, 29 October 11
  22. 22. Monitoring Saturday, 29 October 11
  23. 23. When do I need more servers? Saturday, 29 October 11
  24. 24. Needs to be around from the start! Saturday, 29 October 11
  25. 25. Keep records Saturday, 29 October 11
  26. 26. Spot trends Saturday, 29 October 11
  27. 27. Different types Hardware Performance Software Performance Availability Resourcing Saturday, 29 October 11
  28. 28. Applications New Relic CloudKick ScoutApp Nagios Cacti Circonus Saturday, 29 October 11
  29. 29. Automation Saturday, 29 October 11
  30. 30. Plug into your monitoring Saturday, 29 October 11
  31. 31. Bringing together Monitoring and Elastic behaviour into one beautiful whole! Saturday, 29 October 11
  32. 32. Add some intelligence to add / remove servers as needed based on current information. Saturday, 29 October 11
  33. 33. Just make sure it doesn’t turn into... Saturday, 29 October 11
  34. 34. Skynet!! Saturday, 29 October 11
  35. 35. High Availability Saturday, 29 October 11
  36. 36. Get a highly available and resilient setup by following a few of those recommendations Saturday, 29 October 11
  37. 37. Remember, even Google has outages Saturday, 29 October 11
  38. 38. What to avoid Saturday, 29 October 11
  39. 39. Local Sessions Saturday, 29 October 11
  40. 40. Solution Store sessions in DB / Memcache Saturday, 29 October 11
  41. 41. Local Memory Saturday, 29 October 11
  42. 42. Solution Networked Memcache Saturday, 29 October 11
  43. 43. Local Files Saturday, 29 October 11
  44. 44. Local Uploads Saturday, 29 October 11
  45. 45. Writing to /tmp Saturday, 29 October 11
  46. 46. Solution Store on S3 or a networked FS Saturday, 29 October 11
  47. 47. Solution Serve up static files from CDNs Saturday, 29 October 11
  48. 48. Servers can vanish at any given time Saturday, 29 October 11
  49. 49. Internal APIs Saturday, 29 October 11
  50. 50. Application Internal Storage API GFS S3 FS Saturday, 29 October 11
  51. 51. Application Internal DB API Mongo MySQL Cache Saturday, 29 October 11
  52. 52. SOA Saturday, 29 October 11
  53. 53. Service Oriented Architecture Saturday, 29 October 11
  54. 54. Sort of :-) Saturday, 29 October 11
  55. 55. Eventually Consistent Saturday, 29 October 11
  56. 56. CAP Therom Saturday, 29 October 11
  57. 57. Consistency Availability Partition Tolerance Saturday, 29 October 11
  58. 58. Consistency All nodes see the same data at the same time Saturday, 29 October 11
  59. 59. Availability Node failures do not prevent survivors from continuing to operate Saturday, 29 October 11
  60. 60. Partition Tolerance The system continues to operate despite arbitrary message loss Saturday, 29 October 11
  61. 61. Consistency Availability Partition Tolerance Saturday, 29 October 11
  62. 62. Queue Systems Saturday, 29 October 11
  63. 63. Good for Image Processing Distributed Logs Data Mining Mass Emails Intensive transformation Search Saturday, 29 October 11
  64. 64. Common Tools Gearman Hadoop ZeroMQ RabbitMQ And many others! Saturday, 29 October 11
  65. 65. New York Times 4TB of TIFF files Needed to get 11 million PDF versions Used Hadoop and EC2 100 machines took 24 hours Saturday, 29 October 11
  66. 66. Map/Reduce Saturday, 29 October 11
  67. 67. Map Master gets a problem to solve Breaks into multiple sub-problems Distributed to multiple workers A worker can take the same steps Answer passed back to Master Saturday, 29 October 11
  68. 68. Reduce Takes in answers from the map workers Combines together to get an answer There can be multiple reducers Saturday, 29 October 11
  69. 69. process petabytes of data in few hours on commodity server farm Saturday, 29 October 11
  70. 70. CouchDB Saturday, 29 October 11
  71. 71. CouchDB Highly Concurrent Schema free, document based RESTful API Map/Reduce Views Easy Replication Saturday, 29 October 11
  72. 72. Gearman Saturday, 29 October 11
  73. 73. Your Client Code Gearman Client API (C, PHP, Perl, MySQL UDF, ...) Your App Gearman Job Server Gearman gearmand Gearman Worker API (C, PHP, Perl, Python, ...) Your Worker Code Saturday, 29 October 11
  74. 74. pear.php.net/ net_gearman Saturday, 29 October 11
  75. 75. A Story! Saturday, 29 October 11
  76. 76. Financial Software Saturday, 29 October 11
  77. 77. 3000+ Clients Saturday, 29 October 11
  78. 78. Each one has 5 external data sources Saturday, 29 October 11
  79. 79. Each data source is a web service Saturday, 29 October 11
  80. 80. Ran every 6 hours every day Saturday, 29 October 11
  81. 81. Cron Gearman Web Services Processing 1 Job 1 1 4 5 2 2 5 4 3 3 Saturday, 29 October 11
  82. 82. But! That wasn’t enough Saturday, 29 October 11
  83. 83. Job kicked off on login Saturday, 29 October 11
  84. 84. Supervisord Saturday, 29 October 11
  85. 85. j.mp/supervisord Saturday, 29 October 11
  86. 86. Another Story! Saturday, 29 October 11
  87. 87. CloudSplit Saturday, 29 October 11
  88. 88. Near Real Time Cloud Analytics Saturday, 29 October 11
  89. 89. Clients install logging agent locally Saturday, 29 October 11
  90. 90. syslogd Saturday, 29 October 11
  91. 91. Public API Saturday, 29 October 11
  92. 92. Multiple Persistent Gearman Servers Saturday, 29 October 11
  93. 93. Internal DB API Saturday, 29 October 11
  94. 94. Load Agent syslogd Balanced API Load Balanced Worker Internal API Worker Gearman Worker Gearman CouchDB Persistent Saturday, 29 October 11
  95. 95. CouchDB Setup Saturday, 29 October 11
  96. 96. Write vs Read Saturday, 29 October 11
  97. 97. Writes Multi Master setup Replicated Deals with writes only Saturday, 29 October 11
  98. 98. Reads Multi Master setup Replicated from write cluster Slaves handle website requests Saturday, 29 October 11
  99. 99. Heavy Map/Reduce usage for data Saturday, 29 October 11
  100. 100. Questions? @h helgi@orchestra.io Joind.in: http://joind.in/4326 Saturday, 29 October 11
  • TianpoGao

    Apr. 26, 2015
  • beherca

    Jan. 4, 2014
  • volkanva

    Aug. 18, 2013
  • baresoul

    Jan. 28, 2013
  • manojkumrm

    Nov. 11, 2011
  • grell

    Nov. 7, 2011
  • pilichph

    Nov. 5, 2011
  • medvetalp

    Oct. 31, 2011

Many services / applications now a day are ill equipped with handling a sudden rush of popularity, as is often the case on the internet now a days, to a point where the services either become unavailable or unbearably slow. By taking a chapter from the ant colonies in the wild, where their strength lies in their numbers and the fact that everyone works together towards the same goal, we can apply the same principle to our service by using systems such as: - gearman - memcache - daemons - message queues - load balancers and many more, you can achieve greater performance, more redundancy, higher availability and have the ability to scale your services up and down as required easily. During this talk attendees will be lead through the world of distributed systems and scalability, and shown the how, where and what, of how to take the average application and splitting it into smaller more manageable pieces.

Views

Total views

4,378

On Slideshare

0

From embeds

0

Number of embeds

6

Actions

Downloads

61

Shares

0

Comments

0

Likes

8

×