Downloaded 61 times
Uploaded byHelgi Þormar Þorbjörnsson
Distribute the workload, PHP Barcelona 2011
The document discusses the importance of distributing workloads to increase efficiency, manage budget effectively, and enhance user experience. It covers various aspects of distributed systems, including characteristics like decoupling, elasticity, and high availability, and reviews tools and strategies for implementing such systems, like load balancing, queue systems, and map/reduce methods. The author shares practical examples and applications of distributed computing in real-world scenarios, specifically in cloud analytics and financial software management.
More Related Content
Similar to Distribute the workload, PHP Barcelona 2011
Distribute the workload, PHP Barcelona 2011
- 1. Distribute the workload Helgi Þormar Þorbjörnsson PHP Barcelona, 29th of October 2011 Saturday, 29 October 11
- 2. Who am I? Saturday,29 October 11
- 3. Helgi Co-founded Orchestra.io Developer at PEAR From Iceland @h on Twitter Saturday, 29 October 11
- 4. Why Distribute? Efficiency Budget Perception Saturday, 29 October 11
- 5. Efficiency 10 small servers > 1 big Saturday, 29 October 11
- 6. Budget Spend wisely Commodity servers Cloud Computing (EC2) Saturday, 29 October 11
- 7. Perception Defer intensive processes Give instant feedback Users keep on browsing Saturday, 29 October 11
- 8.
- 9. Ant Colonies Saturday, 29October 11
- 10. Teamwork When faced with a problem they will solve the problem as one. Saturday, 29 October 11
- 11.
- 12.
- 13. Architect for Distribution Saturday, 29 October 11
- 14. Characteristics Decoupling Elasticity High Availability Concurrency Saturday, 29 October 11
- 15.
- 16. Application DB API Cache FE Saturday, 29 October 11
- 17. API Application API DB API Cache FE Cache Saturday, 29 October 11
- 18.
- 19. Cloud Computing Saturday, 29October 11
- 20. Load Balancing Saturday, 29October 11
- 21. My Favourite HA Proxy Nginx Saturday, 29 October 11
- 22.
- 23. When do Ineed more servers? Saturday, 29 October 11
- 24. Needs to bearound from the start! Saturday, 29 October 11
- 25. Keep records Saturday, 29October 11
- 26. Spot trends Saturday, 29October 11
- 27. Different types Hardware Performance Software Performance Availability Resourcing Saturday, 29 October 11
- 28. Applications New Relic CloudKick ScoutApp Nagios Cacti Circonus Saturday, 29 October 11
- 29.
- 30. Plug into your monitoring Saturday, 29 October 11
- 31. Bringing together Monitoring and Elastic behaviour into one beautiful whole! Saturday, 29 October 11
- 32. Add some intelligenceto add / remove servers as needed based on current information. Saturday, 29 October 11
- 33. Just make sureit doesn’t turn into... Saturday, 29 October 11
- 34.
- 35. High Availability Saturday, 29October 11
- 36. Get a highlyavailable and resilient setup by following a few of those recommendations Saturday, 29 October 11
- 37. Remember, even Googlehas outages Saturday, 29 October 11
- 38. What to avoid Saturday,29 October 11
- 39. Local Sessions Saturday, 29October 11
- 40. Solution Store sessions in DB / Memcache Saturday, 29 October 11
- 41. Local Memory Saturday, 29October 11
- 42. Solution Networked Memcache Saturday, 29 October 11
- 43. Local Files Saturday, 29October 11
- 44. Local Uploads Saturday, 29October 11
- 45. Writing to /tmp Saturday,29 October 11
- 46. Solution Store on S3 or a networked FS Saturday, 29 October 11
- 47. Solution Serve up static files from CDNs Saturday, 29 October 11
- 48. Servers can vanishat any given time Saturday, 29 October 11
- 49. Internal APIs Saturday, 29October 11
- 50. Application Internal Storage API GFS S3 FS Saturday, 29 October 11
- 51. Application Internal DB API Mongo MySQL Cache Saturday, 29 October 11
- 52.
- 53. Service Oriented Architecture Saturday, 29 October 11
- 54. Sort of :-) Saturday,29 October 11
- 55. Eventually Consistent Saturday, 29 October 11
- 56. CAP Therom Saturday, 29October 11
- 57. Consistency Availability Partition Tolerance Saturday, 29 October 11
- 58. Consistency All nodes see the same data at the same time Saturday, 29 October 11
- 59. Availability Node failures do not prevent survivors from continuing to operate Saturday, 29 October 11
- 60. Partition Tolerance The system continues to operate despite arbitrary message loss Saturday, 29 October 11
- 61. Consistency Availability Partition Tolerance Saturday, 29 October 11
- 62. Queue Systems Saturday, 29October 11
- 63. Good for Image Processing Distributed Logs Data Mining Mass Emails Intensive transformation Search Saturday, 29 October 11
- 64. Common Tools Gearman Hadoop ZeroMQ RabbitMQ And many others! Saturday, 29 October 11
- 65. New York Times 4TB of TIFF files Needed to get 11 million PDF versions Used Hadoop and EC2 100 machines took 24 hours Saturday, 29 October 11
- 66.
- 67. Map Master gets a problem to solve Breaks into multiple sub-problems Distributed to multiple workers A worker can take the same steps Answer passed back to Master Saturday, 29 October 11
- 68. Reduce Takes in answers from the map workers Combines together to get an answer There can be multiple reducers Saturday, 29 October 11
- 69. process petabytes ofdata in few hours on commodity server farm Saturday, 29 October 11
- 70.
- 71. CouchDB Highly Concurrent Schema free, document based RESTful API Map/Reduce Views Easy Replication Saturday, 29 October 11
- 72.
- 73. Your Client Code Gearman Client API (C, PHP, Perl, MySQL UDF, ...) Your App Gearman Job Server Gearman gearmand Gearman Worker API (C, PHP, Perl, Python, ...) Your Worker Code Saturday, 29 October 11
- 74. pear.php.net/ net_gearman Saturday, 29 October 11
- 75. A Story! Saturday, 29October 11
- 76. Financial Software Saturday, 29October 11
- 77. 3000+ Clients Saturday, 29October 11
- 78. Each one has5 external data sources Saturday, 29 October 11
- 79. Each data sourceis a web service Saturday, 29 October 11
- 80. Ran every 6hours every day Saturday, 29 October 11
- 81. Cron Gearman Web Services Processing 1 Job 1 1 4 5 2 2 5 4 3 3 Saturday, 29 October 11
- 82. But! That wasn’tenough Saturday, 29 October 11
- 83. Job kicked offon login Saturday, 29 October 11
- 84.
- 85.
- 86. Another Story! Saturday, 29October 11
- 87.
- 88. Near Real TimeCloud Analytics Saturday, 29 October 11
- 89. Clients install logging agent locally Saturday, 29 October 11
- 90.
- 91. Public API Saturday, 29October 11
- 92. Multiple Persistent Gearman Servers Saturday, 29 October 11
- 93. Internal DB API Saturday,29 October 11
- 94. Load Agent syslogd Balanced API Load Balanced Worker Internal API Worker Gearman Worker Gearman CouchDB Persistent Saturday, 29 October 11
- 95. CouchDB Setup Saturday, 29October 11
- 96. Write vs Read Saturday,29 October 11
- 97. Writes Multi Master setup Replicated Deals with writes only Saturday, 29 October 11
- 98. Reads Multi Master setup Replicated from write cluster Slaves handle website requests Saturday, 29 October 11
- 99. Heavy Map/Reduce usage for data Saturday, 29 October 11
- 100. Questions? @h helgi@orchestra.io Joind.in: http://joind.in/4326 Saturday, 29 October 11