Distribute the workload, PHP Barcelona 2011
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share

Distribute the workload, PHP Barcelona 2011

  • 3,740 views
Uploaded on

Many services / applications now a day are ill equipped with handling a sudden rush of popularity, as is often the case on the internet now a days, to a point where the services either become......

Many services / applications now a day are ill equipped with handling a sudden rush of popularity, as is often the case on the internet now a days, to a point where the services either become unavailable or unbearably slow.
By taking a chapter from the ant colonies in the wild, where their strength lies in their numbers and the fact that everyone works together towards the same goal, we can apply the same principle to our
service by using systems such as:
- gearman
- memcache
- daemons
- message queues
- load balancers
and many more, you can achieve greater performance, more redundancy, higher availability and have the ability to scale your services up and down as required easily.
During this talk attendees will be lead through the world of distributed systems and scalability, and shown the how, where and what, of how to take the average application and splitting it into smaller more manageable pieces.

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
3,740
On Slideshare
3,736
From Embeds
4
Number of Embeds
2

Actions

Shares
Downloads
55
Comments
0
Likes
7

Embeds 4

http://localhost 3
http://www.w3schools.com 1

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Distribute the workload Helgi Þormar Þorbjörnsson PHP Barcelona, 29th of October 2011Saturday, 29 October 11
  • 2. Who am I?Saturday, 29 October 11
  • 3. Helgi Co-founded Orchestra.io Developer at PEAR From Iceland @h on TwitterSaturday, 29 October 11
  • 4. Why Distribute? Efficiency Budget PerceptionSaturday, 29 October 11
  • 5. Efficiency 10 small servers > 1 bigSaturday, 29 October 11
  • 6. Budget Spend wisely Commodity servers Cloud Computing (EC2)Saturday, 29 October 11
  • 7. Perception Defer intensive processes Give instant feedback Users keep on browsingSaturday, 29 October 11
  • 8. Saturday, 29 October 11
  • 9. Ant ColoniesSaturday, 29 October 11
  • 10. Teamwork When faced with a problem they will solve the problem as one.Saturday, 29 October 11
  • 11. Saturday, 29 October 11
  • 12. Saturday, 29 October 11
  • 13. Architect for DistributionSaturday, 29 October 11
  • 14. Characteristics Decoupling Elasticity High Availability ConcurrencySaturday, 29 October 11
  • 15. DecouplingSaturday, 29 October 11
  • 16. Application DB API Cache FESaturday, 29 October 11
  • 17. API Application API DB API Cache FE CacheSaturday, 29 October 11
  • 18. ElasticitySaturday, 29 October 11
  • 19. Cloud ComputingSaturday, 29 October 11
  • 20. Load BalancingSaturday, 29 October 11
  • 21. My Favourite HA Proxy NginxSaturday, 29 October 11
  • 22. MonitoringSaturday, 29 October 11
  • 23. When do I need more servers?Saturday, 29 October 11
  • 24. Needs to be around from the start!Saturday, 29 October 11
  • 25. Keep recordsSaturday, 29 October 11
  • 26. Spot trendsSaturday, 29 October 11
  • 27. Different types Hardware Performance Software Performance Availability ResourcingSaturday, 29 October 11
  • 28. Applications New Relic CloudKick ScoutApp Nagios Cacti CirconusSaturday, 29 October 11
  • 29. AutomationSaturday, 29 October 11
  • 30. Plug into your monitoringSaturday, 29 October 11
  • 31. Bringing together Monitoring and Elastic behaviour into one beautiful whole!Saturday, 29 October 11
  • 32. Add some intelligence to add / remove servers as needed based on current information.Saturday, 29 October 11
  • 33. Just make sure it doesn’t turn into...Saturday, 29 October 11
  • 34. Skynet!!Saturday, 29 October 11
  • 35. High AvailabilitySaturday, 29 October 11
  • 36. Get a highly available and resilient setup by following a few of those recommendationsSaturday, 29 October 11
  • 37. Remember, even Google has outagesSaturday, 29 October 11
  • 38. What to avoidSaturday, 29 October 11
  • 39. Local SessionsSaturday, 29 October 11
  • 40. Solution Store sessions in DB / MemcacheSaturday, 29 October 11
  • 41. Local MemorySaturday, 29 October 11
  • 42. Solution Networked MemcacheSaturday, 29 October 11
  • 43. Local FilesSaturday, 29 October 11
  • 44. Local UploadsSaturday, 29 October 11
  • 45. Writing to /tmpSaturday, 29 October 11
  • 46. Solution Store on S3 or a networked FSSaturday, 29 October 11
  • 47. Solution Serve up static files from CDNsSaturday, 29 October 11
  • 48. Servers can vanish at any given timeSaturday, 29 October 11
  • 49. Internal APIsSaturday, 29 October 11
  • 50. Application Internal Storage API GFS S3 FSSaturday, 29 October 11
  • 51. Application Internal DB API Mongo MySQL CacheSaturday, 29 October 11
  • 52. SOASaturday, 29 October 11
  • 53. Service Oriented ArchitectureSaturday, 29 October 11
  • 54. Sort of :-)Saturday, 29 October 11
  • 55. Eventually ConsistentSaturday, 29 October 11
  • 56. CAP TheromSaturday, 29 October 11
  • 57. Consistency Availability Partition ToleranceSaturday, 29 October 11
  • 58. Consistency All nodes see the same data at the same timeSaturday, 29 October 11
  • 59. Availability Node failures do not prevent survivors from continuing to operateSaturday, 29 October 11
  • 60. Partition Tolerance The system continues to operate despite arbitrary message lossSaturday, 29 October 11
  • 61. Consistency Availability Partition ToleranceSaturday, 29 October 11
  • 62. Queue SystemsSaturday, 29 October 11
  • 63. Good for Image Processing Distributed Logs Data Mining Mass Emails Intensive transformation SearchSaturday, 29 October 11
  • 64. Common Tools Gearman Hadoop ZeroMQ RabbitMQ And many others!Saturday, 29 October 11
  • 65. New York Times 4TB of TIFF files Needed to get 11 million PDF versions Used Hadoop and EC2 100 machines took 24 hoursSaturday, 29 October 11
  • 66. Map/ReduceSaturday, 29 October 11
  • 67. Map Master gets a problem to solve Breaks into multiple sub-problems Distributed to multiple workers A worker can take the same steps Answer passed back to MasterSaturday, 29 October 11
  • 68. Reduce Takes in answers from the map workers Combines together to get an answer There can be multiple reducersSaturday, 29 October 11
  • 69. process petabytes of data in few hours on commodity server farmSaturday, 29 October 11
  • 70. CouchDBSaturday, 29 October 11
  • 71. CouchDB Highly Concurrent Schema free, document based RESTful API Map/Reduce Views Easy ReplicationSaturday, 29 October 11
  • 72. GearmanSaturday, 29 October 11
  • 73. Your Client Code Gearman Client API (C, PHP, Perl, MySQL UDF, ...) Your App Gearman Job Server Gearman gearmand Gearman Worker API (C, PHP, Perl, Python, ...) Your Worker CodeSaturday, 29 October 11
  • 74. pear.php.net/ net_gearmanSaturday, 29 October 11
  • 75. A Story!Saturday, 29 October 11
  • 76. Financial SoftwareSaturday, 29 October 11
  • 77. 3000+ ClientsSaturday, 29 October 11
  • 78. Each one has 5 external data sourcesSaturday, 29 October 11
  • 79. Each data source is a web serviceSaturday, 29 October 11
  • 80. Ran every 6 hours every daySaturday, 29 October 11
  • 81. Cron Gearman Web Services Processing 1 Job 1 1 4 5 2 2 5 4 3 3Saturday, 29 October 11
  • 82. But! That wasn’t enoughSaturday, 29 October 11
  • 83. Job kicked off on loginSaturday, 29 October 11
  • 84. SupervisordSaturday, 29 October 11
  • 85. j.mp/supervisordSaturday, 29 October 11
  • 86. Another Story!Saturday, 29 October 11
  • 87. CloudSplitSaturday, 29 October 11
  • 88. Near Real Time Cloud AnalyticsSaturday, 29 October 11
  • 89. Clients install logging agent locallySaturday, 29 October 11
  • 90. syslogdSaturday, 29 October 11
  • 91. Public APISaturday, 29 October 11
  • 92. Multiple Persistent Gearman ServersSaturday, 29 October 11
  • 93. Internal DB APISaturday, 29 October 11
  • 94. Load Agent syslogd Balanced API Load Balanced Worker Internal API Worker Gearman Worker Gearman CouchDB PersistentSaturday, 29 October 11
  • 95. CouchDB SetupSaturday, 29 October 11
  • 96. Write vs ReadSaturday, 29 October 11
  • 97. Writes Multi Master setup Replicated Deals with writes onlySaturday, 29 October 11
  • 98. Reads Multi Master setup Replicated from write cluster Slaves handle website requestsSaturday, 29 October 11
  • 99. Heavy Map/Reduce usage for dataSaturday, 29 October 11
  • 100. Questions? @h helgi@orchestra.io Joind.in: http://joind.in/4326Saturday, 29 October 11