Distribute the workload, PHPTek, Amsterdam, 2011
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Distribute the workload, PHPTek, Amsterdam, 2011

on

  • 2,966 views

Many services and applications are ill-equipped to handle a sudden rush of popularity, so their services either become unavailable or unbearably slow. By taking a chapter from the ant colonies in the ...

Many services and applications are ill-equipped to handle a sudden rush of popularity, so their services either become unavailable or unbearably slow. By taking a chapter from the ant colonies in the wild, where their strength lies in their numbers and their ability to work together towards a common goal, you can achieve greater performance, more redundancy, higher availability and have the ability to scale services up and down as required easily. By leveraging systems such as gearman, memcache, daemons, message queues, load balancers and more, you too can enter the world of distributed systems and scalability.

Statistics

Views

Total Views
2,966
Views on SlideShare
2,966
Embed Views
0

Actions

Likes
0
Downloads
25
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Distribute the workload, PHPTek, Amsterdam, 2011 Presentation Transcript

  • 1. Distribute the workload Helgi Þormar Þorbjörnsson PHP Tek, Chicago, 26th May 2011Sunday, 29 May 2011
  • 2. Who am I?Sunday, 29 May 2011
  • 3. HelgiSunday, 29 May 2011
  • 4. Helgi VP of Engineering at Orchestra.ioSunday, 29 May 2011
  • 5. Helgi VP of Engineering at Orchestra.io Developer at PEARSunday, 29 May 2011
  • 6. Helgi VP of Engineering at Orchestra.io Developer at PEAR From IcelandSunday, 29 May 2011
  • 7. Helgi VP of Engineering at Orchestra.io Developer at PEAR From Iceland @h on TwitterSunday, 29 May 2011
  • 8. Why Distribute?Sunday, 29 May 2011
  • 9. Why Distribute? EfficiencySunday, 29 May 2011
  • 10. Why Distribute? Efficiency BudgetSunday, 29 May 2011
  • 11. Why Distribute? Efficiency Budget PerceptionSunday, 29 May 2011
  • 12. Efficiency 10 small servers > 1 bigSunday, 29 May 2011
  • 13. BudgetSunday, 29 May 2011
  • 14. Budget Spend wiselySunday, 29 May 2011
  • 15. Budget Spend wisely Commodity serversSunday, 29 May 2011
  • 16. Budget Spend wisely Commodity servers Cloud Computing (EC2)Sunday, 29 May 2011
  • 17. PerceptionSunday, 29 May 2011
  • 18. Perception Defer intensive processesSunday, 29 May 2011
  • 19. Perception Defer intensive processes Give instant feedbackSunday, 29 May 2011
  • 20. Perception Defer intensive processes Give instant feedback Users keep on browsingSunday, 29 May 2011
  • 21. Perception Defer intensive processes Give instant feedback Users keep on browsingSunday, 29 May 2011
  • 22. Sunday, 29 May 2011
  • 23. Ant ColoniesSunday, 29 May 2011
  • 24. Teamwork When faced with a problem they will solve the problem as one.Sunday, 29 May 2011
  • 25. Sunday, 29 May 2011
  • 26. Sunday, 29 May 2011
  • 27. Architect for DistributionSunday, 29 May 2011
  • 28. CharacteristicsSunday, 29 May 2011
  • 29. Characteristics DecouplingSunday, 29 May 2011
  • 30. Characteristics Decoupling ElasticitySunday, 29 May 2011
  • 31. Characteristics Decoupling Elasticity High AvailabilitySunday, 29 May 2011
  • 32. Characteristics Decoupling Elasticity High Availability ConcurrencySunday, 29 May 2011
  • 33. DecouplingSunday, 29 May 2011
  • 34. Application DB API Cache FESunday, 29 May 2011
  • 35. Application DB API Cache FESunday, 29 May 2011
  • 36. DB Application API Cache FESunday, 29 May 2011
  • 37. DB Application API Cache FE CacheSunday, 29 May 2011
  • 38. API DB Application API Cache FE CacheSunday, 29 May 2011
  • 39. API DB Application API API Cache FE CacheSunday, 29 May 2011
  • 40. ElasticitySunday, 29 May 2011
  • 41. Cloud ComputingSunday, 29 May 2011
  • 42. Load BalancingSunday, 29 May 2011
  • 43. My Favourite HA Proxy NginxSunday, 29 May 2011
  • 44. MonitoringSunday, 29 May 2011
  • 45. When do I need more servers?Sunday, 29 May 2011
  • 46. Needs to be around from the start!Sunday, 29 May 2011
  • 47. Keep recordsSunday, 29 May 2011
  • 48. Spot trendsSunday, 29 May 2011
  • 49. Different typesSunday, 29 May 2011
  • 50. Different types Hardware PerformanceSunday, 29 May 2011
  • 51. Different types Hardware Performance Software PerformanceSunday, 29 May 2011
  • 52. Different types Hardware Performance Software Performance AvailabilitySunday, 29 May 2011
  • 53. Different types Hardware Performance Software Performance Availability ResourcingSunday, 29 May 2011
  • 54. Different types Hardware Performance Software Performance Availability ResourcingSunday, 29 May 2011
  • 55. ApplicationsSunday, 29 May 2011
  • 56. Applications New RelicSunday, 29 May 2011
  • 57. Applications New Relic CloudKickSunday, 29 May 2011
  • 58. Applications New Relic CloudKick ScoutAppSunday, 29 May 2011
  • 59. Applications New Relic CloudKick ScoutApp NagiosSunday, 29 May 2011
  • 60. Applications New Relic CloudKick ScoutApp Nagios CactiSunday, 29 May 2011
  • 61. Applications New Relic CloudKick ScoutApp Nagios Cacti CirconusSunday, 29 May 2011
  • 62. AutomationSunday, 29 May 2011
  • 63. Plug into your monitoringSunday, 29 May 2011
  • 64. Bringing together Monitoring and Elastic behaviour into one beautiful whole!Sunday, 29 May 2011
  • 65. Add some intelligence to add / remove servers as needed based on current information.Sunday, 29 May 2011
  • 66. Just make sure it doesn’t turn into...Sunday, 29 May 2011
  • 67. Skynet!!Sunday, 29 May 2011
  • 68. High AvailabilitySunday, 29 May 2011
  • 69. Get a highly available and resilient setup by following a few of those recommendationsSunday, 29 May 2011
  • 70. Remember, even Google has outagesSunday, 29 May 2011
  • 71. What to avoidSunday, 29 May 2011
  • 72. Local SessionsSunday, 29 May 2011
  • 73. Solution Store sessions in DB / MemcacheSunday, 29 May 2011
  • 74. Local MemorySunday, 29 May 2011
  • 75. Solution Networked MemcacheSunday, 29 May 2011
  • 76. Local FilesSunday, 29 May 2011
  • 77. Local UploadsSunday, 29 May 2011
  • 78. Writing to /tmpSunday, 29 May 2011
  • 79. Solution Store on S3 or a networked FSSunday, 29 May 2011
  • 80. Solution Serve up static files from CDNsSunday, 29 May 2011
  • 81. Servers can vanish at any given timeSunday, 29 May 2011
  • 82. Internal APIsSunday, 29 May 2011
  • 83. Application Internal Storage API GFS S3 FSSunday, 29 May 2011
  • 84. Application Internal DB API Mongo MySQL CacheSunday, 29 May 2011
  • 85. Eventually ConsistentSunday, 29 May 2011
  • 86. CAP TheromSunday, 29 May 2011
  • 87. Consistency Availability Partition ToleranceSunday, 29 May 2011
  • 88. Consistency All nodes see the same data at the same timeSunday, 29 May 2011
  • 89. Availability Node failures do not prevent survivors from continuing to operateSunday, 29 May 2011
  • 90. Partition Tolerance The system continues to operate despite arbitrary message lossSunday, 29 May 2011
  • 91. Consistency Availability Partition ToleranceSunday, 29 May 2011
  • 92. Queue SystemsSunday, 29 May 2011
  • 93. Good forSunday, 29 May 2011
  • 94. Good for Image ProcessingSunday, 29 May 2011
  • 95. Good for Image Processing Distributed LogsSunday, 29 May 2011
  • 96. Good for Image Processing Distributed Logs Data MiningSunday, 29 May 2011
  • 97. Good for Image Processing Distributed Logs Data Mining Mass EmailsSunday, 29 May 2011
  • 98. Good for Image Processing Distributed Logs Data Mining Mass Emails Intensive transformationSunday, 29 May 2011
  • 99. Good for Image Processing Distributed Logs Data Mining Mass Emails Intensive transformation SearchSunday, 29 May 2011
  • 100. Common ToolsSunday, 29 May 2011
  • 101. Common Tools GearmanSunday, 29 May 2011
  • 102. Common Tools Gearman HadoopSunday, 29 May 2011
  • 103. Common Tools Gearman Hadoop ZeroMQSunday, 29 May 2011
  • 104. Common Tools Gearman Hadoop ZeroMQ RabbitMQSunday, 29 May 2011
  • 105. Common Tools Gearman Hadoop ZeroMQ RabbitMQ And many others!Sunday, 29 May 2011
  • 106. GearmanSunday, 29 May 2011
  • 107. Your Client Code Gearman Client API (C, PHP, Perl, MySQL UDF, ...) Your App Gearman Job Server Gearman gearmand Gearman Worker API (C, PHP, Perl, Python, ...) Your Worker CodeSunday, 29 May 2011
  • 108. pear.php.net/ net_gearmanSunday, 29 May 2011
  • 109. A Story!Sunday, 29 May 2011
  • 110. Financial SoftwareSunday, 29 May 2011
  • 111. 3000+ ClientsSunday, 29 May 2011
  • 112. Each one has 5 external data sourcesSunday, 29 May 2011
  • 113. Each data source is a web serviceSunday, 29 May 2011
  • 114. Ran every 6 hours every daySunday, 29 May 2011
  • 115. CronSunday, 29 May 2011
  • 116. Cron GearmanSunday, 29 May 2011
  • 117. Cron Gearman Job 1Sunday, 29 May 2011
  • 118. Cron Gearman Web Services 1 Job 1 4 2 5 3Sunday, 29 May 2011
  • 119. Cron Gearman Web Services Processing 1 Job 1 1 4 5 2 2 5 4 3 3Sunday, 29 May 2011
  • 120. But! That wasn’t enoughSunday, 29 May 2011
  • 121. Job kicked off on loginSunday, 29 May 2011
  • 122. SupervisordSunday, 29 May 2011
  • 123. phpadvent.org/2009/daemonize- your-php-by-sean-coatesSunday, 29 May 2011
  • 124. Questions? @h helgi@orchestra.io Joind.in: http://joind.in/3433Sunday, 29 May 2011