Distribute the                            workload                                Helgi Þormar Þorbjörnsson               ...
Who am I?Saturday, 29 October 11
Helgi                          Co-founded Orchestra.io                          Developer at PEAR                         ...
Why Distribute?                              Efficiency                               Budget                             Per...
Efficiency                          10 small servers > 1 bigSaturday, 29 October 11
Budget                          Spend wisely                          Commodity servers                          Cloud Com...
Perception                          Defer intensive processes                          Give instant feedback              ...
Saturday, 29 October 11
Ant ColoniesSaturday, 29 October 11
Teamwork                  When faced with a problem they will                      solve the problem as one.Saturday, 29 O...
Saturday, 29 October 11
Saturday, 29 October 11
Architect for                          DistributionSaturday, 29 October 11
Characteristics                             Decoupling                             Elasticity                             ...
DecouplingSaturday, 29 October 11
Application                           DB        API                          Cache      FESaturday, 29 October 11
API                                  Application                                                      API                 ...
ElasticitySaturday, 29 October 11
Cloud ComputingSaturday, 29 October 11
Load BalancingSaturday, 29 October 11
My Favourite                             HA Proxy                              NginxSaturday, 29 October 11
MonitoringSaturday, 29 October 11
When do I need                          more servers?Saturday, 29 October 11
Needs to be around                     from the start!Saturday, 29 October 11
Keep recordsSaturday, 29 October 11
Spot trendsSaturday, 29 October 11
Different types                          Hardware Performance                          Software Performance               ...
Applications                            New Relic                            CloudKick                            ScoutApp...
AutomationSaturday, 29 October 11
Plug into your                           monitoringSaturday, 29 October 11
Bringing together Monitoring                          and Elastic behaviour into one                                 beaut...
Add some intelligence to add /                          remove servers as needed based                              on cur...
Just make sure it                          doesn’t turn into...Saturday, 29 October 11
Skynet!!Saturday, 29 October 11
High AvailabilitySaturday, 29 October 11
Get a highly available and                          resilient setup by following a few                             of thos...
Remember, even Google has outagesSaturday, 29 October 11
What to avoidSaturday, 29 October 11
Local SessionsSaturday, 29 October 11
Solution     Store sessions in DB / MemcacheSaturday, 29 October 11
Local MemorySaturday, 29 October 11
Solution                          Networked MemcacheSaturday, 29 October 11
Local FilesSaturday, 29 October 11
Local UploadsSaturday, 29 October 11
Writing to /tmpSaturday, 29 October 11
Solution            Store on S3 or a networked FSSaturday, 29 October 11
Solution          Serve up static files from CDNsSaturday, 29 October 11
Servers can vanish at any                       given timeSaturday, 29 October 11
Internal APIsSaturday, 29 October 11
Application                            Internal Storage API                          GFS       S3         FSSaturday, 29 O...
Application                                  Internal DB API                          Mongo       MySQL         CacheSatur...
SOASaturday, 29 October 11
Service Oriented                            ArchitectureSaturday, 29 October 11
Sort of :-)Saturday, 29 October 11
Eventually                          ConsistentSaturday, 29 October 11
CAP TheromSaturday, 29 October 11
Consistency                          Availability                          Partition ToleranceSaturday, 29 October 11
Consistency                          All nodes see the same data at                                  the same timeSaturday...
Availability                          Node failures do not prevent                          survivors from continuing to  ...
Partition Tolerance                          The system continues to operate                           despite arbitrary m...
Consistency                          Availability                          Partition ToleranceSaturday, 29 October 11
Queue SystemsSaturday, 29 October 11
Good for                          Image Processing                          Distributed Logs                          Data...
Common Tools                            Gearman                            Hadoop                            ZeroMQ       ...
New York Times                     4TB of TIFF files                     Needed to get 11 million PDF versions             ...
Map/ReduceSaturday, 29 October 11
Map                          Master gets a problem to solve                          Breaks into multiple sub-problems    ...
Reduce                          Takes in answers from the map workers                          Combines together to get an...
process petabytes of data in few                          hours on commodity server farmSaturday, 29 October 11
CouchDBSaturday, 29 October 11
CouchDB                          Highly Concurrent                          Schema free, document based                   ...
GearmanSaturday, 29 October 11
Your Client Code                          Gearman Client API                          (C, PHP, Perl, MySQL UDF, ...)      ...
pear.php.net/                          net_gearmanSaturday, 29 October 11
A Story!Saturday, 29 October 11
Financial SoftwareSaturday, 29 October 11
3000+ ClientsSaturday, 29 October 11
Each one has 5 external                               data sourcesSaturday, 29 October 11
Each data source is a                              web serviceSaturday, 29 October 11
Ran every 6 hours every daySaturday, 29 October 11
Cron                              Gearman           Web Services                 Processing                  1            ...
But! That wasn’t enoughSaturday, 29 October 11
Job kicked off on loginSaturday, 29 October 11
SupervisordSaturday, 29 October 11
j.mp/supervisordSaturday, 29 October 11
Another Story!Saturday, 29 October 11
CloudSplitSaturday, 29 October 11
Near Real Time Cloud                               AnalyticsSaturday, 29 October 11
Clients install logging                               agent locallySaturday, 29 October 11
syslogdSaturday, 29 October 11
Public APISaturday, 29 October 11
Multiple Persistent                          Gearman ServersSaturday, 29 October 11
Internal DB APISaturday, 29 October 11
Load                    Agent   syslogd                      Balanced                                                   AP...
CouchDB SetupSaturday, 29 October 11
Write vs ReadSaturday, 29 October 11
Writes                          Multi Master setup                          Replicated                          Deals with...
Reads                          Multi Master setup                          Replicated from write cluster                  ...
Heavy Map/Reduce                            usage for dataSaturday, 29 October 11
Questions?                                     @h                              helgi@orchestra.io                         ...
Upcoming SlideShare
Loading in...5
×

Distribute the workload, PHP Barcelona 2011

3,369

Published on

Many services / applications now a day are ill equipped with handling a sudden rush of popularity, as is often the case on the internet now a days, to a point where the services either become unavailable or unbearably slow.
By taking a chapter from the ant colonies in the wild, where their strength lies in their numbers and the fact that everyone works together towards the same goal, we can apply the same principle to our
service by using systems such as:
- gearman
- memcache
- daemons
- message queues
- load balancers
and many more, you can achieve greater performance, more redundancy, higher availability and have the ability to scale your services up and down as required easily.
During this talk attendees will be lead through the world of distributed systems and scalability, and shown the how, where and what, of how to take the average application and splitting it into smaller more manageable pieces.

Published in: Technology
0 Comments
8 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
3,369
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
59
Comments
0
Likes
8
Embeds 0
No embeds

No notes for slide

Transcript of "Distribute the workload, PHP Barcelona 2011"

  1. 1. Distribute the workload Helgi Þormar Þorbjörnsson PHP Barcelona, 29th of October 2011Saturday, 29 October 11
  2. 2. Who am I?Saturday, 29 October 11
  3. 3. Helgi Co-founded Orchestra.io Developer at PEAR From Iceland @h on TwitterSaturday, 29 October 11
  4. 4. Why Distribute? Efficiency Budget PerceptionSaturday, 29 October 11
  5. 5. Efficiency 10 small servers > 1 bigSaturday, 29 October 11
  6. 6. Budget Spend wisely Commodity servers Cloud Computing (EC2)Saturday, 29 October 11
  7. 7. Perception Defer intensive processes Give instant feedback Users keep on browsingSaturday, 29 October 11
  8. 8. Saturday, 29 October 11
  9. 9. Ant ColoniesSaturday, 29 October 11
  10. 10. Teamwork When faced with a problem they will solve the problem as one.Saturday, 29 October 11
  11. 11. Saturday, 29 October 11
  12. 12. Saturday, 29 October 11
  13. 13. Architect for DistributionSaturday, 29 October 11
  14. 14. Characteristics Decoupling Elasticity High Availability ConcurrencySaturday, 29 October 11
  15. 15. DecouplingSaturday, 29 October 11
  16. 16. Application DB API Cache FESaturday, 29 October 11
  17. 17. API Application API DB API Cache FE CacheSaturday, 29 October 11
  18. 18. ElasticitySaturday, 29 October 11
  19. 19. Cloud ComputingSaturday, 29 October 11
  20. 20. Load BalancingSaturday, 29 October 11
  21. 21. My Favourite HA Proxy NginxSaturday, 29 October 11
  22. 22. MonitoringSaturday, 29 October 11
  23. 23. When do I need more servers?Saturday, 29 October 11
  24. 24. Needs to be around from the start!Saturday, 29 October 11
  25. 25. Keep recordsSaturday, 29 October 11
  26. 26. Spot trendsSaturday, 29 October 11
  27. 27. Different types Hardware Performance Software Performance Availability ResourcingSaturday, 29 October 11
  28. 28. Applications New Relic CloudKick ScoutApp Nagios Cacti CirconusSaturday, 29 October 11
  29. 29. AutomationSaturday, 29 October 11
  30. 30. Plug into your monitoringSaturday, 29 October 11
  31. 31. Bringing together Monitoring and Elastic behaviour into one beautiful whole!Saturday, 29 October 11
  32. 32. Add some intelligence to add / remove servers as needed based on current information.Saturday, 29 October 11
  33. 33. Just make sure it doesn’t turn into...Saturday, 29 October 11
  34. 34. Skynet!!Saturday, 29 October 11
  35. 35. High AvailabilitySaturday, 29 October 11
  36. 36. Get a highly available and resilient setup by following a few of those recommendationsSaturday, 29 October 11
  37. 37. Remember, even Google has outagesSaturday, 29 October 11
  38. 38. What to avoidSaturday, 29 October 11
  39. 39. Local SessionsSaturday, 29 October 11
  40. 40. Solution Store sessions in DB / MemcacheSaturday, 29 October 11
  41. 41. Local MemorySaturday, 29 October 11
  42. 42. Solution Networked MemcacheSaturday, 29 October 11
  43. 43. Local FilesSaturday, 29 October 11
  44. 44. Local UploadsSaturday, 29 October 11
  45. 45. Writing to /tmpSaturday, 29 October 11
  46. 46. Solution Store on S3 or a networked FSSaturday, 29 October 11
  47. 47. Solution Serve up static files from CDNsSaturday, 29 October 11
  48. 48. Servers can vanish at any given timeSaturday, 29 October 11
  49. 49. Internal APIsSaturday, 29 October 11
  50. 50. Application Internal Storage API GFS S3 FSSaturday, 29 October 11
  51. 51. Application Internal DB API Mongo MySQL CacheSaturday, 29 October 11
  52. 52. SOASaturday, 29 October 11
  53. 53. Service Oriented ArchitectureSaturday, 29 October 11
  54. 54. Sort of :-)Saturday, 29 October 11
  55. 55. Eventually ConsistentSaturday, 29 October 11
  56. 56. CAP TheromSaturday, 29 October 11
  57. 57. Consistency Availability Partition ToleranceSaturday, 29 October 11
  58. 58. Consistency All nodes see the same data at the same timeSaturday, 29 October 11
  59. 59. Availability Node failures do not prevent survivors from continuing to operateSaturday, 29 October 11
  60. 60. Partition Tolerance The system continues to operate despite arbitrary message lossSaturday, 29 October 11
  61. 61. Consistency Availability Partition ToleranceSaturday, 29 October 11
  62. 62. Queue SystemsSaturday, 29 October 11
  63. 63. Good for Image Processing Distributed Logs Data Mining Mass Emails Intensive transformation SearchSaturday, 29 October 11
  64. 64. Common Tools Gearman Hadoop ZeroMQ RabbitMQ And many others!Saturday, 29 October 11
  65. 65. New York Times 4TB of TIFF files Needed to get 11 million PDF versions Used Hadoop and EC2 100 machines took 24 hoursSaturday, 29 October 11
  66. 66. Map/ReduceSaturday, 29 October 11
  67. 67. Map Master gets a problem to solve Breaks into multiple sub-problems Distributed to multiple workers A worker can take the same steps Answer passed back to MasterSaturday, 29 October 11
  68. 68. Reduce Takes in answers from the map workers Combines together to get an answer There can be multiple reducersSaturday, 29 October 11
  69. 69. process petabytes of data in few hours on commodity server farmSaturday, 29 October 11
  70. 70. CouchDBSaturday, 29 October 11
  71. 71. CouchDB Highly Concurrent Schema free, document based RESTful API Map/Reduce Views Easy ReplicationSaturday, 29 October 11
  72. 72. GearmanSaturday, 29 October 11
  73. 73. Your Client Code Gearman Client API (C, PHP, Perl, MySQL UDF, ...) Your App Gearman Job Server Gearman gearmand Gearman Worker API (C, PHP, Perl, Python, ...) Your Worker CodeSaturday, 29 October 11
  74. 74. pear.php.net/ net_gearmanSaturday, 29 October 11
  75. 75. A Story!Saturday, 29 October 11
  76. 76. Financial SoftwareSaturday, 29 October 11
  77. 77. 3000+ ClientsSaturday, 29 October 11
  78. 78. Each one has 5 external data sourcesSaturday, 29 October 11
  79. 79. Each data source is a web serviceSaturday, 29 October 11
  80. 80. Ran every 6 hours every daySaturday, 29 October 11
  81. 81. Cron Gearman Web Services Processing 1 Job 1 1 4 5 2 2 5 4 3 3Saturday, 29 October 11
  82. 82. But! That wasn’t enoughSaturday, 29 October 11
  83. 83. Job kicked off on loginSaturday, 29 October 11
  84. 84. SupervisordSaturday, 29 October 11
  85. 85. j.mp/supervisordSaturday, 29 October 11
  86. 86. Another Story!Saturday, 29 October 11
  87. 87. CloudSplitSaturday, 29 October 11
  88. 88. Near Real Time Cloud AnalyticsSaturday, 29 October 11
  89. 89. Clients install logging agent locallySaturday, 29 October 11
  90. 90. syslogdSaturday, 29 October 11
  91. 91. Public APISaturday, 29 October 11
  92. 92. Multiple Persistent Gearman ServersSaturday, 29 October 11
  93. 93. Internal DB APISaturday, 29 October 11
  94. 94. Load Agent syslogd Balanced API Load Balanced Worker Internal API Worker Gearman Worker Gearman CouchDB PersistentSaturday, 29 October 11
  95. 95. CouchDB SetupSaturday, 29 October 11
  96. 96. Write vs ReadSaturday, 29 October 11
  97. 97. Writes Multi Master setup Replicated Deals with writes onlySaturday, 29 October 11
  98. 98. Reads Multi Master setup Replicated from write cluster Slaves handle website requestsSaturday, 29 October 11
  99. 99. Heavy Map/Reduce usage for dataSaturday, 29 October 11
  100. 100. Questions? @h helgi@orchestra.io Joind.in: http://joind.in/4326Saturday, 29 October 11
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×