Distribute the                          workload                             Helgi Þormar Þorbjörnsson                    ...
Who am I?Saturday, 14 May 2011
Saturday, 14 May 2011
VP of Engineering at Orchestra.ioSaturday, 14 May 2011
VP of Engineering at Orchestra.io                        Developer at PEARSaturday, 14 May 2011
VP of Engineering at Orchestra.io                        Developer at PEAR                        From IcelandSaturday, 14...
VP of Engineering at Orchestra.io                        Developer at PEAR                        From Iceland            ...
Why Distribute?Saturday, 14 May 2011
Why Distribute?Saturday, 14 May 2011
Why Distribute?                             BudgetSaturday, 14 May 2011
Why Distribute?                             Budget                            EfficiencySaturday, 14 May 2011
Why Distribute?                             Budget                            Efficiency                           Perceptio...
BudgetSaturday, 14 May 2011
Budget                        Spend wiselySaturday, 14 May 2011
Budget                        Spend wisely                        Commodity serversSaturday, 14 May 2011
Budget                        Spend wisely                        Commodity servers                        Low overhead, h...
Budget                        Spend wisely                        Commodity servers                        Low overhead, h...
Efficiency                        10 small servers > 1 bigSaturday, 14 May 2011
Venue SecuritySaturday, 14 May 2011
1000 people can exit quicker                        through 10 small doors than 1 bigSaturday, 14 May 2011
1000 people can exit quicker                        through 10 small doors than 1 bigSaturday, 14 May 2011
1000 people can exit quicker                        through 10 small doors than 1 bigSaturday, 14 May 2011
1000 people can exit quicker                        through 10 small doors than 1 bigSaturday, 14 May 2011
Saturday, 14 May 2011
Saturday, 14 May 2011
Saturday, 14 May 2011
Saturday, 14 May 2011
PerceptionSaturday, 14 May 2011
Perception                        Defer intensive processesSaturday, 14 May 2011
Perception                        Defer intensive processes                        Give instant feedbackSaturday, 14 May 2...
Perception                        Defer intensive processes                        Give instant feedback                  ...
Perception                        Defer intensive processes                        Give instant feedback                  ...
Saturday, 14 May 2011
Chapter from NatureSaturday, 14 May 2011
Ant ColoniesSaturday, 14 May 2011
AlgorithmsSaturday, 14 May 2011
Algorithms                         SchedulingSaturday, 14 May 2011
Algorithms                         Scheduling                         Vehicle RoutingSaturday, 14 May 2011
Algorithms                         Scheduling                         Vehicle Routing                         AssignmentSa...
Algorithms                         Scheduling                         Vehicle Routing                         Assignment  ...
Algorithms                         Scheduling                         Vehicle Routing                         Assignment  ...
Algorithms                         Scheduling                         Vehicle Routing                         Assignment  ...
How do ants fit?Saturday, 14 May 2011
How do ants fit?                           Strength in numbersSaturday, 14 May 2011
How do ants fit?                           Strength in numbers                           Work togetherSaturday, 14 May 2011
How do ants fit?                           Strength in numbers                           Work together                    ...
Teamwork                  When faced with a problem they will                      solve the problem as one.Saturday, 14 M...
Saturday, 14 May 2011
Saturday, 14 May 2011
What if they were                           bigger?Saturday, 14 May 2011
Types of AntsSaturday, 14 May 2011
Types of Ants                          MilitarySaturday, 14 May 2011
Types of Ants                          Military                          MaidsSaturday, 14 May 2011
Types of Ants                          Military                          Maids                          Tunnel diggersSatu...
Types of Ants                          Military                          Maids                          Tunnel diggers    ...
How does this map                   to my application?Saturday, 14 May 2011
Saturday, 14 May 2011
Colony = ApplicationSaturday, 14 May 2011
Colony = Application                          Ants = ComponentsSaturday, 14 May 2011
Colony = Application                          Ants = Components             Ants do many different types of work to keep   ...
Architect for                        DistributionSaturday, 14 May 2011
CharacteristicsSaturday, 14 May 2011
Characteristics                           DecouplingSaturday, 14 May 2011
Characteristics                           Decoupling                           ElasticitySaturday, 14 May 2011
Characteristics                           Decoupling                           Elasticity                           High A...
Characteristics                           Decoupling                           Elasticity                           High A...
DecouplingSaturday, 14 May 2011
Application                         DB        API                        Cache      FESaturday, 14 May 2011
Application                         DB       API                        Cache      FESaturday, 14 May 2011
DB       Application   API         Cache                        FESaturday, 14 May 2011
DB               Application   API         Cache                                FE                        CacheSaturday, 1...
API               DB               Application         API         Cache                                      FE          ...
API               DB               Application         API                                                     API        ...
ElasticitySaturday, 14 May 2011
Cloud ComputingSaturday, 14 May 2011
Load BalancingSaturday, 14 May 2011
My Favourite                           HA Proxy                            NginxSaturday, 14 May 2011
MonitoringSaturday, 14 May 2011
AutomationSaturday, 14 May 2011
High AvailabilitySaturday, 14 May 2011
BenefitsSaturday, 14 May 2011
Benefits                        Easy managementSaturday, 14 May 2011
Benefits                                 Easy management                        Ability to stop / start servers quicklySat...
Benefits                                 Easy management                        Ability to stop / start servers quickly   ...
Benefits                                 Easy management                        Ability to stop / start servers quickly   ...
Benefits                                 Easy management                        Ability to stop / start servers quickly   ...
What to avoidSaturday, 14 May 2011
What to avoid                          Local:Saturday, 14 May 2011
What to avoid                          Local:                           SessionsSaturday, 14 May 2011
What to avoid                          Local:                           Sessions                           FilesSaturday, ...
What to avoid                          Local:                           Sessions                           Files          ...
What to avoid                          Local:                           Sessions                           Files          ...
What to avoid                          Local:                           Sessions                           Files          ...
SolutionsSaturday, 14 May 2011
Solutions                        Store sessions in DB / MemcacheSaturday, 14 May 2011
Solutions                        Store sessions in DB / Memcache                        Use S3 to store uploaded filesSatur...
Solutions                        Store sessions in DB / Memcache                        Use S3 to store uploaded files     ...
Solutions                        Store sessions in DB / Memcache                        Use S3 to store uploaded files     ...
Solutions                        Store sessions in DB / Memcache                        Use S3 to store uploaded files     ...
Internal APIsSaturday, 14 May 2011
Application                           Internal Storage API                        Mongo     MySQL       FSSaturday, 14 May...
SOASaturday, 14 May 2011
Service Oriented                          ArchitectureSaturday, 14 May 2011
Sort of :-)Saturday, 14 May 2011
Eventually                        ConsistentSaturday, 14 May 2011
CAP TheromSaturday, 14 May 2011
Consistency                        Availability                        Partition ToleranceSaturday, 14 May 2011
Queue SystemsSaturday, 14 May 2011
Good forSaturday, 14 May 2011
Good for                        Image ProcessingSaturday, 14 May 2011
Good for                        Image Processing                        Distributed LogsSaturday, 14 May 2011
Good for                        Image Processing                        Distributed Logs                        Data Minin...
Good for                        Image Processing                        Distributed Logs                        Data Minin...
Good for                        Image Processing                        Distributed Logs                        Data Minin...
Good for                        Image Processing                        Distributed Logs                        Data Minin...
Common ToolsSaturday, 14 May 2011
Common Tools                          GearmanSaturday, 14 May 2011
Common Tools                          Gearman                          HadoopSaturday, 14 May 2011
Common Tools                          Gearman                          Hadoop                          Zero MQ (0MQ)Saturd...
Common Tools                          Gearman                          Hadoop                          Zero MQ (0MQ)      ...
Common Tools                          Gearman                          Hadoop                          Zero MQ (0MQ)      ...
GearmanSaturday, 14 May 2011
Your Client Code                        Gearman Client API                        (C, PHP, Perl, MySQL UDF, ...)         Y...
Story timeSaturday, 14 May 2011
Story timeSaturday, 14 May 2011
Questions?                            helgi@orchestra.io                               Twitter: @h                        ...
Upcoming SlideShare
Loading in...5
×

Think like an ant, distribute the workload - PhpDay, Italy, 2011

2,056

Published on

Many services / applications now a day are ill equipped with handling a sudden rush of popularity, as is often the case on the internet now a days, to a point where the services either become unavailable or unbearably slow.
By taking a chapter from the ant colonies in the wild, where their strength lies in their numbers and the fact that everyone works together towards the same goal, we can apply the same principle to our service by using systems such as
- gearman
- memcache
- daemons
- message queues
- load balancers
and many more, you can achieve greater performance, more redundancy, higher availability and have the ability to scale your services up and down as required easily.
During this talk attendees will be lead through the world of distributed systems and scalability, and shown the how, where and what, of how to take the average application and splitting it into smaller more manageable pieces

Published in: Technology, Business
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,056
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
35
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Think like an ant, distribute the workload - PhpDay, Italy, 2011

  1. 1. Distribute the workload Helgi Þormar Þorbjörnsson PhpDay, Italy, Verona, 13th May, 2011Saturday, 14 May 2011
  2. 2. Who am I?Saturday, 14 May 2011
  3. 3. Saturday, 14 May 2011
  4. 4. VP of Engineering at Orchestra.ioSaturday, 14 May 2011
  5. 5. VP of Engineering at Orchestra.io Developer at PEARSaturday, 14 May 2011
  6. 6. VP of Engineering at Orchestra.io Developer at PEAR From IcelandSaturday, 14 May 2011
  7. 7. VP of Engineering at Orchestra.io Developer at PEAR From Iceland @h on TwitterSaturday, 14 May 2011
  8. 8. Why Distribute?Saturday, 14 May 2011
  9. 9. Why Distribute?Saturday, 14 May 2011
  10. 10. Why Distribute? BudgetSaturday, 14 May 2011
  11. 11. Why Distribute? Budget EfficiencySaturday, 14 May 2011
  12. 12. Why Distribute? Budget Efficiency PerceptionSaturday, 14 May 2011
  13. 13. BudgetSaturday, 14 May 2011
  14. 14. Budget Spend wiselySaturday, 14 May 2011
  15. 15. Budget Spend wisely Commodity serversSaturday, 14 May 2011
  16. 16. Budget Spend wisely Commodity servers Low overhead, high yieldSaturday, 14 May 2011
  17. 17. Budget Spend wisely Commodity servers Low overhead, high yield Cloud Computing (EC2)Saturday, 14 May 2011
  18. 18. Efficiency 10 small servers > 1 bigSaturday, 14 May 2011
  19. 19. Venue SecuritySaturday, 14 May 2011
  20. 20. 1000 people can exit quicker through 10 small doors than 1 bigSaturday, 14 May 2011
  21. 21. 1000 people can exit quicker through 10 small doors than 1 bigSaturday, 14 May 2011
  22. 22. 1000 people can exit quicker through 10 small doors than 1 bigSaturday, 14 May 2011
  23. 23. 1000 people can exit quicker through 10 small doors than 1 bigSaturday, 14 May 2011
  24. 24. Saturday, 14 May 2011
  25. 25. Saturday, 14 May 2011
  26. 26. Saturday, 14 May 2011
  27. 27. Saturday, 14 May 2011
  28. 28. PerceptionSaturday, 14 May 2011
  29. 29. Perception Defer intensive processesSaturday, 14 May 2011
  30. 30. Perception Defer intensive processes Give instant feedbackSaturday, 14 May 2011
  31. 31. Perception Defer intensive processes Give instant feedback Users keep on browsingSaturday, 14 May 2011
  32. 32. Perception Defer intensive processes Give instant feedback Users keep on browsingSaturday, 14 May 2011
  33. 33. Saturday, 14 May 2011
  34. 34. Chapter from NatureSaturday, 14 May 2011
  35. 35. Ant ColoniesSaturday, 14 May 2011
  36. 36. AlgorithmsSaturday, 14 May 2011
  37. 37. Algorithms SchedulingSaturday, 14 May 2011
  38. 38. Algorithms Scheduling Vehicle RoutingSaturday, 14 May 2011
  39. 39. Algorithms Scheduling Vehicle Routing AssignmentSaturday, 14 May 2011
  40. 40. Algorithms Scheduling Vehicle Routing Assignment SetsSaturday, 14 May 2011
  41. 41. Algorithms Scheduling Vehicle Routing Assignment Sets OtherSaturday, 14 May 2011
  42. 42. Algorithms Scheduling Vehicle Routing Assignment Sets OtherSaturday, 14 May 2011
  43. 43. How do ants fit?Saturday, 14 May 2011
  44. 44. How do ants fit? Strength in numbersSaturday, 14 May 2011
  45. 45. How do ants fit? Strength in numbers Work togetherSaturday, 14 May 2011
  46. 46. How do ants fit? Strength in numbers Work together Size benefits themSaturday, 14 May 2011
  47. 47. Teamwork When faced with a problem they will solve the problem as one.Saturday, 14 May 2011
  48. 48. Saturday, 14 May 2011
  49. 49. Saturday, 14 May 2011
  50. 50. What if they were bigger?Saturday, 14 May 2011
  51. 51. Types of AntsSaturday, 14 May 2011
  52. 52. Types of Ants MilitarySaturday, 14 May 2011
  53. 53. Types of Ants Military MaidsSaturday, 14 May 2011
  54. 54. Types of Ants Military Maids Tunnel diggersSaturday, 14 May 2011
  55. 55. Types of Ants Military Maids Tunnel diggers Food gatherersSaturday, 14 May 2011
  56. 56. How does this map to my application?Saturday, 14 May 2011
  57. 57. Saturday, 14 May 2011
  58. 58. Colony = ApplicationSaturday, 14 May 2011
  59. 59. Colony = Application Ants = ComponentsSaturday, 14 May 2011
  60. 60. Colony = Application Ants = Components Ants do many different types of work to keep their colony runningSaturday, 14 May 2011
  61. 61. Architect for DistributionSaturday, 14 May 2011
  62. 62. CharacteristicsSaturday, 14 May 2011
  63. 63. Characteristics DecouplingSaturday, 14 May 2011
  64. 64. Characteristics Decoupling ElasticitySaturday, 14 May 2011
  65. 65. Characteristics Decoupling Elasticity High AvailabilitySaturday, 14 May 2011
  66. 66. Characteristics Decoupling Elasticity High Availability ConcurrencySaturday, 14 May 2011
  67. 67. DecouplingSaturday, 14 May 2011
  68. 68. Application DB API Cache FESaturday, 14 May 2011
  69. 69. Application DB API Cache FESaturday, 14 May 2011
  70. 70. DB Application API Cache FESaturday, 14 May 2011
  71. 71. DB Application API Cache FE CacheSaturday, 14 May 2011
  72. 72. API DB Application API Cache FE CacheSaturday, 14 May 2011
  73. 73. API DB Application API API Cache FE CacheSaturday, 14 May 2011
  74. 74. ElasticitySaturday, 14 May 2011
  75. 75. Cloud ComputingSaturday, 14 May 2011
  76. 76. Load BalancingSaturday, 14 May 2011
  77. 77. My Favourite HA Proxy NginxSaturday, 14 May 2011
  78. 78. MonitoringSaturday, 14 May 2011
  79. 79. AutomationSaturday, 14 May 2011
  80. 80. High AvailabilitySaturday, 14 May 2011
  81. 81. BenefitsSaturday, 14 May 2011
  82. 82. Benefits Easy managementSaturday, 14 May 2011
  83. 83. Benefits Easy management Ability to stop / start servers quicklySaturday, 14 May 2011
  84. 84. Benefits Easy management Ability to stop / start servers quickly Responsibilities are separateSaturday, 14 May 2011
  85. 85. Benefits Easy management Ability to stop / start servers quickly Responsibilities are separate Quickly move to a new clusterSaturday, 14 May 2011
  86. 86. Benefits Easy management Ability to stop / start servers quickly Responsibilities are separate Quickly move to a new cluster Reduced riskSaturday, 14 May 2011
  87. 87. What to avoidSaturday, 14 May 2011
  88. 88. What to avoid Local:Saturday, 14 May 2011
  89. 89. What to avoid Local: SessionsSaturday, 14 May 2011
  90. 90. What to avoid Local: Sessions FilesSaturday, 14 May 2011
  91. 91. What to avoid Local: Sessions Files MemorySaturday, 14 May 2011
  92. 92. What to avoid Local: Sessions Files Memory UploadsSaturday, 14 May 2011
  93. 93. What to avoid Local: Sessions Files Memory Uploads Writing to /tmpSaturday, 14 May 2011
  94. 94. SolutionsSaturday, 14 May 2011
  95. 95. Solutions Store sessions in DB / MemcacheSaturday, 14 May 2011
  96. 96. Solutions Store sessions in DB / Memcache Use S3 to store uploaded filesSaturday, 14 May 2011
  97. 97. Solutions Store sessions in DB / Memcache Use S3 to store uploaded files Use networked MemcacheSaturday, 14 May 2011
  98. 98. Solutions Store sessions in DB / Memcache Use S3 to store uploaded files Use networked Memcache Serve up static files from a CDNSaturday, 14 May 2011
  99. 99. Solutions Store sessions in DB / Memcache Use S3 to store uploaded files Use networked Memcache Serve up static files from a CDN Or... A multi domain solutionSaturday, 14 May 2011
  100. 100. Internal APIsSaturday, 14 May 2011
  101. 101. Application Internal Storage API Mongo MySQL FSSaturday, 14 May 2011
  102. 102. SOASaturday, 14 May 2011
  103. 103. Service Oriented ArchitectureSaturday, 14 May 2011
  104. 104. Sort of :-)Saturday, 14 May 2011
  105. 105. Eventually ConsistentSaturday, 14 May 2011
  106. 106. CAP TheromSaturday, 14 May 2011
  107. 107. Consistency Availability Partition ToleranceSaturday, 14 May 2011
  108. 108. Queue SystemsSaturday, 14 May 2011
  109. 109. Good forSaturday, 14 May 2011
  110. 110. Good for Image ProcessingSaturday, 14 May 2011
  111. 111. Good for Image Processing Distributed LogsSaturday, 14 May 2011
  112. 112. Good for Image Processing Distributed Logs Data MiningSaturday, 14 May 2011
  113. 113. Good for Image Processing Distributed Logs Data Mining Mass EmailsSaturday, 14 May 2011
  114. 114. Good for Image Processing Distributed Logs Data Mining Mass Emails Intensive transformationSaturday, 14 May 2011
  115. 115. Good for Image Processing Distributed Logs Data Mining Mass Emails Intensive transformation SearchSaturday, 14 May 2011
  116. 116. Common ToolsSaturday, 14 May 2011
  117. 117. Common Tools GearmanSaturday, 14 May 2011
  118. 118. Common Tools Gearman HadoopSaturday, 14 May 2011
  119. 119. Common Tools Gearman Hadoop Zero MQ (0MQ)Saturday, 14 May 2011
  120. 120. Common Tools Gearman Hadoop Zero MQ (0MQ) RabbitMQSaturday, 14 May 2011
  121. 121. Common Tools Gearman Hadoop Zero MQ (0MQ) RabbitMQ And many others!Saturday, 14 May 2011
  122. 122. GearmanSaturday, 14 May 2011
  123. 123. Your Client Code Gearman Client API (C, PHP, Perl, MySQL UDF, ...) Your App Gearman Job Server Gearman gearmand Gearman Worker API (C, PHP, Perl, Python, ...) Your Worker CodeSaturday, 14 May 2011
  124. 124. Story timeSaturday, 14 May 2011
  125. 125. Story timeSaturday, 14 May 2011
  126. 126. Questions? helgi@orchestra.io Twitter: @h Joind.in: http://joind.in/3027Saturday, 14 May 2011
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×