Loading…

Flash Player 9 (or above) is needed to view presentations.
We have detected that you do not have it on your computer. To install it, go here.

Like this presentation? Why not share!

Scale like an ant, distribute the workload - DPC, Amsterdam, 2011

on

  • 3,395 views

Many services / applications now a day are ill equipped with handling a sudden rush of popularity, as is often the case on the internet now a days, to a point where the services either become ...

Many services / applications now a day are ill equipped with handling a sudden rush of popularity, as is often the case on the internet now a days, to a point where the services either become unavailable or unbearably slow.
By taking a chapter from the ant colonies in the wild, where their strength lies in their numbers and the fact that everyone works together towards the same goal, we can apply the same principle to our service by using systems such as
- gearman
- memcache
- daemons
- message queues
- load balancers
and many more, you can achieve greater performance, more redundancy, higher availability and have the ability to scale your services up and down as required easily.
During this talk attendees will be lead through the world of distributed systems and scalability, and shown the how, where and what, of how to take the average application and splitting it into smaller more manageable pieces

Statistics

Views

Total Views
3,395
Views on SlideShare
3,393
Embed Views
2

Actions

Likes
8
Downloads
107
Comments
0

1 Embed 2

http://www.thewebhatesme.com 2

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Scale like an ant, distribute the workload - DPC, Amsterdam,  2011 Scale like an ant, distribute the workload - DPC, Amsterdam, 2011 Presentation Transcript

  • Distribute the workload Helgi Þormar Þorbjörnsson Dutch PHP Conference, Amsterdam, 19th May 2011Thursday, 19 May 2011
  • Who am I?Thursday, 19 May 2011
  • HelgiThursday, 19 May 2011
  • Helgi VP of Engineering at Orchestra.ioThursday, 19 May 2011
  • Helgi VP of Engineering at Orchestra.io Developer at PEARThursday, 19 May 2011
  • Helgi VP of Engineering at Orchestra.io Developer at PEAR From IcelandThursday, 19 May 2011
  • Helgi VP of Engineering at Orchestra.io Developer at PEAR From Iceland @h on TwitterThursday, 19 May 2011
  • Why Distribute?Thursday, 19 May 2011
  • Why Distribute?Thursday, 19 May 2011
  • Why Distribute? BudgetThursday, 19 May 2011
  • Why Distribute? Budget EfficiencyThursday, 19 May 2011
  • Why Distribute? Budget Efficiency PerceptionThursday, 19 May 2011
  • BudgetThursday, 19 May 2011
  • Budget Spend wiselyThursday, 19 May 2011
  • Budget Spend wisely Commodity serversThursday, 19 May 2011
  • Budget Spend wisely Commodity servers Low overhead, high yieldThursday, 19 May 2011
  • Budget Spend wisely Commodity servers Low overhead, high yield Cloud Computing (EC2)Thursday, 19 May 2011
  • Efficiency 10 small servers > 1 bigThursday, 19 May 2011
  • Venue SecurityThursday, 19 May 2011
  • 1000 people can exit quicker through 10 small doors than 1 bigThursday, 19 May 2011
  • 1000 people can exit quicker through 10 small doors than 1 bigThursday, 19 May 2011
  • 1000 people can exit quicker through 10 small doors than 1 bigThursday, 19 May 2011
  • 1000 people can exit quicker through 10 small doors than 1 bigThursday, 19 May 2011
  • Thursday, 19 May 2011
  • Thursday, 19 May 2011
  • Thursday, 19 May 2011
  • Thursday, 19 May 2011
  • PerceptionThursday, 19 May 2011
  • Perception Defer intensive processesThursday, 19 May 2011
  • Perception Defer intensive processes Give instant feedbackThursday, 19 May 2011
  • Perception Defer intensive processes Give instant feedback Users keep on browsingThursday, 19 May 2011
  • Perception Defer intensive processes Give instant feedback Users keep on browsingThursday, 19 May 2011
  • “It all depends on how we look at things, and not how they are in themselves.” - Carl G. JungThursday, 19 May 2011
  • Thursday, 19 May 2011
  • Chapter from NatureThursday, 19 May 2011
  • Ant ColoniesThursday, 19 May 2011
  • AlgorithmsThursday, 19 May 2011
  • Algorithms SchedulingThursday, 19 May 2011
  • Algorithms Scheduling Vehicle RoutingThursday, 19 May 2011
  • Algorithms Scheduling Vehicle Routing AssignmentThursday, 19 May 2011
  • Algorithms Scheduling Vehicle Routing Assignment SetsThursday, 19 May 2011
  • Algorithms Scheduling Vehicle Routing Assignment Sets OtherThursday, 19 May 2011
  • Algorithms Scheduling Vehicle Routing Assignment Sets OtherThursday, 19 May 2011
  • How do ants fit?Thursday, 19 May 2011
  • How do ants fit? Strength in numbersThursday, 19 May 2011
  • How do ants fit? Strength in numbers Work togetherThursday, 19 May 2011
  • How do ants fit? Strength in numbers Work together Size benefits themThursday, 19 May 2011
  • Teamwork When faced with a problem they will solve the problem as one.Thursday, 19 May 2011
  • Thursday, 19 May 2011
  • Thursday, 19 May 2011
  • What if they were bigger?Thursday, 19 May 2011
  • Types of AntsThursday, 19 May 2011
  • Types of Ants MilitaryThursday, 19 May 2011
  • Types of Ants Military MaidsThursday, 19 May 2011
  • Types of Ants Military Maids Tunnel diggersThursday, 19 May 2011
  • Types of Ants Military Maids Tunnel diggers Food gatherersThursday, 19 May 2011
  • How does this map to my application?Thursday, 19 May 2011
  • Thursday, 19 May 2011
  • Colony = ApplicationThursday, 19 May 2011
  • Colony = Application Ants = ComponentsThursday, 19 May 2011
  • Colony = Application Ants = Components Ants do many different types of work to keep their colony runningThursday, 19 May 2011
  • Architect for DistributionThursday, 19 May 2011
  • CharacteristicsThursday, 19 May 2011
  • Characteristics DecouplingThursday, 19 May 2011
  • Characteristics Decoupling ElasticityThursday, 19 May 2011
  • Characteristics Decoupling Elasticity High AvailabilityThursday, 19 May 2011
  • Characteristics Decoupling Elasticity High Availability ConcurrencyThursday, 19 May 2011
  • DecouplingThursday, 19 May 2011
  • Application DB API Cache FEThursday, 19 May 2011
  • Application DB API Cache FEThursday, 19 May 2011
  • DB Application API Cache FEThursday, 19 May 2011
  • DB Application API Cache FE CacheThursday, 19 May 2011
  • API DB Application API Cache FE CacheThursday, 19 May 2011
  • API DB Application API API Cache FE CacheThursday, 19 May 2011
  • ElasticityThursday, 19 May 2011
  • Cloud ComputingThursday, 19 May 2011
  • Load BalancingThursday, 19 May 2011
  • My Favourite HA Proxy NginxThursday, 19 May 2011
  • MonitoringThursday, 19 May 2011
  • When do I need more servers?Thursday, 19 May 2011
  • Needs to be around from the start!Thursday, 19 May 2011
  • Keep recordsThursday, 19 May 2011
  • Spot trendsThursday, 19 May 2011
  • Different typesThursday, 19 May 2011
  • Different types Hardware PerformanceThursday, 19 May 2011
  • Different types Hardware Performance Software PerformanceThursday, 19 May 2011
  • Different types Hardware Performance Software Performance AvailabilityThursday, 19 May 2011
  • Different types Hardware Performance Software Performance Availability ResourcingThursday, 19 May 2011
  • Different types Hardware Performance Software Performance Availability ResourcingThursday, 19 May 2011
  • ApplicationsThursday, 19 May 2011
  • Applications New RelicThursday, 19 May 2011
  • Applications New Relic CloudKickThursday, 19 May 2011
  • Applications New Relic CloudKick ScoutAppThursday, 19 May 2011
  • Applications New Relic CloudKick ScoutApp NagiosThursday, 19 May 2011
  • Applications New Relic CloudKick ScoutApp Nagios CactiThursday, 19 May 2011
  • Applications New Relic CloudKick ScoutApp Nagios Cacti CirconusThursday, 19 May 2011
  • AutomationThursday, 19 May 2011
  • Want to sleep easy at night?Thursday, 19 May 2011
  • Want to sleep easy at night? Go out partying without worrying about getting a phone call?Thursday, 19 May 2011
  • Plug into your monitoringThursday, 19 May 2011
  • Bringing together Monitoring and Elastic behaviour into one beautiful whole!Thursday, 19 May 2011
  • Add some intelligence to add / remove servers as needed based on current information.Thursday, 19 May 2011
  • This is why good monitoring is essential or this wouldn’t be possibleThursday, 19 May 2011
  • Just make sure it doesn’t turn into...Thursday, 19 May 2011
  • Skynet!!Thursday, 19 May 2011
  • High AvailabilityThursday, 19 May 2011
  • Get a highly available and resilient setup by following a few of those recommendationsThursday, 19 May 2011
  • Remember, even Google has outagesThursday, 19 May 2011
  • BenefitsThursday, 19 May 2011
  • Benefits Easy managementThursday, 19 May 2011
  • Benefits Easy management Ability to stop / start servers quicklyThursday, 19 May 2011
  • Benefits Easy management Ability to stop / start servers quickly Responsibilities are separateThursday, 19 May 2011
  • Benefits Easy management Ability to stop / start servers quickly Responsibilities are separate Quickly move to a new clusterThursday, 19 May 2011
  • Benefits Easy management Ability to stop / start servers quickly Responsibilities are separate Quickly move to a new cluster Reduced riskThursday, 19 May 2011
  • What to avoidThursday, 19 May 2011
  • Local SessionsThursday, 19 May 2011
  • Solution Store sessions in DB / MemcacheThursday, 19 May 2011
  • Local MemoryThursday, 19 May 2011
  • Solution Networked MemcacheThursday, 19 May 2011
  • Local FilesThursday, 19 May 2011
  • Local UploadsThursday, 19 May 2011
  • Writing to /tmpThursday, 19 May 2011
  • Solution Store on S3 or a networked FSThursday, 19 May 2011
  • Solution Serve up static files from CDNsThursday, 19 May 2011
  • Servers can vanish at any given timeThursday, 19 May 2011
  • Internal APIsThursday, 19 May 2011
  • Application Internal Storage API GFS S3 FSThursday, 19 May 2011
  • Application Internal DB API Mongo MySQL CacheThursday, 19 May 2011
  • SOAThursday, 19 May 2011
  • Service Oriented ArchitectureThursday, 19 May 2011
  • Sort of :-)Thursday, 19 May 2011
  • Eventually ConsistentThursday, 19 May 2011
  • CAP TheromThursday, 19 May 2011
  • Consistency Availability Partition ToleranceThursday, 19 May 2011
  • Consistency All nodes see the same data at the same timeThursday, 19 May 2011
  • Availability Node failures do not prevent survivors from continuing to operateThursday, 19 May 2011
  • Partition Tolerance The system continues to operate despite arbitrary message lossThursday, 19 May 2011
  • Consistency Availability Partition ToleranceThursday, 19 May 2011
  • Queue SystemsThursday, 19 May 2011
  • Good forThursday, 19 May 2011
  • Good for Image ProcessingThursday, 19 May 2011
  • Good for Image Processing Distributed LogsThursday, 19 May 2011
  • Good for Image Processing Distributed Logs Data MiningThursday, 19 May 2011
  • Good for Image Processing Distributed Logs Data Mining Mass EmailsThursday, 19 May 2011
  • Good for Image Processing Distributed Logs Data Mining Mass Emails Intensive transformationThursday, 19 May 2011
  • Good for Image Processing Distributed Logs Data Mining Mass Emails Intensive transformation SearchThursday, 19 May 2011
  • Common ToolsThursday, 19 May 2011
  • Common Tools GearmanThursday, 19 May 2011
  • Common Tools Gearman HadoopThursday, 19 May 2011
  • Common Tools Gearman Hadoop Zero MQ (0MQ)Thursday, 19 May 2011
  • Common Tools Gearman Hadoop Zero MQ (0MQ) RabbitMQThursday, 19 May 2011
  • Common Tools Gearman Hadoop Zero MQ (0MQ) RabbitMQ And many others!Thursday, 19 May 2011
  • GearmanThursday, 19 May 2011
  • Your Client Code Gearman Client API (C, PHP, Perl, MySQL UDF, ...) Your App Gearman Job Server Gearman gearmand Gearman Worker API (C, PHP, Perl, Python, ...) Your Worker CodeThursday, 19 May 2011
  • A Story!Thursday, 19 May 2011
  • Financial SoftwareThursday, 19 May 2011
  • 3000+ ClientsThursday, 19 May 2011
  • Each one has 5 external data sourcesThursday, 19 May 2011
  • Each data source is a web serviceThursday, 19 May 2011
  • Ran every 6 hours every dayThursday, 19 May 2011
  • Cron Gearman Web Services Processing 1 Job 1 1 4 5 2 2 5 4 3 3Thursday, 19 May 2011
  • But! That wasn’t enoughThursday, 19 May 2011
  • Job kicked off on loginThursday, 19 May 2011
  • Another Story!Thursday, 19 May 2011
  • CloudSplitThursday, 19 May 2011
  • Near Real Time Cloud AnalyticsThursday, 19 May 2011
  • Clients install logging agent locallyThursday, 19 May 2011
  • syslogdThursday, 19 May 2011
  • Public APIThursday, 19 May 2011
  • Multiple Persistent Gearman ServersThursday, 19 May 2011
  • Internal DB APIThursday, 19 May 2011
  • AgentThursday, 19 May 2011
  • Agent syslogdThursday, 19 May 2011
  • Agent syslogd APIThursday, 19 May 2011
  • Load Agent syslogd Balanced APIThursday, 19 May 2011
  • Load Agent syslogd Balanced API Gearman GearmanThursday, 19 May 2011
  • Load Agent syslogd Balanced API Gearman Gearman PersistentThursday, 19 May 2011
  • Load Agent syslogd Balanced API Worker Worker Gearman Worker Gearman PersistentThursday, 19 May 2011
  • Load Agent syslogd Balanced API Worker Internal API Worker Gearman Worker Gearman PersistentThursday, 19 May 2011
  • Load Agent syslogd Balanced API Load Balanced Worker Internal API Worker Gearman Worker Gearman PersistentThursday, 19 May 2011
  • Load Agent syslogd Balanced API Load Balanced Worker Internal API Worker Gearman Worker Gearman CouchDB PersistentThursday, 19 May 2011
  • CouchDB SetupThursday, 19 May 2011
  • Write vs ReadThursday, 19 May 2011
  • WritesThursday, 19 May 2011
  • Writes Multi Master setupThursday, 19 May 2011
  • Writes Multi Master setup ReplicatedThursday, 19 May 2011
  • Writes Multi Master setup Replicated Deals with writes onlyThursday, 19 May 2011
  • Writes Multi Master setup Replicated Deals with writes onlyThursday, 19 May 2011
  • ReadsThursday, 19 May 2011
  • Reads Multi Master setupThursday, 19 May 2011
  • Reads Multi Master setup Replicated from write clusterThursday, 19 May 2011
  • Reads Multi Master setup Replicated from write cluster Slaves handle website requestsThursday, 19 May 2011
  • Reads Multi Master setup Replicated from write cluster Slaves handle website requestsThursday, 19 May 2011
  • Heavy Map/Reduce usage for dataThursday, 19 May 2011
  • SupervisordThursday, 19 May 2011
  • phpadvent.org/2009/daemonize- your-php-by-sean-coatesThursday, 19 May 2011
  • Map/ReduceThursday, 19 May 2011
  • MapThursday, 19 May 2011
  • Map Master gets a problem to solveThursday, 19 May 2011
  • Map Master gets a problem to solve Breaks into multiple sub-problemsThursday, 19 May 2011
  • Map Master gets a problem to solve Breaks into multiple sub-problems Distributed to multiple workersThursday, 19 May 2011
  • Map Master gets a problem to solve Breaks into multiple sub-problems Distributed to multiple workers A worker can take the same stepsThursday, 19 May 2011
  • Map Master gets a problem to solve Breaks into multiple sub-problems Distributed to multiple workers A worker can take the same steps Answer passed back to MasterThursday, 19 May 2011
  • ReduceThursday, 19 May 2011
  • Reduce Takes in answers from the map workersThursday, 19 May 2011
  • Reduce Takes in answers from the map workers Combines together to get an answerThursday, 19 May 2011
  • Reduce Takes in answers from the map workers Combines together to get an answer There can be multiple reducersThursday, 19 May 2011
  • process petabytes of data in few hours on commodity server farmThursday, 19 May 2011
  • CouchDBThursday, 19 May 2011
  • CouchDBThursday, 19 May 2011
  • CouchDB Highly ConcurrentThursday, 19 May 2011
  • CouchDB Highly Concurrent Schema free, document basedThursday, 19 May 2011
  • CouchDB Highly Concurrent Schema free, document based RESTful APIThursday, 19 May 2011
  • CouchDB Highly Concurrent Schema free, document based RESTful API Map/Reduce ViewsThursday, 19 May 2011
  • CouchDB Highly Concurrent Schema free, document based RESTful API Map/Reduce Views Easy ReplicationThursday, 19 May 2011
  • HadoopThursday, 19 May 2011
  • Hadoop is a framework for running applications on large clusters of commodity hardware.Thursday, 19 May 2011
  • Thursday, 19 May 2011
  • The Hadoop framework transparently provides applications both reliability and data motionThursday, 19 May 2011
  • Thursday, 19 May 2011
  • Uses Map/Reduce concept to farm out workThursday, 19 May 2011
  • Thursday, 19 May 2011
  • Distributed FS to handled node failure automagicallyThursday, 19 May 2011
  • Thursday, 19 May 2011
  • Join 2 datasets together of a significant sizeThursday, 19 May 2011
  • Thursday, 19 May 2011
  • 500 GB worth of log files with a large location datasetThursday, 19 May 2011
  • ØMQThursday, 19 May 2011
  • ØMQThursday, 19 May 2011
  • ØMQ Async Message SystemThursday, 19 May 2011
  • ØMQ Async Message System Thin and lightweightThursday, 19 May 2011
  • ØMQ Async Message System Thin and lightweight High PerformanceThursday, 19 May 2011
  • ØMQ Async Message System Thin and lightweight High Performance SimpleThursday, 19 May 2011
  • ØMQ Async Message System Thin and lightweight High Performance Simple ScalableThursday, 19 May 2011
  • Thursday, 19 May 2011
  • One socket can load balance to multiple end pointsThursday, 19 May 2011
  • Thursday, 19 May 2011
  • Multiple end points can be funnelled into a single socketThursday, 19 May 2011
  • Thursday, 19 May 2011
  • Handle deployments to multiple serversThursday, 19 May 2011
  • Thursday, 19 May 2011
  • Scale is an example of thatThursday, 19 May 2011
  • Thursday, 19 May 2011
  • Mongrel2 is a web server that uses it in a similar way as fastcgiThursday, 19 May 2011
  • Thursday, 19 May 2011
  • Move around text (JSON) and Binary data for real time communicationThursday, 19 May 2011
  • Thursday, 19 May 2011
  • Could have replaced syslogd and the external API in my previous exampleThursday, 19 May 2011
  • Code time? :-)Thursday, 19 May 2011
  • Questions? helgi@orchestra.io Twitter: @h Joind.in: http://joind.in/3212Thursday, 19 May 2011