SlideShare a Scribd company logo
1 of 139
Download to read offline
Sensible Scaling




                   Rowan Merewood
Sensible Scaling




  “If your application doesn't scale, it's your fault not mine.”
  – Rasmus Lerdorf (@rasmus)

                                                Rowan Merewood
Who?
Who?
●
    @rowan_m
Who?
●
    @rowan_m

●
    Software Engineer
    & Team Lead
Who?
●
    @rowan_m

●
    Software Engineer
    & Team Lead
●
    @ibuildings &
    @techportal
Why?
Why?
  ●
      I've built small apps
Why?
  ●
      I've built small apps
  ●
      and pretty large ones
Why?
  ●
      I've built small apps
  ●
      and pretty large ones

  ●
      I've seen massive
      over-engineering
Why?
  ●
      I've built small apps
  ●
      and pretty large ones

  ●
      I've seen massive
      over-engineering
  ●
      and platforms that
      cannot scale
TDD Principles
TDD Principles
●
    Write only enough
    code to pass the test
TDD Principles
●
    Write only enough
    code to pass the test
●
    Do not over-engineer
TDD Principles
●
    Write only enough
    code to pass the test
●
    Do not over-engineer

●
    Don't give away work
    for free!
Build for now
Build for now
       ●
           Don't be afraid to
           throw code away
Build for now
       ●
           Don't be afraid to
           throw code away
       ●
           XP's spikes
Build for now
       ●
           Don't be afraid to
           throw code away
       ●
           XP's spikes

       ●
           Take advantage of
           current tech.
           without tying
           yourself to it
Who are your users?
Who are your users?
●
    Lots of short visits?
Who are your users?
●
    Lots of short visits?
●
    Anonymous or
    session-based?
Who are your users?
●
    Lots of short visits?
●
    Anonymous or
    session-based?
●
    Long, complex
    interactions?
Who are your users?
●
    Lots of short visits?
●
    Anonymous or
    session-based?
●
    Long, complex
    interactions?
●
    Subscribers or
    free users?
Monitor and Analyse




http://community.plus.net/blog/2011/10/24/a-record-setting-manchester-derby/
Monitor and Analyse
                                        ●
                                            Predict usage
                                            patterns




http://community.plus.net/blog/2011/10/24/a-record-setting-manchester-derby/
Monitor and Analyse
                                        ●
                                            Predict usage
                                            patterns
                                        ●
                                            Plan for peaks and
                                            troughs




http://community.plus.net/blog/2011/10/24/a-record-setting-manchester-derby/
Profiling
Profiling
●
    You do not know
    what is slow
Profiling
●
    You do not know
    what is slow
●
    Ensure you measure
    more than just code!
Profiling
Profiling
●
    PHP - Xdebug – Derick Rethans (@derickr)
    ●
        Kcachegrind, Webgrind, etc.
Profiling
●
    PHP - Xdebug – Derick Rethans (@derickr)
    ●
        Kcachegrind, Webgrind, etc.
●
    PHP – XHProf
Profiling
●
    PHP - Xdebug – Derick Rethans (@derickr)
    ●
        Kcachegrind, Webgrind, etc.
●
    PHP – XHProf
●
    JavaScript – Firebug
Profiling
●
    PHP - Xdebug – Derick Rethans (@derickr)
    ●
        Kcachegrind, Webgrind, etc.
●
    PHP – XHProf
●
    JavaScript – Firebug
●
    JavaScript – Venkman
Profiling
●
    PHP - Xdebug – Derick Rethans (@derickr)
    ●
        Kcachegrind, Webgrind, etc.
●
    PHP – XHProf
●
    JavaScript – Firebug
●
    JavaScript – Venkman

●
    General page display - YSlow
Profiling
●
    CPU Load – top & /proc/loadavg
Profiling
●
    CPU Load – top & /proc/loadavg
●
    Disk IO – iotop
Profiling
●
    CPU Load – top & /proc/loadavg
●
    Disk IO – iotop
●
    Memory – free
Profiling
●
    CPU Load – top & /proc/loadavg
●
    Disk IO – iotop
●
    Memory – free
●
    Network – netstat
Profiling
●
    CPU Load – top & /proc/loadavg
●
    Disk IO – iotop
●
    Memory – free
●
    Network – netstat

●
    Monitoring – watch & time
Profiling under load
Profiling under load
          ●
              Code bottlenecks !=
              Platform bottlenecks
Profiling under load
          ●
              Code bottlenecks !=
              Platform bottlenecks
          ●
              Test in production,
              if possible
Profiling under load
          ●
              Code bottlenecks !=
              Platform bottlenecks
          ●
              Test in production,
              if possible
          ●
              Use multiple VMs
              (Vagrant can help
              automate this)
Profiling under load
          ●
              Code bottlenecks !=
              Platform bottlenecks
          ●
              Test in production,
              if possible
          ●
              Use multiple VMs
              (Vagrant can help
              automate this)
          ●
              Use siege & ab
Finding bottlenecks
Finding bottlenecks
Right           Wrong
Finding bottlenecks
          Right           Wrong
●
    Database
Finding bottlenecks
              Right           Wrong
●
    Database
●
    Disk IO
Finding bottlenecks
              Right           Wrong
●
    Database
●
    Disk IO
●
    External services
Finding bottlenecks
              Right           Wrong
●
    Database
●
    Disk IO
●
    External services
●
    Application work
Finding bottlenecks
              Right               Wrong
●
    Database            ●
                            Autoloaders
●
    Disk IO
●
    External services
●
    Application work
Finding bottlenecks
              Right                Wrong
●
    Database            ●
                            Autoloaders
●
    Disk IO             ●
                            Config files
●
    External services
●
    Application work
Finding bottlenecks
              Right                Wrong
●
    Database            ●
                            Autoloaders
●
    Disk IO             ●
                            Config files
●
    External services   ●
                            Logging
●
    Application work
Finding bottlenecks
              Right                Wrong
●
    Database            ●
                            Autoloaders
●
    Disk IO             ●
                            Config files
●
    External services   ●
                            Logging
●
    Application work    ●
                            Object instantiation
Finding bottlenecks
              Right                Wrong
●
    Database            ●
                            Autoloaders
●
    Disk IO             ●
                            Config files
●
    External services   ●
                            Logging
●
    Application work    ●
                            Object instantiation
                        ●
                            Frameworks
Finding bottlenecks
              Right                   Wrong
●
    Database            ●
                            Autoloaders
●
    Disk IO             ●
                            Config files
●
    External services   ●
                            Logging
●
    Application work    ●
                            Object instantiation
                        ●
                            Frameworks
                            Ask Jo (@juokas)
                            about Doctrine!
Tell users it's slow
Tell users it's slow
          ●
              User experience !=
              Raw speed
Tell users it's slow
          ●
              User experience !=
              Raw speed
          ●
              Keep the UI
              responsive
Tell users it's slow
          ●
              User experience !=
              Raw speed
          ●
              Keep the UI
              responsive
          ●
              Set expectations
Tell users it's slow
          ●
              User experience !=
              Raw speed
          ●
              Keep the UI
              responsive
          ●
              Set expectations

          ●
              Adapt to changes
              (@blongden)
Fail gracefully
Fail gracefully
●
    Actively choose
    to time-out
Fail gracefully
●
    Actively choose
    to time-out
●
    Prioritise
    core/profitable
    functionality
Fail gracefully
●
    Actively choose
    to time-out
●
    Prioritise
    core/profitable
    functionality

●
    Have a
    BIG RED BUTTON
Separate functionality
Separate functionality
 Admin            Batch
Separate functionality
           Admin           Batch
●
    Different users ==
    diff. requirements
Separate functionality
           Admin           Batch
●
    Different users ==
    diff. requirements

●
    No cache
Separate functionality
           Admin           Batch
●
    Different users ==
    diff. requirements

●
    No cache
●
    “Master” storage
Separate functionality
           Admin           Batch
●
    Different users ==
    diff. requirements

●
    No cache
●
    “Master” storage

●
    Security
Separate functionality
           Admin                     Batch
●
    Different users ==   ●
                             Not time critical
    diff. requirements

●
    No cache
●
    “Master” storage

●
    Security
Separate functionality
           Admin                     Batch
●
    Different users ==   ●
                             Not time critical
    diff. requirements   ●
                             CPU/RAM hungry

●
    No cache
●
    “Master” storage

●
    Security
Separate functionality
           Admin                     Batch
●
    Different users ==   ●
                             Not time critical
    diff. requirements   ●
                             CPU/RAM hungry

●
    No cache             ●
                             nice / ionice
●
    “Master” storage

●
    Security
Separate functionality
           Admin                     Batch
●
    Different users ==   ●
                             Not time critical
    diff. requirements   ●
                             CPU/RAM hungry

●
    No cache             ●
                             nice / ionice
●
    “Master” storage     ●
                             Network rate limiting

●
    Security
Separate functionality
           Admin                     Batch
●
    Different users ==   ●
                             Not time critical
    diff. requirements   ●
                             CPU/RAM hungry

●
    No cache             ●
                             nice / ionice
●
    “Master” storage     ●
                             Network rate limiting

●
    Security             ●
                             Read only?
Configure hardware correctly
Configure hardware correctly
              ●
                  What is required for
                  1 request?
Configure hardware correctly
              ●
                  What is required for
                  1 request?
              ●
                  How many
                  concurrent requests?
Configure hardware correctly
              ●
                  What is required for
                  1 request?
              ●
                  How many
                  concurrent requests?

              ●
                  Is it a linear scale?
Configure hardware correctly
              ●
                  What is required for
                  1 request?
              ●
                  How many
                  concurrent requests?

              ●
                  Is it a linear scale?

              ●
                  Ask your
                  hosting company
Aim for services
Aim for services
Obey the “Law of Demeter”
Obey the “Law of Demeter”
●
    Talk to your friends,
    don't talk to
    strangers
Obey the “Law of Demeter”
●
    Talk to your friends,
    don't talk to
    strangers

●
    Promotes
    loose coupling
Obey the “Law of Demeter”
           Good




           Bad
Obey the “Law of Demeter”
                       Good
●   $person->requestPayment($amount);



                       Bad
Obey the “Law of Demeter”
                       Good
●   $person->requestPayment($amount);



                        Bad
●
    $wallet = $person->getWallet();
●
    $wallet->getMoney($amount);
Obey the “Law of Demeter”
           Good




           Bad
Obey the “Law of Demeter”
           Good




           Bad
Obey the “Law of Demeter”
           Good




           Bad
Create immutable objects
Create immutable objects
●
    Objects that cannot change after creation
Create immutable objects
●
    Objects that cannot change after creation
●
    Copy On Write (COW - � ← unicode cow)
Create immutable objects
●
    Objects that cannot change after creation
●
    Copy On Write (COW - � ← unicode cow)
●
    More memory-hungry, but easy to roll back
Create immutable objects
●
    Objects that cannot change after creation
●
    Copy On Write (COW - � ← unicode cow)
●
    More memory-hungry, but easy to roll back
●
    Value objects should be immutable
Create immutable objects
●
    Objects that cannot change after creation
●
    Copy On Write (COW - � ← unicode cow)
●
    More memory-hungry, but easy to roll back
●
    Value objects should be immutable

●
    Think of them like a response from a service
Identify “single server” factors
Identify “single server” factors
●
    File uploads / user content
Identify “single server” factors
●
    File uploads / user content
●
    IP restrictions / server access
Identify “single server” factors
●
    File uploads / user content
●
    IP restrictions / server access
●
    Licensing?
Create services
Create services
Your API should be:
Create services
    Your API should be:
●
    Independent of
    application state
Create services
    Your API should be:
●
    Independent of
    application state
●
    Loosely coupled
Create services
    Your API should be:
●
    Independent of
    application state
●
    Loosely coupled
●
    Accepting/returning
    immutable objects
Use caches
Use caches
     ●
         REST-ful services
         over HTTP let you
         cache easily
Use caches
     ●
         REST-ful services
         over HTTP let you
         cache easily
     ●
         Internal caching is
         trickier, but “out-of-
         the-box” with most
         frameworks
Use caches
                            ●
                                REST-ful services
                                over HTTP let you
                                cache easily
                            ●
                                Internal caching is
                                trickier, but “out-of-
                                the-box” with most
                                frameworks



Proxy other people's services through your own cache!
Use queues
Use queues
●
    Full blown job server:
    Gearman
Use queues
●
    Full blown job server:
    Gearman
●
    AMQP
    implementation:
    RabbitMQ
Use queues
●
    Full blown job server:
    Gearman
●
    AMQP
    implementation:
    RabbitMQ
●
    Light-weight sockets:
    0MQ
Use the right storage
Use the right storage
Use the right storage

           Consistency    Availability




                  Partitioning
Use the right storage

           Consistency            Availability




          Enforced consistency   Eventual consistency
                (RDBMS)               (NoSQL)


                       Partitioning
Scale your storage
Scale your storage
●
    Master/slave
    replication
Scale your storage
●
    Master/slave
    replication
●
    Table partitioning
Scale your storage
●
    Master/slave
    replication
●
    Table partitioning
●
    Row partitioning
Scale your storage
●
    Master/slave
    replication
●
    Table partitioning
●
    Row partitioning

●
    rsync
Scale your storage
●
    Master/slave
    replication
●
    Table partitioning
●
    Row partitioning

●
    rsync
●
    NFS
Scale your storage
●
    Master/slave
    replication
●
    Table partitioning
●
    Row partitioning

●
    rsync
●
    NFS
●
    GlusterFS
Now you can use the cloud
Now you can use the cloud
●
    Full service –
    orchestra.io
Now you can use the cloud
●
    Full service
    orchestra.io
●
    Infrastructure mgmt.
    Scalr
Now you can use the cloud
●
    Full service
    orchestra.io
●
    Infrastructure mgmt.
    Scalr
●
    Manually
    $your_code_here
Now you can use the cloud
●
    Full service
    orchestra.io
●
    Infrastructure mgmt.
    Scalr
●
    Manually
    $your_code_here

●
    Automate!
    Chef & Puppet
Review
Review
   ●
       Start with only what
       you need
Review
   ●
       Start with only what
       you need
   ●
       Identify the
       problems
Review
   ●
       Start with only what
       you need
   ●
       Identify the
       problems
   ●
       Pull the problem out
       to a service
Review
   ●
       Start with only what
       you need
   ●
       Identify the
       problems
   ●
       Pull the problem out
       to a service
   ●
       Distribute
Thank you!
●   Feedback:

    http://joind.in/6324



●   Photos & Transformers:
    Nina Merewood
●   Legal?
    Takara & Hasbro
●   ElePHPant smuggling:
    Johannes Schlüter (@phperror)
●   Vintage photo nonsense:
    http://pixlr.com/o-matic/

More Related Content

Viewers also liked

Protect your users with Circuit breakers
Protect your users with Circuit breakersProtect your users with Circuit breakers
Protect your users with Circuit breakersScott Triglia
 
Designing irresistible apis
Designing irresistible apisDesigning irresistible apis
Designing irresistible apisKirsten Hunter
 
Introducing Azure DocumentDB - NoSQL, No Problem
Introducing Azure DocumentDB - NoSQL, No ProblemIntroducing Azure DocumentDB - NoSQL, No Problem
Introducing Azure DocumentDB - NoSQL, No ProblemAndrew Liu
 
DNS for Developers - NDC Oslo 2016
DNS for Developers - NDC Oslo 2016DNS for Developers - NDC Oslo 2016
DNS for Developers - NDC Oslo 2016Maarten Balliauw
 
Getting Browsers to Improve the Security of Your Webapp
Getting Browsers to Improve the Security of Your WebappGetting Browsers to Improve the Security of Your Webapp
Getting Browsers to Improve the Security of Your WebappFrancois Marier
 
Living With Legacy Code
Living With Legacy CodeLiving With Legacy Code
Living With Legacy CodeRowan Merewood
 
Microservices Minus the Hype: How to Build and Why
Microservices Minus the Hype: How to Build and WhyMicroservices Minus the Hype: How to Build and Why
Microservices Minus the Hype: How to Build and WhyMark Heckler
 
The Evolution and Future of Content Publishing
The Evolution and Future of Content PublishingThe Evolution and Future of Content Publishing
The Evolution and Future of Content PublishingFITC
 
DNS for Developers - ConFoo Montreal
DNS for Developers - ConFoo MontrealDNS for Developers - ConFoo Montreal
DNS for Developers - ConFoo MontrealMaarten Balliauw
 
Get more than a cache back! - ConFoo Montreal
Get more than a cache back! - ConFoo MontrealGet more than a cache back! - ConFoo Montreal
Get more than a cache back! - ConFoo MontrealMaarten Balliauw
 
The Soul in The Machine - Developing for Humans
The Soul in The Machine - Developing for HumansThe Soul in The Machine - Developing for Humans
The Soul in The Machine - Developing for HumansChristian Heilmann
 
Monitoring system with Grafana and StatsD
Monitoring system with Grafana and StatsDMonitoring system with Grafana and StatsD
Monitoring system with Grafana and StatsDArtur Prado
 

Viewers also liked (13)

Protect your users with Circuit breakers
Protect your users with Circuit breakersProtect your users with Circuit breakers
Protect your users with Circuit breakers
 
Designing irresistible apis
Designing irresistible apisDesigning irresistible apis
Designing irresistible apis
 
Introducing Azure DocumentDB - NoSQL, No Problem
Introducing Azure DocumentDB - NoSQL, No ProblemIntroducing Azure DocumentDB - NoSQL, No Problem
Introducing Azure DocumentDB - NoSQL, No Problem
 
DNS for Developers - NDC Oslo 2016
DNS for Developers - NDC Oslo 2016DNS for Developers - NDC Oslo 2016
DNS for Developers - NDC Oslo 2016
 
Getting Browsers to Improve the Security of Your Webapp
Getting Browsers to Improve the Security of Your WebappGetting Browsers to Improve the Security of Your Webapp
Getting Browsers to Improve the Security of Your Webapp
 
Living With Legacy Code
Living With Legacy CodeLiving With Legacy Code
Living With Legacy Code
 
Microservices Minus the Hype: How to Build and Why
Microservices Minus the Hype: How to Build and WhyMicroservices Minus the Hype: How to Build and Why
Microservices Minus the Hype: How to Build and Why
 
The Evolution and Future of Content Publishing
The Evolution and Future of Content PublishingThe Evolution and Future of Content Publishing
The Evolution and Future of Content Publishing
 
DNS for Developers - ConFoo Montreal
DNS for Developers - ConFoo MontrealDNS for Developers - ConFoo Montreal
DNS for Developers - ConFoo Montreal
 
Get more than a cache back! - ConFoo Montreal
Get more than a cache back! - ConFoo MontrealGet more than a cache back! - ConFoo Montreal
Get more than a cache back! - ConFoo Montreal
 
reveal.js 3.0.0
reveal.js 3.0.0reveal.js 3.0.0
reveal.js 3.0.0
 
The Soul in The Machine - Developing for Humans
The Soul in The Machine - Developing for HumansThe Soul in The Machine - Developing for Humans
The Soul in The Machine - Developing for Humans
 
Monitoring system with Grafana and StatsD
Monitoring system with Grafana and StatsDMonitoring system with Grafana and StatsD
Monitoring system with Grafana and StatsD
 

Similar to Sensible scaling

Automating MySQL operations with Puppet
Automating MySQL operations with PuppetAutomating MySQL operations with Puppet
Automating MySQL operations with PuppetKris Buytaert
 
Creating a reasonable project boilerplate
Creating a reasonable project boilerplateCreating a reasonable project boilerplate
Creating a reasonable project boilerplateStanislav Petrov
 
Moby is killing your devops efforts
Moby is killing your devops effortsMoby is killing your devops efforts
Moby is killing your devops effortsKris Buytaert
 
Bug hunting through_reverse_engineering
Bug hunting through_reverse_engineeringBug hunting through_reverse_engineering
Bug hunting through_reverse_engineeringarif
 
ModernWeb 2019: Why we replace TypeScript with Dart
ModernWeb 2019: Why we replace TypeScript with DartModernWeb 2019: Why we replace TypeScript with Dart
ModernWeb 2019: Why we replace TypeScript with DartJumper Chen
 
Pipeline as code for your infrastructure as Code
Pipeline as code for your infrastructure as CodePipeline as code for your infrastructure as Code
Pipeline as code for your infrastructure as CodeKris Buytaert
 
Path Dependent Development (PyCon AU)
Path Dependent Development (PyCon AU)Path Dependent Development (PyCon AU)
Path Dependent Development (PyCon AU)ncoghlan_dev
 
Open Source Monitoring in 2019
Open Source Monitoring in 2019 Open Source Monitoring in 2019
Open Source Monitoring in 2019 Kris Buytaert
 
Passing the Joel Test in the PHP World (phpbnl10)
Passing the Joel Test in the PHP World (phpbnl10)Passing the Joel Test in the PHP World (phpbnl10)
Passing the Joel Test in the PHP World (phpbnl10)Lorna Mitchell
 
Path dependent-development (PyCon India)
Path dependent-development (PyCon India)Path dependent-development (PyCon India)
Path dependent-development (PyCon India)ncoghlan_dev
 
Continuous Infrastructure First
Continuous Infrastructure FirstContinuous Infrastructure First
Continuous Infrastructure FirstKris Buytaert
 
Don't Suck at Building Stuff - Mykel Alvis at Puppet Camp Altanta
Don't Suck at Building Stuff  - Mykel Alvis at Puppet Camp AltantaDon't Suck at Building Stuff  - Mykel Alvis at Puppet Camp Altanta
Don't Suck at Building Stuff - Mykel Alvis at Puppet Camp AltantaPuppet
 
The Bleeding Edge
The Bleeding EdgeThe Bleeding Edge
The Bleeding EdgejClarity
 
Debugging Web Apps on Real Mobile Devices
Debugging Web Apps on Real Mobile DevicesDebugging Web Apps on Real Mobile Devices
Debugging Web Apps on Real Mobile DevicesDale Lane
 
Олексій Павленко. CONTRACT PROTECTION ON THE FRONTEND SIDE: HOW TO ORGANIZE R...
Олексій Павленко. CONTRACT PROTECTION ON THE FRONTEND SIDE: HOW TO ORGANIZE R...Олексій Павленко. CONTRACT PROTECTION ON THE FRONTEND SIDE: HOW TO ORGANIZE R...
Олексій Павленко. CONTRACT PROTECTION ON THE FRONTEND SIDE: HOW TO ORGANIZE R...OdessaJS Conf
 
Liferay portals in real projects
Liferay portals  in real projectsLiferay portals  in real projects
Liferay portals in real projectsIBACZ
 
Python performance profiling
Python performance profilingPython performance profiling
Python performance profilingJon Haddad
 
On the Importance of Infrastructure as Code
On the Importance of Infrastructure as CodeOn the Importance of Infrastructure as Code
On the Importance of Infrastructure as CodeKris Buytaert
 

Similar to Sensible scaling (20)

Automating MySQL operations with Puppet
Automating MySQL operations with PuppetAutomating MySQL operations with Puppet
Automating MySQL operations with Puppet
 
Creating a reasonable project boilerplate
Creating a reasonable project boilerplateCreating a reasonable project boilerplate
Creating a reasonable project boilerplate
 
Moby is killing your devops efforts
Moby is killing your devops effortsMoby is killing your devops efforts
Moby is killing your devops efforts
 
Bug hunting through_reverse_engineering
Bug hunting through_reverse_engineeringBug hunting through_reverse_engineering
Bug hunting through_reverse_engineering
 
ModernWeb 2019: Why we replace TypeScript with Dart
ModernWeb 2019: Why we replace TypeScript with DartModernWeb 2019: Why we replace TypeScript with Dart
ModernWeb 2019: Why we replace TypeScript with Dart
 
Pipeline as code for your infrastructure as Code
Pipeline as code for your infrastructure as CodePipeline as code for your infrastructure as Code
Pipeline as code for your infrastructure as Code
 
Path Dependent Development (PyCon AU)
Path Dependent Development (PyCon AU)Path Dependent Development (PyCon AU)
Path Dependent Development (PyCon AU)
 
Open Source Monitoring in 2019
Open Source Monitoring in 2019 Open Source Monitoring in 2019
Open Source Monitoring in 2019
 
Passing the Joel Test in the PHP World (phpbnl10)
Passing the Joel Test in the PHP World (phpbnl10)Passing the Joel Test in the PHP World (phpbnl10)
Passing the Joel Test in the PHP World (phpbnl10)
 
Path dependent-development (PyCon India)
Path dependent-development (PyCon India)Path dependent-development (PyCon India)
Path dependent-development (PyCon India)
 
Continuous Infrastructure First
Continuous Infrastructure FirstContinuous Infrastructure First
Continuous Infrastructure First
 
Don't Suck at Building Stuff - Mykel Alvis at Puppet Camp Altanta
Don't Suck at Building Stuff  - Mykel Alvis at Puppet Camp AltantaDon't Suck at Building Stuff  - Mykel Alvis at Puppet Camp Altanta
Don't Suck at Building Stuff - Mykel Alvis at Puppet Camp Altanta
 
The Bleeding Edge
The Bleeding EdgeThe Bleeding Edge
The Bleeding Edge
 
The Bleeding Edge
The Bleeding EdgeThe Bleeding Edge
The Bleeding Edge
 
Debugging Web Apps on Real Mobile Devices
Debugging Web Apps on Real Mobile DevicesDebugging Web Apps on Real Mobile Devices
Debugging Web Apps on Real Mobile Devices
 
MSL2008. Debugging
MSL2008. DebuggingMSL2008. Debugging
MSL2008. Debugging
 
Олексій Павленко. CONTRACT PROTECTION ON THE FRONTEND SIDE: HOW TO ORGANIZE R...
Олексій Павленко. CONTRACT PROTECTION ON THE FRONTEND SIDE: HOW TO ORGANIZE R...Олексій Павленко. CONTRACT PROTECTION ON THE FRONTEND SIDE: HOW TO ORGANIZE R...
Олексій Павленко. CONTRACT PROTECTION ON THE FRONTEND SIDE: HOW TO ORGANIZE R...
 
Liferay portals in real projects
Liferay portals  in real projectsLiferay portals  in real projects
Liferay portals in real projects
 
Python performance profiling
Python performance profilingPython performance profiling
Python performance profiling
 
On the Importance of Infrastructure as Code
On the Importance of Infrastructure as CodeOn the Importance of Infrastructure as Code
On the Importance of Infrastructure as Code
 

More from Rowan Merewood

Estimation or, "How to Dig your Grave"
Estimation or, "How to Dig your Grave"Estimation or, "How to Dig your Grave"
Estimation or, "How to Dig your Grave"Rowan Merewood
 
A Dependency Injection Primer
A Dependency Injection PrimerA Dependency Injection Primer
A Dependency Injection PrimerRowan Merewood
 
Practical Applications of Zend_Acl
Practical Applications of Zend_AclPractical Applications of Zend_Acl
Practical Applications of Zend_AclRowan Merewood
 
Algorithm, Review, Sorting
Algorithm, Review, SortingAlgorithm, Review, Sorting
Algorithm, Review, SortingRowan Merewood
 
State Machines to State of the Art
State Machines to State of the ArtState Machines to State of the Art
State Machines to State of the ArtRowan Merewood
 

More from Rowan Merewood (6)

Estimation or, "How to Dig your Grave"
Estimation or, "How to Dig your Grave"Estimation or, "How to Dig your Grave"
Estimation or, "How to Dig your Grave"
 
A Dependency Injection Primer
A Dependency Injection PrimerA Dependency Injection Primer
A Dependency Injection Primer
 
Practical Applications of Zend_Acl
Practical Applications of Zend_AclPractical Applications of Zend_Acl
Practical Applications of Zend_Acl
 
Algorithm, Review, Sorting
Algorithm, Review, SortingAlgorithm, Review, Sorting
Algorithm, Review, Sorting
 
Tools and Talent
Tools and TalentTools and Talent
Tools and Talent
 
State Machines to State of the Art
State Machines to State of the ArtState Machines to State of the Art
State Machines to State of the Art
 

Recently uploaded

Oracle Database 23c Security New Features.pptx
Oracle Database 23c Security New Features.pptxOracle Database 23c Security New Features.pptx
Oracle Database 23c Security New Features.pptxSatishbabu Gunukula
 
How to become a GDSC Lead GDSC MI AOE.pptx
How to become a GDSC Lead GDSC MI AOE.pptxHow to become a GDSC Lead GDSC MI AOE.pptx
How to become a GDSC Lead GDSC MI AOE.pptxKaustubhBhavsar6
 
Key Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfKey Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
 
IT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced ComputingIT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced ComputingMAGNIntelligence
 
Extra-120324-Visite-Entreprise-icare.pdf
Extra-120324-Visite-Entreprise-icare.pdfExtra-120324-Visite-Entreprise-icare.pdf
Extra-120324-Visite-Entreprise-icare.pdfInfopole1
 
.NET 8 ChatBot with Azure OpenAI Services.pptx
.NET 8 ChatBot with Azure OpenAI Services.pptx.NET 8 ChatBot with Azure OpenAI Services.pptx
.NET 8 ChatBot with Azure OpenAI Services.pptxHansamali Gamage
 
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024Alkin Tezuysal
 
EMEA What is ThousandEyes? Webinar
EMEA What is ThousandEyes? WebinarEMEA What is ThousandEyes? Webinar
EMEA What is ThousandEyes? WebinarThousandEyes
 
2024.03.12 Cost drivers of cultivated meat production.pdf
2024.03.12 Cost drivers of cultivated meat production.pdf2024.03.12 Cost drivers of cultivated meat production.pdf
2024.03.12 Cost drivers of cultivated meat production.pdfThe Good Food Institute
 
Automation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projectsAutomation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projectsDianaGray10
 
LF Energy Webinar - Unveiling OpenEEMeter 4.0
LF Energy Webinar - Unveiling OpenEEMeter 4.0LF Energy Webinar - Unveiling OpenEEMeter 4.0
LF Energy Webinar - Unveiling OpenEEMeter 4.0DanBrown980551
 
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - TechWebinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - TechProduct School
 
March Patch Tuesday
March Patch TuesdayMarch Patch Tuesday
March Patch TuesdayIvanti
 
Stobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
Stobox 4: Revolutionizing Investment in Real-World Assets Through TokenizationStobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
Stobox 4: Revolutionizing Investment in Real-World Assets Through TokenizationStobox
 
Novo Nordisk's journey in developing an open-source application on Neo4j
Novo Nordisk's journey in developing an open-source application on Neo4jNovo Nordisk's journey in developing an open-source application on Neo4j
Novo Nordisk's journey in developing an open-source application on Neo4jNeo4j
 
From the origin to the future of Open Source model and business
From the origin to the future of  Open Source model and businessFrom the origin to the future of  Open Source model and business
From the origin to the future of Open Source model and businessFrancesco Corti
 
Flow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First FrameFlow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First FrameKapil Thakar
 
20140402 - Smart house demo kit
20140402 - Smart house demo kit20140402 - Smart house demo kit
20140402 - Smart house demo kitJamie (Taka) Wang
 
Keep Your Finger on the Pulse of Your Building's Performance with IES Live
Keep Your Finger on the Pulse of Your Building's Performance with IES LiveKeep Your Finger on the Pulse of Your Building's Performance with IES Live
Keep Your Finger on the Pulse of Your Building's Performance with IES LiveIES VE
 
The Importance of Indoor Air Quality (English)
The Importance of Indoor Air Quality (English)The Importance of Indoor Air Quality (English)
The Importance of Indoor Air Quality (English)IES VE
 

Recently uploaded (20)

Oracle Database 23c Security New Features.pptx
Oracle Database 23c Security New Features.pptxOracle Database 23c Security New Features.pptx
Oracle Database 23c Security New Features.pptx
 
How to become a GDSC Lead GDSC MI AOE.pptx
How to become a GDSC Lead GDSC MI AOE.pptxHow to become a GDSC Lead GDSC MI AOE.pptx
How to become a GDSC Lead GDSC MI AOE.pptx
 
Key Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfKey Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdf
 
IT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced ComputingIT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced Computing
 
Extra-120324-Visite-Entreprise-icare.pdf
Extra-120324-Visite-Entreprise-icare.pdfExtra-120324-Visite-Entreprise-icare.pdf
Extra-120324-Visite-Entreprise-icare.pdf
 
.NET 8 ChatBot with Azure OpenAI Services.pptx
.NET 8 ChatBot with Azure OpenAI Services.pptx.NET 8 ChatBot with Azure OpenAI Services.pptx
.NET 8 ChatBot with Azure OpenAI Services.pptx
 
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
Design and Modeling for MySQL SCALE 21X Pasadena, CA Mar 2024
 
EMEA What is ThousandEyes? Webinar
EMEA What is ThousandEyes? WebinarEMEA What is ThousandEyes? Webinar
EMEA What is ThousandEyes? Webinar
 
2024.03.12 Cost drivers of cultivated meat production.pdf
2024.03.12 Cost drivers of cultivated meat production.pdf2024.03.12 Cost drivers of cultivated meat production.pdf
2024.03.12 Cost drivers of cultivated meat production.pdf
 
Automation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projectsAutomation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projects
 
LF Energy Webinar - Unveiling OpenEEMeter 4.0
LF Energy Webinar - Unveiling OpenEEMeter 4.0LF Energy Webinar - Unveiling OpenEEMeter 4.0
LF Energy Webinar - Unveiling OpenEEMeter 4.0
 
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - TechWebinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
 
March Patch Tuesday
March Patch TuesdayMarch Patch Tuesday
March Patch Tuesday
 
Stobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
Stobox 4: Revolutionizing Investment in Real-World Assets Through TokenizationStobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
Stobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
 
Novo Nordisk's journey in developing an open-source application on Neo4j
Novo Nordisk's journey in developing an open-source application on Neo4jNovo Nordisk's journey in developing an open-source application on Neo4j
Novo Nordisk's journey in developing an open-source application on Neo4j
 
From the origin to the future of Open Source model and business
From the origin to the future of  Open Source model and businessFrom the origin to the future of  Open Source model and business
From the origin to the future of Open Source model and business
 
Flow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First FrameFlow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First Frame
 
20140402 - Smart house demo kit
20140402 - Smart house demo kit20140402 - Smart house demo kit
20140402 - Smart house demo kit
 
Keep Your Finger on the Pulse of Your Building's Performance with IES Live
Keep Your Finger on the Pulse of Your Building's Performance with IES LiveKeep Your Finger on the Pulse of Your Building's Performance with IES Live
Keep Your Finger on the Pulse of Your Building's Performance with IES Live
 
The Importance of Indoor Air Quality (English)
The Importance of Indoor Air Quality (English)The Importance of Indoor Air Quality (English)
The Importance of Indoor Air Quality (English)
 

Sensible scaling

  • 1. Sensible Scaling Rowan Merewood
  • 2. Sensible Scaling “If your application doesn't scale, it's your fault not mine.” – Rasmus Lerdorf (@rasmus) Rowan Merewood
  • 4. Who? ● @rowan_m
  • 5. Who? ● @rowan_m ● Software Engineer & Team Lead
  • 6. Who? ● @rowan_m ● Software Engineer & Team Lead ● @ibuildings & @techportal
  • 8. Why? ● I've built small apps
  • 9. Why? ● I've built small apps ● and pretty large ones
  • 10. Why? ● I've built small apps ● and pretty large ones ● I've seen massive over-engineering
  • 11. Why? ● I've built small apps ● and pretty large ones ● I've seen massive over-engineering ● and platforms that cannot scale
  • 13. TDD Principles ● Write only enough code to pass the test
  • 14. TDD Principles ● Write only enough code to pass the test ● Do not over-engineer
  • 15. TDD Principles ● Write only enough code to pass the test ● Do not over-engineer ● Don't give away work for free!
  • 17. Build for now ● Don't be afraid to throw code away
  • 18. Build for now ● Don't be afraid to throw code away ● XP's spikes
  • 19. Build for now ● Don't be afraid to throw code away ● XP's spikes ● Take advantage of current tech. without tying yourself to it
  • 20. Who are your users?
  • 21. Who are your users? ● Lots of short visits?
  • 22. Who are your users? ● Lots of short visits? ● Anonymous or session-based?
  • 23. Who are your users? ● Lots of short visits? ● Anonymous or session-based? ● Long, complex interactions?
  • 24. Who are your users? ● Lots of short visits? ● Anonymous or session-based? ● Long, complex interactions? ● Subscribers or free users?
  • 26. Monitor and Analyse ● Predict usage patterns http://community.plus.net/blog/2011/10/24/a-record-setting-manchester-derby/
  • 27. Monitor and Analyse ● Predict usage patterns ● Plan for peaks and troughs http://community.plus.net/blog/2011/10/24/a-record-setting-manchester-derby/
  • 29. Profiling ● You do not know what is slow
  • 30. Profiling ● You do not know what is slow ● Ensure you measure more than just code!
  • 32. Profiling ● PHP - Xdebug – Derick Rethans (@derickr) ● Kcachegrind, Webgrind, etc.
  • 33. Profiling ● PHP - Xdebug – Derick Rethans (@derickr) ● Kcachegrind, Webgrind, etc. ● PHP – XHProf
  • 34. Profiling ● PHP - Xdebug – Derick Rethans (@derickr) ● Kcachegrind, Webgrind, etc. ● PHP – XHProf ● JavaScript – Firebug
  • 35. Profiling ● PHP - Xdebug – Derick Rethans (@derickr) ● Kcachegrind, Webgrind, etc. ● PHP – XHProf ● JavaScript – Firebug ● JavaScript – Venkman
  • 36. Profiling ● PHP - Xdebug – Derick Rethans (@derickr) ● Kcachegrind, Webgrind, etc. ● PHP – XHProf ● JavaScript – Firebug ● JavaScript – Venkman ● General page display - YSlow
  • 37. Profiling ● CPU Load – top & /proc/loadavg
  • 38. Profiling ● CPU Load – top & /proc/loadavg ● Disk IO – iotop
  • 39. Profiling ● CPU Load – top & /proc/loadavg ● Disk IO – iotop ● Memory – free
  • 40. Profiling ● CPU Load – top & /proc/loadavg ● Disk IO – iotop ● Memory – free ● Network – netstat
  • 41. Profiling ● CPU Load – top & /proc/loadavg ● Disk IO – iotop ● Memory – free ● Network – netstat ● Monitoring – watch & time
  • 43. Profiling under load ● Code bottlenecks != Platform bottlenecks
  • 44. Profiling under load ● Code bottlenecks != Platform bottlenecks ● Test in production, if possible
  • 45. Profiling under load ● Code bottlenecks != Platform bottlenecks ● Test in production, if possible ● Use multiple VMs (Vagrant can help automate this)
  • 46. Profiling under load ● Code bottlenecks != Platform bottlenecks ● Test in production, if possible ● Use multiple VMs (Vagrant can help automate this) ● Use siege & ab
  • 49. Finding bottlenecks Right Wrong ● Database
  • 50. Finding bottlenecks Right Wrong ● Database ● Disk IO
  • 51. Finding bottlenecks Right Wrong ● Database ● Disk IO ● External services
  • 52. Finding bottlenecks Right Wrong ● Database ● Disk IO ● External services ● Application work
  • 53. Finding bottlenecks Right Wrong ● Database ● Autoloaders ● Disk IO ● External services ● Application work
  • 54. Finding bottlenecks Right Wrong ● Database ● Autoloaders ● Disk IO ● Config files ● External services ● Application work
  • 55. Finding bottlenecks Right Wrong ● Database ● Autoloaders ● Disk IO ● Config files ● External services ● Logging ● Application work
  • 56. Finding bottlenecks Right Wrong ● Database ● Autoloaders ● Disk IO ● Config files ● External services ● Logging ● Application work ● Object instantiation
  • 57. Finding bottlenecks Right Wrong ● Database ● Autoloaders ● Disk IO ● Config files ● External services ● Logging ● Application work ● Object instantiation ● Frameworks
  • 58. Finding bottlenecks Right Wrong ● Database ● Autoloaders ● Disk IO ● Config files ● External services ● Logging ● Application work ● Object instantiation ● Frameworks Ask Jo (@juokas) about Doctrine!
  • 60. Tell users it's slow ● User experience != Raw speed
  • 61. Tell users it's slow ● User experience != Raw speed ● Keep the UI responsive
  • 62. Tell users it's slow ● User experience != Raw speed ● Keep the UI responsive ● Set expectations
  • 63. Tell users it's slow ● User experience != Raw speed ● Keep the UI responsive ● Set expectations ● Adapt to changes (@blongden)
  • 65. Fail gracefully ● Actively choose to time-out
  • 66. Fail gracefully ● Actively choose to time-out ● Prioritise core/profitable functionality
  • 67. Fail gracefully ● Actively choose to time-out ● Prioritise core/profitable functionality ● Have a BIG RED BUTTON
  • 70. Separate functionality Admin Batch ● Different users == diff. requirements
  • 71. Separate functionality Admin Batch ● Different users == diff. requirements ● No cache
  • 72. Separate functionality Admin Batch ● Different users == diff. requirements ● No cache ● “Master” storage
  • 73. Separate functionality Admin Batch ● Different users == diff. requirements ● No cache ● “Master” storage ● Security
  • 74. Separate functionality Admin Batch ● Different users == ● Not time critical diff. requirements ● No cache ● “Master” storage ● Security
  • 75. Separate functionality Admin Batch ● Different users == ● Not time critical diff. requirements ● CPU/RAM hungry ● No cache ● “Master” storage ● Security
  • 76. Separate functionality Admin Batch ● Different users == ● Not time critical diff. requirements ● CPU/RAM hungry ● No cache ● nice / ionice ● “Master” storage ● Security
  • 77. Separate functionality Admin Batch ● Different users == ● Not time critical diff. requirements ● CPU/RAM hungry ● No cache ● nice / ionice ● “Master” storage ● Network rate limiting ● Security
  • 78. Separate functionality Admin Batch ● Different users == ● Not time critical diff. requirements ● CPU/RAM hungry ● No cache ● nice / ionice ● “Master” storage ● Network rate limiting ● Security ● Read only?
  • 80. Configure hardware correctly ● What is required for 1 request?
  • 81. Configure hardware correctly ● What is required for 1 request? ● How many concurrent requests?
  • 82. Configure hardware correctly ● What is required for 1 request? ● How many concurrent requests? ● Is it a linear scale?
  • 83. Configure hardware correctly ● What is required for 1 request? ● How many concurrent requests? ● Is it a linear scale? ● Ask your hosting company
  • 86. Obey the “Law of Demeter”
  • 87. Obey the “Law of Demeter” ● Talk to your friends, don't talk to strangers
  • 88. Obey the “Law of Demeter” ● Talk to your friends, don't talk to strangers ● Promotes loose coupling
  • 89. Obey the “Law of Demeter” Good Bad
  • 90. Obey the “Law of Demeter” Good ● $person->requestPayment($amount); Bad
  • 91. Obey the “Law of Demeter” Good ● $person->requestPayment($amount); Bad ● $wallet = $person->getWallet(); ● $wallet->getMoney($amount);
  • 92. Obey the “Law of Demeter” Good Bad
  • 93. Obey the “Law of Demeter” Good Bad
  • 94. Obey the “Law of Demeter” Good Bad
  • 96. Create immutable objects ● Objects that cannot change after creation
  • 97. Create immutable objects ● Objects that cannot change after creation ● Copy On Write (COW - � ← unicode cow)
  • 98. Create immutable objects ● Objects that cannot change after creation ● Copy On Write (COW - � ← unicode cow) ● More memory-hungry, but easy to roll back
  • 99. Create immutable objects ● Objects that cannot change after creation ● Copy On Write (COW - � ← unicode cow) ● More memory-hungry, but easy to roll back ● Value objects should be immutable
  • 100. Create immutable objects ● Objects that cannot change after creation ● Copy On Write (COW - � ← unicode cow) ● More memory-hungry, but easy to roll back ● Value objects should be immutable ● Think of them like a response from a service
  • 102. Identify “single server” factors ● File uploads / user content
  • 103. Identify “single server” factors ● File uploads / user content ● IP restrictions / server access
  • 104. Identify “single server” factors ● File uploads / user content ● IP restrictions / server access ● Licensing?
  • 107. Create services Your API should be: ● Independent of application state
  • 108. Create services Your API should be: ● Independent of application state ● Loosely coupled
  • 109. Create services Your API should be: ● Independent of application state ● Loosely coupled ● Accepting/returning immutable objects
  • 111. Use caches ● REST-ful services over HTTP let you cache easily
  • 112. Use caches ● REST-ful services over HTTP let you cache easily ● Internal caching is trickier, but “out-of- the-box” with most frameworks
  • 113. Use caches ● REST-ful services over HTTP let you cache easily ● Internal caching is trickier, but “out-of- the-box” with most frameworks Proxy other people's services through your own cache!
  • 115. Use queues ● Full blown job server: Gearman
  • 116. Use queues ● Full blown job server: Gearman ● AMQP implementation: RabbitMQ
  • 117. Use queues ● Full blown job server: Gearman ● AMQP implementation: RabbitMQ ● Light-weight sockets: 0MQ
  • 118. Use the right storage
  • 119. Use the right storage
  • 120. Use the right storage Consistency Availability Partitioning
  • 121. Use the right storage Consistency Availability Enforced consistency Eventual consistency (RDBMS) (NoSQL) Partitioning
  • 123. Scale your storage ● Master/slave replication
  • 124. Scale your storage ● Master/slave replication ● Table partitioning
  • 125. Scale your storage ● Master/slave replication ● Table partitioning ● Row partitioning
  • 126. Scale your storage ● Master/slave replication ● Table partitioning ● Row partitioning ● rsync
  • 127. Scale your storage ● Master/slave replication ● Table partitioning ● Row partitioning ● rsync ● NFS
  • 128. Scale your storage ● Master/slave replication ● Table partitioning ● Row partitioning ● rsync ● NFS ● GlusterFS
  • 129. Now you can use the cloud
  • 130. Now you can use the cloud ● Full service – orchestra.io
  • 131. Now you can use the cloud ● Full service orchestra.io ● Infrastructure mgmt. Scalr
  • 132. Now you can use the cloud ● Full service orchestra.io ● Infrastructure mgmt. Scalr ● Manually $your_code_here
  • 133. Now you can use the cloud ● Full service orchestra.io ● Infrastructure mgmt. Scalr ● Manually $your_code_here ● Automate! Chef & Puppet
  • 134. Review
  • 135. Review ● Start with only what you need
  • 136. Review ● Start with only what you need ● Identify the problems
  • 137. Review ● Start with only what you need ● Identify the problems ● Pull the problem out to a service
  • 138. Review ● Start with only what you need ● Identify the problems ● Pull the problem out to a service ● Distribute
  • 139. Thank you! ● Feedback: http://joind.in/6324 ● Photos & Transformers: Nina Merewood ● Legal? Takara & Hasbro ● ElePHPant smuggling: Johannes Schlüter (@phperror) ● Vintage photo nonsense: http://pixlr.com/o-matic/