Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Midwest PHP - Scaling Magento


Published on

Scaling Magento - Reaching Peak Performance. An overview of clustered Magento websites. Presented at Midwest PHP by Mathew Beane on 3/15/2015.

Published in: Technology
  • Be the first to comment

Midwest PHP - Scaling Magento

  1. 1. Mathew Beane
  2. 2. Mathew Beane Director, Systems Engineering @ Robofirm
  3. 3. • It is not ready yet…. • We are finalizing a release of the core configuration soon. • Moving to RHEL /CentOS 7 • Performance improvements from move to 7 are worth the wait. • Hoping to see it ready for Lonestar PHP Sneak Peak: • Uses Puppetmaster to build out cluster • Initial release should be the standard (n)webnodes + single db config • Vagrant boxes will be available to allow for development environments
  4. 4. • Open-source e-commerce platform • Based on Zend Framework 1 • Very flexible, it’s built to modify • Extremely scalable, supports huge stores • Market leader and still growing • Magento 2 is right around the corner • It’s is in development beta now •
  5. 5. Before you start:  Application is optimized and clean  Development Pipeline in place  Deployment infrastructure is solid  Rigorous testing protocols are in place Get started:  Add a Load Balancer  Proxy Web Traffic to web(n) servers  Cluster Redis using Sentinel / Twemproxy  Add Varnish if application permits  Add MySQL Read servers  Build in auto-scaling and your done
  6. 6. • PHP 5.4+ - Must use PHP-FPM instead of mod_php • Nginx is easier to use in a clustered environment – The configurations are cleaner – Proxy to PHP-FPM nodes it’s fast, clean and easy • Nexcess Whitepaper: • Redis is the preferred cache for Magento – Using Sentinel and twemproxy you can horizontally scale – Cluster twemproxy to eliminate all single threaded bottlenecks and single points of failure • Turn off MySQL Query Caching – This is single threaded and not of use with Magento • Varnish is great, however your application must be suitable • Use Zend Server 8 Z-Ray to profile and tune your Magento Application
  7. 7. You must be able to package your application in order to deploy it to a cluster. Application Packaging Maintain separate packages for Core Magento and all of your extensions. Composer / Modman Never edit core files. Ensure that you can see any core file changes. Keep a clean core Choose a branching methodology and build around it. Release / Feature Branch Build testing into all of your development cycles, never release without complete testing. Build unit tests where possible. Testing Make pull requests part of your workflow. Use Pull Requests
  8. 8. • When deploying an application to a cluster of application servers this is a requirement • There are many choices: – Capistrano: Written in Ruby but well accepted and great to work with – Jenkins: Another example of Deployment Automation – Bamboo: Part of the Atlassian stack also includes testing and other features. – Roll Your Own: This is more common, using bash scripts and other tools you can build a project deployment tool fairly easily. I highly suggest researching Fabrizo Branca’s work on the subject: Also check out Joshua Warren’s slides on Test Driven Development and his stem to stern tutorial Rock Solid Magento Development.
  9. 9. "Don't rate potential over performance." - Jim Fassel Blaze Meter Using Blazemeter you can easily build repeatable tests, with very nice graphs. (based on JMeter) Gatling On par with Blazemeter. JMeter Very effective, without having to purchase a SaaS Siege Can be used minimally to simulate some types of load.
  10. 10. Vs. HARDWARE • HAProxy: a free fast and reliable solution featuring high availability, load balancing and proxying for TCP and HTTP-based applications. • HAProxy can be used for web servers, database servers, Redis and any other TCP based application. • F5 Hardware load balancers are a standard • Rackspace offers a very easy to use web interface to maintain a hybrid infrastructure • Hardware load balancers offer a turn-key mature solution. SOFTWARE
  11. 11. Before you start:  Application is optimized and clean  Development Pipeline in place  Deployment infrastructure is solid  Rigorous testing protocols are in place Get started:  Add a Load Balancer  Proxy Web Traffic to web(n) servers  Cluster Redis using Sentinel / Twemproxy  Add Varnish if application permits  Add MySQL Read servers  Build in auto-scaling and your done
  12. 12. Hardware Load Balancer HAProxy Load Balancer Sentinel / twemproxy High Availability MySQL / Percona Master/Slave Single Write / Multiple Read Servers Database Apache / Nginx PHP 5.4 + Multiple Web Servers Varnish Web File Server (NFS / NAS) Redis / Memcache Deployment Tools Monitoring Other Typical Cluster Components
  13. 13. Expensive, with many features out of the box that will make for an easy turnkey solution. Using Sentinel and twemproxy wrapped by your load balancer will give you amazing performance out of your Redis cluster. Read servers can be load balanced with Magento. This is easy to achieve, but this doesn’t solve the checkout issue. Defacto software load balancing and High Availability server. This is the most common use for a load balancer, and is a great place to start your setup. • Budget concerns will drive this decision • Hosting Choices will affect availability, costs and toolsets • Start locally with HAProxy and build test clusters using vagrant • HAProxy can still be used, with a hardware load balancer in place.
  14. 14. • Simple to Load Balance, most sites start here • Challenges include the following: – Session Management: should be a no brainer if your using redis for sessions – Shared filesystem: Use NFS for media, keep all code local to web servers – Fencing: easily solved with either software or hardware – Log Collection: Rsyslog or Splunk • Keep the system builds simple, repeatable and automate if possible • How to automate: – Create an complete image to work from - include puppet so it can pull from puppetmaster – Puppetmaster spins up your webserver stack – You have your deployment process in place, so tie it into puppet and pull the code • Be prepared to lose nodes, the more you have the more likely failure is • When a node runs amok, you must be prepared to kill it dead
  15. 15. “Redis clustering using sentinel is easy to set up. Adding twemproxy allows for a highly scalable Redis cluster and you get auto fail over and a ton of other benefits with this configuration. This arrangement can also remove your Redis single point of failure.” • Sentinel Monitors Redis clusters • twemproxy handles sharding • twemproxy agent monitors sentinel • Very robust when setup, nothing is single threaded, everything is HA and the speed…. • Pretty much transparent to Magento despite the complexity
  16. 16. • Percona XtraDB to cluster MySQL • Percona XtraBackup for duplicating and rebuilding nodes • Percona Toolkit to help debug any issues your running into • Difficult to scale Write Servers • Scale out your read servers as needed, but MySQL reads are rarely the bottleneck • Typically Slave server is used for backup and hot swap, NOT clustering. A couple quick tips: • Not all tables in Magento are InnoDB, converting the MyISAM and Memory tables is OK • Usually overtime you will need to be able to kill read servers and refresh (STONITH) • Use your Master server as a read server in the load balancer pool, when you kill all your read servers, it can fall back to master.
  17. 17. • Detailed Auto-Scaling will have to wait until LonestarPHP – 3 Hour tutorial, which will include web server auto-scaling using puppet – April 16th 2015 in Dallas • Insert puzzle building analogy joke here: • Each hosting environment has its own quirks and add on top of that the business logic requirements you will almost always have a unique infrastructure for every client • Build small pieces and work them into the larger picture, you can get a lot of performance with a few minor changes. • Test everything you do, keep detailed notes on the configurations and compare against the previous tests
  18. 18. • Mathew Beane <> • Twitter: @aepod • Blog: Rate this talk: (Slides will be available) Thanks to the following: My Family The Magento Community Robofirm Fabrizo Branca (deployments) Thjis Feryn (sentinel) Rackspace