Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Rock-solid Magento Deployments (and Development)

19,803 views

Published on

In this session we show how to organize Magento projects using Version control and how to have a full development and deployment process in place to assure highest quality with many developers involved and teams spread over different continents. This talk covers how to run a continuous integration pipeline that takes care of testing various aspects of the webshop (unit tests, acceptance tests, performance test,…). Covers: Continuous integration, automation, Vagrant/Chef, Testing pipeline, unit/acceptance/performance tests, monitoring, deployment workflows, development best practices

Published in: Technology

Rock-solid Magento Deployments (and Development)

  1. 1. Magento Deployments Rock-Solid …and Development php[world] 2014 – Washington D.C.
  2. 2. Fabrizio Branca
  3. 3. fbrnc fbrnc
  4. 4. E-Commerce: Magento CMS: TYPO3 Portals: ZF, FLOW,… Mobile Searchperience: SOLR >120 people world-wide High Performance /Scale Global Enterprise Projects
  5. 5. San Francisco, CA Janine Fiona that’s me Leo
  6. 6. Who’s a Magento developer? Who spends a significant amount of his/her time dealing with development/deployment processes? Who thinks he/she’s got it figured out? YOU...?
  7. 7. Picture The BIG
  8. 8. plan code build test deploy operate
  9. 9. time complexity This is when we “deployed” using a FTP client
  10. 10. Keep it simple!
  11. 11. http://xkcd.com/974/
  12. 12. Youain’tgonnaneedit!
  13. 13. code
  14. 14. Vocabulary
  15. 15. CMS Magento Solr ERP Project Applications MMagento 117 Magento 118 Magento 119 Builds Release
  16. 16. Staging Magento 72 Magento 103 Magento 119 Magento 72 Magento 103 Magento 119 Magento 72 Magento 103 Magento 119 Environment Servers Instances
  17. 17. StagingIntegrationDevbox ProductionDeploy transient, potentially broken
  18. 18. StagingIntegrationDevbox ProductionDeploy VirtualBox + Vagrant Jenkins Server identical setup
  19. 19. Build Package Application System Storage User Data media directory (sometimes minified) Database dump Magento core Modules Scripts Settings Tools
  20. 20. Structure Project
  21. 21. project web root root!= This is the complete project This is where your webserver points to.
  22. 22. project web root root
  23. 23. htdocs <Name> <theme> layout template code design core community local <area> <Namespace> <package> app Structure Magento
  24. 24. htdocs <Name> <theme> layout template code design etc locale <area> core community local <area> <language> modules <package> <Namespace> <package> *.xml *.csv <theme> app skin lib Module2 default layout template code design etc locale frontend local frontend de_DE modules base My base My_Module2.xml My_Module2.csv default app skin etc, Block, … js, css, img, … my_module2 my_module2 Magento My_Module My_Module2 Module default layout template code design etc locale frontend local frontend de_DE modules base My base My_Module.xml My_Module.csv default app skin etc, Block, … js, css, img, … my_module my_module
  25. 25. htdocs Module default layout template code design etc locale local frontend de_DE modules My base *.xml app My_Module2.xml My_Module2.csv my_module2 My_Module.xml My_Module.csv etc, Block, … my_module Module2 etc, Block, … my_module2 my_module
  26. 26. modman by @colinmollenhour
  27. 27. htdocs code core community local app app/code/…/My/Module .modman My_Module app/design/… skin/frontend/… app/etc/modules/My_Module.xml modman app/code/…/My/Module My_Module2 app/design/… skin/frontend/… app/etc/modules/My_Module.xml modman <theme> layout template design etc locale <area> <language> modules <package> *.xml modman doesn’t even need to be in the web root!
  28. 28. Vanilla Magento absolutely no core hacks! …and core bugs you’ve fixed (and submitted to Magento, of course…) (except official patches)
  29. 29. “wiring” Intro modman “connects” existing modules to the Magento core, but how do we manage them?
  30. 30. htdocs code core community local app app/code/…/My/Module .modman My_Module app/design/… skin/frontend/… app/etc/modules/My_Module.xml modman app/code/…/My/Module My_Module2 app/design/… skin/frontend/… app/etc/modules/My_Module.xml modman <theme> layout template design etc locale <area> <language> modules <package> *.xml Components tools n98-magerun.phar phpunit.phar modman EnvSettingsTool composer.phar htdocs .modman Configuration Installer (Metadata, Doc,…) My_Module My_Module2 app,… and usually there’s even more stuff… These should come from different origins
  31. 31. Composer
  32. 32. magento-hackathon/ magento-composer-installer Composer
  33. 33. AOEpeople/composer-installers "replace": { "magento-hackathon/magento-composer-installer":"*" } “inspired” by composer/installers “magento-module” “magento-source” handles package types
  34. 34. <?php namespace AoepeopleComposerInstallers; class MagentoInstaller extends BaseInstaller { protected $locations = array( 'module' => '.modman/{$name}/', 'source' => 'htdocs/' ); } AOEpeople/composer-installers
  35. 35. • Fully compatible with magento-hackathon/magento-composer-installer • Adds support for type “magento-source” • Only puts packages into place • Native modman script is used to deploy modules • Wrapped in a simple installer script AOEpeople/composer-installers
  36. 36. { "minimum-stability":"dev", "require":{ "aoepeople/composer-installers": "*", "fbrnc/aoe_scheduler": "*", "fbrnc/aoe_cachecleaner": "*", "aoepeople/magento_enterprise": "1.13.1.0", "aoepeople/envsettingstool": "*" }, "config": { "bin-dir": "bin" } } AOEpeople/composer-installers replaces Hackathon Installer some modules Vanilla Magento Source Tools,…
  37. 37. Directory layout Webroot: /var/www/<projectName>/<environmentName>/current/htdocs var var <projectName> releases <environmentName> shared www media build_117 build_118 build_119 .modman htdocs Configuration tools install.sh app,… var media previous latest current next My_Module My_Module2
  38. 38. Magento Modules 3rd Party
  39. 39. discover use code review add modman add composer git integrate test deploy Magento Connect The Right Thing™ one-click install download good luck with that!
  40. 40. Code Quality Modules Not enterprise ready Not production ready Not cloud ready Bad performance Security
  41. 41. 0% 100% Writing Code Time spent… 5 years ago today Drinking Coffee Drinking Coffee while waiting for builds/tests Twitter VCS Deployment Design / Architecture / Infrastructure Writing Code Ping Pong Think about it
  42. 42. master Feature B Feature A Hotfix (if master is not modified) Hotfix (if master is modified) Feature C build install on integration tests build install on integration tests install on deploy build install on integration tests deploy to stage deploy to prod Git Workflow build install on integration tests deploy to stage deploy to prod build install on integration tests deploy to stage deploy to prod single mainline no commits, only --no-ff merges
  43. 43. build
  44. 44. to recreate What do we need an environment?
  45. 45. Build Packaging
  46. 46. System Storage Database dump media directory (sometimes minified)
  47. 47. StagingIntegrationDevbox ProductionDeploy “Master System”
  48. 48. Settings Urls Database Settings Payment Provider Configuration Feature Flags… Everything that’s different between two environments
  49. 49. EnvSettingsTool
  50. 50. Settings Injection
  51. 51. not go on What should production? Every file is a potential security risk Don’t deploy anything you don’t need to operate the website
  52. 52. .git* ./htdocs/includes ./htdocs/downloader ./htdocs/pkginfo ./htdocs/LICENSE* ./htdocs/RELEASE_NOTES.txt ./htdocs/phpunit.xml* ./htdocs/*.sample ./htdocs/var ./htdocs/media ./.modman/Aoe_TemplateHints ./.modman/Aoe_Profiler ./.modman/EcomDev_PHPUnit …
  53. 53. Extra Base vs. production, staging,… devbox and integration environment
  54. 54. Continuous Integration Deployment Development Delivery Improvement
  55. 55. Jenkins Travis CI Use Jenkins to implement a full deployment pipeline for your projects! Test our Open Source Magento modules with Travis CI!
  56. 56. https://github.com/AOEpeople/MageTestStand MageTestStand
  57. 57. Vagrant Boxes
  58. 58. Awesome stuff Every single line potentially can break production! Bratwurst (Germans) Donut (Americans) Pizza (Italians)
  59. 59. Version Control GIT, SVN,… PHP lint
  60. 60. Version Control Code Reviews Gerrit, Crucible, reVu,…
  61. 61. Code Reviews Build Files DB Installer Settings Triggered by commit, by time or manually Jenkins minify js/css .tar.gz “virtual package” (revision)
  62. 62. Version Control Code Reviews Build Files DB Instal ler Setti ngs
  63. 63. Build Files B ler Settings Static Code Analysis CodeSniffer php-pmd php-cpd php-depend
  64. 64. Static Code Analysis Install on “Integration” First system where code from all developers/teams meets each other unstable system. Might be broken. Will be rebuild without warning
  65. 65. Install on “Integration” Unit Tests PHPUnit Code coverage
  66. 66. Unit Tests Integration Tests AOE PMD (for Magento) broken links, HTTP headers, W3C validity,… Test features that rely on other parts (e.g. framework, external services,…) External components (SSL, Redis, Varnish, ERP, Data Import,…) Basically this is also PHPUnit, but addressing other stuff
  67. 67. Integration Tests Acceptance Tests Selenium Firefox Other Browsers
  68. 68. Acceptance Tests Install on “Staging” Behat (Cucumber) Same setup as production, but different machines Client QA / approval
  69. 69. Install on “Staging” *http://www.slideshare.net/aoemedia/performance-measurement-and-tuning Stress Tests Jmeter Spin up EC2 instances in the cloud to produce traffic*
  70. 70. Again: multiple servers or autoscaling setup Stress Tests Install on “Production” A/B Deployments on different scales Plan B: rollback Zero Downtime
  71. 71. Deployment != Feature Release
  72. 72. Install on “Production” Enable Features
  73. 73. Version Control Code Reviews Build Static Code Analysis Install on “Integration” Unit Tests Integration Tests Acceptance Tests Install on “Staging” Install on “Production” Enable Features
  74. 74. Pipeline Visualization #284 #283 #282 #281 #280 Build Static Code Analysis Unit Tests Install on “Integration” Integration Tests Acceptance Tests #285 for team dashboards
  75. 75. deploy to production deploy to production Continuous Integration Continuous Deployment type “YESIKNOWWHATIMDOING” + click
  76. 76. deploy
  77. 77. Automate!
  78. 78. timespent task size does it manually does it manually gets annoyed writes script to automate runs script wins loses Geeks vs. Non-Geeks Makes fun of geek’s complicated method
  79. 79. Deploy often!
  80. 80. No manual interaction! no creating CMS blocks no creating product attributes no updating of any configuration settings! no clearing the cache no purging CDN or Reverse Proxy content!
  81. 81. Details? High Performance Magento in the Cloud
  82. 82. Deployment Strategy
  83. 83. Deployment Pattern
  84. 84. Build InstallationDeploymentProvisioning
  85. 85. Build What is it? A simple shell script. That’s it! No Ant, no Phing, no fancy PHP tool/framework Where is it used? In the build job of a Jenkins server What does it need to know? Where to get all the source code from What does it do? add metadata, composer, modman, minify, packaging, upload to S3
  86. 86. Provisioning What is it? A couple of chef cookbooks. Run by AWS OpsWorks or Vagrant,… Where is it used? On the target servers What does it need to know? The role of the server (web server, devbox, Redis,…) What does it do? Installing and configuring web server, PHP, MySQL, Redis,…
  87. 87. Deployment What is it? A simple shell script. Or the “deploy” chef cookbook Or EasyDeployWorkflows Where is it used? On the target servers What does it need to know? How to get the build and where to deploy it to What does it do? (Resetting the environment), getting the build, extracting it, managing release folders and symlinks, triggering installer, cleanup old releases
  88. 88. Installation What is it? A simple shell script. Sometimes triggered inside “deploy” before_migrate.rb Where is it used? On the target servers, called from deployment step What does it need to know? What environment this is. Injected settings What does it do? Run modman, connect to shared directories, apply settings, trigger update scripts, customize cache prefix,
  89. 89. Deployment != Installation doesn’t know anything about the app, except where to get it from and where to put it knows how to get the app up and running
  90. 90. Cache Invalidation Cache Warming add build number to id_prefix
  91. 91. Devbox VirtualBox Vagrant Chef
  92. 92. Unibox https://github.com/AOEpeople/chef-devbox
  93. 93. Build InstallationDeploymentProvisioning Chef deploy.sh install.sh Jenkins
  94. 94. Samba VM Host Apache Windows Ubuntu SSH
  95. 95. test
  96. 96. Unit Service UI Acceptance http://www.mountaingoatsoftware.com/blog/the-forgotten-layer-of-the-test-automation-pyramid What the customer thinks the code should do What the developer thinks the code should do Load Test
  97. 97. Menta Selenium 2 Testing Framework for PHP Who is “Menta”? What is “Menta”?
  98. 98. Gallery Screenshot
  99. 99. Perceptual diffs Build X Build X-1 Perceptual Diff
  100. 100. https://plus.google.com/116116065378190790516 Magento Fireside Chats
  101. 101. plan code build test deploy operate
  102. 102. Continuous Improvement
  103. 103. give takeand be part of the community
  104. 104. #SinceIStartedWithMagento
  105. 105. http://joind.in/talk/view/11899
  106. 106. Thank you! Any questions? {firstname}.{lastname}@aoe.com http://www.aoe.com http://www.fabrizio-branca.de @fbrnc Follow me on twitter! My blog

×