Magento Deployments
Rock-Solid
…and Development
php[world] 2014 – Washington D.C.
Fabrizio Branca
fbrnc
fbrnc
E-Commerce:
Magento
CMS:
TYPO3
Portals:
ZF, FLOW,…
Mobile Searchperience:
SOLR
>120 people
world-wide
High
Performance
/Scale
Global
Enterprise
Projects
San Francisco, CA
Janine
Fiona
that’s me
Leo
Who’s a Magento developer? Who spends a significant
amount of his/her time dealing
with development/deployment
processes?
Who thinks he/she’s got it
figured out?
YOU...?
Picture
The BIG
plan
code
build
test
deploy
operate
time
complexity
This is when we
“deployed” using
a FTP client
Keep it
simple!
http://xkcd.com/974/
Youain’tgonnaneedit!
code
Vocabulary
CMS
Magento
Solr
ERP
Project
Applications
MMagento
117
Magento
118
Magento
119
Builds
Release
Staging
Magento
72
Magento
103
Magento
119
Magento
72
Magento
103
Magento
119
Magento
72
Magento
103
Magento
119
Environment
Servers
Instances
StagingIntegrationDevbox ProductionDeploy
transient,
potentially broken
StagingIntegrationDevbox ProductionDeploy
VirtualBox
+ Vagrant
Jenkins
Server
identical
setup
Build Package
Application
System Storage
User Data
media
directory
(sometimes
minified)
Database
dump
Magento
core
Modules
Scripts
Settings
Tools
Structure
Project
project web
root root!=
This is the complete
project
This is where your
webserver points to.
project web
root root
htdocs
<Name>
<theme>
layout
template
code
design
core
community
local
<area>
<Namespace>
<package>
app
Structure
Magento
htdocs
<Name>
<theme>
layout
template
code
design
etc
locale
<area>
core
community
local
<area>
<language>
modules
<package>
<Namespace>
<package>
*.xml
*.csv
<theme>
app
skin
lib
Module2
default
layout
template
code
design
etc
locale
frontend
local
frontend
de_DE
modules
base
My
base
My_Module2.xml
My_Module2.csv
default
app
skin
etc, Block, …
js, css, img, …
my_module2
my_module2
Magento My_Module
My_Module2
Module
default
layout
template
code
design
etc
locale
frontend
local
frontend
de_DE
modules
base
My
base
My_Module.xml
My_Module.csv
default
app
skin
etc, Block, …
js, css, img, …
my_module
my_module
htdocs
Module
default
layout
template
code
design
etc
locale
local
frontend
de_DE
modules
My
base
*.xml
app
My_Module2.xml
My_Module2.csv
my_module2
My_Module.xml
My_Module.csv
etc, Block, …
my_module
Module2
etc, Block, …
my_module2
my_module
modman
by @colinmollenhour
htdocs
code
core
community
local
app
app/code/…/My/Module
.modman
My_Module
app/design/…
skin/frontend/…
app/etc/modules/My_Module.xml
modman
app/code/…/My/Module
My_Module2
app/design/…
skin/frontend/…
app/etc/modules/My_Module.xml
modman
<theme>
layout
template
design
etc
locale
<area>
<language>
modules
<package>
*.xml
modman
doesn’t even need to
be in the web root!
Vanilla
Magento
absolutely no
core hacks!
…and core bugs
you’ve fixed
(and submitted to
Magento, of course…)
(except official
patches)
“wiring”
Intro
modman “connects” existing modules to the Magento core,
but how do we manage them?
htdocs
code
core
community
local
app
app/code/…/My/Module
.modman
My_Module
app/design/…
skin/frontend/…
app/etc/modules/My_Module.xml
modman
app/code/…/My/Module
My_Module2
app/design/…
skin/frontend/…
app/etc/modules/My_Module.xml
modman
<theme>
layout
template
design
etc
locale
<area>
<language>
modules
<package>
*.xml
Components
tools
n98-magerun.phar
phpunit.phar
modman
EnvSettingsTool
composer.phar
htdocs
.modman
Configuration
Installer
(Metadata, Doc,…)
My_Module
My_Module2
app,…
and usually there’s
even more stuff…
These should come
from different origins
Composer
magento-hackathon/
magento-composer-installer
Composer
AOEpeople/composer-installers
"replace": { "magento-hackathon/magento-composer-installer":"*" }
“inspired” by
composer/installers
“magento-module” “magento-source”
handles package types
<?php
namespace AoepeopleComposerInstallers;
class MagentoInstaller extends BaseInstaller
{
protected $locations = array(
'module' => '.modman/{$name}/',
'source' => 'htdocs/'
);
}
AOEpeople/composer-installers
• Fully compatible with magento-hackathon/magento-composer-installer
• Adds support for type “magento-source”
• Only puts packages into place
• Native modman script is used to deploy modules
• Wrapped in a simple installer script
AOEpeople/composer-installers
{
"minimum-stability":"dev",
"require":{
"aoepeople/composer-installers": "*",
"fbrnc/aoe_scheduler": "*",
"fbrnc/aoe_cachecleaner": "*",
"aoepeople/magento_enterprise": "1.13.1.0",
"aoepeople/envsettingstool": "*"
},
"config": {
"bin-dir": "bin"
}
}
AOEpeople/composer-installers
replaces
Hackathon
Installer
some
modules
Vanilla
Magento
Source
Tools,…
Directory layout
Webroot:
/var/www/<projectName>/<environmentName>/current/htdocs
var
var
<projectName>
releases
<environmentName>
shared
www
media
build_117
build_118
build_119
.modman
htdocs
Configuration
tools
install.sh
app,…
var
media
previous
latest
current
next
My_Module
My_Module2
Magento Modules
3rd Party
discover use
code
review
add
modman
add
composer
git integrate test
deploy
Magento Connect
The Right Thing™
one-click
install
download
good luck
with that!
Code Quality
Modules
Not enterprise
ready
Not production
ready
Not cloud
ready
Bad
performance
Security
0%
100%
Writing
Code
Time spent…
5 years ago today
Drinking Coffee
Drinking Coffee
while waiting for builds/tests
Twitter
VCS
Deployment
Design / Architecture /
Infrastructure
Writing Code
Ping Pong
Think about it
master
Feature B
Feature A
Hotfix
(if master is not modified)
Hotfix
(if master is modified)
Feature C
build
install on
integration
tests
build
install on
integration
tests
install on
deploy
build
install on
integration
tests
deploy to
stage
deploy to
prod
Git Workflow
build
install on
integration
tests
deploy to
stage
deploy to
prod
build
install on
integration
tests
deploy to
stage
deploy to
prod
single
mainline
no commits,
only --no-ff
merges
build
to recreate
What do we need
an environment?
Build
Packaging
System Storage
Database
dump
media
directory
(sometimes
minified)
StagingIntegrationDevbox ProductionDeploy
“Master System”
Settings
Urls
Database
Settings
Payment
Provider
Configuration
Feature
Flags…
Everything that’s
different between
two environments
EnvSettingsTool
Settings
Injection
not go on
What should
production?
Every file is a
potential security
risk
Don’t deploy
anything you don’t
need to operate the
website
.git*
./htdocs/includes
./htdocs/downloader
./htdocs/pkginfo
./htdocs/LICENSE*
./htdocs/RELEASE_NOTES.txt
./htdocs/phpunit.xml*
./htdocs/*.sample
./htdocs/var
./htdocs/media
./.modman/Aoe_TemplateHints
./.modman/Aoe_Profiler
./.modman/EcomDev_PHPUnit
…
Extra
Base vs.
production,
staging,…
devbox and
integration
environment
Continuous
Integration
Deployment
Development
Delivery
Improvement
Jenkins Travis CI
Use Jenkins to implement a
full deployment pipeline for
your projects!
Test our Open Source
Magento modules
with Travis CI!
https://github.com/AOEpeople/MageTestStand
MageTestStand
Vagrant
Boxes
Awesome
stuff
Every single
line potentially
can break
production!
Bratwurst
(Germans)
Donut
(Americans)
Pizza
(Italians)
Version Control
GIT, SVN,…
PHP lint
Version Control Code Reviews
Gerrit,
Crucible,
reVu,…
Code Reviews Build
Files
DB
Installer
Settings
Triggered by
commit, by time
or manually
Jenkins
minify js/css
.tar.gz
“virtual package”
(revision)
Version
Control
Code
Reviews
Build
Files
DB
Instal
ler
Setti
ngs
Build
Files
B
ler
Settings
Static Code
Analysis
CodeSniffer
php-pmd
php-cpd
php-depend
Static Code
Analysis
Install on
“Integration”
First system where
code from all
developers/teams
meets each other
unstable system.
Might be broken.
Will be rebuild
without warning
Install on
“Integration”
Unit Tests
PHPUnit
Code
coverage
Unit Tests Integration Tests
AOE PMD
(for Magento)
broken links,
HTTP headers,
W3C validity,…
Test features that rely on other
parts (e.g. framework, external
services,…)
External components
(SSL, Redis, Varnish,
ERP, Data Import,…)
Basically this is also
PHPUnit, but
addressing other stuff
Integration Tests Acceptance Tests
Selenium
Firefox
Other Browsers
Acceptance Tests
Install on
“Staging”
Behat
(Cucumber)
Same setup as production,
but different machines
Client QA /
approval
Install on
“Staging”
*http://www.slideshare.net/aoemedia/performance-measurement-and-tuning
Stress
Tests
Jmeter
Spin up EC2
instances in the
cloud to produce
traffic*
Again:
multiple servers or
autoscaling setup
Stress
Tests
Install on
“Production”
A/B
Deployments
on different
scales
Plan B:
rollback
Zero Downtime
Deployment !=
Feature Release
Install on
“Production”
Enable Features
Version
Control
Code
Reviews
Build Static Code
Analysis
Install on
“Integration”
Unit Tests Integration
Tests
Acceptance
Tests
Install on
“Staging”
Install on
“Production”
Enable
Features
Pipeline Visualization
#284
#283
#282
#281
#280
Build
Static Code
Analysis
Unit
Tests
Install on
“Integration”
Integration
Tests
Acceptance
Tests
#285
for team dashboards
deploy to
production
deploy to
production
Continuous Integration
Continuous Deployment
type “YESIKNOWWHATIMDOING”
+ click
deploy
Automate!
timespent
task size
does it
manually
does it
manually
gets annoyed
writes script to
automate
runs script
wins
loses
Geeks vs. Non-Geeks
Makes fun of geek’s
complicated method
Deploy often!
No manual interaction!
no creating
CMS blocks
no creating
product attributes
no updating of any
configuration settings!
no clearing
the cache
no purging CDN
or Reverse Proxy
content!
Details?
High Performance
Magento in the Cloud
Deployment
Strategy
Deployment
Pattern
Build InstallationDeploymentProvisioning
Build
What is it?
A simple shell script. That’s it!
No Ant, no Phing, no fancy PHP tool/framework
Where is it used?
In the build job of a Jenkins server
What does it need to know?
Where to get all the source code from
What does it do?
add metadata, composer, modman, minify,
packaging, upload to S3
Provisioning
What is it?
A couple of chef cookbooks.
Run by AWS OpsWorks or Vagrant,…
Where is it used?
On the target servers
What does it need to know?
The role of the server (web server, devbox, Redis,…)
What does it do?
Installing and configuring web server, PHP,
MySQL, Redis,…
Deployment
What is it?
A simple shell script.
Or the “deploy” chef cookbook
Or EasyDeployWorkflows
Where is it used?
On the target servers
What does it need to know?
How to get the build and where to deploy it to
What does it do?
(Resetting the environment), getting the build, extracting it,
managing release folders and symlinks, triggering installer,
cleanup old releases
Installation
What is it?
A simple shell script.
Sometimes triggered inside “deploy” before_migrate.rb
Where is it used?
On the target servers, called from deployment step
What does it need to know?
What environment this is. Injected settings
What does it do?
Run modman, connect to shared directories, apply settings,
trigger update scripts, customize cache prefix,
Deployment !=
Installation
doesn’t know anything about the
app, except where to get it from
and where to put it
knows how to get
the app up and running
Cache
Invalidation
Cache
Warming
add build number
to id_prefix
Devbox
VirtualBox Vagrant Chef
Unibox
https://github.com/AOEpeople/chef-devbox
Build InstallationDeploymentProvisioning
Chef
deploy.sh install.sh
Jenkins
Samba
VM
Host
Apache
Windows Ubuntu
SSH
test
Unit
Service
UI
Acceptance
http://www.mountaingoatsoftware.com/blog/the-forgotten-layer-of-the-test-automation-pyramid
What the
customer thinks
the code should do
What the
developer thinks
the code should do
Load
Test
Menta
Selenium 2 Testing Framework for PHP
Who is
“Menta”?
What is
“Menta”?
Gallery
Screenshot
Perceptual diffs
Build X
Build X-1
Perceptual
Diff
https://plus.google.com/116116065378190790516
Magento
Fireside
Chats
plan
code
build
test
deploy
operate
Continuous
Improvement
give takeand
be part of the
community
#SinceIStartedWithMagento
http://joind.in/talk/view/11899
Thank you!
Any questions?
{firstname}.{lastname}@aoe.com
http://www.aoe.com
http://www.fabrizio-branca.de
@fbrnc
Follow me on twitter!
My blog

Rock-solid Magento Deployments (and Development)