Simple Steps to Win the Site
Performance Battle
(and other hopefully interesting stuff)
CHRIS WELLS – CEO – NEXCESS.NET LLC
SOUTHFIELD, MI USA
Quick Facts About Detroit, MI
• It’s the potato chip capital of the world
• based on consumption 
• It had the 1st mile of concrete road (1909)
• Between 6 and 7 mile roads on M-1 (Woodward)
• Home to the only floating Post Office in the U.S.
• J.W.Westcott Co (Zip code: 48222)
• It has the most registered bowlers nationwide
• Biggest municipal bankruptcy in the U.S.
• Which makes us National Bankruptcy Champions!!
• Eat your heart out Jefferson County, AL!! We did 4 x better than you!!
Notes About This Talk
• Lessons learned from MagentoLive Australia
• If masses walk out mid-talk I’m being:
A)WAY too technical
B) Ridiculously boring
• Lessons learned from MagentoLive UK
• If I sweat profusely:
A) DO NOT BEALARMED
B) Assume the technical folks located at the back of the room simply swapped out
the real talk I had prepared with something they thought was more fun and I’m
simply dying a slow 35 minute death on stage before your very eyes.
The Basics NOT On Today’s Agenda
• OS level set up / configuration
• Apache or Nginx
• Magento caching
• Magento EE’s FPC
• Varnish /Turpentine or other reverse proxy caching techniques
• Other performance extensions
• At this point we’re all well aware…
• So let’s dig a little deeper into some fun stuff!
*ON* Today’s Agenda
• PHP – 92%
• MySQL – 3% (if time permits)
• Drives – 5% (if time permits)
• If you leave here today with some food for thought then I’m happy!
PHP – Personal Home Page
• Yes, I know.. It’s now “PHP: Hypertext preprocessor”
• Magento used to be version locked to PHP <= 5.3
• As of January Magento officially supports PHP 5.4! Woohoo!
• You may need to patch your Magento for it to work with 5.4
• So…?
• PHP 5.4 is a good deal faster than 5.3
• Empty hash table optimizations
• Literal tables
• Interned strings
• Zend EngineVM tuning
• But what does this mean for Magento?
Notes On Magento
• Magento is very CPU heavy
• The bulk of this heaviness is within PHP code
• MySQL CPU usage is typically bound by PHP’s CPU usage (speaking in single
server terms)
• Magento is not very I/O heavy (contrary to popular belief)
• Again, I/O is bound by PHP’s CPU usage
• More on I/O later
• Quantitative notes to follow 
Let’s Test Us Some PHP
• 1-hour Siege runs (meant to tax PHP, not MySQL)
• EE tests
• Siege of 20,000 SKUs hitting ~20,000 random URLs
• FPC enabled (Redis)
• CE tests
• Siege of 120 SKUs hitting ~120 URLs.
• Bare metal configuration of:
• Dell R420 – Dual E5-2420 6-core
• H710 512MB NV + 4 x 15K 300GB RAID 10
• 24G RAM
• Ok ok, let’s see what we’ve got!
PHP 5.3.24 (CE Front-end Baseline)
0
20
40
60
80
100
120
140
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 20 25 30 35 40 45 50 55 60
Siegetransacons/second
Dura on (in minutes)
PHP 5.3.24
~42 t/sec
PHP 5.3.24 vs. PHP 5.4.28 (CE Front-end)
0
20
40
60
80
100
120
140
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 20 25 30 35 40 45 50 55 60
Siegetransacons/second
Dura on (in minutes)
PHP 5.3.24
0
20
40
60
80
100
120
140
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 20 25 30 35 40 45 50 55 60
Siegetransacons/second
Dura on (in minutes)
PHP 5.3.24
PHP 5.4.28
~52 t/sec
(~23% increase!)
Magento cache
population
PHP 5.3.24 (EE Front-end Baseline)
0
100
200
300
400
500
600
700
800
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 20 25 30 35 40 45 50 55 60
Siegetransacons/second
Dura on (in minutes)
PHP 5.3.24
~550 t/sec
GO GO FPC!
PHP 5.3.24 vs. PHP 5.4.28 (EE Front-end)
0
100
200
300
400
500
600
700
800
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 20 25 30 35 40 45 50 55 60
Siegetransacons/second
Dura on (in minutes)
PHP 5.3.24
0
100
200
300
400
500
600
700
800
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 20 25 30 35 40 45 50 55 60
Siegetransacons/second
Dura on (in minutes)
PHP 5.3.24
PHP 5.4.28
~615 t/sec
(~12% increase!!)FPC / Magento cache
population amplifies
base PHP version
improvements
PHP 5.3.24 vs. PHP 5.4.28/.14 (EE Front-end)
0
100
200
300
400
500
600
700
800
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 20 25 30 35 40 45 50 55 60
Siegetransacons/second
Dura on (in minutes)
PHP 5.3.24
PHP 5.4.14
PHP 5.4.28
Even upgrading from
5.4.14 shows gains of ~4%
PHP 5.3.24 vs. PHP 5.4.28 (Re-index from CLI)
0
10
20
30
40
50
60
70
80
90
100
PHP 5.3.24
PHP 5.3.24
Time In Seconds
86
88
90
92
94
96
98
100
PHP 5.3.24
PHP 5.4.28
~8% improvement
Let’s Push Things a Little Further...
PHP 5.4.28 vs. PHP 5.5.12 (EE/Front-end)
0
100
200
300
400
500
600
700
800
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 20 25 30 35 40 45 50 55 60
Siegetransacons/second
Dura on (in minutes)
PHP 5.4.28
PHP 5.5.12
~781 t/sec
~25% better than 5.4
~41% better than 5.3
WOW 
FPC / Magento cache
population again
amplifies base PHP
version improvements
Notes On PHP 5.5.x+
• !! Warning – Magento does not yet officially support 5.5+ !!
• APC goes out
• Magento configs may need to be tweaked if you’re using APC’s key/val caching
• Some extensions will need to be updated / re-coded for sure
• OPcache is introduced
• OPcache is the name of the bundled ZendOptimizer+ opcode caching system
• Seems to work out of the box without too much fuss
• More research is needed here – was very surprised with the performance results
• !! Warning – Magento does not yet officially support 5.5+ !!
PHP 5.4.28 vs. PHP 5.5.12 (Re-index from CLI)
85
86
87
88
89
90
91
PHP 5.4.28
PHP 5.5.12
Time In Seconds
~4% improvement
Let’s Push Things a Little Further…
PHP 5.5.12 vs. PHP 5.6.0-BETA2 (EE/Front-end)
0
100
200
300
400
500
600
700
800
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 20 25 30 35 40 45 50 55 60
Siegetransacons/second
Dura on (in minutes)
PHP 5.5.12
PHP 5.6.0 (BETA 2)
Essentially equal within
margin of error
PHP 5.5.12 vs. PHP 5.6.0-BETA2 (Re-index from CLI)
72
74
76
78
80
82
84
86
88
PHP 5.5.12
PHP 5.6.0 BETA2
Time In Seconds
BUT, we got ~11%
improvement on the re-index!
~21.5% better than 5.3.24!
Let’s…
PHP 5.5.12 vs. PHP-NG (EE/Front-end)
Notes on PHP-NG
• Removes numerous heap allocations (and de-allocations)
• Stores more native data directly on the stack
• Removes the need to garbage collect basic primitives (bool, long, etc)
• PHP’s reportedWordpress benchmarks show very good results
• 26.75 sec -> 18.95 sec (36% improvement)
• 9.5M instructions -> 5M internal instructions executed (90% reduction)
• Take some comfort in knowing that more gains are on the way from
PHP folks directly
Simple Step #1 – Use PHP 5.4.28 (for now)
**But watch out for future PHP releases!
MySQL – “My SQL”
• My is the Michael Widinius’s daughter’s name (this was news to me)
• Lots of “brand” options on the MySQL front unlike PHP
• MariaDB
• Percona Server / PerconaCluster
• Oracle’s MySQL
• Equally many options on the storage engine front
• MyISAM
• InnoDB
• XtraDB
Let’s Test Us Some MySQL
• ~1 hour re-index runs (ideally meant to put some load on MySQL)
• Full CLI-based re-index
• Tested latest EE with various versions/brands of MySQL
• DB contained 800,000 SKUs (thanks Magento performance team)
• Bare metal configuration of:
• Dell R420 – Dual E5-2420 6-core
• H710 512MB NV + 4 x 15K 300GB RAID 10
• 24G RAM
• Ok ok, let’s see what we’ve got!
Magento CLI Re-index (EE)
Time In Seconds
3100
3150
3200
3250
3300
3350
3400
3450
3500
3550
MySQL 5.1.71
Percona Cluster 5.5.34
MariaDB 10.0.10
MySQL 5.6.17
Percona Server 5.6.17
Magento CLI Re-index (EE)
Time In Seconds
0
5
10
15
20
25
Percona Server 5.6.17
MariaDB 10.0.10
MySQL 5.1.71
Percona Cluster 5.5.34
MySQL 5.6.17
Time Spent in "sys"
Thoughts On The Tests
• Re-index runs, while DB intensive, are still PHP heavy
• Oracle’s MySQL did surprisingly well
• If concurrency were higher I’d expect Percona/MariaDB to excel
• DB bound writes (beyond the re-index) were missing
• This would tax the DB servers considerably
• TCP (and other) overhead for Percona Cluster doesn’t seem to kill
performance
Simple Step #2 –
Use a modern version of MySQL (ideally XtraDB based)
Hard and Solid State Disk Drives
• Hard drive vs. disk drive vs. solid state drive vs. SAS, etc.
• IOPs (I/O operations / sec)
• More is better, but don’t throw money away
• All SSDs not created equal
• Don’t believe quoted specs!
• Don’t buy consumer grade SSDs!
• Quality SSDs are ~2x the cost of quality HDD of same size
• Know your workload profile!
Live Client Case Study
• Magento EE 1.12.0.1
• ~700,000 SKUs
• 350+ orders / hour
• 15 x web servers (bare metal)
• Dell R420, Dual E5-2450 8-core, 24GB RAM
• 2 x DB servers (replicated) – peak @ ~20,000 qps
• 1 x file server (shared storage for media)
• FPC / Magento cache in Redis
• ~1 week sample size
Database Server IO Requirements
Read Write
Average IOPs: 3.66 471
Min IOPs: 0.0 6.87
Max IOPs: 830 2250
Standard deviation: 34.5 197.0
95% below: 60.4 795
99.999% below: 151 1310
• Percona Server 5.5.35
• 128GB RAM / 60GB MySQL DB (innodb_buffer_pool = 75G)
• 8 x 300GB 15k HDD/SAS RAID 10 - H700/1G RAID card
• Replicating to a hot-standby DB server (unused by Magento)
???
File Server IO Requirements
Read Write
Average IOPs: 13.7 7.65
Min IOPs: 5.58 2.97
Max IOPs: 85.8 186
Standard deviation: 6.72 11.0
95% below: 24.7 25.8
99.999% below: 42.3 54.6
• Serving /media and /var only over NFS
• 24GB RAM - 600GB data on disk
• 4 x 600GB 15k HDD/SAS (RAID 10) - H700/512M RAID card
Notes On Drives
• Get quality drives regardless of technology
• No consumer SSDs!
• Caches are very good at relieving disk-bound I/O
• MySQL query cache
• Redis/memcaches
• RAID card caches
• On-disk controller caches
• Invest in CPU over IOPs unless you can afford both
• Magento is typically bound by CPU, not I/O – Know your workload
profile
Simple Step #3 –
Use HDD & SSD drives as needs/costs dictate
Thank you!
** Thanks for not walking out while I sweat. Come collect your free bag of chips!!
CHRIS WELLS – CLWELLS -AT- NEXCESS.NET

Nexcess Magento Imagine 2014 Performance Breakout

  • 1.
    Simple Steps toWin the Site Performance Battle (and other hopefully interesting stuff) CHRIS WELLS – CEO – NEXCESS.NET LLC SOUTHFIELD, MI USA
  • 2.
    Quick Facts AboutDetroit, MI • It’s the potato chip capital of the world • based on consumption  • It had the 1st mile of concrete road (1909) • Between 6 and 7 mile roads on M-1 (Woodward) • Home to the only floating Post Office in the U.S. • J.W.Westcott Co (Zip code: 48222) • It has the most registered bowlers nationwide • Biggest municipal bankruptcy in the U.S. • Which makes us National Bankruptcy Champions!! • Eat your heart out Jefferson County, AL!! We did 4 x better than you!!
  • 3.
    Notes About ThisTalk • Lessons learned from MagentoLive Australia • If masses walk out mid-talk I’m being: A)WAY too technical B) Ridiculously boring • Lessons learned from MagentoLive UK • If I sweat profusely: A) DO NOT BEALARMED B) Assume the technical folks located at the back of the room simply swapped out the real talk I had prepared with something they thought was more fun and I’m simply dying a slow 35 minute death on stage before your very eyes.
  • 4.
    The Basics NOTOn Today’s Agenda • OS level set up / configuration • Apache or Nginx • Magento caching • Magento EE’s FPC • Varnish /Turpentine or other reverse proxy caching techniques • Other performance extensions • At this point we’re all well aware… • So let’s dig a little deeper into some fun stuff!
  • 5.
    *ON* Today’s Agenda •PHP – 92% • MySQL – 3% (if time permits) • Drives – 5% (if time permits) • If you leave here today with some food for thought then I’m happy!
  • 6.
    PHP – PersonalHome Page • Yes, I know.. It’s now “PHP: Hypertext preprocessor” • Magento used to be version locked to PHP <= 5.3 • As of January Magento officially supports PHP 5.4! Woohoo! • You may need to patch your Magento for it to work with 5.4 • So…? • PHP 5.4 is a good deal faster than 5.3 • Empty hash table optimizations • Literal tables • Interned strings • Zend EngineVM tuning • But what does this mean for Magento?
  • 7.
    Notes On Magento •Magento is very CPU heavy • The bulk of this heaviness is within PHP code • MySQL CPU usage is typically bound by PHP’s CPU usage (speaking in single server terms) • Magento is not very I/O heavy (contrary to popular belief) • Again, I/O is bound by PHP’s CPU usage • More on I/O later • Quantitative notes to follow 
  • 8.
    Let’s Test UsSome PHP • 1-hour Siege runs (meant to tax PHP, not MySQL) • EE tests • Siege of 20,000 SKUs hitting ~20,000 random URLs • FPC enabled (Redis) • CE tests • Siege of 120 SKUs hitting ~120 URLs. • Bare metal configuration of: • Dell R420 – Dual E5-2420 6-core • H710 512MB NV + 4 x 15K 300GB RAID 10 • 24G RAM • Ok ok, let’s see what we’ve got!
  • 9.
    PHP 5.3.24 (CEFront-end Baseline) 0 20 40 60 80 100 120 140 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 20 25 30 35 40 45 50 55 60 Siegetransacons/second Dura on (in minutes) PHP 5.3.24 ~42 t/sec
  • 10.
    PHP 5.3.24 vs.PHP 5.4.28 (CE Front-end) 0 20 40 60 80 100 120 140 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 20 25 30 35 40 45 50 55 60 Siegetransacons/second Dura on (in minutes) PHP 5.3.24 0 20 40 60 80 100 120 140 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 20 25 30 35 40 45 50 55 60 Siegetransacons/second Dura on (in minutes) PHP 5.3.24 PHP 5.4.28 ~52 t/sec (~23% increase!) Magento cache population
  • 11.
    PHP 5.3.24 (EEFront-end Baseline) 0 100 200 300 400 500 600 700 800 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 20 25 30 35 40 45 50 55 60 Siegetransacons/second Dura on (in minutes) PHP 5.3.24 ~550 t/sec GO GO FPC!
  • 12.
    PHP 5.3.24 vs.PHP 5.4.28 (EE Front-end) 0 100 200 300 400 500 600 700 800 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 20 25 30 35 40 45 50 55 60 Siegetransacons/second Dura on (in minutes) PHP 5.3.24 0 100 200 300 400 500 600 700 800 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 20 25 30 35 40 45 50 55 60 Siegetransacons/second Dura on (in minutes) PHP 5.3.24 PHP 5.4.28 ~615 t/sec (~12% increase!!)FPC / Magento cache population amplifies base PHP version improvements
  • 13.
    PHP 5.3.24 vs.PHP 5.4.28/.14 (EE Front-end) 0 100 200 300 400 500 600 700 800 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 20 25 30 35 40 45 50 55 60 Siegetransacons/second Dura on (in minutes) PHP 5.3.24 PHP 5.4.14 PHP 5.4.28 Even upgrading from 5.4.14 shows gains of ~4%
  • 14.
    PHP 5.3.24 vs.PHP 5.4.28 (Re-index from CLI) 0 10 20 30 40 50 60 70 80 90 100 PHP 5.3.24 PHP 5.3.24 Time In Seconds 86 88 90 92 94 96 98 100 PHP 5.3.24 PHP 5.4.28 ~8% improvement
  • 15.
    Let’s Push Thingsa Little Further...
  • 16.
    PHP 5.4.28 vs.PHP 5.5.12 (EE/Front-end) 0 100 200 300 400 500 600 700 800 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 20 25 30 35 40 45 50 55 60 Siegetransacons/second Dura on (in minutes) PHP 5.4.28 PHP 5.5.12 ~781 t/sec ~25% better than 5.4 ~41% better than 5.3 WOW  FPC / Magento cache population again amplifies base PHP version improvements
  • 17.
    Notes On PHP5.5.x+ • !! Warning – Magento does not yet officially support 5.5+ !! • APC goes out • Magento configs may need to be tweaked if you’re using APC’s key/val caching • Some extensions will need to be updated / re-coded for sure • OPcache is introduced • OPcache is the name of the bundled ZendOptimizer+ opcode caching system • Seems to work out of the box without too much fuss • More research is needed here – was very surprised with the performance results • !! Warning – Magento does not yet officially support 5.5+ !!
  • 18.
    PHP 5.4.28 vs.PHP 5.5.12 (Re-index from CLI) 85 86 87 88 89 90 91 PHP 5.4.28 PHP 5.5.12 Time In Seconds ~4% improvement
  • 19.
    Let’s Push Thingsa Little Further…
  • 20.
    PHP 5.5.12 vs.PHP 5.6.0-BETA2 (EE/Front-end) 0 100 200 300 400 500 600 700 800 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 20 25 30 35 40 45 50 55 60 Siegetransacons/second Dura on (in minutes) PHP 5.5.12 PHP 5.6.0 (BETA 2) Essentially equal within margin of error
  • 21.
    PHP 5.5.12 vs.PHP 5.6.0-BETA2 (Re-index from CLI) 72 74 76 78 80 82 84 86 88 PHP 5.5.12 PHP 5.6.0 BETA2 Time In Seconds BUT, we got ~11% improvement on the re-index! ~21.5% better than 5.3.24!
  • 22.
  • 23.
    PHP 5.5.12 vs.PHP-NG (EE/Front-end)
  • 24.
    Notes on PHP-NG •Removes numerous heap allocations (and de-allocations) • Stores more native data directly on the stack • Removes the need to garbage collect basic primitives (bool, long, etc) • PHP’s reportedWordpress benchmarks show very good results • 26.75 sec -> 18.95 sec (36% improvement) • 9.5M instructions -> 5M internal instructions executed (90% reduction) • Take some comfort in knowing that more gains are on the way from PHP folks directly
  • 25.
    Simple Step #1– Use PHP 5.4.28 (for now) **But watch out for future PHP releases!
  • 26.
    MySQL – “MySQL” • My is the Michael Widinius’s daughter’s name (this was news to me) • Lots of “brand” options on the MySQL front unlike PHP • MariaDB • Percona Server / PerconaCluster • Oracle’s MySQL • Equally many options on the storage engine front • MyISAM • InnoDB • XtraDB
  • 27.
    Let’s Test UsSome MySQL • ~1 hour re-index runs (ideally meant to put some load on MySQL) • Full CLI-based re-index • Tested latest EE with various versions/brands of MySQL • DB contained 800,000 SKUs (thanks Magento performance team) • Bare metal configuration of: • Dell R420 – Dual E5-2420 6-core • H710 512MB NV + 4 x 15K 300GB RAID 10 • 24G RAM • Ok ok, let’s see what we’ve got!
  • 28.
    Magento CLI Re-index(EE) Time In Seconds 3100 3150 3200 3250 3300 3350 3400 3450 3500 3550 MySQL 5.1.71 Percona Cluster 5.5.34 MariaDB 10.0.10 MySQL 5.6.17 Percona Server 5.6.17
  • 29.
    Magento CLI Re-index(EE) Time In Seconds 0 5 10 15 20 25 Percona Server 5.6.17 MariaDB 10.0.10 MySQL 5.1.71 Percona Cluster 5.5.34 MySQL 5.6.17 Time Spent in "sys"
  • 30.
    Thoughts On TheTests • Re-index runs, while DB intensive, are still PHP heavy • Oracle’s MySQL did surprisingly well • If concurrency were higher I’d expect Percona/MariaDB to excel • DB bound writes (beyond the re-index) were missing • This would tax the DB servers considerably • TCP (and other) overhead for Percona Cluster doesn’t seem to kill performance
  • 31.
    Simple Step #2– Use a modern version of MySQL (ideally XtraDB based)
  • 32.
    Hard and SolidState Disk Drives • Hard drive vs. disk drive vs. solid state drive vs. SAS, etc. • IOPs (I/O operations / sec) • More is better, but don’t throw money away • All SSDs not created equal • Don’t believe quoted specs! • Don’t buy consumer grade SSDs! • Quality SSDs are ~2x the cost of quality HDD of same size • Know your workload profile!
  • 33.
    Live Client CaseStudy • Magento EE 1.12.0.1 • ~700,000 SKUs • 350+ orders / hour • 15 x web servers (bare metal) • Dell R420, Dual E5-2450 8-core, 24GB RAM • 2 x DB servers (replicated) – peak @ ~20,000 qps • 1 x file server (shared storage for media) • FPC / Magento cache in Redis • ~1 week sample size
  • 34.
    Database Server IORequirements Read Write Average IOPs: 3.66 471 Min IOPs: 0.0 6.87 Max IOPs: 830 2250 Standard deviation: 34.5 197.0 95% below: 60.4 795 99.999% below: 151 1310 • Percona Server 5.5.35 • 128GB RAM / 60GB MySQL DB (innodb_buffer_pool = 75G) • 8 x 300GB 15k HDD/SAS RAID 10 - H700/1G RAID card • Replicating to a hot-standby DB server (unused by Magento) ???
  • 35.
    File Server IORequirements Read Write Average IOPs: 13.7 7.65 Min IOPs: 5.58 2.97 Max IOPs: 85.8 186 Standard deviation: 6.72 11.0 95% below: 24.7 25.8 99.999% below: 42.3 54.6 • Serving /media and /var only over NFS • 24GB RAM - 600GB data on disk • 4 x 600GB 15k HDD/SAS (RAID 10) - H700/512M RAID card
  • 36.
    Notes On Drives •Get quality drives regardless of technology • No consumer SSDs! • Caches are very good at relieving disk-bound I/O • MySQL query cache • Redis/memcaches • RAID card caches • On-disk controller caches • Invest in CPU over IOPs unless you can afford both • Magento is typically bound by CPU, not I/O – Know your workload profile
  • 37.
    Simple Step #3– Use HDD & SSD drives as needs/costs dictate
  • 38.
    Thank you! ** Thanksfor not walking out while I sweat. Come collect your free bag of chips!! CHRIS WELLS – CLWELLS -AT- NEXCESS.NET