Scaling PHP to 40 Million Uniques

  • 5,441 views
Uploaded on

This is a talk I gave at Northeast PHP 2013 in Boston, MA. …

This is a talk I gave at Northeast PHP 2013 in Boston, MA.

Links: jkle.in/nephp

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
5,441
On Slideshare
0
From Embeds
0
Number of Embeds
5

Actions

Shares
Downloads
43
Comments
0
Likes
14

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Scaling PHP to 40 Million Uniques Northeast PHP 2013 Jonathan Klein @jonathanklein Saturday, August 17, 13
  • 2. Scaling PHP to 40 Million Uniques Northeast PHP 2013 Jonathan Klein @jonathanklein Saturday, August 17, 13
  • 3. Scaling PHP to 60 Million Uniques Northeast PHP 2013 Jonathan Klein @jonathanklein Saturday, August 17, 13
  • 4. Who Am I? Saturday, August 17, 13
  • 5. Who Am I? • Performance Engineer at Etsy Saturday, August 17, 13
  • 6. Who Am I? • Performance Engineer at Etsy • Boston Web Perf Meetup Organizer Saturday, August 17, 13
  • 7. Who Am I? • Performance Engineer at Etsy • Boston Web Perf Meetup Organizer • Previously at Wayfair Saturday, August 17, 13
  • 8. Who Am I? • Performance Engineer at Etsy • Boston Web Perf Meetup Organizer • Previously at Wayfair • Led team that converted to PHP Saturday, August 17, 13
  • 9. Slides, Links: jkle.in/nephp Saturday, August 17, 13
  • 10. codeascraft.com Saturday, August 17, 13
  • 11. Some Etsy Stats Saturday, August 17, 13
  • 12. Some Etsy Stats • 1.5 billion page views/month Saturday, August 17, 13
  • 13. Some Etsy Stats • 1.5 billion page views/month • Almost $1B in sales last year Saturday, August 17, 13
  • 14. Some Etsy Stats • 1.5 billion page views/month • Almost $1B in sales last year • Over 1M lines of PHP Saturday, August 17, 13
  • 15. Some Etsy Stats • 1.5 billion page views/month • Almost $1B in sales last year • Over 1M lines of PHP • 60M+ unique visitors/month Saturday, August 17, 13
  • 16. 2012 Traffic Saturday, August 17, 13
  • 17. August 2012 Traffic Saturday, August 17, 13
  • 18. August 2012 Traffic 50% Higher Saturday, August 17, 13
  • 19. Holidays: ~90M Uniques Saturday, August 17, 13
  • 20. Saturday, August 17, 13
  • 21. Saturday, August 17, 13
  • 22. A few others... Saturday, August 17, 13
  • 23. A few others... • Search (Solr) Saturday, August 17, 13
  • 24. A few others... • Search (Solr) • Gearman Saturday, August 17, 13
  • 25. A few others... • Search (Solr) • Gearman • Redis Saturday, August 17, 13
  • 26. A few others... • Search (Solr) • Gearman • Redis • Postgres (legacy) Saturday, August 17, 13
  • 27. Saturday, August 17, 13
  • 28. Hardware (Supermicro) Saturday, August 17, 13
  • 29. Hardware (Supermicro) 2x 8-core Intel E5-2960 CPUs Saturday, August 17, 13
  • 30. Hardware (Supermicro) 2x 8-core Intel E5-2960 CPUs 24GB of RAM Saturday, August 17, 13
  • 31. Hardware (Supermicro) 2x 8-core Intel E5-2960 CPUs 24GB of RAM 160GB SSD Saturday, August 17, 13
  • 32. Apache Saturday, August 17, 13
  • 33. Apache • Apache 2.2 Saturday, August 17, 13
  • 34. Apache • Apache 2.2 • Prefork MPM Saturday, August 17, 13
  • 35. Apache • Apache 2.2 • Prefork MPM • mod_php5 Saturday, August 17, 13
  • 36. Apache • Apache 2.2 • Prefork MPM • mod_php5 StartServers 30 MinSpareServers 30 MaxSpareServers 60 ServerLimit 60 MaxClients 60 MaxRequestsPerChild 0 Saturday, August 17, 13
  • 37. PHP Saturday, August 17, 13
  • 38. PHP • PHP 5.4 Saturday, August 17, 13
  • 39. PHP • PHP 5.4 • Zend OPCache (3GB Memory Segment) Saturday, August 17, 13
  • 40. PHP • PHP 5.4 • Zend OPCache (3GB Memory Segment) • memory_limit: 128M Saturday, August 17, 13
  • 41. PHP • PHP 5.4 • Zend OPCache (3GB Memory Segment) • memory_limit: 128M • max_execution_time: 30 Saturday, August 17, 13
  • 42. Optimizing PHP http://www.slideshare.net/jnklein/northeast-php-high-performance-php Saturday, August 17, 13
  • 43. Optimizing PHP • See last year’s talk http://www.slideshare.net/jnklein/northeast-php-high-performance-php Saturday, August 17, 13
  • 44. Optimizing PHP • See last year’s talk • Use an Opcode Cache http://www.slideshare.net/jnklein/northeast-php-high-performance-php Saturday, August 17, 13
  • 45. Optimizing PHP • See last year’s talk • Use an Opcode Cache • xhprof http://www.slideshare.net/jnklein/northeast-php-high-performance-php Saturday, August 17, 13
  • 46. Optimizing PHP • See last year’s talk • Use an Opcode Cache • xhprof • StatsD/Graphite http://www.slideshare.net/jnklein/northeast-php-high-performance-php Saturday, August 17, 13
  • 47. Optimizing PHP • See last year’s talk • Use an Opcode Cache • xhprof • StatsD/Graphite • Find hotspots http://www.slideshare.net/jnklein/northeast-php-high-performance-php Saturday, August 17, 13
  • 48. Optimizing PHP • See last year’s talk • Use an Opcode Cache • xhprof • StatsD/Graphite • Find hotspots • Upgrade it http://www.slideshare.net/jnklein/northeast-php-high-performance-php Saturday, August 17, 13
  • 49. Static Arrays Saturday, August 17, 13
  • 50. Translations class language_de { static $translations = array( "<hash>" => array("content" => 'Accessoires', "file" => "TAXONOMY"), "<hash>" => array("content" => 'Keramik', "file" => "CATEGORY"), "<hash>" => array("content" => 'Dekorieren', "file" => "CATEGORY"), "<hash>" => array("content" => 'Getaggt %s', "file" => "Foo.php"), "<hash>" => array("content" => 'Badezusatz', "file" => "CATEGORY"), "<hash>" => array("content" => 'Datum:{% $date %}', "file" => "bar.tpl") ... snip ... ); } Saturday, August 17, 13
  • 51. Saturday, August 17, 13
  • 52. Saturday, August 17, 13
  • 53. Atomic Deploys Saturday, August 17, 13
  • 54. Deploying PHP http://codeascraft.com/2013/07/01/atomic-deploys-at-etsy/ Saturday, August 17, 13
  • 55. Deploying PHP • mod_realdoc - Apache module http://codeascraft.com/2013/07/01/atomic-deploys-at-etsy/ Saturday, August 17, 13
  • 56. Deploying PHP • mod_realdoc - Apache module • incpath - PHP Extension http://codeascraft.com/2013/07/01/atomic-deploys-at-etsy/ Saturday, August 17, 13
  • 57. Deploying PHP • mod_realdoc - Apache module • incpath - PHP Extension • A/B symlink swap http://codeascraft.com/2013/07/01/atomic-deploys-at-etsy/ Saturday, August 17, 13
  • 58. Deploying PHP • mod_realdoc - Apache module • incpath - PHP Extension • A/B symlink swap • Avoid recompilation to opcodes http://codeascraft.com/2013/07/01/atomic-deploys-at-etsy/ Saturday, August 17, 13
  • 59. Upgrading PHP Saturday, August 17, 13
  • 60. PHP 5.4 vs. 5.3 - CPU PHP 5.3 PHP 5.4 Saturday, August 17, 13
  • 61. PHP 5.4 vs. 5.3 - Memory PHP 5.3 PHP 5.4 Saturday, August 17, 13
  • 62. Saturday, August 17, 13
  • 63. Understand Framework Overhead Saturday, August 17, 13
  • 64. http://systemsarchitect.net/performance-benchmark-of-popular-php-frameworks/ Saturday, August 17, 13
  • 65. Saturday, August 17, 13
  • 66. Saturday, August 17, 13
  • 67. Saturday, August 17, 13
  • 68. Frameworks are Harder to Scale Saturday, August 17, 13
  • 69. Saturday, August 17, 13
  • 70. Memcached Saturday, August 17, 13
  • 71. Memcached • Roughly same hardware as webs Saturday, August 17, 13
  • 72. Memcached • Roughly same hardware as webs • More memory (48GB) Saturday, August 17, 13
  • 73. Memcached • Roughly same hardware as webs • More memory (48GB) • Shard keys across servers Saturday, August 17, 13
  • 74. Memcached • Roughly same hardware as webs • More memory (48GB) • Shard keys across servers • mctop - https://github.com/etsy/mctop Saturday, August 17, 13
  • 75. Saturday, August 17, 13
  • 76. MySQL Saturday, August 17, 13
  • 77. MySQL • MySQL 5.5 (http://jkle.in/nephp) Saturday, August 17, 13
  • 78. MySQL • MySQL 5.5 (http://jkle.in/nephp) • ~20 master/master pairs Saturday, August 17, 13
  • 79. MySQL • MySQL 5.5 (http://jkle.in/nephp) • ~20 master/master pairs • innodb Saturday, August 17, 13
  • 80. MySQL • MySQL 5.5 (http://jkle.in/nephp) • ~20 master/master pairs • innodb • thread_cache=800 Saturday, August 17, 13
  • 81. MySQL • MySQL 5.5 (http://jkle.in/nephp) • ~20 master/master pairs • innodb • thread_cache=800 • max_connections=2500 Saturday, August 17, 13
  • 82. MySQL • MySQL 5.5 (http://jkle.in/nephp) • ~20 master/master pairs • innodb • thread_cache=800 • max_connections=2500 • [Almost] no joins Saturday, August 17, 13
  • 83. Theoretically we can serve ~9000 req/sec Saturday, August 17, 13
  • 84. Saturday, August 17, 13
  • 85. Static Content Saturday, August 17, 13
  • 86. Saturday, August 17, 13
  • 87. ~500 Million Images Saturday, August 17, 13
  • 88. Offloading Requests Saturday, August 17, 13
  • 89. Offloading Requests • Multiple CDNs Saturday, August 17, 13
  • 90. Offloading Requests • Multiple CDNs • Img hit rate > 90% Saturday, August 17, 13
  • 91. Offloading Requests • Multiple CDNs • Img hit rate > 90% • CSS/JS hit rate ~100% Saturday, August 17, 13
  • 92. Offloading Requests • Multiple CDNs • Img hit rate > 90% • CSS/JS hit rate ~100% • Edge serves ~7000 reqs/sec Saturday, August 17, 13
  • 93. What about misses? Saturday, August 17, 13
  • 94. Saturday, August 17, 13
  • 95. Change Management Saturday, August 17, 13
  • 96. Feature Flags Saturday, August 17, 13
  • 97. Saturday, August 17, 13
  • 98. // Owner: Username $server_config['foo']['bar'] = array( 'users' => array('jsmith', 'jdoe'), ); // Owner: Username $server_config['baz'] = [ 'enabled' => [ 'variant1' => 0, 'variant2' => 0, 'variant3' => 50 ], ]; // Owner: Username $server_config['qux'] = array( 'enabled' => 100, 'data' => array( 'somedata' ) ); Saturday, August 17, 13
  • 99. if (Feature::isEnabled('foo.bar')) { // Code } Saturday, August 17, 13
  • 100. Ramp-up: 1%, 5%, 25%, 100% Saturday, August 17, 13
  • 101. Deployinator Saturday, August 17, 13
  • 102. Saturday, August 17, 13
  • 103. Schemanator Saturday, August 17, 13
  • 104. Saturday, August 17, 13
  • 105. Recent Improvements Saturday, August 17, 13
  • 106. Saturday, August 17, 13
  • 107. Sandy Bridge Saturday, August 17, 13
  • 108. Baseline Performance (median) Saturday, August 17, 13
  • 109. Small code change Baseline Performance (median) Saturday, August 17, 13
  • 110. Small code change PHP 5.4 Baseline Performance (median) Saturday, August 17, 13
  • 111. Small code change PHP 5.4 Disabled Hyperthreading Baseline Performance (median) Saturday, August 17, 13
  • 112. Challenges Saturday, August 17, 13
  • 113. Saturday, August 17, 13
  • 114. Options Saturday, August 17, 13
  • 115. Options • Scale Horizontally (more shards) Saturday, August 17, 13
  • 116. Options • Scale Horizontally (more shards) • Scale Vertically (SSD, bigger drives) Saturday, August 17, 13
  • 117. Options • Scale Horizontally (more shards) • Scale Vertically (SSD, bigger drives) • Change architecture Saturday, August 17, 13
  • 118. Saturday, August 17, 13
  • 119. Architecture Changes Saturday, August 17, 13
  • 120. Architecture Changes • Ongoing conversation Saturday, August 17, 13
  • 121. Architecture Changes • Ongoing conversation • Separate logical/physical shards Saturday, August 17, 13
  • 122. Architecture Changes • Ongoing conversation • Separate logical/physical shards • Complex Saturday, August 17, 13
  • 123. Architecture Changes • Ongoing conversation • Separate logical/physical shards • Complex • Server failure Saturday, August 17, 13
  • 124. Architecture Changes • Ongoing conversation • Separate logical/physical shards • Complex • Server failure • Migrating data Saturday, August 17, 13
  • 125. Architecture Changes • Ongoing conversation • Separate logical/physical shards • Complex • Server failure • Migrating data • Consistency Saturday, August 17, 13
  • 126. Monitoring Saturday, August 17, 13
  • 127. Graphite Saturday, August 17, 13
  • 128. Stacked Search Timers Saturday, August 17, 13
  • 129. Ganglia Saturday, August 17, 13
  • 130. Saturday, August 17, 13
  • 131. Nagios Saturday, August 17, 13
  • 132. Saturday, August 17, 13
  • 133. Saturday, August 17, 13
  • 134. Takeaways Saturday, August 17, 13
  • 135. K.I.S.S. Saturday, August 17, 13
  • 136. Use Proven Technologies Saturday, August 17, 13
  • 137. Saturday, August 17, 13
  • 138. Understand Your Stack Saturday, August 17, 13
  • 139. Measure Everything Saturday, August 17, 13
  • 140. Work at Etsy ;-) Saturday, August 17, 13
  • 141. Connect http://www.meetup.com/Web-Performance-Boston/ www.etsy.com/careers jonathan@etsy.com @jonathanklein Saturday, August 17, 13