Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Oh Crap, My Code is Slow Chris Tankersley @dragonmantank Madison PHP Conference, September-October 2016 Madison PHP Confer...
Madison PHP Conference, September-October 2016 2 1.79 MHz 8-bit Processor 128K RAM 640x192 max resolution 64 color palette...
Madison PHP Conference, September-October 2016 3 520 Mhz Apple S1 512MB RAM 390x312 resolution (~303 ppi density) 16 milli...
Madison PHP Conference, September-October 2016 4 32 Cores 512GB RAM 1-10Gbps NICs 10 Terabyte FusionIO Cards
Madison PHP Conference, September-October 2016 5
“premature optimization is the root of all evil.” Donald Knuth, “Structured Programming With Go To Statements” Madison PHP...
Madison PHP Conference, September-October 2016 7 www.phpbench.com
It Doesn’t Matter Madison PHP Conference, September-October 2016 8
We Are the 3% Madison PHP Conference, September-October 2016 9
Who is at fault? • 3rd Party Connections • I/O Performance • Database Madison PHP Conference, September-October 2016 10
Madison PHP Conference, September-October 2016 11
Madison PHP Conference, September-October 2016 12
Madison PHP Conference, September-October 2016 13
4 Hours • Cache Ad Campaign Data • Cache Analytics Data • Run Numbers • Sync Products Madison PHP Conference, September-Oc...
4 Hours • Cache Ad Campaign Data • Cache Analytics Data • Run Numbers • Sync Products Madison PHP Conference, September-Oc...
The Problems Madison PHP Conference, September-October 2016 16
Running Numbers was heavy • Primary server would spike under load • Secondary servers would get out of sync and go into “r...
Madison PHP Conference, September-October 2016 18 0 500 1000 1500 2000 2500 3000 5:20am 5:25am 5:30am 5:35am 5:40am 5:45am...
Madison PHP Conference, September-October 2016 19 0 1000 2000 3000 4000 5000 6000 5:20am 5:25am 5:30am 5:35am 5:40am 5:45a...
Product Sync was Slow • Just took hours to run Madison PHP Conference, September-October 2016 20
Madison PHP Conference, September-October 2016 21
Product Sync Madison PHP Conference, September-October 2016 22
Just Start Logging • Add DEBUG log messages with timestamps • Where is it slow? Madison PHP Conference, September-October ...
Seldaek/monolog use MonologLogger; use MonologHandlerStreamHandler; // create a log channel $log = new Logger(‘job_debug')...
Culprits Madison PHP Conference, September-October 2016 25
What did we find? • All calls to Product API had to be full sets, couldn’t subset • Calls to Product API were slow, but no...
Our Workflow // Original Workflow Get Page X from API For Each Product: Extract Data from XML Transmogrify the Data into a...
Solution – Out of Band Processing // Original Workflow Get Page X from API For Each Product: Extract Data from XML Transmo...
Solution – Out of Band Processing // Job 1 - Cache Product API Calls Get Page X…X+10 from API Cache XML to Database If No ...
Solution – Out of Band Processing // Job 2 – Insert Products Get Page X…X+10 from DB For Each Product: Extract Data from X...
Run Totals Madison PHP Conference, September-October 2016 31
Madison PHP Conference, September-October 2016 32
Background • Was originally PHP • Turned into a MongoDB Script because it was too slow Madison PHP Conference, September-O...
Madison PHP Conference, September-October 2016 34 0 500 1000 1500 2000 2500 3000 5:20am 5:25am 5:30am 5:35am 5:40am 5:45am...
Madison PHP Conference, September-October 2016 35
What did we find? Madison PHP Conference, September-October 2016 36
Check the Server Metrics Madison PHP Conference, September-October 2016 37 https://aws.amazon.com/blogs/aws/new-cloudwatch...
Suspect Madison PHP Conference, September-October 2016 38
Our Solution – Throw Hardware At It Madison PHP Conference, September-October 2016 39
Our Solution – Throw Hardware At It • Increased IOPs on the SSD’s • Larger Instances on AWS Madison PHP Conference, Septem...
Our Solution – Move out of MongoDB • Rewrite the script back into PHP • Run in our worker system Madison PHP Conference, S...
The Result Madison PHP Conference, September-October 2016 42
The New Bug Madison PHP Conference, September-October 2016 43 It took 5 hours to run
Suspect Madison PHP Conference, September-October 2016 44
Madison PHP Conference, September-October 2016 45
Madison PHP Conference, September-October 2016 46
What did we find? Madison PHP Conference, September-October 2016 47
Code Profiling Madison PHP Conference, September-October 2016 48
xhprof • Low-cost dynamic analysis for PHP • PHP Extension • Store results in a DB • Has a pretty good GUI • https://www.d...
Pretty Graphs Madison PHP Conference, September-October 2016 50
Useful Metrics Madison PHP Conference, September-October 2016 51
What we find? • Hydrating objects was expensive • We were doing deep hydration, resulting in extra DB and hydration calls ...
The result? Madison PHP Conference, September-October 2016 53 It brought it down to around 3.5 hours
Valgrind • General programming tool for checking memory debugging, memory leaks, and code profiling • Supported by xdebug ...
Enable it in xdebug zend_extension=/usr/lib/php/20151012/xdebug.so xdebug.profiler_enable=1 xdebug.profiler_output_dir=/va...
Function Calls and Code Flow Madison PHP Conference, September-October 2016 56
What we find? • We were looping a lot • We were looping big loops inside small loops • We were looping through a lot of th...
The Result – Reduce the Looping Madison PHP Conference, September-October 2016 58 Runtime was reduced to 30 minutes
Tips for Slow Code • Use Monolog to add Debugging messages • Use xhprof to profile “live” code • Use xdebug and Valgrind t...
Thank You! • https://github.com/dragonmantank • Author of “Docker for Developers” • https://leanpub.com/dockerfordevs • ht...
Credits • Slide 13 – Andrei.D40 – Stacks of Books, Flickr • Slide 34 – Upper Snake River Valley Historical Society – 3339 ...
Upcoming SlideShare
Loading in …5
×

Oh Crap, My Code is Slow - Madison PHP 2016

210 views

Published on

There are many excuses that developers use for inefficient code - CPU and RAM is cheap these days, or PHP is by default a slow language. These are just a few of those. What happens when your code is actually to slow to scale? Most of us will not deal with things on the scale of Facebook or Google, but there will come a time where we will need to figure out why code is slow. Thankfully there are are many different tools to help us out and properly optimize our code for those times when we need to dig deep into our code.

Published in: Technology
no profile picture user

  • Be the first to comment

  • Be the first to like this

Oh Crap, My Code is Slow - Madison PHP 2016

  1. 1. Oh Crap, My Code is Slow Chris Tankersley @dragonmantank Madison PHP Conference, September-October 2016 Madison PHP Conference, September-October 2016 1
  2. 2. Madison PHP Conference, September-October 2016 2 1.79 MHz 8-bit Processor 128K RAM 640x192 max resolution 64 color palette RS-232 Serial Port Cartridge Bay 2 Joystick Ports Disk Extended Color Basic 2.1
  3. 3. Madison PHP Conference, September-October 2016 3 520 Mhz Apple S1 512MB RAM 390x312 resolution (~303 ppi density) 16 million colors WatchOS
  4. 4. Madison PHP Conference, September-October 2016 4 32 Cores 512GB RAM 1-10Gbps NICs 10 Terabyte FusionIO Cards
  5. 5. Madison PHP Conference, September-October 2016 5
  6. 6. “premature optimization is the root of all evil.” Donald Knuth, “Structured Programming With Go To Statements” Madison PHP Conference, September-October 2016 6
  7. 7. Madison PHP Conference, September-October 2016 7 www.phpbench.com
  8. 8. It Doesn’t Matter Madison PHP Conference, September-October 2016 8
  9. 9. We Are the 3% Madison PHP Conference, September-October 2016 9
  10. 10. Who is at fault? • 3rd Party Connections • I/O Performance • Database Madison PHP Conference, September-October 2016 10
  11. 11. Madison PHP Conference, September-October 2016 11
  12. 12. Madison PHP Conference, September-October 2016 12
  13. 13. Madison PHP Conference, September-October 2016 13
  14. 14. 4 Hours • Cache Ad Campaign Data • Cache Analytics Data • Run Numbers • Sync Products Madison PHP Conference, September-October 2016 14
  15. 15. 4 Hours • Cache Ad Campaign Data • Cache Analytics Data • Run Numbers • Sync Products Madison PHP Conference, September-October 2016 15
  16. 16. The Problems Madison PHP Conference, September-October 2016 16
  17. 17. Running Numbers was heavy • Primary server would spike under load • Secondary servers would get out of sync and go into “rollback” Madison PHP Conference, September-October 2016 17
  18. 18. Madison PHP Conference, September-October 2016 18 0 500 1000 1500 2000 2500 3000 5:20am 5:25am 5:30am 5:35am 5:40am 5:45am Replication Lag Lag in Seconds
  19. 19. Madison PHP Conference, September-October 2016 19 0 1000 2000 3000 4000 5000 6000 5:20am 5:25am 5:30am 5:35am 5:40am 5:45am Replication Lag Lag in ms
  20. 20. Product Sync was Slow • Just took hours to run Madison PHP Conference, September-October 2016 20
  21. 21. Madison PHP Conference, September-October 2016 21
  22. 22. Product Sync Madison PHP Conference, September-October 2016 22
  23. 23. Just Start Logging • Add DEBUG log messages with timestamps • Where is it slow? Madison PHP Conference, September-October 2016 23
  24. 24. Seldaek/monolog use MonologLogger; use MonologHandlerStreamHandler; // create a log channel $log = new Logger(‘job_debug'); $log->pushHandler(new StreamHandler('path/to/your.log', Logger::DEBUG)); // add records to the log $log->debug(date(‘Y-m-d H:i:s’) . ‘ – Contacted API’); // Do our business logic $log->debug(date(‘Y-m-d H:i:s’) . ‘ – Finished with page’); Madison PHP Conference, September-October 2016 24
  25. 25. Culprits Madison PHP Conference, September-October 2016 25
  26. 26. What did we find? • All calls to Product API had to be full sets, couldn’t subset • Calls to Product API were slow, but not horrid • Generating and inserting the Products were slow due to business logic • Blocked Operations: • Getting next page from API • Processing products Madison PHP Conference, September-October 2016 26
  27. 27. Our Workflow // Original Workflow Get Page X from API For Each Product: Extract Data from XML Transmogrify the Data into a Product Object Save Object to DB If No Next Page: Break Else: Page++ Continue Madison PHP Conference, September-October 2016 27
  28. 28. Solution – Out of Band Processing // Original Workflow Get Page X from API For Each Product: Extract Data from XML Transmogrify the Data into a Product Object Save Object to DB If No Next Page: Break Else: Page++ Continue Madison PHP Conference, September-October 2016 28
  29. 29. Solution – Out of Band Processing // Job 1 - Cache Product API Calls Get Page X…X+10 from API Cache XML to Database If No Next Page: Break Else: Page++ Continue Call Job 2 Respawn Job Madison PHP Conference, September-October 2016 29
  30. 30. Solution – Out of Band Processing // Job 2 – Insert Products Get Page X…X+10 from DB For Each Product: Extract Data from XML Transmogrify the Data into a Product Object Save Object to DB If No Next Page: Break Else: Page++ Continue Madison PHP Conference, September-October 2016 30
  31. 31. Run Totals Madison PHP Conference, September-October 2016 31
  32. 32. Madison PHP Conference, September-October 2016 32
  33. 33. Background • Was originally PHP • Turned into a MongoDB Script because it was too slow Madison PHP Conference, September-October 2016 33
  34. 34. Madison PHP Conference, September-October 2016 34 0 500 1000 1500 2000 2500 3000 5:20am 5:25am 5:30am 5:35am 5:40am 5:45am Replication Lag Lag in ms
  35. 35. Madison PHP Conference, September-October 2016 35
  36. 36. What did we find? Madison PHP Conference, September-October 2016 36
  37. 37. Check the Server Metrics Madison PHP Conference, September-October 2016 37 https://aws.amazon.com/blogs/aws/new-cloudwatch- metrics-for-amazon-ebs-volumes/
  38. 38. Suspect Madison PHP Conference, September-October 2016 38
  39. 39. Our Solution – Throw Hardware At It Madison PHP Conference, September-October 2016 39
  40. 40. Our Solution – Throw Hardware At It • Increased IOPs on the SSD’s • Larger Instances on AWS Madison PHP Conference, September-October 2016 40
  41. 41. Our Solution – Move out of MongoDB • Rewrite the script back into PHP • Run in our worker system Madison PHP Conference, September-October 2016 41
  42. 42. The Result Madison PHP Conference, September-October 2016 42
  43. 43. The New Bug Madison PHP Conference, September-October 2016 43 It took 5 hours to run
  44. 44. Suspect Madison PHP Conference, September-October 2016 44
  45. 45. Madison PHP Conference, September-October 2016 45
  46. 46. Madison PHP Conference, September-October 2016 46
  47. 47. What did we find? Madison PHP Conference, September-October 2016 47
  48. 48. Code Profiling Madison PHP Conference, September-October 2016 48
  49. 49. xhprof • Low-cost dynamic analysis for PHP • PHP Extension • Store results in a DB • Has a pretty good GUI • https://www.digitalocean.com/community/tutorials/how-to-set-up- xhprof-and-xhgui-for-profiling-php-applications-on-ubuntu-14-04 Madison PHP Conference, September-October 2016 49
  50. 50. Pretty Graphs Madison PHP Conference, September-October 2016 50
  51. 51. Useful Metrics Madison PHP Conference, September-October 2016 51
  52. 52. What we find? • Hydrating objects was expensive • We were doing deep hydration, resulting in extra DB and hydration calls • We had authentication checking happening in a loop, due to bad logging code Madison PHP Conference, September-October 2016 52
  53. 53. The result? Madison PHP Conference, September-October 2016 53 It brought it down to around 3.5 hours
  54. 54. Valgrind • General programming tool for checking memory debugging, memory leaks, and code profiling • Supported by xdebug • KCacheGrind/QCacheGrind to view output Madison PHP Conference, September-October 2016 54
  55. 55. Enable it in xdebug zend_extension=/usr/lib/php/20151012/xdebug.so xdebug.profiler_enable=1 xdebug.profiler_output_dir=/var/www/tests/xdebug Madison PHP Conference, September-October 2016 55
  56. 56. Function Calls and Code Flow Madison PHP Conference, September-October 2016 56
  57. 57. What we find? • We were looping a lot • We were looping big loops inside small loops • We were looping through a lot of the same data multiple times Madison PHP Conference, September-October 2016 57
  58. 58. The Result – Reduce the Looping Madison PHP Conference, September-October 2016 58 Runtime was reduced to 30 minutes
  59. 59. Tips for Slow Code • Use Monolog to add Debugging messages • Use xhprof to profile “live” code • Use xdebug and Valgrind to get deeper profiling Madison PHP Conference, September-October 2016 59
  60. 60. Thank You! • https://github.com/dragonmantank • Author of “Docker for Developers” • https://leanpub.com/dockerfordevs • http://ctankersley.com • chris@ctankersley.com • @dragonmantank Madison PHP Conference, September-October 2016 60
  61. 61. Credits • Slide 13 – Andrei.D40 – Stacks of Books, Flickr • Slide 34 – Upper Snake River Valley Historical Society – 3339 loggin, Flickr Madison PHP Conference, September-October 2016 61

×