Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

You Were Lied To About Optimization

122 views

Published on

Many know of the famous quote, "Premature optimization is the root of all evil," but most people do not know the full quote or understand the context in which optimization is considered evil. As with anything in programming optimization is evil, maybe. Stop using excuses for slow code, and start to think about the places and tools that you can use to optimize. Thankfully there are are many different tools like xhprof, Valgrind, and others to help us out and properly optimize our code for those times when we need to dig deep into our code.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

You Were Lied To About Optimization

  1. 1. You Were Lied To About Optimization Chris Tankersley @dragonmantank 1
  2. 2. Madison PHP Conference, September-October 2016 2 1.79 MHz 8-bit Processor 128K RAM 640x192 max resolution 64 color palette RS-232 Serial Port Cartridge Bay 2 Joystick Ports Disk Extended Color Basic 2.1
  3. 3. 3 520 Mhz Apple S1 512MB RAM 390x312 resolution (~303 ppi density) 16 million colors WatchOS
  4. 4. 4 32 Cores 512GB RAM 1-10Gbps NICs 10 Terabyte FusionIO Cards
  5. 5. Madison PHP Conference, September-October 2016 5
  6. 6. “premature optimization is the root of all evil.” Donald Knuth, “Structured Programming With Go To Statements” 6
  7. 7. 7 www.phpbench.com
  8. 8. It Doesn’t Matter 8
  9. 9. We Are the 3% 9
  10. 10. Who is at fault? • 3rd Party Connections • I/O Performance • Database 10
  11. 11. 11
  12. 12. 12
  13. 13. 13
  14. 14. 4 Hours • Cache Ad Campaign Data • Cache Analytics Data • Run Numbers • Sync Products 14
  15. 15. 4 Hours • Cache Ad Campaign Data • Cache Analytics Data • Run Numbers • Sync Products 15
  16. 16. The Problems 16
  17. 17. Running Numbers was heavy • Primary server would spike under load • Secondary servers would get out of sync and go into “rollback” 17
  18. 18. 18 5:20am 5:25am 5:30am 5:35am 5:40am 5:45am 0 500 1000 1500 2000 2500 3000 Replication Lag Lag in Seconds
  19. 19. 19 5:20am 5:25am 5:30am 5:35am 5:40am 5:45am 0 1000 2000 3000 4000 5000 6000 Replication Lag Lag in ms
  20. 20. Product Sync was Slow • Just took hours to run 20
  21. 21. 21
  22. 22. Product Sync 22
  23. 23. Just Start Logging • Add DEBUG log messages with timestamps • Where is it slow? 23
  24. 24. Seldaek/monolog use MonologLogger; use MonologHandlerStreamHandler; // create a log channel $log = new Logger(‘job_debug'); $log->pushHandler(new StreamHandler('path/to/your.log', Logger::DEBUG)); // add records to the log $log->debug(date(‘Y-m-d H:i:s’) . ‘ – Contacted API’); // Do our business logic $log->debug(date(‘Y-m-d H:i:s’) . ‘ – Finished with page’); 24
  25. 25. Culprits 25
  26. 26. What did we fnd? • All calls to Product API had to be full sets, couldn’t subset • Calls to Product API were slow, but not horrid • Generating and inserting the Products were slow due to business logic • Blocked Operations: • Getting next page from API • Processing products 26
  27. 27. Our Workfow // Original Workfow Get Page X from API For Each Product: Extract Data from XML Transmogrify the Data into a Product Object Save Object to DB If No Next Page: Break Else: Page++ Continue 27
  28. 28. Solution – Out of Band Processing // Original Workfow Get Page X from API For Each Product: Extract Data from XML Transmogrify the Data into a Product Object Save Object to DB If No Next Page: Break Else: Page++ Continue 28
  29. 29. Solution – Out of Band Processing // Job 1 - Cache Product API Calls Get Page X…X+10 from API Cache XML to Database If No Next Page: Break Else: Page++ Continue Call Job 2 Respawn Job 29
  30. 30. Solution – Out of Band Processing // Job 2 – Insert Products Get Page X…X+10 from DB For Each Product: Extract Data from XML Transmogrify the Data into a Product Object Save Object to DB If No Next Page: Break Else: Page++ Continue 30
  31. 31. Run Totals 31
  32. 32. Madison PHP Conference, September-October 2016 32
  33. 33. Background • Was originally PHP • Turned into a MongoDB Script because it was too slow 33
  34. 34. 34 5:20am 5:25am 5:30am 5:35am 5:40am 5:45am 0 500 1000 1500 2000 2500 3000 Replication Lag Lag in ms
  35. 35. 35
  36. 36. What did we fnd? 36
  37. 37. Check the Server Metrics 37 https://aws.amazon.com/blogs/aws/new- cloudwatch-metrics-for-amazon-ebs-volumes/
  38. 38. Suspect 38
  39. 39. Our Solution – Throw Hardware At It 39
  40. 40. Our Solution – Throw Hardware At It • Increased IOPs on the SSD’s • Larger Instances on AWS 40
  41. 41. Our Solution – Move out of MongoDB • Rewrite the script back into PHP • Run in our worker system 41
  42. 42. The Result 42
  43. 43. The New Bug 43 It took 5 hours to run
  44. 44. Suspect 44
  45. 45. 45
  46. 46. 46
  47. 47. What did we fnd? 47
  48. 48. Code Profling 48
  49. 49. Xhprof/Tideways • Low performance cost dynamic analysis for PHP • PHP Extension • Store results in a DB • Has a pretty good GUI • https://www.digitalocean.com/community/tutorials/how-to -set-up-xhprof-and-xhgui-for-profling-php-applications -on-ubuntu-14-04 49
  50. 50. Pretty Graphs 50
  51. 51. Useful Metrics 51
  52. 52. What we fnd? • Hydrating objects was expensive • We were doing deep hydration, resulting in extra DB and hydration calls • We had authentication checking happening in a loop, due to bad logging code 52
  53. 53. The result? 53 It brought it down to around 3.5 hours
  54. 54. Valgrind • General programming tool for checking memory debugging, memory leaks, and code profling • Supported by xdebug • KCacheGrind/QCacheGrind to view output 54
  55. 55. Enable it in xdebug zend_extension=/usr/lib/php/20151012/xdebug.so xdebug.profler_enable=1 xdebug.profler_output_dir=/var/www/tests/xdebug 55
  56. 56. Function Calls and Code Flow 56
  57. 57. What we fnd? • We were looping a lot • We were looping big loops inside small loops • We were looping through a lot of the same data multiple times 57
  58. 58. The Result – Reduce the Looping 58 Runtime was reduced to 30 minutes
  59. 59. Tips for Slow Code • Use Monolog to add Debugging messages • Use xhprof/Tideways to profle “live” code • Use xdebug and Valgrind to get deeper profling 59
  60. 60. Thank You! • https://github.com/dragonmantank • Author of “Docker for Developers” • https://leanpub.com/dockerfordevs • http://ctankersley.com • chris@ctankersley.com • @dragonmantank 60
  61. 61. Credits • Slide 13 – Andrei.D40 – Stacks of Books, Flickr • Slide 34 – Upper Snake River Valley Historical Society – 3339 loggin, Flickr 61

×